I'm currently working on a desktop application that may have a future need to download data files from the web.  Some of these files will be quite large, over 1GB so its very possible that a lengthy download will be interrupted. In this post I present some code to download a file and resume previously interrupted download.

The Range header defined in HTTP/1.1 allows a client to request only part of a file. Not all servers will honor this request for every file. For example during testing I found that some servers will respond with the entire file for very small files but will honor the range request for larger files.

The StartDownload() method is the primary function of the class.  It first opens an output stream to the destination file and saves the current length of the stream (line 26). This length indicates the starting position to begin downloading.  If the file doesn't exist this will of course be 0 but a partially (or completely) downloaded file will return some length greater than 0.

Next the code sends a request to the server to get the length of the file (line 27).  The GetContentLength() function (lines 54-61) requests the file and reads the response headers but it doesn't actually fetch any other data from the server.

The code then compares the current length of the output stream and the expected content length to decide whether it needs to download the file or resume a partially completed download.

The OpenReadStream() function (lines 63-73) uses the starting position and content length to create an HttpWebRequest and open a stream to the file. The key part is line 66 where it specifies the range to download. This function also verifies that the response headers indicate that the server will honor the request for a partial download (if the start position is greater than 0). If the response from the server is going to send the entire file again the code rewinds the output stream to the start position.

Finally the Copy() method copies a block at a time from the server to the local output stream.

To test this code I pointed it towards the Microsoft .NET 3.5 redistributable installer which is under 3MB. You can run it and repeatedly hit Ctrl+C to kill it before it finishes or set a breakpoint in the Copy() method and repeatedly kill it there. Eventually the code will download the entire file.  But how do we know its correct?  Just download the file manually and compare the file check sums.

The code needs more error checking (not shown to keep it small) and a progress event raised in the Copy() function would be a nice touch, but otherwise the complete code sample below demonstrates how to download a file and resume a partially completed download.

   1: using System;

   2: using System.Net;

   3: using System.IO;

   4:  

   5: namespace RestartableDownload

   6: {

   7:     internal class RestartableDownload

   8:     {

   9:         private Uri _uri;

  10:         private string _destFile;

  11:         private FileStream _writeStream;

  12:         private Stream _readStream;

  13:  

  14:         internal RestartableDownload(string uri, string destinationFile)

  15:         {

  16:             _uri = new Uri(uri);

  17:             _destFile = destinationFile;

  18:         }

  19:  

  20:         internal void StartDownload()

  21:         {

  22:             if (_uri.Scheme.Equals("http"))

  23:             {

  24:                 try

  25:                 {

  26:                     long start = OpenWriteStream();

  27:                     long length = GetContentLength();

  28:                     if (start < length)

  29:                     {

  30:                         OpenReadStream(start, length);

  31:                         Copy();

  32:                     }

  33:                 }

  34:                 catch (System.Exception ex)

  35:                 {

  36:                     Console.WriteLine(ex.ToString());

  37:                 }

  38:                 finally

  39:                 {

  40:                     if (_writeStream != null)

  41:                         _writeStream.Close();

  42:                     if (_readStream != null)

  43:                         _readStream.Close();

  44:                 }

  45:             }

  46:         }

  47:  

  48:         private long OpenWriteStream()

  49:         {

  50:             _writeStream = new FileStream(_destFile, FileMode.Append, FileAccess.Write);

  51:             return _writeStream.Length;

  52:         }

  53:  

  54:         private long GetContentLength()

  55:         {

  56:             HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_uri);

  57:             HttpWebResponse response = (HttpWebResponse)request.GetResponse();

  58:             long length = response.ContentLength;

  59:             response.Close();

  60:             return length;

  61:         }

  62:  

  63:         private void OpenReadStream(long start, long length)

  64:         {

  65:             HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_uri);

  66:             request.AddRange((int)start, (int)length);

  67:             HttpWebResponse response = (HttpWebResponse)request.GetResponse();

  68:             if (response.ContentLength == length)

  69:             {

  70:                 _writeStream.Seek(0, SeekOrigin.Begin);

  71:             }

  72:             _readStream = response.GetResponseStream();

  73:         }

  74:  

  75:         private void Copy()

  76:         {

  77:             byte[] buffer = new byte[1024];

  78:             int count = _readStream.Read(buffer, 0, buffer.Length);

  79:             while (count > 0)

  80:             {

  81:                 _writeStream.Write(buffer, 0, count);

  82:                 _writeStream.Flush();

  83:                 count = _readStream.Read(buffer, 0, buffer.Length);

  84:             }

  85:         }

  86:     }

  87:  

  88:     class Program

  89:     {

  90:         static void Main(string[] args)

  91:         {

  92:             string uri = @"http://download.microsoft.com/download/7/0/3/" + 

  93:                 "703455ee-a747-4cc8-bd3e-98a615c3aedb/dotNetFx35setup.exe";

  94:             string output = @"downloaded-dotNetFx35setup.exe";

  95:             

  96:             RestartableDownload download = new RestartableDownload(uri, output);

  97:             download.StartDownload();

  98:         }

  99:     }

 100: }

I have been reading through Robert L. Glass's book Facts and Fallacies of Software Engineering.  In this book Glass presents a number of fundamental truths about software development.

While reading the book I am continually struck by how much historical perspective it offers. The software development industry is young but it has a history. Its helpful to have a good historical perspective to avoid repeating past mistakes but also to moderate expectations for the future.

For example, today there are scores of websites devoted to sharing small bits of useful code.  Individual developers post everything from small code fragments to large sample applications and explanatory articles. One might think this is a new phenomenon made possible only by the emergence of the world wide web but in fact developers have been sharing code since the beginning and we can expect this will continue for years into the future. To illustrate this consider Fact #15 in the book about code reuse-in-the-small. Glass claims the very first reusable code library emerged in the 1950's and consisted of user contributed subroutines and a catalog of available code. To add more reference points to the historical timeline recall that Grace Hopper wrote the first compiler in 1952, and Fortran, Lisp, and COBOL were all developed later in the same decade.

Glass's conclusion of Fact #15 is that code reuse-in-the-small is a solved problem.  I certainly agree and also think it shows sharing code is natural.  Today of course we have to respect copyright law and software patents because code has monetary value, but scores of websites devoted to sharing small bits of useful code complement that value because its not so much the code that is being shared but the knowledge and experience behind it.

Rather than writing my own blog software from scratch I'm using BlogEngine.NET. This seemed like a natural choice because its both open source and written in ASP.NET. This solves all the little problems that I would have to solve were I to write my own from scratch plus it leaves the door open for me to customize since its open source. The initial setup is very easy and it basically works out of the box once uploaded to a server and after configuring a couple things.  I'm using Windows Live Writer to author the posts and a code snippet plugin to format code.