Using ShadowSpawn and Robocopy to copy in use files over a low bandwidth connection.

I had a Mail Server that needed to be moved from one location to another without interrupting service. The new server and the old server are in two different locations separated by many miles. The only connection between the two is a 4mb synchronous Internet connection which is also needed to serve web sites, applications and the like. So I was limited to about 1mb of bandwidth to synchronize these systems.

The job basically entails synchronization of the old servers files with the new servers files and required that 20gb of data be sent. This much data over a 4mb pipe, with no bandwidth control, would lead to a monopolizing of the bandwidth essentially shutting the pipe down until the copy is complete. This is complicated by the fact that many of these files will be in use.

Robocopy is usually my tool of choice. It comes with Windows and there is not much it cannot do, except backup files that are in use. Ouch! Now what? If only I could take advantage of Volume Shadow Copy! Then along comes ShadowSpawn by Craig Andera. You may know his work on Hobocopy, which uses VSS to copy files , but has some limitations. Well, ShadowSpawn was meant to be used with Robocopy and it does so awesomely!

Since this was my initial synchronization, I wanted to replace all files, including any that would be in use, so I stopped all services and applications on the destination computer. Future synchronizations will only need to get user data and emails that have changed so I will make adjustments in my robocopy commands to skip in use files in the future and there will be no need for VSS to do those, so that discussion is outside of the context of this article.

So, On with job…

On the source server I needed to install, or in this case simply extract and copy the ShadowSpawn files into the Windows folder. I do this so that the files will be in the path. I also had to install the Visual C++ run-time, which is very conveniently located on the same download page.  I also mapped the destination root as drive Z:, I could have simply mapped the destination folder, but I’m more comfortable this way.

Once installed I needed to build a command that will do the job in a way that used ShadowSpawn to initiate a Volume ShadowCopy, Create a drive letter and then hand the job off to RoboCopy to complete the file copy operation using on 1mb of bandwidth. Completing this means we need an understanding of how robocopy uses the /IPG (Inter Packet Gap) switch.  Here is an excerpt from the Wikipedia Robocopy article:

Bandwidth throttling

Robocopy’s “inter-packet gap” (IPG) option allows some control over the network bandwidth used in a session. In theory, the following formula expresses the delay (D, in milliseconds) required to simulate a desired bandwidth (BD, in kilobits per second), over a network link with an available bandwidth of BA kbps:

D = {B_A - B_D \over B_A \times B_D} \times 512 \times 1000

In practice however, some experimentation is usually required to find a suitable delay, due to factors such as the nature and volume of other traffic on the network. 

Basically, do the math and figure out what number to plug in to the /IPG switch.  For my purposes /ipg:375 was worked out based on a desired bandwidth (Bd) of 1mb (1024 bits) over a 4mb (4096bits) connection (Ba).

So here’s how I structured the command:
shadowspawn C:\mailserver Q: robocopy q:\ z:\mailserver /B /MIR /IPG:375

This command tells ShadowSpawn to Send a VSS copy of C:\Mailserver to Drive Letter Q: then launches RoboCopy which copies the files from the VSS copy (Q: drive) to the destination in Backup (Backup Mode ensures that you do not get access denied errors) and Mirror Mode.

The copy operation took a long time to complete, but I was able to preserve enough bandwidth for our systems to function and still get the job done.

This procedure could be modified for use in my backup script to allow for a full system backup or simply as a workaround for in use files.