Need some critique on my backup plans using robocopy

asusx205ta · 25 Dec 2016

I am trying to make better file backups. So in order to have different backup frequencies on different files, I decided to write scripts to run robocopy to perform the backups (I'm running these scripts manually using scheduler to display reminders).

Here is what I came up with:

I have a few locations that I need more frequent backups than the rest:
Documents
Dropbox
OneDrive
Google Drive

Less frequent backups:
Pictures
Videos

I already have a schedule for windows 7 style file backup including system image backup.

Due to the natures of different files, some files in the frequent backup folders are large and don't need as frequently backed up as others. An example would be books I downloaded. They are large files but I don't cry a river if I lose them and have to find them again online. I still want them under Documents for organizational purposes. Other files are less important, such as past versions of a document that I've published. They are only somewhat important since I can always go back to a certain version if someone asks about it. I zipped files like that using winrar.

So here is how I organized the backup jobs.

small backups: running every two weeks only on files that are newer and more important.
medium backups: running every 2 months
complete backups: running every 6 months to back up everything
All backups are kept until further decisions.

There will be two identical drives on a dual bay (not WD external dual drive with forced hardware AES encryption). One will be backing up files. The second one is synced with /MIR option of robocopy (haven't decided on how frequently).

Since all my files are within respective folders, I decided to write a robocopy job file for each of these folders and save the job file inside each folder. Say my Dropbox folder will contain small and medium backup job files. They will have specific files and folders they wish to exclude to make small and medium backups take a lot less space and time than complete backups.
Then in a central location, say my Dropbox folder, I will have three power shell scripts to run small, medium, and complete backups.

I think this is better than keeping one job file since the job file in my scenario goes with the folder the job is supposed to back up.

Any advice on my scenario? I've done some tests on these jobs and they seem to work. Robocopy is not very fast but at least I'm only backing up files I want backed up, instead of trying to back up everything every time.

Caledon Ken · 27 Dec 2016

My two cents even though we don't have pennies where I live anymore.

Windows 7 for Image - I think you will find the Internet laced with issues about how reliable this process is. Some have had good success and some have been left very disappointment. If you read the forums a lot you will see Macrium Reflect is recommended and used by a number of members. There is a free version and that is what is recommended. Search Macrium Reflect in the forums and you can judge for yourself.

Frequency - Everyone's frequency is going to be different. The question I ask myself is how much effort would it take to recreate and can I even recreate the data. Sounds like you create documents to publish. If you had a disk failure at a bad time could you make your deadline and how much effort would be involved. I run a Raid 1 for a little safety and copy out my weeks work to another drive. I can recreate most but simply don't have the time and storage is so cheap.

For Backups I call them all copies if they are are located together and are online. Fire, theft and water damage are hopefully localized but if they are all right there they are all gone. I like the fact you backup to offsite as well. Then there is the "Online" or connected. Some of today's Ransomware is aggressive and if you can readily access it so can it. I have two monthly backups of data that are offline. One I keep at my house, one I keep somewhere else. The one that is offsite is encrypted.

I also like the fact that you are keeping versions of your backups, then if one fails you can go back another generation. Many people miss this point.

On Robocopy I would ensure you exclude Junction points as that can cause the process to run long or should I say very long.

Good luck and thanks for sharing. Hopefully it will stir others into action.

Bree · 27 Dec 2016

Caledon Ken said:

Windows 7 for Image - I think you will find the Internet laced with issues about how reliable this process is. Some have had good success and some have been left very disappointment. If you read the forums a lot you will see Macrium Reflect is recommended and used by a number of members.

@asusx205ta, I am one who has persevered with the Backup & Restore (Windows 7) System Image, I use it regularly, both to save and restore images. Problems mostly involve not being able to find and recognise the image you want to restore. Moving the system image to another drive or renaming it will often 'break' the ability to restore it. The actual restore seems reliable though, once you can get it to see the image. I've found the two most reliable ways to use the built in 'system imaging' are to save the image to a second partition on the same drive, or save to multiple DVDs. Make the restore disk when offered at the end of making the image, this one has the best chance of 'seeing' the image.

Even I would recommend using something else - Macrium is more flexible and reliable, making it the popular choice on these boards.

Caledon Ken said:

On Robocopy I would ensure you exclude Junction points as that can cause the process to run long or should I say very long.

Not just long - it can produce infinitely long path names. I use a batch file based on Robocopy for monthly Full and weekly Incremental backups of my user files. Including junction points can under some circumstances create a copy of a folder that contains a copy of that same folder (that in turn contains a copy of itself, and so on...) until the path length gets too long! The option switch is /xjd

I would also recommend setting retries to a low number (I set /R:0). You could have a very long wait if a file cannot be copied! From Robocopy /? (my bold)

/R:n :: number of Retries on failed copies: default 1 million.
/W:n :: Wait time between retries: default is 30 seconds.

Samuria · 27 Dec 2016

If files are on Google drive one drive they are backed up straight away and safe on the cloud so there is no need to back them up

asusx205ta · 27 Dec 2016

Samuria said:

If files are on Google drive one drive they are backed up straight away and safe on the cloud so there is no need to back them up

I trust them just enough with my files but not with backing them up though.

asusx205ta · 27 Dec 2016

Thanks Caledon. I will investigate Macrium Reflect. Right now I use Paragon backup and recovery 2014 free edition, only occasionally. I only restored windows image a couple of times so I don't know how reliable they are. Once I thought my backup included an image but was only able to restore one image that was second most recent version for some reason. The result wasn't bad. I was migrating a system so I went right back with my screw driver, swapped in the original drive, did a system image and swapped in the new drive.

Yes, I keep all copies of my backup, only occasionally sorting them out to delete useless and large files, such as an install file that had duplicates in many backups. I figured, at the rate I am generating files, I can afford to add more drives if I need them, at a very reasonable cost.

I don't connect my backup drives to my computer unless I need to perform the backups. It saves from wear and tear and also prevents mishaps, such as ransomware or fire/theft. I also plan to only mirror to my second backup maybe every 6 months and store it in the bank for the rest of the time.

Thanks for the junction advice. I hit that roadblock right after I thought I figured out robocopy and started a small-scale test on my Document folder. It turns out that this folder has junctions to Pictures, Videos, and a couple other folders (full back, yeah!). So I exclude with:

/SL
/XJD
/XJF

:)

asusx205ta · 27 Dec 2016

Thanks Bree,

I will change the retry. 1Million times seems unnecessary :). I got the junction options.

Bree · 27 Dec 2016

asusx205ta said:

1Million times seems unnecessary...

Yes, I thought that too :)

If you would like to look into using Full and Incremental backups, I use Robocopy's /M option for that.

File Selection Options :
/M :: copy only files with the Archive attribute and reset it.

For a Full backup I first use the command attrib +A *.* /s then Robocopy with /M to clear their archive attribute as it copies them. Subsequent use of Robocopy /M will only copy files that have changes since the last backup.