How do you manage data on very large discs?

Ford Prefect · 28 Oct 2019

Hi Folks,
I'd be very interested to know how other people manage very large discs and data, i.e. 1TB and above.
As a photographer and videographer/film maker I am almost always dealing with large volumes of data and hence have migrated to a couple of terabyte drives.
I always keep my system disc free of large data and back it up to an external drive weekly.
I currently make great use of partitioning and find it's an excellent way to keep everything separate and after a large photo or video project I can just delete the whole workspace without worrying that I'm going to delete something else by accident.
So my current set up is a separate partition for all of the following:
C: System
D: Pro photos
E: Family photos
F: Music
G: Software and downloads
H: Pro videos
I: Family videos
J: Video workspace
K: Video archives
L: Photo workspace
M: Photo archives
N: Multimedia
But by the time you add in a couple of letters for CD/DVD drives and another four for a multi-card reader, I'm up to T.
So when I plug in a usb flash drive it shows up at U.
All well and good so far and it works for me BUT I've now decided I should back up these partitions perhaps every 3 months or so, but if I plug in a large external drive, I can't partition it into enough partitions because windows runs out of letters!
So as I said, I'd be really interested to know what others do.
Regards.

dalchina · 28 Oct 2019

Hi, considering the number of partitions I'd suggest that's far too many, and very restrictive- comparatively inflexible if you get short of space on one.

Better, say, to have one partition for 'media' containing relevant folders (photos, videos, music...).

Backup strategy- as base, start with disk imaging. A disk image file can contain multiple partitions.
Differential images are smaller than the initial base image, and incrementals smaller again.
Macrium reflect (free) only offers differential imaging: the paid option has some much more advanced features.

For fast changing data add another backup method so data not backed up by periodic disk imaging- any decent free backup program, or use one that syncs data, e.g. to the cloud or to other storage.

perhaps every 3 months or so,

- that seems quite a large interval.

Consider too using disk monitoring software such as Hard Disk Sentinel (not free, but excellent) or Crystal Diskinfo - these can monitor and report if disk problems start to become apparent giving early warning.

Note that the act of disk imaging itself checks disk integrity for the used part of the disks imaged.

Ford Prefect · 28 Oct 2019

Thanks very much for your interest and for your "suggestions", I've tried fewer partitions and using folders but it doesn't give the separation and it didn't work for me, it's too easy to end up with hundreds of folders which become a nightmare to monitor.
Do you actually manage very large discs and data?
If so, I'm interested in what you actually do, rather than suggestions.
I'd like to here from people who have been there and done it!
Thanks again.

dalchina · 28 Oct 2019

I have a 1Tb drive with 4 partitions... and I use folders within a partition to categorise my data oh ye of little faith..

Suggest it's a matter of discipline- sounds as if you need the partition constraints to enforce that, if you see what I mean.

Good luck.

digitalcurisity · 28 Oct 2019

I find using a stand alone HD to put my files on i have a 3Tb WD Hard Drive.

Ford Prefect · 28 Oct 2019

Hi Dal,
Yes I guess you are right regarding the question of discipline but I have 4TBs of space with about 2.5TBs of current data and when I'm doing films and editing I often load in excess of 50GBs per film and I can have several on the go at the same time, then as I finish them I need to free up the space. I've found that using partitions (mine are all around 100GBs) aids organisation and separation and can prevent accidental erasures.
Just wondered what other people did.
Thanks.

Ztruker · 28 Oct 2019

You're partitioning scheme works for you so why change it. Change your backup scheme to a method that doesn't require a one to one correspondence of partitions, by using a backup program program like Macrium Reflect Premium. You can include multiple partitions in a single backup image. You can browse these images by mounting them via Macrium to recover folders and files as well as restore entire images.

This gives you the safety you need and the access down to the individual file level you would want.

SIW2 · 28 Oct 2019

It is an issue with windows using letters - not an issue for Linux.

I have the same problem when I bring another large disk online. I sometimes release letters from partitions I am not currently accessing and reassign them to partitions I do want to access. It only takes a few seconds. You can assign A and B, which helps a little - but still not difficult to run out of letters.

I don't think there is way round that in windows.

You didn't say what program you are using for your backups. Most imaging programs can create images of unlettered partitions, but will expect a lettered partition as the destination/target.

Ford Prefect said:

Hi Folks,
I'd be very interested to know how other people manage very large discs and data,...
But by the time you add in a couple of letters for CD/DVD drives and another four for a multi-card reader, I'm up to T.
So when I plug in a usb flash drive it shows up at U.
All well and good so far and it works for me BUT I've now decided I should back up these partitions perhaps every 3 months or so, but if I plug in a large external drive, I can't partition it into enough partitions because windows runs out of letters!
So as I said, I'd be really interested to know what others do.
Regards.

RolandJS · 28 Oct 2019

I hope you gave each data partition a terse, brief, unique name. That way, during backups / restores, you know exactly what partition is being acted upon during backup, and which partition is receiving a particular restore.

Berton · 28 Oct 2019

So as I said, I'd be really interested to know what others do.

My main computer has a 1TB internal HDD and 2 External USB HDDs, 2TB and 4TB. For use with a number of computers I have 2 x 2TB NAS HDDs attached by Ethernet cable to my Router, I can Map the Public Folder on those drives from any computer desired that can access the Network and also the Internet, whether Wired or Wireless/Wi-Fi. I haven't a need to multi-partition those 4 drives, the internal 1TB HDD came partitioned 150GB [C:] and 780GB [D:] from ASUS.