I have discovered an easy way to compacting VHDs

jimbo45 said:

. These days even ssd sizes are big so allocating a fixed vhdx storage size shouldn't be a problem....

When you have, as I do, more than a dozen VMs and a host machine with a 500GB SSD, even with each one being given a minimal 64GB virtual disk there's not enough room for each to have a fixed size .vhdx. Typically my VMs all have 128GB drives, so even a 1TB SSD would be too small.

And that's not even taking account of the vast increase in time and external storage required to make a system image of the host machine each month.

My monthly routine before making a Macrium Reflect image of the host machine is to go through each of its VM, delete any no longer needed checkpoints, and compact its .vhdx.

jimbo45 said:

When you create a vhdx - my view is if you are using it for windows is to create one with a fixed maximum size right at the start. Then it just behaves as a standard Windows physical disk. These days even ssd sizes are big so allocating a fixed vhdx storage size shouldn't be a problem. The trick is to have the OS as small as possible and all your data etc on other disks, partitions etc.

AS far as Linux is concerned -- if you choose the xfs file system rather than ext3 / 4 then it's "automatically self adjusting", no defragging etc required. (Better more "unbreakable" file system. Note though you can't reduce partition sizes once allocated but they can be extended with standard partition manangers like GPARTED,.

Cheers
jimbo

You do have a tendency to assume we all have superfast systems with lots of storage space. You might be lucky enough to afford that but I certainly cannot.

Bree said:

When you have, as I do, more than a dozen VMs and a host machine with a 500GB SSD, even with each one being given a minimal 64GB virtual disk there's not enough room for each to have a fixed size .vhdx. Typically my VMs all have 128GB drives, so even a 1TB SSD would be too small.

And that's not even taking account of the vast increase in time and external storage required to make a system image of the host machine each month.

My monthly routine before making a Macrium Reflect image of the host machine is to go through each of its VM, delete any no longer needed checkpoints, and compact its .vhdx.

Totally agree - I have 6 vhdx files that take approx 300 GB, but if I did them as fixed, they would all take twice as much space 600 GB (I like to allow for vhdx to double in size IF NECESSARY but I would not expect all to double).

Frankly, since earlier posts, I found easiest way to reduce size is to create a vhdx smaller than current one but bigger than nominal storage space, then clone old vhdx to new one using Macrium Reflect. One of Reflect's quirks is it automatically defrags if cloning to a smaller drive (but not if same size or larger). Then on completion of clone, I expand vhdx back to my nominal maximum size from hyper-v edit menu.

I cannot say if this would work for systems with Linux partitions though.

cereberus said:

....I cannot say if this would work for systems with Linux partitions though.

Unlikely.....

The following applies identically to image backups and clones. For brevity, though cloning is discussed, what follows applies equally to both.....

Linux file systems (XFS, JFS, BTRFS, etc) are not natively supported by Reflect. You will still be able to perform the clone, however, a Forensic Sector Copy will be performed. This will take more time due all sectors being copied, not just the in use sectors.

Macrium Software | Cloning and imaging Linux file systems

Bree said:

Unlikely.....

Macrium Software | Cloning and imaging Linux file systems

Then, as that would take a lot of space, I guess I would exclude the Linux partition from my normal Reflect backups which I do to a second internal drive fairly frequently (I do back up to external drive now and then as well), and only back up them to one of my older hdd backup drives (still good as backup workhorses).

Interesting topic.

On my Windows VMs I used sdelete many times successfully, like mentioned here:
How to Shrink a VirtualBox Virtual Machine and Free Up Disk Space
* this method demonstrates it in VirtualBox but also works everywhere, in Hyper-V, VMWare, KVM... all zeroing out Windows VM disks with sdelete but with each hypervisor with its own method of compacting of course.
Only zeroing out free space uses a single pass so it should be the fastest way. Unlike securely wiping data multiple times.

But it's a good mention about the defrag step; never bothered to use it in the past with smaller disks but it is definitely needed nowadays on larger NTFS vhdx drives.

Note:
I did use defrag from within VMs but there are chances indeed not to allow a full compact afterwards.
What may disturb a full defrag from within the VM is the fact that afterwards one still needs to shutdown and may allow Windows to move some files then.

Bree said:

I'm revisiting this thread now because of that 'almost' in the sentence above - one of my VMs is Linux Mint and you can't use Defrag on its EXT4 .vhdx, nor can Diskpart compact it without some help.

No need to defrag ext4 filesystem.

I assume zeroing out free space is enough in this case.
So you need a Linux equivalent for "sdelete -z x:" --> try the zerofree method.

cereberus said:

...I cannot say if this would work for systems with Linux partitions though.

Hopachi said:

....No need to defrag ext4 filesystem.....

I have one Hyper-V VM for Linux Mint, its vhdx file is difficult to compact. Although there's only about 12GB of data on its Ext4 partition, until now the best I've managed is to compact the vhdx to 37GB.

Diskpart has a 'compact vdisk' command that will shrink a vhdx, but for maximum effectiveness it's first necessary to consolidate the free space as much as possible. For any windows OS that can be easily be done by mounting the vhdx and using the Defrag command. That (of course) is where it all falls down for a vhdx with an Ext4 Linux system.

Yes, I've found out that you can defragment all files in Linux, but that does nothing for free space consolidation. I've also learned that cat /dev/zero > zero.dat ; sync ; sleep 1 ; sync ; rm -f zero.dat can zero all unused space. That only helps a little though. The clue that led to a solution was this (surprising to me) statement that Ext4 by design deliberately spreads data across the whole partition....

ext2/3/4 deliberately distribute files across all the block groups in the filesystem. Directories and the files they contain tend to be allocated in the same or nearby block groups, and keeping the free space scattered means there will likely be nearby free space to do that.

LinuxQuestions.org | [SOLVED] consolidate free space on ext4

So armed with that, I looked for a Linux based solution to squeeze all the data into the smallest possible part of the partition. The rather convoluted workaround I came up with was to boot my Linux VM from a GParted Live ISO, use it to shrink the partition as much as was allowed, then expand it back to its original size. Now when I used Diskpart to compact the vhdx it shrank from 37GB to just 17GB.