My Virtualization Dilemna

I have been using VMWare Workstation for ages, and for the most part, I don't have a lot to complain about. Lately, with a lot of the new features in Windows 10, I really want to enable Hyper-V to take advantage of these new features, like WSL2 and Docker for Windows. But as everybody reading this already probably knows, you can't run any Type 2 hypervisor software like VMWare Workstation in Windows 10 if Hyper-V is enabled.

I have an Intel NUC not currently being used, so I thought I'd look into running a standalone virtualization server. I work as a S/W contractor, and I create VMs to segregate the work I do for different clients. While I'm usually only working with a single VM at a time, I do like to keep the other VMs up to date, so I tend to run these on occasion in the background. This is where a standalone server for virtualization would be nice. Running multiple VMs on my work machine can really affect performance. I like that with VMWare I can carry a VM or two with me on removable storage for when I visit the customer onsite. That means that being able to carry a VM with me is major factor in my end decision. The following is the result of my investigations and I thought this might interest some others. Perhaps you even have a suggestion or two that I didn't think of.

Right off, I tossed out using KVM since there is no Type 2 hypervisor S/W that would allow me to run a KVM guest on my Windows 10 work laptop.

I looked at ESXi, or vSphere from VMWare. The free version was more than capable of doing what I required.

One problem with the ESXi server is that it requires you to store your VMs in storage pools which are formatted in their own VMFS format. Using external drives is not officially supported, but I could work around that, and I could use an NFS share on my NAS as a storage pool. The NFS share was the only type of storage pool that I could use to copy my VMs between the server and my Windows 10 machine.

The second and biggest problem was, ESXi and Workstation/Player VMs are not compatible. You had to convert the VMs to move them between the two. So ESXi was not going to work.

Next, I looked into running a Windows Hyper-V Server which is also free to use.

The first problem is the Hyper-V server prefers to work in a domain and not workgroup environment. This meant there were some limitations in storage solutions and some setup issues. I could use local and external storage for the VMs, but not the SMB shares on my NAS. Both ESXi and Hyper-V servers would support iSCSI drives on my NAS, but data from an iSCSI drive is not easily accessed by other machines.

Another problem with Hyper-V is that support for Linux VMs is just downright terrible. Compared to VMware, the performance is terrible, especially the graphics performance, and sound and access to USB devices is difficult if not impossible. Using VMware Linux VMs I routinely format and create bootable SD cards for ARM devices like RPI. You can't do that with a Hyper-V Linux VM. I thought about using VirtualBox for my Linux VMs since they say it can run on Windows with Hyper-V enabled, but I found out that was false. Don't waste your time for now. Maybe one day they'll have it working reliably.

While the VM formats are the same between Windows 10 and the Hyper-V server, at least for now, Hyper-V requires you to import/export VMs in order to move them. There is NO GOOD reason for this and MS really screwed the pooch here. You should be able to carry a VM on a USB drive from machine to machine just like you can with VMWare Workstation/Player/Fusion. Moving VMs is a hassle, and sometimes you even had to recreate the VMs if the import/export function failed, and it did quite often. So, I abandoned the idea of moving my virtual machines to Hyper-V.

What I found out after spending a lot of time looking at this is there is no good solution. I'm sticking with VMWare workstation. I may just load a Linux distro on that unused NUC and install VMWare Workstation on that machine to run a few VMs in the background. If I want to take advantage of WSL2 or Docker for Windows, I'll simply have to reboot with Hyper-V enabled or disabled, or I can run those utilities in a Windows 10 VM using VMWare Workstation since this nested virtualization does work.
I

Not using hyperv in recent 6 years so I try to make only suggestions related to vmware.

You're right that esxi/vsphere VMs and wksta/player VMs are incompatible. You can use Vmware Converter to "copy" them between different hypervisors however this operation is time consuming and inconvenient so it is better to use only one type of VMs. There are two options, both have its drawbacks and both require disabled hyperv:

1. Using wksta/player VMs; you need to install linux or windows at your home host and then vmware workstation or player in it (so sadly no real type 1 hypervisor). Transferring of VM directories is very easy; furthermore, you can automate starting of VMs and other tasks at your home host using vmrun command from VIX package.

2. Using esxi/vsphere VMs; you can have real type 1 hypervisor at home (esxi installed on bare metal); however, you need to use nested virtualization (hypervisor-in-hypervisor) at your notebook, for example: win10 host - vmplayer - esxi - linux guest. Very simple, there are many step-by-step guides how to do it. In esxi, you have always full control over your VMs but transferring VMs between hosts may be somewhat slow and tricky: you need to use Datastore Browser from esxi webclient or old vSphere Client (not supported for 6.5 and 6.7 versions) to copy folders. If you have some vSphere license you're lucky; you can simply use vMotion migration (or cloning for some reasons) to move (even running) virtual machines, and use vSphere Update Manager to update your esxi hosts.

Nothing is perfect... Nested virtualization requires SLAT capable proccessor for 64-bit guests; of course it is always slower than single tier virtualization but acceptable. And for very basic esxi without vsphere, updating/upgrading hosts requires some handwork with esxcli command.

------------------------------------------

If you really want to use some hyperv-based features of win10 and don't want to reboot frequently, you can install esxi host as a guest in hyperv VM; not tested by me, still nested hypervisors, no idea of performance in this case.
Install ESXi 6.0 in a Hyper-V VM : A step-by-step guide

I agree. I'm going with #1 and will use a Linux host since I want the smallest memory/performance footprint. The main purpose of this machine will be running VMWare Workstation.

I looked at nesting ESXi in hyper-v, but there were too many issues with that approach. First, ESXi won't install without some patching to start with. Then there is a big problem with ESXi and external storage devices and the need to create/use storage pools.

I didn't consider nesting ESXi inside of VMWare Workstation since that would still require me to disable hyper-v. Since I would be disabling hyper-v, and running Workstation, I might as well just stick with Workstation. The performance hit of running Workstation on a Linux box instead of using a Type 1 hypervisor on my server is not a big issue. Neither is the performance hit of running Windows 10 in a VM and running those hyper-v dependent features in that VM.

He knows, of course...

If I want to take advantage of WSL2 or Docker for Windows, I'll simply have to reboot with Hyper-V enabled or disabled

If you really want to use some hyperv-based features of win10 and don't want to reboot frequently, you can install esxi host as a guest in hyperv VM

muchomurka said:

Update - see this thread:
VMWARE and Ms to have HYPER-V and Workstation / Player to co-exist
Sadly, no release date given.

Yea, I tried VirtualBox with Hyper-V and that doesn't work, at least not for me. In theory, this should work, but the differences in how the two virtualize H/W, like the GPU for instance, will likely result in a less than optimal solution. I think you would end up with a Linux VM running on Workstation with the terrible GPU performance of a Hyper-V VM.

Currently, I think KVM has one of the best desktop Type 1 solutions. Like Windows, you can run a desktop workstation connected physically to the hypervisor server using GPU passthrough. Unlike Windows, you can have that desktop workstation run whatever O/S you prefer. The problem with KVM is the lack of a Type 2 hypervisor that you could run on top of another O/S like Windows. Maybe they should get together with Virtual Box and offer such a solution.

As I said earlier, the Type 2 hypervisor is crucial for me to do the work that I do. I can go on-site anywhere, regardless of whether the customer has Windows or Linux boxes, and I can load VMWare Player and run a VM from a USB drive. This gives me the development environment I need without wasting a lot of time setting up a new machine.

Yea, I tried VirtualBox with Hyper-V and that doesn't work, at least not for me. In theory, this should work, but the differences in how the two virtualize H/W, like the GPU for instance, will likely result in a less than optimal solution.

Perhaps vmware & ms co-operation will result in some form of gpu etc passthru... we'll see. Furthermore, not sure how long we'll have to wait.

johnnyZ said:

But as everybody reading this already probably knows, you can't run any Type 2 hypervisor software like VMWare Workstation in Windows 10 if Hyper-V is enabled.

You definitely can run VBox when Hyper-V is enabled (performance isn't great but you can).

You definitely can run VMWare when Hyper-V is enabled (performance isn't great but you can).

Perhaps you are thinking of the past.