ReFS and OneDrive incompatibility - odd/expensive workaround?

cjfraz · 08 Dec 2019

I have a two disk ReFS storage pool set up as a mirror in storage spaces. It's been running well for sometime, and is accessed frequently by multiple users over a network share.

I'd like to start syncing data in the pool to the cloud using onedrive, but onedrive is incompatible with ReFS (very frustrating).

I don't want to convert the pool to NTFS... so...

I'm thinking I add 1 more physical disk to the computer, format it as NTFS, and then set it up as a mirror of the storage pool using the 'add mirror' option in the disk management control panel.

Yes... downside here is I have to use a whole additional disk, and make another local copy of my already locally mirrored data, just to get it on an NTFS drive that OneDrive will sync... and yes.. maybe this is a Frankenstein mix of storage spaces and software RAID.... but will it work? will it have performance impact on people using the ReFS pool over the network?

anybody have any better ideas how to solve this problem?

Thanks for any insight,

J...

muchomurka · 09 Dec 2019

I'm thinking I add 1 more physical disk to the computer, format it as NTFS, and then set it up as a mirror of the storage pool using the 'add mirror' option in the disk management control panel.

You cannot form mirror from different filesystems; the same write operations at sector level are performed on both disks. Mirroring is not dirsync. Furthermore, probably you will see msg "All disks holding extents for a given volume must have the same sector size" during attempt of creating mirrored volume because of StorageSpaces Virtualdisk different "hardware" characteristics.

sygnus21 · 09 Dec 2019

I know nothing about ReFS but I found this post interesting. Perhaps Brink could shed some light.

Possibly a compatibility concern? In particular non-supported OS's accessing OneDrive? (strictly a guess and nothing more).

cjfraz · 10 Dec 2019

Point well taken on the directory mirroring not being dirsync, and file system issues. Thank you for saving me some time on that!

It looks like with respect to this ReFS storage pool, i will settle for using sharepoint online library as an offsite backup only (backed up regularly with syncback pro), and may revisit this in the future if OneDrive ever starts supporting ReFS (and larger libraries).

Thanks again for the helpful comments.

J...

jimbo45 · 11 Dec 2019

Hi there

@cjfraz

Not sure if this would work so it's only an idea.

Set up a small Linux Virtual Machine (VM) and attach the two HDD's you want to use as your shared storage pool as RAW devices on the Linux VM --then you can use Software RAID 0 (very fast and much more stable than Windows storage spaces) -- since you are going to sync data to the cloud your RAID 0 would be backed up.

Format the two disks with xfs file system

On Linux to create the xfs file system run : mkfs.xfs </dev/device> -f for each device, then create raid 0 array,and finally start samba to share HDD's . (You might need to install xfsprogs if xfs file system isn't initially built in to the kernel or use ext4 - imo xfs is far superior and resilient though.)

After formatting then sync with one drive as per similar to this :

How to Sync a NAS Drive with OneDrive - ePINIONATED

IMO Windows storage spaces are an abomination both in performance terms, and reliability (sometimes break when upgrading Windows) . Linux RAID (via software mdadm) is reliable , very fast and easily shareable over your network.

Mdadm RAID 0 is also faster than LVM (another technology for multiple volumes)

reasonable discussion here on this :

Pick Your Pleasure: RAID-0 mdadm Striping or LVM Striping? - Linux Today Blog

- although of course it has a risk that if a disk fails then you have to build the whole array again but if you are sync'ing data with one drive - in the cloud the backup is available -- also if you have one of those small passport external self powered USB devices you can take local backups regularly too.

VM's can be extremely efficient these days --for the linux VM you don't even need a GUI ---> just enough for the network drive, RAID and samba. Use HYPER-V, VMWARE or VBOX. HYPER-V might be your best bet in this case since all you need is the HDD's to be shareable and sync'd with one drive -- you don't need to run a load of stuff on the Linux VM.

For backup (local) use the linux rsync built in command. You can also easily control your Linux VM via SSH from the host - you don't need a GUI for the VM but if you really want one then you can still access it via standard Windows RDP (needs a bit more set up on the VM though and you'll need to install something like a VNC server --tigervnc is often used. personally for your job I wouldn't bother with a GUI -- K.I.S.S is the motto (Keep it Simple St...D) !!!

BTW the sharers of the drives on the NAS system don't need to have any account on the HOST Windows system either once they have access to the shared drives on the NAS (Linux) system.

There's absolutely nothing wrong in having Servers as VM's -- it's often done commercially anyway !!!.

Don't forget to backup existing data before starting -- then you can restore to your RAID 0 array. Not sure of the volume you have but those 4TB usb3 external self powered disks are brilliant for local backup and can be in standard NTFS format. Linux can read / write ntfs these days so no trouble when restoring the data.

Added : Using Linux RAID you can use other levels such as RAID 1, RAID10, RAID5 etc depending on the resiliency you need on the NAS - personally in your case since you are sync'ing with onedrive go for RAID 0 - fast speed.

If also you don't like using hideous editor commands on the Linux system to edit config files (if no GUI you'll have to use line editors like vi) there's nothing to stop you editing these on the windows host via notepad or whatever and then SSH or FTP'ing them back to the VM.

Cheers
jimbo