ReFS cannot fix corruption--can file be accessed at all?

With Windows 10 Pro for Workstaions:
If integrity streams settings are enabled and an error is returned instead of requested data because ReFS is unable to correct a corruption, can the data still be accessed by disabling integrity streams? If not, is there some other way to access the data? If I have a 10GB video file and just a couple of bits are rotted, I’d like to be alerted to the corruption, but I certainly don’t want the entire file to be discarded!

That was my reading of the only docs I could find too, but what if I turn off integrity streams? With integrity streams off after the corruption detection, would ReFS still be aware that the file is corrupted?

I'm leaning toward using ReFS because I have large files and lots of data, but I certainly won't use it if it would trash a mult-gig video file because of 1 bit. Sure I want corruption corrected if possible and I would always want to be aware that corruption exists, but I don't think the file should be gone. I can think of many cases where a corrupted file is still better than nothing at all.

I can't believe I'm the only one thinking along these lines.

The title of my original post may be a bit misleading--I don't currently have a case of corruption. I need to know what happens if there is corruption in the future. Otherwise I'm not paying for 10 Pro for Workstations.

I would give it a shot except that the needed functionality has been removed from Windows 10 Pro. I was hoping someone would know. It doesn't seem I'm asking a strange question. I certainly hope the lack of docs isn't an indication of the quality of ReFS in general.

Thanks for the response. Hopefully the thread will stay alive. Something else to think about is that introducing test corruption might not be the simplest thing to do.

mjohnsonn2 said:

I would give it a shot except that the needed functionality has been removed from Windows 10 Pro.

Only formatting was removed - 10 Pro can read/write ReFS volumes just fine.

You can format your volume ReFS from PE or persuade normal 10 Pro it is PE by following these instructions: Format a Drive with ReFS in Windows 10. Just remember to reset it back to normal after.

I don't know but from the documentation if you turn off integrity streams it implies that it isn't going to check the checksum of the file (it is off by default). It should be easy enough to turn on for one file, edit one bit with HxD and then see.

I will try it but it will be at least a week before I'll be trying much. My file server and workstations are all working on video pretty much non-stop and I don't want to monkey with the registry and be rebooting. Perhaps I'll just do the upgrade before I test. It's only around $125 from what I hear. I will get to it and report back in case someone else cares. I still can't imagine why this hasn't come up before. Thanks

There doesn't seem to be an easy way to introduce bit-rot. HxD looks to be performing only changes acceptable to ReFS.

I purchased Pro for Workstations and tried the test. I set file integrity and checked that it was both enabled and enforced for a large file. I can fully access the file both before and after altering a few bits using HxD. No error occurs.

I think the problem is that HxD performs only bit changes that are legitimate according the file system. In other words, when I change some bits to simulate bit rot, then save the file, ReFS must be recalculating the checksum. The "date last modified" attribute in File Explorer shows the correct update time which further indicates that HxD performed a valid change.

Any ideas for how to fake some file corruption so we can see if ReFS does it's job at all?

mjohnsonn2 said:

Any ideas for how to fake some file corruption so we can see if ReFS does it's job at all?

Sorry, I don't know. I thought HxD could specifically write to specific location on the disk (that was my theory anyway). I suppose you could boot some Linux and use dd to write zero (or something else) at random but whether that would work I don't know (nor how to identify where to write them).

Something like this to write a zero to sector 10 on the disk.

Code:

dd if=/dev/zero of=/dev/sdb1 bs=1 seek=10 count=1

I've no idea if it would work though (you have the controller to deal with so what you think you write may not in fact be true) and even if it did whether it would prove anything either way is beyond me.

I suppose we would need some input from someone who knows more about it.

I believe I came up with a way to create some corruption in order to do some testing. The surface test function within Hard Disk Sentinel (great product!) has a way to perform a destructive write test to random blocks on a drive. Apparently it does this at a low level without any legitimate reporting to the ReFS file system.

This is what I set up for a quick test.

1. I formatted a 1TB drive with ReFS using "format /fs:refs /I:enable <volume>" to turn on integrity checking disk-wide.

2. I nearly filled the drive with large video files. Less than 10GB unused space.

3. I ran the Hard Disk Sentinel random write test for a few seconds. The graphic showed 10 or twelve blocks as written to. They were spread out all over the place so I was sure I hit some files.

Some tests...

I started copying a large bunch of the files from the ReFS formatted drive over to an NTFS drive. After a minute or two, the process stopped on a file with this: "Error 0x80070143: A data integrity checksum error occurred. Data in the file stream is corrupt."

Terrific! We have corruption that got caught with a bad checksum.

The filename was clearly shown too. A quick check with Get-FileIntegrity showed integrity was still enabled for the file. Even with the checksum error I could still access it and view it in VLC so my concern of losing my data is certainly lessened. The video ran for a while in VLC and then failed with no error message (just the VLC cone on screen), I assume when it hit the corruption introduced by the destructive write test.

I tried this a few more times, wondering if ReFS had somehow deleted even part of a file. Apparently not because I checked the number of bytes in a folder having a bad file both before and after I saw an error on copying the folder's files and they were the same. I can't be sure though because I have no idea if ReFS was doing something in the background to check integrity before the test.

All in all my tests are encouraging but they certainly aren't very extensive or definitive. I checked the event log and saw some ReFS errors there, but interestingly not all of the ones that popped up during my testing.

I sure hope some folks with the ability and time to do a better testing job than I can will figure this all out. It still bugs me that what will happen to our data with a "resilient" file system is so terribly documented. We shouldn't have to hire a homicide investigator to figure out what's going on with a paid product.