How to safely remove an external drive?

@Enter and @TV2: Sorry, I'm also not set up for email notification. Very interesting discussion since my last post.

A few additional thoughts/questions ...

Enter said:

FreeCommander is closed but there are still shown some P: processes in Process Monitor:


How could I delete / close them?

Hmmm... I don't know and I can't figure out how to replicate or create something similar on my system, I suspect because I don't have the 'Operation' column in Process Explorer. I've never seen that column before, and I can't figure out how to activate it on my Win7 system. How did you get it to display? (Is it perhaps only available in Win10?)

Do those FreeCommander P:\ line entries have any associated handles if you open handles in the lower pane (Ctrl+H)? When you see this kind of entry after FreeCommander is closed, does a Process Explorer search (Ctrl+F) for 'P:\' find any open handles associated with FreeCommander? If not, then it doesn't seem like deleting/closing the P:\ line entries would be necessary before removing the external USB drive.

Enter said:

In Process Explorer this is shown:


Looks somehow risky to close these handles.

I haven't previously noticed handles like the AIMP.exe handles shown here (without the drive P\: in the handle Name, and with 'Section' as the handle Type). Do those handles close if you close AIMP?

At any rate, I'm guessing they are not relevant to the inability to safely remove the external USB drive, as it appears you searched for "P:" rather than "P:\", and those handles were found because they include "P:" in the handle Name (AIMP:MouseHookShare and AIMP:KeyboardHookShare).

enter said:

And what if windows is just accessing the drive / ntfs system or so?

enter said:

But what if windows scans / rebuilds or whatever that ntfs?

Without researching it further (which I don't have time or energy to do right now), I'm still not 100% certain what produces the ...$Extend\$RmMetadata\$Txf handles. I previously mentioned from recollection of my research a few years ago that these handles are associated with NTFS journaling, but if I remember correctly, commenters in some forums/blogs thought they might be associated with Windows Indexing or Recycle Bin. My best recollection is that after sifting through everything I could find, I was about 90-95% certain the culprit was/is NTFS journaling.

I've since forgotten much of what I learned at that time, but this MSDN blog post by Raymond Chen (Jan 1, 2013) seems to explain the relevant basics of NTFS journaling in regard to removing an external USB drive. If I understand it correctly, the journaling is the first of several steps whenever changes are made to a file on an NTFS drive, and "provides file system recoverability by logging ... operations required for any transaction that alters important file system data structures ... before these operations are carried through on the disk" (e.g., before all other steps necessary to complete copying/altering/deleting a file). File data loss/corruption problems can occur if an external USB drive is removed before all the other steps are completed. As you obviously already understand, with write caching enabled, it is possible that the remaining steps may not actually be completed when the OS informs the user that the copying is complete, as the file changes cached in RAM may not yet written to the removable drive (presumably depending on how busy the OS is, data transfer speed, etc.). So their could be data corruption if the drive is unplugged before those steps are completed.

But disabling write caching is supposed to avoid such problems, so file operations are supposed to be complete when the OS tells you they are complete:

"Write-behind caching is disabled by default on removable drives. You get into this mess only if you override the default. And on the dialog box that lets you override the default, there is a warning message that says that when you enable write-behind caching, you must use the Safely Remove Hardware icon instead of just yanking the drive."

It seems to me that if journaling is the first step in any file-system change, then once you can be confident the other steps involved in file changes are completed, obviously no further journaling is necessary and the drive can be removed. Basically, as I understand it, journaling protects the file system structures (preventing file system corruption), while disabling write-caching protects the data contents inside those already protected file system structures (preventing file data corruption due to premature unplugging).

So my reasoning is that *IF*, in fact, the ...$Extend\$RmMetadata\$Txf handles are created/used by NTFS journaling as I have surmised, then "yanking the drive" with those journaling handles still open shouldn't cause any problems, as long as you verify other handles are closed and you have allowed enough time for all other file operations to complete. (Usually should be a few seconds at most, but pretty cautious and I figure I'm safe if I've waited 10-15 minutes.)

...But this reasoning hinges on whether I'm correct about the source of the ...$Extend\$RmMetadata\$Txf handles.

Given the ubiquity of external USB drives and the frequency of the ...$Extend\$RmMetadata\$Txf handles problem, it has always seemed strange to me that Microsoft has not (AFAIK) at a minimum:

• authoritatively clarified the issue, so users can be confident about what is going on (Is it NTFS journaling?)
• provided some guidance on 'safe' practices or workarounds (Is it 'safe' to unplug if those are the only open handles?)

...But I don't think those minimums are actually sufficient (especially since most average users would never see them or probably understand them. Instead, if these ...$Extend\$RmMetadata\$Txf handles are, in fact, associated with journaling, it seems to me Microsoft should build into Windows some way of automatically closing the handles when the user attempts to "Safely Remove Hardware and Eject Media", or provide a GUI option to do so, if/when that is the only issue preventing safe removal.

Enter said:

Just had the same probleme with another drive, it couldn't be removed safely.

This reminds me that when I previously researched this, there was also some discussion in the Everything Search forum about whether NTFS indexing by Everything Search might be preventing external USB drive removal (and maybe something about whether Everything made use of the ...$Extend\$RmMetadata\$Txf handles ???). I don't recall the issues clearly right now, but I think there were some suggestions about how to tweak the Everything NTFS Index settings that seemed to make sense. ...But IIRC correctly I tried the suggested remedies without success.

Enter said:

There obviously were two recycle bins on it. After some tries / error messages

I can see that this refers to what appears to be a Recycle Bin (S-1-5-21-263......), but any chance you could post an English translation of the first 5 lines?

Enter said:

I made it to remove the bins.


Suddenly I could remove that drive safely.

Ohhh, very interesting!!!

Did you check what happened to the ...$Extend\$RmMetadata\$Txf handles in Process Explorer after you deleted the $Recycle.bin? ...I'm assuming they must have been closed/gone, but I'm trying to understand if deleting the $Recycle.bin closed the handles. If so, maybe they don't have to do with NTFS journaling, or perhaps NTFS journaling and Recycle Bin interact somehow (???)

I'm glad you've been looking at this so closely and willing to put so much time into researching it and posting your results, as it's a very frequent and vexing problem, and it would be nice to get a clearer understanding. I put a couple questions in boldface and blue above to make them easier to pick out. Like I said earlier, I'm not set up for email notification, but I'll try to check back regularly for the next few days to continue our discussion if you have anything further to post.

gdv said:

I don't have the 'Operation' column in Process Explorer. I've never seen that column before, and I can't figure out how to activate it on my Win7 system.

'Operation' is a default column in ProcMon (that is Process Monitor as in the screen shot - not Process Explorer). If you mean ProcMon and you don't have the 'Operation' column you can add it by going to Options > Select Columns.

As for unplugging drives I have the same issue with the journal apparently locking volumes but I couldn't ever figure it out. It is more of a problem for me disconnecting the (SATA) disk in my docking station than USB but happens with both.

I use a shortcut to sync.exe pinned to the taskbar to flush everything to disk and then just disconnect.

Sorry to be so long getting back. This took a long time to write (too long), and I ended up doing some research during the writing. Then when I was about 2/3 done, the forum software somehow lost my reply content in the edit box during a preview, so I had to rewrite a lot of it.

There almost certainly errors and typos, but I'm too wasted to proof it any more carefully right now.

lx07 said:

'Operation' is a default column in ProcMon (that is Process Monitor as in the screen shot - not Process Explorer).

I knew (and almost said in previous post) the answer was going to be obvious! ... .... ...But thanks!

Part of my problem was that the image doesn't display full size in my browser, and I was having a hard time reading the relevant P: lines in question. By the time I downloaded the image from imgur.com so I could display it full size to read the P: lines, I had already given up trying to find the column in Process Explorer. ...So I didn't even look at the top left corner where I can now clearly see "Process Monitor". Worse, I'm already familiar with Process Monitor; but to be fair, I rarely use it, or the 'Time of Day' column (which I wondered about but didn't search for), the registry keys, etc. in the image would have tipped me off.

lx07 said:

I use a shortcut to sync.exe pinned to the taskbar to flush everything to disk and then just disconnect.

Ahhh, thanks! That seems like a good additional precaution. Although I always just make sure I've waited more than long enough before unplugging, I will incorporate that into my practices. Should especially be relevant if/when there is a need to unplug in a hurry.

Enter said:

...May be "Process Monitor" is not that reliable, I do not know.

To the contrary, although I rarely use it, I think it is very powerful and reliable... ...but just not the right tool for the job of finding open drive handles.

gdv said:

If not, then it doesn't seem like deleting/closing the P:\ line entries would be necessary before removing the external USB drive.

Enter said:

Yes, but forcing to close such a line would have the same effect, I assume, as just plugging off the drive, the risk of data loss / data loss.

When I thought the screenshot was from Process Explorer I was saying those P:\ line entries would not need to be closed before removing drive (because probably have nothing to do with handles on the P:\ drive). Now that I realize the screenshot is from Process Monitor , I would point out that the P:\ line entries cannot be forced closed.

Process Monitor displays a running list (log) of the filtered process events that occur on the system between when you start and stop monitoring, so each of those lines simply represents a single process event that occurred at the time listed in the 'Time of Day' column.

I don't know enough about it, but it seems likely to me that there are process events when a P:\ drive handle is opened or closed (and that might even be part of what the CreateFile and CloseFile P:\ line entries in the screenshot indicate ???). But if so, the resulting line entry in Process Monitor line simply shows when that event occurred. I guess if you knew the right kind of event to filter for, you could conceivably monitor for only handle open and close events, and then would be able to view when those occurred on the P:\ drive and check for a close following each open. But that would be an extremely inefficient way to observe handles, and you still wouldn't be able to force the handles closed from within Process Monitor.

gdv said:

File data loss/corruption problems can occur if an external USB drive is removed before all the other steps are completed.

Enter said:

Yes, or if a handle is close by force, I guess.

Yes <- - - - - - - [Gee, I can almost never answer that briefly.]

gdv said:

it is possible that the remaining steps may not actually be completed when the OS informs the user that the copying is complete, as the file changes cached in RAM may not yet written to the removable drive (presumably depending on how busy the OS is, data transfer speed, etc.).

Enter said:

Yes, and / or besides of process belonging to that special transfer, there might be other processes, e.g. rebuilding / changing the NTFS, the Journal or whatever.

Not completely sure I'm understanding you here, but if you have already verified in Process Explorer that all other handles are closed on the external drive, that particular transfer should be the only thing left to complete. The journaling crucial to NTFS drive recovery will have already occurred first, and there shouldn't be any other processes besides completing the operations of writing of any file changes to the drive (flushing any remaining data from the cache and writing it to the drive). No further journaling required, at least for drive recovery.

Per NTFS Transaction Journal:

NTFS log provides file system recoverability by logging, or recording, the operations required for any transaction that alters important file system data structures. This is done before these operations are carried through on the disk. This process ensures that if the system crashes, partially completed transactions can be redone or undone when the system comes back online

As I understand it, the idea behind journaling is that any NTFS structural changes are logged to the drive first, before the file data changes (held in a cache) are actually written to/deleted from the drive. This protects the NTFS data structures so recovery to a stable state is possible in the event of system crashes, power failure, etc. (...including unplugging a USB NTFS drive). But obviously it doesn't protect the actual data to be stored in those structures if the data isn't yet written to the drive.

So as I understand it, even if the file is quite large, the NTFS journaling includes logging of files names, logging where on the drive data is to be written/deleted, etc. Logging these events involves a trivial amount of metadata that can be written to the drive almost instantly/immediately from a human perspective, and is written before the writing of relevant data, so the NTFS logging events are not really a meaningful source of delays the user has to be concerned about. I could be wrong, but I'm assuming the OS won't represent to the user that the operation is completed before this crucial NTFS logging occurs. Then the remaining operations necessary to write the actual data to the drive occur (with each step in the process is also logged almost instantly in NTFS journal). What could take longer is flushing data from the cache and writing it to the drive, transferring large amounts of data over a limited USB transfer rate connection, etc. But unless another change is made to the file, no further NTFS logging is needed to protect drive file system recoverability, even if data is still being written (although still unwritten file data won't be recoverable).

Again per NTFS Transaction Journal:

After a transaction is logged, the NTFS performs the sub-operations on the volume itself, in the cache. Finally, after the cache has been updated, NTFS writes another record to the log file, recording that the transaction has been completed.

When a system is being recovered after a system failure, the NTFS reads through the log file and redoes each committed transaction. After that, the NTFS locates all of the transactions in the log file that weren't committed at the failure and rolls back.

As I understand it, after the data has been flushed form cache and written to disk, the final NTFS logging event occurs to mark the process as complete. Again this involves a trivial amount of metadata that can be written to the drive almost instantly. If the file data itself hasn't yet been completely written to the drive, the final NTFS log file record showing the file is complete won't be there, and NTFS rollback will occur upon recovery. But once the data writing is complete, the NTFS log entry should occur almost instantly, and the drive can then be safely removed. (I'm pretty sure, but not 100%, this also means that if a very large file is only partly written, most of the written data will be recoverable. Depending on the type of data it may or may not be usable: i.e., a plain text file, an .mp3 file, or some types of video files that can be split anywhere might work as expected up to the point where the transfer ended.)

gdv said:

So there could be data corruption if the drive is unplugged before those steps are completed.

Enter said:

Yes, that's why one has to click "safely remove" to write the content of the cache to the drive. Actually I would assume, flushing the cache would have the same effect, but the error message keeps on popping up.

I assume probably the same re flushing the cache. But if the ...$Extend\$RmMetadata\$Txf handles exist for NTFS journaling (per our hypothesis) and are the only problem (and I think we can be confident they are once enough time has passed), the error message is popping up not because of uncompleted data writing, but because those handles are "locking" the drive. And the OS apparently does not have any means to automatically close them when no longer needed, and the user UI has no manual option to unlock/close them.

Yes, sounds like a reasonable assumption, of course, but if not...one never knows, not only that a single file might get corrupt, if the NTFS breaks...

See NTFS journaling info above. The NTFS logging is written almost instantly to the drive before, during, and immediately after the data is written, so there is no reason for NTFS to 'break'. At worst, MTFS will rollback to the last completed/logged transaction and the file system will be stable (not broken) going forward from there. At worst, any unwritten data in the one file that did not finish writing will be missing from that file, but the file metadata will be intact. Maybe an analogy would be a paper file folder (NTFS data structure) being placed in a file cabinet (HDD), but not all the relevant sheets of paper (data) get added to the file folder before someone closes the drawer and locks the file cabinet (system crash, power loss, unplug USB). When you unlock and open the file cabinet (reboot), the file folders are all intact, complete, and usable, except the one folder that was incomplete when the drawer got closed and locked. But even it's contents are in the proper folder and usable, although incomplete.

If you allow enough time (or flush the cache), it seems even that last file written to the drive should be complete and sound. I fact after researching this a bit more and trying to write it up, I think I've convinced myself that I have been being way overly cautious in waiting at least 10-15 minutes (and often hours) before removing a drive. Unless I have been writing a hug file to the drive, if only the ...$Extend\$RmMetadata\$Txf handles exist, I think I've convinced myself that 1-2 minutes should be more than sufficient

Or watching if the LED of the drive is still blinking, I could imagine.

Might be helpful for those drives that have a blinking LED, but not all do (and most USB flash drives don't, but still could be formatted NTFS), and I don't know if they all behave the same way. I think I'd have more confidence in checking the handles in Process Explorer and adding in the additional step of flushing the cache with Sync or FFB (see below).

Or rather do some programming (or what ever) to solve the problem.

That's what I meant when I said,

gdv said:

...But I don't think those minimums are actually sufficient (especially since most average users would never see them or probably understand them. Instead, if these ...$Extend\$RmMetadata\$Txf handles are, in fact, associated with journaling, it seems to me Microsoft should build into Windows some way of automatically closing the handles when the user attempts to "Safely Remove Hardware and Eject Media", or provide a GUI option to do so, if/when that is the only issue preventing safe removal. (emphasis added)

To make that a bit clearer, it seems like Microsoft should build in a way to detect when ...$Extend\$RmMetadata\$Txf handles are the only issue preventing drive removal, and then either automatically close the handles when the user clicks "Safely Remove Hardware..." or provide a UI checkbox or button for user to manually close those handles (maybe with brief description of the issue and guidance to help user decide whether to use the checkbox/button).

Ooops, the best search program I know, may be it was me asking that.

Yeah, I love Everything Search! I'll try to go back and check those Indexing discussions as soon as I get a chance. Do you use the same username there? (I'm gdv there as well)

Everything would not cause failing savely removing, it would automatically close or release such handles to let the drive be safely removed.

Are you saying that is the way it now works? ...or that settings can now be tweaked to do that? I haven't checked the site for a while, and I'm running v1.4.0.713b, which I know is not the most recent version. I'll try to check for updates as soon as I get a chance.

Access to folder denied.
Admin rights needed to process.
You have to get the rights from admin to make changes to the folder.

Date of creation.
Button, left: Try again and "Cancel", right.

Ahhh, thanks for the translation. Helps me better understand the whole $Recycle.bin issue.

gdv said:

Did you check what happened to the ...$Extend\$RmMetadata\$Txf handles in Process Explorer after you deleted the $Recycle.bin?

Enter said:

If I remember it right, they disappeared.

If that is easy to replicate, or if you run into a similar situation, it would be nice to know for sure, as it might imply Recycle Bins are keeping the ...$Extend\$RmMetadata\$Txf handles open. Apparently it is established that a corrupted Recycle Bin is one cause for the handles not closing, I would then want to get some idea of how often that is the cause. What if it is the main cause, or even the only cause? I don't think that is the case as I don't think I'm likely to have corrupted Recycle Bins because I nearly always delete files completely (Shift+Delete) rather than sending them to the Recycle Bin

gdv said:

I'm assuming they must have been closed/gone, but I'm trying to understand if deleting the $Recycle.bin closed the handles. If so, maybe they don't have to do with NTFS journaling, or perhaps NTFS journaling and Recycle Bin interact somehow (???)

Enter said:

I would assume, the Journaling / NTFS works the same way for bins like it / them does with normal files.

So would I, but if it is corrupted Recycle Bins that cause the handles to not be released, that is an important finding, and if deleting the corrupted Recycle Bin closes the handles (and results in them not staying open during future attempts to "Safely Remove Hardware...", that would be even more important.

lx07 said:

I use a shortcut to sync.exe pinned to the taskbar to flush everything to disk and then just disconnect.

Enter said:

That one I use, too. But it is just flushing the cache, it does not effect / stop the journaling, NTFS actions, I would say.

Agreed, both Sync & FFB just flush the data in the cache to the drive, but theoretically that should stop any NTFS journaling as well, as that logging is done almost instantly before data is written, updated almost instantly while data is being written, and finalized almost instantly when the data-writing is completed (see discussion above). But what Sync and FFB don't do is close the NTFS ...$Extend\$RmMetadata\$Txf handles on the drive.

To stay with the admittedly imperfect paper file folder analogy, the file folder (NTFS) is labeled (NTFS) and slotted into the correct section (NTFS) of the correct drawer (NTFS) of the filing cabinet (HDD), and all the relevant sheets of paper (data) are filed in the file folder, but the filing secretary (OS) is still holding onto the file folder (...$Extend\$RmMetadata\$Txf handles) inside the drawer, so the drawer cannot be closed and locked ("Safely Remove Hardware").

Enter said:

This one can be used, too: FFB.zip

Ahhh, I see you have also found USB expert, Uwe Sieber (although you probably use his German page). I don't think I've visited his site since I was researching the ...$Extend\$RmMetadata\$Txf handles a few years ago, and I don't think I grasped the issues well enough at the time to have realized FFB might be useful. (...Might have even been before FFB was first introduced in April 2013 but probably since then.) ...But I had been thinking while drafting each of my posts in this thread that I would go back and check his site for anything new on the issue as soon as I get a chance.

According to Google, there is no mention of "$Extend\$RmMetadata\$Txf" (or "$Extend" or "$RmMetadata" or "$Txf") anywhere on his site, but he's the only one I've run across on the internet who I'm guessing could probably authoritatively clarify the ...$Extend\$RmMetadata\$Txf handles issue. I remember thinking about trying to email him about it a few years ago, but never got around to it.

I notice he mentions a couple of limitations of Sysinternals Sync that FFB overcomes:

"The Sysinternals sync tool can flush volumes only which have a drive letter. And it forgets drive Z: when called without parameters"

It would be interesting to know if the FFB -f switch simply does the same thing as closing handles in Process Explorer:

-f force dismount (open handles become invalid)

If so, the switch would presumably force close all open handles on the drive. That, along with detection of what is preventing 'Safely Remove Hardware", would be the kind of function I think Microsoft should build in to automatically use (or manually selectable by user), if Windows detects that only the four ...$Extend\$RmMetadata\$Txf handles are preventing "Safely Remove Hardware".