File explorer --Very Slow for moving large number of small files


  1. Posts : 11,247
    Windows / Linux : Arch Linux
       #1

    File explorer --Very Slow for moving large number of small files


    Hi there

    If you have Loads (and I mean Loads) of small files - e-books, excel data. word docs, emails etc etc then File explorer runs REALLY slow on even fast devices

    I gave up moving this lot after about 4 hours. (W10 machine).

    Switched to Linux system -- did it in 45 mins similar hardware to W10 machine - both Real machines --not VM's and direct transfer (not via LAN).

    What gives !!!!!

    I know it's only a 1 time move but similar hardware with identical HDD should not have a HUGE difference in the move --note in fact Linux actually has hard a harder job since the file system was NTFS which isn't a native Linux file system either. Source system external HDD on USB 3 device,, target systm larger HDD device on USB3 device -- no hubs - USB 3 transfer working OK on both W10 and Linux so reasonable comparison.

    IMO the NTFS file system has serious problems if you start messing around with this type of data. !!!! Perhaps Ms need to look at a better file system for Mass storage devices. 5 and 6 TB HDD's are quite common now even for consumer sector and not expensive either.

    I know this is an exceptional job but it does show that something in File explorer isn't very efficient once you start manipulating large volumes of data - especially in small files. (Approx 290 TB of data consisting of 128,000 files in 55,000 odd directories)

    File explorer --Very Slow for moving large number of small files-snapshot10.png

    I'm usually quite happy with Windows - but there is definitely a problem here. Fortunately it was a 1 off job though !!!!.

    Cheers

    jimbo
      My Computer


  2. Posts : 822
    Microsoft Windows 10 Pro 64-bit
       #2

    In my quest to find out what my network transfer speeds were with a program called iPerf3 I found several posts on Stack Exchange that came to the same conclusion as you, In most cases there was literally a x2 increase in speed using Linux.

    I transfer quite a few VM's once or twice a month and I find that it is faster to zip the whole 15 to 25 Gig folder and then transfer it.
      My Computer


  3. Posts : 13,899
    Win10 Version 22H2 Pro and Home, Win11 Pro and Home
       #3

    I've found that using USB 3.0 devices is faster than USB 2.0 as it should be but the key is in having USB 3.0 ports, cables and drives. To that end I've added USB 3.0 Add-in PCIe X1 cards with 4 ports on it and for convenience a USB 3.0 Hub in the front of the case that takes power from the computer's power supply, not from the USB ports. Amazing what investing about $50 can help.
      My Computers


  4. Posts : 11,247
    Windows / Linux : Arch Linux
    Thread Starter
       #4

    Berton said:
    I've found that using USB 3.0 devices is faster than USB 2.0 as it should be but the key is in having USB 3.0 ports, cables and drives. To that end I've added USB 3.0 Add-in PCIe X1 cards with 4 ports on it and for convenience a USB 3.0 Hub in the front of the case that takes power from the computer's power supply, not from the USB ports. Amazing what investing about $50 can help.
    Hi there

    I decided for this type of stuff In future to use a dedicated USB3 external 2 bay mains powered enclosure with (optionally switchable) RAID 0 / 1 for the HDD's. Using RAID 0 on a USB 3 enclosure might be considered "Bonkers" by some but it speeded up the job no end. The RAID card is internal in the USB enclosure (can disable if I want to use the HDD's as separate HDD's --easier to have one logical HDD !!! I wasn't worried about data integrity with the RAID 0 - just wanted fastest possible transfer speed.

    On Windows the RAID 0 also speeded up transfer but was a real Slouch compared to Linux.

    Decided to use XFS (Linux) file system and use the Linux machine for this type of transfer in future.

    Windows NTFS needs a beef up certainly for large volumes of data transfer.

    I suppose that's why Linux still rules for large servers out in the commercial world. Don't get me wrong I LIKE windows too -- but seems for some jobs it's just not man enough for the task in hand.

    BTW type in my original post --Data transfer amount of course approx 290 GB -- not 290 TB !!!!

    Cheers
    jimbo
      My Computer


 

  Related Discussions
Our Sites
Site Links
About Us
Windows 10 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 10" and related materials are trademarks of Microsoft Corp.

© Designer Media Ltd
All times are GMT -5. The time now is 09:23.
Find Us




Windows 10 Forums