1.    29 May 2016 #1

    File explorer --Very Slow for moving large number of small files


    Hi there

    If you have Loads (and I mean Loads) of small files - e-books, excel data. word docs, emails etc etc then File explorer runs REALLY slow on even fast devices

    I gave up moving this lot after about 4 hours. (W10 machine).

    Switched to Linux system -- did it in 45 mins similar hardware to W10 machine - both Real machines --not VM's and direct transfer (not via LAN).

    What gives !!!!!

    I know it's only a 1 time move but similar hardware with identical HDD should not have a HUGE difference in the move --note in fact Linux actually has hard a harder job since the file system was NTFS which isn't a native Linux file system either. Source system external HDD on USB 3 device,, target systm larger HDD device on USB3 device -- no hubs - USB 3 transfer working OK on both W10 and Linux so reasonable comparison.

    IMO the NTFS file system has serious problems if you start messing around with this type of data. !!!! Perhaps Ms need to look at a better file system for Mass storage devices. 5 and 6 TB HDD's are quite common now even for consumer sector and not expensive either.

    I know this is an exceptional job but it does show that something in File explorer isn't very efficient once you start manipulating large volumes of data - especially in small files. (Approx 290 TB of data consisting of 128,000 files in 55,000 odd directories)

    Click image for larger version. 

Name:	snapshot10.png 
Views:	1 
Size:	190.3 KB 
ID:	82209

    I'm usually quite happy with Windows - but there is definitely a problem here. Fortunately it was a 1 off job though !!!!.

    Cheers

    jimbo
      My ComputerSystem Spec
  2.    29 May 2016 #2
    Join Date : Jun 2014
    Posts : 554
    Microsoft Windows 10 Pro 64-bit

    In my quest to find out what my network transfer speeds were with a program called iPerf3 I found several posts on Stack Exchange that came to the same conclusion as you, In most cases there was literally a x2 increase in speed using Linux.

    I transfer quite a few VM's once or twice a month and I find that it is faster to zip the whole 15 to 25 Gig folder and then transfer it.
      My ComputerSystem Spec
  3.    29 May 2016 #3
    Join Date : Aug 2014
    Forever West
    Posts : 3,983
    Win10 Home and Pro, Win10 Insider Preview, Win7 Home, Linux Mint

    I've found that using USB 3.0 devices is faster than USB 2.0 as it should be but the key is in having USB 3.0 ports, cables and drives. To that end I've added USB 3.0 Add-in PCIe X1 cards with 4 ports on it and for convenience a USB 3.0 Hub in the front of the case that takes power from the computer's power supply, not from the USB ports. Amazing what investing about $50 can help.
      My ComputerSystem Spec
  4.    29 May 2016 #4

    Quote Originally Posted by Berton View Post
    I've found that using USB 3.0 devices is faster than USB 2.0 as it should be but the key is in having USB 3.0 ports, cables and drives. To that end I've added USB 3.0 Add-in PCIe X1 cards with 4 ports on it and for convenience a USB 3.0 Hub in the front of the case that takes power from the computer's power supply, not from the USB ports. Amazing what investing about $50 can help.
    Hi there

    I decided for this type of stuff In future to use a dedicated USB3 external 2 bay mains powered enclosure with (optionally switchable) RAID 0 / 1 for the HDD's. Using RAID 0 on a USB 3 enclosure might be considered "Bonkers" by some but it speeded up the job no end. The RAID card is internal in the USB enclosure (can disable if I want to use the HDD's as separate HDD's --easier to have one logical HDD !!! I wasn't worried about data integrity with the RAID 0 - just wanted fastest possible transfer speed.

    On Windows the RAID 0 also speeded up transfer but was a real Slouch compared to Linux.

    Decided to use XFS (Linux) file system and use the Linux machine for this type of transfer in future.

    Windows NTFS needs a beef up certainly for large volumes of data transfer.

    I suppose that's why Linux still rules for large servers out in the commercial world. Don't get me wrong I LIKE windows too -- but seems for some jobs it's just not man enough for the task in hand.

    BTW type in my original post --Data transfer amount of course approx 290 GB -- not 290 TB !!!!

    Cheers
    jimbo
      My ComputerSystem Spec

 


Similar Threads
Thread Forum
Laptop gets very slow while copying/moving files
My laptop used to work great in every aspect on windows 10 days ago .... 2 days ago i installed linux mint to give it a try and formatted the hard disk, but i couldn't acclimate to linux os. so i downloaded W10 iso and did a fresh install...but i...
Performance & Maintenance
File Explorer search, display only files using column number
When I have a long number of files in a folder and want to use the search function to show only the file names that contain a certain number or letter at an exact column number (in the file name) how can I accomplish that? example kit12.txt...
General Support
Customization Use Large or Small Taskbar Buttons in Windows 10
The taskbar is the long horizontal bar at the bottom of your desktop. You can use large or small taskbar buttons. This tutorial will show you how to use large or small taskbar buttons for your account in Windows 10. If you turned on Use small...
Tutorials
Solved File explorer slow with loading thumnails and files
I installed Windows 10 last week as an upgrade from Windows 7, and the internet part is working good. But something with the file explorer didn't work. The Document, Pictures, Music, and Video files are still there, but the thumbnails load slowly,...
General Support
Increasing number of items to be pinned to File explorer
I have recently upgraded to Windows 10 and would like to know how I go about increasing the number of items to be pinned to File Explorer It would be appreciated if someone could kindly assist me in advising me how to set the maximum number of...
General Support
Our Sites
Site Links
About Us
Windows 10 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 10" and related materials are trademarks of Microsoft Corp.

© Designer Media Ltd
All times are GMT -5. The time now is 11:30.
Find Us
Twitter Facebook Google+ Ten Forums iOS App Ten Forums Android App



Windows 10 Forums