Best way to Clean-up a huge File Structure keeping only the good.

questorfla · 15 Dec 2018

Doing this manually is nearly impossible. The folder structure contains over 160,000 files that have accumulated over many years. Less than half of these need to be kept. But they are buried among all the other files that need to be removed.

I think it might be possible using some vartiation of robocopy to move only the specified filetypes from folder A to folder B and create the entire path as it moved the files. I don't know what combination of switches would be needed to accomplish this though and it is important that the original folder structure be maintained.

After I have moved as many types as i can think of, I could examine what was left in folder A to see if I missed anything. If anyone knows of a utility or tool made to do this I would appreciate the name

AndreTen · 15 Dec 2018

You could use wiztree. It shows extensions too and about zillion other info. I prefer treesize for simple treemap, but Wiztree offers more info. There are some nice pictures on their site.

Antibody - Wiztree

ddelo · 16 Dec 2018

questorfla said:

Doing this manually is nearly impossible. The folder structure contains over 160,000 files that have accumulated over many years. Less than half of these need to be kept. But they are buried among all the other files that need to be removed.

I think it might be possible using some vartiation of robocopy to move only the specified filetypes from folder A to folder B and create the entire path as it moved the files. I don't know what combination of switches would be needed to accomplish this though and it is important that the original folder structure be maintained.

After I have moved as many types as i can think of, I could examine what was left in folder A to see if I missed anything. If anyone knows of a utility or tool made to do this I would appreciate the name

Please provide an example and we'll create the robocopy command/commands.
PowerShell can also be very helpful in filtering and copying file types.

Try3 · 16 Dec 2018

Please define your criteria for "need to be kept".

Please confirm that you have a file backup.

Denis

Bree · 16 Dec 2018

questorfla said:

I think it might be possible using some vartiation of robocopy to move only the specified filetypes from folder A to folder B and create the entire path as it moved the files...

ddelo said:

Please provide an example and we'll create the robocopy command/commands.

I think I'd use the older Xcopy for this (in fact, I have) as RoboCopy needs a few more switches to cope with unexpected things like junction points or locked files. It's defaults can be less than useful in such circumstances. I wrote myself a 'user files backup' batch file with RoboCopy and until I got the switches right it could hang or go into an infinite loop.

Both RoboCopy and Xcopy have a /S option that will copy subdirectories. For Xcopy a simple command to copy all .mp4 files from folder A and recreate the folder structure in Folder B would be:

XCOPY /S "Folder A\*.mp4" "Folder B\"

questorfla · 16 Dec 2018

Several have asked for an example. I made this as complete as I could along with all the things that i need to accomplish when run. (Please excuse typos i tried to catch them alI)
It should MOVE only a single file type such as *.doc to the same path in B2 as it existed in A1. Nothing else would get moved or changed in A1. All the *.doc files in every subfolder in A1 would be moved to their same path in B2.

B2 is currently an empty copy of the full directory tree of A1 and I only need to populate it with the desired files in their correct places. However, if the script could create the subfolder paths as needed when it runs that would save me time on future uses. That would also avoid a lot of empty subfolders in B2 that I am sure to end up with and will have to remove anyway.

Subsequent runs of the same script should be able to next move *.txt or any other file extension to and from the same folders without duplicating any previously created paths by using the paths where they exist in B2. No duplicate or (2) subfolders created.

When job is complete, A1 would have its complete original folder tree lacking only the files that have been moved to their same locations in B2.
Here is the simplest possible layout showing what i am trying to do

A1\
item.zip
other.doc

sub1\
pic.doc
file.ini

sub2\
text.doc
text.exe
text.cfg

sub3\
temp.fil
_______________________________________________
B2\
sub1\
sub2\
sub3\

First run would move only all *.doc files from C:\A1\ to C:\B2 in the same subdirectory paths as they previously existed in A1. Shown below is C:\B2 after the *.doc move

C:\B2\
other.doc

sub1\
pic.doc

sub2\
text.doc

sub3\

This needs to be run as a batch script and would be Super if it asked which file type to move as well as the name of the original directory and destination directory and used them as a variables in the script.
Example:

Set org = "enter folder to move from: " (this needs to require full path as c:\a\b\c for example
Set dst = "enter folder to move to: " (this also needs to require full path as c:\a\b\c )
Set ty = "Enter file type to move: "
Then use %org% and %dst% with %ty% in the script.
(this is to allow the script to be in the same folder i keep all my scripts in but be run at a prompt anywhere)

At the end of the first run it should loop back to ask for "file type" again and run through the same move
And keep this up until the File Type I enter is "stop"
This has to be done on several folders for different reasons and if it works as needed I would want something I could save and easily reuse for future needs.

PS: Here is the script that got me the Empty dir-tree copied from A1 to B2 but I cannot find the right switches to get it populated as described above. It does work perfect though and if i could get the right switches in it in the right places I believe it could do it all.

Code:

robocopy.exe /R:0 /W:0 "C:\A1" "C:\B2" /E /Z /Copy:DT /XF "*.doc" /LOG:"C:\SCRIPTS\EMPTY.LOG"

questorfla · 16 Dec 2018

Thanks BREE!!
I got so focused on trying to force Robocopy to work that i totally missed your Simple Xcopy routine.
It Worked perfectly (I think). And a batch file using it wil be easy to modify with the needed variables. I can't believe it was that easy but sometimes the hardest things are. Tunnel-vision is a terrible thing to get!
The Only part that Xcopy didn't do was remove the original files after copying, which is what i need to happen to get things as I need them. The starting folder should have all the chosen filestypes Moved to the new folder and kept in their correct paths. After copying, the files themselves, (not the folders they were in) need to be removed from the original so it ends up with only the original directory tree which would then include only filetypes that were not moved out of it.
I am hoping there is a "switch for that" in Xcopy and i'm looking right now.
Or, as an altrnative, since i already have a way to get an empty directory tree, maybe the MOVE command can be used to get the files into it? But i have never gotten out of "move" what I expected to get.

Bree · 16 Dec 2018

questorfla said:

This needs to be run as a batch script and would be Super if it asked which file type to move as well as the name of the original directory and destination directory and used them as a variables in the script.
Example:
Set org = "enter folder to move from: " (this needs to require full path as c:\a\b\c for example
Set dst = "enter folder to move to: " (this also needs to require full path as c:\a\b\c )
Set ty = "Enter file type to move: "
Then use %org% and %dst% with %ty% in the script.
(this is to allow the script to be in the same folder i keep all my scripts in but be run at a prompt anywhere)

Couple of point there, the SET command as used above requires the /P option. Also, you need to perform some sanity checks on the answers provided. eg.

Code:

:ORG
SET /P org="enter folder to move from: "
if not exist %org% echo the %org% folder does not exist
if not exist %org% goto ORG
echo .
echo Folder "%org%" selected.

For the file type you'll need to check if it was entered with or without a leading 'dot', ie was doc or .doc typed?

Code:

:TY
SET /P ty="Enter file type to move: "
SET _1st=%ty:~0,1%
if "%_1st%" == "." goto TY-OK
echo "please enter a file type with a leading '.' eg: .txt"
 goto TY
:TY-OK
echo.
echo file extension %ty% selected.

reference: variable substring - Windows CMD - SS64.com

Bree · 16 Dec 2018

questorfla said:

The Only part that Xcopy didn't do was remove the original files after copying, which is what i need to happen to get things as I need them. ... I am hoping there is a "switch for that" in Xcopy and i'm looking right now.
Or, as an altrnative, since i already have a way to get an empty directory tree, maybe the MOVE command can be used to get the files into it? But i have never gotten out of "move" what I expected to get.

No, neither Xcopy or Move have the options you want.

I'm always wary of deleting files from within a batch command as there's no recycle bin 'safety net' to fall back on*. However, if you want to delete the original files after a successful copy, then the DEL command has a /S option too.

* this came up earlier, powershell or 3rd party utilities can make the recycle bin a viable option
Command Prompt to delete to recycle bin - Windows 10 Forums

questorfla · 16 Dec 2018

Thanks for catching all my dumb errors The set missing /p etc. is what i get for trying to post before even test/running. While I agree 100% with the dangers of using delete in batch, the worst that could happen is i would just make another copy of the original and try again. I think i will go with your del /s After the rest is done though

I never run any kind of experimental scripts on original folders anyway. When finally completed i would be running it on a copy of the folder i need to clean. But I am always one of the 'fools to rush in where angels fear to tread' and always appreciate the 'heads-up'.
Like i said, it is the EASY ways that I seem to have complete overlooked. While examing Xcopy option i found i could have much easier used it with /T /E to create my Empty Directory tree as well. Again Thanks.

One final: I decided to enter the original path to be including file-type to avoid the other problem with missing (.) etc. Since it will only be Me running it i will remember much of the needs but you are 100% correct. I should consider my own limitations and possible errors if doing this for a "keep around" script.