Transfer latest dated file names and remove earlier dated duplicates [closed] - date

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
Currently, I have to upload a bunch of excel sheets to a network shared folder. Each of these files has the date they were created appended at the end of the filename. Then I have to remove the earlier duplicates leaving just the latest dated versions.
Basically it looks likes this...
Before:
apples 2019.07.01.xlsx
apples 2019.07.07.xlsx
oranges 2019.07.01.xlsx
bananas 2019.07.01.xlsx
After:
apples 2019.07.07.xlsx
oranges 2019.07.01.xlsx
bananas 2019.07.01.xlsx
I stumbled upon a possible solution, which was to create a batch-file to recursively go through the folder and do this. However, I am unsure where to start.
I read this other stackoverflow article, which is pretty close to what I want to do but I am having trouble adjusting it to my needs. Any assistance would be appreciated.
Edit2: this code worked for me:
#(
SetLocal EnableDelayedExpansion
ECHO OFF
SET "_PathToCheck=Y:\T\DT"
SET "_FileGlob=PLOG - * - ????.??.?? - *.xlsx"
SET "_CurrentFile="
)
FOR /F "Tokens=1-2* Delims=-" %%A IN ('DIR /A-D /O-N /B "%_PathToCheck%\%_FileGlob%"') DO (
IF /I "!_CurrentFile!" EQU "%%A-%%B" (
ECHO.Deleting: "%_PathToCheck%\%%A-%%B-%%C"
DEL /F /Q "%_PathToCheck%\%%A-%%B-%%C"
) ELSE (
ECHO.
ECHO.New File Found: "%%A-%%B"
ECHO.-----------
ECHO.Retaining: "%_PathToCheck%\%%A-%%B-%%C"
SET "_CurrentFile=%%A-%%B"
)
)

Answer 1

You may use the same approach you would use if you do this job by hand: review the file list and every time that a file appear with the same name than the previous one, remove the previous one... Simple! Isn't it? ;)
#echo off
setlocal EnableDelayedExpansion
rem Initialize the "previous name"
set "lastName="
rem Process files in natural order, that is, the same order showed in the question
rem and set %%a to name and %%b to rest: date plus extension
for /F "tokens=1*" %%a in ('dir /B /A:-D /O:N *.xlsx') do (
rem If previous name is not the same as current one
if "!lastName!" neq "%%a" (
rem Just update previous name and date
set "lastName=%%a"
set "lastDate=%%b"
) else (
rem Remove the previous file
ECHO del "!lastName! !lastDate!"
rem and update the previous date
set "lastDate=%%b"
)
)
This solution assumes that the name and the date parts are separated by exactly one space...
EDIT: New method added, after several confusing changes made by the OP
#echo off
setlocal EnableDelayedExpansion
set "lastName="
for /F "delims=" %%a in ('dir /B /A:-D /O:N *.xlsx') do (
set "currName="
set "currFile="
for %%b in (%%~Na) do (
set "part=%%b"
set "currFile=!currFile! !part!"
if "!part:.=!" equ "!part!" set "currName=!currName! !part!"
)
if "!lastName!" neq "!currName!" (
set "lastName=!currName!"
set "lastFile=!currFile!"
) else (
ECHO del "!lastFile:~1!.xlsx"
set "lastFile=!currFile!"
)
)
Example of input files:
apples 2019.07.01.xlsx
apples 2019.07.07.xlsx
oranges 2019.07.01.xlsx
bananas 2019.07.01.xlsx
apples 2019.07.01 proof1.xlsx
apples 2019.07.07 proof1.xlsx
PLOG - Organic Valley - 2019.07.01 - (DAI) OG Cream Cheese.xlsx
PLOG - Organic Valley - 2019.07.07 - (DAI) OG Cream Cheese.xlsx
PLOG - Organic Valley - 2019.07.10 - (DAI) OG Cream Cheese.xlsx
Output:
del "apples 2019.07.01.xlsx"
del "apples 2019.07.01 proof1.xlsx"
del "PLOG - Organic Valley - 2019.07.01 - (DAI) OG Cream Cheese.xlsx"
del "PLOG - Organic Valley - 2019.07.07 - (DAI) OG Cream Cheese.xlsx"

Answer 2

Here is the newest version for you Greg.
Although originally I and a couple of others wrote versions which would work for date modified using a different logic (create variables for all of the files' names then sort them again, or do sets of compares) I realized we could still accomplish the goal in a single loop and without needing so many temp variables and without having to have more complex logic in that scenario and this one, so I took a few minutes and created that version.
Essentially we just need to define a variable with the File's name that has already been found, since we know they are ordered correctly date-wise, and only need to worry about removing the duplicate named files.
To do so we can use SEt or IF DEFINED, I prefer IF DEFINED here since I can use regular IF ( ) THEN ( ) ELSE ( ) logic as already defined in the script. (Note the items in Italic here are not terms that can be used in a CMD script, but I am writing them to clarify the normal logic of the IF construct)
We could use SET "[Variable Name]" instead, and test if success or failure using || or &&, but that would be more re-write and unnecessary here.
#(
SetLocal EnableDelayedExpansion
ECHO OFF
SET "_PathToCheck=C:\T\DT"
SET "_FileGlob=PLOG - * - ????.??.?? - *.xlsx"
SET "_CurrentFile="
SET "_MatchList="
)
FOR /F "Tokens=1-3* Delims=-" %%A IN ('
DIR /A-D /O-N /B "%_PathToCheck%\%_FileGlob%"
') DO (
SET "_CurrentFile=%%A-%%B-%%D"
SET "_MatchList=!_CurrentFile: =_!"
IF DEFINED _MatchList_!_MatchList! (
ECHO.Deleting: "%_PathToCheck%\%%A-%%B-%%C-%%D"
DEL /F /Q "%_PathToCheck%\%%A-%%B-%%C-%%D"
) ELSE (
ECHO.
ECHO.New File Found: "!_MatchList!" Date-Stamp: %%C
ECHO.-----------
ECHO.Retaining: "%_PathToCheck%\%%A-%%B-%%C-%%D"
SET "_MatchList_!_MatchList!=%%A-%%B-%%D"
)
)
The previous version which conformed to the standards for only the left side of the file name being unique.
#(
SetLocal EnableDelayedExpansion
ECHO OFF
SET "_PathToCheck=Y:\T\DT"
SET "_FileGlob=PLOG - * - ????.??.?? - *.xlsx"
SET "_CurrentFile="
)
FOR /F "Tokens=1-2* Delims=-" %%A IN ('
DIR /A-D /O-N /B "%_PathToCheck%\%_FileGlob%"
') DO (
IF /I "!_CurrentFile!" EQU "%%A-%%B" (
ECHO.Deleting: "%_PathToCheck%\%%A-%%B-%%C"
DEL /F /Q "%_PathToCheck%\%%A-%%B-%%C"
) ELSE (
ECHO.
ECHO.New File Found: "%%A-%%B"
ECHO.-----------
ECHO.Retaining: "%_PathToCheck%\%%A-%%B-%%C"
SET "_CurrentFile=%%A-%%B"
)
)
Example Output:
Y:\>Y:\t\DT.cmd
New File Found: "PLOG - File Three For yoU "
-----------
Retaining: "Y:\T\DT\PLOG - File Three For yoU - 2019.08.11 - (something) AAA 1 .xlsx"
New File Found: "PLOG - File Number Two "
-----------
Retaining: "Y:\T\DT\PLOG - File Number Two - 2019.12.19 - Ending ABDC 1111 AB.xlsx"
Deleting: "Y:\T\DT\PLOG - File Number Two - 2019.07.30 - Ending ABDC 1111 AB.xlsx"
Deleting: "Y:\T\DT\PLOG - File Number Two - 2019.03.12 - Ending Number 3 .xlsx"
New File Found: "PLOG - File Number One "
-----------
Retaining: "Y:\T\DT\PLOG - File Number One - 2020.01.01 - Ending BBB .xlsx"
Deleting: "Y:\T\DT\PLOG - File Number One - 2019.12.19 - Ending BBB 2 .xlsx"
Deleting: "Y:\T\DT\PLOG - File Number One - 2019.09.07 - Ending AAA1.xlsx"
Deleting: "Y:\T\DT\PLOG - File Number One - 2017.01.03 - Ending AAA 1 .xlsx"
Y:\>
Screenshot confirming the Script works and showing the Output and results:
Essentially this does the same thing as in my original version only now we know that we should be looking for the Hyphens
IE:
We use DIR to Sort the File names in a reversed sort order, this will mean that files that have a newer date it will appear before those with older dates.
This simplifies the logic for deleting the files, and is the crux of my original solution as well.
Because of using that method we only need to check if the first part of the file name (the part before the date) is the same as the previous file found.
We do this by creating a variable to hold the name of the current file _CurrentFile and set it empty, so on the initial check, it will not match any file name.
If _CurrentFile matches the first part of the file name (again, the part before the date) o the file dir found, then we can safely delete it.
If _CurrentFile does not match the interesting portion of the file reported by the DIR cmd, then we update the _CurrentFile variable to that new value and move on to the next file result to test.
As you are unfamiliar with cmd/batch scripting, I would like to take a minute to go into more detail about what the script is doing and why for you so you can go forward yourself:
First I should note that we have a few options on how to iterate the files, most commonly for, for/F, and For files are common go-to for looping over files, sometimes with a DIR cmd in a for /F alternatively with a WMIC file list (although, thankfully WMIC is finally getting deprecated in favor of Powershell).
As we know you simply wat to Choose based off its Filename and the date stored in the file name, then using a dir cmd to sort by Name will be a pragmatic method to do the matching quickly
Now onto what each part of the script is doing
#(
Parenthesis create code blocks in CMD and Batch Script, everything within a given Parenthesis will be evaluated at the same time.
By placing an # in front of the parenthesis any commands with it ( And not within further parenthesis, or after a DO ) will not be echoed to the screen. This is to stop this section form showing up and cluttering output.
SetLocal EnableDelayedExpansion
We are turning on Delayed Expansion, to allow us to easily evaluate the contents of the variables inside of a for loop by referencing them with !_var! instead of %_Var%, technically we can get away without this if any of your filenames have ! in them, we should disable this and re-write it a bit, if not then it's fine.
ECHO OFF
I am stopping the script from echoing every line it is doing so we have less cluttered output. Setting this command means I no longer have to use # in front of further commands within this code block or future code outside this block.
SET "_PathToCheck=Y:\T\DT"
SET "_FileGlob=PLOG - * - ????.??.?? - *.xlsx"
SET "_CurrentFile="
)
Setting the variables and closing the code block with a closing parenthesis seems self-explanatory except for one _FileGlob
This is a standard File Glob it is used to match the name of the file you want to have considered for comparison.
* matches any character any number of times, ? matches any character once.
This ensures that if we encounter files which don't conform to the format we expect we can skip them.
If the need required a more explicit matching, we might use a glob of *.xlsx and use FINDStr to check against a regex pattern to make sure the format was very exactly the one needed.
in this next part
FOR /F "Tokens=1-2* Delims=-" %%A IN ('
DIR /A-D /O-N /B "%_PathToCheck%\%_FileGlob%"
') DO (
[Code]
)
Now I am going to go a little out of order here:
We are using DIR to quickly sort the files by Their Name in reverse order and return just the filenames. DIR is extremely fast at doing this, so it's preferable if you are doing a little sorting rather than matching the files using IF compares later. We utilize the file glob as mentioned above to ensure only files we want to evaluate are returned.
The Option /A-D ignores directories, /B will only output the file name (since we aren't recursing) Then, we have /O-N -- /O is "Order By" the Option N sorts by Name ascending, while -N sorts by name in Reverse (Descending) Order (IE Z-A 9-0), so we can be assured that the file with the name that has the newest date will be the first one we find.
This is all placed inside a For /F Loop which is a way to parse the output of a command. We use Delims=- to "Tokenize" or Split-up the strings FOR is receiving from the DIR command. We Tell FOR what Variable names to store the Tokes in using %%A (Variables are as follows: "? # A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ]" OR "_ `` a b c d e f g h i j k l m n o p q r s t u v w x y z" { ( More info here https://ss64.com/nt/for_f.html ) ), Variables we be assigned to Tokens starting with the one you chose.
When we specify the tokens to pick, Tokens=1-2*", specifically 1-2 means to take the first Token through the second token, and store them in the First N variables (where N = the number of variables in the set 1-2, ie %%A and %%B for our purposes), and * means stop tokenizing anything after any tokens mentioned prior to this point, and place all of the remaining portions of the line into the next variable (%%C).
Because we are tokenizing use the Hyphen as a delimiter, we now that the first two tokens will be PLOG and [Name to Compare while the date and the rest of the file name will be in the 3rd token.
In the DO ( ) section we are going to go on and process the info returned by each line and stored in our tokens.
Lets go on to examine the code within the DO ( )
IF /I "!_CurrentFile!" EQU "%%A-%%B" (
ECHO.Deleting: "%_PathToCheck%\%%A-%%B-%%C"
DEL /F /Q "%_PathToCheck%\%%A-%%B-%%C"
) ELSE (
ECHO.
ECHO.New File Found: "%%A-%%B"
ECHO.-----------
ECHO.Retaining: "%_PathToCheck%\%%A-%%B-%%C"
SET "_CurrentFile=%%A-%%B"
)
This is probably familiar to you enough as you are used to VBA, but we are testing the value of the variable _CurrentFile to the First Two Portions of the string, which we know are the entire portion of the file name up to the Date, and we need to add the Hyphen back in because when FOR splits by tokens it removes those tokens.
We check is the _CurrentFile variable ia a match for the currently returned file name's portion up to, but not including the date.
If this matches, we delete (Del) the file because we have already seen the file once before so this is one that is older.
We use the /F Option to Force deleting read-Only Files, and we use /Q to stop it from prompting us to confirm the deletion of each file.
We also ECHO. that we are deleting the file we found to note what the script is doing.
) ELSE (
If this does not match, that means this is a new file we haven't encountered for, and must be the first one returned, in which case we want to keep it because we know from the sort of Dir that it will be the interesting file.
Therefore on a non-match, we change the _CurrentFile variable to hold the value of the first two tokens %%A-%%B to use in future checks of the results returned.
We also ECHO. that we found the file and are retaining it to give a nice little indicator of what the script is doing.
A further note on ECHO -- Although I like how Echo. looks, ECHO( is safer to use, and I prefer it for that reason, but it is more confusing for folks who are unfamiliar with cmd scripts as the Open parenthesis looks like I have either a typo or an unclosed code block and can lead to people thinking it causes some problem. So for this reaosn, I try to avoid using ECHO( in favor of ECHO. when ECHO. will do.
Original Post and Versions Which used the incorrect format
You can make this a quite simple script that basically finds each unique name and keeps the 1st one so long as your names are in YYYY.MM.DD.xlsx format by pre-sorting the names so that the one with the newest date in the name one is always the first file encountered.
The Space is guaranteed? Optional?
to do this you need to use a FOR /F loop to parse the output from DIR ordered by (/O) Name Descending (-N)
DT.CMD:
#(
SetLocal EnableDelayedExpansion
ECHO OFF
SET "_PathToCheck=Y:\T\DT"
SET "_FileGlob=* ????.??.??.xlsx"
SET "_CurrentFile="
)
FOR /F "Tokens=*" %%A IN ('DIR /A-D /O-N /B "%_PathToCheck%\%_FileGlob%"') DO (
SET "_TFile=%%~nA"
SET "_TFile=!_TFile:~0,-10!"
IF /I "!_CurrentFile!" EQU "!_TFile!" (
ECHO.Deleting: "%_PathToCheck%\%%~A"
DEL /F /Q "%_PathToCheck%\%%~A"
) ELSE (
ECHO.
ECHO.New File Found: !_TFile!
ECHO.-----------
ECHO.Retaining: "%_PathToCheck%\%%~A"
SET "_CurrentFile=!_TFile!"
)
)
We then simply need to compare the names of the files except for the Trailing YYYY.MM.DD.xlsx, and if the File is the 1st with that name we keep it, as we know it will be the newest.
If the name is a duplicate we can delete it because we know we already skipped the newest.
Example Output:
Y:\>Y:\t\DT.cmd
New File Found: bananas
-----------
Retaining: "Y:\T\DT\bananas 2019.07.01.xlsx"
New File Found: oranges
-----------
Retaining: "Y:\T\DT\oranges 2019.09.01.xlsx"
Deleting: "Y:\T\DT\oranges 2019.07.11.xlsx"
New File Found: apples
-----------
Retaining: "Y:\T\DT\apples 2019.07.07.xlsx"
Deleting: "Y:\T\DT\apples 2019.07.01.xlsx"
If your Date format is instead YYYY.DD.MM.Xlsx
Then you will need to go through an extra hoop or two.
Essentially in that scenario, we can do the following:
save the File name as a variable with the corrected (sortable) version of the file name (YYYY.MM.DD format) and then sort it and then compare the array of variables, deleting the ones which are not newest.
Here is that version DT_DM.CMD:
#(
SetLocal EnableDelayedExpansion
ECHO OFF
SET "_PathToCheck=Y:\T\DT"
SET "_FileGlob=* ????.??.??.xlsx"
SET "_CurrentFile="
SET "_MatchList= "
)
FOR /F "Tokens=*" %%A IN ('DIR /A-D /ON /B "%_PathToCheck%\%_FileGlob%"') DO (
SET "_TFile=%%~nA"
SET "_TFileMD=!_TFile:~-5!"
SET "_TVar=__!_TFile:~0,-5!!_TFileMD:~-2!.!_TFileMD:~0,2!"
REM ECHO.Storing File: "%%~A" As: "!_TVar!"
SET "!_TVar!=%%~A"
IF /I "!_CurrentFile!" NEQ "!_TFile:~0,-10!" (
ECHO.New File Found, Adding to Sort List: "!_TFile:~0,-10!"
SET "_CurrentFile=!_TFile:~0,-10!"
SET "_MatchList=!_MatchList! "__!_TFile:~0,-10!""
)
)
ECHO.
ECHO.Delete Old Files
ECHO.-----------------
REM Loop the Matched Files:
FOR %%a IN (%_MatchList%) DO (
ECHO.
ECHO.Delete Old %%a Files
ECHO.-----------------
REM Loop the SET sorted for each File Found and Skip the First one (Newest), deleting the others.
FOR /F "Skip=1 Tokens=1-2 Delims==" %%A IN ('SET "%%~a" ^| SORT /R') DO (
ECHO.Deleting: "%_PathToCheck%\%%~B"
DEL /F /Q "%_PathToCheck%\%%~B"
REM Remove the deleted file variable so we can print a list of retained files at the end:
SET "%%A="
)
)
ECHO.
ECHO.Retained Files:
ECHO.-----------------
FOR %%a IN (%_MatchList%) DO ( SET "%%~a" )
Here is example output from that:
Y:\>Y:\t\DT_DM.cmd
New File Found, Adding to Sort List: "apples "
New File Found, Adding to Sort List: "bananas "
New File Found, Adding to Sort List: "oranges "
Delete Old Files
-----------------
Delete Old "__apples " Files
-----------------
Deleting: "Y:\T\DT\apples 2019.07.07.xlsx"
Deleting: "Y:\T\DT\apples 2019.12.01.xlsx"
Delete Old "__bananas " Files
-----------------
Delete Old "__oranges " Files
-----------------
Retained Files:
-----------------
__apples 2019.12.01=apples 2019.01.12.xlsx
__bananas 2019.01.07=bananas 2019.07.01.xlsx
__oranges 2019.11.07=oranges 2019.07.11.xlsx
Now, both of these examples Assume that you ALWAYS want whatever file is Named with the Newest Date, not the most recently modified file
This is probably the case as I know I usually want to have that scenario when working with my own dated files, in case someone or some process came along and modified the files, or I saved more than one out of order.
But just in case you really wanted to just retain the most recently modified file, we can Use the same concept as in the second version and save the Real Modified time to the Variables instead of the date on them.
DT_Modified.CMD:
#(
SetLocal EnableDelayedExpansion
ECHO OFF
SET "_PathToCheck=Y:\T\DT"
SET "_FileGlob=*.xlsx"
SET "_CurrentFile="
SET "_MatchList= "
)
FOR %%A IN ("%_PathToCheck%\%_FileGlob%") DO (
ECHO.%%A| FINDStr /I " [0-9][0-9][0-9][0-9]\.[0-9][0-9]\.[0-9][0-9]\.xlsx$" >NUL && (
SET "_TFile=%%~nA"
SET "_TVar=__!_TFile:~0,-10!%%~tA"
ECHO.Storing File: "%%~A" As: "!_TVar!"
SET "!_TVar!=%%~A"
IF /I "!_CurrentFile!" NEQ "!_TFile:~0,-10!" (
ECHO.
ECHO.New File Found, Adding to Sort List: "!_TFile:~0,-10!"
ECHO.
SET "_CurrentFile=!_TFile:~0,-10!"
SET "_MatchList=!_MatchList! "__!_TFile:~0,-10!""
)
)
)
ECHO.
ECHO.Delete Old Files
ECHO.-----------------
REM Loop the Matched Files:
FOR %%a IN (%_MatchList%) DO (
ECHO.
ECHO.Delete Old %%a Files
ECHO.-----------------
REM Loop the SET sorted for each File Found and Skip the First one (Newest), deleting the others.
FOR /F "Skip=1 Tokens=1-2 Delims==" %%A IN ('SET "%%~a" ^| SORT /R') DO (
ECHO.Deleting: "%_PathToCheck%\%%~B"
DEL /F /Q "%_PathToCheck%\%%~B"
REM Remove the deleted file variable so we can print a list of retained files at the end:
SET "%%A="
)
)
ECHO.
ECHO.Retained Files:
ECHO.-----------------
FOR %%a IN (%_MatchList%) DO ( SET "%%~a" )
Example of First Script Running ad results: