``
I have to copy files from source folder to target folder both are in the same storage account(ADL). The files in the source folder are of in .txt format and have date appended in the file name,
eg: RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
and
RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt
(20221201 and 20221202 is date in file name , date format: yyyymmdd)
I have to create a pipeline that will sort and store files in the folders in ADL's in this hierarchy
ex: adl/2022/12/01/RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
adl/2022/12/02/RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt
even if we have multiple files on same date in file name based on that date in file name it has to create year(YYYY) folder and in year(YYYY) folder it should create month(MM) folder and in month(MM) folder it should create date(DD) folder like above example. Each File should copy into respective yyyy and respective mm and respective date folder.
What I have done:
In Get Metadata - Given argument to extract **childitems**
For each activity that contains a Copy activity.
In Copy activity source wildcard path is given as *.txt
for sink took concat expression using split and substring functions
Please check the screenshots of all activities and expressions
but this pipeline is creating the folders based on date in file name (like adl/2022/12/01)
but problem is it was copying all files into all date(DD) folders
(like adl/2022/12/01/RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt
adl/2022/12/02/RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt)
1.[GET META to extract child items](https://i.stack.imgur.com/GVYgZ.png)
2.[Giving GET META output to FOREACH](https://i.stack.imgur.com/cbo30.png)
3.[Inside FOREACH using COPY ](https://i.stack.imgur.com/U5LK5.png)
4.[Source Data Set](https://i.stack.imgur.com/hyzuC.png)
5.[Sink Data Set](https://i.stack.imgur.com/aiYYm.png) Expression used in Data Set in Folder Path '#concat('adl','/'dataset().FolderName)
6.[Took parameter for Sink](https://i.stack.imgur.com/QihZR.png)
7.[Sink in copy activity ](https://i.stack.imgur.com/4OzT5.png)
Expression used in sink for dynamic folders using split and substring function
#concat(substring(split(item().name,'.')[3],0,4),'/',
substring(split(item().name,'.')[3],4,2),'/',
substring(split(item().name,'.')[3],6,2)
)
**OUTPUT for this pipeline**
adl/2022/12/01/RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt
adl/2022/12/02/RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt
**Required Output is**
adl/2022/12/01/RAINBOW.IND.EXPORT.20221201.WIFI.NETWORK.SCHOOL.txt
adl/2022/12/02/RAINBOW.IND.EXPORT.20221202.WIFI.NETWORK.SCHOOL.txt
(i.e each file should copy to respective date folders only even if we have multiple files in same date, they should copy to date folders based on date in file name)
I have reproduced the above and got same result when I followed the steps that you have given.
Copy activity did like this because, in source or sink you did not gave #item().name(file name for that particular iteration) and you have given *.txt in the wildcard path of source in copy activity.
It means for every iteration(for every file name) it copies all .txt files from source into that particular target folder(same happened for you).
To avoid this,
Give #item().name in source wild card file name
It means we are giving only one that iteration file name in the source for the copy.
(OR)
Keep the wildcard file name in source as it is(*.txt) and create a sink dataset parameter for file name.
and give #item().name to it in copy activity sink.
You can do any of the above and if you want you can do both at a time. I have checked all the 3 scenarios like
1.#item().name in wild card sink file name.
2. #item().name in dataset file name by keeping wildcard path same.
3. combining both 1 and 2(#item().name in wild card file name and in sink dataset parameter).
All are working fine and giving desired result.
I downloaded all the pictures from an old spanish social network and the Last modified date is the date that I downloaded the file instead of the real date. I would like to order all the pictures by Last modified date to upload to google photos using the correct date.
I know the real date because the image name has the date and time and I would make a script to change de last modified date using the name of the file. I don't care about the time.
Name of the file format: "yyyy-mm-dd hh-mm-ss.jpg"
I'm using windows, but is not a problem to install Unix to execute some bash script, because it may probabbly be easier.
Is there any way to do this?
Thanks in advance
You can use powershell $(Get-Item *Filename*).creationtime=$(Get-Date "mm/dd/yyyy hh:mm am/pm") in a batch file or command prompt to change the creation date of a file. for more information look at this. In your case you could do:
for %%i in (<folder with all the files path>) do (
set name=%%~i
set name=%name:~0,19%
powershell $(Get-Item %%~i).creationtime=$(Get-Date "%name:~5,2%/%name:~8,2%/%name:~0,4% %name:~11,2%:%name:~14,2%")
)
Note: I have not tested this since I do not have the data set for it.
Im trying to create a batch script to call a .exe to carry out analysis on multiple data files in a same folder. The syntax should be like:
"D:\Softwares\Analyzer.exe" [DataFile1].dat [DataFile2].dat ... [DataFileN].dat AnalysisFile.pdo"
Currently I have tried to use a FOR loop to scan each *.dat file in a specified folder. (I don't know how many data files in that folder, so I cannot type the filenames directly in the command line)
For example:
#ECHO OFF
FOR /r %%i in (*.dat) DO (
"D:\Softwares\Analyzer.exe" %%~ni.dat TestAnalysis.pdo
)
PAUSE
However, the analysis is carried out on seperate datafiles, and the .exe file will pop-up and open every time when a new .dat file is detected. Is there any way I could use *.dat or any other methods to represent [DataFile1].dat [DataFile2].dat ... [DataFileN].dat in one line seperated by a space (not a new line)?
I have also tried to use #tilte, which does not work as well. Since the .exe window keep pop-up whenever a new .dat file is detected and I have to close each of them in order to continue to next .dat file.
In general, I would like to do an automatic scan in a folder, get the names of the datafiles, and write a command line to call these .dat files in one line.
Any ideas/helps appreciated!!!
try this, remove the word echo if the output is OK:
#echo off &setlocal enabledelayedexpansion
set "line="
for %%i in (*.dat) do set "line=!line! "%%i""
echo "D:\Softwares\Analyzer.exe" %line% TestAnalysis.pdo
The code doesn't work with *dat files with exclams ! in the file name. This may be fixed if needed.
Hey guys, I'm looking for a batch file to tell me if certain folders have been modified today (I'll run it every morning). I'm happy to specify each of the folders to be to be queried, I just haven't been able to find anything that meets my requirements yet. If anyone knows off the top of their head what the code for the .bat would be, that would be awesome :) Thanks in advance.
Here's a batch file that should do the trick:
#echo off
setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
for /D %%Q IN (*.*) DO (
set FILETIME=%%~tQ
if "!FILETIME:~0,10!"=="%DATE:~4%" echo %%Q
)
This works by comparing the date-part of the file timestamp against the current date.
As written, it checks directories in the current directory, but you could replace *.* with whatever filespec you want to test (or pass it in as an argument).
I don't know if this will work on systems with anything other default English/US regional settings, but it could probably be tweaked to make it work, if it doesn't already. I also don't know what would happen if the system is shared between users in different time-zones.
you can use a vbscript
Set objFS = CreateObject( "Scripting.FileSystemObject" )
strFolder = WScript.Arguments(0)
Set objFolder = objFS.GetFolder(strFolder)
If DateDiff("d", Now, objFolder.DateLastModified ) = 0 Then
WScript.Echo "0"
End If
in your batch file (or command line)
C:\test>cscript //nologo test.vbs myFolderName
use a for loop to catch the output. (Or you can even do everything in vbscript )
Project moving forwards, I can see why creating .bat files to do things can become addictive!
I can now save somefile.txt at regular intervals, I then rename somefile.txt by adding the time and date to create a unique file name
ren somefile.txt somefile_%time:~0,2%%time:~3,2%-%date:~-10,2%%date:~3,2%%date:~-4,4%.txt
As an example, the code above has just renamed somefile.txt to somefile_1317_13022011.txt (1317hrs on 13th February 2011)
I ran
ren somefile.txt somefile_%time:~0,2%%time:~3,2%-%date:~-10,2%%date:~7,2%%date:~-4,4%.txt
yesterday, it ran successfully until midnight, and then it crashed (syntax error) although it was saving as 12012011 for the date (12th Jan 2011) instead of the correct date of 12022011.
Will the current version ran past midnight? Am I confusing myself with UK vs US date format?
Animuson gives a decent way to do it, but no help on understanding it. I kept looking and came across a forum thread with this commands:
Echo Off
IF Not EXIST n:\dbfs\doekasp.txt GOTO DoNothing
copy n:\dbfs\doekasp.txt n:\history\doekasp.txt
Rem rename command is done twice (2) to allow for 1 or 2 digit hour,
Rem If before 10am (1digit) hour Rename starting at location (0) for (2) chars,
Rem will error out, as location (0) will have a space
Rem and space is invalid character for file name,
Rem so second remame will be used.
Rem
Rem if equal 10am or later (2 digit hour) then first remame will work and second will not
Rem as doekasp.txt will not be found (remamed)
ren n:\history\doekasp.txt doekasp-%date:~4,2%-%date:~7,2%-%date:~10,4%_#_%time:~0,2%h%time:~3,2%m%time:~6,2%s%.txt
ren n:\history\doekasp.txt doekasp-%date:~4,2%-%date:~7,2%-%date:~10,4%_#_%time:~1,1%h%time:~3,2%m%time:~6,2%s%.txt
I always name year first YYYYMMDD, but wanted to add time. Here you will see that he has given a reason why 0,2 will not work and 1,1 will, because (space) is an invalid character. This opened my eyes to the issue. Also, by default you're in 24hr mode.
I ended up with:
ren Logs.txt Logs-%date:~10,4%%date:~7,2%%date:~4,2%_%time:~0,2%%time:~3,2%.txt
ren Logs.txt Logs-%date:~10,4%%date:~7,2%%date:~4,2%_%time:~1,1%%time:~3,2%.txt
Output:
Logs-20121707_1019
Digging up the old thread because all solutions have missed the simplest fix...
It is failing because the substitution of the time variable results in a space in the filename, meaning it treats the last part of the filename as a parameter into the command.
The simplest solution is to just surround the desired filename in quotes "filename".
Then you can have any date pattern you want (with the exception of those illegal characters such as /,\,...)
I would suggest reverse date order YYYYMMDD-HHMM:
ren "somefile.txt" "somefile-%date:~10,4%%date:~7,2%%date:~4,2%-%time:~0,2%%time:~3,2%.txt"
following should be your right solution
ren somefile.txt somefile_%time:~0,2%%time:~3,2%-%DATE:/=%.txt
I took the above but had to add one more piece because it was putting a space after the hour which gave a syntax error with the rename command.
I used:
set HR=%time:~0,2%
set HR=%Hr: =0%
set HR=%HR: =%
rename c:\ops\logs\copyinvoices.log copyinvoices_results_%date:~10,4%-%date:~4,2%-%date:~7,2%_%HR%%time:~3,2%.log
This gave me my format I needed:
copyinvoices_results_2013-09-13_0845.log
problem in %time:~0,2% can't set to 24 hrs format, ended with space(1-9), instead of 0(1-9)
go around with:
set HR=%time:~0,2%
set HR=%Hr: =0% (replace space with 0 if any <has a space in between : =0>)
then replace %time:~0,2% with %HR%
good luck
ls | xargs -I % mv % %_`date +%d%b%Y`
One line is enough.
ls all files/dirs under current dir and append date to each file.
I tried to do the same:
<fileName>.<ext> --> <fileName>_<date>_<time>.<ext>
I found that :
rename 's/(\w+)(\.\w+)/$1'$(date +"%Y%m%d_%H%M%S)'$2/' *