I have a list of files in a adls container which contain date in the name as given below:
TestFile-Name-20221120. csv
TestFile-Name-20221119. csv
TestFile-Name-20221118. csv
and i want to copy files which contain today date only like TestFile-Name-20221120. csv on today and so on.
I've used get metedata activity to get list of files and then for each to iterate over each file and then used set variable to extract name from the file like 20221120 but not sure how to proceed further.
We have something similar running. We check an SFTP folder for the existanc e of files, using the Get Metadata activity. In our case, there can be folders or files. We only want to process files, and very specific ones for that matter (I.e. we have 1 pipeline per filename we can process, as the different filenames would contain different columns/datatypes etc).
Our pipeline looks like this:
Within our Get Metadata component, we basically just filter for the name of the object we want, and we only want files ending in .zip, meaning we added a Filename filter:
:
In your case, the first part would be 'TestFile-Name-', and the second part would be *.csv'.
We then have a For Each loop set up, to process anything (the child items) we retrieved in the Get Metadata step. Within the For Each we defined an If Condition to only process files, and not folders.
In our cases, we use the following expression:
#equals(item().type, 'File')
In your case, you could use something like:
#endsWith(item().name, concat(<variable containing your date>, '.csv'))
Assuming all the file names start with TestFile-Name-,
and you want to copy the data of file with todays date,
use get metadata activity to check if the file exists and the file name can be dynamic like
#concat('TestFile-Name-',utcnow(),'.csv')
Note: you need to fromat utcnow as per the needed format
and if file exists, then proceed for copy else ignore
I have a folder say, source folder, containing 1000+ xml files with some ambiguous names, like:
_MIM_15646432635_6664684
_MIM_54154548557_6568436 etc.
Out of these thousands of XML files I’ve to select some 10-12 xml files with a particular node in them and move them to another folder (destination folder) and rename the files respectively with some meaningful names.
For example:
There is an xml file with name _MIM_15646432635_6664684 and it contains a node pattern like: “bab6e7h835468eg” and I’ve to rename it to name like: {1FE9909E-4450-B98665362022}
So for this I’ve to write a script which will search the file in source folder and if it finds my desired node pattern then move this file to destination folder post renaming it to some meaningful name.
Provided I’ve an excel sheet where I do have a list of two columns, one containing the specific node pattern and column two has a respective new name list.
Currenty, I’ve a script which can search a file and move it to another folder provided I’m giving input to the script with the node pattern and file name from that excel sheet:
Select-String – Path “\Dubwta01\AIR\Invalid*.xml” – Pattern ‘bab6e7h835468eg’ | %{Copy-Item – Path $_.Path – Destination ‘\Dubwta01\AIR\Invalid{1FE9909E-4450-B98665362022}.xml’
What now I need is that a new script which will pick all 10 files from source folder and move it to destination folder by renaming it and I don’t have to hard code the pattern and new name in script rather it shall fetch the details from that excel or I can save the details in text files whatever is suitable for script to pick the name and pattern from.
I have a mapping file in import.CSV as following:
Name|Business
Jack|Carpenter
Rose|Secretary
William|Clerk
Now, I have a directory which contains files like
90986883#Jack#Sal#1000.dat
76889992#Rose#Sal#2900.dat
67899279#William#Sal#1900.dat
12793298#Harry#Sal#2500.dat
Please note #Sal will always be there after Name. I need to pick these files and put into another directory and end result in second directory should look like.
90986883#Carpenter#Sal#1000.dat
76889992#Secretary#Sal#2900.dat
67899279#Clerk#Sal#1900.dat
Basically Files need to be renamed based upon CSV file and if Name is not there in file name , then there is no action required. Please note that source file should not be changed.
I will appreciate all kind of help.
I'd like to add a specific field from a PDF into its file name in a batch file.
For example, there is a date on page 1 in the PDF file and I'd like to include that date in the name. How do I create the batch file?
In the following screenshot:
Good day,
In Kettle Pentaho, if use Move files..., then we can edit the file name in the destination, like specify date time format, so the file like abc.txt will become abc02012015.txt in the destination.
However, in Copy files..., I cant see this option. Anyway to do the same things as Move files...?
You can try "Copy or Move result filenames" Step. Here you can append a date pattern while copying files.
Hope it helps :)
You can check "Destination is file" in Settings in Copy file step and then in "Destination File/Folder" write folder with specify filename