I just starting with DataStage, developing a project to process files I'm getting an strange error when the variable is mapped from a sequence job to parallel job, if I check the path in the error message the path exist and the file is there.. Any insight that can help to solve this problem. I'd really appreciate.. Below you can see with the pictures how the job is implemented and the error.
Thanks in Advance
The white space between your file path and the : in the error message are suspicious. Copy the file path directly from the log and paste into an editor that you can do character examination. You are likely having whitespace issues when looking to the path.
Please indicate what you are doing to the file path in the User Variables activity. The log appears to contain a valid Windows pathname, and reports "no such file or directory". Can you please validate that the file does indeed exist at that location? If it does not, then your ls | awk | tr pipeline or your sequence parameter TestFile_$DestinationPath may need some work.
I ran into the same situation where the file name being passed from the sequencer to the job looks to resolve correctly, but it gives the same error of file not found. Even though I copy the resolved name (taken from job log of the job) and open that path/file, it opens successfully. However, you do not see that DS is adding an extra "space" to the end of the resolved parameter value in the job. If you were to switch the input file stage from specific file to file pattern and just add * to the end of your file name, you will see that DS is trying to resolve the parameter value plus a space plus an *. For example, your sequencer is passing file name as "D:\abc.txt", but the job will see it as "D:\abc.txt *".
Actual job log entry of the job is:
ls: File or directory "D:\abc.txt *" is not found
The fix that worked for me is to do an Exchange of "0D" to "FF" in the User Variable stage. For some reason, DS was adding a carriage return in the User Variable stage. I wrote the file name to a text file to see what the value is being passed along the way and it always showed up as the correct value with no special characters added throughout my sequencer. By adding this Exchange in the User Variable expression, it worked for me as what you are expecting (passing the true file name value to the next job and able to read the file successfully). There is no need to actually switch to file pattern, but this was needed for debugging purposes to see the value actually being resolved in the job.
Related
I'm trying to (ultimately) modify a batch of files but getting stuck in the basics as I try to modify a single file before running a batch command.
If someone could help me troubleshoot the command I'm inputting, that would be fantastic. I'm sure it's something very simple.
Thanks a lot for any help you can provide!
Here's the abbreviated image exif data:
-ExifToolVersion=10.10
-FileName=2018_11_13_1.jpeg
-Directory=.
-FileSize=2.8 MB
-FileModifyDate=2019:07:12 15:40:38-07:00
-FileAccessDate=2019:07:12 15:40:38-07:00
-FileInodeChangeDate=2019:07:23 10:38:02-07:00
-FilePermissions=rw-rw-r--
-FileType=JPEG
-FileTypeExtension=jpg
-MIMEType=image/jpeg
[...]
-ModifyDate=2018:11:13 12:00:53
[...]
-DateTimeOriginal=2018:11:13 12:00:53
-CreateDate=2018:11:13 12:00:53
My current input is: exiftool "-FileModifyDate<$filename00000" ./2018_11_13_1.jpeg
And the error message is:
Warning: No writable tags set from 2018_11_13_1.jpeg
0 image files updated
1 image files unchanged
And the exif data is, of course, unchanged.
I've confirmed that I can write a value to this tag, so there's definitely something going wrong in pulling from the filename.
( Continued from How to compensate for incomplete date/time info in filename )
The problem here is that you are trying to write from a tag named filename00000. If you check the example in the other post, you will see that there is a space after Filename. This sets it apart so that exiftool knows which is a tag name and which is other data.
There is possibly an additional problem here, though. Your filename has an extra number that is not the date. When exiftool tries to write the time stamp from the filename, it is going to end up with a value of "2018:11:13 10:00:00", which might become especially problematic if that last digit hits a value of 3 or more, resulting in a timestamp of "2018:11:13 30:00:00".
I would suggest using exiftool's Advanced Formatting Feature (a fancy way of saying that you can use perl code in the command) to strip the excess data. Something like
exiftool "-FileModifyDate<${filename;s/^(.*\d{4}_\d\d_\d\d).*/$1/} 000000" ./2018_11_13_1.jpeg
Though take note, if the filenames are in any other format, then it would require a different command.
Attempting to write a Wget to get and save Vanguard pricing data. So far I have 2 statements that both work correctly from the Command Line when I paste the string. When I save the string as a bat file one works and the other gives an unexpected result.
The string that works correctly in both places is:
Wget64 --output-document=C:\Users\Default\downloads\VVA_Daily_Portfolio-%DATE:~-4%-%DATE:~4,2%-%DATE:~7,2%.html "https://personal.vanguard.com/us/funds/annuities/variable"
The string that only works in the Command Line and not as a bat file is:
Wget64 --output-document=C:\Users\Default\downloads\VVA_Fund64_History-%DATE:~-4%-%DATE:~4,2%-%DATE:~7,2%.html "https://personal.vanguard.com/us/funds/tools/pricehistorysearch?radio=1&results=get&FundType=VVAP&FundIntExt=INT&FundId=0064&fundName=0064&fundValue=0064&radiobutton2=1&beginDate=03%2F01%2F2017&endDate=12%2F31%2F2017&year=#res"
Can someone help me write the script so that the expected result is achieved. I suspect that the Vanguard website can tell the difference between a Command Line vs bat file query, or that there is something inherently different between the two methods of execution.
ANy assistance is appreciated. Dan
The cmd command parser behaves differently in command line and batch files. In this case, the main problem is the variable expansion. In command line when a variable does not contain a value (it is undefined), the variable read operation is not removed, but inside batch files the read operation is removed.
That means that something like echo(%thisDoesNotExist% will output (under the assumption the variable does not exist) %thisDoesNotExist% in command line and nothing in batch file.
What relation has this with your problem?
If we split your wget in parts you have
Wget64
--output-document="C:\Us ... y-%DATE:~-4%-%DATE:~4,2%-%DATE:~7,2%.html"
^........^ ^.........^ ^.........^
"https://pe ... h?radio=1& ... &beginDate=03%2F01%2F2017&endDate=12%2F31%2F2017&year=#res"
^....^ ^....^
You can see where the parser tries to resolve variables, correctly in the output case and incorrectly (from the purpouse of the command point of view) in the URL.
You need to escape (by doubling them) the percent signs that are not part of a variable read operation, ex. ... beginDate=03%%2F01%%2F2017&...
Recently I have lost my database files after server (Qnap) reboot.
I need to recover the MYD files.
I'm using the Photorec to recover my files which uses file signatures.
I know that MYD files doesn't have common file signature.
I've recreated the database from the earlier backup (on local machine) just to check if the signature changes when new data inserted and it seams like the signature bit is not changing.
I'm not sure if that means I have the signature of the file that I'm looking for.
I've tried to create a photorec.sig file and entered MYD 0 0x00010065fe00000100003101aa70010f00 which are the first 34 chars of the hexdump result.
I've saved this file into the current working directory and ~/.photorec.sig and tried to recognize the MYD file with fidentify_static which always returns unknown.
I think there is something wrong with the signature file that I've created. Can anyone tell me how to make sure that the signature file is created correctly and placed to the right place on qnap?
Run fidentify with the file/directory as the parameter:
[kmaster#adsl ~]$ fidentify /home/kmaster/src/testfiles/sample.pfi
See http://www.cgsecurity.org/wiki/Add_your_own_extension_to_PhotoRec
I'd like to append the relevant fail2ban log entry to the notification email I already receive for any given incident.
Does anybody know how this can be done?
It depends on what information you would like - you may edit the appropriate action.d configuration file's actionban segment by copying the .conf version to a .local version which will override the .conf version as per the fail2ban documentation, and edit it to include whatever information you would like. For example, I have personally amended my sendmail-whois.conf (which is the main sendmail action I use - you could do likewise with sendmail.conf however if you use that for example) by copying it to sendmail-whois.local which I then edited to include the server hostname on the 'From:' line.
You could also include commands to be executed with their output passed to the email to be sent, as long as you follow the correct syntax and fully qualify the path to the relevant commands - for example, you will see that the sendmail-whois action configuration contains the line, within the actionban segment;
`/usr/bin/whois <ip>`\n
Note, as I have mentioned above - the full path to the relevant command is included (in this case, for whois), and the entire command with its options must be delimited by backquotes. the \n at the end of the line indicates that a new line be printed following this one in the output.
Hope that clarifies things for you!
I am quite stuck; I need to compress the content of a folder, where I have multiple files (extension .dat). I went for shell scripting.
So far I told myself that is not that hard: I just need to recursively read the content of the dir, get the name of the file and zip it, using the name of the file itself.
This is what I wrote:
for i in *.dat; do zip $i".zip" $i; done
Now when I try it I get a weird behavior: each file is called like "12/23/2012 data102 test1.dat"; and when I run this sequence of commands; I see that zip instead of recognizing the whole file name, see each part of the string as single entity, causing the whole operation to fail.
I told myself that I was doing something wrong, and that the i variable was wrong; so I have replaced echo, instead than the zip command (to see which one was the output of the i variable); and the $i output is the full name of the file, not part of it.
I am totally clueless at this point about what is going on...if the variable i is read by zip it reads each single piece of the string, instead of the whole thing, while if I use echo to see the content of that variable it gets the correct output.
Do I have to pass the value of the filename to zip in a different way? Since it is the content of a variable passed as parameter I was assuming that it won't matter if the string is one or has spaces in it, and I can't find in the man page the answer (if there is any in there).
Anyone knows why do I get this behavior and how to fix it? Thanks!
You need to quote anything with spaces in it.
zip "$i.zip" "$i"
Generally speaking, any variable interpolation should have double quotes unless you specifically require the shell to split it into multiple tokens. The internal field separator $IFS defaults to space and tab, but you can change it to make the shell do word splitting on arbitrary separators. See any decent beginners' shell tutorial for a detailed account of the shell's quoting mechanisms.