I am using Matlab to print a small text file (temp_script.exec) that will be used to run GrADS commands. The script looks like the following:
'reinit'
'open temp_ctl.ctl'
'set lon -100 -80'
'set lat 20 30'
'define prc = var'
'set sdfwrite data_out.nc'
'sdfwrite prc'
The script is called via cshell:
#!/bin/csh -f
grads -lbc << EOF
temp_script.exec
EOF
exit
The script seems to execute properly, but the output (data_out.nc) is not generated. Strangely, if I edit it using VI and replace the first character -- the single quotation before the command "reinit" -- by typing another single quotation, then re-run the script, data is generated properly.
My question is, what could be different? The scripts look identical in several different text editors, but the "modified" script (by typing) is 1 byte larger. I am using the "fprintf" function to generate the single quotes in Matlab. Could it be some problem with that function?
Thanks for reading.
To see if the files are really the same (the generated one and the one edited with vi):
od -c -t x1 temp_script.exec > temp_script.lis
od -c -t x1 vi_script.exec > vi_script.lis
diff exec_script.lis vi_script.lis
There could be a UNICODE BOM at the beginning of the file, or a missing newline at the end of file that is causing your issue.
Related
How do I get the q script output to be displayed in QPad64's (QInsightPad's) output window. I am presently starting q and QPad 64 with the following .bat file :
set QHOME=C:\Q\q
set QINIT=C:\code\server.q
set PATH=%PATH%;%QHOME%;%QHOME%\w32
START C:\QInsightPad-2.2_FREE-x64\qpad64.exe
Result: q opens in cmd window, but all output from scipts run in Qpad show up there ( in cmd ) instead of in Qpad's output window. How do I fix this ?
Qpad isn't really designed to be used like that. You could redirect the stdout and read it in to view in qpad.
\1 qproc.log
Your scripts could then have output to the log as certain things are done like so:
1"tables_loaded"
// \n for new line
1"\nfuncs_loaded"
read0 `:qproc.log
"tables_loaded"
"funcs_loaded"
I am writing a data import script in Tcl (from SQL Server to Postgres) and have to call command line unix tr to scrub out null characters in a data file. I write the data to a temp file and then use exec to process the file through tr.
The tr call I would like Tcl to generate looks like this on the command line:
tr -d '\000' < blah >blah.notnull
The Tcl code I use to make the above is this, with $STATE(TMP) holding the temp file:
set ret [catch {exec tr -d '\\000' < $STATE(TMP) > $STATE(TMP).clean}]
However, sometimes this doesn't work and the PostgreSQL COPY fails because of x00 characters. If I run the command line version on the file, then COPY succeeds.
Could someone help me out understanding the exec call and quoting and backslashes? I am a bit stumped.
The error message, a reformatted version of the PG error:
Problem with COPY on blahblah: PGRES_FATAL_ERROR, ERROR: invalid byte sequence for encoding "UTF8": 0x00
Annoyingly, the Tcl exec code often works, but not always.
(We are hand rolling an import system using Tcl, Linux, BCP, SQL server, etc. beause all the off-the-shelf tools fail with the size of our data.)
Thanks to all who read or answer!
The thing is that Tcl doesn't ascribe any special meaning at all to single quotes. The equivalent in Tcl is braces, so use {\000} instead of '\000'. With what you wrote, you were sending three characters (a ', a NUL, and another ') in as that argument, and that causes all sorts of trouble since literal NUL characters don't go well as C strings.
Thus, you should be doing:
exec tr -d {\000} < blah >blah.notnull
or:
set ret [catch {
exec tr -d {\000} < $STATE(TMP) > $STATE(TMP).clean
}]
Tcl can also do that operation directly.
# Read binary data
set f [open $STATE(TMP) "rb"]
set data [read $f]
close $f
# Write transformed binary data
set f [open $STATE(TMP).clean "wb"]
puts -nonewline $f [string map [list \u0000 ""] $data]
close $f
[EDIT]: When the amount of data being transformed is large, it's better to do a bit at a time.
set fIn [open $STATE(TMP) "rb"]
set fOut [open $STATE(TMP).clean "wb"]
while true {
# 128kB chunk size; a bit arbitrary, but big enough to be OK
set data [read $fIn 131072]
# If we didn't read anything and instead got EOF, stop the loop
if {[eof $fIn]} break
puts -nonewline $fOut [string map [list \u0000 ""] $data]
}
close $fIn
close $fOut
You could also use a Tcl 8.6 channel transform to do the work and then fcopy to move things over, but there wouldn't be much difference in performance.
-- Edit : Resolved. See answer.
Background:
I'm writing a shell that will perform some extra actions required on our system when someone resizes a database.
The shell is written in ksh (requirement), the OS is Solaris 5.10 .
The problem is with one of the checks, which verifies there's enough free space on the underlying OS.
Problem:
The check reads the df -k line for root, which is what I check in this step, and prints it to a file. I then "read" the contents into variables which I use in calculations.
Unfortunately, when I try to run an arithmetic operation on one of the variables, I get an error indicating it is null. And a debug output line I've placed after that line verifies that it is null... It lost it's value...
I've tried every method of doing this I could find online, they work when I run it manually, but not inside the shell file.
(* The file does have #!/usr/bin/ksh)
Code:
df -k | grep "rpool/ROOT" > dftest.out
RPOOL_NAME=""; declare -i TOTAL_SIZE=0; USED_SPACE=0; AVAILABLE_SPACE=0; AVAILABLE_PERCENT=0; RSIGN=""
read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN < dftest.out
\rm dftest.out
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=$TOTAL_SIZE/1024))
This is the result:
DBResize.sh[11]: TOTAL_SIZE=/1024: syntax error
I'm pulling hairs at this point, any help would be appreciated.
The code you posted cannot produce the output you posted. Most obviously, the error is signalled at line 11 but you posted fewer than 11 lines of code. The previous lines may matter. Always post complete code when you ask for help.
More concretely, the declare command doesn't exist in ksh, it's a bash thing. You can achieve the same result with typeset (declare is a bash equivalent to typeset, but not all options are the same). Either you're executing this script with bash, or there's another error message about declare, or you've defined some additional commands including declare which may change the behavior of this code.
None of this should have an impact on the particular problem that you're posting about, however. The variables created by read remain assigned until the end of the subshell, i.e. until the code hits a ), the end of a pipe (left-hand side of the pipe only in ksh), etc.
About the use of declare or typeset, note that you're only declaring TOTAL_SIZE as an integer. For the other variables, you're just assigning a value which happens to consist exclusively of digits. It doesn't matter for the code you posted, but it's probably not what you meant.
One thing that may be happening is that grep matches nothing, and therefore read reads an empty line. You should check for errors. Use set -e in scripts to exit at the first error. (There are cases where set -e doesn't catch errors, but it's a good start.)
Another thing that may be happening is that df is splitting its output onto multiple lines because the first column containing the filesystem name is too large. To prevent this splitting, pass the option -P.
Using a temporary file is fragile: the code may be executed in a read-only directory, another process may want to access the same file at the same time... Here a temporary file is useless. Just pipe directly into read. In ksh (unlike most other sh variants including bash), the right-hand side of a pipe runs in the main shell, so assignments to variables in the right-hand side of a pipe remain available in the following commands.
It doesn't matter in this particular script, but you can use a variable without $ in an arithmetic expression. Using $ substitutes a string which can have confusing results, e.g. a='1+2'; $((a*3)) expands to 7. Not using $ uses the numerical value (in ksh, a='1+2'; $((a*3)) expands to 9; in some sh implementations you get an error because a's value is not numeric).
#!/usr/bin/ksh
set -e
typeset -i TOTAL_SIZE=0 USED_SPACE=0 AVAILABLE_SPACE=0 AVAILABLE_PERCENT=0
df -Pk | grep "rpool/ROOT" | read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=TOTAL_SIZE/1024))
Strange...when I get rid of your "declare" line, your original code seems to work perfectly well (at least with ksh on Linux)
The code :
#!/bin/ksh
df -k | grep "/home" > dftest.out
read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN < dftest.out
\rm dftest.out
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=$TOTAL_SIZE/1024))
print $TOTAL_SIZE
The result :
32962416 5732492 25552588 19% /home
5598
Which are the value a simple df -k is returning. The variables seem to last.
For those interested, I have figured out that it is not possible to use "read" the way I was using it.
The variable values assigned by "read" simply "do not last".
To remedy this, I have applied the less than ideal solution of using the standard "while read" format, and inside the loop, echo selected variables into a variable file.
Once said file was created, I just "loaded" it.
(pseudo code:)
LOOP START
echo "VAR_A="$VAR_A"; VAR_B="$VAR_B";" > somefile.out
LOOP END
. somefile.out
I need to prepare list of strings for translation of my iPhone application.
I have extracted strings from *.m files using genstring and from the XIB files using ibtool command.
But I have also lots of texts to translate in plist files (String field types enclosed in string tag).
Is there a nice bash script / command to extract those strings into a flat txt file?
I could review and filter it so my translators can work with nice list but not with alien looking XML file.
I made a custom shell script which tries to figure out the values needed. You can then use the localize.py script in a modified way (see below) to automatically create the translation files. (The line break where somehow very important) If there more entities to be translated, the shell script can be modified accordingly
#!/bin/bash
rm -f $2
sed -n 'N;/<key>Title<\/key>/{N;/<string>.*<\/string>/{s/.*<string>\(.*\)<\/string>.*/\/* \1 *\/\
"\1" = "\1";\
/p;};}' $1 >> $2
sed -n 'N;/<key>FooterText<\/key>/{N;/<string>.*<\/string>/{s/.*<string>\(.*\)<\/string>.*/\/* \1 *\/\
\"\1" = "\1";\
/p;}
;}' $1 >> $2
sed -n 'N;/<key>Titles<\/key>/{N;/<array>/{:a
N;/<\/array>/!{
/<string>.*<\/string>/{s/.*<string>\(.*\)<\/string>.*/\/* \1 *\/\
\"\1" = "\1";\
/p;}
ba
;};};}' $1 >> $2
the localize.py script needed some modification. Therefore I created a small package containing the localizer for the source code and for the plist Files. The new script even supports Duplikates (meaning it will kick them)
We recently made a small online application to do that, please take a look on: http://www.icapps.be/plist-translator/
I can't think of any command off the top of my head. However, plists are glorified xml files and there are various parsers available for them.
It shouldn't be too difficult to create a simple python script to get all the strings from the file.
Does this help?
http://www.icanlocalize.com/site/tutorials/how-to-translate-plist-files/
We much prefer paying clients who use our translation system with our translators, but you can translate yourself in our GUI at no charge.
I have a file full of ascii data. How would I append a string to the first line of the file? I cannot find that sort of functionality using fopen (it seems to only append at the end and nothing else.)
The following is a pure MATLAB solution:
% write first line
dlmwrite('output.txt', 'string 1st line', 'delimiter', '')
% append rest of file
dlmwrite('output.txt', fileread('input.txt'), '-append', 'delimiter', '')
% overwrite on original file
movefile('output.txt', 'input.txt')
Option 1:
I would suggest calling some system commands from within MATLAB. One possibility on Windows is to write your new line of text to its own file and then use the DOS for command to concatenate the two files. Here's what the call would look like in MATLAB:
!for %f in ("file1.txt", "file2.txt") do type "%f" >> "new.txt"
I used the ! (bang) operator to invoke the command from within MATLAB. The command above sequentially pipes the contents of "file1.txt" and "file2.txt" to the file "new.txt". Keep in mind that you will probably have to end the first file with a new line character to get things to append correctly.
Another alternative to the above command would be:
!for %f in ("file2.txt") do type "%f" >> "file1.txt"
which appends the contents of "file2.txt" to "file1.txt", resulting in "file1.txt" containing the concatenated text instead of creating a new file.
If you have your file names in strings, you can create the command as a string and use the SYSTEM command instead of the ! operator. For example:
a = 'file1.txt';
b = 'file2.txt';
system(['for %f in ("' b '") do type "%f" >> "' a '"']);
Option 2:
One MATLAB only solution, in addition to Amro's, is:
dlmwrite('file.txt',['first line' 13 10 fileread('file.txt')],'delimiter','');
This uses FILEREAD to read the text file contents into a string, concatenates the new line you want to add (along with the ASCII codes for a carriage return and a line feed/new line), then overwrites the original file using DLMWRITE.
I get the feeling Option #1 might perform faster than this pure MATLAB solution for huge text files, but I don't know that for sure. ;)
How about using the frewind(fid) function to take the pointer to the beginning of the file?
I had a similar requirement and tried frewind() followed by the necessary fprintf() statement.
But, warning: It will overwrite on whichever is the 1st line. Since in my case, I was the one writing the file, I put a dummy data at the starting of the file and then at the end, let that be overwritten after the operations specified above.
BTW, even I am facing one problem with this solution, that, depending on the length(/size) of the dummy data and actual data, the program either leaves part of the dummy data on the same line, or bring my new data to the 2nd line..
Any tip in this regards is highly appreciated.