I am trying to run a bash command like this one in scala:
cat "example file.txt" | grep abc
Scala has a special syntax for process piping, so this was my first approach:
val filename = "example file.txt"
(Process(Seq("cat", filename)) #| Process(Seq("grep", "abc"))).run()
As far as I understand, this executes the first process, reads the output back to scala and feeds it to the second process. The problem is that, for performance reasons, I would like to execute both processes without leaving the terminal. The file is huge and I don't need the whole output, that's why I am using grep in the first place. So, my second approach was this one:
val filename = "example file.txt"
(Process(Seq("bash", "-c", "cat " + filename + " | grep abc"))).run()
The problem here is that it breaks if the filename has spaces. I could try to escape spaces, but I would rather have scala doing it for me (there are many other characters that I would also need to escape).
Is there another way to run this command?
It is easy enough to escape the filename:
val escapedFilename = "'" + filename.replace("'", "'\\''") + "'"
But the right way to do this is to pass the filename directly to grep:
Process(Seq("grep", "abc", filename)).run()
Which is equivalent to this in the shell:
grep abc "example file.txt"
Related
I would like to join strings for file name and shell script so I can run the command line in kdb for ftp transfer.
But there are quotations in quotation marks. not sure how to add / in there.
This is the code I have:
host:"abc.com";
usr:"def";
path:"get /home/eddie/abc.csv /home/terry/";
cmd:" " sv ("/home/kdb/eddie/ftp.sh";host;usr;path);
system cmd;
So the path will not have quotation mark and will be running error. How can I solve this problem?
You can escape quotes with \ e.g. "\"Matt\"" but I don't think that's your issue. It looks like you are attempting to use get in the system command. This is a kdb keyword and your OS will not recognise it. You should just be passing the location of the csv to your ftp script.
Edit:
You may also need sh in the system command.
cat test.sh
echo $1
system "test.sh hello"
sh: ./test.sh: Permission denied
'os
system "sh test.sh hello"
"hello"
Assuming that you simply want quotes within a string, it may just be as simple as using .Q.s1 aka -3!, see https://code.kx.com/q/ref/dotq/#qs1-string-representation
q)" " sv ("/home/kdb/eddie/ftp.sh";host;usr;.Q.s1 path)
"/home/kdb/eddie/ftp.sh abc.com def \"get /home/eddie/abc.csv /home/terry/\""
q)" " sv ("/home/kdb/eddie/ftp.sh";host;usr;-3!path)
"/home/kdb/eddie/ftp.sh abc.com def \"get /home/eddie/abc.csv /home/terry/\""
I'm working with a command line utility that requires passing the name of a file to write output to, e.g.
foo -o output.txt
The only thing it writes to stdout is a message that indicates that it ran successfully. I'd like to be able to pipe everything that is written to output.txt to another command line utility. My motivation is that output.txt will end up being a 40 GB file that I don't need to keep, and I'd rather pipe the streams than work on massive files in a stepwise manner.
Is there any way in this scenario to pipe the real output (i.e. output.txt) to another command? Can I somehow magically pass stdout as the file argument?
Solution 1: Using process substitution
The most convenient way of doing this is by using process substitution. In bash the syntax looks as follows:
foo -o >(other_command)
(Note that this is a bashism. There's similar solutions for other shells, but bottom line is that it's not portable.)
Solution 2: Using named pipes explicitly
You can do the above explicitly / manually as follows:
Create a named pipe using the mkfifo command.
mkfifo my_buf
Launch your other command with that file as input
other_command < my_buf
Execute foo and let it write it's output to my_buf
foo -o my_buf
Solution 3: Using /dev/stdout
You can also use the device file /dev/stdout as follows
foo -o /dev/stdout | other_command
Named pipes work fine, but you have a nicer, more direct syntax available via bash process substitution that has the added benefit of not using a permanent named pipe that must later be deleted (process substitution uses temporary named pipes behind the scenes):
foo -o >(other command)
Also, should you want to pipe the output to your command and also save the output to a file, you can do this:
foo -o >(tee output.txt) | other command
For the sake of making stackoverflow happy let me write a long enough sentence because my proposed solution is only 18 characters long instead of the required 30+
foo -o /dev/stdout
You could use the magic of UNIX and create a named pipe :)
Create the pipe
$ mknod -p mypipe
Start the process that reads from the pipe
$ second-process < mypipe
Start the process, that writes into the pipe
$ foo -o mypipe
foo -o <(cat)
if for some reason you don't have permission to write to /dev/stdout
I use /dev/tty as the output filename, equivalent to using /dev/nul/ when you want to output nothing at all. Then | and you are done.
I need help with a very specific question. I need to replace the following UNIX-line to a windows equivalent without installation of third-party software.
awk -F ";" '$6 ~/2019-03-11/ {print}' myInputFile.csv >> myOutputFile.csv
What the line does:
It scans myInputFile.csv, where the column delimiter is a ;, for the text 2019-03-11 in the 6th column and appends the line to myOutputFile.csv
Unfortunately, I cannot install gawk or anything like it on the client's machine.
I haven't used a windows machine for many years and am totally lost. :-(
I believe what you are after is the following in Powershell:
$ Get-Content .\myInputFile.csv | %{ if ($_.Split(';')[5] -match "2019-03-11") { $_; } } >> .\myOutputFile.csv
I will not put my hands in fire for the outcome of this.
If you want to call this from a bat-file, you have to do some ugly-ness.
findstr.exe has rather limited RegEx capabilities, but this should be sufficient:
findstr "^[^;][^;]*;[^;][^;]*;[^;][^;]*;[^;][^;]*;[^;][^;]*;2019-03-11" <myInputFile.csv >>myOutputFile.csv
Another pure cmdline way (provided there are at least 7 columns):
For /F "tokens=1-6* delims=;" %A in ('findstr "2019-03-11" ^<myInputFile.csv') do #if "%F"=="2019-03-11" (>>myOutputFile.csv Echo %A;%B;%C;%D;%E;%F;%G)
In a batch file you'll have to double all percent signs.
On Command Prompt you can create a text file inline by doing this:
copy con file.txt
Hello World
^Z
Or:
type con > file.txt
Hello World
^Z
Is there an equivalent command in Powershell? Neither of the two commands I listed above work.
Pipe content to the out-file cmdlet to emulate this.
"Hello World" | out-file hello.txt
To get multiple lines, open the quotes but don't close them right away
"hello
>> is it me you're looking for" | out-file hello2.txt
The >> will appear on the second line after hitting enter
Another way is using "here-strings" for this instead of opening quotes.
#'
hello
is it me you're looking for?
I can even use ' # $blabla etc in here
and no substitutes will be placed
You want var substitutes, use #"..."# instead of #'...'#
'# | out-file hello.txt
I currently have a directory of files with date based names in the format mmddyy:
nxd060815a.html
nxd060915a.html
nxd061015a.html
and would like to change file name format to yyyy-mm-dd:
nxd2015-06-08a.html
nxd2015-06-09a.html
nxd2015-06-10a.html
How would I do this from the command line?
Any help is appreciated.
ls | awk '{ system("mv " $0 " " substr($0,1,3) "20" substr($0,8,2) "-" substr($0,4,2) "-" substr($0,6,2) substr($0,10,6))}'
run the above from the dir the files are in. You should be able to figure out whats going on, so no other comment.