I have a log file which is written by several threads and therefore not in order.
The log contains entries like this:
[2018-07-11 15:30:39.139] [ddm.iolib.InputWorker] [debug] pop cmd from in queue...
[2018-07-11 15:30:39.140] [ddm.iolib.InputWorker] [debug] push cmd to out queue...
I would like to sort the lines by time stamp.
I found this great answers for sorting in bash: sort logfile by timestamp
This works fine, but I would like to sort this in Emacs. I tried using sort-regexp-fields, but it seems I am not clever enough to get it to work.
What I tried is:
^.*\([0-9]+:[0-9]+:[0-9]+\.[0-9]+\]\).*$
as records specifier and \1 as key. Nothing at all was sorted.
Could please someone tell me, what am I doing wrong?
I am also open for other possibilities than sort-regexp-fields.
#Toby Speight Thanks for the hint. I understand the problem and will try to avoid such questions in future. I am just debugging a lot with logging and this sorting will be very helpful for me in a lot of future cases. I am new to SO, so please accept my apologize.
#Picaud Vincent I would upvote if I could. Never used shell-command-on-region. Actually sort-lines was sufficient for this case, but your approach will be useful in the future, so your answer is accepted. Thanks.
You can apply any bash command to a selected region:
Please follow these steps:
select your region (or the whole buffer with C-x h)
type C-u M-|, Emacs will prompt you for your shell command, enter sort -k1 -r for instance
And that's it! (attention, it is M-| and not M-!)
Extract from Emacs doc (you can get it with C-h k M-|):
M-| runs the command shell-command-on-region (found in global-map),
which is an interactive compiled Lisp function in ‘simple.el’.
It is bound to M-|, .
(shell-command-on-region START END COMMAND &optional OUTPUT-BUFFER
REPLACE ERROR-BUFFER DISPLAY-ERROR-BUFFER REGION-NONCONTIGUOUS-P)
Execute string COMMAND in inferior shell with region as input.
Normally display output (if any) in temp buffer ‘Shell Command
Output’; Prefix arg means replace the region with it. Return the
exit code of COMMAND.
In other terms M-| runs the shell command and displays output in the ‘Shell Command Output’ buffer. If you want this output to replace the selected region you must prefix the M-| command, this is our C-u command (step 2).
Answering to #Toby Speight comment here is an example. I use as initial buffer these lines (your question provided link)
freeswitch.log:2011-09-08 12:21:07.282236 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda3525c0 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-08-08 13:21:07.514261 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda354460 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-06-04 16:21:08.998227 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda356300 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-09-08 12:21:10.374238 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda3581a0 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
I type: C-x h C-u M-| sort -k1 -r <RET> (where <RET> stands for the "return" keyboard key) as result my buffer contains now:
freeswitch.log:2011-09-08 12:21:10.374238 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda3581a0 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-09-08 12:21:07.282236 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda3525c0 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-08-08 13:21:07.514261 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda354460 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
freeswitch.log:2011-06-04 16:21:08.998227 [ERR] ftdm_queue.c:136 Failed to enqueue obj 0x7f2cda356300 in queue 0x7f2ce8005990, no more room! windex == rindex == 58!
Even though Picaud Vincent's answer describes an important method for emacs users to have in their toolbelt, I want to record for posterity the answer from the comments, since it is simple and solves the OP's problem exacly with a minimum of fuss:
Mark the appropriate region (perhaps with C-x h to mark the whole buffer).
M-x sort-lines
You could use sort-regexp-fields, but given that the timestamp is at the beginning of the line (and your use of ISO 8601 format for the timestamp), that means that they can be sorted quite simply with sort-lines. Assuming you want to sort the whole buffer, that's just
(sort-lines nil (point-min) (point-max))
The approach you had would match only the time part of the timestamp, so you could end up with different days intermingled.
Related
Using only the features of the POSIX shell, is there a "simple command" that does nothing and does not change the value of $? People usually describe : as a no-op command for shell, but this always sets $? to zero, so it's not what I want.
This is needed by a program that generates shell scripts. In several places it needs to emit an if-then-else block
if CONDITION
then
TRUE-COMMANDS
else
FALSE-COMMANDS
fi
but, because of further complications that I'd rather not try to explain right now, it cannot reliably tell whether TRUE-COMMANDS and FALSE-COMMANDS are empty. An empty then- or else-clause will be a shell syntax error.
: can be put at the beginning of the then-clause to take care of TRUE-COMMANDS being empty, but that won't work for FALSE-COMMANDS because it clobbers $? and the FALSE-COMMANDS might want to look at the value $? was set to by the condition. For the same reason, : can't be put after FALSE-COMMANDS—the code after the if-then-else might want to see $? from the last operation within the if-then-else.
Bonus points if you can avoid:
Forking: (exit $?) does the job, but there are so many of these conditional blocks in a generated script that it produces a measurable slowdown.
Functions: given nop () { return $? } then nop does the job, but due to more complications that I'd rather not get into, it's not practical to have nop defined sufficiently early for all of the places that would need it.
The easiest would be to make use of a simple assignment. Instead of using :, do _rc=$?.
if condition; then
[ list-true ] # optional
_rc=$?
else
[ list-false ] # optional
_rc=$?
fi
( exit $_rc ) # optional
list-post-if
Using this variable _rc, you have stored the exit status of the last executed command, whether this is condition or the last command in list-true or list-false.
The arguments in favour of this method is the low overhead of an assignment.
The argument against is the need to at least rewrite list-post-if to make use of _rc instead of $?.
If the latter is not possible, or too tedious, you might concider to add a (exit $_rc) just after the conditional statement. This, however, requires a sub-shell, but it is only one.
Don't let anyone tell you there are only functions and sub-shells.
Can you create or distribute another tiny file? If so, you can create a file with just
return $?
and then source it as an "empty" command keeping the exit status:
$ echo 'return $?' > keepstatus
$ ls foobar
ls: fooobar: No such file or directory
$ . ./keepstatus
$ echo $?
2
$ sleep 100
^C
$ . ./keepstatus
$ echo $?
130
Doesn't fork, doesn't use functions, no extra variable, keeps the status and is as POSIXly as they come.
There's even a fourth way I can see, when I sacrifice the bonus points for forking and assuming, since you are in autoconf m4 territory, that finding and using a host compiler is a breeze.
cat > keepstatus.c <<EOF
#include <stdlib.h>
int main(int argc, char **argv) {
return argv[1] ? atoi(argv[1]) : 0;
}
EOF
$CC -o keepstatus keepstatus.c
Then use /path/to/keepstatus $?.
It’s certainly not possible in a POSIX-compliant shell to write a command that does not touch $?. By reading through IEEE Std 1003.1-2017 vol. 3 chapter 2 ‘Shell Command Language’, we learn that:
§2.8.2 says that ‘[e]ach command has an exit status’ (so no command has no exit status).
§2.5.2 ‘Special Parameters’ says that $? ‘[e]xpands to the decimal exit status of the most recent pipeline (see Section 2.9.2)’.
§2.9.2 ‘Pipelines’ says that a pipeline is a sequence of one or more commands, optionally preceded (as a whole) by !, with commands joined by |. About exit status, it says
If the pipeline does not begin with the ! reserved word, the exit status shall be the exit status of the last command specified in the pipeline. Otherwise, the exit status shall be the logical NOT of the exit status of the last command.
§2.9 ‘Shell Commands’ defines a ‘command’ and says that ‘[u]nless otherwise stated, the exit status of a command shall be that of the last simple command executed by the command’. A ‘command’ may be any of the following:
A simple command (§2.9.1), which is simply an external program, a shell built-in or a variable assignment (without a command name). The former will of course return the exit status of the command executed. About the latter, the specification says:
If there is no command name, but the command contained a command substitution, the command shall complete with the exit status of the last command substitution performed. Otherwise, the command shall complete with a zero exit status.
A pipeline, described by §2.9.2 mentioned above.
A compound-list (§2.9.3), which is a sequence of one or more {sequences of one or more {sequences of one or more pipelines (see above), joined by && or ||}, each terminated by ; or & (with a final ; optional)}, joined by newline characters. A compound-list returns the exit status of the last pipeline executed synchronously; asynchronously-executed pipelines (those terminated by &) set exit status to zero. None of the sequences may be empty, which guarantees at least one pipeline will be executed.
A compound command (§2.9.4), which is either:
A subshell or a braced compound-list (§2.9.4.1), which returns the exit status of the underlying compound-list (see above).
A conditional construct (case, §2.9.4.3 or if, §2.9.4.4) or a loop (for, §2.9.4.2; while, §2.9.4.5; until, §2.9.4.6). If the body of this construct is not executed, it returns an exit status of zero.
A function definition (§2.9.5), which returns an exit status of zero if the definition is accepted, and a non-zero status otherwise.
And finally, §2.9.4.4 ‘The if Conditional Construct’ defines the condition and the body of the if construct to be a compound-list, described by §2.9.3 mentioned above. This means that the body of an if construct will always contain at least one command that will clobber $?. The specification does not even leave any wiggle room for ‘implementation-defined behaviour’.
All of the above means that the only way to write a no-op command that preserves $? is by reading out the value in $? and returning it back. There are three constructs in POSIX shell that are capable of this, two of which are already mentioned in the question body: an exit from a subshell or a function invocation.
The third is .: the shell sourcing statement mentioned in #Jens’ answer. By sourcing a script containing just a return "$?" command, the value of $? can be preserved. However, this requires you to arrange that a suitable script is found at some known location, which I assume to be just as inconvenient as ensuring that a no-op function has been defined early enough in the file (if not in fact more).
If you bend the strict POSIX requirement a little, even this can be overcome, though:
. /dev/stdin <<EOF
return "$?"
EOF
/dev/stdin is not a POSIX feature, but is widely available; it is explicitly listed in IEEE Std 1003.1-2017 vol. 1 §2.1.1 as an extension. The above snippet has been tested to work in bash 5.0.8, dash 0.5.11 and busybox sh 1.30.1 on Linux (with appropriately set-up /dev and so on).
I am iterating through a large test matrix in MATLAB and calling second-party proprietary software (running in MATLAB) each time. I cannot edit the software source code. Sometimes, the software hangs, so I want to exit it after a certain amount of time and move on to the next iteration.
In pseudocode, I'm doing this:
for i = 1:n
output(i) = proprietary_software(input(i));
end
How can I skip to the next iteration (and possibly save output(i)='too_long') if the proprietary software is taking too long?
You will need to call Matlab from another instance of Matlab. The other instance of Matlab will run the command and release control to the first instance of Matlab to wait while it either saves the results or reaches a certain time. In this case, it will wait 30 seconds.
You will need 1 additional function. Make sure this function is on the Matlab path.
function proprietary_software_caller(input)
hTic=tic;
output=proprietary_software(input);
hToc=toc(hTic);
if hToc<30
save('outfile.mat','output');
end
exit;
end
You will need to modify your original script this way
[status,firstPID] = str2double(system('for /f "tokens=2 delims=," %F in (''tasklist /nh /fi "imagename eq Matlab.exe" /fo csv) do #echo %~F'')'));
for i = 1:n
inputStr=num2str(input(i));
system(['matlab.exe -nodesktop -r proprietary_software_caller\(',inputStr,'\)&']);
hTic=tic;
hToc=toc(hTic);
while hToc<30 || ~(exist('outfile.mat','file')==2)
hToc=toc(hTic);
end
if hToc>=30
output(i)= 'too_long';
[status,allPIDs]=str2double(system('for /f "tokens=2 delims=," %F in (''tasklist /nh /fi "imagename eq Matlab.exe" /fo csv) do #echo %~F'')'));
allPIDs(allPIDs==firstPID)=[];
for a=1:numel(allPIDs)
[status,cmdout]=system(['taskkill /F /pid ' sprintf('%i',allPIDs(a))]);
end
elseif exist('outfile.mat','file')==2
loadedData=load('outfile.mat');
output(i)=loadedData.output;
delete('outfile.mat');
end
end
I hope this helps.
You are essentially asking for a way to implement a timeout on MATLAB code. This can be surprisingly tricky to implement. The first thing to state is that if the MATLAB code in question cannot terminate itself, either by exiting cleanly or throwing an error, then it is not possible to terminate the code without quitting or killing the MATLAB process in question. For example, throwing an error in an externally created timer does not work; the error is caught.
The first question to ask is therefore:
Can the over-running code be made to terminate itself?
This depends on the cause to the over-run, and also your access to the source code:
If the program gets stuck in an infinite (or very long-running) loop, either in MATLAB code or a mex file for which you have source code, or which calls a user-defined callback each iteration, then you can get this code to terminate itself.
If the program gets stuck inside a MATLAB builtin, or a p-code file or mex file for which you don't have the source code, and doesn't have support for calling a callback regularly, then it won't be possible for you to get the code to terminate itself.
Let's address the first case. The easiest way to get the code to terminate itself is to get it to throw an error, which is caught by the caller, if it exceeds the timeout time. E.g. in the OP's case:
for i = 1:n
tic();
try
output(i) = proprietary_software(input(i));
catch
end
end
with the following code somewhere in the over-running loop, or called in a loop callback or mex file:
assert(toc() < 10, 'Timed out');
Now for the second case. You need to kill this MATLAB process, so it makes sense for this to be a MATLAB process you have spawned from your current MATLAB session. You can do this using a system call similar to this:
system('matlab -nodisplay -r code_to_run()')
While it is possible for a MATLAB process to quit itself in some situations which could be of use here (e.g. a timer function calling quit('force')), the most reliable way of killing a MATLAB process is to do it with a system call, using taskkill (Windows) or kill (Linux/Mac).
A framework using the approach of spawning and killing timed-out MATLAB processes might work like this:
Using system calls, launch one or more new MATLAB processes from your MATLAB session, running the code you want.
Use the file system or a memory mapped file to communicate between the MATLAB processes the function inputs, loop progress, outputs, process ids and timeout times.
Use the original MATLAB process to check the timeout times haven't been reached, or if so to terminate the process in question and instantiate a new one.
Use the original MATLAB process to collect up the function outputs (either from the filesystem or memory mapped file) and exit. Workers should terminate when there is no more work left
I provide a sketch only because a full working implementation of this approach is fairly involved, and in fact it has already been implemented and is publicly available in the batch_job toolbox. In the OP's case, using this toolbox (with a 10 second timeout) you'd call:
output = batch_job(#proprietary_software, input(:)', '-timeout', 10);
Note that for the toolbox to work, its root directory needs to be on your MATLAB path at startup.
I am trying to write a simple server program in Ocaml that communicates with other processes via a socket. I currently have a problem that the strings the server reads (with input_line ic) do not seem to compare with other strings properly. For example, I run the server program and use telnet to connect to it, if I send "end" as a line, the server program trys to match with "end", but this doesn't work as expected. The function that handles communication is service (below), which is called to handle a client as part of a forking server (something like the double fork treatment here).
let service ic oc
try while true do
let instr = input_line ic in
match instr with
| "end" -> print_endline "matching end" (* this never runs *)
| _ -> output_string oc ((String.uppercase instr) ^ "\n") ; flush oc
done
with End_of_file -> print_endline "Input stream ended."
;;
In fact, if I do print_endline (string_of_bool ("end" = instr)) I always get false (even when I send "end" via telnet). To try and get some sense of what is going I printed out the result of different comparison operations between the let-binding and the try block:
print_endline instr ;
print_endline "end" ;
print_endline (string_of_bool ("end" = instr)) ;
print_endline (string_of_bool ("end" == instr)) ;
print_endline (string_of_int (String.compare "end" instr)) ;
When I send "end" the server now prints out
end
end
false
false
-1
I'm really lost as to what could be going on - I presume it must be something about getting the instr via reading from a socket, as usually you can compare strings just fine.
I don't think I actually had a question in all that background so here are a few variants that could work:
What am I doing wrong?
Why can't I test the input in order to take different actions?
Is this a bug in Ocaml?
Do you need the complete source to figure this out?
My guess is that there are carriage returns in the strings coming in from telnet. As I recall, the old protocols tend to send CRLF at the ends of lines.
You might try printing the string out using String.escaped.
It's pretty unlikely you're seeing a bug in OCaml.
I am iterating through a large test matrix in MATLAB and calling second-party proprietary software (running in MATLAB) each time. I cannot edit the software source code. Sometimes, the software hangs, so I want to exit it after a certain amount of time and move on to the next iteration.
In pseudocode, I'm doing this:
for i = 1:n
output(i) = proprietary_software(input(i));
end
How can I skip to the next iteration (and possibly save output(i)='too_long') if the proprietary software is taking too long?
You will need to call Matlab from another instance of Matlab. The other instance of Matlab will run the command and release control to the first instance of Matlab to wait while it either saves the results or reaches a certain time. In this case, it will wait 30 seconds.
You will need 1 additional function. Make sure this function is on the Matlab path.
function proprietary_software_caller(input)
hTic=tic;
output=proprietary_software(input);
hToc=toc(hTic);
if hToc<30
save('outfile.mat','output');
end
exit;
end
You will need to modify your original script this way
[status,firstPID] = str2double(system('for /f "tokens=2 delims=," %F in (''tasklist /nh /fi "imagename eq Matlab.exe" /fo csv) do #echo %~F'')'));
for i = 1:n
inputStr=num2str(input(i));
system(['matlab.exe -nodesktop -r proprietary_software_caller\(',inputStr,'\)&']);
hTic=tic;
hToc=toc(hTic);
while hToc<30 || ~(exist('outfile.mat','file')==2)
hToc=toc(hTic);
end
if hToc>=30
output(i)= 'too_long';
[status,allPIDs]=str2double(system('for /f "tokens=2 delims=," %F in (''tasklist /nh /fi "imagename eq Matlab.exe" /fo csv) do #echo %~F'')'));
allPIDs(allPIDs==firstPID)=[];
for a=1:numel(allPIDs)
[status,cmdout]=system(['taskkill /F /pid ' sprintf('%i',allPIDs(a))]);
end
elseif exist('outfile.mat','file')==2
loadedData=load('outfile.mat');
output(i)=loadedData.output;
delete('outfile.mat');
end
end
I hope this helps.
You are essentially asking for a way to implement a timeout on MATLAB code. This can be surprisingly tricky to implement. The first thing to state is that if the MATLAB code in question cannot terminate itself, either by exiting cleanly or throwing an error, then it is not possible to terminate the code without quitting or killing the MATLAB process in question. For example, throwing an error in an externally created timer does not work; the error is caught.
The first question to ask is therefore:
Can the over-running code be made to terminate itself?
This depends on the cause to the over-run, and also your access to the source code:
If the program gets stuck in an infinite (or very long-running) loop, either in MATLAB code or a mex file for which you have source code, or which calls a user-defined callback each iteration, then you can get this code to terminate itself.
If the program gets stuck inside a MATLAB builtin, or a p-code file or mex file for which you don't have the source code, and doesn't have support for calling a callback regularly, then it won't be possible for you to get the code to terminate itself.
Let's address the first case. The easiest way to get the code to terminate itself is to get it to throw an error, which is caught by the caller, if it exceeds the timeout time. E.g. in the OP's case:
for i = 1:n
tic();
try
output(i) = proprietary_software(input(i));
catch
end
end
with the following code somewhere in the over-running loop, or called in a loop callback or mex file:
assert(toc() < 10, 'Timed out');
Now for the second case. You need to kill this MATLAB process, so it makes sense for this to be a MATLAB process you have spawned from your current MATLAB session. You can do this using a system call similar to this:
system('matlab -nodisplay -r code_to_run()')
While it is possible for a MATLAB process to quit itself in some situations which could be of use here (e.g. a timer function calling quit('force')), the most reliable way of killing a MATLAB process is to do it with a system call, using taskkill (Windows) or kill (Linux/Mac).
A framework using the approach of spawning and killing timed-out MATLAB processes might work like this:
Using system calls, launch one or more new MATLAB processes from your MATLAB session, running the code you want.
Use the file system or a memory mapped file to communicate between the MATLAB processes the function inputs, loop progress, outputs, process ids and timeout times.
Use the original MATLAB process to check the timeout times haven't been reached, or if so to terminate the process in question and instantiate a new one.
Use the original MATLAB process to collect up the function outputs (either from the filesystem or memory mapped file) and exit. Workers should terminate when there is no more work left
I provide a sketch only because a full working implementation of this approach is fairly involved, and in fact it has already been implemented and is publicly available in the batch_job toolbox. In the OP's case, using this toolbox (with a 10 second timeout) you'd call:
output = batch_job(#proprietary_software, input(:)', '-timeout', 10);
Note that for the toolbox to work, its root directory needs to be on your MATLAB path at startup.
I wanted to write my results into a file which is produced in a recursive subroutine. And I also wanted to assign the data(read) in the file to an array in my main program in fortran90.
program permutations
implicit none
call generate (position_min)
open(unit=20, file="a.dat", status="old")
do i=1,720
read(20,*)(G(i,j),j=1,6)
end do
contains
recursive subroutine generate (position)
implicit none
integer, intent (in) :: position
integer :: value
if (position > position_max) then
open(unit=20, file="a.dat", status="unknown")
write (20, *) permutation
else
call generate(position+1)
end if
end subroutine generate
end program permutations
This program gives me the following runtime error.
At line 19 of file p2.f90 (unit = 20, file = 'a.dat')
Fortran runtime error: End of file
How do I fix this?
I think the answer is primarily my comment to the question. If you look at your code (neglecting the undeclared variable issue), in particular the if-statement of the recursive subroutine, you should note that you have
if (position > position_max) then
open(unit=20, file="a.dat", status="unknown")
write (20, *) permutation
else
call generate(position+1)
end if
that is, you are only writing to file if position > position_max. Satisfying this condition writes one line to a.dat and then completes all of the previous if statements. What you probably meant to have was it writing to file each time through the recursive loop; to do that, you would want something like
open(20,file="a.dat",status="unknown")
write(20,*) permutation
close(20)
if(position > position_max) then
return
else
call generate(position+1)
endif
In running this, I found I was getting 2 extra lines (due to writing at position=position_min and at position=position_max). You probably could tweak that to get exactly 720, but I think that this part is irrelevant because you can change your read loop to the following
i=1
do
read(20,*,iostat=ierr) G(i,:)
if(ierr/=0) exit
i = i+1
enddo
A normal read returns an iostat of 0 and an end-of-file returns -1, so as long as you can read you will continue the loop and break when the EOF is found.
After fixing up the undeclared variables, adding the close(20) statement, and adjusting as I commented above, I had no problems writing and reading in the recursive subroutine.