I would like to use Monolog to log a single process, eg the progress of a command, and return that log when the command is finished. The process is not necessarily a console command.
Is there a handler in monolog that allows me to log to a variable - in memory ? Or, alternatively, is it easy / pretty to log to a temp file and read that temp file when done (and clear that temp file when starting?).
I created a job that could be reusable for new files. The entire activities in the job, the maps and everything else will remain the same except for the file name. I already tried it once but it seems that i need to re "load" the file and remap everything again. It's inefficient. Is there any way for me to pass different file in a job without remaping, reconfiguring and reloading anything?
You have multiple options for allowing a DataStage parallel job to use a different filename for input on each job run:
When using either Sequential File stage or File Connector stage, in stead of typing the actual filename, you can input the name of a job parameter which has been defined on the Parameters tab of the job properties dialog. For example, if you define string parameter myFile, then in the filename field of input stage you would enter #myFile# and at job run-time that would be replaced by whatever is the current value of the myFile parameter. If you run job manually from Director/Designer clients, you will have job run dialog where you can specify a value for job parameters. If you start job via dsjob command, there are options to pass in job parameters on command line. You also have option to use parameterset files that you can modify prior to job run.
Another option would be to use a file location and pattern instead of a specific file name. Both Sequential File stage and File Connector stage let you specify a pattern, for example: /data/my_input_files/*.txt
Then, each time you run job it will input any files at that location matching the above pattern, so it can process multiple files. However, to prevent re-processing files from prior job runs, you will want to clean up any files at that location after job completes. Then when you have new files to process just put them in that directory and re-run the job.
In case if all the files contains a similar data structure, you need to implement one parallel job and if you have a similar pattern of file name for all file names Such as 1234ab.xls, 1234vd.xls, 1234gd.xls, ... you could pass the file name as 1234??.xls In the sequential job file name parameter (Use this as file name in parallel job) which contains the above parallel job to be executed.
For instance, I am creating reboot scripts for about 400 servers to reboot every night. I already have the task scheduler portion done with a script.
What I need is, how to insert "shutdown /r /m \servername-001 /f" into a file called "servername-001_Reboot.bat".
and in the same script change the 001 to 002 into the next batch file and so on and so forth.
Unless someone has a more efficient way of doing an automated reboot schedule.
Have a variable $ServerNumber = 1 and inside your for-loop just keep incrementing it.
The filename for the batchfile can be made like so:
$filename = "servername-$( "{0:D3}" -f $ServerNumber )_Reboot.bat"
(Read here to learn more about formatting numbers in powershell)
Any variable including a string, let's call it $str, can be piped to any file:
$str | Out-File "path/to/file/$filename"
use the -Append flag if you want to append data to the file if it already exists (as opposed to overwriting it).
I used to use a server with LSF but now I just transitioned to one with SLURM.
What is the equivalent command of bpeek (for LSF) in SLURM?
bpeek
bpeek Displays the stdout and stderr output of an unfinished job
I couldn't find the documentation anywhere. If you have some good references for SLURM, please let me know as well. Thanks!
You might also want to have a look at the sattach command.
I just learned that in SLURM there is no need to do bpeek to check the current standard output and standard error since they are printed in running time to the files specified for the stdout and stderr.
Here's a workaround that I use. It mimics the bpeek functionality from LSF
Create a file bpeek.sh:
#!/bin/bash
# take as input an argument - slurm job id - and save it into a variable
jobid=$1
# run scontrol show job $jobid and save the output into a variable
#find the string that starts with StdOut= and save it into a variable without the StdOut= part
stdout=$(scontrol show job $jobid | grep StdOut= | sed 's/StdOut=//')
#show last 10 rows of the file if no argument 2 is given
nrows=${2:-10}
tail -f -n $nrows $stdout
Then you can use it:
sh bpeek.sh JOBID NROWS(optional)
Or add an alias to ~/.bashrc file:
alias bpeek="sh ~/bpeek.sh $1 $2"
and then use it:
bpeek JOBID NROWS(optional)
In command prompt - How do I export all the content of the screen to a text file(basically a copy command, just not by using right-clicking and the clipboard)
This command works, but only for the commands you executed, not the actual output as well
doskey /HISTORY > history.txt
If you want to append a file instead of constantly making a new one/deleting the old one's content, use double > marks. A single > mark will overwrite all the file's content.
Overwrite file
MyCommand.exe>file.txt
^This will open file.txt if it already exists and overwrite the data, or create a new file and fill it with your output
Append file from its end-point
MyCommand.exe>>file.txt
^This will append file.txt from its current end of file if it already exists, or create a new file and fill it with your output.
Update #1 (advanced):
My batch-fu has improved over time, so here's some minor updates.
If you want to differentiate between error output and normal output for a program that correctly uses Standard streams, STDOUT/STDERR, you can do this with minor changes to the syntax. I'll just use > for overwriting for these examples, but they work perfectly fine with >> for append, in regards to file-piping output re-direction.
The 1 before the >> or > is the flag for STDOUT. If you need to actually output the number one or two before the re-direction symbols, this can lead to strange, unintuitive errors if you don't know about this mechanism. That's especially relevant when outputting a single result number into a file. 2 before the re-direction symbols is for STDERR.
Now that you know that you have more than one stream available, this is a good time to show the benefits of outputting to nul. Now, outputting to nul works the same way conceptually as outputting to a file. You don't see the content in your console. Instead of it going to file or your console output, it goes into the void.
STDERR to file and suppress STDOUT
MyCommand.exe 1>nul 2>errors.txt
STDERR to file to only log errors. Will keep STDOUT in console
MyCommand.exe 2>errors.txt
STDOUT to file and suppress STDERR
MyCommand.exe 1>file.txt 2>nul
STDOUT only to file. Will keep STDERR in console
MyCommand.exe 1>file.txt
STDOUT to one file and STDERR to another file
MyCommand.exe 1>stdout.txt 2>errors.txt
The only caveat I have here is that it can create a 0-byte file for an unused stream if one of the streams never gets used. Basically, if no errors occurred, you might end up with a 0-byte errors.txt file.
Update #2
I started noticing weird behavior when writing console apps that wrote directly to STDERR, and realized that if I wanted my error output to go to the same file when using basic piping, I either had to combine streams 1 and 2 or just use STDOUT. The problem with that problem is I didn't know about the correct way to combine streams, which is this:
%command% > outputfile 2>&1
Therefore, if you want all STDOUT and STDERR piped into the same stream, make sure to use that like so:
MyCommand.exe > file.txt 2>&1
The redirector actually defaults to 1> or 1>>, even if you don't explicitly use 1 in front of it if you don't use a number in front of it, and the 2>&1 combines the streams.
Update #3 (simple)
Null for Everything
If you want to completely suppress STDOUT and STDERR you can do it this way. As a warning not all text pipes use STDOUT and STDERR but it will work for a vast majority of use cases.
STD* to null
MyCommand.exe>nul 2>&1
Copying a CMD or Powershell session's command output
If all you want is the command output from a CMD or Powershell session that you just finished up, or any other shell for that matter you can usually just select that console from that session, CTRL + A to select all content, then CTRL + C to copy the content. Then you can do whatever you like with the copied content while it's in your clipboard.
Just see this page
in cmd type:
Command | clip
Then open a *.Txt file and Paste. That's it. Done.
If you are looking for each command separately
To export all the output of the command prompt in text files. Simply follow the following syntax.
C:> [syntax] >file.txt
The above command will create result of syntax in file.txt. Where new file.txt will be created on the current folder that you are in.
For example,
C:Result> dir >file.txt
To copy the whole session, Try this:
Copy & Paste a command session as follows:
1.) At the end of your session, click the upper left corner to display the menu.
Then select.. Edit -> Select all
2.) Again, click the upper left corner to display the menu.
Then select.. Edit -> Copy
3.) Open your favorite text editor and use Ctrl+V or your normal
Paste operation to paste in the text.
If your batch file is not interactive and you don't need to see it run then this should work.
#echo off
call file.bat >textfile.txt 2>&1
Otherwise use a tee filter. There are many, some not NT compatible. SFK the Swiss Army Knife has a tee feature and is still being developed. Maybe that will work for you.
How about this:
<command> > <filename.txt> & <filename.txt>
Example:
ipconfig /all > network.txt & network.txt
This will give the results in Notepad instead of the command prompt.
From command prompt Run as Administrator. Example below is to print a list of Services running on your PC run the command below:
net start > c:\netstart.txt
You should see a copy of the text file you just exported with a listing all the PC services running at the root of your C:\ drive.
If you want to output ALL verbosity, not just stdout. But also any printf statements made by the program, any warnings, infos, etc, you have to add 2>&1 at the end of the command line.
In your case, the command will be
Program.exe > file.txt 2>&1