I keep getting wget errors such as "Bad file descriptor" when trying to write to a filesystem that can't handle ! in filenames.
Unfortunately this ends the whole job rather than moving onto the next file. How can I make wget ignore/skip any errors instead of quit completely?
(edit: seems I'm getting this error for files even without special characters but still the same issue with wget quitting half-way through a long job rather than continuing to the next file.)
Related
I have a perlscript file was running fine in crontab but suddenly it stopped running without any modification.
cd /home/user/public_html/crons && ./script.pl 2>&1 >/dev/null
The top of the script file is #!/usr/bin/perl -X
The output expect from this script is changes in database
I have another script file with the same modification and still works fine
When I run the file in the browser it works fine and execute all lines without any problem
I tried full path /usr/bin/perl but it didn't work
I tried Perl at the beginning but it didn't work
I run the command from SSH using putty but nothing happened
I checked log file /var/log/cron but no errors at all
I created temporary log file cd /home/user/public_html/crons/script.pl> /tmp/temp.log 2>&1 to see the errors but the log is empty
Here is the solution:-
I found the issue, There is was a stuck process for the same cron file , so i killed this process and its fixed
You can find your file process like this
ps aux | grep 'your cron file here'
This is a really common antipattern people seem to tend toward with cron.
Cron sends you an email with the output of your script, if it generates any output. People often redirect output to /dev/null to prevent cron from sending the email. This is bad because now the output of your script is lost entirely. Even if the script has some built-in logging, it might generate errors before it gets the log file opened and those are lost. It also might crash in a way that doesn't get written to the logging mechanism.
At a bare minimum, you should just remove 2>&1 >/dev/null to start receiving the email. (and also, test your mail setup using a temporary cron job like 1 * * * * echo "Test" )
The next better solution is to change it to >> /var/log/myscript/current.log and then also set up something to rotate the log files (like logrotate) and also make sure to create that directory with permissions that the script user is allowed to write to it. By only redirecting STDOUT of the script, any errors or warnings it writes to STDERR cause you to get an email, and if there are no errors/warnings the output goes to the log file and no email gets sent.
Neither of those changes solve the root problem though, which is that when cron runs your script it does so with a different environment than you have on the command line. What you really want is a way to run the script with a consistent environment, and log it. The "ultimate solution" is to define your task in some kind of service manager, and then use cron to occasionally start it. For instance, you could use systemd and define a service that doesn't restart, then use systemctl start my_custom.service in your cron job. Now you can test independent of cron, and your tests will have the same exact environment, and be logged by the service manager. As extra bonuses, you are protected from accidentally running your script twice at once, and you get a clean way to stop a running cron job without the danger of stale pid files.
I don't particularly advocate systemd myself, but thankfully there are lots of alternatives:
Runit : http://smarden.org/runit/runsvdir.8.html
S6 : https://skarnet.org/software/s6/
Perp : http://b0llix.net/perp/site.cgi?page=perpd.8
(but installing and configuring a service manager is a bigger task than just using systemd if your distro is based on systemd) Each of these allows you to define a service that doesn't restart. Then you use a shell command to issue a "run once" directive to the supervisor, which runs the task as a child. Now you can easily launch the jobs yourself and see all the errors in the log, and then add that command to the crontab and know that it will run identically when cron starts it.
Back to your original problem, once you get some logging you are likely to discover it is a permission problem or a upgraded module in the system perl.
I'm a Python/programming novice and this is probably an novice mistake, but I've scoured the internet for answers and have found none! I'm learning on a Rosalind module that's about opening data files. http://rosalind.info/problems/ini5/ I'm pretty sure I understand everything clearly, so I'm frustrated about my inability to do this simple task.
I'm using Python 3.6.2 and IDLE. The assignment is to simply open a .txt file and read a few lines.
I downloaded the .txt file to my working directory. Then, I opened up IDLE Shell and made sure I was in the right working directory (using ls & cd). I then opened a new IDLE .py file and wrote a script:
f = open('filename.txt', 'r')
f.readlines()[2]
I saved the script as p5.py. Then, I tried to run the script by calling F5. In the Shell, I got this message:
================ RESTART: /Users/liv/Desktop/Rosalind/p5.py =================
Is that an error? I think it's just a message from IDLE that IDLE has opened p5.py. Therein lies my problem, because now I have the wrong file open.
I started realizing that when I used the Shell and called it to print, and it came back with an empty string.
What am I doing wrong?? How do I get IDLE to open the filename.txt file? ...not the .py file.
The RESTART line means that IDLE has sent p5.py to Python to be run, which is exactly what you want and what you asked. Python should have then opened the text file, read it, retrieved the 3rd line, and stopped. Since p5.py has no output statements and does not raise any exceptions, you will not see anything. If you change the 2nd line to
print('line is ', f.readlines()[2])
then you should see something.
I am attempting to use the SVN_Load_Dirs.PL script file (https://svn.apache.org/repos/asf/subversion/trunk/contrib/client-side/svn_load_dirs/) to attempt to merge a platform drop.
However, I can't get the --glob_ignores flag to behave as I'd expect, and I know so little perl that I can't dig into the script to understand why. The format I am using is:
--glob_ignores=*.jazzignore
Where I want to ignore all .jazzignore files (although I am fine with anything with "jazzignore" in either the extension or name being ignored. I've looked for examples but can't find any actual usage of this flag anywhere. What I am looking for is a way to ignore all .jazzignore files and a few entire directories (like jazz5 for an example)
I assumed the flag would then be --glob_ignores="*.jazzignore *.jazz5" but that doesn't appear to be working.
It turns out that the script was failing silently because I was running it from a windows cmd (and therefore never getting to the flag option). Apparently there are some issues running it this way and it was designed to be run from a linux command line. After switching it works perfectly.
I am trying to download the contents of this link >
http://goldsmr2.sci.gsfc.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2Fs4pa%2FMERRA%2FMAT1NXSLV.5.2.0%2F2014%2F01%2FMERRA300.prod.assim.tavg1_2d_slv_Nx.20140103.hdf&FORMAT=TmV0Q0RGLw&BBOX=-16%2C-44%2C-11%2C-40&LABEL=MERRA300.prod.assim.tavg1_2d_slv_Nx.20140103.SUB.nc&FLAGS=&SHORTNAME=MAT1NXSLV&SERVICE=SUBSET_LATS4D&LAYERS=&VERSION=1.02&VARIABLES=u10m%2Cu50m%2Cv10m%2Cv50m
through wget but I get an internal error 500 instead. It works when I paste the url into firefox.
I have tried some options with wget and even using a proxy but nothing seems to work. Help is very much appreciated.
Thank you.
Certain characters have special meaning when used on the command line. For example, if your URL there are several & characters, which tells the shell to run the command before it, and run the next command without waiting for the first one to finish, basally terminating the URL early, and running several other commands from the rest of the URL.
You can avoid these issues by enclosing the URL in single quotes like this, which will prevent the shell from parsing special characters or variables.
wget 'YOUR_URL_HERE'
I'm using org-mode and am looking to export my agenda (example of an agenda) to a text file so that I can display it with conky. The org-mode manual features this command line example to do this:
emacs -batch -l ~/.emacs -eval '(org-batch-agenda "t")' | lpr
I've modified this like so:
emacs -batch -l ~/.emacs -eval '(org-batch-agenda "e")' \
> ~/org/aux/agenda-export.txt
I have this set as a cron job to run every 5 minutes. It works great unless I actually have emacs open. Then I noticed that the file agenda-export.txt was empty. In running this manually from the coammand line vs. through cron, I get this error (or similar depending on the files I have open):
...~/org/file.org locked by jwhendy (pid 10935): (s, q, p, ?)?
I was going to write a script to perhaps export to agenda-export-test.txt, then perhaps check for an empty file or no lines (wc -l == 0 maybe?). If true, leave the existing file alone and delete agenda-export-test.txt. If not, then move agenda-export-test.txt to agenda-export.txt.
But... when I try to run such a script, I'm met with the same emacs inquiry about whether to steal the lock, proceed, or quit. I'm fine with proceeding, as I don't think org-agenda does anything to the files and thus it wouldn't harm anything... but I don't know how to tell emacs to "force" or choose "proceed" if problems are encountered. I need something non-interactive.
So, to summarize, my thoughts were to try:
passing a --force option or similar to emacs to make it proceed at the pause
see if the exported tmp file has any lines and deal with it accordingly
tell emacs to run in "read only mode" (but I don't think it exists)
The main issue is that with cron, I'm not there to tell the process what to do, and so it just just make an empty file as the exported results. How can I handle this locked file business with a "blind" process like cron that can't respond?
I've tried asking the mailing list as well without a working outcome. [1] I wondered if someone here might have ideas.
[1] http://www.mail-archive.com/emacs-orgmode#gnu.org/msg45056.html
Have you tried copying file without using emacs?
put in your crontab:
cp ~/org/file.org /tmp/export.org && emacs -batch .... /tmp/export.org ..
A regular "cp" command should not copy emacs locks. Of course, once in a while you might get a damaged file if you save agenda just during cp command, but this should not be too bad.