I have a question illustrated by the following command line interaction:
$ wget www.google.com -nv >> out.log
2014-10-28 21:41:43 URL:http://www.google.com/ [17700] -> "index.html.1" [1]
So wget www.google.com, and using -nv (nonverbose, but still printing error information), and i redirected all the output to out.log, so nothing should print on stdout, but information still gets printed to the terminal, which i can only assume is coming from stderr. Does anyone know why wget does that? How would i go about turning it off and still preserve error logging when there are actual errors?
Thanks a lot!
Jason
Like the manual says, the option you are looking for is -q. "Non-verbose" merely turns off verbose status reporting.
The somewhat weird design decisions in wget are one reason to prefer curl.
Use cURL instead:
$ curl -Ss http://www.stackoverflow.com -o /dev/null
(no output)
$ curl -Ss http://www.stackoverflow.invalid -o /dev/null
curl: (6) Couldn't resolve host 'www.stackoverflow.invalid'
If you for whichever reason really need to use wget, you can capture output and only show it on failure:
errors=$(2>&1 wget -nv http://www.stackoverflow.com) || echo "$errors" >&2
Related
I have a download link to a large file.
You need to be logged in to the site, so a cookie is used.
The download link redirects to another URL.
I'm able to download the file with wget but I only want the output of the "real" direct download link.
wget does exactly this before starting the download
Location: https://foo.com/bar.zip [following]
Is there a way to make wget stop and not actually downloading the file?
The solutions I found recommend redirecting to dev/null but this would still download the file. What I want is wget following the redirects but not actually starting the download.
I couldn't find a way to do it with wget, but I found a way to do it with curl:
curl https://openlibrary.org/data/ol_dump_latest.txt.gz -s -L -I -o /dev/null -w '%{url_effective}'
This only downloads the HEAD of the page (and sends it to /dev/null), so the file itself is never downloaded.
(src: https://stackoverflow.com/a/5300429/2317712 )
Going off of #qqilihq's comment to the curl answer, this will first strip out the line starting with "Location:" then remove the "Location: " from the beginning and the " [following]" from the end using awk. Not sure if I would use this as it looks like a small change in the wget output could make it blow up. I would use the curl answer myself.
wget --max-redirect=0 http://example.com/link-to-get-redirec-url-from 2>&1 | awk '/Location: /,// { print }' | awk '{print $2}'
I'm trying to print http headers to a text file. I tried :
wget -S --spider -O SESSIONS.txt 'mysite.com'
wget -S --spider 'mysite.com' > SESSIONS.txt
In both cases SESSIONS.txt remains empty. why?
"--spider" option does not download anything.
You can try this -
wget -S --spider -q mysite.com 2>Sessions.txt
This will save only the headers to "Sessions.txt"
However, you will have to use echo and other commands to figure out which request generated which headers.
Or, you can remove the -q option and then parse the file to remove unnecessary lines.
Another way is to use "curl -I". However, this sends a HEAD request instead of a GET request. So, it will only work if the server supports and responds to HEAD requests.
I would like to be able to
Display STDERR on the screen
Copy STDOUT and STDERR in files (and if possible in the same file)
For information, I am using Msys to do that.
.
After doing some research on the SO, I have tried to use something like
<my command> > >(tee stdout.log) 2> >(tee stderr.log)
Bu I got the following error:
sh: syntax error near unexpected token `>'
.
Any idea on how to do that?
There might be no direct solution in Msys but the >(tee ... ) solution works fine in *Nix, OSX, and probably Cygwin.
The workaround is to grep all the errors and warnings we want to keep them on the screen.
I have successfully used the following command for a makefile to compile C code:
make 2>&1 | tee make.log | grep -E "(([Ee]rror|warning|make):|In function|undefined)"
I have a simple script (test.sh) that generates STDOUT and STDERR:
#!/bin/bash
echo hello
rm something
exit
Then, to do what you want execute with the following:
./test.sh > stdout.log 2> >(tee stderr.log >&2)
You'll get the STDERR on the screen, and two separated log files with STDERR and STDOUT. I used part of the answer given here
Note that I am assuming you don't have a file called something on the current directory :)
If you want both STDOUT and STDERR to go to the same file, use the -a option on tee:
./test.sh > std.log 2> >(tee -a std.log >&2)
I'm looking to use wget to retrieve a perl file and execute it in one line. Does anyone know if this is possible/how I would go about doing this?
In order to use wget for this purpose, you would use the -O flag and give it the '-' character as an argument. From the manpage:
-O file
--output-document=file
Giving '-' as the "file" option to -O tells it to send it's output to stdout, which can then be piped into the Perl command.
You can provide the -q flag as well to turn off wget's own warning and message output:
-q
--quiet
Turn off Wget's output.
This will make things look cleaner in the shell.
So you would end up with something like:
wget -qO - http://127.0.0.1/myscript.pl | perl -
For more information on I/O redirection take a look at this:
http://www.tldp.org/LDP/abs/html/io-redirection.html
Just download and pipe to perl
curl -L http://your_location.pl | perl -
You'll sometimes see code like for install modules like cpanm.
I'm new to Wget. Following online examples, I am trying to log in to a simple page using the following command:
wget --post-data='entry=85482564&submit3=LOGIN' \ --save-cookies=my-cookies.txt --keep-session-cookies \ https://www.abczyx.com
I get the following error:
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
'submit3' is not recognized as an internal or external command, operable program or batch file.
I'm guessing that it doesn't quite recognize the &, but I am not sure how to fix it. I'm running Windows 7 cmd line. A side question, why use "\"? I see some examples with it, and some without it. I get issues with it.
After doing some reading, I found that because it is MS DOS, they do not interpret the special characters correctly. Adding quotes around it ("&") did the trick.
In Windows the escape sign is the caret, ^, not backslash, \. So in the batch file it should look like 'entry=85482564^&submit3=LOGIN'.
For me what worked was change & to %26
as in
--post-data 'login=foo%26pass=bar'
also if you are posting an email addrress be sure to change the # to %40
Other codes:
https://en.wikipedia.org/wiki/Percent-encoding
Yes, there is a mistake(I'd say a very serious mistake) in wget's manual. In the manual it says:
Log in to the server.
This can be done only once. wget --save-cookies cookies.txt
--post-data 'user=foo&password=bar'
http://example.com/auth.php
So you do something like
wget --save-cookies cookies.txt \
--post-data 'user=yourUser12%23125&password=yourPassword12%241' \
http://www.websitelink.com/
Which ovbiously doesn't work for multiple reasons. First, you have to remove the \ symbols because they get in the way, second, you have to remove line breaks themselves because when you paste it in your command line tool, it will execute them just as if you pressed enter after each of the lines, which will result in trying to execute that command as 3 separate commands:
First:
wget --save-cookies cookies.txt \
Second:
--post-data 'user=yourUser12%23125&password=yourPassword12%241' \
Third:
http://www.websitelink.com/
Ok, so you remove the slashes and then realize that you have to also remove line breaks by yourself, but it still doesn't work. At this point it's pepehands in the air. So what do you do now? Somehow you have to automagically realize that the & symbol should be also percent-encoded. So you turn
Log in to the server.
This can be done only once. wget --save-cookies cookies.txt
--post-data 'user=foo&password=bar'
http://example.com/auth.php
To this:
wget --save-cookies cookies.txt --post-data 'user=yourUser12%23125%26password=yourPassword12%241' http://www.websitelink.com/
And it starts working!