I'm trying to crawl a website which needs to be logged in with wget but it stops everytime it finds a logout url (https://example.com/logout/).
I've tried excluding the directories but without success.
This is my command:
wget --content-disposition --header "Cookie: session_cookies" -k -m -r -E -p --level=inf --retry-connrefused -D site.com -X */logout/*,*/settings/* -o log.txt https://example.com/
I've tried with -R option instead of -X but that didn't work.
Can be solved by the keyword "--reject-regex", like this: "--reject-regex logout", see:wget-devTips
I wanted to write a little script using wget to get some files. I was using the direct URL and everything seemed to work nice. After the script finished its work I checked the directories for the files and some were missing. I took a closer look at the terminal and I found that wget somehow managed to change the source URL. It either just typed some random characters or it changed 4 characters from the original URL to wget.
Here's a screen so you know what I mean:
Is this a known problem?
I tried wget with several added options none of them worked for me:
wget
wget -c
wget -r -np -nc --restrict-file-names=nocontrol
If you need to have a closer look at the whole source code, here it is:
mkdir ETH
cd ETH
mkdir Analysis\ I Lineare\ Algebra Diskrete\ Mathematik Einfuehrung\ in\ die\ Programmierung
cd Analysis\ I
mkdir Serien Loesungen Skript
cd ..
cd Lineare\ Algebra
mkdir Serien Loesungen Skript
cd ..
cd Diskrete\ Mathematik
mkdir Serien Loesungen
cd ..
cd Einfuehrung\ in\ die\ Programmierung
mkdir Serien Loesungen Probeklausuren Vorlesungsfolien Uebungsfolien
cd ..
cd Analysis\ I
cd Serien
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie01-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie02-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie03-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie04-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie05-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie06-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie07-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie08-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie09-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie10-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie11-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie12-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie13-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie14-Ana1-D-INFK.pdf
cd ..
cd Loesungen
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung01-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung02-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung03-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung04-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung05-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung06-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung07-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung08-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung09-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung10-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung11-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung12-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung13-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung14-Ana1-D-INFK.pdf
cd ..
cd Skript
wget -c http://www.math.ethz.ch/~struwe/Skripten/InfAnalysis-bbm-8-11-2010.pdf
cd ../..
cd Lineare\ Algebra
cd Serien
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s01.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s02.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s02.zip
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s03.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s04.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s05.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s05.zip
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s06.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s07.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s08.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s09.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s10.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s11.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s12.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s13.pdf
cd ..
cd Loesungen
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l01.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l02.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l03.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l04.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l05.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l06.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l07.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l08.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l09.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l10.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l11.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l12.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l01.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l02.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l03.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l04.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l05.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l06.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l07.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l08.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l09.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l10.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l11.zip
cd ..
cd Skript
wget -c http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/LAS07.pdf
cd ../..
cd Diskrete\ Mathematik
cd Serien
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u01.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u02.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u03.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u04.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u05.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u06.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u07.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u08.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u09.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u10.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u11.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u12.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u13.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u14.pdf
cd ..
cd Loesungen
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l01.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l02.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l03.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l04.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l05.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l06.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l07.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l08.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l09.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l10.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l11.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l12.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l13.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l14.pdf
cd ../..
cd Einfuehrung\ in\ die\ Programmierung
cd Serien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/01/assignment_1.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/02/assignment_2.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/03/assignment_3.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/04/assignment_4.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/05/assignment_5.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/06/assignment_6.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/07/assignment_7.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/08/assignment_8.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/09/assignment_9.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/10/assignment_10.pdf
cd ..
cd Loesungen
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/02/assignment_2_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/03/assignment_3_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/04/assignment_4_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/05/assignment_5_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/06/assignment_6_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/07/assignment_7_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/08/assignment_8_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/09/assignment_9_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/10/assignment_10_solution.pdf
cd ..
cd Probeklausuren
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/1/exam.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/1/solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/2/exam.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/2/solution.pdf
cd ..
cd Vorlesungsfolien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/01_overview.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/02_objects.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/03_conventions.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/04_interface.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/05_logic.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/06_creation.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/07_references_assignments.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/08_bnf.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/08_control.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/09_abstraction.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/10_dynamic_model.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/11_inheritance.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/12_recursion.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/13_data_structures.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/15_topological_sort.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/14_multiple_inheritance.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/18_undo.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/17_agents.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/19_software_engineering.pdf
cd ..
cd Uebungsfolien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_02cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_03cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_04cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_05cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_06cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_07cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_09cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_10cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_13cm.pdf
Thank you very much.
The corruption is happening in your terminal window, long before wget sees any command.
Corruption is very common when you do large copy-and-paste operations into terminal sessions running bash. Copy and paste doesn't play well with the way bash reads from the terminal.
You should run your script file directly, using the source command (also called .), or by starting a subshell.
Here is my wget command where i am trying to rename the file which i am downloading but it is not working. I am using -O option here but somehow it is not working.
access="http://mvn:8081/nexus/content/com/mvn/"
wget -r -np -nd -l1 -O "access.war" "$access" -A "com.infa.products.ldm.ingestion.access.web-"$n"-.-1-ldm-access-web.war"
Here i am renaming it to access.war. I can only use wget to do this job due to some restrictions.
Thanks for the help.
The option -A is "comma separated", but you are using dots to separate the extensions!
Instead of
-A "com.infa.products.ldm.ingestion.access.web-"$n"-.-1-ldm-access-web.war"
Try
-A "com,infa,products,ldm,ingestion,access,web-"$n"-,-1-ldm-access-web,war"
If this is not the solution to your problem, I suggest you simplify your wget-call down to something like this
wget -r -np -nd -l1 -O "access.war" "$access"
Just to verify that all else is working.
Or even better (to get fewer files)
wget -r -np -nd -l1 -O "access.war" "$access" -A "war"
When using inline C with Varnish I've not been able to get /etc/varnish/default
to be happy at startup.
I've tested inline C with varnish for two things: GeoIP detection and Anti-Site-Scraping functions.
The DAEMON_OPTS always complains even though I'm following what other seem
to indicate works fine.
My problem is that this command line start up works:
varnishd -f /etc/varnish/varnish-default.conf -s file,/var/lib/varnish/varnish_storage.bin,512M -T 127.0.0.1:2000 -a 0.0.0.0:8080 -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'
But it errors out with trying to start up from default start scripts:
/etc/default/varnish has this in it:
DAEMON_OPTS="-a :8080 \
-T localhost:2000 \
-f /etc/varnish/varnish-default.conf \
-s file,/var/lib/varnish/varnish_storage.bin,512M \
-p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"
The error is:
# /etc/init.d/varnish start
Starting HTTP accelerator: varnishd failed!
storage_file: filename: /var/lib/varnish/vbox.local/varnish_storage.bin size 512 MB.
Error:
Unknown parameter "'cc_command".
If I try change the last line to:
-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"
It's error is now:
# /etc/init.d/varnish start
Starting HTTP accelerator: varnishd failed!
storage_file: filename: /var/lib/varnish/vbox.local/varnish_storage.bin size 512 MB.
Error: Unknown storage method "hared"
It's trying to interpret the '-shared' as -s hared and 'hared' is not a storage type.
For both GeoIP and the Anti-Site-Scrape I've used the exact recommended daemon options
plus have tried all sorts of variations like adding ' and '' but no joy.
Here is a link to the instruction I've followed that work fine except the DAEMON_OPTS part.
http://drcarter.info/2010/04/how-fighting-against-scraping-using-varnish-vcl-inline-c-memcached/
I'm using Debian and the exact DAEMON_OPTS as stated in the instructions.
Can anyone help with a pointer on what's going wrong here?
Even if Jacob will probably never read this, visitors from the future might appreciate what I'm going to write.
I believe I know what's wrong, and it looks like a Debian-specific problem, at least verified on Ubuntu 11.04 and Debian Squeeze.
I traced the execution from my /etc/default/varnish that contains the $DAEMON_OPTS to the init script.
In the init script /etc/init.d/varnish, the start_varnishd() function is:
start_varnishd() {
log_daemon_msg "Starting $DESC" "$NAME"
output=$(/bin/tempfile -s.varnish)
if start-stop-daemon \
--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
-P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
log_end_msg 0
else
log_end_msg 1
cat $output
exit 1
fi
rm $output
}
So I modified it to print the full start-stop-daemon command line, like:
start_varnishd() {
log_daemon_msg "Starting $DESC" "$NAME"
output=$(/bin/tempfile -s.varnish)
+ echo "start-stop-daemon --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"
if start-stop-daemon \
--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
-P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
log_end_msg 0
So I got a command line echoed on STDOUT, and copied-pasted it into my shell. And, surprise! It worked. WTF?
Repeated again to be sure. Yes, it works. Mmh. Could it be another of those bash/dash corner cases?
Let's try feeding the start-stop-daemon command line to bash, and see how it reacts:
start_varnishd() {
log_daemon_msg "Starting $DESC" "$NAME"
output=$(/bin/tempfile -s.varnish)
if bash -c "start-stop-daemon \
--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
-P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"; then
log_end_msg 0
else
log_end_msg 1
cat $output
exit 1
fi
rm $output
}
Yes, it works just fine, at least for my case.
Here's the relevant part of my /etc/default/varnish:
...
## Alternative 2, Configuration with VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request. Use a 1GB
# fixed-size cache file.
#
DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/geoip-example.vcl \
-S /etc/varnish/secret \
-s malloc,100M \
-p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o %o %s'"
...
I've seen posts where someone tried to work around this problem by moving the compile command into a separated shell script. Unfortunately that doesn't change the fact that start-stop-daemon is going to pass the $DAEMON_OPTS var through dash, and that will result in mangled options.
Would be something along the lines of:
-p 'cc_command=exec /etc/varnish/compile.sh %o %s'"
And then the compile.sh script as:
#!/bin/sh
cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o $#
but it doesn't work, so just patch your init scripts, and you're good to go!
Hope you can find this information useful.
You can try using :-
DAEMON_OPTS="-a :8080 \
-T localhost:2000 \
-f /etc/varnish/varnish-default.conf \
-s file,/var/lib/varnish/varnish_storage.bin,512M \
-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"
Obviously, your startup script interpreting the DAEMON_OPTS is not prepared for whitespace (even within single quotes). At my Fedora (15) installation, the suggested solution works fine; the arguments get interpreted correctly because the "$*" bash parameter is passed in /etc/init.d/varnish and in /etc/init.d/functions in daemon().
Did you get your startup scripts from a package or did you make custom scripts?
This isn't directly related to the question, but you may find yourself here if you are working through the Varnish Tutorial - Put Varnish on port 80.
For recent installs of Varnish on Debian systems the configuration for varnishd startup options can be found in /etc/systemd/system/multi-user.target.wants/varnish.service. The documented way of changing the port via /etc/default/varnish still exists, but is no longer functional unless you change your system to use init scripts rather than systemd.
After you've changed your options in /etc/systemd/system/multi-user.target.wants/varnish.service, don't forget to run systemctl daemon-reload, which will catalog the changes for executing the program.
I'm trying to download all of the pdfs and ppts from this website: http://mlss2011.comp.nus.edu.sg/index.php?n=Site.Slides
I do in Cygwin:
wget --no-parent -A.pdf,.pptx -r -l1 http://mlss2011.comp.nus.edu.sg/index.php?n=Site.Slides
but no files are downloaded.
What do I need to change in the above wget command for this to work?
needed to use the -e robots=off code, so this worked
wget -e robots=off -A.pdf,.pptx -r -l1 http://mlss2011.comp.nus.edu.sg/index.php?n=Site.Slides
Also in general, use the --debug flag for more help.