I have tried to find a solution to this without success. We have this app and we successfully can make HTTP posts with wget like these:
wget -qO- --keep-session-cookies --save-cookies cookies.txt https://example.com/usuarios/login/26960 | grep "name='__csrf_magic'" | sed 's/.value="(.)".*/\1/' > csrf.txt
wget -qO- --keep-session-cookies --load-cookies cookies.txt --save-cookies cookies.txt --post-data "email=<email>&code=<code>" https://example.com//usuarios/login
wget --keep-session-cookies --load-cookies cookies.txt --post-data "dados" 'https://example.com/pesquisas/insert'
wget -qO- --keep-session-cookies --save-cookies cookies.txt https://example.com/usuarios/logoff
How to convert this wget calls to Zapier?
Thanks in advance,
Related
I was using below command to download list of urls
function() { wget "$(echo "$1" | some command)"; }
cat url | xargs -I {} bash -c 'function "{}"'
How can i log failed download url?
Tried
wget -a log
But not working
I'm trying to crawl a website which needs to be logged in with wget but it stops everytime it finds a logout url (https://example.com/logout/).
I've tried excluding the directories but without success.
This is my command:
wget --content-disposition --header "Cookie: session_cookies" -k -m -r -E -p --level=inf --retry-connrefused -D site.com -X */logout/*,*/settings/* -o log.txt https://example.com/
I've tried with -R option instead of -X but that didn't work.
Can be solved by the keyword "--reject-regex", like this: "--reject-regex logout", see:wget-devTips
I wanted to write a little script using wget to get some files. I was using the direct URL and everything seemed to work nice. After the script finished its work I checked the directories for the files and some were missing. I took a closer look at the terminal and I found that wget somehow managed to change the source URL. It either just typed some random characters or it changed 4 characters from the original URL to wget.
Here's a screen so you know what I mean:
Is this a known problem?
I tried wget with several added options none of them worked for me:
wget
wget -c
wget -r -np -nc --restrict-file-names=nocontrol
If you need to have a closer look at the whole source code, here it is:
mkdir ETH
cd ETH
mkdir Analysis\ I Lineare\ Algebra Diskrete\ Mathematik Einfuehrung\ in\ die\ Programmierung
cd Analysis\ I
mkdir Serien Loesungen Skript
cd ..
cd Lineare\ Algebra
mkdir Serien Loesungen Skript
cd ..
cd Diskrete\ Mathematik
mkdir Serien Loesungen
cd ..
cd Einfuehrung\ in\ die\ Programmierung
mkdir Serien Loesungen Probeklausuren Vorlesungsfolien Uebungsfolien
cd ..
cd Analysis\ I
cd Serien
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie01-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie02-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie03-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie04-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie05-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie06-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie07-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie08-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie09-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie10-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie11-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie12-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie13-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie14-Ana1-D-INFK.pdf
cd ..
cd Loesungen
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung01-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung02-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung03-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung04-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung05-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung06-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung07-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung08-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung09-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung10-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung11-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung12-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung13-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung14-Ana1-D-INFK.pdf
cd ..
cd Skript
wget -c http://www.math.ethz.ch/~struwe/Skripten/InfAnalysis-bbm-8-11-2010.pdf
cd ../..
cd Lineare\ Algebra
cd Serien
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s01.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s02.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s02.zip
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s03.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s04.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s05.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s05.zip
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s06.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s07.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s08.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s09.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s10.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s11.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s12.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s13.pdf
cd ..
cd Loesungen
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l01.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l02.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l03.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l04.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l05.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l06.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l07.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l08.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l09.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l10.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l11.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l12.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l01.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l02.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l03.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l04.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l05.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l06.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l07.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l08.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l09.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l10.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l11.zip
cd ..
cd Skript
wget -c http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/LAS07.pdf
cd ../..
cd Diskrete\ Mathematik
cd Serien
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u01.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u02.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u03.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u04.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u05.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u06.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u07.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u08.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u09.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u10.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u11.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u12.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u13.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u14.pdf
cd ..
cd Loesungen
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l01.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l02.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l03.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l04.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l05.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l06.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l07.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l08.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l09.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l10.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l11.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l12.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l13.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l14.pdf
cd ../..
cd Einfuehrung\ in\ die\ Programmierung
cd Serien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/01/assignment_1.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/02/assignment_2.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/03/assignment_3.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/04/assignment_4.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/05/assignment_5.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/06/assignment_6.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/07/assignment_7.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/08/assignment_8.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/09/assignment_9.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/10/assignment_10.pdf
cd ..
cd Loesungen
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/02/assignment_2_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/03/assignment_3_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/04/assignment_4_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/05/assignment_5_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/06/assignment_6_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/07/assignment_7_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/08/assignment_8_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/09/assignment_9_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/10/assignment_10_solution.pdf
cd ..
cd Probeklausuren
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/1/exam.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/1/solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/2/exam.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/2/solution.pdf
cd ..
cd Vorlesungsfolien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/01_overview.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/02_objects.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/03_conventions.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/04_interface.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/05_logic.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/06_creation.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/07_references_assignments.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/08_bnf.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/08_control.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/09_abstraction.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/10_dynamic_model.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/11_inheritance.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/12_recursion.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/13_data_structures.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/15_topological_sort.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/14_multiple_inheritance.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/18_undo.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/17_agents.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/19_software_engineering.pdf
cd ..
cd Uebungsfolien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_02cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_03cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_04cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_05cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_06cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_07cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_09cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_10cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_13cm.pdf
Thank you very much.
The corruption is happening in your terminal window, long before wget sees any command.
Corruption is very common when you do large copy-and-paste operations into terminal sessions running bash. Copy and paste doesn't play well with the way bash reads from the terminal.
You should run your script file directly, using the source command (also called .), or by starting a subshell.
Right now my wget is:
wget -Ncq -e \"convert-links=off\" --load-cookies /dev/null --tries=50 --timeout=45 --no-check-certificate \"$download\" -O $prefix$title.webm &
and it runs in the background, i dont want it to be in the background. How can i fix this?
wget -Ncq -e \"convert-links=off\" --load-cookies /dev/null --tries=50 --timeout=45 --no-check-certificate \"$download\" -O $prefix$title.webm #&
& sends to background
# is a comment
I'm trying to download all of the pdfs and ppts from this website: http://mlss2011.comp.nus.edu.sg/index.php?n=Site.Slides
I do in Cygwin:
wget --no-parent -A.pdf,.pptx -r -l1 http://mlss2011.comp.nus.edu.sg/index.php?n=Site.Slides
but no files are downloaded.
What do I need to change in the above wget command for this to work?
needed to use the -e robots=off code, so this worked
wget -e robots=off -A.pdf,.pptx -r -l1 http://mlss2011.comp.nus.edu.sg/index.php?n=Site.Slides
Also in general, use the --debug flag for more help.