I was using below command to download list of urls
function() { wget "$(echo "$1" | some command)"; }
cat url | xargs -I {} bash -c 'function "{}"'
How can i log failed download url?
Tried
wget -a log
But not working
Related
I'm trying to execute this code:
sudo su sh <(curl https://mirror.cyberpanel.net/install-test.sh || wget -O - https://mirror.cyberpanel.net/install-test.sh)
but get this output:
...
su: user sh does not exist
....
curl: (23) Failed writing body (0 != 8192)
.....
Cannot write to ‘-’ (Success).
Can anyone help resolve?
In the command su sh sh is a user name. If you want su to interpret it as a command to run add option -c:
sudo su -c "sh…"
See https://linux.die.net/man/1/su and https://www.sudo.ws/man.html
Save the file to disk first
url=https://mirror.cyberpanel.net/install-test.sh
curl -O "$url" || wget -O - "$url"
# Examine the contents of install-test.sh first, *then* run ...
sudo sh install-test.sh
"Examine" could me reading it yourself to verify that it doesn't do anything you don't want it to do, or comparing its MD5 checksum to an expected value.
I'm trying to crawl a website which needs to be logged in with wget but it stops everytime it finds a logout url (https://example.com/logout/).
I've tried excluding the directories but without success.
This is my command:
wget --content-disposition --header "Cookie: session_cookies" -k -m -r -E -p --level=inf --retry-connrefused -D site.com -X */logout/*,*/settings/* -o log.txt https://example.com/
I've tried with -R option instead of -X but that didn't work.
Can be solved by the keyword "--reject-regex", like this: "--reject-regex logout", see:wget-devTips
I use this script to install swiftlint on travis:
set -e
SWIFTLINT_PKG_PATH="/tmp/SwiftLint.pkg"
SWIFTLINT_PKG_URL="https://github.com/realm/SwiftLint/releases/download/0.25.1/SwiftLint.pkg"
wget --output-document=$SWIFTLINT_PKG_PATH $SWIFTLINT_PKG_URL
if [ -f $SWIFTLINT_PKG_PATH ]; then
echo "SwiftLint package exists! Installing it..."
sudo installer -pkg $SWIFTLINT_PKG_PATH -target /
else
echo "SwiftLint package doesn't exist. Compiling from source..." &&
git clone https://github.com/realm/SwiftLint.git /tmp/SwiftLint &&
cd /tmp/SwiftLint &&
git submodule update --init --recursive &&
sudo make install
fi
It's not the best way to always update it if needed in this script - is it possible to point to always the latest released pkg?
You can get info about last release from API: https://api.github.com/repos/realm/SwiftLint/releases/latest
To get tag name:
curl --silent "https://api.github.com/repos/realm/SwiftLint/releases/latest" |
grep -Po '"tag_name": "\K.*?(?=")'
To get download URL for pkg file:
curl --silent "https://api.github.com/repos/realm/SwiftLint/releases/latest" |
grep -Po '"browser_download_url": "\K.*?.pkg(?=")'
Source of one-liners.
I wanted to write a little script using wget to get some files. I was using the direct URL and everything seemed to work nice. After the script finished its work I checked the directories for the files and some were missing. I took a closer look at the terminal and I found that wget somehow managed to change the source URL. It either just typed some random characters or it changed 4 characters from the original URL to wget.
Here's a screen so you know what I mean:
Is this a known problem?
I tried wget with several added options none of them worked for me:
wget
wget -c
wget -r -np -nc --restrict-file-names=nocontrol
If you need to have a closer look at the whole source code, here it is:
mkdir ETH
cd ETH
mkdir Analysis\ I Lineare\ Algebra Diskrete\ Mathematik Einfuehrung\ in\ die\ Programmierung
cd Analysis\ I
mkdir Serien Loesungen Skript
cd ..
cd Lineare\ Algebra
mkdir Serien Loesungen Skript
cd ..
cd Diskrete\ Mathematik
mkdir Serien Loesungen
cd ..
cd Einfuehrung\ in\ die\ Programmierung
mkdir Serien Loesungen Probeklausuren Vorlesungsfolien Uebungsfolien
cd ..
cd Analysis\ I
cd Serien
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie01-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie02-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie03-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie04-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie05-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie06-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie07-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie08-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie09-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie10-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie11-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie12-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie13-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Serie14-Ana1-D-INFK.pdf
cd ..
cd Loesungen
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung01-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung02-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung03-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung04-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung05-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung06-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung07-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung08-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung09-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung10-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung11-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung12-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung13-Ana1-D-INFK.pdf
wget -c https://www2.math.ethz.ch/education/bachelor/lectures/hs2015/other/analysis1_infk/Loesung14-Ana1-D-INFK.pdf
cd ..
cd Skript
wget -c http://www.math.ethz.ch/~struwe/Skripten/InfAnalysis-bbm-8-11-2010.pdf
cd ../..
cd Lineare\ Algebra
cd Serien
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s01.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s02.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s02.zip
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s03.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s04.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s05.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s05.zip
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s06.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s07.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s08.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s09.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s10.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s11.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s12.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/exercisesheets/s13.pdf
cd ..
cd Loesungen
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l01.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l02.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l03.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l04.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l05.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l06.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l07.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l08.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l09.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l10.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l11.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l12.pdf
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l01.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l02.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l03.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l04.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l05.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l06.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l07.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l08.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l09.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l10.zip
wget -c http://cvg.ethz.ch/teaching/linalg/mastersolutions/l11.zip
cd ..
cd Skript
wget -c http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/LAS07.pdf
cd ../..
cd Diskrete\ Mathematik
cd Serien
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u01.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u02.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u03.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u04.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u05.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u06.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u07.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u08.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u09.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u10.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u11.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u12.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u13.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/u14.pdf
cd ..
cd Loesungen
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l01.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l02.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l03.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l04.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l05.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l06.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l07.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l08.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l09.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l10.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l11.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l12.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l13.pdf
wget -c http://www.crypto.ethz.ch/teaching/lectures/DM15/l14.pdf
cd ../..
cd Einfuehrung\ in\ die\ Programmierung
cd Serien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/01/assignment_1.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/02/assignment_2.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/03/assignment_3.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/04/assignment_4.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/05/assignment_5.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/06/assignment_6.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/07/assignment_7.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/08/assignment_8.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/09/assignment_9.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/10/assignment_10.pdf
cd ..
cd Loesungen
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/02/assignment_2_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/03/assignment_3_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/04/assignment_4_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/05/assignment_5_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/06/assignment_6_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/07/assignment_7_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/08/assignment_8_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/09/assignment_9_solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/assignments/10/assignment_10_solution.pdf
cd ..
cd Probeklausuren
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/1/exam.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/1/solution.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/2/exam.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/mock_exams/2/solution.pdf
cd ..
cd Vorlesungsfolien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/01_overview.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/02_objects.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/03_conventions.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/04_interface.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/05_logic.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/06_creation.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/07_references_assignments.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/08_bnf.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/08_control.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/09_abstraction.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/10_dynamic_model.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/11_inheritance.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/12_recursion.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/13_data_structures.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/15_topological_sort.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/14_multiple_inheritance.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/18_undo.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/17_agents.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/lectures/19_software_engineering.pdf
cd ..
cd Uebungsfolien
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_02cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_03cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_04cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_05cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_06cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_07cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_09cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_10cm.pdf
wget -c http://se.inf.ethz.ch/courses/2015b_fall/eprog/exercise_sessions/exercise_session_13cm.pdf
Thank you very much.
The corruption is happening in your terminal window, long before wget sees any command.
Corruption is very common when you do large copy-and-paste operations into terminal sessions running bash. Copy and paste doesn't play well with the way bash reads from the terminal.
You should run your script file directly, using the source command (also called .), or by starting a subshell.
Here is my wget command where i am trying to rename the file which i am downloading but it is not working. I am using -O option here but somehow it is not working.
access="http://mvn:8081/nexus/content/com/mvn/"
wget -r -np -nd -l1 -O "access.war" "$access" -A "com.infa.products.ldm.ingestion.access.web-"$n"-.-1-ldm-access-web.war"
Here i am renaming it to access.war. I can only use wget to do this job due to some restrictions.
Thanks for the help.
The option -A is "comma separated", but you are using dots to separate the extensions!
Instead of
-A "com.infa.products.ldm.ingestion.access.web-"$n"-.-1-ldm-access-web.war"
Try
-A "com,infa,products,ldm,ingestion,access,web-"$n"-,-1-ldm-access-web,war"
If this is not the solution to your problem, I suggest you simplify your wget-call down to something like this
wget -r -np -nd -l1 -O "access.war" "$access"
Just to verify that all else is working.
Or even better (to get fewer files)
wget -r -np -nd -l1 -O "access.war" "$access" -A "war"