I have a list of url paths saved as, say, listOfFiles.txt containing the following:
https://domain.path/level1/level2/name-of-file-01/index.html
https://domain.path/level1/level2/name-of-file-02/index.html
https://domain.path/level1/level2/name-of-file-03/index.html
...
where the name-of-file-xx has no pattern. For example,
https://domain.path/level1/level2/cR2xcet/index.html
https://domain.path/level1/level2/fse4scx/index.html
...
Question: How do you download the index.html here and saving each of the with names using their name-of-file-xx names using wget?
EDIT: What other options/arguments do we need to add in the following code to solve this problem?
wget -np -l1 -i listOfFiles.txt
Related
I am looking to download all quality_variant_[accession_name].txt files from the Salk Arabidopsis 1001 Genomes site using wget in Bash shell.
Main page with list of accessions: http://signal.salk.edu/atg1001/download.php
Each accession links to a page (e.g., http://signal.salk.edu/atg1001/data/Salk/accession.php?id=Aa_0 where Aa_0 is the accession ID) containing three more links: unsequenced_[accession], quality_variant_[accession], and quality_variant_filtered_[accession]
I am only interested in the quality_variant_[accession] link (not quality_variant_filtered_[accession] link), which takes you to to a .txt file with sequence data (e.g., http://signal.salk.edu/atg1001/data/Salk/quality_variant_Aa_0.txt)
Running the command below, the files of interest are eventually outputted (but not downloaded because of the --spider argument), demonstrating that wget can move through the page's hyperlinks to the files I want.
wget --spider --recursive "http://signal.salk.edu/atg1001/download.php
I have not let the command run long enough to determine whether the files of interest are downloaded, but the command below does begin to download the site recursively.
# Arguments in brackets do not impact the performance of the command
wget -r [-e robots=off] [-m] [-np] [-nd] "http://signal.salk.edu/atg1001/download.php"
However, whenever I try to apply filters to pull out the .txt files of interest, whether with --accept-regex, --accept, or many other variants, I cannot get past the initial .php file.
# This and variants thereof do not work
wget -r -A "quality_variant_*.txt" "http://signal.salk.edu/atg1001/download.php"
# Returns:
# Saving to: ‘signal.salk.edu/atg1001/download.php.tmp’
# Removing signal.salk.edu/atg1001/download.php.tmp since it should be rejected.
I could make a list of the accession names and loop through those names modifying the URL in the wget command, but I was hoping for a dynamic one-liner that could extract all files of interest even if accession IDs are added over time.
Thank you!
Note: the data files of interest are contained in the directory http://signal.salk.edu/atg1001/data/Salk/, which is also home to a .php or static HTML page that is displayed when that URL is visited. This URL cannot be used in the wget command because, although the data files of interest are contained here server side, the HTML page contains no reference to these files but rather links to a different set of .txt files that I don't want.
I am trying to download bulk images from URL'S listed in text file.
The command I am using is
wget -i linksfile.txt
The url structure of images in linksfile.txt is like below
www.domainname.com/197507/1-foto-000.jpg?20180711125016
www.domainname.com/197507/2-foto-000.jpg?20180711125030
www.domainname.com/197507/3-foto-000.jpg?20180711125044
www.domainname.com/197507/4-foto-000.jpg?20180711125059
Download images are being saved with filenames as
1-foto-000.jpg?20180711125016
2-foto-000.jpg?20180711125030
3-foto-000.jpg?20180711125044
4-foto-000.jpg?20180711125059
How can I omit all the text after .jpg ? I want file names to be saved as
1-foto-000.jpg
2-foto-000.jpg
3-foto-000.jpg
4-foto-000.jpg
and If possible can filenames be saved as
197507-1-foto-000.jpg
197507-2-foto-000.jpg
197507-3-foto-000.jpg
197507-4-foto-000.jpg
197507 is the folder name where images are hosted on server
I read tutorials on file name changing, Most of them are focused on downloading single file and using wget -o to change file name,, Is there any way we implement in above scenario ?
Maybe --content-disposition would do the trick.
I am new to wget.Let's get straight to the question. I want to download all images from a website directory. The directory contains no index file. The image name follows a pattern like ABCXXXX where XXXX= any four digit number. So how to download all images under the directory?
I've tried
wget -p http://www.example.com
but it's downloading an index.html file instead of multiple images.
Using wget:
wget -r -A "*.jpg" http://example.com/images/
Using cURL:
curl "http://example.com/images/ABC[0000-9999].jpg" -o "ABC#1.jpg"
According to man curl:
You can specify multiple URLs or parts of URLs by writing part sets
within braces as in:
http://site.{one,two,three}.com
or you can get sequences of alphanumeric series by using [] as in:
ftp://ftp.numericals.com/file[1-100].txt
ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
ftp://ftp.letters.com/file[a-z].txt
And explanation for #1:
-o, --output <file>
Write output to instead of stdout. If you are using {} or [] to
fetch multiple documents, you can use '#' followed by a number in the
specifier. That variable will be replaced with the current
string for the URL being fetched. Like in:
curl http://{one,two}.site.com -o "file_#1.txt"
or use several variables like:
curl http://{site,host}.host[1-5].com -o "#1_#2"
You may use this option as many times as the number of URLs you have.
See also the --create-dirs option to create the local directories
dynamically. Specifying the output as '-' (a single dash) will force
the output to be done to stdout.
In order to download a page I use the following command:
wget -kmnH -p http://www.example.com/my/own/folder/page.htm
I get the following directory structure:
/js_folder
/css_folder
/my
/own
/folder
page.htm
What I need is the following directory structure:
/js_folder
/css_folder
/page.htm
I tried using --cut-dirs=3 but it puts all the js and css in the same folder as page.htm.
Is there a way to bypass this problem calling wget only once?
Other solutions are welcome as well!
I'm trying to retrieve working webpages with wget and this goes well for most sites with the following command:
wget -p -k http://www.example.com
In these cases I will end up with index.html and the needed CSS/JS etc.
HOWEVER, in certain situations the url will have a query string and in those cases I get an index.html with the query string appended.
Example
www.onlinetechvision.com/?p=566
Combined with the above wget command will result in:
index.html?page=566
I have tried using the --restrict-file-names=windows option, but that only gets me to
index.html#page=566
Can anyone explain why this is needed and how I can end up with a regular index.html file?
UPDATE: I'm sort of on the fence on taking a different approach. I found out I can take the first filename that wget saves by parsing the output. So the name that appears after Saving to: is the one I need.
However, this is wrapped by this strange character â - rather than just removing that hardcoded - where does this come from?
If you try with parameter "--adjust-extension"
wget -p -k --adjust-extension www.onlinetechvision.com/?p=566
you come closer. In www.onlinetechvision.com folder there will be file with corrected extension: index.html#p=566.html or index.html?p=566.html on *NiX systems. It is simple now to change that file to index.html even with script.
If you are on Microsoft OS make sure you have latter version of wget - it is also available here: https://eternallybored.org/misc/wget/
To answer your question about why this is needed, remember that the web server is likely to return different results based on the parameters in the query string. If a query for index.html?page=52 returns different results from index.html?page=53, you probably wouldn't want both pages to be saved in the same file.
Each HTTP request that uses a different set of query parameters is quite literally a request for a distinct resource. wget can't predict which of these changes is and isn't going to be significant, so it's doing the conservative thing and preserving the query parameter URLs in the filename of the local document.
My solution is to do recursive crawling outside wget:
get directory structure with wget (no file)
loop to get main entry file (index.html) from each dir
This works well with wordpress sites. Could miss some pages tho.
#!/bin/bash
#
# get directory structure
#
wget --spider -r --no-parent http://<site>/
#
# loop through each dir
#
find . -mindepth 1 -maxdepth 10 -type d | cut -c 3- > ./dir_list.txt
while read line;do
wget --wait=5 --tries=20 --page-requisites --html-extension --convert-links --execute=robots=off --domain=<domain> --strict-comments http://${line}/
done < ./dir_list.txt
The query string is required because of the website design what the site is doing is using the same standard index.html for all content and then using the querystring to pull in the content from another page like with script on the server side. (it may be client side if you look in the JavaScript).
Have you tried using --no-cookies it could be storing this information via cookie and pulling it when you hit the page. also this could be caused by URL rewrite logic which you will have little control over from the client side.
use -O or --output-document options. see http://www.electrictoolbox.com/wget-save-different-filename/