I am trying to download bulk images from URL'S listed in text file.
The command I am using is
wget -i linksfile.txt
The url structure of images in linksfile.txt is like below
www.domainname.com/197507/1-foto-000.jpg?20180711125016
www.domainname.com/197507/2-foto-000.jpg?20180711125030
www.domainname.com/197507/3-foto-000.jpg?20180711125044
www.domainname.com/197507/4-foto-000.jpg?20180711125059
Download images are being saved with filenames as
1-foto-000.jpg?20180711125016
2-foto-000.jpg?20180711125030
3-foto-000.jpg?20180711125044
4-foto-000.jpg?20180711125059
How can I omit all the text after .jpg ? I want file names to be saved as
1-foto-000.jpg
2-foto-000.jpg
3-foto-000.jpg
4-foto-000.jpg
and If possible can filenames be saved as
197507-1-foto-000.jpg
197507-2-foto-000.jpg
197507-3-foto-000.jpg
197507-4-foto-000.jpg
197507 is the folder name where images are hosted on server
I read tutorials on file name changing, Most of them are focused on downloading single file and using wget -o to change file name,, Is there any way we implement in above scenario ?
Maybe --content-disposition would do the trick.
Related
I have a list of url paths saved as, say, listOfFiles.txt containing the following:
https://domain.path/level1/level2/name-of-file-01/index.html
https://domain.path/level1/level2/name-of-file-02/index.html
https://domain.path/level1/level2/name-of-file-03/index.html
...
where the name-of-file-xx has no pattern. For example,
https://domain.path/level1/level2/cR2xcet/index.html
https://domain.path/level1/level2/fse4scx/index.html
...
Question: How do you download the index.html here and saving each of the with names using their name-of-file-xx names using wget?
EDIT: What other options/arguments do we need to add in the following code to solve this problem?
wget -np -l1 -i listOfFiles.txt
I have a text file C:\folder\filelist.txt containing a list of numbers, for example:
345651
342679
344000
349080
I want to append the URL as shown below, download only the files that are >1000KB, and strip the parameters after "-a1" from the filename, for example:
URL
Size
Output File
https://some.thing.com/gab/abc-345651-def-a1?scl=1&fmt=jpeg
1024kb
C:\folder\abc-345651-def-a1.jpeg
https://some.thing.com/gab/abc-342679-def-a1?scl=1&fmt=jpeg
3201kb
C:\folder\abc-342679-def-a1.jpeg
https://some.thing.com/gab/abc-342679-def-a1?scl=1&fmt=jpeg
644kb
-
https://some.thing.com/gab/abc-349080-def-a1?scl=1&fmt=jpeg
2312kb
C:\folder\abc-349080-def-a1.jpeg
This is the code I currently have, which works for downloading the files and appending the .jpeg extension, given the full URL is in the text file. It does not filter out the smaller images or strip the parameters following "-a1".
cd C:\folder\
wget --adjust-extension --content-disposition -i C:\folder\filelist.txt
I'm running Windows and I'm a beginner at writing batch scripts. The most important thing 'm trying to accomplish is to avoid downloading images <1000kb: it would be acceptable if I had to manually append the URL in the text file and rename the files after the fact. Is it possible to do what I'm trying to do? I've tried modifying the script by referencing the posts below, but I can't seem to get it to work. Thanks in advance!
Wget images larger than x kb
Downloading pdf files with wget. (characters after file extension?)
Spider a Website and Return URLs Only
#change working directory
cd /c/folder/
#convert input file list to unix
dos2unix filelist.txt
for image in $(cat filelist.txt)
do
imageURL="https://some.thing.com/gab/abc-$image-def-a1?scl=1&fmt=jpeg"
size=`wget -d -qO- "$imageURL" 2>&1 | grep 'Content-Length' | awk {'print $2'}`
if [[ $size -gt 1024000 ]] ;then
imgname="/c/folder/abc-$image-def-a1.jpeg"
wget -O $imgname $imageURL
fi
done
I am looking to download all quality_variant_[accession_name].txt files from the Salk Arabidopsis 1001 Genomes site using wget in Bash shell.
Main page with list of accessions: http://signal.salk.edu/atg1001/download.php
Each accession links to a page (e.g., http://signal.salk.edu/atg1001/data/Salk/accession.php?id=Aa_0 where Aa_0 is the accession ID) containing three more links: unsequenced_[accession], quality_variant_[accession], and quality_variant_filtered_[accession]
I am only interested in the quality_variant_[accession] link (not quality_variant_filtered_[accession] link), which takes you to to a .txt file with sequence data (e.g., http://signal.salk.edu/atg1001/data/Salk/quality_variant_Aa_0.txt)
Running the command below, the files of interest are eventually outputted (but not downloaded because of the --spider argument), demonstrating that wget can move through the page's hyperlinks to the files I want.
wget --spider --recursive "http://signal.salk.edu/atg1001/download.php
I have not let the command run long enough to determine whether the files of interest are downloaded, but the command below does begin to download the site recursively.
# Arguments in brackets do not impact the performance of the command
wget -r [-e robots=off] [-m] [-np] [-nd] "http://signal.salk.edu/atg1001/download.php"
However, whenever I try to apply filters to pull out the .txt files of interest, whether with --accept-regex, --accept, or many other variants, I cannot get past the initial .php file.
# This and variants thereof do not work
wget -r -A "quality_variant_*.txt" "http://signal.salk.edu/atg1001/download.php"
# Returns:
# Saving to: ‘signal.salk.edu/atg1001/download.php.tmp’
# Removing signal.salk.edu/atg1001/download.php.tmp since it should be rejected.
I could make a list of the accession names and loop through those names modifying the URL in the wget command, but I was hoping for a dynamic one-liner that could extract all files of interest even if accession IDs are added over time.
Thank you!
Note: the data files of interest are contained in the directory http://signal.salk.edu/atg1001/data/Salk/, which is also home to a .php or static HTML page that is displayed when that URL is visited. This URL cannot be used in the wget command because, although the data files of interest are contained here server side, the HTML page contains no reference to these files but rather links to a different set of .txt files that I don't want.
I want to upload files from my system to a directory in github repo using the api .Is there any api endpoint which allows me to do that.
You should use the GitHub CRUD API which wans introduced in .May 2013
It includes:
File Create
PUT /repos/:owner/:repo/contents/:path
File Update
PUT /repos/:owner/:repo/contents/:path
File Delete
DELETE /repos/:owner/:repo/contents/:path
There's deffinitly a way to do it.
This is how I did it;
curl -i -X PUT -H ‘Authorization: token 9xxxxxxxxxxxxxxxxxxxxxxxe2’ -d
‘{“message”: “uploading a sample pdf”,
“content”:”bXkgbm……………………………..”
}’ https://api.github.com/repos/batman/toys/contents/sample.pdf
Where the content property is a base64 encoded string of characters. I used this tool to encode my pdf file. https://www.freeformatter.com/base64-encoder.html
Notice, "batman" is the owner, "toys" is my repo, "contents" has to be there by default, and sample.pdf would the name of the file you want to upload your file as.
In short, stick to this format: /repos/:owner/:repo/contents/:path
And you can run the identical step for any of these files:
PNG (.png)
GIF (.gif)
JPEG (.jpg)
Log files (.log)
Microsoft Word (.docx), Powerpoint (.pptx), and Excel (.xlsx) documents
Text files (.txt)
PDFs (.pdf)
ZIP (.zip, .gz)
Good luck.
Btw, I have these same details added on here: http://www.simplexanswer.com/2019/05/github-api-how-to-upload-a-file/
I'm currently stuck with this problem where my .gz file is "some_name.txt.gz" (the .gz is not visible, but can be recognized with File::Type functions),
and inside the .gz file, there is a FOLDER with the name "some_name.txt", which contains other files and folders.
However, I am not able to extract the archive as you would manually (the folder with the name "some_name.txt" is extracted along with its contents) when calling the extract function from the Archive::Extract because it will just extract the "some_name.txt" folder as a .txt file.
I've been searching the web for answers, but none are correct solutions. Is there a way around this?
From Archive::Extract official doc
"Since .gz files never hold a directory, but only a single file;"
I would recommend using tar on the folder and then gz it.
That way you can use Archive::Tar to easily extract specific file:
Example from official docs:
$tar->extract_file( $file, [$extract_path] )
Write an entry, whose name is equivalent to the file name provided to disk. Optionally takes a second parameter, which is the full native path (including filename) the entry will be written to.
For example:
$tar->extract_file( 'name/in/archive', 'name/i/want/to/give/it' );
$tar->extract_file( $at_file_object, 'name/i/want/to/give/it' );
Returns true on success, false on failure.
Hope this helps.
Maybe you can identify these files with File::Type, rename them with .gz extension instead of .txt, then try Archive::Extract on it?
A gzip file can only contain a single file. If you have an archive file that contains a folder plus multiple other files and folders, then you may have a gzip file that contains a tar file. Alternatively you may have a zip file.
Can you give more details on how the archive file was created and a listing of it contents?