I managed to do a backup of my ftp website to my Synology, with the task manager, using this script :
wget -m ftp://user:pwd#example.com/folder/* -P /volume1/homes/admin/
Unfortunately, when a file is deleted on the ftp server, it will not be deleted on the Synology. Is it possible to do it with a wget option ?
Thank you in advance !
Related
I am trying to download all files starting with traceroute from https://data-store.ripe.net/datasets/atlas-daily-dumps/ via wget.
I am running the following command:
wget -A traceroute* -m -np https://data-store.ripe.net/datasets/atlas-daily-dumps/ --no-check-certificate
It creates the directories, checks index.html's and then within 5 minutes it stops, without downloading any traceroute files.
When I try another type of file via
wget -A connection* -m -np https://data-store.ripe.net/datasets/atlas-daily-dumps/ --no-check-certificate
it donwloads the connection files no problem. What can be the issue?
You probably have a local file that matches the glob traceroute*; you need to put single quotes around it so the shell won't match anything:
wget -A 'traceroute*' -m -np https://data-store.ripe.net/datasets/atlas-daily-dumps/ --no-check-certificate
specifying traceroute*.bz2 seems to have fixed the problem
Scenario: there are multiple folders and many files stored in storage bucket that is accessible by project team members. Instead of downloading individual files one at a time (which is very slow and time consuming), is there a way to download entire folders? Or at least multiple files at once? Is this possible without having to use one of the command consoles? Some of the team members are not tech savvy and need to access these files as simple as possible. Thank you for any help!
I would suggest downloading the files with gsutil. However if you have a large number of files to transfer you might want to use the gsutil -m option, to perform a parallel (multi-threaded/multi-processing) copy:
gsutil -m cp -R gs://your-bucket .
The time reduction for downloading the files can be quite significant. See this Cloud Storage documentation for complete information on the GCS cp command.
If you want to copy into a particular directory, note that the directory must exist first, as gsutils won't create it automatically. (e.g: mkdir my-bucket-local-copy && gsutil -m cp -r gs://your-bucket my-bucket-local-copy)
I recommend they use gsutil. GCS's API deals with only one object at a time. However, its command-line utility, gsutil, is more than happy to download a bunch of objects in parallel, though. Downloading an entire GCS "folder" with gsutil is pretty simple:
$> gsutil cp -r gs://my-bucket/remoteDirectory localDirectory
To download files to local machine need to:
install gsutil to local machine
run Google Cloud SDK Shell
run the command like this (example, for Windows-platform):
gsutil -m cp -r gs://source_folder_path "%userprofile%/Downloads"
gsutil rsync -d -r gs://bucketName .
works for me
I am moving a large number of files from a local drive into a bucket using "gsutil -m mv". However during the transfer process it would appear to only be running one transfer at a time. I have checked top and only see one process from python running the command. I have modified both "parallel_process_count"
"parallel_thread_count" in the boto config file and do not observe any change in the transfers behavior. Even when running gsutil with the -m option i still receive the message below:
"==> NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m -m ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous."
Has anyone else run into this issue before?
OS: Centos 6.6
gsutil version: 4.15
python version: 2.6.6
This is a bug in gsutil 4.14-4.15, where the -m flag was not getting propagated correctly for the mv command. It is fixed in gsutil 4.16.
We are moving web servers. (LAMP)
Webserver 1 has hundreds of symbolic links pointing to files in a different directory (eg. ../../files/001.png). When we move over to the new server (have downloaded the site to my computer then reuploading to Webserver 2 using SFTP client Transmit). It does not copy the symbolic links...
Is there a better way to get the symbolic links from one server to another apart from recreating them on the new server?
rsync -a from one server to another will preserve file attributes and symlinks.
rsync -av user#server:/path/to/source user#server2:/path/to/target
Something like the following? On Webserver 1:
tar czf - the_directory | (ssh Webserver2 "cd /path/to/wherever && tar xzf -")
This creates a tar of the stuff to copy to STDOUT, and pipes it to an untar on the other server over ssh. It can be faster than a recursive ssh copy too.
What's the best way of updating data files from a website that has moved on to a new domain, with changes in their folder structure.
The old URL for example is http://folder.old-domain.com while the new URL is http://new-domain.com/directory1/directory2. My data is stored locally in ~/Data_Backup/folder.old-domain.com folder.
Data was originally downloaded using:
$ wget -S -t 0 -c --mirror –w 2 –k http://folder.old-domain.com
I was thinking of using mv to rename the old folder to follow the new URL pattern, but is there a better way of doing this?
Will this work? I'm not particular with the directory structure. What's important is to update the contents of the target folder (and its sub-folders.)
$ wget -S -t 0 -c -m –w 2 –k -N -np -P ~/Data_Backup/folder.old-domain.com http://new-domain.com/directory/directory
Thanks in advance.
Got it!
I need to add the following options:
-nH --cut-dirs=2
and now it works.