crc32c signature computed for local file (Rgw3kA==) doesn't match cloud-supplied digest (5A+KjA==). Local file (/home/blah/pgdata.tar) will be deleted.
I did a bit of diagnosing, and I noticed that it was always on the cloud-supplied digest of "5A+KjA==" but usually at a different point in the file with different local crc32c. This is using either:
gsutil -m rsync gs://bucket/ /
or
gsutil -m cp gs://bucket/pgdata.tar /
I seem to get this error almost all the time transferring a large 415GB tar database file. It always exits in error at a different part, and it doesn't resume. Is there any workarounds for this? If it is legitimate file corruption, I would expect it to fail at the same point in the file?
File seems fine as I've loaded this onto various instances and postgresql about a week ago.
I'm not sure of the verision of gsutils, but it is the natively installed one on GCE Ubuntu 14.04 image, following the GCE provided instructions for crcmod installation on Debian/Ubuntu.
Related
I have a shell script which refreshes my emulators data to the latest data from prod.
Part of the script is removing the existing bucket and then re exporting it to avoid the Path already exists error.
I know that I can manually add version buckets like /firestore_data/v1 but that would require me to find out what the last version is from the console and then update the shell script each time I need to refresh the emulators data.
Ideally I would like to be able to run gsutil -m cp -r gs://my-app.appspot.com/firestore_data#latest
Is there any way to version storage buckets, or to leave tags that can be used when adding and copying down?
I have a zillion machines in different places (home network, cloud, ...) and I use fish on each of them. The problem is that I have to synchronize their configuration every time I change something in there.
Is there a way to load the configuration from a remote repository? (= a place where it would be stored, not necessarily git but ideally I would manage them in GitHub). In such a case I would just have a one liner everywhere.
I do not care too much about startup time, loading the config each time would be acceptable
I cannot push the configuration to the machines (via Ansible for instance) - not of them are reachable from everywhere directly - but all of them can reach Internet
There are two parts to your question. Part one is not specific to fish. For systems I use on a regular basis I use Dropbox. I put my ~/.config/fish directory in a Dropbox directory and symlink to it. For machines I use infrequently, such as VMs I use for investigating problems unique to a distro, I use rsync to copy from my main desktop machine. For example,
rsync --verbose --archive --delete -L --exclude 'fishd.*' krader#macpro:.config .
Note the exclusion of the fishd.* pattern. That's part two of your question and is unique to fish. Files in your ~/.config/fish directory named with that pattern are the universal variable storage and are currently unique for each machine. We want to change that -- see https://github.com/fish-shell/fish-shell/issues/1912. The problem is that file contains the color theme variables. So to copy your color theme requires exporting those vars on one machine:
set -U | grep fish_color_
Then doing set -U on the new machine for each line of output from the preceding command. Obviously if you have other universal variables you want synced you should just do set -U and import all of them.
Disclaimer: I wouldn't choose this solution myself. Using a cloud storage client as Kurtis Rader suggested or a periodic cron job to pull changes from a git repository (+ symlinks) seems a lot easier and fail-proof.
On those systems where you can't or don't want to sync with your cloud storage, you can download the configuration file specifically, using curl for example. Some precious I/O time can be saved by utilizing HTTP cache control mechanisms. With or without cache control, you will still need to create a connection to a remote server each time (or each X times or each Y time passed) and that wastes quite some time already.
Following is a suggestion for such a fish script, to get you started:
#!/usr/bin/fish
set -l TMP_CONFIG /tmp/shared_config.fish
curl -s -o $TMP_CONFIG -D $TMP_CONFIG.headers \
-H "If-None-Match: \"$SHARED_CONFIG_ETAG\"" \
https://raw.githubusercontent.com/woj/dotfiles/master/fish/config.fish
if test -s $TMP_CONFIG
mv $TMP_CONFIG ~/.config/fish/conf.d/shared_config.fish
set -U SHARED_CONFIG_ETAG (sed -En 's/ETag: "(\w+)"/\1/p' $TMP_CONFIG.headers)
end
Notes:
Warning: Not tested nearly enough
Assumes fish v2.3 or higher.
sed behavior varies from platform to platform.
Replace woj/dotfiles/master/fish/config.fish with the repository, branch and path that apply to your case.
You can run this from a cron job, but if you insist to update the configuration file on every init, change the script to place the configuration in a path that's not already automatically loaded by fish, e.g.:
mv $TMP_CONFIG ~/.config/fish/shared_config.fish
and in your config.fish run this whole script file, followed by a
source ~/.config/fish/shared_config.fish
I recently configured a little server for test some services, now, before an upgrade or install new software, I want to make an exact copy of my files, with owners, groups and permissions, also the symlinks.
I tried with rsync to keep the owner and group but in the machine who receives the copy I lost them.
rsync -azp -H /directorySource/ myUser#192.168.0.30:/home/myUser/myBackupDirectory
My intention is to do it with the / folder, to keep all my configurations just in case, I have 3 services who have it's own users and maybe makes modifications in folders outside it's home.
In the destination folder appear with my destination user, whether I do the copy from the server as if I do it from the destination, it doesn't keep the users and groups!, I create the same user, tried with sudo, even a friend tried with 777 folder :)
cp theoretically serves the same but doesn't work over ssh, anyway I tried to do it in the server but have many errors. As I remembered the command tar also keep the permissions and owners but have errors because the server it's working and it isn't so fast the process to restore. I remember too the magic dd command, but I made a big partition. Rsync looked the best option to do it, and to keep synchronized the backup. I saw rsync in the new version work well with owners but I have the package upgraded.
Anybody have some idea how I do this, or how is the normal process to keep my own server well backuped, to restore just making the partition again?
The services are taiga, a project manager platform, a git repository, a code reviewer, and so on, all are working well with nginx over Ubuntu Server. I haven't looked other backup methods because I thought rsync with a cron job do the work.
Your command would be fine, but you need to run as root user on the remote end (only root has permission to set file owners):
rsync -az -H /directorySource/ root#192.168.0.30:/home/myUser/myBackupDirectory
You also need to ensure that you use rsync's -o option to preserve owners, and -g to preserve groups, but as these are implied by -a your command is OK. I removed -p because that's also implied by -a.
You'll also need root access, on the local end, to do the reverse transfer (if you want to restore your files).
If that doesn't work for you (no root access), then you might consider doing this using tar. A proper archive is probably the correct tool for the job, and will contain all the correct user data. Again, root access will be needed to write that back to the file-system.
My source is an old external hard drive formatted HFS+ (used to be data from server running 10.4.11) connected to an iMac running 10.8.5 with an upgraded version of rsync 3.0.9.
The destination is a Centos 6.4 server running rsync 3.0.9 as well.
We have tried to transfer a FONTS folder (Source size = 4.7GB) to the destination but the size of the folder is not kept (Destination size = 655MB).
Below is the command that I run to preserve hard links - ACLs.....
/usr/local/bin/rsync -aHpEXotg --devices --specials --ignore-errors /Users/london/Desktop/FONTS root#192.168.200.253:/home/TEST
Also getting errors: rsync: rsync_xal_set: lsetxattr(""/home/TEST/FONTS/ Folder/Kfz-EURO Schrift MAC+PC/MAC/FE Mittelschrift.image"","user.com.apple.FinderInfo") failed: Operation not supported (95)
Most of the files are showing as Unix files and can't be open.
This issue has been time consuming so if someone can guide me.
Thanks..
Ran across this today as I encountered similar errors. I ended up running rsync with minimum options to complete the copy:
rsync -r --progress /path/to/source /path/to/destination
-r is recursive
--progress shows additional copy info (versus v for verbose output)
If you leave out --progress, rsync will only show you files that error and will transfer the rest - that can be useful to know which files you're not getting if there aren't very many with errors. Course, alternatively if there are a lot of errors, that can indicate bad sectors on the drive.
HI,
Actually i am trying to reduce the time taken to transfer N-number of files from remote machine local machine using secure file transfer, previously i used scp system command it establishes a connection for each file transfer.
Thanks in advance.
Unless you have a bandwidth cap on each individual TCP connection, you are not going to get a significant reduction in download time by using multiple SCP connections.
You can check whether you get a speed up by putting a separate scp command for each file in a shell script and timing the script. Then rerun the shell script with & at the end of each scp line. If this speeds up the transfer and you want to really do this in Perl, look into fork or Parallel::ForkManager.
I think this would create a separate connection each time. However, scp with the -r flag (which Net::SCP uses) recursively copies all of the files in a directory with a single connection. That might be the way to go, if you have your files in a few directories and you want to copy all of the files in those directories.
Otherwise, rsync with the --files-from option should use only one connection. (Don't forget -z for compression, or -a).
If the only reason you're considering using perl to do this is that you want a single session, then just use the command line rsync to get this effect (with --files-from). If you want perl power to generate the list of files-from, File::Rsync supports that.