We're migrating home folders to a new filesystem, and I am looking for a way to automate it using Perl or a shell script. I don't have much choice in programming languages as the systems are proprietary storage clusters that should remain as unchanged as possible.
Task: Under directory /home/ I have various users' home folders aaa, bbb, ccc, ... and they have certain permissions and user/group ownership that need to remain intact upon migration to /newhome/. Here's example of what needs to be migrated from /home:
drwxr-xr-x 3 aaaaa xxxxxxxxx 4096 Feb 26 2008 aaaaa/
drwxrwxrwx 88 bbbbbbb yyyyyy 8192 Dec 16 16:32 bbbbbbb/
drwxr-xr-x 6 ccccc yyyyyy 4096 Nov 24 04:38 ccccc/
drwxr-xrwx 36 dddddd yyyyyy 4096 Jun 20 2008 dddddd/
drwxr-xr-x 27 eee yyyyyy 4096 Dec 16 02:56 eee/
So, exact same folders with permissions and ownerships should be created under /newhome. Copying/moving files should not be a concern, as it will be handled later.
Anyone has worked on such script? I am really new to Perl, so I need help.
cp's -a flag will maintain permission, modification times etc. You should for be able to do something like:
for a in `ls /home`; do cp -a "/home/$a" "/newhome/$a" ; done
Try it with one directory to see if does what you need before automating it.
EDIT: You can disable recursive file copying by using rsync or tar as mentioned by Paul. With rsync, subdirectories are still preserved, but files aren't copied:
sudo rsync -pgodt /home/ /newhome/
I haven't tried tar's --no-recursion, so can't comment on it.
EDIT 2: Another way
find /home/ -maxdepth 1 -print | sudo cpio -pamVd /newhome
Reference
You can only preserve the owner and group if you do the copying operation as root. Most of the commands given will work - the tar and the cp -rp options will.
The only trick to worry about is non-writable directories, but that's an issue for non-root users. Then, I tend to use:
(cd /home; find . -depth) | cpio -pvdumB /newhome
The -depth option means that file and sub-directories are processed before the directories themselves, so the no-write permission on the directory is only set after all the contents of the directory have been copied into it. You can also use a 'sort -r' to list files in reverse order, which ensures that directories appear after their contents.
This will create the directories and copy all the files.
cd /home; tar cvBf - . | (cd /newhome; tar xvpBf -)
If you don't want to copy all the files, you might be able to do that by adding a "--no-recursion" to the first tar command.
If these directories are on the same filesystem, why not simply
cp -p /home/* /newhome/
Related
I am using MobaXterm portable.
I found a strange setup, summarized here.
External commands in /bin work fine. E.g., with /bin/ssh.exe I can ssh ok.
Internal commands are
"redirected" to busybox, as
$ which cat
/bin/cat
$ ll /bin/cat
lrwxrwxrwx 1 USER001 UsersGrp 16 Jul 24 07:42 /bin/cat -> /bin/busybox.exe
at the same time aliased to files that apparently do not exist.
$ type cat
cat is aliased to `/bin/cat.exe'
These aliases apparently take precedence over files in PATH, so the commands do not work.
$ cat myfile
bash: /bin/cat.exe: No such file or directory
If I unalias, cat does not look for /bin/cat.exe but for /bin/busybox.exe, and everything is "back to normal".
$ unalias cat
$ cat myfile
Hello world
...
How can I get normal behaviour (either without aliases or with the presence of the alias targets)?
I mean not to write my own unaliases in .bashrc, this shouldn´t be needed.
Moreover, perhaps I would be breaking something.
Why would MobaXterm setup things like this?
PS: In the initial state, even ls does not work, for the same reason.
But ll works, since
$ type ll
ll is aliased to `_bbf ls -l'
$ type _bbf
_bbf is a function
...
How can I get normal behaviour?
Workarounds:
unaliasing by hand, so /bin/busybox.exe is actually used.
Below I add a script for that.
Copying .exe files from the temporary root dir when it is available, so the external versions are used.
Why would MobaXterm setup things like this?
When not using a Persistent root (/) directory, this is obtained
$ which cat
/bin/cat
$ ll /bin/cat
-rwxr-xr-x 1 RY16205 UsersGrp 49703 jul. 28 07:12 /bin/cat
$ type cat
cat is aliased to `/bin/cat.exe'
$ ll /bin/cat.exe
-rwxr-xr-x 1 USER001 UsersGrp 49703 jul. 28 07:12 /bin/cat.exe
$ cat myfile
Hello world
...
$ unalias cat
$ type cat
cat is hashed (/bin/cat)
$ cat myfile
Hello world
...
So any of the two cats work (internal busybox and external versions; I do not know if they are exactly the same).
This is because /bin points to C:\Users\user001\AppData\Local\Temp\Mxt108\bin and cat.exe is there.
But when using a Persistent root (/) directory, /bin points to <Persistent root (/) directory\bin, and cat.exe is not created there.
The former temporary root dir is removed as soon as MXT is closed.
So this is probably a configuration error from MobaXterm.
If so, the only option seems a workaround, as above.
Script for unaliasing:
#!/bin/bash
export ROOTDIR_WIN=$(cygpath -wl /)
if [[ ${ROOTDIR_WIN##*\\} == "Mxt108" ]] ; then
# Not using a Persistent root dir. Do not need to unalias.
echo "Not using a Persistent root dir. Do not need to unalias."
else
# Using a Persistent root dir. Need to unalias.
exe_linux_list="bash busybox chmod cygstart cygtermd cygwin-console-helper dircolors dwm_w32 echo grep ls MoTTY ssh ssh-pageant test twm_w32 wc xkbcomp_w32 XWin_MobaX"
for exe_linux in ${exe_linux_list} ; do
if [[ $(type -t ${exe_linux}) == "alias" ]] ; then
#type ${exe_linux}
unalias ${exe_linux}
fi
done
fi
On my MobaXterm system, /etc/profile sources /etc/baseprofile which includes aliases for all of these sorts of things, i.e.
alias "cat"="_bbf cat"
and checking that from my command prompt yields what I would expect:
$ type cat
cat is aliased to `_bbf cat'
Have you changed your system somehow so that /etc/baseprofile is not being sourced? Or have you changed /etc/baseprofile?
It also appears that you've installed the regular GNU Coreutils package, as I don't have a /bin/cat.exe.
$ ll /bin/cat.exe
ls: /bin/cat.exe: No such file or directory
Perhaps that's where your problem started but the _bbf function is supposed to handle that. Which again leads me to the belief that you've changed /etc/baseprofile somehow.
At most time, it is cool. This error maybe caused by wrong path match of cat.exe.
As for me, when I run git log, the same error message comes out. It is due to PATH variable. There are two dirs and both of them contain git.exe. One of them is half-done with a small size. And Mobaxterm choose it. :D
I confirm this by run which git and it will give the actual path out.
I fix it by
alias git='/drives/C/Program\ Files/Git/mingw64/bin/git.exe'
The following is my dirs.
├─cmd
│ git-gui.exe
│ git-lfs.exe
│ git.exe # oops
│ gitk.exe
│ start-ssh-agent.cmd
│ start-ssh-pageant.cmd
├─mingw64
│ ├─bin
│ │ git-upload-pack.exe
│ │ git.exe # the right one
I need to free up some disk space on my web server and would like to ask if running the command below would break anything?
My server is running centos 6 with cpanel/whm.
$ find / -type f -name "*.tar.gz" -exec rm -i {} \;
Any help or advice will be greatly appreciated.
Thanks.
Well, you'll lose logs if they are already compressed, or uploaded files if any. By default there shouldn't be any of those files on installed system. Personally I think this is wrong to just jettison what you can instead of trying to find the cause.
You can try finding what's occupying space by running:
du -hs / # shows how much root directory occupies
Compare that to the output of:
df -h # shows used space on disks
If the number didn't match by a far - you probably have unclosed deleted files and a simple reboot will reclaim this space for you.
If not you can proceed by recursively doing:
cd <dir>; du -hs * # enter directory and calculate size of its contents
You can do that starting from / and proceeding to the biggest dir. After all you'll find your source of free space. :)
PS: CentOS doesn't compress logs by default. You will not detect those logs by searching for archived files, but they can be huge. Compressing them is an easy way to get some space:
Turn on compression in /etc/logrotate.conf:
compress
Compress already rotated logs with:
cd /var/log; find . -type f | grep '.*-[0-9]\+$' | xargs -n1 gzip -9
How can I pipe a text stream into a file and, while the file is still in use, wipe it for job rotation?
Long version:
I've been struggling for a while onto an apparently minor issue, that's making my experiments impossible to continue.
I have a software collecting data continuously from external hardware (radiotelescope project) and storing in a csv format. Being the installation at a remote location I would, once a day, copy the saved data in a secure place and wipe the file content while, for the same reason, I can NOT to stop the hardware/software, thus software such as log rotation wouldn't be an option.
For as much effort spent see my previous post, it seems the wiped file keeps growing although empty.
Bizarre behavior, showing file size, truncate file, show file size again:
pi#tower /media/data $ ls -la radio.csv ;ls -s radio.csv;truncate radio.csv -s 1; ls -la radio.csv ;ls -s radio.csv
-rw-r--r-- 1 pi pi 994277 Jan 18 21:32 radio.csv
252 radio.csv
-rw-r--r-- 1 pi pi 1 Jan 18 21:32 radio.csv
0 radio.csv
Then, as soon as more data comes in:
pi#tower /media/data $ ls -la radio.csv ;ls -s radio.csv
-rw-r--r-- 1 pi pi 1011130 Jan 18 21:32 radio.csv
24 radio.csv
I thought to pipe the output into a sed command and save right away, with no luck altogether. Also, filesystem/hardware doesn't seems buggy (tried different hardware/distro/filesystem).
Would anyone be so nice to give me a hint how to proceed?
Thank you in advance.
Piped into tee with -a option. The file was kept open by originating source.
Option APPEND of tee helped to stick at the EOF new data; in the given issue, appending to the beginning when file zeroed.
For search engine and future reference:
sudo rtl_power -f 88M:108M:10k -g 1 - | tee -a radio.csv -
Now emptying the file with
echo -n > radio.csv
gets the file zeroed as expected.
Does anybody know if there is an rsync option, so that directories that are being traversed do not show on stdout.
I'm syncing music libraries, and the massive amount of directories make it very hard to see which file changes are actually happening.
I'v already tried -v and -i, but both also show directories.
If you're using --delete in your rsync command, the problem with calling grep -E -v '/$' is that it will omit the information lines like:
deleting folder1/
deleting folder2/
deleting folder3/folder4/
If you're making a backup and the remote folder has been completely wiped out for X reason, it will also wipe out your local folder because you don't see the deleting lines.
To omit the already existing folder but keep the deleting lines at the same time, you can use this expression :
rsync -av --delete remote_folder local_folder | grep -E '^deleting|[^/]$'
I'd be tempted to filter using by piping through grep -E -v '/$' which uses an end of line anchor to remove lines which finish with a slash (a directory).
Here's the demo terminal session where I checked it...
cefn#cefn-natty-dell:~$ mkdir rsynctest
cefn#cefn-natty-dell:~$ cd rsynctest/
cefn#cefn-natty-dell:~/rsynctest$ mkdir 1
cefn#cefn-natty-dell:~/rsynctest$ mkdir 2
cefn#cefn-natty-dell:~/rsynctest$ mkdir -p 1/first 1/second
cefn#cefn-natty-dell:~/rsynctest$ touch 1/first/file1
cefn#cefn-natty-dell:~/rsynctest$ touch 1/first/file2
cefn#cefn-natty-dell:~/rsynctest$ touch 1/second/file3
cefn#cefn-natty-dell:~/rsynctest$ touch 1/second/file4
cefn#cefn-natty-dell:~/rsynctest$ rsync -r -v 1/ 2
sending incremental file list
first/
first/file1
first/file2
second/
second/file3
second/file4
sent 294 bytes received 96 bytes 780.00 bytes/sec
total size is 0 speedup is 0.00
cefn#cefn-natty-dell:~/rsynctest$ rsync -r -v 1/ 2 | grep -E -v '/$'
sending incremental file list
first/file1
first/file2
second/file3
second/file4
sent 294 bytes received 96 bytes 780.00 bytes/sec
total size is 0 speedup is 0.00
I'm trying to tar up all the *.class files only on a Solaris box under a certain directory.
Reading the man pages for tar made it seem like the -I option is what I wanted.
This is what I've tried from the dir in question:
find . -name "*.class" >> ~/includes.txt
tar cvf ~/classfiles.tar -I ~/includes.txt
From that I get:
tar: Removing leading `/' from member names
/home/myhomedir/includes.txt
And the ~/classfiles.tar files is garbage.
I don't have write permission on the dir where the *.class files are so I need to have the tar written to my home dir. Could someone tell me where I have gone wrong? What tar magic should I use?
Check which tar you are running. That message about removing the leading slash is a gtar (GNU tar) message, and the -I option you are trying to use is a Sun tar option (which lives in /bin/tar).
(at least the above is all true on my Solaris machine)