I use wget to download an entire website with all included assets, the problem is that wget does not download included videos.
For example with this website, if I execute the following command :
wget -q -r ‐‐page-requisites http://videohtml5.byethost11.com/index.html
It download almost everything but if you open the web page, you'll see that the video is not downloaded.
I have tried the following options without results:
-r : for recursion
--page-requisites : to download all included assets
However if I directly put the link to the video as an option of wget it works :
wget -q -r ‐‐page-requisites http://videohtml5.byethost11.com/movie.mp4
But I would like to download everything in one command. I have read the wget manual but I didn't see any other option that could do that. That's why I am asking your help.
EDIT : I change the url to really match my need
SOLUTION : Because I am using Windows, I didn't get the latest released which has the fix for the bug. Do not download wget from http://gnuwin32.sourceforge.net/packages/wget.htm, but use https://eternallybored.org/misc/wget/ instead.
The video is hosted at a different domain: you need the -H parameter.
See the manpage section about spanning hosts: https://www.gnu.org/software/wget/manual/wget.html#Spanning-Hosts
== Update ==
It seems wget has a bug preventing to download the <source> of the <video> tag. See https://lists.gnu.org/archive/html/bug-wget/2013-06/msg00070.html
This works as you expect:
wget -H -r --level=1 -k -p http://camendesign.com/code/video_for_everybody/
...
drwxr-xr-x 24 root root 4096 Apr 17 10:08 camendesign.com
drwxr-xr-x 2 root root 4096 Apr 17 10:08 clips.vorwaerts-gmbh.de
drwxr-xr-x 2 root root 4096 Apr 17 10:08 forum.camendesign.com
-rw-r--r-- 1 root root 13700 May 12 2013 test.html
drwxr-xr-x 2 root root 4096 Apr 17 10:08 www.youtube.com
root#test /tmp/test# cd clips.vorwaerts-gmbh.de/
root#test /tmp/test/clips.vorwaerts-gmbh.de# ll
total 5396
-rw-r--r-- 1 root root 5510872 Feb 9 2010 big_buck_bunny.mp4
Related
I am trying to configure an user account using one cloud-init yaml file that include a call to write_files module, like this:
write_files:
#passwd file for vncserver
- path: /home/ubuntu/.vnc/passwd
owner: ubuntu:ubuntu
permissions: '0600'
defer: true
encoding: b64
content: bmtzZGN1eQo=
The file is created as expected, but the problem is that the parent directory is owned by root, and not by ubuntu user.
$ ls -la .vnc/
total 12
drwxr-xr-x 2 root root 4096 Dec 20 16:24 .
drwxr-x--- 5 ubuntu ubuntu 4096 Dec 20 16:24 ..
-rw------- 1 ubuntu ubuntu 8 Dec 20 16:24 passwd
I tried to manually create the /home/ubuntu/.vnc/ directory prior to create the passwd file to be able to set the ownership of the directory, just to find that documentation of write_files does not explain how to create (empty) directories.
I know that I could do this using runcmd module to insert a command like this:
runcmd:
- mkdir --mode 0600 --parents /home/ubuntu/.vnc
- echo bmtzZGN1eQo | base64 -d > /home/ubuntu/.vnc/passwd
- chmod 0600 /home/ubuntu/.vnc/passwd
but this seems to be too complex to do such small task.
It is possible to use write_files module to create directories or change ownership/permission of existing directories?
I'm currently reading a book on programming with C, I got to a part where I've got to write a program which will display the real uid and effective uid that the file is being executed on. After compiling the code with gcc, I input the command to see the current uOwner and gOwner ls- l id_demo the output is this:
-rwxrwxr-x 1 user user 8629 Sep 21 13:04 id_demo
I then execute the program itself, this is what I get:
real uid: 1000 effective uid: 1000
...so far so good. I then input a command to change the owner of the file:
sudo chown root:root ./id_demo
The ls -l confirms that the owner has been changed to root:
-rwxrwxr-x 1 root root 8629 Sep 21 13:04 id_demo
Again, executing the program shows real uid and uid as 1000. The last step after which the uid must be 0 is this: sudo chmod u+s ./uid_demo but for me they stay as 1000, where in the book the output is clearly show to be this:
real uid: 1000
effective uid: 0
Any ideas why is this happening?
UPDATE
id_demo source code:
#include <stdio.h>
int main ()
{
printf("real uid: %d\n", getuid());
printf("effective uid: %d\n", geteuid());
}
UPDATE 2
Screen shots
PLEASE HELP. I'm going crazy I spent 6+hour looking for the solution and I need to move on.
We've figured it out. The cause is an ecryptfs-mounted home directory. The mount output contains the following line:
/home/evgeny/.Private on /home/evgeny type ecryptfs
That means that the home directory isn't actually part of the root filesystem (that has the necessary suid flag), but its own virtual filesystem that apparently doesn't support setuid binaries by default. I have successfully reproduced the issue with a test user that has an encrypted home directory.
It is possible to add the suid flag to the ecryptfs with the following command:
sudo mount -i -o remount,suid /home/evgeny
I'm not certain though how safe that is, nor how to change it permanently so that it would survive reboots.
This works for me:
compile
$ gcc uid_demo.c -o uid_demo
$ ll
total 12
-rwxrwxr-x 1 saml saml 6743 Sep 21 17:05 uid_demo
-rw-rw-r-- 1 saml saml 116 Sep 21 16:58 uid_demo.c
chown
$ sudo chown root:root uid_demo
$ ll
total 12
-rwxrwxr-x 1 root root 6743 Sep 21 17:05 uid_demo
-rw-rw-r-- 1 saml saml 116 Sep 21 16:58 uid_demo.c
chmod
$ sudo chmod u+s uid_demo
$ ll
total 12
-rwsrwxr-x 1 root root 6743 Sep 21 17:05 uid_demo
-rw-rw-r-- 1 saml saml 116 Sep 21 16:58 uid_demo.c
run
$ ./uid_demo
real uid: 500
effective uid: 0
I have a yum repository I've set up where I store custom rpms.
I have no problem finding information about other packages that were built and stored in this custom repo.
#yum --disablerepo=rhui-us-east-rhel-server-1y,epel,epel-testing --enablerepo=customrepo install php53-pecl-xdebug
php53-pecl-xdebug x86_64 2.2.1-2 customrepo 132 k
No problem.
Now I drop somerpm.rpm in centos/repo/5/noarch, run createrepo --update . in this directory and try the same command, and yet it shows no results.
I tried running createrepo --update in the root of the repo as well, but that did not work either (I'm actually not sure where to run it and if it needs a repodata directory in each subdir).
[root#reposerver mnt]# ls -l /var/www/repo/
total 12
-rw-r--r-- 1 root root 203 Jun 8 00:13 REPO_README
drwxr-xr-x 3 root root 4096 Jun 10 2011 centos
drwxr-xr-x 2 root root 4096 Oct 18 20:02 repodata
[root#reposerver mnt]# ls -l /var/www/repo/centos/5/
SRPMS/ i386/ noarch/ repodata/ x86_64/
[root#reposerver mnt]# ls -l /var/www/repo/centos/5/noarch/
total 7324
-rw-r--r-- 1 root root 1622 Jun 28 2011 compat-php-5.1.6-1.noarch.rpm
drwxr-xr-x 2 root root 4096 Oct 18 19:55 repodata
-rw-r--r-- 1 root root 1066928 Oct 18 19:54 salt-0.10.3-1.noarch.rpm
-rw-r--r-- 1 root root 6363197 Oct 18 19:54 salt-0.10.3-1.src.rpm
-rw-r--r-- 1 root root 21822 Oct 18 19:54 salt-master-0.10.3-1.noarch.rpm
-rw-r--r-- 1 root root 14294 Oct 18 19:54 salt-minion-0.10.3-1.noarch.rpm
I also tried adding the exactarch=0 flag to my repo config to ignore arch restrictions and this did not work either, it was a shot in the dark, since my rpm is noarch, it should show regardless.
# cat /etc/yum.repos.d/mycompany.repo
[mycompany]
name=mycompany custom repo
baseurl=http://config/repo/centos/5/$basearch
enabled=1
exactarch=0
I'm at a loss at this point. Usually createrepo --update does the trick, but for some reason it cannot find the new rpms.
repo]# find . -type f -name "*.gz" | xargs zcat | grep salt-minion
returns results as well, so it's definitely in the repo data.
yum clean all on the server I was trying to install on worked.
Also make sure to do createrepo --update on the specific subdir instead of the root of the repo.
I'm trying to get gitweb setup on a CentOS 6.2 server with git/gitweb 1.7.1 and httpd 2.2.15 installed.
gitweb's default project root (verified in the CGI script) is /var/lib/git, so I've created that and a bare git repository in there:
$ ls -laF /var/lib/git
total 12
drwxrwxr-x. 3 git git 4096 Feb 8 16:37 ./
drwxr-xr-x. 15 root root 4096 Feb 8 14:20 ../
drwxrwxr-x. 7 git git 4096 Feb 8 15:37 foo/
$ git init --bare --shared foo
Initialized empty shared Git repository in /var/lib/git/foo/
$ ls -lF foo
total 32
drwxrwsr-x. 2 git git 4096 Feb 8 17:16 branches/
-rw-rw-r--. 1 git git 126 Feb 8 17:16 config
-rw-rw-r--. 1 git git 73 Feb 8 17:16 description
-rw-rw-r--. 1 git git 23 Feb 8 17:16 HEAD
drwxrwsr-x. 2 git git 4096 Feb 8 17:16 hooks/
drwxrwsr-x. 2 git git 4096 Feb 8 17:16 info/
drwxrwsr-x. 4 git git 4096 Feb 8 17:16 objects/
drwxrwsr-x. 4 git git 4096 Feb 8 17:16 refs/
$ cat foo/HEAD
ref: refs/heads/master
However on viewing http://localhost/git/, I see "404 No projects found".
I've debugged through the script and can see that it's finding /var/lib/git/foo, but Perl's -e operator fails on /var/lib/git/foo/HEAD. At the same place in the file, a backticked call to ls shows that the file is visible there, but I cannot make Perl -e see the file.
Any idea what might be making this fail? This makes no sense to me.
EDIT: note that SELinux extensions on this CentOS box appear to be disabled:
$ sudo sestatus
SELinux status: disabled
EDIT: moving everything from /var/lib/git to /git hasn't helped. I've changed the apache user to have a real shell, logged in as that user, and verified that it has access to all the directories and files in question.
It was in fact SELinux. Even though SELinux reported it was disabled, it was somehow preventing access to some files for CGI scripts running under httpd. Enabling SELinux and setting it to permissive mode, it started working.
This seems highly non-intuitive and frustrates me, but at least it's working.
I'm still thinking it's an issue with permissions... but I could be wrong. Have you ensured that all of the parent directories leading up to your /var/lib/git directory are permission-accessible?
Someone else had a similar problem here and it might be worth trying a completely different directory... maybe even /opt.
By following the official instructions http://www.mongodb.org/display/DOCS/Quickstart+Unix and this post http://blog.phy5ics.com/2010/03/27/installing-mongodb-on-mediatemple-dv/ I've just about managed to get mongodb installed on MediaTemples DV 4.0 server (I think).
I am however having problems installing the PHP driver http://www.mongodb.org/display/DOCS/PHP+Language+Center
In SSH I get this:
[root#xxx]# cd /var/tmp
[root#xxx]# pecl install mongo
downloading mongo-1.1.4.tgz ...
Starting to download mongo-1.1.4.tgz (68,924 bytes)
.................done: 68,924 bytes
18 source files, building
running: phpize
Configuring for:
PHP Api Version: 20090626
Zend Module Api No: 20090626
Zend Extension Api No: 220090626
/usr/bin/phpize: /var/tmp/mongo/build/shtool: /bin/sh: bad interpreter: Permission denied
Cannot find autoconf. Please check your autoconf installation and the $PHP_AUTOCONF environment variable. Then, rerun this script.
ERROR: `phpize' failed
I am logged in as the root user - I don't understand why it's failing and what steps I need to take to install the PHP driver?
Thanks
Run the following commands on your server's command line:
$ mkdir /root/tmp
$ mount --bind /root/tmp /tmp
$ umount /tmp; umount /var/tmp
$ pecl install mongo
A few things:
/root/tmp is just an arbitrary temp directory. You can use whatever you want, provided it exists.
Some instructions say to use --host instead of --bind. On RHEL/CentOS mount says --host is an unrecognized option.
If you're on a VM, it's likely that you'll have to do this each time you restart your VM/Container.
For Media Temple customers, I can confirm that this works on both (dv) and (ve) servers with CentOS 5 and 6.
From media temple support: Need to create a temporary directory (/root/tmpz):
$ mkdir /root/tmpz
$ mount --host /root/tmpz /tmp
$ umount /tmp; umount /var/tmp
$ pecl install mongo
Build complete.
Don't forget to run 'make test'.
running: make INSTALL_ROOT="/var/tmp/pear-build-root/install-mongo-1.1.4" install
Installing shared extensions: /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php /modules/
running: find "/var/tmp/pear-build-root/install-mongo-1.1.4" | xargs ls -dils
69094140 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4
69275176 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr
69275177 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64
69290445 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php
69290447 4 drwxr-xr-x 2 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php/modules
69290448 676 -rwxr-xr-x 1 root root 684126 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php/modules/mongo.so
Build process completed successfully
Installing '/usr/lib64/php/modules/mongo.so'
install ok: channel://pecl.php.net/mongo-1.1.4
configuration option "php_ini" is not set to php.ini location
You should add "extension=mongo.so" to php.ini
Do you have php-dev installed? phpize is basically "compiling" the MongoDB driver, but unless you have the -dev installed, this may not work.