I know if you run in rsync
rsync some.domain.com::
It will return me a list of the rsync depots. Is there any way of getting it to return the details of the depot, the path specifically.
Thanks
No, rsyncd is specifically designed not to reveal the physical path of the modules. Now, if you have shell access to the rsyncd server, you can read /etc/rsyncd.conf for that information.
(But, there may be ways to exploit rsyncd to reveal the path, if the use chroot setting is off. Don't quote me on that, though.)
Related
I am attempting to regularly archive a few file types hosted on a community website where our admin has been MIA for years, in case he dies or just stops paying for the hosting.
I am able to download all of the files I need using wget -r -np -nd -e robots=off -l 0 URL but this leaves me with about 60,000 extra files to waste time both downloading and deleting.
I am really only looking for files with the extensions "tbt" and "zip". When I add in -A tbt,zip to the input, wget then only downloads a single file, "index.html.tmp". It immediately deletes this file because it doesn't match the file type specified, and then the process stops entirely, with wget announcing that it is finished. It does not attempt to download any of the other files that it grabs when the -A flag is not included.
What am I doing wrong? Why does specifying file types in the way that I did cause it to finish after only looking at one file?
Possibly you're hitting the same problem I've hit when trying to do something similar. When using --accept, wget determines whether a links refers to a file or directory based on whether or not it ends with a /.
For example, say I have a directory named files, and a web page that has:
Lots o' files!
If I were to request this with wget -r, then I wget would happily GET /files, see that it was an HTML document containing a bunch of links, and continue to download those links.
However, if I add -A zip to my command line, and run wget with --debug, I see:
appending ‘http://localhost:8080/files’ to urlpos.
[...]
Deciding whether to enqueue "http://localhost:8080/files".
http://localhost:8080/files (files) does not match acc/rej rules.
Decided NOT to load it.
In other words, wget thinks this is a file (no trailing /) and it doesn't match our acceptance criteria, so it gets rejected.
If I modify the remote file so that it looks like...
Lots o' files!
...then wget will follow the link and download files as desired.
I don't think there's a great solution to this problem if you need to use wget. As I mentioned in my comment, there are other tools available that may handle this situation more gracefully.
It's also possible you're experiencing a different issue; the output of adding --debug to your command line clarify things in that case.
I also experienced this issue, on a page where all the download links looked something like this: filedownload.ashx?name=file.mp3. The solution was to match for both the linked file, and the downloaded file. So my wget accept flag looked like this: -A 'ashx,mp3'. I also used the --trust-server-names flag. This catches all the .ashx that are linked in the webpage, then when wget does the second check, all the mp3 files that were downloaded will stay.
As an alternative to --trust-server-names, you may also find the --content-disposition flag helpful. Both flags help rename the file that gets downloaded from filedownload.ashx?name=file.mp3 to just file.mp3.
I'm writing a syntax check tool to parse several files on different branches.
Is there a way for me to read the contents without checking out the file?
The tool is written in Perl.
`p4 print //depot/path/to/file`;
(Usual requirements for running a p4 command apply -- make sure the p4 executable is in your PATH, make sure you're authenticated with p4 login, make sure you're connecting to the right server, etc.)
See p4 help print for more info on the print command -- you might find the -q and/or -o flags helpful depending on what exactly you need to do with the output.
Scenario:
I have a process where many files are being copied (scp'd) to a DestinationServer by Host1, Host2, Host3, Host4 for example. Going to the same common directory: DestinationServer:/home/target. All the files are unique so no files will be overwritten. Host1-Host4 will have a cronjob that will launch their scp script to DestinationServer. The caveat is the Hosts are in different time zones, locations. So, they will finish at different times.
Need:
Since the files are being scp'd to Destination:/home/target, what is the best way to programmatically check when those scp's from the other Hosts are done??
Options:
My options are to programmatically do this either in perl or shell if possible.
What do I look for, what unix commands or perl modules could I use to help determine when the processes would finish? Any ideas, examples would be great! Thanks.
Use a Maildir kind of approach: copy all files to a temporary directory, then after the transfer is complete have the originating host perform a rename into the target directory via ssh. That way when a file appears in the target directory, you know that it is complete.
I suggest this because if you just scp files into the target directory and monitor the directory in whatever way, you cannot distinguish a complete transfer from an interrupted scp command or a network failure.
SGI::FAM, Sys::Gamin
Similar but alternative way to Jouni is to use semaphore files. Before scp-ing files originating host puts up semaphore-file and when finished, remove it. So you know, it's time.
I have a project that depends upon some other binaries to be downloaded from web at install time.For this what i do is:
if ( file-present-in-src/)
# skip that file
else
# use wget to download the file
The problem with this approach is that when I interrupt a download in middle, and do invoke the script next time, the partially downloaded file is also skipped (which is not desired), also I want wget to resume the download of the partially downloaded file.
How should I go about it:
Possible Solutions I could think of:
Let the file to be downloaded to some file say download_tmp. Move to original file
if successful.
Handle SIG{'INT'} to write proper cleanup code.
But none of these could help resume the partial file download,
Any insights?
Fist, I don't understand what this has to do with Perl, since you're using wget to do the dowloading ... You could use libwww-perl (perldoc LWP) and have more control about the download process.
Then I second your idea of downloading to a "tmp" filename and move the file on success.
However I think you need to go further and verify the integrity of the files. Doing an MD5 or SHA hash is very easy, and match the downloaded one with what you're expecting. You can have a short file on server containing the checksum (filename.md5). Determine success only when you have a match.
Note that catching all the signals and generally trying to make the process unkillable, and then expecting it to have worked is bound to fail at one point or another. There could be a network timeout, a crash, power failure, configuration problem on the server ... you should instead assume downloads can fail, because they will, and code so that your process can recover.
Finally you're not telling us what kind of binaries you're downloading and what you're doing with them. Since you use wget I'm going to assume you're on Unix; you should consider using RPM+Yum or the likes, they handle all this for you. RPM are easy to write, really.
use your first approach ..
download to "FileName".tmp
move "FileName".tmp to "FileName" move! not copy
once per diem clean out all .tmp files (paranoia rulez)
You could just use wget's -N and -c options and remove the entire "if file exists" logic.
My hosted scripts have been moved and no longer work.
The specified CGI application
misbehaved by not returning a complete
set of HTTP headers.
I notice that someone at my host company has modified my scripts so that where I used to have
use lib 'd:/myorig/LIB';
I now have
use lib '//newhost/LIB';
Should this work?
I tried 1800 INFORMATION's suggestion and ran the minimal script of
#!perl -w
use lib '//whatever/lib';
print "success";
...which gave the same result.
Update: ysth's suggestion of FatalsToBrowser did indeed reveal more information. It looks like the path (added by someone from the hosting company) might be wrong.
Update2: The hosting company now says that these scripts, unchanged from the previous host mind, are throwing lots of syntax errors. "Since we cannot debug your scripts for you we suggest you contact the original programmer and ask them for help". <grinds teeth>
Partial Resolution: The hosting company finally realised they hadn't set permissions correctly. They still aren't right, and (aargh) they don't allow site owners to set folder permissionsn, not even on folders within their own sites.
I don't know if it should work or not, but my intuition is that it would be okay. However, the two use lib lines you posted are not equivalent.
# go to the 'd' drive and use the 'myorigLIB' directory on that drive
use lib 'd:/myorigLIB';
# go to the 'newhostLIB' server - no path is specified - this looks invalid to me
use lib '//newhostLIB';
Perhaps you need to specify the path to the share on the server? Also, you might need to look at permissions? Maybe the user the CGI is running as cannot access that network path?
Also, you could write a simple (non CGI) program to test your theory and just run it:
#!perl -w
use lib '//whatever/lib';
print "success";
Then just run that on the server if you can and see what happens.
No the path is incomplete it needs both a server name and a complete path. It is a bad practice as well because it requires that two machines be monitored rather than one for your application to function.
The specified CGI application misbehaved by not returning a complete set of HTTP headers.
That's a non-error. If you are lucky, your hosting company will make an error log available to you that will show the actual error that perl is dying with. If not,
consider using
use CGI::Carp "fatalsToBrowser";
for testing. (If you are paranoid (which is not a bad thing to be), you will refrain from leaving that enabled once you are done testing, since errors can commonly provide information about your code or even your database that may help a black hat exploit security holes.)
I know I ran into trouble trying to use mapped drives and unc paths from apache because the apache user was not allowed to use network drives. That was difficult to figure out -- but it's possible to do it. That may be a related problem.
#!perl -w
print "HTTP/1.0 200 OK\nContent-Type: text/plain\n\n";
my $path = "//whatever/lib";
print "\nExists ", -e $path;
print "\nDirectory ", -d $path;
print "\nReadable ", -r $path;
print "\nListing:\n";
print "\t$_\n" for glob "$path/*";