I'm looking for a virtual filesystem layer in Perl. Something that would provide a general abstraction for basic filesystem routines like ls, mkdir and so on, regardless how the actual filesystem is implemented.
I'd like an interface like this:
# create a directory "/some/path/tmp" in my current filesystem
my $plainfs = Module::new->(type => 'local', root=>'/some/path);
$plainfs->mdkir("/tmp");
# create "tmp" dir on a remote filesystem
my $sshfs = Module::new->(type=>'ssh', root=>'user:password#example.com:~/pub')
$sshfs->mdkir("/tmp");
I found the VFS package on MetaCPAN, unfortunately there are only empty, unimplemented modules.
Is something already implemented? Right now, I'm looking for only “local” filesystems and ftp or ssh—I don't need a database “filesystem” or any other exotic “filesystem” like CVS or so. Searching 20k MetaCPAN modules is painful without any tagging system or alike…
Perhaps File::System is what you're looking for. It provides basic functionalities found in common operating systems for managing a virtual file system (not necessarily comprised only of files and directories).
Most of the functionalities are presented as method of the File::System::Object package.
what about some FUSE implementation? ( file system in userspace ) ? I would guess there is at least one pseudo-filesystem implemented in perl based on that. After all, it should be quite easy to implement, basically it's no more than some set of operations like mount, ls, df, stat and so on. I was once through autofs sources in C, looked pretty straightforward. You might want to see http://code.google.com/p/mogilefs/ as well.
Don't be too stuck up on the module approach. All you need is some utility that mounts SSH/FTP filesystem as a local filesystem and then you will simply use standard commands like cd, mkdir and so on. The reason why you don't see any modules for this is that this approach is generally preferred.
Look at http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FileSystems
You will simply use FUSE to mount any of those file systems and that is it. Here are some links to look at, but most of those can be got as packages in most distributions too.
http://sourceforge.net/projects/lufs/
http://lftpfs.sourceforge.net
Here is module to simply mount FUSE file systems within perl:
http://search.cpan.org/~dpavlin/Fuse/Fuse.pm
There are a LOT of File::* modules which handle different parts of cross-platform filesystem management.
For example:
use File::Spec::Functions qw(catfile);
Will let you get my $filename = catfile $root, $path, "$filename.$ext"; or my $new_directory = catfile $path, "new_sub_directory"; and be sure to use the correct separators, e.g. / or \, et cetera.
Another thing you seem to want can be had with:
use File::Path qw(make_path);
which is pretty handy, and can be called like make_path($new_directory, { mode => 0755 });
I'm not really sure if File::System actually handles remote systems the way you want.
A couple different ways occur to me to handle that, but I think Net::SSH::Expect is what I've used in the past, and isn't too bad, although you'd probably have an easier time if you could somehow mount the remote filesystem locally, do what you have to do, then unmount it.
Related
Problem outline:
need allow upload ZIP files (and tgz and more compressed directory trees) via web-from
the zip files should be extracted for their content handling
planning to use Archive::Extract for the extracting
here are things like ZIP BOMBS and like...
From the manual
Archive::Extract can use either pure perl modules or command line
programs under the hood. Some of the pure perl modules (like
Archive::Tar and Compress::unLZMA) take the entire contents of the
archive into memory, which may not be feasible on your system.
Consider setting the global variable $Archive::Extract::PREFER_BIN to
1 , which will prefer the use of command line programs and won't
consume so much memory.
The questions are:
When I set the $Archive::Extract::PREFER_BIN = 1 - i'm enough protected againist ZIP-BOMB like things?
$Archive::Extract::PREFER_BIN protect me againist much memory usage - but, the standard unzip, tar -z unrar binaries are safe againist zip bomb like attacks?
If not - how to handle safely uploaded compressed directory tree? (so here is not only one file inside the e.g zip archive).
$Archive::Extract::PREFER_BIN = 1 doesn't protect you against zip bombs, you are passing the problem to the binary unzip tool of your system.
This SO question may helps you. I like the idea of running a second process with ulimit.
What module would you recommend to get a disk name on Linux? I've done some search on CPAN but all modules I've found are too old. In Bash I can use something like:
disk_name=$(df |grep -w '/'|awk '{print $1}'|cut -d/ -f3)
echo $disk_name
sda6
Please help me to understand how to do same in Perl.
Thanks.
The "proper" way to list mounted disks on Linux is through the getmntent() system call, which can be accessed from Perl using the Quota module:
use Quota;
Quota::setmntent();
while (my ($dev, $path, $type, $opts) = Quota::getmntent()) {
print "The root device is $dev.\n" if $path eq "/";
}
Quota::endmntent();
As a bonus, using the Quota module to list device mount points should be fairly portable to other Unixish systems, which parsing various system files or the output of df may not be. Unfortunately, this seemingly basic module is not included in the standard Perl distribution, so you have to get it from CPAN (or from your distro's package repository — for example, Debian / Ubuntu have the libquota-perl package).
Ps. Simply splitting the device name on / and taking the third element (as your cut command does) is not a safe way to turn, say, /dev/sdb1 into sdb1. Some issues with it are that:
Not all block devices have to live under /dev — it's really just a convention.
Even if the device file is under /dev, it might be in a subdirectory of it. For example, my root filesystem is on the device /dev/disk/by-uuid/627f8512-f037-4c6c-9892-6130090c0e0f.
Sometimes, the device name might not even be an actual filesystem path: for example, virtual or in-memory filesystems such as tmpfs are often mounted with the device name none, but it's possible to use any device name with them.
If you do want to get rid of the /dev/ part, I'd suggest a conservative approach using a regexp, for example like this:
if ($dev =~ m(^/dev/(.*)$)s) {
print "The directory $path is mounted from device $1 under /dev.\n";
} else {
print "The directory $path is not mounted from a device under /dev.\n"
}
What you're describing is not the disk name but the device name of the block device representing the partition mounted at root (/). On a regular computer it would normally be something like /dev/sdXN or /dev/hdXN with X being the disk number (primary hard drive is usually A, secondary is B, etc.) and N is the partition number on that device.
Provided you're always running on a unix system, you can try reading /etc/mtab file, which lists all mounted partitions, or the special file /proc/mounts, which pretty much does the same. You'll need to parse it afterwards to find the one you need and get the device name from it.
Alternatively, you can just run df as a process and get its input into perl, something like
open(DF, "df|");
#mount_points = <DF>;
close(DF);
and then iterate over the data to find what you need. I'm not aware of any modules of the top of my head that would do the job for you, but the code seems pretty simple to me anyway.
P.S. Note that Max OS X, while being a derivative of BSD, doesn't have the same file structure and therefore this approach wouldn't work. On Mac OS X, you can read file /etc/fstab.hd, which contains similar info but in a slightly different format.
One way to do just what you are doing in the question
df / | perl -ne 'm"^/\w+/(\w+)";print "$1\n" if defined $1;'
but using a CPAN library to do it is probably better.
Scenario:
I have a process where many files are being copied (scp'd) to a DestinationServer by Host1, Host2, Host3, Host4 for example. Going to the same common directory: DestinationServer:/home/target. All the files are unique so no files will be overwritten. Host1-Host4 will have a cronjob that will launch their scp script to DestinationServer. The caveat is the Hosts are in different time zones, locations. So, they will finish at different times.
Need:
Since the files are being scp'd to Destination:/home/target, what is the best way to programmatically check when those scp's from the other Hosts are done??
Options:
My options are to programmatically do this either in perl or shell if possible.
What do I look for, what unix commands or perl modules could I use to help determine when the processes would finish? Any ideas, examples would be great! Thanks.
Use a Maildir kind of approach: copy all files to a temporary directory, then after the transfer is complete have the originating host perform a rename into the target directory via ssh. That way when a file appears in the target directory, you know that it is complete.
I suggest this because if you just scp files into the target directory and monitor the directory in whatever way, you cannot distinguish a complete transfer from an interrupted scp command or a network failure.
SGI::FAM, Sys::Gamin
Similar but alternative way to Jouni is to use semaphore files. Before scp-ing files originating host puts up semaphore-file and when finished, remove it. So you know, it's time.
Okay this is and isn't programming related I guess...
I've got a whole bunch of little useful console utilities scattered across a suite of projects that I wrote and I want to dump them all to a single directory to make using them simpler. The only issue is that I have them all compiled in both Debug and Release mode.
Given that I only want the release mode versions in my utilities directory, what switch would allow me to specify that I want all executables from my tree structure but only from within Release folders:
Example:
Projects\
Project1\
Bin\
Debug\
Project1.exe
Release\
Project1.exe
Project2\
etc etc...
To
Utilities\
Project1.exe
Project2.exe
Project3.exe
Project4.exe
...
etc etc...
I figured this would be a cinch with XCopy - but it doesn't seem to allow me to exclude the Debug directories - or rather - only include items in my Release directories.
Any ideas?
You can restrict it to only release executables with the following. However, I do not believe the other requirement of flattening is possible using xcopy alone. To do the restriction:
First create a file such as exclude.txt and put this inside:
\Debug\
Then use the following command:
xcopy /e /EXCLUDE:exclude.txt *.exe C:\target
You can, however, accomplish what you want using xxcopy (free for non-commercial use). Read technical bulletin #16 for an explanation of the flattening features.
If the claim in that technical bulletin is correct, then it confirms that flattening cannot be accomplished with xcopy alone.
The following command will do exactly what you want using xxcopy:
xxcopy /sgfo /X:*\Debug\* .\Projects\*.exe .\Utilities
I recommend reading the technical bulletin, however, as it gives more sophisticated options for the flattening. I chose one of the most basic above.
Sorry, I haven't tried it yet, but shouldn't you be using:
xcopy release*.exe d:\destination /s
I am currently on my Mac so, I cant really check to be for sure.
This might not help you with assembling them all in one place now, but going forward have you considered adding a post-build event to the projects in Visual Studio (I'm assuming you are using it based on the directory names)
xcopy /Y /I /E "$(TargetDir)\$(TargetFileName)" "c:\somedirectory\$(TargetFileName)"
Ok, this is probably not going to work for you since you seem to be on a windows machine.
Here goes anyway, for the logic.
# From the base directory
mkdir Utilities
find . -type f | grep -w Release > utils.txt
for f in $(<utils.txt); do cp $f Utilities/; done
You can combine the find and cp lines into one, I split them for readability.
To do this on a windows machine you'll need Cygwin or some such Unix Utilities handy.
Maybe there are tools in the Windows shell to do this...
This may help get you started:
C:\>for %i in (*) do dir "%~dpi\*.exe"
Used in the dir command as a modifier to i, ~dp uses the drive and path of everything found in (*). If I run the above in a folder that has several subfolders containing executables, I get a dir list of all of the executables in each folder.
You should be able to modify that to add '\bin\release\' following the ~dpi portion and change dir to xcopy. A little experimentation should make it pretty easy.
To use the for statement above in a batch file, change '%' to '%%' in both places.
My hosted scripts have been moved and no longer work.
The specified CGI application
misbehaved by not returning a complete
set of HTTP headers.
I notice that someone at my host company has modified my scripts so that where I used to have
use lib 'd:/myorig/LIB';
I now have
use lib '//newhost/LIB';
Should this work?
I tried 1800 INFORMATION's suggestion and ran the minimal script of
#!perl -w
use lib '//whatever/lib';
print "success";
...which gave the same result.
Update: ysth's suggestion of FatalsToBrowser did indeed reveal more information. It looks like the path (added by someone from the hosting company) might be wrong.
Update2: The hosting company now says that these scripts, unchanged from the previous host mind, are throwing lots of syntax errors. "Since we cannot debug your scripts for you we suggest you contact the original programmer and ask them for help". <grinds teeth>
Partial Resolution: The hosting company finally realised they hadn't set permissions correctly. They still aren't right, and (aargh) they don't allow site owners to set folder permissionsn, not even on folders within their own sites.
I don't know if it should work or not, but my intuition is that it would be okay. However, the two use lib lines you posted are not equivalent.
# go to the 'd' drive and use the 'myorigLIB' directory on that drive
use lib 'd:/myorigLIB';
# go to the 'newhostLIB' server - no path is specified - this looks invalid to me
use lib '//newhostLIB';
Perhaps you need to specify the path to the share on the server? Also, you might need to look at permissions? Maybe the user the CGI is running as cannot access that network path?
Also, you could write a simple (non CGI) program to test your theory and just run it:
#!perl -w
use lib '//whatever/lib';
print "success";
Then just run that on the server if you can and see what happens.
No the path is incomplete it needs both a server name and a complete path. It is a bad practice as well because it requires that two machines be monitored rather than one for your application to function.
The specified CGI application misbehaved by not returning a complete set of HTTP headers.
That's a non-error. If you are lucky, your hosting company will make an error log available to you that will show the actual error that perl is dying with. If not,
consider using
use CGI::Carp "fatalsToBrowser";
for testing. (If you are paranoid (which is not a bad thing to be), you will refrain from leaving that enabled once you are done testing, since errors can commonly provide information about your code or even your database that may help a black hat exploit security holes.)
I know I ran into trouble trying to use mapped drives and unc paths from apache because the apache user was not allowed to use network drives. That was difficult to figure out -- but it's possible to do it. That may be a related problem.
#!perl -w
print "HTTP/1.0 200 OK\nContent-Type: text/plain\n\n";
my $path = "//whatever/lib";
print "\nExists ", -e $path;
print "\nDirectory ", -d $path;
print "\nReadable ", -r $path;
print "\nListing:\n";
print "\t$_\n" for glob "$path/*";