I'm just curious if when writing PL/Perl functions if I can have a use My::Lib; statement, or enable pragma's and features (e.g. 'use strict; use feature 'switch';).
Not when using PL/Perl. It restricts the use of require and use, so you cannot import modules. However, you can install PL/Perlu (for unrestricted mode) which allows you to load modules.
plperlu can be considered a security risk, however, as it also allows filesystem commands such as open.
For security purposes you cannot run a use/require statement within a function under plperl, but you can under plperlu.
IF you want to use modules in a secure way, you can add plperl.on_init = 'require "myperlinit.pl";' to the postgresql.conf file, then create a perl script called myperlinit.pl in the data directory which contains your uses. This will require a restart of the database server and these modules are available to all of your functions.
If you want strict mode turned on, you can plperl.use_strict = true will add it.
Note: this script is executed once per connection when the first perl function is called, and not when the connection is created.
Related
Is it possible to read, write, delete OS files with PL/pgSQL?
Can I run OS commands?
I've seen some examples that you can copy files like CSV but can you read/write/delete OS files? Can you execute OS commands?
No, that's not possible.
PL/pgSQL is a trusted language and as such does not allow access to server resources, let alone running OS commands.
Explanation of "trusted language"
The optional key word TRUSTED specifies that the language does not grant access to data that the user would not otherwise have. Trusted languages are designed for ordinary database users (those without superuser privilege) and allows them to safely create functions and procedures. Since PL functions are executed inside the database server, the TRUSTED flag should only be given for languages that do not allow access to database server internals or the file system
There are some some SQL functions available that enable roles with superuser privilege to read files on the server - but that is independent of PL/pgSQL.
If you do want to open up your database server for all kind of attacks, use a non-trusted language, like PL/Python or if you are really adventurous PL/sh
PostgreSQL has some functions to read files in the data directory: pg_read_file and pg_read_binary_file
The “adminpack” extension has a function to write files: pg_file_write
Perhaps you can abuse COPY ... TO PROGRAM to run code on the server.
But the smart thing to do is to write a function in PL/PerlU or PL/Python.
I'm going to build a extremly small script for dumping a Sybase database in perl. The problem is that Perl doesn't come with preinstalled Sybase-support. I don't have access to the servers root so I can't install any packages and I can't reach the perl-folder. The server is not configured for internet access so I have to deliver the packages "manually" thorugh FTP.
So, my question is if there are any easy ways of doing this. The only library I need is DBI::Sybase or Sybase standalone (maybe I haven't done my research enough and doesn't even need this much?) which means I would love to just be able to put the .pm file there, loading it through
use localModule
and then run my small script.
The solution has to work on both Red hat and Solaris if I understood my supervisor correctly.
Best regards
Since you are primarily concerned with dumping the database, and not data retrieval and manipulation, you could probably get by without having to use DBI::Sybase or other perl module that is not preinstalled.
Without more details, it's hard to be very specific, but here's the overview. Your perl script can execute some SQL scripts which can dump the databases.
You can either put the list of databases you wish to dump in a config file (or env file), or you can generate it dynamically by calling isql using the -b option to suppress headers, and nocount to suppress footers, and store the output in an array.
Once you have the list of databases, just loop them, running another isql command to dump each database.
Has anyone done or even no if its possible to use NLTK within a Postgres Python Stored Procedure or trigger
You can use pretty much any Python library in a PL/Python stored procedure or trigger.
See the PL/Python documentation.
Concepts
The crucial point to understand is that PL/Python is CPython (in PostgreSQL up to and including 9.3, anyway); it uses exactly the same interpreter that the normal standalone Python does, it just loads it as a library into the PostgreSQL backed. With a few limitations (outlined below), if it works with CPython it works with PL/Python.
If you have multiple Python interpreters installed on your system - versions, distributions, 32-bit vs 64-bit etc - you might need to make sure you're installing extensions and libraries into the right one when running distutils scripts, etc, but that's about it.
Since you can load any library available to the system Python there's no reason to think NLTK would be a problem unless you know it requires things like threading that aren't really recommended in a PostgreSQL backend. (Sure enough, I tried it and it "just worked", see below).
One possible concern is that the startup overhead of something like NLTK might be quite big, you probably want to preload PL/Python it in the postmaster and import the module in your setup code so it's ready when backends start. Understand that the postmaster is the parent process that all the other backends fork() from, so if the postmaster preloads something it's available to the backends with greatly reduced overheads. Test performance either way.
Security
Because you can load arbitrary C libraries via PL/Python and because the Python interpreter has no real security model, plpythonu is an "untrusted" language. Scripts have full and unrestricted access to the system as the postgres user and can fairly simply bypass access controls in PostgreSQL. For obvious security reasons this means that PL/Python functions and triggers may only be created by the superuser, though it's quite reasonable to GRANT normal users the ability to run carefully written functions that were installed by the superuser.
The upside is that you can do pretty much anything you can do in normal Python, keeping in mind that the Python interpreter's lifetime is that of the database connection (session). Threading isn't recommended, but most other things are fine.
PL/Python functions must be written with careful input sanitation, must set search_path when invoking the SPI to run queries, etc. This is discussed more in the manual.
Limitations
Long-running or potentially problematic things like DNS lookups, HTTP connections to remote systems, SMTP mail delivery, etc should generally be done from a helper script using LISTEN and NOTIFY rather than an in-backend job in order to preserve PostgreSQL's performance and avoid hampering VACUUM with lots of long transactions. You can do these things in the backend, it just isn't a great idea.
You should avoid creating threads within the PostgreSQL backend.
Don't attempt to load any Python library that'll load the libpq C library. This could cause all sorts of exciting problems with the backend. When talking to PostgreSQL from PL/Python use the SPI routines not a regular client library.
Don't do very long-running things in the backend, you'll cause vacuum problems.
Don't load anything that might load a different version of an already loaded native C library - say a different libcrypto, libssl, etc.
Don't write directly to files in the PostgreSQL data directory, ever.
PL/Python functions run as the postgres system user on the OS, so they don't have access to things like the user's home directory or files on the client side of the connection.
Test result
$ yum install python-nltk python-nltk
$ psql -U postgres regress
regress=# CREATE LANGUAGE plpythonu;
regress=# CREATE OR REPLACE FUNCTION nltk_word_tokenize(word text) RETURNS text[] AS $$
import nltk
return nltk.word_tokenize(word)
$$ LANGUAGE plpythonu;
regress=# SELECT nltk_word_tokenize('This is a test, it''s going to work fine');
nltk_word_tokenize
-----------------------------------------------
{This,is,a,test,",",it,'s,going,to,work,fine}
(1 row)
So, as I said: Try it. So long as the Python interpreter PostgreSQL is using for plpython has nltk's dependencies installed it will work fine.
Note
PL/Python is CPython, but I'd love to see a PyPy based alternative that can run untrusted code using PyPy's sandbox features.
I'm inheriting a file transfer environment with a collection of scripts written in Perl running on Linux. In a nutshell, these scripts just transfer files between sites using SFTP and SMB/CIFS protocols.
I've noticed that the scripts use Net::SFTP::Foreign for the SFTP connection handling.
Are there any advantages to using Perl modules to accomplish connections and transfers as opposed to just calling an external commands like lftp or smbclient?
You usually get better error detection and reporting using a module. I can't think of any good reason to change already working code to use an external command instead.
I am wondering if anyone has a Perl script (or can write one) to execute on multiple hosts at once via ssh, without any modules. I used to have something like this but cannot find it now and can't remember how it was done.
Are you looking for ClusterSSH? It's Perl, and it's used to run the same commands on several hosts at once, so this might be what you're looking for...
You might want to try using Expect.pm which is similar to #cnicutar's suggestion of calling an Expect script from Perl, except that you write it all in Perl. (This of course down not fit the requirement of "without any modules", but that requirement leads to bad Perl )
Learn how to install and use modules even when you don't have admin privileges on the host
Use Net::OpenSSH::Parallel
If you cannot use any additional modules from CPAN or any other source , all I can recommend you are:
1) Use Expect script and call it internally in your Perl script [Only if you are not willing to use Expect.pm module]
2) Use SSH keygen in all the servers to which you will connect to , so that password wont be necessary in the script. As mentioned by "cnicutar"
3) Use "remsh" if SSH usage is not that necessary.