How to install and launch PostgreSQL as part of a stand-alone Python-based application? - postgresql

I have created a Python application that utilizes a PostgreSQL database and now I'd like to package it as a stand-alone application. I have worked out how to package everything as a single executable using PyInstaller however it will fail upon launch when attempting to establish a connection to the database if it doesn't exist. To address this my assumption is that I need to also include a mechanism for installing and launching a PostgreSQL database instance if it's not present.
I'm trying to provide a single installation process that creates/configures/launches the database (if not present) and installs the application's executable. I want to create installers for Linux, Windows, and MacOS.
How do I best go about this? I am hoping to find documentation for a solution to this use case -- my Google-Fu appears to be weak and/or the solution is obscure.

Handling external software is not the purpose of Pyinstaller. So you need to handle its dependencies by yourself. So about your question, you need to first find a portable version of PostgreSQL, then in your script, run and configure it manually. For example, there is a good portable version in here.
Just extract the contents of PGRE portable to a directory like PostgreSQL next to your python script (app.py).
Then you can use the below code to launch Database:
import subprocess
import os
import sys
PGRE_EXE = "PostgreSQL/PostgreSQLPortable.exe"
def resource_path(relative_path):
if hasattr(sys, '_MEIPASS'):
return os.path.join(sys._MEIPASS, relative_path)
return os.path.join(os.path.abspath("."), relative_path)
def start_pgre():
pgre_path = os.path.join(resource_path("."), PGRE_EXE)
print(pgre_path)
p = subprocess.Popen(
[pgre_path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
print(stdout)
print(stderr)
if __name__ == "__main__":
start_pgre()
Then add the whole PostgreSQL as external data with --add-data:
pyinstaller -F --add-data "./PostgreSQL:PostgreSQL" app.py

Related

Configure Stata 16 to work with Postgres WRDS Database on a Apple Silicon Mac

I am currently trying to run some Stata code related to the WRDS database for my Ph.D. It is a postgres database requiring some configuration via ODBC. While isql connection via Terminal works, the command odbc list in Stata returns the following error:
The ODBC file libodbc.dylib could not be found on this system. Setting the unix LD_LIBRARY_PATH environment variable may correct this error.
I mainly followed the tutorial on https://gist.github.com/JonathanWillitts/7b5a519bd40dd730b98ce1ad75e859e8, trying to adapt it to the Postgres requirements for the Database via homebrew and psqlodbc. I assume the error comes from the variable export via Terminal that I use:
export LD_LIBRARY_PATH=/opt/homebrew/lib/ && /Applications/Stata/StataSE.app/Contents/MacOS/StataSE &
I have had success downloading WRDS data into R using the RPostgres package.
If you do not figure out how to download WRDS data directly into Stata, a possible workaround is to:
Use R and the RPostgres package, and download the data into R
Then export the data from R into Stata, see:
https://www.statmethods.net/input/exportingdata.html
https://datascienceplus.com/exporting-data-from-r/
I am happy to share R code for obtaining WRDS data. Let me know if this interest you and I can share my R code here.

How can I import a large (multi-GB) sql file into postgres using dotnet core?

My database needs to mirror another, to which I have no access except for a nightly export of the sql file. I could script the import using psql.exe, but would prefer everything to be under the control of the dotnet core application.
I can't use the COPY command, because the file contains ALL the sql to set up the schemas and tables, as well as all the sql commands to insert/alter/copy the data.
I can't use \i because that is a postgresql console command, not something I can run through npgsql.
Is what I'm trying to do possible? Is it inherently a bad idea, and should I run a script to import it outside of the dotnet application? Should the dotnet application run and talk to the psql.exe program directly?
You could theoretically parse the SQL file in .NET and send it to PostgreSQL, but this is a very non-trivial thing to do, since you'd need to understand where statements end (identify semicolons) in order to send chunks.
You could, of course, send the entire file as a single chunk, but if it's huge, that may be a bad idea.
At the end of the day, I don't think there's any particular issue with launching psql.exe as an external process from .NET, and properly inspecting its exit code for error handling. Any reason you think you need to avoid that?

How to connect to PostgreSQL in Erlang using epgsql driver?

I would like to access a PostgreSQL database in Erlang. I downloaded the epgsql driver, it was a few directories and files, but I don't understand how to use it.
How can I write an Erlang program and use the epgsql driver to access a PostgreSQL database?
I made a new folder and copied all files from src/ in the driver and pgsql.hrl to my new folder. Then I created a simple test program:
-module(dbtest).
-export([dbquery/0]).
dbquery() ->
{ok,C} = pgsql:connect("localhost", "postgres", "mypassword",
[{database, "mydatabase"}]),
{ok, Cols, Rows} = pgsql:equery(C, "select * from mytable").
Then I started erl and compiled the modules with c(pgsql). and c(dbtest).
But then when I exeute dbtest:dbquery(). I get this error:
** exception error: undefined function pgsql:connect/4
in function dbtest:dbquery/0
Any suggestions on how I can connect to a PostgreSQL database using Erlang?
Rebar is a good tool to use but, I'm finding it's good to know how your project should be structured so you can tell what to do when things go wrong. Try organizing your project like this:
/deps/epqsql/
/src/dbtest.erl
/ebin
Then cd into deps/epqsql and run make to build the library.
Your dbtest.erl file should also explicitly reference the library, add this near the top:
-include_lib("deps/epgsql/include/pgsql.hrl").
You'll probably want to use a Makefile (or rebar) that compiles your code when you make changes but, try this to compile things right now: erlc -I deps/epqsql/ebin -o ebin src/dbtest.erl.
When testing, make sure your load paths are set correctly, try: erl -pz deps/epqsql/ebin/ ebin/. When the erl console loads up, try dbtest:dbquery(). and see what happens!
I don't have Postgresql setup on my machine but, I was able to get more reasonable looking errors with this setup.
I recommend to use ejabber pgsql driver https://svn.process-one.net/ejabberd-modules/pgsql/trunk/

Portable PostgreSQL for development off a usb drive

In order to take some development work home I have to be able to run a PostgreSQL database.
I don't want to install anything on the machine at home. Everything should run off the usb drive.
What development tools do you carry on your USB drive?
That question covers pretty much everything else, but I have yet to find a guide to getting postgresql portable. It doesn't seem easy if it's even possible.
So how do I get PostgreSQL portable? Is it even possible?
EDIT:
PostgreSQL Portable works. It's very slow on the usb-drive I have, but it works. I can't recommend doing constant development with it but for what I need it's great.
Perhaps if I pick up a full speed external drive I'll try out virtualization. Given the poor performance of just running the database off this drive, a full virtual OS running off of it would be unusable.
Here's how you can do this on your own:
http://www.postgresonline.com/journal/archives/172-Starting-PostgreSQL-in-windows-without-install.html
An alternate route would be to use something like VirtualBox and just install your development environment (database, whatever) on there.
There are 2 projects to try in 2014: http://sourceforge.net/projects/pgsqlportable/ and http://sourceforge.net/projects/postgresqlportable/?source=recommended.
I can't vouch for the second, but I'm using the first and it works right out of the box.
After unzipping using 7-zip (http://www.7-zip.org/download.html):
1) Run "start service without usuario.bat" ( english translation )
2) Then run "pgadmin3.bat"
The only minimal problem for me was that its in spanish. I've been able to change the language to english by following Change language of system and error messages in PostgreSQL. Using google translate the instructions are:
Description
This is a zip to automatically run postgresql 9.1.0.1 for windows. This version already has pgagent and pldebugger. To run must: 1) unzip
the zip 2) run the "start service without usuario.bat" found in the
pgsql directory within the folder you just unzipped. 3) Optional. If
you want to run the agent works postgresql (pgagent) should only run
the "start pgagent.bat" found in the pgsql directory inside the folder
you just unzipped. 4) Optional. To manage and / or develop the bd you
can run the pgadmin3.bat 5 files) Optional. To stop and / or restart
the server correctly use file "service without stopping usuario.bat"
usuario.bat or restart service without depending on the case.
Now option for Linux (file. Tar.gz). Postgresql portable Linux 9.2
Please use the tickets for your answer bugs.
Username: postgres Password: 123
Just a Note : on a new computer , to get pgadminIII working you may need to add a db. The settings are in attached screenshot.
Hope it helps.
I agree with virtualization solution, but maybe you can find useful this link from portable freeware collection, I have used this locally, not from usb though
1.download and extract : zip version
2.inside pgsql folder create data folder(put any name,I used 'data')
3.initalize data folder: c:\pgsql\bin\initdb.exe -D c:\pgsql\data -U postgres -W -E UTF8 -A scram-sha-256
4.to start/stop see next cmd code that I use (press any key inside it to stop)
c:\pgsql\bin\pg_ctl.exe -D c:\pgsql\data -l logfile start
pause
c:\pgsql\bin\pg_ctl.exe -D c:\pgsql\data stop
more info

How do I copy data from a remote system without using ssh or FTP Perl modules?

I have to write a Perl script to automatically copy data from remote server to my local system. The directory structure on remote systems is:
../log/D1/<date>.tar.gz
../log/D2/<date>.gz
../log/D3/<date>.tar.gz
../log/D4/<date>
and same on other server. I want to copy the data on local system in below format.
../log/S1/D1/<date>.tar.gz
../log/S1/D2/<date>.gz
../log/S1/D3/<date>.tar.gz
../log/S1/D4/<date>
and same for other servers i.e. S2, S3, etc
Also, no ssh supported Perl modules are available on remote server as well on local server and I dont have permission to install any Perl modules. The only good thing is that the connectivity is through password-less ssh keys.
Can anyone please suggest me any Perl code to get this done?
I believe you can access to shell command from perl.
So you can do this:
$cmd = "/usr/bin/scp remotefile localfile";
system $cmd;
NOTE: scp is secure-copy -- a buddy of ssh.
This does not require ssh-perl module but it require ssh support on both (which I have).
Hope this helps.
I started to suggest the scp command line program, but it seems that there's a CPAN module for that (no surprise). Check out Net::SCP.
By using scp on your client (where you can install new Perl modules) you can copy files without having to install any new software on the remote system. It just needs to have the ssh server running - which you've said it does.
I'd say stop trying to make life difficult for yourself and get the system to support the features you require.
Trying to develop for such a limited/ locked down platform is not going to be cost-effective in the long run - you'll develop stuff more slowly and it will have more bugs.
A little developer time is way more expensive than a decent hosted VM / hardware box.
Get a proper host, it will definitely save money (talk to your manager about this).
From your query above I understand that you don't have much permissions to install perl modules or do any changes which require administrative privileges. I love perl but to automate things like this you should use bash instead of perl. Below is the sample code I am using with password less ssh keys.
#!/bin/bash
DATE=`date`
BASEDIR="/basedir"
cd $BASEDIR
for HOST in S1 S2 S3
do
scp -q $HOST:$BASEDIR/D1/$DATE.tar.gz $HOST/D1/
echo "Data copy from $HOST done"
done
exit 0
You can use different date formats like date +%Y%m%d for current date in format YYYYMMDD. Also you can use this link to learn different date formats.
Hope this helps.
You may not be able to install anything in system-wide lib directories, but there is nothing preventing you from installing modules in a location to which you have write-access. See How do I keep my own module/library directory?
This creates no more of a security issue than allowing you to write scripts on this system in the first place.
So, go forth and install Net::SCP.
It sounds like you want rsync. You shouldn't have to do any programming at all.