I am using a Jupyter Notebook on IBM Data Science Experience. Is it possible to enable SQL Magics/IPython-sql? How can I install it?
I want to connect to dashDB/DB2 and run SQL statements.
Yes, it is possible to use the IPython-sql (SQL Magics) module in the Jupyter Notebooks. The trick is to install it into the user space. Run the following in a code cell:
!pip install --user ipython-sql
If you want to connect to DB2 or dashDB, then you would need to install the related database drivers. Because the SQL Magics depend on SQLAlchemy, use these commands (same cell as the command above works):
!pip install --user ibm_db
!pip install --user ibm_db_sa
Once everything is installed, you need to load the SQL Magics extension:
%load_ext sql
I took the instructions on installing SQL Magics in the Data Science Experience from this blog post. It also has an example on how to connect to the database.
There is also another way to run SQLs against dashDB from IBM Data Science Experience. It has already pre-deployed the ibmdbpy and ibmdbR libraries for Python and R notebooks, respectively. So you don't have to set up anything before using it.
Here is a sample for Python:
https://apsportal.ibm.com/analytics/notebooks/5a59ba9b-02b2-40e4-b955-9727cb68c88b/view?access_token=09240b783432f1a62004bcc82b48a7aed07afc401e2f94a77c7e087b74d8c053
And here is one for R:
https://apsportal.ibm.com/analytics/notebooks/4ff39dad-f497-40c6-941c-43162c347819/view?access_token=9b2ae23b8ec4d8223a7f88950db66a72c736b269ef6cf1d658bb1fcd49c78f35
Related
I have an auditing and slowly changing dimensions (SCD) use case that requires the installation of the PostgreSQL temporal_tables and cyanaudit extensions.
However my PostgreSQL server is on Windows and after searching high and low, I still haven't found a way to install these extensions in PostgreSQL for Windows.
Has anyone succeeded in installing these extensions in the Windows environment and can you please share your methodology?
Thanks a million,
JDaniel
I have not tried to install these extensions on Windows but I suggest you have a look on the related source code repositories.
temporal_tables is stored on GitHub: if you search in open and closed issues about Windows installation, I have found: https://github.com/arkhipov/temporal_tables/issues/19.
cyanaudit is stored on bitbucket and I have not found anything related to Windows.
However installation should be easier because it is mainly SQL and PL/PGSQL code: you need to be able to use tar and perl because the installer is using Perl.
I have a data ware house running on PostgreSQL and I would like to check what all are the missing indices are in my database.
I tried to install the extension pg_qualstats but it is giving the below error.
root#Ubuntu-1604-xenial-64-minimal ~ # sudo pip install pg_qualstats
Collecting pg_qualstats
Could not find a version that satisfies the requirement pg_qualstats (from versions: )
No matching distribution found for pg_qualstats
You are using pip version 18.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root#Ubuntu-1604-xenial-64-minimal ~ #
So how can i install this extension? I tried the CREATE EXTENSION pg_qualstats but it was giving error.
Also is there any other database maintenance need to be done for the database regularly?
What all parameter I have to check?
Can we automate the maintenance activity?
I was a SQL server DB admin and it was much easier to find out the missing index, understand the execution plan, DB maintenance but I find it hard, when it comes to PostgreSQL.
So any guidance will be of great help.
I am only answering the immediate question regarding the installation of pg_qualstats - the rest of the questions are way too broad for a platform like stackoverflow (or dba.stackexchange).
Many interesting extensions are provided as source code in Postgres (that's one of the reasons why it's highly recommended to run Postgres on Linux, because compiling the extensions is way easier in Linux than it is on Windows, and may extensions are only developed for Linux).
pg_qualstats is no different.
It is provided together with PoWA and the installation of the extension is documented as part of their installation guide
In a nutshell:
Download the source:
wget https://github.com/powa-team/pg_qualstats/archive/1.0.7.tar.gz -O pg_qualstats-1.0.7.tar.gz
tar zxvf pg_qualstats-1.0.7.tar.gz
cd pg_qualstats-1.0.7
the compile it:
make
make install
Register the shared libraries by editing postgresql.conf and after restarting Postgres the extension can be created using CREATE EXTENSION pg_qualstats;
I'm trying to analyze my datasets on DB2 on Cloud in the Jupyter notebook created in Watson Studio. When using the "%sql" magic for connecting DB2 doesn't work naturally, showing no such module. According to an IBM tutorial, it is required to run the "%run db2re.ipynb" command in a Jupyter cell before connecting DB2. But when I run this cell nothing happens and the "%sql" magic still not available. Any advise is appreciated.
In general, there are two ways of accessing libraries in Watson Studio:
- Install or import a library, then reference it. Note that you need to specify the --user option.
- First save your own scripts, then import them.
There are also the built-in line and cell magics.
With that, I think I got it to work the following way:
1st cell, download db2re.ipynb to your environment:
%%sh
wget https://raw.githubusercontent.com/DB2-Samples/Db2re/master/db2re.ipynb
2nd cell, install necessary library:
!pip install --user qgrid
3rd cell, run the db2re.ipynb notebook extension:
%run db2re.ipynb
Thereafter, I was able to run a %sqlcommand.
I have python2 env. on windows 10 with jupyter notebook.
after following instructions in this tutorial I managed to install spark on windows 10:
https://medium.com/#GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c
but when I try to run cell magic for SQL I get the following error :
ERROR:root:Cell magic %%sql not found.
when I used %lsmagic I could not find sql cell magic among them.
also I noticed there was no option for pyspark kernel when starting new notebook in Jupyter.
Are you trying to use SQL or Spark-SQL? I've used iPython-SQL which was great, and there's also SparkMagic which sounds like what you're looking for. Try installing SparkMagic, which does use %%sql magic.
This one may be for the experts!
So here is the problem:
I have a flask app that gets its data from Postgresql and it runs fine in my normal environment. I tried to deploy it locally using virtualenv and after pip installing all the requirements the only one that gave me trouble is psycopg2 which appears to be dependency of Postgresql.
I then used this amazing article to help me install it, by putting export PATH=/Library/PostgreSQL/9.3/bin:$PATH to the .bash_profile file.
But now I get this error:
Library not loaded: libssl.1.0.0.dylib ...Image not found
What is going on?