I'm writing an sqlalchemy import/export script using the serializer dumps and loads.
The export works, but I have problems with the import, mainly due to foreign key issues.
I'm using sorted_tables to get the list of tables sorted based on dependencies and this makes sure I won't have cross tables foreign key issues but is there something similar to handle internal foreign keys (a table pointing to itself)?
I'm basically thinking about 2 possible solutions:
Find a way to sort the rows based on the dependencies
Disable all constraints -> insert the data -> enable all constraints
again
but I'm not sure how to do this properly...
a table example:
class Employee(Base):
__tablename__ = "t_employee"
id = sa.Column(Identifier, sa.Sequence('%s_id_seq' % __tablename__), primary_key=True, nullable=False)
first_name = sa.Column(sa.String(30))
last_name = sa.Column(sa.String(30))
manager_id = sa.Column(Identifier, sa.ForeignKey("t_employee.id", ondelete='SET NULL'))
and here is my script:
def export_db(tar_file):
print "Exporting Database. This may take some time. Please wait ..."
Base.metadata.create_all(engine)
tables = Base.metadata.tables
with tarfile.open(tar_file, "w:bz2") as tar:
for tbl in tables:
print "Exporting table %s ..." % tbl
table_dump = dumps(engine.execute(tables[tbl].select()).fetchall())
ti = tarfile.TarInfo(tbl)
ti.size = len(table_dump)
tar.addfile(ti, StringIO(table_dump))
print "Database exported! Exiting!"
exit(0)
def import_db(tar_file):
print "Importing to Database. This may take some time. Please wait ..."
print "Dropping all tables ..."
Base.metadata.drop_all(engine)
print "Creating all tables ..."
Base.metadata.create_all(engine)
tables = Base.metadata.sorted_tables
with tarfile.open(tar_file, "r:bz2") as tar:
for tbl in tables:
try:
entry = tar.getmember(tbl.name)
print "Importing table %s ..." % entry.name
fileobj = tar.extractfile(entry)
table_dump = loads(fileobj.read(), Base.metadata, db)
for data in table_dump:
db.execute(tbl.insert(), strip_unicode(dict(**data)))
except:
traceback.print_exc(file=sys.stdout)
exit(0)
db.commit()
print "Database imported! Exiting!"
exit(0)
For mass dumps, the standard technique is to disable constraints, do the import, then re-enable them. You'll also get much faster performance on the inserts.
Related
class datalog(display_clock):
def con_mysql(self):
cat = mysql.connector.connect(
host="localhost", user="subramanya", passwd="Sureshbabu#4155", database="CFM")
if (cat):
datacursor = cat.cursor()
todaydate = d
check_table = (
"SELECT count(*) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME=%s")
datacursor.execute(check_table, (todaydate,))
result = datacursor.fetchone()
if (result):
self.success_login()
else:
datacursor.execute(
"CREATE TABLE {today}(Sl_no INT NOT NULL AUTO_INCREMENT PRIMARY KEY,date DATE,Start_time TIME,End_time TIME,Item CHAR(255),Weight FLOAT, Amount INTEGER(10))".format(today=todaydate))
self.success_login()
else:
datacursor.Terminate
self.error_display.insert(0.0, "Connecting Database failed!!!")
I tried to check whether any table exists for today's date or not.
if not create the same.
no error occurred. But table not created for sysdate.
Welcome to stackoverflow!
I believe there is a small misconception here. You don't need to check if the table exists beforehand and create it afterward. Most of the current database technologies accept the condition IF NOT EXISTS on CREATE TABLE CLAUSE.
CREATE TABLE IF NOT EXISTS sales (
sale_id INT NOT NULL,
);
It means the table sales will be only created IF NOT EXISTS previously.
Also, I strongly recommend refactoring your code a wee bit. Take as a suggestion (please adapt accordingly your project needs):
from datetime import datetime
class Settings:
# please, avoid hard-coded credentials.
DB_HOST = "localhost"
DB_USER = "subramanya"
DB_PASSWD = "Sureshbabu#4155"
DB_SCHEMA = "CFM"
class datalog(display_clock):
def db_connect(self):
conn = mysql.connector.connect(
host=Settings.DB_HOST,
user=Settings.DB_USER,
passwd=Settings.DB_PASSWD,
database=Settings.DB_SCHEMA
)
if not conn:
raise Exception("Connecting Database failed!!!")
return conn
def ensure_table(self):
conn = self.db_connect()
conn.datacursor.execute("""
CREATE TABLE IF NOT EXISTS `{0}`(
Sl_no INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
date DATE,
Start_time TIME,
End_time TIME,
Item CHAR(255),
Weight FLOAT,
Amount INTEGER(10)
);
""".format(datetime.today().strftime('%Y%m%d')) # format 20200915
)
def run(self):
self.ensure_table()
self.success_login()
There are plenty of ways to write this code, but keep in mind that readability matters a lot.
I created a temporary table with sqlalchemy (with an underlying postgres database) that is going to be joined with a database table. However, in some cases when a value is empty '' then postgres throws the error:
failed to find conversion function from unknown to text
SqlAlchemy assembles everything to the following context
[SQL: 'WITH temp_table AS \n(SELECT %(param_1)s AS id, %(param_2)s AS email, %(param_3)s AS phone)\n SELECT campaigns_contact.id, campaigns_contact.email, campaigns_contact.phone \nFROM campaigns_contact JOIN temp_table ON temp_table.id = campaigns_contact.id AND temp_table.email = campaigns_contact.email AND temp_table.phone = campaigns_contact.phone'] [parameters: {'param_1': 83, 'param_2': '', 'param_3': '+1234567890'}]
I assemble the temporary table as follows
stmts = []
for row in import_data:
row_values = [literal(row[value]).label(value) for value in values]
stmts.append(select(row_values))
subquery = union_all(*stmts)
subquery = subquery.cte(name="temp_table")
The problem seems to be the part here
...%(param_2)s AS email...
which after replacing the param_2 results in
...'' AS email...
which will cause the error mentioned above.
One way to solve the issue is to perform a cast
...''::text AS email...
However, I don't know how to perform ::text cast with sqlalchemy!?
I'm trying to use python for stored procs in sql, I have tested my sql code and it works fine, but when I execute it via python, the values are not inserted into my table
Note: I don't have any errors when executing
My code below:
import psycopg2
con = psycopg2.connect(dbname='dbname'
, host='host'
, port='5439', user='username', password='password')
def executeScriptsfromFile(filename):
#Open and read the file as a single buffer
cur = con.cursor()
fd = open(filename,'r')
sqlFile = fd.read()
fd.close()
#all SQL commands(split on ';')
sqlCommands = filter(None,sqlFile.split(';'))
#Execute every command from the input file
for command in sqlCommands:
# This will skip and report errors
# For example, if the tables do not yet exist, this will skip over
# the DROP TABLE commands
try:
cur.execute(command)
con.commit()
except Exception as inst:
print("Command skipped:", inst)
cur.close()
executeScriptsfromFile('filepath.sql')
Insert comment in sql:
INSERT INTO schema.users
SELECT
UserId
,Country
,InstallDate
,LastConnectDate
FROM #Source;
Note: As I said the sql works perfectly fine when I tested it.
Recently converted from mysql to mysqli and things were working fine. Connect, select still work fine but now one query fails. Here is the basic code:
Connect to database ($con) - successful.
mysqli_query to select some data from table1 (fn1, ln1, yr1) - successful.
Table data goes to $fn, $ln, $yr after mysql_fetch_array - successful.
Use the data to form an insert:
$sql = "insert into table2 (fn2, ln2, yr2) values ('$fn', '$ln', '$yr')";
mysql-query($con, $sql) or die ("Insert failed: " . mysqli_error($con));
The query fail with the Insert failed message but no reason from mysql_error.
What have I missed?
I've try it. It work. Insert it's ok!
$link = new mysqli($dbhost, $dbuser, $dbpass, $dbname);
$qry = "insert into reputazione (iduser,star,votante,commento)
values(10,5,\"pippo\",\"commento\")";
mysqli_query($link, $qry);
if (mysqli_affected_rows($link) > 0) {
echo "ok!";
}
Okay, I solved the problem by coding all the mysqli commands in-line and not calling functions and passing the sql statements to them. I had functions for connecting to the DB, selecting from one table, inserting into another table and then deleting from the first table.
In Oracle OCI and OCCI there are API facilities to perform array inserts where you build up an array of values in the client and send this array along with a prepared statement to the server to insert thousands of entries into a table in a single shot resulting in huge performance improvements in some scenarios. Is there anything similar in PostgreSQL ?
I am using the stock PostgreSQL C API.
Some pseudo code to illustrate what i have in mind:
stmt = con->prepare("INSERT INTO mytable VALUES ($1, $2, $3)");
pg_c_api_array arr(stmt);
for triplet(a, b, c) in mylongarray:
pg_c_api_variant var = arr.add();
var.bind(1, a);
var.bind(2, b);
var.bind(3, c);
stmt->bindarray(arr);
stmt->exec()
PostgreSQL has similar functionality - statement COPY and COPY API - it is very fast
libpq documentation
char *data = "10\t20\40\n20\t30\t40";
pres = PQexec(pconn, "COPY mytable FROM stdin");
/* can be call repeatedly */
copy_result = PQputCopyData(pconn, data, sizeof(data));
if (copy_result != 1)
{
fprintf(stderr, "Copy to target table failed: %s\n",
PQerrorMessage(pconn));
EXIT;
}
if (PQputCopyEnd(pconn, NULL) == -1)
{
fprintf(stderr, "Copy to target table failed: %s\n",
PQerrorMessage(pconn));
EXIT;
}
pres = PQgetResult(pconn);
if (PQresultStatus(pres) != PGRES_COMMAND_OK)
{
fprintf(stderr, "Copy to target table failed:%s\n",
PQerrorMessage(pconn));
EXIT;
}
PQclear(pres);
As Pavel Stehule points out, there is the COPY command and, when using libpq in C, associated functions for transmitted the copy data. I haven't used these. I mostly program against PostgreSQL in Python, have have used similar functionality from psycopg2. It's extremely simple:
conn = psycopg2.connect(CONN_STR)
cursor = conn.cursor()
f = open('data.tsv')
cusor.copy_from(f, 'incoming')
f.close()
In fact I've often replaced open with a file-like wrapper object that performs some basic data cleaning first. It's pretty seamless.
I like this way of creating thousands of rows in a single command:
INSERT INTO mytable VALUES (UNNEST($1), UNNEST($2), UNNEST($3));
Bind an array of the values of column 1 to $1, an array of the values of column 2 to $2 etc.! Providing the values in columns may seem a bit strange at first when you are used to thinking in rows.
You need PostgreSQL >= 8.4 for UNNEST or your own function to convert arrays into sets.