I am trying to move my Python application from my PC (which is using SQLITE3) to Heroku (Postgres). I have declared Boolean columns in my tables and, in SQLITE I need to give these a value of 0 or 1. However, I believe in Postgres this needs to be True or False.
Is there a way that this conversion can be done easily such that I can continue to develop on SQLITE and deploy to Postgres without changing the code.
I guess the bottom line is amending the columns to Int and constraining their values to 0 or 1 but is there an easier answer?
# For example with SQLITE (adminuser is a column of Users, described as
# Boolean):-
admin_user = Users.query.filter_by(club = form.club.data, adminuser = 1)"
# For Postgres:-
admin_user = Users.query.filter_by(club = form.club.data, adminuser =
True)"
Related
I need to migrate tables from MS Access to Postgres. I'd like to use pyodbc to do this as it allows me to connect to the Access database using python and query the data.
The problem I have is I'm not exactly sure how to programmatically create a table with the same schema other than just creating a SQL statement using string formatting. pyodbc provides the ability to list all of the fields, field types and field lengths, so I can create a long SQL statement with all of the relevant information, however how can I do this for a bunch of tables? would I need to build SQL string statements for each table?
import pyodbc
access_conn_str = (r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)}; 'r'DBQ=C:\Users\bob\access_database.accdb;')
access_conn = pyodbc.connect(access_conn_str)
access_cursor = access_conn.cursor()
postgres_conn_str = ("DRIVER={PostgreSQL Unicode};""DATABASE=access_database;""UID=user;""PWD=password;""SERVER=localhost;""PORT=5433;")
postgres_conn = pyodbc.connect(postgres_conn_str)
postgres_cursor = postgres_conn.cursor()
table_ditc = {}
row_dict = {}
for row in access_cursor.columns(table='table1'):
row_dict[row.column_name] = [row.type_name, row.column_size]
table_ditc['table1'] = row_dict
for table, values in table_ditc.items():
print(f"Creating table for {table}")
access_cursor.execute(f'SELECT * FROM {table}')
result = access_cursor.fetchall()
postgres_cursor.execute(f'''CREATE TABLE {table} (Do I just put a bunch of string formatting in here?);''')
postgres_cursor.executemany(f'INSERT INTO {table} (Do I just put a bunch of string formatting) VALUES (string formatting?)', result)
postgres_conn.commit()
As you can see, with pyodbc I'm not exactly sure how to build the SQL statements. I know I could build a long string by hand, but if I were doing a bunch of different tables, with different fields etc. that would not be realistic. Is there a better, easier way to create the table and insert rows based off of the schema of the Access database?
I ultimately ended up using a combination of pyodbc and pywin32. pywin32 is "basically a very thin wrapper of python that allows us to interact with COM objects and automate Windows applications with python" (quoted from second link below).
I was able to programmatically interact with Access and export the tables directly to Postgres with DoCmd.TransferDatabase
https://learn.microsoft.com/en-us/office/vba/api/access.docmd.transferdatabase
https://pbpython.com/windows-com.html
import win32com.client
import pyodbc
import logging
from pathlib import Path
conn_str = (r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)}; 'rf'DBQ={access_database_location};')
conn = pyodbc.connect(conn_str)
cursor = conn.cursor()
a = win32com.client.Dispatch("Access.Application")
a.OpenCurrentDatabase(access_database_location)
table_list = []
for table_info in cursor.tables(tableType='TABLE'):
table_list.append(table_info.table_name)
for table in table_list:
logging.info(f"Exporting: {table}")
acExport = 1
acTable = 0
db_name = Path(access_database_location).stem.lower()
a.DoCmd.TransferDatabase(acExport, "ODBC Database", "ODBC;DRIVER={PostgreSQL Unicode};"f"DATABASE={db_name};"f"UID={pg_user};"f"PWD={pg_pwd};""SERVER=localhost;"f"PORT={pg_port};", acTable, f"{table}", f"{table.lower()}_export_from_access")
logging.info(f"Finished Export of Table: {table}")
logging.info("Creating empty table in EGDB based off of this")
This approach seems to be working for me. I like how the creation of the table/fields as well as insertion of data is all handled automatically (which was the original problem I was having with pyodbc).
If anyone has better approaches I'm open to suggestions.
I am loading data to Cassandra through YCSB using this command -
bin/ycsb load cassandra-10 -p hosts="132.67.105.254" -P workloads/workloada > workloada_res.txt
I just want what "sort of data" is loaded using above command. I mean a single character or a string.
Have you tried to run the same command with the basic switch instead of the cassandra-10 one?
From the documentation:
The basic parameter tells the Client to use the dummy BasicDB layer. [...]
If you used BasicDB, you would see the insert statements for the database [in the terminal]. If you used a real DB interface layer, the records would be loaded into the database.
You will then notice that YCSB generates rowkeys like user###, with ### < recordcount. The columns are named field0 to fieldN with N = fieldcount - 1 and the content of each cell is a random string of fieldlength characters.
recordcount is given in workloada and equals 1000.
fieldcount defaults to 10 and fieldlength to 100 but you can overwrite both in you workload file or using the -p switch.
]4[]5
I created one thing which access my database table name is sensordata from PostgreSQL. Now I have to send data to these table how. How can I do this?
I did the connection part of thingworx composer and PostgreSQL db on local setup.
I am trying to send sensor data from thingworx to PostgreSQL db but i am not able to sent it
You must to do two things:
1 Create a service to insert row in postgresql_conn thing;
Select 'SQL (Command)' as script type.
Put somthing like this into script area
INSERT INTO sensordata
(Temperature, Humidity, Vibration)
VALUES ([[TemperatureField]], [[HumidityField]], [[VibrationField]]);
TemperatureField, HumidityField, VibrationField are input fields of the service.
2 Create Subscriptions to sensordata thing.
As event set AnyDataChange;
Put something like this in the script area:
var params = {
TemperatureField: me.Temperature,
HumidityField: me.Humidity,
VibrationField: me.Vibration
};
var result = Things["postgresql_conn"].InsertRecord(params);
Now when data of sensordata change one row is add to the postgress table.
Sorry for my english
I have a PostgreSQL database. I had to extend an existing, big table with a few more columns.
Now I need to fill those columns. I tought I can create an .csv file (out of Excel/Calc) which contains the IDs / primary keys of existing rows - and the data for the new, empty fields. Is it possible to do so? If it is, how to?
I remember doing exactly this pretty easily using Microsoft SQL Management Server, but for PostgreSQL I am using PG Admin (but I am ofc willing to switch the tool if it'd be helpfull). I tried using the import function of PG Admin which uses the COPY function of PostgreSQL, but it seems like COPY isn't suitable as it can only create whole new rows.
Edit: I guess I could write a script which loads the csv and iterates over the rows, using UPDATE. But I don't want to reinvent the wheel.
Edit2: I've found this question here on SO which provides an answer by using a temp table. I guess I will use it - although it's more of a workaround than an actual solution.
PostgreSQL can import data directly from CSV files with COPY statements, this will however only work, as you stated, for new rows.
Instead of creating a CSV file you could just generate the necessary SQL UPDATE statements.
Suppose this would be the CSV file
PK;ExtraCol1;ExtraCol2
1;"foo",42
4;"bar",21
Then just produce the following
UPDATE my_table SET ExtraCol1 = 'foo', ExtraCol2 = 42 WHERE PK = 1;
UPDATE my_table SET ExtraCol1 = 'bar', ExtraCol2 = 21 WHERE PK = 4;
You seem to work under Windows, so I don't really know how to accomplish this there (probably with PowerShell), but under Unix you could generate the SQL from a CSV easily with tools like awk or sed. An editor with regular expression support would probably suffice too.
I'm trying to run jasper reports against a live and reporting database, but any reports run against the live database throw exceptions about not finding the right tables (although the default PUBLIC schema is found). It looks like the main DataSource connection isn't honoring the H2 connection settings which specify IGNORECASE=true, as the generated columns and tables are capitalized, by my queries are not.
DataSource.groovy dataSource:
dataSource {
hibernate {
cache.use_second_level_cache = false
cache.use_query_cache = false
}
dbCreate = "create-drop" // one of 'create', 'create-drop','update'
pooled = true
driverClassName = "org.h2.Driver"
username = "sa"
password = ""
url = "jdbc:h2:mem:testDb;MODE=PostgreSQL;IGNORECASE=TRUE;DATABASE_TO_UPPER=false"
jndiName = null
dialect = null
}
Datasources.groovy dataSource:
datasource(name: 'reporting') {
environments(['development', 'test'])
domainClasses([SomeClass])
readOnly(false)
driverClassName('org.h2.Driver')
url('jdbc:h2:mem:testReportingDb;MODE=PostgreSQL;IGNORECASE=TRUE;DATABASE_TO_UPPER=false')
username('sa')
password('')
dbCreate('create-drop')
logSql(false)
dialect(null)
pooled(true)
hibernate {
cache {
use_second_level_cache(false)
use_query_cache(false)
}
}
}
What fails:
JasperPrint print = JasperFillManager.fillReport(compiledReport, params,dataSource.getConnection())
While debugging, the only difference I've found is that the live dataSource, when injected or looked up with DatasourcesUtils.getDataSource(null), is a TransactionAwareDatasourceProxy, and DatasourcesUtils.getDataSource('reporting') is a BasicDataSource
What do I need to do for Jasper to operate on the active in-memory H2 database?
This failure is not reproducible against a real postgres database.
Probably you are opening a different database. Using the database URL jdbc:h2:mem:testDb will open an in-memory database within the same process and class loader.
Did you try already using a regular persistent database, using the database URL jdbc:h2:~/testDb?
To use open an in-memory database that is running in a different process or class loader, you need to use the server mode. That means, you need to start a server where the database is running, and connect to it using jdbc:h2:tcp://localhost/mem:testDb.
See also the database URL overview.
H2 doesn't currently support case-insensitive identifiers (table names, column names). I know other databases support it, but currently H2 uses regular java.util.HashMap<String, ..> for metadata, and that's case sensitive (whether or not IGNORECASE is used).
In this case, the identifier names are case-sensitive. I tried with the database URL jdbc:h2:mem:testReportingDb;MODE=PostgreSQL;IGNORECASE=TRUE;DATABASE_TO_UPPER=false using the H2 Console:
DROP TABLE IF EXISTS UPPER;
DROP TABLE IF EXISTS lower;
CREATE TABLE UPPER(NAME VARCHAR(255));
CREATE TABLE lower(name VARCHAR(255));
-- ok:
SELECT * FROM UPPER;
SELECT * FROM lower;
-- fail (table not found):
SELECT * FROM upper;
SELECT * FROM LOWER;
So, the question is: when creating the tables, were they created with uppercase identifiers or a different database URL? Is it possible to change that? If not: is it possible to use a different database URL?
Just don't run reports against in-memory datasources, and this won't be an issue.