I'm experimenting with Tabpy in Tableau and writing data back to a database via a users selection in a dashboard.
My data connections work fine. However, I'm having trouble passing a variable back into the SQL query and getting an error saying the "variable name" doesn't exist in the table that I'm trying to update. Here is my code.
The error states that "dlr_status" does not exist on the table. This is the variable I'm trying to pass back to the query. The database is a Postgres database. Any help is greatly appreciated. I've been researching this for several days and can't find anything.
SCRIPT_STR("
import psycopg2
import numpy as np
from datetime import date
con = psycopg2.connect(
dbname='edw',
host='serverinfo',
port='5439',
user='username',
password='userpassword')
dlr_no_change = _arg1[0]
dlr_status = _arg2[0]
update_trigger = _arg3[0]
sql = '''update schema.table set status = dlr_status where dlr_no = dlr_no_change'''
if update_trigger == True:
cur = con.cursor()
cur.execute(sql)
cur.commit",
ATTR([Dlr No]), ATTR([dlr_status]), ATTR([Update_Now]))
Your commit is missing "()". Or add a con.autocommit = True after creating the connection if you don't want to commit each step.
Related
I need to migrate tables from MS Access to Postgres. I'd like to use pyodbc to do this as it allows me to connect to the Access database using python and query the data.
The problem I have is I'm not exactly sure how to programmatically create a table with the same schema other than just creating a SQL statement using string formatting. pyodbc provides the ability to list all of the fields, field types and field lengths, so I can create a long SQL statement with all of the relevant information, however how can I do this for a bunch of tables? would I need to build SQL string statements for each table?
import pyodbc
access_conn_str = (r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)}; 'r'DBQ=C:\Users\bob\access_database.accdb;')
access_conn = pyodbc.connect(access_conn_str)
access_cursor = access_conn.cursor()
postgres_conn_str = ("DRIVER={PostgreSQL Unicode};""DATABASE=access_database;""UID=user;""PWD=password;""SERVER=localhost;""PORT=5433;")
postgres_conn = pyodbc.connect(postgres_conn_str)
postgres_cursor = postgres_conn.cursor()
table_ditc = {}
row_dict = {}
for row in access_cursor.columns(table='table1'):
row_dict[row.column_name] = [row.type_name, row.column_size]
table_ditc['table1'] = row_dict
for table, values in table_ditc.items():
print(f"Creating table for {table}")
access_cursor.execute(f'SELECT * FROM {table}')
result = access_cursor.fetchall()
postgres_cursor.execute(f'''CREATE TABLE {table} (Do I just put a bunch of string formatting in here?);''')
postgres_cursor.executemany(f'INSERT INTO {table} (Do I just put a bunch of string formatting) VALUES (string formatting?)', result)
postgres_conn.commit()
As you can see, with pyodbc I'm not exactly sure how to build the SQL statements. I know I could build a long string by hand, but if I were doing a bunch of different tables, with different fields etc. that would not be realistic. Is there a better, easier way to create the table and insert rows based off of the schema of the Access database?
I ultimately ended up using a combination of pyodbc and pywin32. pywin32 is "basically a very thin wrapper of python that allows us to interact with COM objects and automate Windows applications with python" (quoted from second link below).
I was able to programmatically interact with Access and export the tables directly to Postgres with DoCmd.TransferDatabase
https://learn.microsoft.com/en-us/office/vba/api/access.docmd.transferdatabase
https://pbpython.com/windows-com.html
import win32com.client
import pyodbc
import logging
from pathlib import Path
conn_str = (r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)}; 'rf'DBQ={access_database_location};')
conn = pyodbc.connect(conn_str)
cursor = conn.cursor()
a = win32com.client.Dispatch("Access.Application")
a.OpenCurrentDatabase(access_database_location)
table_list = []
for table_info in cursor.tables(tableType='TABLE'):
table_list.append(table_info.table_name)
for table in table_list:
logging.info(f"Exporting: {table}")
acExport = 1
acTable = 0
db_name = Path(access_database_location).stem.lower()
a.DoCmd.TransferDatabase(acExport, "ODBC Database", "ODBC;DRIVER={PostgreSQL Unicode};"f"DATABASE={db_name};"f"UID={pg_user};"f"PWD={pg_pwd};""SERVER=localhost;"f"PORT={pg_port};", acTable, f"{table}", f"{table.lower()}_export_from_access")
logging.info(f"Finished Export of Table: {table}")
logging.info("Creating empty table in EGDB based off of this")
This approach seems to be working for me. I like how the creation of the table/fields as well as insertion of data is all handled automatically (which was the original problem I was having with pyodbc).
If anyone has better approaches I'm open to suggestions.
I have a Mirth (v3.10) Database Reader channel source that grabs some test records (from an SQL Server source) using the query...
select *
from [mydb].[dbo].[lab_test_MIRTHTEST_001]
where orc_2_1_placer_order_number
in (
'testid_001', 'testid_002', 'testid_003'
)
Even though the channel appears to function properly and messages are getting written to the channel destination, I am seeing SQL errors in the server logs in the dashboard when deploying the channel:
[2020-12-16 08:16:28,266] ERROR (com.mirth.connect.connectors.jdbc.DatabaseReceiver:268): Failed to process row retrieved from the database in channel "MSSQL2SFTP_TEST"
com.mirth.connect.connectors.jdbc.DatabaseReceiverException: com.microsoft.sqlserver.jdbc.SQLServerException: The index 1 is out of range.
at com.mirth.connect.connectors.jdbc.DatabaseReceiverQuery.runPostProcess(DatabaseReceiverQuery.java:233)
at com.mirth.connect.connectors.jdbc.DatabaseReceiver.processRecord(DatabaseReceiver.java:260)
...
I can run this query fine in the SQL Server Mgmt Studio itself (and the messages seem to be transmitting fine), so not sure why this error is popping up but am concerned there is something I'm missing here.
Anyone with more experience know what is going on here? How to fix?
The issue looks to be in the post-process SQL section of the Database Reader, so it makes sense that the messages appear to be working.
Did you intend to enable the post-process section at the bottom of your source tab?
Kindly share the code that you are using to process data in the result set. In the meantime, you can consider the code below as a staring point. You can place this in Javascript transformer step in the source connector of your channel.
//Declaring variables to hold column values returned from the result set
var variable1;
var variable2;
//defining the sql read command
var Query = "select * from [mydb].[dbo].[lab_test_MIRTHTEST_001]";
Query += " where orc_2_1_placer_order_number in";
Query += " ('testid_001', 'testid_002', 'testid_003')";
var result = dbconn.executeCachedQuery(Query);
//where dbconn is your database connection string
//looping through the results
while(result.next())
{
variable1=result.getString("variable1");
variable2 = result.getString("variable2");
}
//optionally place the returned values in a channel map for use later
$c('variable1',variable1);
$c('variable2',variable2);
Im trying to get Values from a MySQL Table and save them as a dictionary in Python, using MySQL connector since I couldnt manage to install anythingelse...
My problem is that it prints all key-value pairs, but only the last one/last row gets saved in the dictionary f[]. How do I get it to save every key-Value pair in the dictionary?
Thanks in advance!
import mysql.connector
import mysql
conn = mysql.connector.connect(user="F", password="", host="localhost", db="f")
mycursor=conn.cursor(dictionary=True)
query = ("SELECT * FROM 'test')
mycursor.execute(query)
f={}
for row in mycursor:
f = {row["JT"]: row ["f"]}
print f
I found the solution:
for row in mycursor:
f[row['JT']]=row["f"]
]4[]5
I created one thing which access my database table name is sensordata from PostgreSQL. Now I have to send data to these table how. How can I do this?
I did the connection part of thingworx composer and PostgreSQL db on local setup.
I am trying to send sensor data from thingworx to PostgreSQL db but i am not able to sent it
You must to do two things:
1 Create a service to insert row in postgresql_conn thing;
Select 'SQL (Command)' as script type.
Put somthing like this into script area
INSERT INTO sensordata
(Temperature, Humidity, Vibration)
VALUES ([[TemperatureField]], [[HumidityField]], [[VibrationField]]);
TemperatureField, HumidityField, VibrationField are input fields of the service.
2 Create Subscriptions to sensordata thing.
As event set AnyDataChange;
Put something like this in the script area:
var params = {
TemperatureField: me.Temperature,
HumidityField: me.Humidity,
VibrationField: me.Vibration
};
var result = Things["postgresql_conn"].InsertRecord(params);
Now when data of sensordata change one row is add to the postgress table.
Sorry for my english
One of my users wants to get data into Excel from SQL 2008 query/stored proc.
I never actually did it before.
I tried a sample using ADO and got data but user reasonably asked - where are the column names?
How do I connect a spreadsheet to an SQL resultset and get it with column names?
Apparently the field names are in the recordset object already.. just needed to pull them out.
i = 1
For Each objField In rs.Fields
Sheet1.Cells(1, i) = objField.Name
i = i + 1
Next objField
I don't know which version of Excel you are using but in Excel 2007 you can just connect to the SQL DB by going to Data -> From Other Sources -> From SQL Server. After you select your server and database, your connection will be created. Then you can edit it (Data -> Connections -> Properties) where in the Definition tab you change the Command type to SQL and enter your query in the Command text box. You can also create a view on the server and just point to that from Excel.
This should do it unless I misunderstood your question.