I would like to execute multiple statements by mysql connector. The code is as below,
import mysql.connector
conn = mysql.connector.connect(user='root', password='******',
database='dimensionless_ideal')
cursor = conn.cursor()
sql = ("Select * From conditions_ld; "
"Select * From conditions_fw; "
"Select * From results")
cursor.execute(sql, multi=True)
conn.close()
When I run this python script, it is keep running for a very long time unless I kill it. No output and error info. What is the problem?
You should take a look here : https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
It seems that when you use the Multi=True option, the cursor.execute returns an iterator. There is an example on how to use it.
Related
I have set up a connection between pyodbc and the aws rds (postgresql database) and have installed psqlodbc (which is what the Postgres Unicode(x64) odbc driver is). Everything looks fine until I run a SQL query. It returns a syntax error but there is nothing wrong with my syntax. I'm not exactly sure what would be the issue.
This is Python 3.7 by the way.
import pyodbc
mypw = 'skjhaf234234dkjhkjx'
string = 'Driver={PostgreSQL Unicode(x64)};Server=myfakeserveraddress.rds.amazonaws.com;Database=mydb;UID=myusername;PWD='+mypw+';'
connection = pyodbc.connect(string)
c = connection.cursor()
c.execute("SELECT * FROM schema_table.test_table;")
Error Message:
Traceback (most recent call last):
File "", line 1, in
pyodbc.ProgrammingError: ('42601', '[42601] ERROR: syntax error at or near "'schema_table.test_table'";\nError while executing the query (1) (SQLExecDirectW)')
Without the single quotation marks ' surrounding the table name, I get this error
c.execute("SELECT * from schema_table.test_table")
Traceback (most recent call last): File "", line 1, in
pyodbc.ProgrammingError: ('25P02', '[25P02] ERROR: current
transaction is aborted, commands ignored until end of transaction
block;\nError while executing the query (1) (SQLExecDirectW)')
PS My company has disabled pip installs so I cannot upgrade my packages and am limited to using only a few packages (including this one).
How can I execute my commands without errors?
It seems I have figured it out.... I added autocommit=False to the connection initialization and it seems fine now.... Perhaps it has something to do with the underlying parsing of the sql commands. Keeping the question in case it helps someone.
import pyodbc
mypw = 'skjhaf234234dkjhkjx'
string = 'Driver={PostgreSQL Unicode(x64)};Server=myfakeserveraddress.rds.amazonaws.com;Database=mydb;UID=myusername;PWD='+mypw+';'
connection = pyodbc.connect(string, autocommit=False)
c = connection.cursor()
c.execute("SELECT * FROM schema_table.test_table;")
Today I was stress testing multiprocessing in Python against our cloud MongoDB (Atlas).
It's currently running at 100%, and I'd like to do something like a "restart".
I have found a "Shutdown" command, but I can't find a command to start it up after it has shutdown, so I'm afraid to run just the "Shutdown".
I have tried killing processes one at a time in the lower right section of the screen below, but after refreshing the page, the same processes numbers are there, and I think there are more on the bottom of the list. I think they are all backed up.
Doing an insert of a large document does not return to the Python program in 5 minutes. I need to get that working again (should update in 10-15 seconds as it has in the past).
I am able to open a command window and connect to that server. Just unclear what commands to run.
Here is an example of how I tried to kill some of the processes:
Also note that the "Performance Adviser" page is not recommending any new indexes.
Update 1:
Alternatively, can I kill all running, hung, or locked processes?
I was reading about killop here:(https://docs.mongodb.com/manual/tutorial/terminate-running-operations/), but found it confusing with the versions, and the fact that I'm using Atlas.
I'm not sure if there is an easier way, but this is what I did.
First, I ran a Python program to extract all the desired operation IDs based on my database and collection name. You have to look at the file created to understand the if statements in the code below. NOTE: it says that db.current_op is deprecated, and I haven't found out how to do this without that command (from PyMongo).
Note the doc page warns against killing certain types of operations, so I was careful to pick ones that were doing inserts on one specific collection. (Do not attempt to kill all processes in the JSON returned).
import requests
import os
import sys
import traceback
import pprint
import json
from datetime import datetime as datetime1, timedelta, timezone
import datetime
from time import time
import time as time2
import configHandler
import pymongo
from pymongo import MongoClient
from uuid import UUID
def uuid_convert(o):
if isinstance(o, UUID):
return o.hex
# This get's all my config from a config.json file, not including that code here.
config_dict = configHandler.getConfigVariables()
cluster = MongoClient(config_dict['MONGODB_CONNECTION_STRING_ADMIN'])
db = cluster[config_dict['MONGODB_CLUSTER']]
current_ops = db.current_op(True)
count_ops = 0
for op in current_ops["inprog"]:
count_ops += 1
#db.kill- no such command
if op["type"] == "op":
if "op" in op:
if op["op"] == "insert" and op["command"]["insert"] == "TestCollectionName":
#print(op["opid"], op["command"]["insert"])
print('db.adminCommand({"killOp": 1, "op": ' + str(op["opid"]) + '})')
print("\n\ncount_ops=", count_ops)
currDateTime = datetime.datetime.now()
print("type(current_ops) = ", type(current_ops))
# this dictionary has nested fields
# current_ops_str = json.dumps(current_ops, indent=4)
# https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior
filename = "./data/currents_ops" + currDateTime.strftime("%y_%m_%d__%H_%M_%S") + ".json"
with open(filename, "w") as file1:
#file1.write(current_ops_str)
json.dump(current_ops, file1, indent=4, default=uuid_convert)
print("Wrote to filename=", filename)
It writes the full ops file to disk, but I did a copy/paste from the command windows to a file. Then from the command line, I ran something like this:
mongo "mongodb+srv://mycluster0.otwxp.mongodb.net/mydbame" --username myuser --password abc1234 <kill_opid_script.js
The kill_opid_script.js looked like this. I added the print(db) because the first time I ran it didn't seem to do anything.
print(db)
db.adminCommand({"killOp": 1, "op": 648685})
db.adminCommand({"killOp": 1, "op": 667396})
db.adminCommand({"killOp": 1, "op": 557439})
etc... for 400+ times...
print(db)
def fetch_holidays_in_date_range(src):
query = "SELECT * from holiday_tab where id = src"
db = dbconnect.connect()
# defining the cursor for reading data
cursor = db.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
# query the database
cursor.execute(query.format(src));
rows = cursor.fetchall()
dbconnect.destroy(cursor, db)
return rows
Could someone help, how to mock this code in pytest or in unittest. I've googled for mock db and I hardly found anything.
Pytest doesn't support you to run the test on production/main DB if you are using pytest-django.
There is a better approach to solve this issue.
Pytest DB resolve method
This says that whenever you run the test with a marker #pytest.mark.django_db, tests are run on another newly created db with name test_your production_db_name.
So if your db name is hello, pytest will create a new db called test_hello and runs tests on it
I need assistance with understanding why my hivevar is not being set in my query?
This is my beeline statement in a shell script:
Start_Date="20180423"
End_Date="20180424"
beeline -u 'jdbc:hive2://#####/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=######;' -f ${my_queries}/Count_Query --showHeader=false --outputformat=csv2 --silent=false --hivevar start_date=$Start_Date --hivevar end_date=$End_Date 1>${my_data}/Data_File 2>${my_log}/Log_File
The Query
use sample_db;
select count(*) from sample_table where data_dt>=${start_date} and data_dt<${end_date};
When I look at the data file, which provides a dump of the query, the variables are not properly set to the values.
0: jdbc:hive2://####> use sample_db;
0: jdbc:hive2://####> select count(*) from sample_table where data_dt>=${start_date} and data_dt<${end_date};
The issue is following part
**--hivevar start_date=$Start_Date --hivevar end_date=$End_Date**
Remove ** and you are good to go.
Shell Script.
Start_Date="20180423"
End_Date="20180424"
beeline_cmd="beeline -u 'jdbc:hive2://#####/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=######;' --showHeader=false --outputformat=csv2 --silent=false"
${beeline_cmd} -f ${my_queries}/Count_Query --hivevar start_date=${Start_Date} --hivevar end_date=${End_Date} 1>${my_data}/Data_File 2>${my_log}/Log_File
Hive Query
There are 4 postgresql instances running on 1 server. I want to make ddl change in all of them at same time.
How can I do this?
Write a shell script (if you're on *nix) or .cmd batch file / .vbs (if you're on Windows) to do it. Have the script invoke psql -f /path/to/ddl.sql and the IP/port, database name, etc.
Alternately, write a script in a language like Python that has proper PostgreSQL bindings. Have the script loop through the databases and run the DDL for each. In Python, for example, the following (untested) script should do the trick:
import psycopg2
conn_definitions = [
"dbname=db1 port=5432 host=127.0.0.1",
"dbname=db2 port=5432 host=127.0.0.1",
"dbname=db3 port=5432 host=127.0.0.1",
"dbname=db4 port=5432 host=127.0.0.1",
]
ddl = """
CREATE TABLE blah (
blah integer
);
CREATE INDEX blah_blah_idx ON blah(blah);
"""
connections = []
cursors = []
for conn_info in conn_definitions:
conn = psycopg2.connect(conn_info)
curs = conn.cursor()
cursors.append(curs)
connections.append(conn)
for curs in cursors:
curs.execute("BEGIN;")
for curs in cursors:
curs.execute(ddl)
for curs in cursors:
curs.execute("COMMIT;")
for conn in connections:
conn.close()
Enhance if desired by doing things like splitting the DDL into an array of statements you loop over so you can do per-statement error handling.
You could also generate the connections dynamically by connecting to one DB for each host and running select datname from pg_database to get a listing of other DBs.