I'm new to alembic so this is very likely a beginners fallacy. I have a FastAPI server which runs SQL migrations on startup.
The problem is, that these migrations are being executed every time the server is starting up. This results in an error where the table is already created:
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1050, "Table 'signal' already exists")
Why is alembic running this migration again and how would it even know that the migration already happened? For example, I wouldn't want to run INSERT statements repeatedly either.
Important detail here (maybe): I am attaching to a Docker container which starts the server each time I connect.
Here's the relevant code:
def run_sql_migrations():
# retrieves the directory that *this* file is in
migrations_dir = os.path.join(os.path.dirname(os.path.realpath(db.__file__)), "alembic")
# this assumes the alembic.ini is also contained in this same directory
config_file = os.path.join(migrations_dir, "..", "alembic.ini")
db_url = f"mysql://{os.environ['DB_USER']}:{os.environ['DB_PASSWORD']}" \
f"#{os.environ['DB_HOST']}:{os.environ['DB_PORT']}/{os.environ['DB_NAME']}"
config = Config(file_=config_file)
config.set_main_option("script_location", migrations_dir)
config.set_main_option("sqlalchemy.url", db_url)
# upgrade the database to the latest revision
upgrade(config, "head")
app = FastAPI()
run_sql_migrations()
My first migration is nothing more than the creation of one table:
"""testing revisioning
Revision ID: 2287ab5f2d12
Revises:
Create Date: 2023-01-27 11:02:48.341141
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '2287ab5f2d12'
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
'signal',
sa.Column('id', sa.Integer, primary_key=True),
)
def downgrade() -> None:
op.drop_table("signal")
Related
I'm following the Diesel Getting Started Tutorial.
When I run the migrations the schema.rs file generated is empty.
I checked postgres and the database and table is created.
What am I doing wrong? I followed every step of the tutorial.
Edit: in the up.sql
CREATE TABLE posts (
id SERIAL PRIMARY KEY,
title VARCHAR NOT NULL,
body TEXT NOT NULL,
published BOOLEAN NOT NULL DEFAULT FALSE
)
Cargo.toml
[package]
name = "diesel_demo"
version = "0.1.0"
edition = "2021"
[dependencies]
diesel = { version = "2.0.0", features = ["postgres"] }
dotenvy = "0.15"
diesel.toml
[print_schema]
file = "src/schema.rs"
[migrations_directory]
dir = "migrations"
.env
DATABASE_URL=postgres://postgres:123#localhost/diesel_demo
Output of diesel migration run and diesel migration redo. I only get an output in the first time I ran diesel migration run.
Edit2: I solved by reeinstaling rust, postgresql and the c++ build tools. Thanks everyone for the help.
Today I was stress testing multiprocessing in Python against our cloud MongoDB (Atlas).
It's currently running at 100%, and I'd like to do something like a "restart".
I have found a "Shutdown" command, but I can't find a command to start it up after it has shutdown, so I'm afraid to run just the "Shutdown".
I have tried killing processes one at a time in the lower right section of the screen below, but after refreshing the page, the same processes numbers are there, and I think there are more on the bottom of the list. I think they are all backed up.
Doing an insert of a large document does not return to the Python program in 5 minutes. I need to get that working again (should update in 10-15 seconds as it has in the past).
I am able to open a command window and connect to that server. Just unclear what commands to run.
Here is an example of how I tried to kill some of the processes:
Also note that the "Performance Adviser" page is not recommending any new indexes.
Update 1:
Alternatively, can I kill all running, hung, or locked processes?
I was reading about killop here:(https://docs.mongodb.com/manual/tutorial/terminate-running-operations/), but found it confusing with the versions, and the fact that I'm using Atlas.
I'm not sure if there is an easier way, but this is what I did.
First, I ran a Python program to extract all the desired operation IDs based on my database and collection name. You have to look at the file created to understand the if statements in the code below. NOTE: it says that db.current_op is deprecated, and I haven't found out how to do this without that command (from PyMongo).
Note the doc page warns against killing certain types of operations, so I was careful to pick ones that were doing inserts on one specific collection. (Do not attempt to kill all processes in the JSON returned).
import requests
import os
import sys
import traceback
import pprint
import json
from datetime import datetime as datetime1, timedelta, timezone
import datetime
from time import time
import time as time2
import configHandler
import pymongo
from pymongo import MongoClient
from uuid import UUID
def uuid_convert(o):
if isinstance(o, UUID):
return o.hex
# This get's all my config from a config.json file, not including that code here.
config_dict = configHandler.getConfigVariables()
cluster = MongoClient(config_dict['MONGODB_CONNECTION_STRING_ADMIN'])
db = cluster[config_dict['MONGODB_CLUSTER']]
current_ops = db.current_op(True)
count_ops = 0
for op in current_ops["inprog"]:
count_ops += 1
#db.kill- no such command
if op["type"] == "op":
if "op" in op:
if op["op"] == "insert" and op["command"]["insert"] == "TestCollectionName":
#print(op["opid"], op["command"]["insert"])
print('db.adminCommand({"killOp": 1, "op": ' + str(op["opid"]) + '})')
print("\n\ncount_ops=", count_ops)
currDateTime = datetime.datetime.now()
print("type(current_ops) = ", type(current_ops))
# this dictionary has nested fields
# current_ops_str = json.dumps(current_ops, indent=4)
# https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior
filename = "./data/currents_ops" + currDateTime.strftime("%y_%m_%d__%H_%M_%S") + ".json"
with open(filename, "w") as file1:
#file1.write(current_ops_str)
json.dump(current_ops, file1, indent=4, default=uuid_convert)
print("Wrote to filename=", filename)
It writes the full ops file to disk, but I did a copy/paste from the command windows to a file. Then from the command line, I ran something like this:
mongo "mongodb+srv://mycluster0.otwxp.mongodb.net/mydbame" --username myuser --password abc1234 <kill_opid_script.js
The kill_opid_script.js looked like this. I added the print(db) because the first time I ran it didn't seem to do anything.
print(db)
db.adminCommand({"killOp": 1, "op": 648685})
db.adminCommand({"killOp": 1, "op": 667396})
db.adminCommand({"killOp": 1, "op": 557439})
etc... for 400+ times...
print(db)
I am using alembic to generate database migration scripts for a mysql database. I've noticed that the syntax of the generated upgrade and downgrade scripts differ slightly, whereas I thought they would basically be the same.
models.py- before
class Message_User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(20), nullable=True)
models.py-after table modification
class Message_User(db.Model):
id = db.Column(db.Integer, primary_key=True)
tag = db.Column(db.String(15), nullable=True)
migration file - original - shows table creation
def upgrade():
op.create_table('message_user',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=20), nullable=True)
sa.PrimaryKeyConstraint('id', name=op.f('pk_message_user'))
)
def downgrade():
op.drop_table('message_user')
migration file - after - shows table modification
def upgrade():
op.add_column('message_user', sa.Column('tag', sa.String(length=15), nullable=True))
op.drop_column('message_user', 'name')
def downgrade():
op.add_column('message_user', sa.Column('name', mysql.VARCHAR(collation='utf8_bin',
length=20), nullable=True))
op.drop_column('message_user', 'tag')
The upgrade scripts describe the changes purely in sqlalchemy terms, whereas the downgrade scripts add mysql dialect specific changes. Specifically, the upgrade script defines the type as sa.String(length=15) whereas the downgrade defines it as mysql.VARCHAR(collation='utf8_bin', length=20). In create table statements in the downgrade scripts, the autogenerated script also includes mysql_collate, mysql_default_charset and mysql_engine whereas these aren't in create table statements for upgrade scripts. I didn't see any mention of this in the alembic documentation. Does anyone know why this differs?
def fetch_holidays_in_date_range(src):
query = "SELECT * from holiday_tab where id = src"
db = dbconnect.connect()
# defining the cursor for reading data
cursor = db.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
# query the database
cursor.execute(query.format(src));
rows = cursor.fetchall()
dbconnect.destroy(cursor, db)
return rows
Could someone help, how to mock this code in pytest or in unittest. I've googled for mock db and I hardly found anything.
Pytest doesn't support you to run the test on production/main DB if you are using pytest-django.
There is a better approach to solve this issue.
Pytest DB resolve method
This says that whenever you run the test with a marker #pytest.mark.django_db, tests are run on another newly created db with name test_your production_db_name.
So if your db name is hello, pytest will create a new db called test_hello and runs tests on it
I designed a GUI application using wxPython that communicate with a local database (Mongodb) located in the same folder. My main application has the relative path to the database daemon to start it every time the GUI is lunched.
This is the main.py:
import mongodb
class EVA(wx.App):
# wxPython GUI here
pass
if __name__ == "__main__":
myMongodb = mongodb.Mongodb()
myMongodb.start()
myMongodb.connect()
app = EVA(0)
app.MainLoop()
This is the mongodb.py module:
from pymongo import Connection
import subprocess, os , signal
class Mongodb():
pid = 0
def start(self):
path = "/mongodb-osx-x86_64-1.6.5/bin/mongod"
data = "/data/db/"
cmd = path + " --dbpath " + data
MyCMD = subprocess.Popen([cmd],shell=True)
self.pid = MyCMD.pid
def connect(self):
try:
connection = Connection(host="localhost", port=27017)
db = connection['Example_db']
return db
except Exception as inst:
print "Database connection error: " , inst
def stop(self):
os.kill(self.pid,signal.SIGTERM)
Every thing works fine from the terminal. However, when I used py2app to make a standalone version of my program on Mac OS (OS v10.6.5, Python v2.7), I am able to lunch the GUI but can't start the database. It seems py2app changed the location of Mongodb executable folder and broke my code.
I use the following parameters with py2app:
$ py2applet --make-setup main.py
$ rm -rf build dist
$ python setup.py py2app --iconfile /icons/main_icon.icns -r /mongodb-osx-x86_64-1.6.5
How to force py2app to leave my application structure intact?
Thanks.
Py2app changes the current working directory to the foo.app/Content/Resources folder within the app bundle when it starts up. It doesn't seem to be the case from the code you show above, but if you have any paths that are dependent on the CWD (including relative pathnames) then you'll have to deal with that somehow. One common way to deal with it is to also copy the other stuff you need into that folder within the application bundle, so it will then truly be a standalone bundle that is not dependent on its location in the filesystem and hopefully also not dependent on the machine it is running upon.