Creating postgres schemas using psycopg cur.execute - postgresql

My python application allows users to create schemas of their naming. I need a way to protect the application from sql injections.
The SQL to be executed reads
CREATE SCHEMA schema_name AUTHORIZATION user_name;
The psycopg documentation (generally) recommends passing parameters to execute like so
conn = psycopg2.connect("dbname=test user=postgres")
cur = conn.cursor()
query = 'CREATE SCHEMA IF NOT EXISTS %s AUTHORIZATION %s;'
params = ('schema_name', 'user_name')
cur.execute(query, params)
But this results in a query with single quotes, which fails:
CREATE SCHEMA 'schema_name' AUTHORIZATION 'user_name';
> fail
Is there a way to remove the quotes, or should I just settle for stripping non-alphanumeric characters from the schema name and call it a day? The later seems kind of ugly, but should still work.

To pass identifiers use AsIs. But that exposes to SQL injection:
import psycopg2
from psycopg2.extensions import AsIs
conn = psycopg2.connect(database='cpn')
cursor = conn.cursor()
query = """CREATE SCHEMA %s AUTHORIZATION %s;"""
param = (AsIs('u1'), AsIs('u1; select * from user_table'))
print cursor.mogrify(query, param)
Output:
CREATE SCHEMA u1 AUTHORIZATION u1; select * from user_table;

Here's a boilerplate that might help. I've used environment variables but you can use a .conf or whatever you like.
Store your connection variables in a .env file:
db_host = "localhost"
db_port = "5432"
db_database = "postgres"
db_user = "postgres"
db_password = "postgres"
db_schema = "schema2"
Load params in your app.py and assign them to variables, then use the variables where required:
import psychopg2
from dotenv import load_dotenv
import database
# Load your environment variables here:
load_dotenv()
db_host = os.environ["db_host"]
db_port = os.environ["db_port"]
db_database = os.environ["db_database"]
db_user = os.environ["db_user"]
db_password = os.environ["db_password"]
db_schema = os.environ["db_schema"]
# Build Connection:
connection = psycopg2.connect(host=db_host,
port=db_port,
database=db_database,
user=db_user,
password=db_password
)
# Build Query Strings:
CREATE_SCHEMA = f"CREATE SCHEMA IF NOT EXISTS {schema};"
CREATE_TABLE1 = f"CREATE TABLE IF NOT EXISTS {schema}.table1 (...);"
CREATE_TABLE2 = f"CREATE TABLE IF NOT EXISTS {schema}.table2 (...);"
# Create Schema and Tables:
with connection:
with connection.cursor() as cursor:
cursor.execute(CREATE_SCHEMA)
cursor.execute(CREATE_TABLE1)
cursor.execute(CREATE_TABLE2)

As of psycopg2 >= 2.7, psycopg2.sql can be used to compose dynamic statements, which also guards from SQL injection.

Related

Flask Rest Api SQL Alchemy connection Cloud Sql Postgresq

I have a connection problem with Cloud Sql Postgres from my Flask Rest API app.
I have a db.py file:
import os
from flask_sqlalchemy import SQLAlchemy
import sqlalchemy
db = SQLAlchemy()
def connect_unix_socket() -> sqlalchemy.engine.base.Engine:
""" Initializes a Unix socket connection pool for a Cloud SQL instance of Postgres. """
# Note: Saving credentials in environment variables is convenient, but not
# secure - consider a more secure solution such as
# Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
# keep secrets safe.
db_user = os.environ["DB_USER"] # e.g. 'my-database-user'
db_pass = os.environ["DB_PASS"] # e.g. 'my-database-password'
db_name = os.environ["DB_NAME"] # e.g. 'my-database'
unix_socket_path = os.environ["INSTANCE_UNIX_SOCKET"] # e.g. '/cloudsql/project:region:instance'
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<INSTANCE_UNIX_SOCKET>/.s.PGSQL.5432
# Note: Some drivers require the `unix_sock` query parameter to use a different key.
# For example, 'psycopg2' uses the path set to `host` in order to connect successfully.
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user,
password=db_pass,
database=db_name,
query={"unix_sock": "{}/.s.PGSQL.5432".format(unix_socket_path)},
),
# [START_EXCLUDE]
# Pool size is the maximum number of permanent connections to keep.
pool_size=5,
# Temporarily exceeds the set pool_size if no connections are available.
max_overflow=2,
# The total number of concurrent connections for your application will be
# a total of pool_size and max_overflow.
# 'pool_timeout' is the maximum number of seconds to wait when retrieving a
# new connection from the pool. After the specified amount of time, an
# exception will be thrown.
pool_timeout=30, # 30 seconds
# 'pool_recycle' is the maximum number of seconds a connection can persist.
# Connections that live longer than the specified amount of time will be
# re-established
pool_recycle=1800, # 30 minutes
# [END_EXCLUDE]
)
return pool
I import the db.py file in my app.py file:
import os
import sqlalchemy
from flask import Flask
from flask_smorest import Api
from flask_sqlalchemy import SQLAlchemy
from db import db, connect_unix_socket
import models
from resources.user import blp as UserBlueprint
# pylint: disable=C0103
app = Flask(__name__)
def init_connection_pool() -> sqlalchemy.engine.base.Engine:
# use a Unix socket when INSTANCE_UNIX_SOCKET (e.g. /cloudsql/project:region:instance) is defined
if unix_socket_path:
return connect_unix_socket()
raise ValueError(
"Missing database connection type. Please define one of INSTANCE_HOST, INSTANCE_UNIX_SOCKET, or INSTANCE_CONNECTION_NAME"
)
db = None
#app.before_first_request
def init_db() -> sqlalchemy.engine.base.Engine:
global db
db = init_connection_pool()
api = Api(app)
#app.route("/api")
def user_route():
return "Welcome user API!"
api.register_blueprint(UserBlueprint)
if __name__ == '__main__':
server_port = os.environ.get('PORT', '8080')
app.run(debug=True, port=server_port, host='0.0.0.0')
The app run correctly, when i call the end point to Get or Post users, the app crash and give me this error:
"The current Flask app is not registered with this 'SQLAlchemy'"
RuntimeError: The current Flask app is not registered with this 'SQLAlchemy' instance. Did you forget to call 'init_app', or did you create multiple 'SQLAlchemy' instances?
This is my User.py class:
from sqlalchemy.exc import SQLAlchemyError, IntegrityError
from db import db
from models import UserModel
from schemas import UserSchema
blp = Blueprint("Users", "users", description="Operations on users")
#blp.route("/user/<string:user_id>")
class User(MethodView):
#blp.response(200, UserSchema)
def get(self, user_id):
user = UserModel.query.get_or_404(user_id)
return user
def delete(self, user_id):
user = UserModel.query.get_or_404(user_id)
db.session.delete(user)
db.session.commit()
return {"message": "User deleted"}, 200
#blp.route("/user")
class UserList(MethodView):
#blp.response(200, UserSchema(many=True))
def get(self):
return UserModel.query.all()
How i can fix this issue?
#dev_ Your issue is that your are trying to intermingle the use of SQLAlchemy Core with SQLAlchemy ORM as if they are the same thing, leading to your issues. SQLAlchemy connection pools created using sqlalchemy.create_engine use the CORE API while Flask-SQLAlchemy uses the SQLAlchemy ORM model. This is the core reason for you issue. It is easier to use one or the other.
I would recommend using purely Flask-SQLALchemy with the use of the cloud-sql-python-connector library for your use-case. It will make your life much easier.
For simplicity, I am getting rid of your db.py leading to your app.py file being as follows:
from flask import Flask
from flask_smorest import Api
from flask_sqlalchemy import SQLAlchemy
from google.cloud.sql.connector import Connector, IPTypes
from resources.user import blp as UserBlueprint
# load env vars
db_user = os.environ["DB_USER"] # e.g. 'my-database-user'
db_pass = os.environ["DB_PASS"] # e.g. 'my-database-password'
db_name = os.environ["DB_NAME"] # e.g. 'my-database'
instance_connection_name = os.environ["INSTANCE_CONNECTION_NAME"] # e.g. 'project:region:instance'
# Python Connector database connection function
def getconn():
with Connector() as connector:
conn = connector.connect(
instance_connection_name, # Cloud SQL Instance Connection Name
"pg8000",
user=db_user,
password=db_pass,
db=db_name,
ip_type= IPTypes.PUBLIC # IPTypes.PRIVATE for private IP
)
return conn
app = Flask(__name__)
# configure Flask-SQLAlchemy to use Python Connector
app.config['SQLALCHEMY_DATABASE_URI'] = "postgresql+pg8000://"
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {
"creator": getconn
}
# initialize db (using app!)
db = SQLAlchemy(app)
# rest of your code
api = Api(app)
# ...
Hope this helps resolve your issue!

Karate + Gradle : Is there a way to get environment variable value in a .java file and define Postgres DB configuration based on env variable value? [duplicate]

I have a DB utils java file where I need to load DB username password based on environment I am running the code on , and these environment values I need to import from karate-config.js . How to achieve this ?
Just use embedded expressions ! So if you have dbusername and dbpassword set in karate-config.js:
* def config = { username: '#(dbusername)', password: '#(dbpassword)', url: 'jdbc:h2:mem:testdb', driverClassName: 'org.h2.Driver' }
* def DbUtils = Java.type('com.mycompany.DbUtils')
* def db = new DbUtils(config)

How can I add a custom variable to Sqitch, to be used in target postgres

I would like to add few variables:"username' and "database" in my sqitch.conf on a defined target.
file sqitch.conf=>
engine = pg
[core "variables"]
username = jv_root
database = test
[target "dev_1"]
uri = db:pg://username#sqlhost:5432/database
[target "dev_2"]
uri = db:pg://username#sqlhost2:5432/database
where I run:
sqitch deploy -t dev_1
it throw an error =>
ERROR: no such user: username
You can add environment specific variables like this.
[target.dev_1.variables]
username = jv_root
password = test
How you address them in your sql files depends on the sql dialect.

Azure python create user database

I am trying some azure documentation Quickstart tutorial to create a resource group with one SQL Server and one database. The code runs just fine and I am able to create all the resource. Now I was curious how can I run in the same script the code to create a readonly user inside the database I am creating?
This is the code I have:
import os
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.sql import SqlManagementClient
REGION = 'westus'
GROUP_NAME = 'resource-group-name'
SERVER_NAME = 'server-name'
DATABASE_NAME = 'sample-db'
def run_example():
subscription_id = os.environ.get(
'AZURE_SUBSCRIPTION_ID',
'11111-11-1111-11111-111111') # your Azure Subscription Id
credentials = ServicePrincipalCredentials(
client_id='my-client-id',
secret='my-secret',
tenant='tenant'
)
resource_client = ResourceManagementClient(credentials, subscription_id)
sql_client = SqlManagementClient(credentials, subscription_id)
# You MIGHT need to add SQL as a valid provider for these credentials
# If so, this operation has to be done only once for each credentials
resource_client.providers.register('Microsoft.Sql')
# Create Resource group
print('Create Resource Group')
resource_group_params = {'location': 'westus'}
print_item(resource_client.resource_groups.create_or_update(
GROUP_NAME, resource_group_params))
# Create a SQL server
print('Create a SQL server')
server = sql_client.servers.create_or_update(
GROUP_NAME,
SERVER_NAME,
{
'location': REGION,
'version': '12.0', # Required for create
'administrator_login': 'server-login', # Required for create
'administrator_login_password': 'pass-word' # Required for create
}
)
print_item(server)
print('\n\n')
# Get SQL server
print('Get SQL server')
server = sql_client.servers.get_by_resource_group(
GROUP_NAME,
SERVER_NAME,
)
print_item(server)
print("\n\n")
# List SQL servers by resource group
print('List SQL servers in a resource group')
for item in sql_client.servers.list_by_resource_group(GROUP_NAME):
print_item(item)
print("\n\n")
# List SQL servers by subscription
print('List SQL servers in a subscription')
for item in sql_client.servers.list():
print_item(item)
print("\n\n")
# List SQL servers usage
print('List SQL servers usage')
for item in sql_client.servers.list_usages(GROUP_NAME, SERVER_NAME):
print_metric(item)
print("\n\n")
# Create a database
print('Create SQL database')
async_db_create = sql_client.databases.create_or_update(
GROUP_NAME,
SERVER_NAME,
DATABASE_NAME,
{
'location': REGION
}
)
# Wait for completion and return created object
database = async_db_create.result()
print_item(database)
print("\n\n")
# Get SQL database
print('Get SQL database')
database = sql_client.databases.get(
GROUP_NAME,
SERVER_NAME,
DATABASE_NAME
)
print_item(database)
print("\n\n")
# List SQL databases by server
print('List SQL databases in a server')
for item in sql_client.databases.list_by_server(GROUP_NAME, SERVER_NAME):
print_item(item)
print("\n\n")
# List SQL database usage
print('List SQL database usage')
for item in sql_client.databases.list_usages(GROUP_NAME, SERVER_NAME, DATABASE_NAME):
print_metric(item)
print("\n\n")
def print_item(group):
"""Print an Azure object instance."""
print("\tName: {}".format(group.name))
print("\tId: {}".format(group.id))
print("\tLocation: {}".format(group.location))
if hasattr(group, 'tags'):
print("\tTags: {}".format(group.tags))
if hasattr(group, 'properties'):
print_properties(group.properties)
def print_metric(group):
"""Print an SQL metric."""
print("\tResource Name: {}".format(group.resource_name))
print("\tName: {}".format(group.display_name))
print("\tValue: {}".format(group.current_value))
print("\tUnit: {}".format(group.unit))
def print_properties(props):
"""Print a ResourceGroup properties instance."""
if props and props.provisioning_state:
print("\tProperties:")
print("\t\tProvisioning State: {}".format(props.provisioning_state))
print("\n\n")
if __name__ == "__main__":
run_example()
I am missing this last bit where I want to create this readonly user inside the database I am creating. Thank you very much for your time and help guys
Create user in Azure SQL database is very different with create database instance. It needs the admin account or the enough permission, and the user binds the login, the login must be created in master DB, and the user must be created in current user D, then alter the database role to the user. The code you user is not suitable for create the user.
Even with pyodbc script, you still need the connection string, specify the database/user,/password. The limit is you can't access master DB and user database with one connection string or SQL database connection.
I'm afraid to say we can't do that with the code.

How to load Postgres QgsVectorLayer

I have a QGIS script that I am trying to load a vector layer that is stored in a Postgres database. When I print the layer's isValid() method I get False. Here is my code:
from qgis.core import *
db_client = 'postgres'
db_host = 'localhost'
db_port = '5432'
db_name = 'database'
db_user = 'user'
db_password = 'pass123'
db_schema = 'public'
tablename = 'Geo_Layer'
geometrycol = 'geom'
tract_number_index = 3
QgsApplication.setPrefixPath('/usr/bin/qgis', True)
qgs = QgsApplication([], False)
qgs.initQgis()
geo_uri = QgsDataSourceUri()
geo_uri.setConnection(db_host, db_port, db_name, db_user, db_password)
geo_uri.setDataSource(db_schema, tablename, geometrycol, '', 'id')
geo_layer = QgsVectorLayer(geo_uri.uri(False), "Test", "postgres")
# Other configurations I have tried
# geo_layer = QgsVectorLayer(geo_uri.uri(), "Test", "postgres")
# geo_layer = QgsVectorLayer(geo_uri.uri(), "Test", "ogr")
# geo_layer = QgsVectorLayer(geo_uri.uri(False), "Test", "ogr")
print(geo_layer.isValid())
qgs.exitQgis()
I have provided the other QgsVectorLayer configurations I have tried. All print that the layer is not valid.
QGIS Version: 3.16.3-Hannover
Python Version: 3.8.5
Ubuntu Version: 20.04.02 LTS
I have check my credentials with DBeaver and I am able to connect.
I once faced this issue when my geometry column in postgis contains multiple geometry type. In this case you can first filter the column for geometry types, and then for each geometry type construct a layer for qgis:
for geom in geometry_types:
uri.setDataSource(schema, table, column, "GeometryType(%s)= '%s'" % (column, geom))
vlayer = QgsVectorLayer(uri.uri(), layer_name, "postgres")
print(vlayer.isValid())
You can check for the geometry types in postgis using following query:
SELECT DISTINCT(GeometryType("%s"::geometry)) FROM "%s";""" % (column, table)