I've got an Actix-web server that connects to a Postgres DB.
I've noticed that after a 1000 requests my Postgres DB's RAM usage has spiked.
When I stop actix-web, the RAM held by the db is cleared. This leads me to believe that my code is not releasing the connection.
I cannot find an example of connections actually being released. It looks like it's inferred in everyone else's code.
Here's mine:
async fn hellow_world(a : f32, b : f32, pool: &Pool) -> Result<Value, PoolError> {
let client: Client = pool.get().await?;
let sql = format!("select \"json\" from public.table_a WHERE a={} and b={}", a, b);
let stmt = client.prepare(&sql).await?;
let row = client.query_one(&stmt, &[]).await?;
let result : Value = row.get(0);
Ok(result)
}
#[derive(Deserialize)]
pub struct MyRequest {
a: f32,
b: f32
}
#[get("/hello")]
async fn sv_hellow_world(info: web::Query<MyRequest>, db_pool: web::Data<Pool>) -> Result<HttpResponse, Error> {
let response : Value = hellow_world(info.a, info.b, &db_pool).await?;
Ok(HttpResponse::Ok().json(response))
}
#[actix_rt::main]
async fn main() -> std::io::Result<()> {
dotenv().ok();
let config = Config::from_env().unwrap();
let pool = config.pg.create_pool(tokio_postgres::NoTls).unwrap();
env_logger::from_env(Env::default().default_filter_or("info")).init();
let server = HttpServer::new(move || App::new().wrap(Logger::default()).wrap(Logger::new("%a %{User-Agent}i")).data(pool.clone()).service(sv_hellow_world))
.bind("0.0.0.0:3000")?
.run();
server.await
}
Based on further testing, #Werner determined that the code was piling up server-side prepared statements.
It is not clear whether these statements can be closed using this library.
Either of two approaches can be used to avoid this problem:
Use a single, shared prepared statement
Use the direct query form instead of the prepared statement
I recommend the first approach on principle as it is more efficient and protects against SQL Injection. It should look something like this:
async fn hellow_world(a : f32, b : f32, pool: &Pool) -> Result<Value, PoolError> {
let client: Client = pool.get().await?;
let stmt = client.prepare("select \"json\" from public.table_a WHERE a=$1::numeric and b=$2::numeric").await?;
let row = client.query_one(&stmt, &[&a, &b]).await?;
let result : Value = row.get(0);
Ok(result)
}
Using this code, only one prepared statement should be created on each of the pool's connections.
Related
I've viewed so many stack overflows, but can't get my head around how to fix this.
Here are my deps:
[dependencies]
mongodb = { version = "2.3.1"}
tokio = "1.22.0"
twilio-openapi = "1.0.0"
async-trait = "0.1.58"
futures = "0.3.25"
I've created this async_trait repository trait for implementing various data access repositories:
use std::pin::Pin;
use async_trait::async_trait;
use futures::Future;
use mongodb::{bson::Document, error::Error, options::FindOneOptions, Client};
pub mod notification_settings;
#[async_trait]
pub trait Repository<'a> {
fn with_client(client: &'a mut &Client) -> Self;
async fn find_one<'b>(
self,
filter: Document,
find_one_options: FindOneOptions,
) -> Pin<Box<(dyn Future<Output = Result<Option<Document>, Error>> + Send + 'b)>>;
}
It seems like async_trait sort of requires returning a Pin<Box<dyn Future<Output = X> + Send + 'b)>> Where you can specify a certain lifetime on the borrow. I'm not sure exactly why or if there are ways around this, but again still learning this.
And then here is my struct implementing the trait:
use std::{marker::Send, pin::Pin};
use async_trait::async_trait;
use futures::Future;
use mongodb::{bson::Document, error::Error, options::FindOneOptions, Client};
pub struct NotificationSettingsRepository<'a> {
pub(crate) client: &'a Client,
}
#[async_trait]
impl<'a> super::Repository<'a> for NotificationSettingsRepository<'a> {
fn with_client(client: &'a mut &Client) -> Self {
NotificationSettingsRepository { client: client }
}
async fn find_one<'b>(
self,
filter: Document,
find_one_options: FindOneOptions,
) -> Pin<Box<(dyn Future<Output = Result<Option<Document>, Error>> + Send + 'b)>> {
let collection = self
.client
.database("Occasionally")
.collection::<Document>("NotificationSettings");
let document = collection.find_one(filter, find_one_options);
Box::pin(async { document.await })
}
}
Problem is on collection.find_one(filter, find_one_options); I get an error about:
`collection` does not live long enough
borrowed value does not live long enoughrustcClick for full compiler diagnostic
notification_settings.rs(30, 5): `collection` dropped here while still borrowed
notification_settings.rs(17, 23): lifetime `'b` defined here
notification_settings.rs(21, 10): type annotation requires that `collection` is borrowed for `'b`
This makes some sense because the definition of find_one looks like this:
find_one(&self, filter: impl Into<Option<Document>>, options: impl Into<Option<FindOneOptions>>) -> Result<Option<T>>
So it is borrowing the collection via &self
I've tried a few things like adding an async { } wrapper on the Future of find_one. But I'm not sure what I'm doing there. Please help and thank you!
This is my async function that uses rust to connect to an existing mongoDB database. Is there a way to return / export the client variable / object and make it usable in other Files / Functions?
async fn connect_to_db() -> Result<(), Box<dyn Error>> {
// Load the MongoDB connection string from an environment variable (or string):
let client_uri = "mongodb://localhost:27017";
let options =
ClientOptions::parse_with_resolver_config(&client_uri, ResolverConfig::cloudflare())
.await?;
let client = Client::with_options(options)?;
let db = client.database("fiesta");
// Select Collection(s)
let user_col: Collection<User> = db.collection("users");
let skills_col: Collection<Skill> = db.collection("skills");
//ok.
Ok(())
}
Help would be appreciated.
Dependencies:
rocketrs
mongodb
tokio & serde
I have tried multiple things such as changing the lifetimes and changing the return type of the function, but to no avail.
There is such a module code (for working with a database):
use tokio_postgres::{NoTls, Error};
pub async fn hello() -> Result<(), Error> {
// Connect to the database.
let (client, connection) =
tokio_postgres::connect("host=localhost user=postgres", NoTls).await?;
// The connection object performs the actual communication with the database,
// so spawn it off to run on its own.
tokio::spawn(async move {
if let Err(e) = connection.await {
eprintln!("connection error: {}", e);
}
});
// Now we can execute a simple statement that just returns its parameter.
let rows = client
.query("SELECT $1::TEXT", &[&"hello world"])
.await?;
// And then check that we got back the same string we sent over.
let value: &str = rows[0].get(0);
assert_eq!(value, "hello world");
Ok(())
}
Question:
How, in this case, the access to the database should be written?
(the guide doesn't say anything about it - or I didn't fully understand it.)
https://docs.rs/tokio-postgres/0.5.5/tokio_postgres/
What mechanisms in this case will protect access to the database from sql injections?
The simplest general use case is needed.
client.query(statement, params) will convert the first argument statement to a prepared statement and execute it with the params.
To be safe from sql injection, make sure that all user data is passed in the second params argument.
DO NOT DO THIS:
let id = "SOME DATA FROM THE USER";
let rows = client
.query(format!("SELECT * FROM SomeTable WHERE id = {}", id), &[])
.await?;
DO THIS:
let id = "SOME DATA FROM THE USER";
let rows = client
.query("SELECT * FROM SomeTable WHERE id = $1", &[&id])
.await?;
Explanation:
In tokio-postgres most client methods (query* or execute*) can accept either a &str or Statement for the sql statement. If passed a &str it will create a prepared statement (Statement object) for you.
I used r2d2_postgres to create a connection pool:
fn get_connection_pool(
) -> Result<r2d2::Pool<r2d2_postgres::PostgresConnectionManager<postgres::tls::NoTls>>, Error> {
let manager = PostgresConnectionManager::new(
"host=localhost user=someuser password=hunter2 dbname=mydb"
.parse()
.unwrap(),
NoTls,
);
let pool = r2d2::Pool::new(manager).unwrap();
Ok(pool)
}
And then cloned the connection pool into a warp web request
if let Ok(pool) = pool_conns {
let hello = warp::path!("get_quote" / "co_num" / String / "ctrl_num" / String)
.map(move |co, ctrl| autorate::get_quote(co, ctrl, pool.clone()));
warp::serve(hello).run(([127, 0, 0, 1], 8889)).await;
}
Then called pool.get() inside the request
let mut client = pool.get().unwrap();
but received the runtime error
thread 'main' panicked at 'Cannot start a runtime from within a runtime. This happens because a function
(like 'block_on') attempted to block the current thread while the thread is being used to drive asynchronous tasks.
My question is: in Rust, how should these two concepts work together? Specifically I mean a postgres connection pool and an async web server. I'm thinking I should have a connection pool and be able to pass it into each request to dole out connections as needed. Am I using the wrong connection pool, or just passing it in the wrong way?
Several kind folks on reddit steered me in the right direction. Instead of r2d2, I needed an async connection pool so I switched to deadpool_postgres. Ended up looking like this:
#[tokio::main]
async fn main() {
let mut cfg = Config::new();
cfg.host("yourhost");
cfg.user("youruser");
cfg.password("yourpass");
cfg.dbname("yourdb");
let mgr = Manager::new(cfg, tokio_postgres::NoTls);
let pool = Pool::new(mgr, 16);
let get_quote = warp::path!("get_quote" / "co_num" / String / "ctrl_num" / String)
.and(warp::any().map(move || pool.clone()))
.and_then(autorate::get_quote);
warp::serve(get_quote).run(([127, 0, 0, 1], 8889)).await;
}
And then to use a connection:
pub async fn get_quote(
co: String,
ctrl: String,
pool: deadpool_postgres::Pool,
) -> Result<impl warp::Reply, std::convert::Infallible> {
let co_result = Decimal::from_str(&co);
let ctrl_result = Decimal::from_str(&ctrl);
let client = pool.get().await.unwrap();
if let (Ok(co_num), Ok(ctrl_num)) = (co_result, ctrl_result) {
let orders_result = get_orders(&client, &co_num, &ctrl_num).await;
if let Ok(orders) = orders_result {
if let Ok(rated_orders) = rate_orders(orders, &client).await {
return Ok(warp::reply::json(&rated_orders));
}
}
}
Ok(warp::reply::json(&"No results".to_string()))
}
async fn get_orders(
client: &deadpool_postgres::Client,
co: &Decimal,
ctrl: &Decimal,
) -> Result<Vec<Order>, Error> {
for row in client
.query().await
...
I am experimenting with a standalone script that will query a Postgres database using Vapor and Fluent. In a normal Vapor API application this is simply done by:
router.get("products") { request in
return Product.query(on: request).all()
}
However, in a standalone script, since there is no "request", I get stuck on what to replace the "request" or DatabaseConnectable with. Here's where I get stuck on:
import Fluent
import FluentPostgreSQL
let databaseConfig = PostgreSQLDatabaseConfig(hostname: "localhost",
username: "test",
database: "test",
password: nil)
let database = PostgreSQLDatabase(config: databaseConfig)
let foo = Product.query(on: <??WhatDoIPutHere??>)
I tried creating an object that conforms to DatabaseConnectable, but couldn't figure out how to correctly get that object to conform.
You will need to create an event loop group to be able to make database requests. SwiftNIO's MultiThreadedEventLoopGroup is good for this:
let worker = MultiThreadedEventLoopGroup(numberOfThreads: 2)
You can change the number of threads used as you need.
Now you can create a connection to the database with that worker:
let conn = try database.newConnection(on: worker)
The connection is in a future, so you can map it and pass the connection in your query:
conn.flatMap { connection in
return Product.query(on: connection)...
}
Make sure you shutdown your worker when you are done with it using shutdownGracefully(queue:_:)
The above is very good, but just clarify how simple it is, when you get it, I have made a small test example for this. Hope it helps you.
final class StandAloneTest : XCTestCase{
var expectation : XCTestExpectation?
func testDbConnection() -> Void {
expectation = XCTestExpectation(description: "Wating")
let databaseConfig = PostgreSQLDatabaseConfig(hostname: "your.hostname.here",
username: "username",
database: "databasename",
password: "topsecretpassword")
let database = PostgreSQLDatabase(config: databaseConfig)
let worker = MultiThreadedEventLoopGroup(numberOfThreads: 2)
let conn = database.newConnection(on: worker)
let sc = SomeClass( a:1, b:2, c:3 ) //etc
//get all the tupples for this Class type in the base
let futureTest = conn.flatMap { connection in
return SomeClass.query(on: connection).all()
}
//or save a new tupple by uncommenting the below
//let futureTest = conn.flatMap { connection in
// return someClassInstantiated.save(on: connection)
//}
//lets just wait for the future to test it
//(PS: this blocks the thread and should not be used in production)
do{
let test = try futureTest.wait()
expectation?.fulfill()
worker.syncShutdownGracefully()
print( test )
}catch{
expectation?.fulfill()
print(error)
}
}
}
//Declare the class you want to test here using the Fluent stuff in some extension