Configure PostgreSQL DB with Vapor 3 on Heroku - swift

I've built a simple Vapor 3 API that I'd like to deploy on Heroku. I'd like it to be backed by a PostgreSQL database which is also attached to another Heroku app (I have successfully attached the DB in the Heroku dashboard – and the DB works correctly in the other application). However, my Vapor app never completes starting up, crashing with the following error:
Fatal error: Error raised at top level: ⚠️ PostgreSQL Error: no pg_hba.conf entry for host "[the IP addr]", user "[heroku postgres username here]", database "[heroku psql db here]", SSL off
- id: PostgreSQLError.server.fatal.ClientAuthentication
I used vapor heroku init to set up the Heroku app. I've Googled around, and tried adding a Procfile and messing with configure.swift, but so far no luck. Here are all the relevant files I can think of:
Procfile:
web: Run serve --env production --hostname 0.0.0.0 --port $PORT --config:postgresql.url $DATABASE_URL
configure.swift:
import FluentPostgreSQL
import Vapor
/// Called before your application initializes.
public func configure(_ config: inout Config, _ env: inout Environment, _ services: inout Services) throws {
/// Register providers first
try services.register(FluentPostgreSQLProvider())
var contentConfig = ContentConfig.default()
/// Create custom JSON encoder
let jsonEncoder = JSONEncoder()
if #available(OSX 10.12, *) {
jsonEncoder.dateEncodingStrategy = .iso8601
} else {
jsonEncoder.dateEncodingStrategy = .millisecondsSince1970
}
// jsonEncoder.keyEncodingStrategy = .convertToSnakeCase
/// Register JSON encoder and content config
contentConfig.use(encoder: jsonEncoder, for: .json)
services.register(contentConfig)
/// Register routes to the router
let router = EngineRouter.default()
try routes(router)
services.register(router, as: Router.self)
/// Register middleware
var middlewares = MiddlewareConfig() // Create _empty_ middleware config
/// middlewares.use(FileMiddleware.self) // Serves files from `Public/` directory
middlewares.use(ErrorMiddleware.self) // Catches errors and converts to HTTP response
services.register(middlewares)
// Configure a database
let dbConfig: PostgreSQLDatabaseConfig
if let url = Environment.get("DATABASE_URL"), let psqlConfig = PostgreSQLDatabaseConfig(url: url) {
dbConfig = psqlConfig
} else {
dbConfig = PostgreSQLDatabaseConfig(hostname: "localhost", port: 5432, username: "admin", database: "development", password: nil)
}
let postgresql = try PostgreSQLDatabase(config: dbConfig)
/// Register the configured SQLite database to the database config.
var databases = DatabasesConfig()
databases.add(database: postgresql, as: .psql)
services.register(databases)
/// Configure migrations
var migrations = MigrationConfig()
migrations.add(model: Visit.self, database: .psql)
services.register(migrations)
}
Package.swift:
// swift-tools-version:4.0
import PackageDescription
let package = Package(
name: "SubwayNyc",
dependencies: [
// 💧 A server-side Swift web framework.
.package(url: "https://github.com/vapor/vapor.git", from: "3.0.0"),
// 🔵 Swift ORM (queries, models, relations, etc) built on PostgreSQL.
.package(url: "https://github.com/vapor/fluent-postgresql.git", from: "1.0.0"),
.package(url: "https://github.com/vapor/sql.git", from: "2.1.0")
],
targets: [
.target(name: "App", dependencies: ["FluentPostgreSQL", "Vapor"]),
.target(name: "Run", dependencies: ["App"]),
.testTarget(name: "AppTests", dependencies: ["App"])
]
)
How can I get PostgreSQL hooked up to my Vapor 3 app on Heroku?

For Heroku we need unverifiedTLS transport.
https://api.vapor.codes/postgresql/latest/PostgreSQL/Classes/PostgreSQLConnection/TransportConfig.html
let pgURL = Environment.get("DATABASE_URL") ?? "postgres://user:password#host:port/database"
let pgConfig = PostgreSQLDatabaseConfig(url: pgURL, transport: PostgreSQLConnection.TransportConfig.unverifiedTLS)!
:D

The original error is the key here, in particular: SSL off.
This error is thrown by Heroku Postgres when the client is attempting to connect without SSL. Not familiar with Vapor myself, but a quick look around suggests that configure.swift is where you can make configuration changes. Once you enable SSL from the client, you should be able to connect to this database instance without issue.

Related

Wasm-rust and postgres

is there a way to get data from a database in postgres using wasm?. I'd tried to get it using a library in rust but I got some errore when I build the package using "wasm-pack building--target web". The idea is to build a function in lib.rs file that return data from a db. I have the below code inside lib.rs:
use postgres::{Client, Error, NoTls};
use wasm_bindgen::prelude::*;
...
struct Author {
_id: i32,
name: String,
}
#[wasm_bindgen]
pub fn select_name(name: &String) -> Result<(), Error> {
let mut client = Client::connect("postgresql://user:1234#localhost:5432/db", NoTls)?;
for row in client.query(
"SELECT id, name FROM author WHERE name = $1",
&[&name],
)? {
let author = Author {
_id: row.get(0),
name: row.get(1),
};
println!(
"Select_Name => Author {} :",
author.name
);
}
Ok(())
}
but I get some errors:
error[E0432]: unresolved import `crate::sys::IoSourceState`
error[E0432]: unresolved import `crate::sys`
...
It is not possible directly (as Njuguna Mureithi rightly wrote) but it can be circumvented.
We can use the project: https://github.com/PostgREST/postgrest
and expose the API to our sql server.
We download the latest version of postgrest
https://github.com/PostgREST/postgrest/releases/tag/v9.0.0
in case of Linux we unpack
tar -xf postgrest-v9.0.0-linux-static-x64.tar.xz
then run help
./postgrest -h
create a configuration file for ./postgrest
postgrest -e > cfg.psqlrest
change the user and password for the database in the configuration file.
e.g. with
db-uri = "postgres://user:pasword#localhost:5432/dbname"
to your dbname database access user:pasword configuration
db-uri = "postgres://postgres:zaqwsxc#localhost:5432/dbname"
and run the server which will issue the api to our postgres
./postgrest cfg.psqlrest
The address http://localhost:3000 will start accessing the database dbname, which must be created in the database server beforehand.
Here is a description of the libraries needed to call the query using the API.
https://rustwasm.github.io/wasm-bindgen/examples/fetch.html
examples of API
https://postgrest.org/en/stable/api.html

How can I help Vapor successfully SSL-handshake my PostgreSQL server?

I'm using Vapor on a Ubuntu server to connect to my DigitalOcean-managed PostgreSQL database.
From the command-line, running the following works fine:
psql postgresql://user:password#host:port/dbname?sslmode=require
But running the equivalent with the following code gives me:
Fatal error: Error raised at top level: NIOOpenSSL.NIOOpenSSLError.handshakeFailed(NIOOpenSSL.OpenSSLError.sslError([Error: 337047686 error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed])): file /home/buildnode/jenkins/workspace/oss-swift-5.1-package-linux-ubuntu-18_04/swift/stdlib/public/core/ErrorType.swift, line 200
Here is the code:
let postgres = PostgreSQLDatabase(config: PostgreSQLDatabaseConfig(
hostname: Environment.get("POSTGRESQL_HOSTNAME")!,
port: Int(Environment.get("POSTGRESQL_PORT")!)!,
username: Environment.get("POSTGRESQL_USERNAME")!,
database: Environment.get("POSTGRESQL_DATABASE")!,
password: Environment.get("POSTGRESQL_PASSWORD")!,
transport: .standardTLS
))
Switching the transport argument to .unverifiedTLS works.
I need help to let Vapor work out the SSL connection fine, but I have no idea where to start.
I recently got this working with Vapor 4 and MySQL on Digital Ocean, I suspect the same will work for PostgreSQL. The main bit was configuring Vapor to trust Digital Ocean's certificate.
Download the CA certificate from the managed database dashboard on Digital Ocean (the connection details section).
Configure the database tlsConfigurataion to trust that certificate. Here's an example of what that could look like:
import NIOSSL
public func configure(_ app: Application) throws {
app.databases.use(.postgres(
hostname: Environment.get("DATABASE_HOST") ?? "localhost",
port: Environment.get("DATABASE_PORT").flatMap(Int.init(_:)) ?? PostgresConfiguration.ianaPortNumber,
username: Environment.get("DATABASE_USERNAME") ?? "vapor_username",
password: Environment.get("DATABASE_PASSWORD") ?? "vapor_password",
database: Environment.get("DATABASE_NAME") ?? "vapor_database",
tlsConfiguration: try makeTlsConfiguration()
), as: .psql)
// ...
}
private func makeTlsConfiguration() throws -> TLSConfiguration {
var tlsConfiguration = TLSConfiguration.makeClientConfiguration()
if let certPath = Environment.get("DATABASE_SSL_CERT_PATH") {
tlsConfiguration.trustRoots = NIOSSLTrustRoots.certificates(
try NIOSSLCertificate.fromPEMFile(certPath)
)
}
return tlsConfiguration
}
In this example, I use the DATABASE_SSL_CERT_PATH environment variable to set the path of the downloaded ca-certificate.crt file.

Deleting test database in Vapor 3

I want to write some integration tests for Vapor 3 server and I need to have clean Postgre database each time I run my tests. How can I achieve this? It seems migrations isn't the right way to go as they've been running once if database doesn't exist yet.
Have a look at https://github.com/raywenderlich/vapor-til/tree/master/Tests
This requires a DB to be running before you run the tests, but it reverts all migrations at the start of each test run, which gives you a clean DB each time. (Specifically here)
There's also a docker-compose.yml in the root directory for spinning up a completely isolated test environment on Linux
I've found a solution that is less resource-intensive, then reverting all migrations every time.
RSpec has a configuration (use_transactional_fixtures) that allows wrapping every test in an SQL transaction. When testing is over it will rollback the transaction and in consequence revert all the changes that happened during testing. Relevant documentation is here.
We can implement a similar solution in Vapor. My example test looks like this.
final class VaporTests: XCTestCase {
var app: Application!
override func setUp() {
super.setUp()
app = try! Application.buildForTesting()
let conn = try! app.requestPooledConnection(to: .psql).wait()
try! conn.simpleQuery("BEGIN TRANSACTION").wait()
try! app.releasePooledConnection(conn, to: .psql)
}
override func tearDown() {
let conn = try! app.requestPooledConnection(to: .psql).wait()
try! conn.simpleQuery("ROLLBACK").wait()
try! app.releasePooledConnection(conn, to: .psql)
super.tearDown()
}
func testExample() throws {
let request = HTTPRequest(method: .GET, url: "my/endpoint/example")
let wrapper = Request(http: request, using: app)
let response = try ExampleController().example(wrapper).wait()
XCTAssertEqual(response, .ok)
}
}
To make sure that I don't encounter issues with concurrency I'm limiting database pool to 1 connection in the test application.
func configure(_ config: inout Config, _ env: inout Environment, _ services: inout Services) throws {
// ... other configurations
let poolConfig = DatabaseConnectionPoolConfig(maxConnections: 1)
services.register(poolConfig)
}
Many thanks to Jakub Jatczak for helping me to find out how this happens in Rails.
Quite late to the party but following way also does the revert and migrate command work. This code does the similar commands as answer given by #0xTim. But I have made use of Console.framework:
Mostly we use a configure.swift file like below:
import FluentPostgreSQL
import Vapor
public func configure(_ config: inout Config, _ env: inout Environment, _ services: inout Services) throws {
// Register providers first
try services.register(FluentPostgreSQLProvider())
...
/// Configure commands
var commandConfig = CommandConfig.default()
commandConfig.useFluentCommands()
services.register(commandConfig)
...
/// Configure migrations
services.register { container -> MigrationConfig in
var migrationConfig = MigrationConfig()
try migrate(migrations: &migrationConfig)
return migrationConfig
}
}
Quite late to the party but following code does execute revert and migrate
commands: (I am using Quick and Nimble so beforeSuite. Commented code is there because unless you use above configure.swift you can just uncomment the code and make use of CommandConfig directly.)
import Quick
import Vapor
import Console
import FluentPostgreSQL
...
configuration.beforeSuite {
let console: Console = Terminal()
// var commandConfig = CommandConfig()
// commandConfig.use(RevertCommand(), as: "revert")
// commandConfig.use(MigrateCommand(), as: "migrate")
var config = Config.default()
var env = Environment.testing
var services = Services.default()
do {
// try App.configure(&config, &env, &services)
let container = try Application(config: config, environment: env, services: services)
let commandConfig = try container.make(CommandConfig.self)
let commands = try commandConfig.resolve(for: container).group()
var input = CommandInput(arguments: ["vapor","revert","--all", "-y"])
try console.run(commands, input: &input, on: container).wait()
input = CommandInput(arguments: ["vapor","migrate","-y"])
try console.run(commands, input: &input, on: container).wait()
} catch let error {
console.error(error.localizedDescription)
exit(1)
}
}
For thoses who are seeking another approach that doesnt involve registering new migrations ( and, to me, adding more code complexity ) you can use a Pre-Action script for tests target ( ⌘ + < )
By using a bash script you can create a brand new postgresql database that will be used to build the project for tests only :
# Variables
export IS_TEST=true
export DB_USERNAME="`whoami`"
export DB_DBNAME="BARTENDER_TEST_DB"
#Creating dedicated Postgres DB
echo "deleting & recreating $DB_DBNAME for user $DB_USERNAME"
psql postgres<< EOF
DROP DATABASE "$DB_DBNAME";
CREATE DATABASE "$DB_DBNAME";
\list
EOF
Then in configure.swift file you create a PostgreSQLDatabaseConfig that matches the newly created database
if let _ = Environment.get("IS_TEST") { // IS_TEST is defined only in Pre-Action script
guard let username = Environment.get("DB_USERNAME") else {
fatalError("Failed to create PostgresConfig - DB_USERNAME in Environment variables")
}
guard let databasename = Environment.get("DB_DBNAME") else {
fatalError("Failed to create PostgresConfig - DB_DBNAME in Environment variables")
}
postgresqlConfig = PostgreSQLDatabaseConfig(
hostname: "127.0.0.1",
port: 5432,
username: username,
database: databasename,
password: nil
)
}
else { /* your other config here */ }
let database = PostgreSQLDatabase(config: postgresqlConfig)
...
The big advantage I found in this it that I can even trigger a vapor build and vapor run from another project that will create a brand new Webservice environment for my continuous integration testings, just by inserting the correct environment variables
Simplest way I found was to add PostgresKit to the test target and use it to setup a connection to call my "clean-up" queries.
#testable import App
import Vapor
import XCTest
import PostgresKit
final class UserTests: XCTestCase {
var pools: EventLoopGroupConnectionPool<PostgresConnectionSource>!
var postgresDb: PostgresDatabase!
var eventLoopGroup: EventLoopGroup!
override func setUp() {
let configuration = PostgresConfiguration(
hostname: "localhost",
username: "postgres",
password: "password",
database: "db_name"
)
eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: 2)
pools = EventLoopGroupConnectionPool(
source: PostgresConnectionSource(configuration: configuration),
on: eventLoopGroup
)
postgresDb = pools.database(logger: Logger.init(label: "TestLogger"))
}
override func tearDown() {
let _ = try! postgresDb.query("DELETE FROM \(User.schema)").wait()
try! pools.syncShutdownGracefully()
try! eventLoopGroup.syncShutdownGracefully()
}
func testUploadUser() throws {
let app = Application(.testing)
defer { app.shutdown() }
try configure(app)
try app.testable(method: .running).test(.POST, "api/users", beforeRequest: { req in
try req.content.encode(["firstName" : "Dwide", "lastName" : "Shrewd"])
}, afterResponse: { res in
XCTAssertEqual(res.status, .ok)
let user = try res.content.decode(User.self)
XCTAssertEqual(user, User(id: user.id, firstName: "Dwide", lastName: "Shrewd"))
})
}
}
And this is my Package.swift
// swift-tools-version:5.2
import PackageDescription
let package = Package(
name: "MyVaporProject",
platforms: [
.macOS(.v10_15)
],
dependencies: [
// 💧 A server-side Swift web framework.
.package(url: "https://github.com/vapor/vapor.git", from: "4.0.0"),
.package(url: "https://github.com/vapor/fluent.git", from: "4.0.0"),
.package(url: "https://github.com/vapor/fluent-postgres-driver.git", from: "2.0.0"),
.package(url: "https://github.com/vapor/postgres-kit.git", from: "2.0.0")
],
targets: [
.target(
name: "App",
dependencies: [
.product(name: "Fluent", package: "fluent"),
.product(name: "FluentPostgresDriver", package: "fluent-postgres-driver"),
.product(name: "Vapor", package: "vapor")
],
swiftSettings: [
// Enable better optimizations when building in Release configuration. Despite the use of
// the `.unsafeFlags` construct required by SwiftPM, this flag is recommended for Release
// builds. See <https://github.com/swift-server/guides#building-for-production> for details.
.unsafeFlags(["-cross-module-optimization"], .when(configuration: .release))
]
),
.target(name: "Run", dependencies: [.target(name: "App")]),
.testTarget(
name: "AppTests",
dependencies: [
.target(name: "App"),
.product(name: "XCTVapor", package: "vapor"),
.product(name: "PostgresKit", package: "postgres-kit")
]
)
]
)
As in previous answers, this requires a stood-up Postgres database, already migrated, ready to take connections before the tests start.

Xcode error when building Vapor 3 framework

I had this error when trying to build from XCode , albeit it builds from terminal
clang: error: no such file or directory: '/Users/++/Desktop/vapor/til/.build/checkouts/sqlite.git--8232814251736334455/Sources/CSQLite/sqlite3.c'
clang: error: no input files
I'm using pgsql and this is my configure.swift
// 1
import FluentPostgreSQL
import Vapor
public func configure(
_ config: inout Config,
_ env: inout Environment,
_ services: inout Services
) throws {
// 2
try services.register(FluentPostgreSQLProvider())
let router = EngineRouter.default()
try routes(router)
services.register(router, as: Router.self)
var middlewares = MiddlewareConfig()
middlewares.use(ErrorMiddleware.self)
services.register(middlewares)
// 1
var databases = DatabasesConfig()
// 2
let hostname = Environment.get("DATABASE_HOSTNAME")
?? "localhost"
let username = Environment.get("DATABASE_USER") ?? "vapor"
let databaseName = Environment.get("DATABASE_DB") ?? "vapor"
let password = Environment.get("DATABASE_PASSWORD")
?? "password"
// 3
let databaseConfig = PostgreSQLDatabaseConfig(
hostname: hname,
username: username,
database: databasesName,
password: password)
// 4
let database = PostgreSQLDatabase(config: databaseConfig)
// 5
databases.add(database: database, as: .psql)
// 6
services.register(databases)
var migrations = MigrationConfig()
// 4
migrations.add(model: Acronym.self, database: .psql)
services.register(migrations)
}
I removed XCode and reinstalled it but no effect
After making sure that you removed any traces of the fluent-sqlite package from your Package.swift file (since your code suggests using PostgreSQL now), please try deleting your .build folder, along with a few regeneratable items with the following commands:
rm -rf .build
rm -rf til.xcodeproj # or however is your Xcode project file called
rm Package.resolved
Then you can re-generate your Xcode project with vapor xcode and/or you can try building from command line again.

Connecting to Oracle Database Using Server Side Swift

Is it possible yet to connect to an Oracle Database using Swift (raw or a Swift framework) on Linux? What I have done is tried to build a Docker VM, install the Oracle binaries, add the OCILIB package and then connect [tried] using a package called SwiftOracle, which seems unsupported (lots of build issues) and just exposes the OCILIB C code to Swift using a module map and wrapper.
I tried this using the Kitura framework and none of this seemed to work - Xcode can't compile because it isn't unable to either find the C library or create the module.
Here are the steps that I have tried to no avail:
Build Docker VM (which includes Oracle binaries): https://github.com/wnameless/docker-oracle-xe-11g
Download and install OCILIB: https://github.com/vrogier/ocilib
Add SwiftOracle package, fix build issues and try to build.
you can use the following method to connect to oracle database: ( This was possible by the help of vapor community.)
-----to make oracle driver work, I have Tied this method in Ubuntu-------------
-- oracle client need to be installed such that the header and library path can be defined, you can get these from oracle website.
oracle-instantclinet*-basic-*.rpm
oracle-instantclinet*-devel-*.rpm
oracle-instantclinet*-sqlplus-*.rpm
--install thus downloaded package using following command
sudo alien -i oracle-instantclinet*-basic-*.rpm
sudo alien -i oracle-instantclinet*-devel-*.rpm
sudo alien -i oracle-instantclinet*-sqlplus-*.rpm
-- Install libaio1 in ubuntu
sudo apt install libaio1
-- this path should be in ~/.bashrc
#oracle home and library path
export ORACLE_HOME=/usr/lib/oracle/12.2/client64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/oracle/12.2/client64/lib:/usr/local/lib
--download OCILIB library from Github
git clone https://github.com/vrogier/ocilib.git ( or download the latest version / tested on ocilib 4.5.2 )
-- extract ocilib file cd to ocilib folder, configure make and make install
tar -zxf ocilib-4.5.2-gnu.tar.gz
cd ocilib-4.5.2
./configure --with-oracle-headers-path=/usr/include/oracle/12.2/client64/ --with-oracle-lib-path=/usr/lib/oracle/12.2/client64/lib CFLAGS="-O2 -m64"
make
sudo make install
-- use this configuration if you need to deal with unicodes, generally you don't need this
./configure --with-oracle-headers-path=/usr/include/oracle/12.2/client64/ --with-oracle-lib-path=/usr/lib/oracle/12.2/client64/lib --with-oracle-charset=wide CFLAGS="-O2 -m64"
-- The above method installs OCILIB in your machine.
-- To user OCILIB library in your Vapor project Include the following in you Package.swift file
// swift-tools-version:4.0
import PackageDescription
let package = Package(
name: "myAPIProject",
dependencies: [
// 💧 A server-side Swift web framework.
.package(url: "https://github.com/vapor/vapor.git", from: "3.0.0"),
// 🔵 Swift ORM (queries, models, relations, etc) built on SQLite 3.
.package(url: "https://github.com/vapor/fluent-sqlite.git", from: "3.0.0"),
//Oracle wrapper for swift
.package(url: "https://github.com/h1257977/SwiftOracle.git", from: "0.1.7")
],
targets: [
.target(name: "App", dependencies: ["FluentSQLite","SwiftOracle", "Vapor"]),
.target(name: "Run", dependencies: ["App"]),
.testTarget(name: "AppTests", dependencies: ["App"])
]
)
-- In Routes.swift , include the following:
import Vapor
import SwiftOracle
let service = OracleService(host: "192.168.1.12", port:"1521", service: "orcl")
let b = Connection(service: service, user:"test", pwd:"oracle")
final class VReq: Content {
var name: String?
var age: String?
init (NAME: String, AGE: String) {
self.name = NAME
self.age = AGE
}
final class VMdata {
func getData() throws -> [VReq] {
try! b.open()
b.autocommit = true
let cursor = try! b.cursor()
try! cursor.execute("select * from userlist")
//iterates each row in the cursor and maps only the values (keys are unique) from the dictionary of each rows, if its nil it will replace with "null"
var items = cursor.map { row in row.dict.mapValues { "\($0 ?? "NULL")" }} // output as [[String:String]]
//takes each dictionary in the items array and returns a VReq
let result = items.map { dict in VReq(NAME: dict["NAME"] ?? "NULL", ADDRESS: dict["ADDRESS"] ?? "NULL", USER_AGE: dict["USER_AGE"] ?? "NULL")}
return result
}
}
public func routes(_ router: Router) throws {
router.get("test") { req -> [VReq] in
let val = VMdata()
let vdata = try! val.getData()
return vdata
}
}
--To run Vapor 3 you need to link the library files
swift build -Xlinker -L/usr/local/lib && ./.build/x86_64-unknown-linux/debug/Run --hostname 0.0.0.0
-- To display unicode characters from any database column you may have to set NLS_LANG in the server hosting Vapor application.
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8