How to connect to Postgres with Kotlin/Ktor with plugin start - postgresql

I have decided to start learning Kotlin/Ktor. I have created a project in Intellij and selected the Postgres plugin. I'm reading the instructions of how to set it up from the file it created:
/**
* Makes a connection to a Postgres database.
*
* In order to connect to your running Postgres process,
* please specify the following parameters in your configuration file:
* postgres.url -- Url of your running database process.
* postgres.user -- Username for database connection
* postgres.password -- Password for database connection
**/
But I cant figure out where to set the username or password (or url). It refers to a configuration file but I have no idea where that is.
Here are all my file structure if it helps.
All files part 1
All files part 2
All files part 3
I have tried the Navigate - > Search everywhere" option for keywords such as password and postgres but found nothing that looks like where I can configure the password or username.
Any help appreciated.

There is supposed to be an application.conf file in your resources folder, if it is not there, that means you chose your configurations to be in code when you created the project the first time.
Look at the following image: Ktor Project creation, in the bottom it asks where to set up your configurations, choose HOCON file instead of Code. By this way, you will have application.conf file in your resources folder.
In that file, add the following code snippet to configure Postgres:
postgres {
url = "Your URL"
user = "Your USERNAME"
password = "Your PASSWORD"
}
However, for security purposes, you should not write your database credentials as plaintext in your configuration file. For your production, consider using environment variables and configure Postgres like this:
postgres {
url = ${pg_url}
user = ${pg_username}
password = ${pg_password}
}
where pg_url, pg_username, and pg_password are the defined environment variables in your system.
Finally, to access the url, user, and password in your code, Do it like this:
val url = environment.config.property("postgres.url").getString()
val user = environment.config.property("postgres.user").getString()
val password = environment.config.property("postgres.password").getString()
For more info about configuring your project in the file, refer to the documentation. Hope my answer helps.

Related

Prisma.Schema will not Update

I am having the dumbest issue ever with my schema file. It has nothing to do with the Models or any properties or actual information, but rather the file is just refusing to save, or update the saved changes.
The prisma.schema file worked perfectly and we had to make a couple of minor changes, literally just changing the names of some of the models' values. For some reason, this made my file unable to read the contents of my .env file... fine, I only needed it for the URL anyway. This being said I then changed the following code from this...
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
}
to this...
datasource db {
provider = "postgresql"
url = "postgresql://nick:Abc!123#localhost:5432/tom?schema=public"
}
But whenever I run any command that will involve the schema file in some way (node server.js, npx prisma migrate dev etc) it will give me the following error...
PrismaClientInitializationError: error: Environment variable not found: DATABASE_URL.
--> schema.prisma:8
|
7 | provider = "postgresql"
8 | url = env("DATABASE_URL")
|
which is clearly just not what I have in my code anymore. I've never seen anything like this, I've closed vscode, I've restarted my machine, any ideas?
I believe you need to run prisma generate command after you make any changes to the schema.prisma file.

How do I store a SQL connection password in Airflow cfg?

In the .cfg file, I connected sql alchemy to Postgres with user: airflow_admin and password: pass:
sql_alchemy_conn = postgresql+psycopg2://airflow_admin:pass#localhost:5432/airflow_backend
How do I anonymize this so that the password doesn't show? Do I create a .env file and store the password as a variable and then reference that variable in .cfg conn string?
I read the following but an example would be helpful: https://airflow.readthedocs.io/en/stable/howto/set-config.html
There are several way to do it:
1. change the configuration file permision to read only by airflow account
2. used airflow by docker mode, and encrpyt the configurationfile by docker secret create and map it.
THe mode 1 is easy and convinent. The mode 2 is flexible and can be used in production environment.
Good luck.
If you think the answer is good, pls up vote

How do you include postgresql.conf on docker container when using org.testcontainers

Is it possible to give postgresql testcontainer a custom postgresql.conf file via config?
I have included maven dependency
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<version>1.10.6</version>
</dependency>
And using 'Database containers launched via JDBC URL scheme' for DB url
As such have the setting in my Spring Boot app as:
datasource:
url: jdbc:tc:postgresql:10-alpine:///databasename
driver-class-name: org.testcontainers.jdbc.ContainerDatabaseDriver
I need to have a custom setting in postgresql.conf.
Is there a way of pushing postgresql.conf to the docker container started by testcontainers?
EDIT 1
Thanks #vilkg I did know about the TC_INITSCRIPT script option and SET function however:
I am wanting a custom setting such as my.key
ALTER system does not work for your own settings eg: ALTER SYSTEM SET my.key = 'jehe'; get error Could not execute the SQL command.
Message returned: `ERROR: unrecognized configuration parameter "my.key"
I had previously try SET and ALTER DATABASE as below
SET my.key = 'new value 8'; -- sets for current session
ALTER DATABASE test SET my.key = 'new value 8'; -- sets for subsequent sessions
select current_setting('my.key');
PROBLEM IS
when testcontainer starts postgres container and I pass it an init script to run
url: jdbc:tc:postgresql:10-alpine:///databasename?TC_INITSCRIPT=init_pg.sql
and I can include the above SQL its happy..
I know setting of that secret.key is working correctly in this script because it will fail on the line select current_setting('my.key'); if other two are commented out
I also know that runing it against db name test is correct eg: 'ALTER DATABASE test' because if I use a different name it fails
Testcontainers automatically connects the app to db named test
So with all of the above I believe the DB is setup nicely and all should be good
BUT
When I use 'current_setting('my.key')' within application code it fails
If you want to continue launching Postgres container using JDBC URL scheme, test containers can execute init script for you. The script must be on the classpath, referenced as follows:
jdbc:tc:postgis:9.6://hostname/databasename?TC_INITSCRIPT=somepath/init.sql
ALTER SYSTEM SET command was introduced in postgres 9.4, so you could use it in your init script.
Another option would be to start postgres container using database containers objects and use withCopyFileToContainer() method. Example:
JdbcDatabaseContainer<?> postgisContainer = new PostgisContainerProvider()
.newInstance()
.withDatabaseName( POSTGRES_DATABASE_NAME )
.withUsername( POSTGRES_CREDENTIALS )
.withPassword( POSTGRES_CREDENTIALS )
.withCopyFileToContainer(MountableFile.forClasspathResource("postgresql.conf"), "/var/lib/postgresql/data"));
EDIT:
If none of the above works, you can reconfigure Postgres command and pass your custom configuration keys. All you need is extending PostgreSQLContainer and overriding configure() method.
#Override
protected void configure()
{
setCommand( "postgres -c $your_config_key=$your_config_value" );
}
we have to create our database and the connection’s user. This is done by using environment variables from the Docker image. To change postgres.conf we can use DockerFIle where we will replace the existing postgres.conf by the new configuration.
#ClassRule
public static GenericContainer postgresql= new GenericContainer(
new ImageFromDockerfile("postgresql:10-alpine")
.withDockerfileFromBuilder(dockerfileBuilder -> {
dockerfileBuilder.from("myPostgresql:10-alpine")
// root password is mandatory
.env("PG_ROOT_PASSWORD", "root_password")
.env("PG_DATABASE", "postgres")
.env("PG_USER", "postgres")
.env("PG_PASSWORD", "postgres")
})
Next, we have to create a database schema and populate the database. From the image documentation, the directory /docker-entrypoint-initdb.d is scanned at startup and all files with .sh, .sql et .sql.gz extension are executed. So, we just have to put our files schema.sql and data.sql in this directory. Somme parameteres can be set by sql requesting running database
.withFileFromClasspath("a_schema.sql", "db/pg/schema.sql")
.withFileFromClasspath("b_data.sql", "db/pg/data.sql"))

How to connect to postgresql, without specifying plain-text password, with libpq's pgpass?

Postgres' C library libpq documentation talks about a more secure way to connect to a DB without specifying password in source code.
I was not able to find any examples of how to do it. How to make my Postgre Server use this file? Please help.
You don't import it into your Python program. The point of .pgpass is that it is a regular file subject to the system's file permissions, and the libpq driver which libraries such as psycopg2 use to connect to Postgres will look to this file for the password instead of requiring the password to be in the source code or prompting for it.
Also, this is not a server-side file, but a client-side one. So, on a *nix box, you would have a ~/.pgpass file containing the credentials for the various connections you want to be able to make.
Edit in response to comment from OP:
Two main things need to happen in order for psycopg2 to correctly authenticate via .pgpass:
Do not specify a password in the string passed to psycopg2.connect
Make sure the correct entry is added to the .pgpass file for the user who will be connecting via psycopg2.
For example, to make this work for all databases for a particular user on localhost port 5432, you would add the following line to that user's .pgpass file:
localhost:5432:*:<username>:<password>
And then the connect call would be of this form:
conn = psycopg2.connect("host=localhost dbname=<dbname> user=<username>")
The underlying libpq driver that psycopg2 uses will then utilize the .pgpass file to get the password.
just adding to #khampson's answer, I could only get this to work (in windows) if I added the PGPASSFILE environment variable (and subsequently restart pycharm) even though my pgpass file is in the default location (%APPDATA%\postgresql\pgpass.conf).
Well. You asked for .pgpass but...
There is also "The Connection Service File"
By default, the per-user service file is named ~/.pg_service.conf.
A different file name can be specified by setting the environment variable PGSERVICEFILE
Look at
https://www.postgresql.org/docs/15/libpq-pgservice.html
Format is ".INI". Parameters are connection parameters.
Example:
# this is a comment in my service file
[a_service]
host=dbserver.loc
port=5432
dbname=thedbname
user=thebest
password=bho
application_name=ifyoulike
# Here is another service
[another_service]
host=etc
port=etc
Access via psql:
psql service=a_service
Access via psycopg2:
db = psycopg2.connect('service=a_service')
Access via sqlalchemy:
eng = sqlalchemy.create_engine('postgresql:///?service=another_service')
#!/usr/bin/python
import psycopg2
import sys
import pprint
def main():
conn_string = "host='127.0.0.1' dbname='yourdatabsename' user='yourusername' password='yourpassword'"
print "Connecting to database\n ->%s" % (conn_string)
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
cursor.execute("SELECT * FROM tablename")
records = cursor.fetchall()
pprint.pprint(records)
if __name__ == "__main__":
main()

Moving ejabberd default DB to MySQL shows authentication failure

I am trying to setup an ejabberd server on my Amazon EC2 Ubuntu instance.
With the default DB Provided by ejabberd, I can easily setup my connection. But I need to replace the mnesia DB with MySQL. I found some tutorials over the internet. From those tutorial I found out a solution. I will explain it as step by step.
I am using ejabberd 2.1.11. I made the following changes on ejabberd.cfg file
Commented the following line :
{auth_method, internal}
Uncommented this:
{auth_method, odbc}
Configured my MySQL DB
{odbc_server, {mysql, "localhost", "students", "root", ""}} // No Password set
Change mod_last to mod_last_odbc
Change mod_offline to mod_offline_odbc
Change mod_roster to mod_roster_odbc
Change mod_private to mod_private_odbc
Change mod_privacy to mod_privacy_odbc
Change mod_pubsub to mod_pubsub_odbc
Change mod_vcard to mod_vcard_odbc
Then I installed ejabberd-mysql driver from the following link
http://stefan-strigler.de/2009/01/14/ejabberd-mysql-drivers-for-debian-and-ubuntu/
After making all these changes I restarted my ejabberd server.
Then I tried to login to my ejabberd server. It shows me the login prompt.
After entering the credentials it takes a lot time and then displays authentication failed.
Any help on the topic is appreciated.
Let's dig into problem
Your setup is working that means your config file is fine. But then
Why does auth fails ?
What schema you have in your students database ?
If you have a proper schema installed then does the user present in ur db's users table?
Have you also updated conf/odbc.ini with proper mysql details.
Even if both the conditions meet then I'll advice you to set mysql password and try again.
Let me know if that helps or not.
Update :-
update your config with {loglevel, 5}
then hit the login and tail all the log files.
odbc.ini
1 [ejabberd]
2 Driver = MySQL
3 DATABASE = students
4 PWD =
5 SERVER = localhost
6 SOCKET = /tmp/mysql.sock
7 UID = root
One Major basic part that one can easily miss is that data which was previously stored in mnesia database will no longer will be available for your new configuration so again you have to create one admin user like this to access your admin account.
./ejabberdctl register admin "password"