I am fairly new to Eclipse Ditto and have just started using it for my project.
I am trying to connect Cloud hosted mongodb instance to ditto.
Following the documentation I know that I need to add some variables and pass them to docker-compose. The problem is that I do not know what should be the values of these variables as there are no examples.
Are all these variables necessary or will just the URI work?
This is my current .env file config
MONGO_DB_URI=mongodb+srv://username:pass#IP
MONGO_DB_READ_PREFERENCE=primary
MONGO_DB_WRITE_CONCERN=majority
The command I am using to start ditto is
docker-compose --env-file .env up
I have removed mongodb service from docker-compose.yml
Nice to hear that you started using Ditto in your project.
You need to set the following env variables to connect to your Cloud hosted MongoDB.
MONGO_DB_URI: Connection string to MongoDB
For more detailes see: https://docs.mongodb.com/manual/reference/connection-string/
If you have a ReplicaSet your MongoDB URI should look like this: mongodb://[username:password#]mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myRepl
I assume you also need to enable SSL to connect to your MongoDB.
To do so set this env var.
MONGO_DB_SSL_ENABLED: true
If you want to use a specific Ditto version you can set the following env var
DITTO_VERSION= e.g. 2.1.0-M3
If you use .env as file name you can start Ditto with:
docker-compose up
The other options for pool size, read preference and write concern aren't necessary as there are default values in place.
Related
So basically I have my cloud hasura with existing schema, relations tables etc... and i want to offline it using docker and try using metadata export and import that seems not working how can I do it or is there other ways to do it?
this is the docker i want to offline
this is my cloud i want to get the schemas or metadata
OR MAYBE I JUST MANUALLY RECREATE THE TABLES AND RELATIONS??
When using the steps outlined in the Hasura Quickstart with Docker page then the following steps would help get all the table definitions, relationships etc., setup on the local instance just like it is set up on hasura cloud instance.
Migrate all the database schema and metadata using the steps mentioned in Setting up migrations
Since you want to migrate from hasura cloud use the URL of the cloud instance in step 2. Perform steps 3-6 as described in the above link.
Bring up the local docker environment. Ideally edit the docker-compose.yaml file to set HASURA_GRAPHQL_ENABLE_CONSOLE: "false" before running docker-compose up -d.
Resume the process of applying migrations from step 7. Use the endpoint from local instance. For example,
$ hasura metadata apply --endpoint http://localhost:8080
$ hasura migrate apply --endpoint http://localhost:8080
I deploy a mongodb in a container. and I also successfully connected to mongodb when my backend runs in the local environment. However, when I tried to move the backend to a container as well, it cannot connect mongodb anymore. The connection string remains the same but it failed.
The error shows that mongodb cannot find the user. However, in my local environment, the same connection string works.
After my googling, most guys said I need to change authSource. I have tried to change authSource to admin, it doesn't work even in my local environment. And I am sure the user exists in my admin database. not in api_dev_db. One wired thing is I can connect to mongodb in my local environment but not work in docker.
Could you kindly help me out?
Here I post some pictures.
These are the env variables I defined in my local environment and they work
This is how I deploy backend service
This is my backend dockerfile
This is the error when I deploy backend in container
This is how I initial mongodb
The problem is I need to remove '' in env variable, which is MONGO_URL=mongodb://api_user:api1234#mongodb:27017/api_dev_db?authSource=api_dev_db
I would like to configure Conjur with ssl_mode=verify-full to connect to my postgres database.
I use the Docker image cyberark/conjur:1.8.1#sha256:01d601d763edf1d98ca81dda36d4744e78244a4836cfa804570a47da5fd50405
Adding it as a string parameter (like that DATABASE_URL=postgres://conjur:$CONJURDBPASSWORD#postgres-conjur:5432/conjurdb?sslmode=verify-full) does not seem to work.
The db library used by Conjur is Sequel and it supports it https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-postgres
How can I achieve that without altering the Conjur code ? Ideally, via ENV or mounting a config file.
A project like Gemstash uses the same library and gives a way to achieve that easily. with a config.yml file containing (for instance):
:db_adapter: postgres
:db_url: postgres://{{ .Env.DB_HOST }}/gemstashdb?user=gemstash&password={{ .Env.DB_PASSWD }}
:db_connection_options:
:connect_timeout: 10
:read_timeout: 5
:timeout: 30
:sslmode: 'verify-full'
:sslrootcert: '{{ .Env.HOME }}/.ssl/root.crt'
I didn't find anything similar in Conjur.
Any of you with experience with PostgREST and Cloud SQL ?
I have my SQL instance ready with open access (0.0.0.0/0) and I can access it with local PostGREST using the Cloud proxy app.
Now I want to run Postgrest from an instance of the same project but
I can't find an URI format for Postgrest that supports Cloud SQL format, as
Google SQL Cloud uses only unix sockets like /cloudsql/INSTANCE_CONNECTION_NAME
Config 1
db-uri = "postgres://postgres:password#/unix(/cloudsql/INSTANCE_CONNECTION_NAME)/mydatabase"
db-schema = "api"
jwt-secret = "OOcJ7VoSY1mXqod4MKtb9WCCwt9erJkRQ2tzYmLb4Xe="
db-anon-role = "web_anon"
server-port=3000
Returns {"details":"could not translate host name \"unix(\" to address: Unknown host\n","code":"","message":"Database connection error"}
Config 2
db-uri = "postgres://postgres:password#/mydatabase?unix_socket=/cloudsql/INSTANCE_CONNECTION_NAME"
db-schema = "api"
jwt-secret = "OOcJ7VoSY1mXqod4MKtb9WCCwt9erJkRQ2tzYmLb4Xe="
db-anon-role = "web_anon"
server-port=3000
The parser rejects the question mark
{"details":"invalid URI query parameter: \"unix_socket\"\n","code":"","message":"Database connection error"}
Config 3
db-uri = "postgres://postgres:password#/mydatabase"
db-schema = "api"
jwt-secret = "OOcJ7VoSY1mXqod4MKtb9WCCwt9erJkRQ2tzYmLb4Xe="
db-anon-role = "web_anon"
server-port=3000
server-unix-socket= "/cloudsql/INSTANCE_CONNECTION_NAME"
server-unix-socket appears to only take socket lock file path. Feeding it /cloudsql/INSTANCE_CONNECTION_NAME tries to delete file as in `postgrest.exe: /cloudsql/INSTANCE_CONNECTION_NAME: DeleteFile "/cloudsql/INSTANCE_CONNECTION_NAME": invalid argument t (The filename, directory name, or volume label syntax is incorrect.)
Documentation
Cloud SQL Doc
https://cloud.google.com/sql/docs/mysql/connect-run
PostgREST
http://postgrest.org/en/v6.0/configuration.html
https://github.com/PostgREST/postgrest/issues/1186
https://github.com/PostgREST/postgrest/issues/169
Environment
PostgreSQL version:11
PostgREST version: 6.0.2
Operating system: Win10 and Alpine
First you have to add the Cloud SQL connection to the Cloud Run instance:
https://cloud.google.com/sql/docs/postgres/connect-run#configuring
After that, the DB connection will be available in the service on a Unix domain socket at path /cloudsql/<cloud_sql_instance_connection_name> and you can set the PGRST_DB_URI environment variable to reflect that.
Here's the correct format:
postgres://<pg_user>:<pg_pass>#/<db_name>?host=/cloudsql/<cloud_sql_instance_connection_name>
e.g.
postgres://postgres:postgres#/postgres?host=/cloudsql/project-id:zone-id-1:sql-instance
According with Connecting with CloudSQL, the example is:
# postgres+pg8000://<db_user>:<db_pass>#/<db_name>?unix_sock=/cloudsql//.s.PGSQL.5432
Then you can try with (Just as #marian.vladoi mentioned):
db-uri = "postgres://postgres:password#/mydatabase?unix_socket=/cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432"
Keep in mind that the connection name should include:
ProjectID:Region:DatabaseName
For example: myproject:myregion:myinstance
Anyway, you can find here more options to connect from external applications and from within Google Cloud.
I tried many variations but couldn't get it to work out of the box, however I'll post this workaround.
FWIW I was able to use an alternate socket location with postgrest locally, but then when trying to use the cloudsql location it doesn't seem to interpret it right - perhaps the colons in the socket path are throwing it off?
In any case as #Steve_Chávez mentions, this approach does work db-uri = postgres:///user:password#/dbname and defaults to the postgrest default socket location (/run/postgresql/.s.PGSQL.5432). So in the docker entrypoint we can symlink this location to the actual socket injected by Cloud Run.
First, add the following to the Dockerfile (above USER 1000):
RUN mkdir -p /run/postgresql/ && chown postgrest:postgrest /run/postgresql/
Then add an executable file at /etc/entrypoint.bash containing:
set -eEux pipefail
CLOUDSQL_INSTANCE_NAME=${CLOUDSQL_INSTANCE_NAME:-PROJECT_REGION_INSTANCE_NAME}
POSTGRES_SOCKET_LOCATION=/run/postgresql
ln -s /cloudsql/${CLOUDSQL_INSTANCE_NAME}/.s.PGSQL.5432 ${POSTGRES_SOCKET_LOCATION}/.s.PGSQL.5432
postgrest /etc/postgrest.conf
Change the Dockefile entrypoint to CMD /etc/entrypoint.sh. Then add CLOUDSQL_INSTANCE_NAME as an env var in cloud run. The PGRST_DB_URI env var is like so postgres://authenticator:password#/postgres
An alternative approach if you don't like this, would be to connect via serverless vpc connector.
I struggled with this too.
I end up doing a one-liner for DB-URI env variable
host=/cloudsql/project-id:zone:instance-id user=user port=5432 dbname=dbname password=password
However, I have postgrest running on cloud run that lets you specify the instance connection name via
INSTANCE_CONNECTION_NAME=/cloudsql/project-id:zone:instance-id
Maybe you can host it there and you end up doing it serverless Im not sure where are you running it currently.
https://cloud.google.com/sql/docs/mysql/connect-run
I would like to find out how to connect to an external MongoDB instance in Meteor.
I have added this environment
Meteor.startup(function () {
process.env.MONGO_URL = 'mongodb://[UN]:PW]#[host]:[port]/meteorTest'
});
but still the data is coming from the local database.
I want to move all the collections from my local db to this external db. I read all the tutorials, its all telling me to setup this evn variable but nothing really working. How do I test whether its connected or not?
In my own experience; I have needed to set the environment variable before starting the meteorjs server app. To do this you will need to pass the environment variable on the command-line as you invoke meteor or preset the environment for the profile that is running the meteor app on your system.
So you would start your app with this kind of a command:
MONGO_URL='mongodb://user:password#remote.domain.com:12345/' meteor
You should also make sure that the mongodb is reachable and that your user credentials are correct! I am assuming you are trying to run meteor on your local machine using a remote mongodb instance.
On Windows
You will have to create a batch file in your meteor application folder to invoke the environment variable. There is an example of this here: https://stackoverflow.com/a/29833177/1997579
I don't like to use big repeating command and I was searching for a solution where I will be setting a variable embedded with something so every time I start my meteor app; the MONGO_URL will set to environment automatically. So this what I did:
In the package.json file I replaced the start parameter as below:
"scripts": {
"start": "MONGO_URL=mongodb://username:password#host_url:portnumber/dbname meteor run"
},
Now every time I want to run my app; I run npm start instead of meteor or meteor run
Note: there is a disadvantage with that. Your db credentials will be exposed if you put your db credentials to package.json file and add this file to version control.
run it in command prompt:
"MONGO_URL=mongodb://<USER>:<PASSWORD>#<SERVER>:<PORT>/<DB> meteor"
or
save this url in run.sh file in project folder and run meteor
On windows, I set MONGO_URL in my system's environment variable and it worked fine for me.
I have created a new environment variable and it's value as MONGO_URL=mongodb://username:password#host_url:portnumber/dbname
And in path variable, I have added %MONGO_URL%
Then in meteor app root folder, I have run $meteor