I'm using Nest.js deployed Google cloud app engine and Cloud SQL postgres typeorm
Here is my config file
{
"type":"postgres",
"extra":{"socketPath":"/cloudsql/xxx:us-central1:xxx"},
"username":"xxx",
"password":"xxx",
"database":"xxx",
"synchronize":false,"logging":false,
"entities":["dist/**/*.entity{ .ts,.js}"],
"migrations":["/workspace/dist/src/db/migrations/*.js"],
"migrationsTableName":"migrations_typeorm",
"migrationsRun":false,
"cli":{"migrationsDir":"src/db/migrations"}
}
But app engine ignores socketPath and automatically use DEFAULT host and port 127.0.0.1:5432
What is wrong with my config options?
When running the project locally, everything is okay but when deployed to the cloud it's not working. The reason is that locally you don't need "host" property but cloud server needs "host" property which should be the same socketPath ("/cloudsql/xxx:us-central1:xxx").
So when you are on localhost then remove "host" property.
When deploying to server attach "host" property.
Related
I have setup an Rancher (RKE) (kuberbetes) for my application.
and application using the postgres so i have setup Crunchydata postgres operator and create postgres cluster using that.
everything fine but now i want to see the pg_activity for my postgresql.
how i can see the activity of whole postgres ?
you use the monitoring tools in rancher to monitor the Postgres.
apart from that you can SSH inside the respective pod of the database and use the cli command and check the output.
In rancher, you can also use the client tool to connect with the rancher and run the cli command to check the pg_activity.
Client docker image : https://hub.docker.com/r/jbergknoff/postgresql-client/
you can also deploy the GUI docker client on rancher and use it
GUI postgress client : https://hub.docker.com/r/dpage/pgadmin4/
GUI Example : https://dataedo.com/kb/query/postgresql/list-database-sessions#:~:text=Using%20pgAdmin,all%20connected%20sessions%20(3).
I'm having issues when trying to connect to my Cloud SQL instance. I created a SQL Server instance, downloaded the cloud sql proxy, and everything seems to start to connect, but I keep getting the following error:
errors parsing config:
invalid "instance-connection-name": unsupported network: unix
I'm specifying the tcp port to use, but it still complains about UNIX. Here is the command I'm using when trying to connect (I replaced the actual instance connection name for privacy/security):
./cloud_sql_proxy.exe -instances=[instance-connection-name]=tcp:3306
Any help would be appreciated.
Thanks!
I tried this and it works
Rename cloud_sql_proxy_xxx to cloud_sql_proxy
Open cmd in your cloud_sql_proxy's location
Run the following command: cloud_sql_proxy -instances=[project:region:instance-name]=tcp:1433 without [ ]
From Connecting to a Cloud SQL for SQL Server using a Cloud SQL Proxy:
Depending on your language and environment, you can start the proxy using either TCP sockets or Unix sockets.
TCP sockets:
Copy your instance connection name from the Instance details page
For example: myproject:us-central1:myinstance.
If you are using a service account to authenticate the proxy, note the location on your client machine of the private key file that was created when you created the service account.
Start the proxy.
Some possible proxy invocation strings:
a) Using Cloud SDK authentication:
./cloud_sql_proxy -instances=<INSTANCE_CONNECTION_NAME>=tcp:1433
The specified port must not already be in use, for example, by a local database server.
b) Using a service account and explicit instance specification (recommended for production environments):
./cloud_sql_proxy -instances=<INSTANCE_CONNECTION_NAME>=tcp:1433 \
-credential_file=<PATH_TO_KEY_FILE> &
The docs describe that hasura needs the postgres connection string with the HASURA_GRAPHQL_DATABASE_URL env var.
Example:
docker run -d -p 8080:8080 \
-e HASURA_GRAPHQL_DATABASE_URL=postgres://username:password#hostname:port/dbname \
hasura/graphql-engine:latest
It looks like that my problem is that the server instance connection name for google cloud sql looks like PROJECT_ID:REGION:INSTANCE_ID is not TCP
From the cloud run docs (https://cloud.google.com/sql/docs/postgres/connect-run) I got this example:
postgres://<db_user>:<db_pass>#/<db_name>?unix_sock=/cloudsql/<cloud_sql_instance_name>/.s.PGSQL.5432 but it does not seem to work. Ideas?
I'm currently adding the cloud_sql_proxy as a workaround to the container so that I can connect to TCP 127.0.0.1:5432, but I'm looking for a direct connection to google-cloud-sql.
// EDIT Thanks for the comments, beta8 did mostly the trick, but I also missed the set-cloudsql-instances parameter: https://cloud.google.com/sdk/gcloud/reference/beta/run/deploy#--set-cloudsql-instances
My full cloud-run command:
gcloud beta run deploy \
--image gcr.io/<PROJECT_ID>/graphql-server:latest \
--region <CLOUD_RUN_REGION> \
--platform managed \
--set-env-vars HASURA_GRAPHQL_DATABASE_URL="postgres://<DB_USER>:<DB_PASS>#/<DB_NAME>?host=/cloudsql/<PROJECT_ID>:<CLOUD_SQL_REGION>:<INSTANCE_ID>" \
--timeout 900 \
--set-cloudsql-instances <PROJECT_ID>:<CLOUD_SQL_REGION>:<INSTANCE_ID>
As per v1.0.0-beta.8, which has better support for Postgres connection string parameters, I've managed to make the unix connection to work, from Cloud Run to Cloud SQL, without embedding the proxy into the container.
The connection should look something like this:
postgres://<user>:<password>#/<database>?host=/cloudsql/<instance_name>
Notice that the client will add the suffix /.s.PGSQL.5432 for you.
Make sure you added also the Cloud SQL client permission.
If the Hasura database requires that exact connection string format, you can use it. However, you cannot use Cloud Run's Cloud SQL support. You will need to whitelist the entire Internet so that your Cloud Run instance can connect. Cloud Run does not publish a CIDR block of addresses. This method is not recommended.
The Unix Socket method is for Cloud SQL Proxy that Cloud Run supports. This is the connection method used internally to your container when Cloud Run is managing the connection to Cloud SQL. Note, for this method IP based hostnames are not supported in your client to connect to Cloud Run's Cloud SQL Proxy.
You can embed the Cloud SQL Proxy directly in your container. Then you can use 127.0.0.1 as the hostname part for the connection string. This will require that you create a shell script as your Cloud Run entrypoint to launch both the proxy and your application. Based on your scenario, I recommend this method.
The Cloud SQL Proxy is written in Go and the source code is published.
If you choose to embed the proxy, don't forget to add the Cloud SQL Client role to the Cloud Run service account.
I have a flask application which is running as docker container. Flask application uses local mongodb. This docker container is not able to connect to local mongodb.
I have tried following option :
set --network="host" in docker run ... command
set MONGO_URI = "mongodb://host-ip-address:27017/model-service-sample",
set MONGO_URI = "mongodb://container-gateway-ip-address:27017/model-service-sample"
None of the above options worked.
Can anyone please suggest a way to accomplish this?
For MacOS you should use:
host.docker.internal or gateway.docker.internal
for connecting from a container to a service on the host.
refer : https://docs.docker.com/docker-for-mac/networking/#/known-limitations-use-cases-and-workarounds
I am attempting to deploy my first MEAN stack application ('weatherapp') to production on AWS.
I deployed my NodeJS/Express/Angular app to AWS Elastic Beanstalk (preconfigured Linux machine running Node). This works fine and I can view the app in the browser.
Separately I created a docker container running MongoDB and deployed it to AWS / EC2 following the steps in this post:
https://blog.codeship.com/running-mean-web-application-docker-containers-aws/
My question is - how do I connect the two?
In my NodeJS app I was connecting to my local Mongo instance locally like this:
'mongodb://localhost:27017/weatherapp'
What steps can I take to find out what the connection string should be for my production Mongo instance on docker?
Thanks in advance!
The answer to this is two-fold. We need to set some options on the Docker side in the EC2 instance and then some security groups and configuration on the AWS side. First, we'll start on on the Docker container side.
Container
When you run the MongoDB container, you will want to do two things:
Persist the data to disk.
Open the MongoDB port to the container.
To persist the data to disk you will want to do something like -v /data/db:/data/db. This will make the MongoDB data available at /data/db on the host. This makes sure that an accidental deletion or upgrade of the container doesn't lose any data.
Next, we need to publish the MongoDB port so that applications external of Docker can connect to it. The default MongoDB port is 27017 so let's publish that using -p 27017:27017.
If your original command for starting MongoDB was:
docker run --name mymongodb -d mongo
Then the new one would be:
docker run --name mymongodb -d -p 27017:27017 -v /data/db:/data/db
AWS
Now, we need to edit the security group of your EC2 instance and configuration of Elastic Beanstalk.
Security Groups
First, take a look at your Security Groups in the EC2 console. You will have a group for the Elastic Beanstalk application named similar to awseb-e-xanf9hqrw3-stack-AWSEBSecurityGroup-1N2T1AI2H05I8 with a ID similar to sg-07fb8c43. We'll use this ID in the next step so copy it somewhere.
Now find the Security Group attached to your EC2 instance running the Docker container. You will need to add a new rule to this group allowing access to the MongoDB container. Edit the group and add a new inbound rule for:
Type: Custom TCP
Protocol TCP
Port range: 27017
Source: sg-07fb8c43
This will allow the Elastic Beanstalk EC2 instances (using sg-07fb8c43) to access the MongoDB port on your Docker EC2 instance.
Elastic IP
You'll likely want a more static IP address for your EC2 instance in case it reboots. Navigate to the Elastic IPs section of the EC2 console and allocate a new address to your Docker EC2 instance.
The new Elastic IP will be the address you use in your Elastic Beanstalk configuration to connect to MongoDB. If your address was 54.67.29.50 then your application would connect to mongodb://54.67.29.50:27017.
Elastic Beanstalk
Now, instead of hardcoded this address in your Node.js application, you should configure your application to pull the information from an environment variable. In your application, you should read the MongoDB URL from something like process.env.MONGO_URI. Then, in your Elastic Beanstalk application configuration, navigate to the Software Configuration and then down to Environment Properties. Here, you create a property name of MONGO_URI and the value as mongodb://54.67.29.50:27017. This will allow you to easily change the MongoDB instance should it ever change or if you launch multiple environments with different databases.