While trying to get an external JSON file using python on PythonAnywhere
import http.client
conn = http.client.HTTPSConnection("www.tesourodireto.com.br")
conn.request("GET", "/json/br/com/b3/tesourodireto/service/api/treasurybondsinfo.json")
I got CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
I know that I need to import the certificate chain used by this website. I found this bash script to get it:
openssl s_client -showcerts -connect www.tesourodireto.com.br:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >mycertfile.pem
Now I need to import it in the trusted cas on the os:
import certifi
certifi.where()
'/usr/local/lib/python3.10/site-packages/certifi/cacert.pem'
But it requires me root permissions:
cat mycertfile.pem > /usr/local/lib/python3.10/site-packages/certifi/cacert.pem
bash: /usr/local/lib/python3.10/site-packages/certifi/cacert.pem: Permission denied
Sudo command is out of question:
sudo su -
bash: sudo: command not found
How can I accomplish this on PythonAnywhere, being a normal user? I don't want to disable SSL chain verification...
If you have a complete cert file, you can pass it directly to requests:
requests.get('https://example.com', verify='/path/to/certfile')
Related
I usually dump my mongodb database using this command and it works perfectly:
mongodump --uri mongodb+srv://name:password#cluster0.fklgt.mongodb.net/database_name --archive="dump-copy-name" --forceTableScan
However, I have been trying to dump different databases and it just keeps logging this:
2021-06-21T18:43:39.206+0100 error dialing
cluster0-shard-00-00.fklgt.mongodb.net:27017: SSL errors: SSL
routines:ssl3_get_server_certificate:certificate verify failed
Any idea what's going-on?
The mongodump tool does not implicitly trust your system certificate store. You will need to pass it the root CA certificate so that it can validate the cluster's certificate.
To find out which certificate is the root, use openssl:
openssl s_client -connect cluster0-shard-00-00.fklgt.mongodb.net:27017
The output should include a certificate chain section like:
---
Certificate chain
0 s:/C=US/ST=New York/L=New York/O=MongoDB, Inc./CN=*.mongodb.com
i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
(this is an example, test with your cluster to get the real certificates)
The highest numbered certificate in the chain should be the root. Once you have that, google for the CN part (in my example that was "DigiCert SHA2 Secure Server CA"), and you should find a link to download the certificate.
The root should be one of the two listed here: https://docs.atlas.mongodb.com/reference/faq/security/#hard-coded-certificate-authority
Once you have the root certificate in .pem format, use the command line option --sslCAFile=<filename> to pass it to mongodump.
Windows 10 Pro x64
Mongodb 4.4
Compass GUI
I set up Mongodb as normal (no security) and verified everything was working. Now trying to add TLS/SSL. I followed the instructions at:
https://docs.mongodb.com/manual/appendix/security/appendixA-openssl-ca/
https://docs.mongodb.com/manual/appendix/security/appendixB-openssl-server/
No errors. Everything succeeded. Verified the pems with:
openssl verify -CAfile mongodb.pem mongodb01.pem
mongodb01.pem: OK
Updated my config file with:
net:
port: 27017
bindIp: mongodb01.xxx.com
tls:
mode: requireTLS
certificateKeyFile: C:\Program Files\MongoDB\Server\4.4\bin\mongodb01.pem
CAFile: C:\Program Files\MongoDB\Server\4.4\bin\mongodb.pem
The CA cert was created for *.xxx.com and the server cert was created for mongodb01.xxx.com. NOTE: mongodb01.xxx.com is faked in my hosts file to 127.0.0.1. I've done this many times before and the fake dns name trick does work when its all on the same machine.
I then run mongodb with:
mongod --config "C:\Program Files\MongoDB\Server\4.4\bin\mongod.cfg"
But when I try:
openssl s_client -connect mongodb01.xxx.com:27017
verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate
EDIT:
So I made SOME progress. I don't think MongoDB is picking up the CAFile. When I added the root cert and the intermediate cert into the mongodb01.pem (and all 3 signed with the same key), then the cert chain passes, but its returning a self signed cert error (as expected).
I haven't tried calling it from Java yet, I was trying to get it to work in Compass first. However, Compass complains about the self signed cert. I tried putting the certs in the my trusted store, but Compass still refuses to connect.
You need to provide the CA certificate to s_client, unless the certificate was issued by a well-known CA and its CA cert is in the bundle that ships with openssl (assuming such a bundle even exists).
Use -CApath or -CAfile options to specify the CA certificate.
openssl does not usually have any default root certification authorities.
Try passing your CA with the -CAfile option, like
openssl s_client -CAfile mongodb.pem -connect mongodb01.orionsoftware.com:27017
Note that mongodb.pem should not contain the private key for this usage.
I have a local server without any domain or public IP for that. I'm gonna to setup SSL self sign certificate for BigBlueButton. How I can do it in my local server?
Without host and domain names, self-signed certificates will be the only option which means they will not be valid SSL certificates. I don't know BigBlueButtom but it's documentation doesn't recommend this set up for production environments. Not every browser will accept it either.
However, if you want to give it a try, you can generate self-signed SSL certs on Linux using this command:
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout selfsigned.key -out selfsigned.crt
These options will create both a key file and a certificate. You will be asked a few questions about the server in order to embed the information correctly in the certificate.
And then you can try to adapt the instructions here.
I was setting up BBB environment recently.
Self-signed certificate is no good. To get it working I had to:
Use a real server setup (with let's encrypt) and a real domain to get real certificates
copy the certificates to my local development setup (and update nginx config of course)
set up /etc/hosts locally
Use real SSL certificate. I had to:
Install BBB. Use ip instead hostname. See
https://docs.bigbluebutton.org/2.2/install.html#configure-nginx-to-use-https
Example:
wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | bash -s -- -v bionic-230 -s 10.211.55.9 -e me#example.com -a -w
Configure nginx to use HTTPS for you real domain (Order of certificates is very important). See
https://docs.bigbluebutton.org/2.2/install.html#configure-nginx-to-use-https
Add to hosts file ip and you domain. Example:
10.211.55.9 example.com
Use command to change domain.
bbb-conf --setip example.com
I have set up a SonarQube and configured SSL certificates to make the URL always HTTPS using CertBot. As of now, the PostgreSQL database has a public IP and below are the values changed in sonar.properties file:
sonar.jdbc.username=weakusername
sonar.jdbc.password=strongpassword
sonar.web.host=127.0.0.1
sonar.jdbc.url=jdbc:postgresql://xx.xxx.xxx.xxx/sonarqube
sonar.search.javaOpts=-Xms512m -Xmx512m
# Change max limits
sysctl -w vm.max_map_count=262144
I am using Cloud SQL PostGres as the database. I would like to allow Only SSL Only Connections to the database and here the way how to do it, generate client certificate, etc.
After setting "Allow only SSL Connections" to true I understand there is a way to connect to the database using the client certificate described here.
Below is the command to start the psql client:
psql "sslmode=verify-ca sslrootcert=server-ca.pem \
sslcert=client-cert.pem sslkey=client-key.pem \
hostaddr=[INSTANCE_IP] \
user=postgres dbname=[DB_NAME]"
However, SonarQube is not able to connect to the Database (Not sure how to tell SonarQube to use the client certificates).
What changes are required in the configuration file to make SonarQube use appropriate client certificate and connect to the database using SSL?
You should add the following to the URL:
jdbc:postgresql://xx.xxx.xxx.xxx/sonarqube?ssl=true&sslmode=verify-ca&sslrootcert=/path/to/server-ca.pem&sslkey=/path/to/client-key.pem&sslcert=/path/to/client-cert.pem
See the documentation for the available SSL connection parameters and SSL client configuration.
Convert client key format from PEM to PK8:
openssl pkcs8 -topk8 -inform PEM -outform DER -in client-key.pem -out client-key.pk8 -nocrypt
Be sure to update the value for the sslkey query param in sonar.jdbc.url with the new path/filename.
How to make connection from mongo-spark connector to mongodb when only TLS/ssl enabled for mongo DB ?
How to pass the uri and collection name in read config to make connection with TLS/ssl enabled mongodb instance?
Thanks in advance ?
To make the ssl connection from Spark to the Mongo server you will need to trust the Mongo certificate, or the CA (certificate authority) that has signed that certificate. This is the most important part, and the trickiest one for me to figure it out.
Spark is a Java application, so it get the certificates from a jks trustStore. you will need to import the Mongo certificate (only the public part) into a trustStore to make it available for spark. To do so:
Get the Mongo certificate: Ask the DBA or the sysadmin who has setup the mongo to provide the certificate to you. Other aproach is to get it with openssl:
$ openssl s_client -connect mongodb:27017
CONNECTED(00000003)
depth=0 C = ES, ST = Madrid, L = Madrid, O = HOME, OU = HOME, CN=mongodb mongo.hostname.local
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/C=ES/ST=Madrid/L=Madrid/O=COMPANY/OU=AREA/CN=mongo.hostname.local
i:/C=ES/ST=Madrid/L=Madrid/O=COMPANY/OU=AREA/CN=mongo.hostname.localIssuing CA
---
Server certificate
-----BEGIN CERTIFICATE-----
[..... A bunch of base64 text....]
-----END CERTIFICATE-----
Get the part from the -----BEGIN CERTIFICATE----- to -----BEGIN CERTIFICATE----- and save it in a .cert file
Import it into a trustStore
$ keytool -import -file /path/to/your/mongodb.crt -alias mongodb -keystore /path/to/your/trustStore.jks
Enter keystore password: 123456
...
...
Trust this certificate? [no]: yes
Certificate was added to keystore
Make sure the keystore is accesible from all your spark cluster nodes.
Now, you have your server certificate imported. If you need mutual TLS you will need to provide a valid client certificate. This certificate, and the certificate private key, should be in a jks keyStore (it could be in the same trustStore file you have stored the Mongo server certificate because it uses the same format). If are not going to use mutual TLS you don't need to do this, but you have to check that the MongoDB instance is able to accept connections without client certificates. This is with the flag sslAllowConnectionsWithoutCertificates
The next step is specifying in the connection URI that you want to use TLS. This is fairly simple, just add the ?ssl=true to your connection string. So the connection URI will be something like this
mongodb://user:pw#host:port/db.collection?ssl=true
Now you can summit your job. When summiting the job we also need to specify the location of our trustStore, and the libraries for the mongo connector:
/spark/bin/spark-submit \
--master spark://spark-master:7077 \
--packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 \
--conf spark.executor.extraJavaOptions="-Djavax.net.ssl.trustStore=/path/to/your/trustStore.jks -Djavax.net.ssl.trustStorePassword=yourPassword" \
--conf spark.driver.extraJavaOptions="-Djavax.net.ssl.trustStore=/path/to/your/trustStore.jks -Djavax.net.ssl.trustStorePassword=yourPassword" \
/yourJob.jar
We use the extraJavaOptions for the driver and the executor to pass these parameters. If you are using mutual TLS, include the following extra java options:
-Djavax.net.ssl.keyStore=/path/to/your/keyStore.jks
-Djavax.net.ssl.keyStorePassword=yourPassword
The /path/to/your/keyStore.jks is where you have stored your client certificates.
If the spark connector library is not already installed, you may run into trouble. The spark process will go to maven to download the library, but it will not be able to verify the maven certificates because we have specified another keyStore with just our certificate. One workaround is to import our certificate directly into the default keystore located at $JAVA_HOME/jre/lib/security/cacerts. The default password is changeit. Remember to do this in every worker node too.
I hope it helps!
Sources:
https://github.com/brunocfnba/spark-mongo-ssl
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_spark-component-guide/content/spark-encryption.html
https://community.hortonworks.com/articles/147113/how-to-configure-your-spark-application-to-use-mon.html
https://mapr.com/support/s/article/Unable-to-find-valid-certification-path-to-requested-target-error-while-accessing?language=en_US