I am trying to set up Openshift 4.9 and running into issues configuring the mirror registry. I have narrowed down the issue to cert error with quay.io
$ wget "https://quay.io/openshift-release-dev/ocp-release:4.8.15-x86_64"
--2021-10-25 16:57:27-- https://quay.io/openshift-release-dev/ocp-release:4.8.15-x86_64
Resolving quay.io (quay.io)... 35.172.159.14, 34.224.196.162, 3.216.152.103, ...
Connecting to quay.io (quay.io)|35.172.159.14|:443... connected.
ERROR: The certificate of âquay.ioâ is not trusted.
ERROR: The certificate of âquay.ioâ has been revoked.
I have downloaded the cert chain from quay.io and copied it to
/etc/pki/ca-trust/source/anchors/
Then I ran update-ca-trust as well as update-ca-trust extract
I checked the bundle and certs are present.
/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
However, I keep getting the cert for quay.io is not trusted.
Any pointers to fix this would be appreciated.
Two things may help:
First of all, make sure you added the right CA file to the anchors folder:
DigiCert High Assurance EV Root CA Self-signed
Fingerprint SHA256: 7431e5f4c3c1ce4690774f0b61e05440883ba9a01ed00ba6abd7806ed3b118cf
Pin SHA256: WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18=
Then check the result in /etc/pki/tls/certs/ca-bundle.crt
Related
I have created a docker container that is using Sagemaker via the java sdk. This container is deployed on a k8s cluster with several replicas.
The container is doing simple requests to Sagemaker to list some models that we have trained and deployed. However we are now having issues with some java certificate. I am quite novice with k8s and certificates so I will appreciate if you could provide some help to fix the issue.
Here are some traces from the log when it tries to list the endpoints:
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy67.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
... 70 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
... 97 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 103 common frames omitted
This might most likely to do with some custom SSL certification path added to your network by your admin. You might want to inspect the SSL root certificates by opening any secured website on your browser and click on the Secure link to the left of the address bar ( atleast this is how it is in chrome ) . You will see a popup showing certificate and certification information. Go to its Certificate Path and see the ROOT certificate , if it is something of custom certificate then you will need to add the same to your cacerts file. Read this link for more details
I think I have found the answer to my problem. I have set up another k8s cluster and deployed the container there as well. They are working fine and the certificate issues does not happen. When investigating more I have noticed that they were some issues with DNS resolution on the first k8s cluster. In fact the containers with certificate issues could not ping google.com for example.
I fixed the DNS issue by not relying on core-dns and setting the DNS configuration in the deployment.yaml file. I am not sure to understand why exactly but this seems to have fixed the certificate issue.
The error message you're receiving occurs when Java does not know about the root certificate returned by an TLS endpoint. This often occurs if you change the root certificates available.
Per https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#Customization:
"If a truststore named <java-home>/lib/security/jssecacerts is found, it is used.
If not, then a truststore named <java-home>/lib/security/cacerts is searched for and used (if it exists).
Finally, if a truststore is still not found, then the truststore managed by the TrustManager will be a new empty truststore."
Openssl is a good tool for debugging such certificate issues. You can use the following command to retrieve the certificate returned by an endpoint. This may help you determine what the certificate chain looks like.
openssl s_client -showcerts -connect www.example.com:443 </dev/null
You can view the list of certificates that Java knows about using keytool, a utility vended with the JRE.
keytool -list -cacerts
Some system administrators will override the default certificates by writing an alternative truststore file into the default location. Other times, teams may override the default using the javax.net.ssl.trustStore system property.
Finally, you can use the jps utility, also vended with the JRE, to see the system properties set on a running Java process.
jps -v
I reinstalled some nodes and a master. Now on the master I am getting:
Sep 15 04:53:58 master kube-apiserver[803]: I0915 04:53:58.413581 803 logs.go:41] http: TLS handshake error from $ip:54337: remote error: bad certificate
Where $ip is one of the nodes.
So I likely need to delete or recreate certificates. What would the location of those be? Any recommended commands to recreate or remove those or copy them from node to master or vice versa? Whatever gets me past this error message...
Take a look through the Creating Certificates section of authentication.md. It walks you through the certificates that you need to create and how to pass them to the system components, and you should be able to use that to re-generate certificates for your cluster.
I am not sure either what I am trying to do is possible or correct way.
One of my colleague spinup kubernetes gce cluster (with 1 master and 4 minions.) in a project which is shared with me as owner access.
After setup he shared his ~/.kubernetes_auth keys along with .kubecfg.crt, .kubecfg.ca.crt and .kubecfg.key. I copied all of the at my home folder and setup the kubernetes workspace.
I also set the project name as a default project in geconfig. and now I can connect to the master and slaves using 'gcutil ssh --zone us-central1-b kubernetes-master'
But when I try to list of existing pods using 'cluster/kubecfg.sh list pods'
I see
"F1017 21:05:31.037148 18021 kubecfg.go:422] Got request error: Get https://107.178.208.109/api/v1beta1/pods?namespace=default: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ChangeMe")
I tried to debug from my side but failed to come any conclusion. Any sort of clue will be helpful.
You can also copy the cert files off of the master again. They are located in /usr/share/nginx on the master.
It is probably due to a not implemented feature, see this issue:
https://github.com/GoogleCloudPlatform/kubernetes/issues/1886
you can copy the files from /usr/share/nginx/... on the master
into your home dir and try again.
I figured out a workaround: set the -insecure_skip_tls_verify option
In kubecfg.sh, change the code near the bottom to
else
auth_config=(
"-insecure_skip_tls_verify"
)
fi
Obviously this is insecure and you are putting yourself at risk of a man in the middle attack, etc.
I have tried several ways to get some PEM files to be used by CouchDB. I have generated a cert with powershell, exported it with key to a pfx and then used openssl to convert to 2 pem files and installed them in Couch. With this approach it seems to work in IE11, but it doesn't work with firefox or other browsers. Firefox produces this error:
The key does not support the requested operation. (Error code:
sec_error_invalid_key)
I've also gotten a free ssl cert from ssl.com, (they gave me three CRT files) and tried converting it using openssl, but to no avail.
I've also followed the instructions on the page from CouchDB to generate a self signed cert specifically for this purpose, but it will not load the page. (http://docs.couchdb.com/en/latest/config/http.html#config-ssl about half way down)
Has anybody had success with this? How do I get my certs into a format that will play nice with Couch and will all browsers?
UPDATE:
Now I'm getting this error
A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot.
Not sure if this is a step forward or backwards...
It seems like CouchDB versions predating 1.7 or 1.6 are not able to have intermediate certificates specified for certificate verification. Since you are writing about having received three .crt files, (s)ome of those might be required as intermediate certificate(s). CouchDB not knowing about them can be the cause of your problem.
Apparently, one way to work around this is to concatenate your certificate file along with the intermediate certificate file(s). Simply cat them together like
$ cat yours.crt theirs.crt > couchdb.crt
...and use CouchDB's certfile configuration option to point to couchdb.crt's location.
If you prefer to convert .crt to .pem first, use sth like
$ openssl openssl x509 -in yours.crt -inform der -outform pem -out yours.pem
In a new enough version, you can probably use an intermediate certificate by setting CouchDB's cacertfile option. Have a look at this for further information.
Profiles installed by MDM service are showing as "Not Verified" after upgrading the device to iOS 6. These profiles were signed by a InCommon cert issued by AddTrust before being pushed to the devices. They were showing as "Verified" before the upgrade. Any ideas what might have caused this?
I got exactly the same problem so it is probably a bug in the iOS profile system because the very same SSL certificate is trusted by the browser. Note: Our certificate is of the "*.host.org" type.
This might be an untrusted CA in the certificate chain from the cert provider you bought it from. Looks like some CAs are untrusted or missing from iOS6. I had the same problem and included the whole of the cert chain in our cert signing bundle and the issue was resolved.
Suggest you open a support case with your cert provider to see if it's a known issue or dig around to see if you can find a list of trusted CAs used in iOS6 - I couldn't find one. Synching the device to iTunes may also refresh the CA list but this didn't work for me this time.
I have exactly the same problem as your.
My chain is GeoTrust -> RapidSSL -> MyCert. I have included the full chain in my .crt, but it stills show "not verified" when I try to install the configuration profil.
I don't know how to insert the whole path.
I use an openssl command to sign my file :
openssl smime -sign -signer #{crt_path} -inkey #{private_key_path} -nodetach -outform der -in #{file_to_sign_path} -out #{file_signed_path}
My crt_path is a .crt file, including the three certs.
EDIT
I found out the problem with my openssl command. My full chain was in the #{crt_path} but was not used by the command. I added the *-certfile #{crt_path}* and things works well !
Including the intermediate cert in the pkcs7_sign call (php openssl_pkcs7_sign() in my case) revolved the problem.
Yes! Adding the entire path (-root) did the trick.
Verify that the signature created by the MDM SW actually contains the path. Since it wasn't needed before...