RJDBC - Could not open client transport with JDBC Uri - ibm-cloud

I'm trying to connect to my Biginsights Enterprise cluster from RJDBC:
hiveconnection <- dbConnect(drv,
"jdbc:hive2://xxxxx:10001/default",
"xxxxx",
"xxxxx",
ssl="true",
sslTrustStore="mytruststore.jks",
trustStorePassword="xxxxx",
hive.server2.transport.mode="http",
hive.server2.thrift.http.path="gateway/default/hive
)
However, I'm hitting the issue:
Could not open client transport with JDBC Uri

This works for me:
username = 'changeme'
password = 'changeme'
# change this to your hostname
url = 'jdbc:hive2://bi4c-xxxx-master-3.bi.services.bluemix.net:10000/default;ssl=true;sslTrustStore=/change/to/yourpath/to/truststore.jks;trustStorePassword=mypassword;'
library(rJava)
library(RJDBC)
hive.class.path = list.files(path=c('/change/to/yourpath/to/build/hivedrivers/'), pattern="jar", full.names=T);
.jinit(classpath=hive.class.path,parameters="")
drv <- JDBC("org.apache.hive.jdbc.HiveDriver","hive-jdbc-2.0.0.jar",identifier.quote="`")
conn <- dbConnect(drv,url,username,password)
show_databases <- dbGetQuery(conn, "show databases")
print(show_databases)
Setup Steps
I used gradle to download all the hive jdbc dependencies to a folder. I next created a build.gradle file:
repositories {
mavenCentral()
}
configurations {
drivers
}
dependencies {
drivers "org.apache.hive:hive-jdbc:2.0.0"
}
task CopyDrivers(type: Copy) {
from configurations.drivers
into "$buildDir/hivedrivers"
}
Then in the terminal window I changed to directory of my build.gradle file, I executed:
$ gradle CopyDrivers
I created the truststore with:
$ BI_HOST=bi4c-xxxxx-master-3.bi.services.bluemix.net
$ openssl s_client -showcerts -connect $BI_HOST:9443 < /dev/null | openssl x509 -outform PEM > certificate
$ rm -f truststore.jks
$ keytool -import -trustcacerts -alias biginsights -file certificate -keystore truststore.jks -storepass mypassword -noprompt

Related

How to properly setup my Ionic (Angular) dev machine with self-signed certificate?

I'm working on a Ionic-Angular app and faced quite a few issues due to clear text traffic. So I decided to switch to https even while coding but it wasn't so easy.
I open this question and propose the answer I found to keep a trace and hopefully save you some time if you would like to do the same.
Edit:
I now have a repo for certificates generation scripts: https://github.com/ch4mpy/self-signed-certificate-generation
Prerequisite
You need a hostname for your dev machine. This name will be declared in certificates and https services will be accessed using this hostname (URLs have to be like https://[hostname]:... for certificate check to pass).
If your network doesn't have a DNS already, you might use something like MaraDNS hosted on your dev-machine (see P.S. for sample configuration).
Generate certificate (and signing key) in various formats
self_signed_template.config:
[req]
default_bits = 2048
default_md = sha256
prompt = no
default_keyfile = [hostname]_self_signed_key.pem
encrypt_key = no
distinguished_name = dn
req_extensions = v3_req
x509_extensions = v3_req
[dn]
C = PF
ST = Tahiti
L = Papeete
O = c4-soft
emailAddress = ch4mp#c4-soft.com
CN = [hostname]
[v3_req]
subjectAltName = critical, #alt_names
basicConstraints = critical, CA:false
keyUsage = critical, keyCertSign, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = critical, serverAuth, clientAuth
[alt_names]
DNS.1 = [hostname]
DNS.2 = localhost
DNS.3 = 10.0.2.2
self_signed.sh
#!/bin/bash
if [ -z "$1" ]
then
echo "Usage:"
echo ""
echo "self_signed.sh key_password [java_home] [hostname] [store_password] [certificates_directory_path] [cacerts_password]"
echo ""
echo " - java_home is defaulted to $JAVA_HOME"
echo " - hostname is defaulted to $HOSTNAME"
echo " - store_password is defaulted to key_password"
echo " - certificates_directory_path is defaulted to current diretory"
echo " - cacerts_password is defaulted to changeit"
echo ""
echo "Sample:"
echo "./self_signed.sh \"secr3!\" \"C:/Java/jdk1.8.0_281\" \"bravo-ch4mp\""
echo ""
exit 1
else
echo "#------------------------------------------"
echo "# This is a no-op script"
echo "# Copy / paste output to:"
echo "# - generate certificate files"
echo "# - import certificates into cacerts file"
echo "#------------------------------------------"
KEY_PASSWORD="${1}"
echo "# key password: $KEY_PASSWORD"
if [ -z "$2" ]
then
if [ -z "$JAVA_HOME" ]
then
echo "ERROR: could not locate java home"
exit 1
else
JAVA=$JAVA_HOME
fi
else
JAVA=$2
fi
JAVA=$(echo $JAVA | sed 's/\\/\//g')
echo "# java home: $JAVA"
if [ -f "${JAVA}/lib/security/cacerts" ]
then
# recent JDKs and JREs style
CACERTS="${JAVA}/lib/security/cacerts"
elif [ -f "${JAVA}/jre/lib/security/cacerts" ]
then
# legacy JDKs style (1.8 and older)
CACERTS="${JAVA}/jre/lib/security/cacerts"
else
echo "ERROR: could not locate cacerts under ${JAVA}"
exit 1
fi
echo "# cacerts path: $CACERTS"
if [ -z "${3}" ]
then
HOST="$HOSTNAME"
else
HOST="${3}"
fi
echo "# host (certificate CN): $HOST"
if [ -z "${4}" ]
then
STORE_PASSWORD="$KEY_PASSWORD"
else
STORE_PASSWORD="${4}"
fi
echo "# store password : $STORE_PASSWORD"
if [ -z "${5}" ]
then
CERTIF_DIR="."
else
CERTIF_DIR="${5}"
fi
echo "# certificates directory path: $CERTIF_DIR"
CERTIF_DIR=$(echo $CERTIF_DIR | sed 's/\\/\//g')
if [ -z "${6}" ]
then
CACERTS_PASSWORD="changeit"
else
CACERTS_PASSWORD="${6}"
fi
echo "# cacerts password: $CACERTS_PASSWORD"
echo "#------------------------------------------"
fi
echo ""
rm -f ${HOST}_self_signed.config;
sed 's/\[hostname\]/'${HOST}'/g' "${CERTIF_DIR}/self_signed_template.config" > "${CERTIF_DIR}/${HOST}_self_signed.config"
echo openssl req -config \"${CERTIF_DIR}/${HOST}_self_signed.config\" -new -keyout \"${CERTIF_DIR}/${HOST}_self_signed_key.pem\" -out \"${CERTIF_DIR}/${HOST}_self_signed_cert.pem\" -reqexts v3_req
echo ""
echo openssl x509 -req -days 365 -extfile \"${CERTIF_DIR}/${HOST}_self_signed.config\" -in \"${CERTIF_DIR}/${HOST}_self_signed_cert.pem\" -extensions v3_req -signkey \"${CERTIF_DIR}/${HOST}_self_signed_key.pem\" -out \"${CERTIF_DIR}/${HOST}_self_signed.crt\"
echo ""
echo openssl pkcs12 -export -in \"${CERTIF_DIR}/${HOST}_self_signed.crt\" -inkey \"${CERTIF_DIR}/${HOST}_self_signed_key.pem\" -name ${HOST}_self_signed -password pass:${KEY_PASSWORD} -out \"${CERTIF_DIR}/${HOST}_self_signed.pfx\"
echo ""
echo \"${JAVA}/bin/keytool\" -importkeystore -srckeystore \"${CERTIF_DIR}/${HOST}_self_signed.pfx\" -srcstorepass \"${STORE_PASSWORD}\" -srcstoretype pkcs12 -srcalias ${HOST}_self_signed -destkeystore \"${CERTIF_DIR}/${HOST}_self_signed.jks\" -deststoretype PKCS12 -deststorepass ${STORE_PASSWORD} -destalias ${HOST}_self_signed
echo ""
echo \"${JAVA}/bin/keytool\" -importkeystore -srckeystore \"${CERTIF_DIR}/${HOST}_self_signed.pfx\" -srcstorepass \"${STORE_PASSWORD}\" -srcstoretype pkcs12 -srcalias ${HOST}_self_signed -destkeystore \"${CACERTS}\" -deststorepass ${CACERTS_PASSWORD} -destalias ${HOST}_self_signed
echo ""
Then run something like ./self_signed.sh "secr3!" C:/Java/jdk1.8.0_281.
Execute ./self_signed.sh for each of your JDKs / JREs and then
Simply copy / paste / run all output commands at 1st execution and last command only (import certificate in JDK / JRE cacerts file) from 2nd execution on (otherwise you'll loose previous certificates).
Admin privileges could be required to import the certificate in Java's cacerts.
On windows, Git Bash has all of sed, openssl and keytool on the path.
Import this certificate as trusted root authority
If you add this certificate to trusted root authorities, your browser will display no error nor warning when navigating URLs like https://[hostname]:....
On windows, this can be done with certmgr.msc (right click trusted root authorities and then import). Please comment if you successfully do the same on other OS.
Configure Ionic-Angular to serve over https with this certificate
Edit angular.json to set "sslCert" and "sslKey" under your-project/architect/serve/options/ and point it to respectively [hostname]_self_signed.crt and [hostname]_self_signed_key.pem generated earlier.
This is enough for the right certificate to be picked when running ionic serve --ssl --host=[hostname] or ionic capacitor run android -l --ssl --host=[hostname]
Embed certificate in Android project
Reminder: android resource folder is android/app/src/main/res/ under your project or app/res/ in Android Studio
First, copy [hostname]_self_signed.crt to raw resources, replacing -, if any, with _ in hostname.
Create network_security_config.xml in xml resources (careful with modified hostname)
<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
<base-config cleartextTrafficPermitted="true">
<trust-anchors>
<certificates src="#raw/[hostname]_self_signed"/>
<certificates src="system"/>
</trust-anchors>
</base-config>
</network-security-config>
important note: if all your trafic is served over https, you should set cleartextTrafficPermitted to false (which is default value since Android 9). Consider doing so for prod build at least.
Finally, edit AndroidManifest.xml and add android:networkSecurityConfig="#xml/network_security_config" to your <application > tag
Embed certificate in iOS project
I have no experience with iOS, please feel free to comment or add an answer if you get it working.
Configure your backends to serve over https with self-signed certificate
Well... it really depends on the stack you use. A few samples:
For Kestrel (.Net app debugged in Visual Studio), set ASPNETCORE_Kestrel__Certificates__Default__Password and ASPNETCORE_Kestrel__Certificates__Default__Path, the second pointing to the [hostname]_self_signed.pfx
For spring-boot, copy [hostname]_self_signed.jks into src/main/resources/ and set server.ssl properties
Keycloak has comprensive doc to setup the server with custom certificate
Please comment if you get other backend types working
P.S. The dwood3rc.txt file I use for MaraDNS:
#upstream_servers = {}
#upstream_servers["."]="8.8.8.8, 8.8.4.4" # Servers we connect to
root_servers = {}
# ICANN DNS root servers
root_servers["."]="198.41.0.4, 199.9.14.201, 192.33.4.12, 199.7.91.13,"
root_servers["."]+="192.203.230.10, 192.5.5.241, 192.112.36.4, "
root_servers["."]+="198.97.190.53, 192.36.148.17, 192.58.128.30, "
root_servers["."]+="193.0.14.129, 199.7.83.42, 202.12.27.33"
# local DNS server
root_servers["bravo-ch4mp."]="192.168.1.181"
root_servers["local."]="192.168.1.181"
# The IP this program has
bind_address="127.0.0.1, 192.168.1.181, 192.168.1.132"
# The IPs allowed to connect and use the cache
recursive_acl = "127.0.0.1/16, 192.168.0.1/16"
chroot_dir = "/etc/maradns"
# This is the file Deadwood uses to read the cache to and from disk
cache_file = "dw_cache_bin"
filter_rfc1918 = 0
ip4 = {}
ip4["bravo-ch4mp."] = "192.168.1.181"
ip6 = {}
Once the DNS server is up (net start deadwood) on my dev machine (and also firewall configured...), I configure clients to use it as primary DNS (edit wifi network properties which does not require rooted device) et voilĂ !
P.S.2 Keycloak standalone configuration allowing test devices to connect to OpenId endpoints over https
Copy [hostname]_self_signed.jks to standalone/configuration/.
Edit standalone/configuration/standalone.xml to replace ${jboss.bind.address:127.0.0.1} with ${jboss.bind.address:0.0.0.0}. Save and close.
Start Keycloak with bin/standalone[.bat|.sh], then using bin/jboss-cli[.bat|.sh]:
connect
/subsystem=keycloak-server/spi=hostname/provider=default:write-attribute(name=properties.frontendUrl,value="https://[hostname]:8443/auth")
/core-service=management/security-realm=UndertowRealm:add()
/core-service=management/security-realm=UndertowRealm/server-identity=ssl:add(keystore-path=[hostname]_self_signed.jks, keystore-relative-to=jboss.server.config.dir, keystore-password=[keystore_password])
/subsystem=undertow/server=default-server/https-listener=https:write-attribute(name=security-realm, value=UndertowRealm)
reload

Created scala docker image using sbt native plugin. I have to add LDAP CA cert. How to add that in the image

Scala docker plug in creates the docket image but how to add LDAP CA certificate in the image. Is there any property to set the LDAP certificate. My application requires ldap authentication during login
i achieved it by
dockerCommands+=Cmd("USER", "root")
dockerCommands += Cmd("RUN" ,"apk update && apk add openssl")
dockerCommands += Cmd("RUN" ,"apk add --no-cache curl")
dockerCommands += Cmd("RUN" ,"curl http://crl.xyz.com/xCertBundle.p7b | openssl pkcs7 -
print_certs -outform PEM -out CertBundle.pem")
dockerCommands+= Cmd("RUN" , "keytool -importcert -noprompt -alias 'xyzBundle' -file
CertBundle.pem -storepass changeit -keystore /usr/lib/jvm/java-1.8-
openjdk/jre/lib/security/cacerts")
You need to add a mapping for your certificate file:
https://sbt-native-packager.readthedocs.io/en/latest/formats/universal.html#universal-plugin-getting-started-with-packaging

Faild when I try to send an email in Jenkins

I'm trying to send an email in Jenkins. I'm doing next steps:
Jenkins -> Configure System
In "E-mail Notification" area I'm providing next data, click "Test Configuration" and many errors appears:
Could you please help me with this issue?
It looks like you need gmail smtp server's certificate
Please try the following steps from http://notepad2.blogspot.com/2012/04/import-gmail-certificate-into-java.html (1)
Copied from the (1):
"
The following procedures are to import the gmail smtp certificate into the default Java keystore (Depends on the java mail application, the location of keystore may be vary):
Connect to smtp.gmail.com:465 to display the certificate in a terminal window:
For Linux:
openssl s_client -connect smtp.gmail.com:465
For Mac OS:
openssl s_client -connect smtp.gmail.com:465
For Windows
Install openssl first
Run command:
s_client -connect smtp.gmail.com:465
Copy and save the lines between "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" into a file, say, gmail.cert
Import the certificate into java keystore(Default location):
sudo keytool -import -alias smtp.gmail.com -keystore /path/to/keystore -file /Users/wilson/gmail.cert
For Windows:
keytool -import -alias smtp.gmail.com -keystore "%JAVA_HOME%/jre/lib/security/cacerts" -file C:\Users\wilson\gmail.cert
For Mac OS:
sudo keytool -import -alias smtp.gmail.com -keystore /System/Library/Frameworks/JavaVM.framework//Versions/CurrentJDK/Home/lib/security/cacerts -file /Users/wilson/gmail.cert
For Linux:
sudo keytool -import -alias smtp.gmail.com -keystore $JAVA_HOME/jre/lib/security/cacerts -file /Users/wilson/gmail.cert
Note: your need to provide the password to access the keystore. The password for the default java keystore is changeit
Answer Yes when it ask "Trust this certificate? [no]: yes"
Note: if your java mail client application uses its own keystore, you need to change the location of the application's keystore rather than JVM's keystore in the keytool command.
"

How to configure a Play application to use Let's Encrypt certificate and to convert let's encrypt certificate so that play can understand it?

I have been using play framework as my server and react webpage as a client. I have already created and set up certificate for react webpage using letsencrypt. But, now i have to configure this certificate work also for play application. How can i configure application.conf to use it?
I myself found answer answer to my Question.
First create certificate for nginx from https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04#step-5-enabling-the-changes-in-nginx
After creating certificate, follow following step:
1) First stop your play server
2) Do sudo su
3) Do ssh and go to the location where your sslcertificate is which will be like: /etc/letsencrypt/live/example.com
4)Type following command
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out cert_and_key.p12 -CAfile chain.pem -caname root -passout pass:your_password
5)Type following command
keytool -importkeystore -srcstorepass your_password -destkeystore keyStore.jks -srckeystore cert_and_key.p12 -srcstoretype PKCS12 -storepass your_password
6) After .jks is created put following code in your application.conf file:
play.crypto.secret="changethissosomethingsecret"
play.server.https.keyStore.path = "Path to your .jks file"
play.server.https.keyStore.type = "JKS"
play.server.https.keyStore.password = "yourKeyStorePassword"
5) Now change your play run command to following
nohup /home/ubuntu/webserver/manpowercompany-1.0-SNAPSHOT/bin/manpowercompany -J-Xms128M -J-Xmx1024M -Dplay.crypto.secret=anyKey -Dhttps.port=9002 &
Note: Change the directory according to your file locations
6) Start the server with the file containing above run command. You might need to do sudo.
7) You are good to go. Now your server is served as https at port 9002
Above procedure are the step i did according to my project. So, do reference this and make changes according to your project.
Thank you!!!

Github peer not authenticated when I issue g8 command

I am using red hat linux. I am trying to run this command:
g8 typesafehub/play-scala
And I am getting this response:
Exception fetching from github peer not authenticated
But when I check the connection using
openssl s_client -connect github.com:443
I get this:
Verify return code: 0 (ok)
Which means that I am able to connect with github. Why doesn't this command work?
g8 typesafehub/play-scala
I also ran into this issue on an RHEL 5 VM image where I am using openjdk 6. It was the other note to look at TrustManager clued me in on a fix. I tweak the invocation to add a trust setting for github; in my situation it resolves the peer authentication issue.
First grab the github certificate using openssl and keytool to make it accessible to java.
echo "" | openssl s_client -connect www.github.com:443 \
-showcerts 2>/dev/null | openssl x509 -out github.cert
keytool -import -alias github \
-file github.cert -storepass g8g8g8 \
-keystore $HOME/g8.truststore
Now to rewrite the invocation with a script I call "G8":
g8 \
\ -Djavax.net.ssl.trustStore=$HOME/g8.truststore \
\ -Djavax.net.ssl.trustStorePassword=g8g8g8 \
$*
Now try executing G8 -v typesafehub/akka-scala-sbt and I see things are much happier now. I imagine setting a systemwide default truststore would may be better but I haven't figured that one out yet.
If it really is an authentication issue, check your ~/.g8/config file for authentication purpose, but you shouldn't need it for anonymous access.
Note that, according to issue 32 of giter8, it can also depends on the Java you are using.
For instance:
Sorry, that preview release of openjdk 7 is not fit for general use. (There's also giter8 issue #27 specific to openjdk on mac.) I have tested openjdk 7~b147-2.0-0ubuntu0.11.10.1 with giter8 and that worked fine, so when there is a final release available for mac you should be able to use it.
For now, please try with jdk 6 and reopen if you are still having trouble.
Another JDK (openjdk) might end up using the wrong TrustManager, as described in "Avoiding the "javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated" with HttpClient"
I had the same issue as B Evans (thanks for this!), but in Windows, so here is the equivalent code in case someone else has this issue and doesn't know how to do it from windows cmd. I also had to get openssl from http://www.openssl.org/related/binaries.html
openssl s_client -connect www.github.com:443 -showcerts > out.txt
openssl x509 -out github.cert < out.txt
keytool -import -alias github -file github.cert \
-storepass g8g8g8 -keystore C:\tmp\g8.truststore
Then add the same to JAVA_OPTS (I also had to deal with our corporate firewall and hence proxy as well...)
SET JAVA_OPTS=-Dhttp.proxyHost=our.proxy.com -Dhttp.proxyPort=8080 \
-Dhttps.proxyHost=our.proxy.com -Dhttps.proxyPort=8080 \
-Djavax.net.ssl.trustStore=C:\tmp\g8.truststore \
-Djavax.net.ssl.trustStorePassword=g8g8g8