Jenkins/Groovy: Why does withCredentials() introduce a NotSerializableException exception? - rest

Question

This question is a follow-up to Groovy/Jenkins: how to refactor sh(script:"curl ...") to URL?.
Though I intend to try invoking REST API with HTTP Request - as the one answerer has suggested - I'd like to also specifically learn about/understand the original problem RE: non-serializability.
I've reduced the code in the linked Q&A to more minimally demonstrate the problem.
This code:
#Library('my-sandbox-libs#dev') sandbox_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { echo "hello" } } }
stage( "2" ) {
steps {
script {
try {
my_lib.v4()
}
catch(Exception e) {
echo "Jenkinsfile: ${e.toString()}"
throw e
}
}
}
}
stage( "3" ) { steps { script { echo "world" } } }
}
}
// vars/my_lib.groovy
import groovy.json.JsonOutput
def v4() {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
...generates HTTP error 401. This is expected, because it invokes a Bitbucket REST API without necessary authentication.
What's missing is the "Bearer: xyz" secret-text. In the past, I've gotten this secret-text using Jenkins'/Groovy's withCredentials function, as in the modified function v4() below:
// vars/my_lib.groovy
import groovy.json.JsonOutput
def v4() {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
post.setRequestProperty("Authorization", "Bearer " + auth_token)
}
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
...but I've discovered that the addition of that withCredentials-block, specifically, introduces a java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl runtime exception.
At the linked Stack Overflow Q&A, one commenter has informed me that the problem has to do with serializable vs. non-serializable objects in this function. But I'm having a hard time understanding what non-serializable object is introduced by use of the withCredentials-block.
As a point of reference, use of the same withCredentials-block works just fine when I invoke the same REST API using curl instead of "native" Jenkins/Groovy functions. I.e. the following code works fine:
def v1() {
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
def cmd = "curl -f -L " +
"-H \"Authorization: Bearer ${auth_token}\" " +
"-H \"Content-Type:application/json\" " +
"-X POST https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d " +
"-d \'${JsonOutput.toJson(dict)}\'"
sh(script: cmd, returnStatus: true)
}
}
So, in summary, this question is why does withCredentials() introduce non-serializable objects (and what is the non-serializable object), which causes NonSerializableException exceptions with use of URL and/or HttpURLConnection, and does one work around this?
I'm not unwilling to use a different solution, such as httpRequest objects, but this question is about learning the nature of this specific problem, and how to work around it with the existing objects, i.e. URL and HttpURLConnection objects.
Update: As it turns out, I'm unable to use the suggested alternate solution using httpRequest objects, because the HTTP Request plugin is only available to Jenkins versions 2.222.4 or newer, which our Jenkins does not meet. It's outside my privilege to update our Jenkins version, and basically I'll need to assume an inability to upgrade Jenkins.
Our Jenkins version is 2.190.3, which has Groovy version 2.4.12.

Answer 1

When having such issues I usually try to put the "low level" code that calls down to Groovy/Java API, into #NonCPS functions. Objects in such functions don't need to be serializable, so we can freely use any Groovy/Java API.
Background reading: Pipeline CPS Method Mismatches
Make sure you don't call any Jenkins pipeline steps from #NonCPS functions (nor any other function that is not marked #NonCPS) - such code could silently fail or do unexpected stuff! (There are some "safe" functions like echo though.)
As withCredentials is a pipeline step, it has to be called from a "regular" function (not marked as #NonCPS), one level up the call chain.
Note that I'm passing auth_token as an argument to v4_internal. If you need other Jenkins variables in the code, these should also be passed as arguments.
// vars/my_lib.groovy
import groovy.json.JsonOutput
def v4() {
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
v4_internal(auth_token)
}
}
#NonCPS
def v4_internal( def auth_token ) {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
post.setRequestProperty("Authorization", "Bearer " + auth_token)
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}

Answer 2

This sucks - and I hope someone comes along with a better answer - but it looks like the only way I can pull this off is as follows:
#Library('my-sandbox-libs#dev') sandbox_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { echo "hello" } } }
stage( "2" ) {
steps {
script {
try {
my_lib.v5(my_lib.getBitbucketCred())
}
catch(Exception e) {
echo "Jenkinsfile: ${e.toString()}"
throw e
}
}
}
}
stage( "3" ) { steps { script { echo "world" } } }
}
}
// vars/my_lib.groovy
import groovy.json.JsonOutput
def getBitbucketCred() {
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
return auth_token
}
}
def v5(auth_token) {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
req.setRequestProperty("Authorization", "Bearer " + auth_token)
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
Specifically, I must invoke withCredentials() completely separate from the scope of the function that uses URL and/or HttpURLConnection.
I don't know if this is considered acceptable in Jenkins/Groovy, but I'm dissatisfied by the inability to call withCredentials() from the v5() function itself. I'm also unable to call a withCredentials()-wrapper function from v5().
When I was trying to call withCredentials() either directly in v5() or from a wrapper function called by v5(), I tried every combination of #NonCPS between v5() and the wrapper function, and that didn't work. I also tried explicitly setting the URL and HttpURLConnection objects to null before the end of the function (as suggested at Jenkins/Groovy: Why does withCredentials() introduce a NotSerializableException exception?), and neither did that work.
I'd be disappointed in Jenkins/Groovy if this is the only solution. This feels like an artificial limit on how one can choose to organize his code.
Updating with more detail in response to #daggett:
RE: calling withCredentials() directly from my_lib.v5() or calling it from a wrapper function, let's start with mylib.groovy set up as follows (let me also take the opportunity to give the functions better names):
def withCredWrapper() {
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
return auth_token
}
}
def callRestFunc() {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
// version 1:
withCredentials([string(credentialsId: 'bb_auth_bearer_token_cred_id',
variable: 'auth_token')]) {
post.setRequestProperty("Authorization", "Bearer " + auth_token)
}
// version 2:
//post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
With the above code, function callRestFunc() can either call withCredentials() directly, as above, or indirectly by the wrapper function withCredWrapper(), i.e.:
...
// version 1:
//withCredentials([string(credentialsId: 'bb_auth_bearer_token_cred_id',
// variable: 'auth_token')]) {
// post.setRequestProperty("Authorization", "Bearer " + auth_token)
//}
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
Further, #NonCPS can be applied to one of withCredWrapper() or callRestFunc(), both, or neither.
Below are the specific failures with all 8 combinations thereof:
1.
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
2.
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/ Masking supported pattern matches of $auth_token, Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
3.
#NonCPS
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
4.
#NonCPS
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/ Masking supported pattern matches of $auth_token, Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
5.
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
6.
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching my_lib.withCredWrapper; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/
7.
#NonCPS
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: expected to call my_lib.withCredWrapper but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/ Masking supported pattern matches of $auth_token, Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
8.
#NonCPS
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/