I am trying to put some data in kudu, but the worker cannot find the kerberos token, so I am not able to put some data into the kudu database.
here you can see my spark2-submit statement
spark2-submit --master yarn "spark.yarn.maxAppAttempts=1" --conf "spark.authenticate=true" --deploy-mode cluster --properties-file ./custom.properties --keytab /home/stream_ingest/stream_ingest.keytab --principal xxx#xxx --class app.test.Task
the exception is the following:
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
ERROR: exception occured during an insert to kudu, sending record ConsumerRecord(topic = custom, partition = 0, offset = 120, CreateTime = 1528444250889, checksum = 3907376269, serialized key size = -1, serialized value size = 1067790, key = null, value = [B#1623bffa) to error topic custom
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx:7051,xxx:7051,xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:352)
at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:222)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:246)
at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:365)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:354)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:246)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:300)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:221)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:60)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:132)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:128)
at org.apache.kudu.client.Connection.cleanup(Connection.java:715)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:460)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.client.Connection.handleUpstream(Connection.java:244)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at org.apache.kudu.shaded.org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
It looks like that the kudu client does not find the keytab file. In the kudu documentation they mentioned, that you only have to specify the keytab and principal parameter.
If you execute a openTable on the KuduClient in the driver, everything works fine.
Related
I trying to make Apache Kafka protected using SASL_SSL and GSSAPI mecanism. Everythnig is workin properly apart from the fact that the Authentication names used by Kafka are the "pre-windows 2000" formatted names instead of the "standard" new ones.
For instance, I declare a new kafka broker in our Active Directory (I forgot to say that it's a Windows 10 version...):
User logon Name: kafka/kafka1.myfqdn.com#MYFQDN.COM
User logon name (pre-Windows 2000): FAKE_USER1
When I login into kafka using this user keytab, I have this into the logs:
[2020-11-21 17:05:50,168] INFO Successfully authenticated client: authenticationID=FAKE_USER1#MYFQDN.COM; authorizationID=kafka/kafka1.myfqdn.com#MYFQDN.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2020-11-21 17:09:50,909] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2020-11-21 17:12:00,672] INFO Successfully authenticated client: authenticationID=FAKE_USER1#MYFQDN.COM; authorizationID=kafka/kafka1.myfqdn.com#MYFQDN.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2020-11-21 17:12:00,772] INFO Successfully authenticated client: authenticationID=FAKE_USER1#MYFQDN.COM; authorizationID=kafka/kafka1.myfqdn.com#MYFQDN.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2020-11-21 17:12:00,799] DEBUG No acl found for resource ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL), authorized = false (kafka.authorizer.logger)
[2020-11-21 17:12:00,799] INFO Principal = User:FAKE_USER1 is Denied Operation = DescribeConfigs from host = xxx.xxx.xxx.xxx on resource = Cluster:LITERAL:kafka-cluster for request = DescribeConfigs with resourceRefCount = 1 (kafka.authorizer.logger)
Of course, the Denied at the end is normal because my rules expect to extract "kafka" from the kafka/kafka1.myfqdn.com#MYFQDN.COM user.
Could you tell me what I do not do properly ?
When starting up the windows service for the agent following error is logged in the event log:
Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.
(Level: Error; Source: VstsAgentService)
The agents are configured using a PAT.
The windows services are run as Network Service.
Configured according to: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops
The agents have been running successfully for weeks but s omething has broken our agents. Of course it will afterwards be connected to some type of change. Thanks in advance for any troubleshooting hints.
Extractions from Agent_20200421-125954-utc.log.
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2020-04-21 12:59:58Z WARN VisualStudioServices] Authentication failed with status code 401.
...
[2020-04-21 12:59:58Z ERR VisualStudioServices] POST request to https://dev.azure.com/{organization}/_apis/distributedtask/pools/1/sessions is not authorized. Details: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR MessageListener] Catch exception during create session.
[2020-04-21 12:59:58Z ERR MessageListener] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z INFO MessageListener] Non-retriable exception: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR Terminal] WRITE ERROR: Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.
Something has broken our agents.
Since completely know nothing to what are the changes and how's the broken of agent now, just give some possible suggestion and you can have a try.
Go D:\agent5, and open .credentials with text.
Make sure its content is like below:
{
"scheme": "OAuth",
"data": {
"clientId": "xxxxxx", # This are generated automatically by
"authorizationUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token",
"oauthEndpointUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token"
}
}
Then save it, and try to re-run agent again.
If this way still could not make your agent available, or the .credentials file has contained the contents I mentioned above, now I guess the broken should caused on .credentials_rsaparams file.
.credentials_rsaparams is a file which has a RSA private key for OAuth token exchange between agent and server. The server hold the public key, and the server will send a new token to agent encrypted by the public key every 50 mins.
It is the file which generated by server automatically, which means you can not modified it manually since you don't know what you should input.
At present, you'd better re-configure this agent by following this steps:
1) Run Command line/Powershell line by using Run as Administrator, then cd D:\agent5.
2) Run .\config remove. And do agent configuration steps after the remove succeed.
When using Vault Agent with a secret ID file, I received the following error message:
$ ./vault agent --config auth_config.hcl
==> Vault server started! Log data will stream in below:
==> Vault agent configuration:
Api Address 1: http://127.0.0.1:8300
Cgo: disabled
Log Level: info
Version: Vault v1.3.0
2020-02-04T14:08:28.352-0800 [INFO] auth.handler: starting auth handler
2020-02-04T14:08:28.352-0800 [INFO] auth.handler: authenticating
2020-02-04T14:08:28.352-0800 [INFO] sink.server: starting sink server
2020-02-04T14:08:28.352-0800 [INFO] template.server: starting template server
2020-02-04T14:08:28.352-0800 [INFO] template.server: no templates found
2020-02-04T14:08:28.352-0800 [INFO] template.server: template server stopped
2020-02-04T14:08:28.354-0800 [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://127.0.0.1:8200/v1/auth/approle/login
Code: 400. Errors:
* invalid secret id" backoff=2.190384035
The command I executed was:
vault agent --config auth_config.hcl
The contents of my auth_config.hcl file is:
vault {
address = "http://127.0.0.1:8200"
}
auto_auth {
method "approle" {
config {
role_id_file_path = "./role_id"
secret_id_file_path = "./secret_id"
remove_secret_id_file_after_reading = false
}
}
}
cache {
use_auto_auth_token = true
}
listener "tcp" {
address = "127.0.0.1:8300"
tls_disable = true
}
My secret ID was generated using the following command:
vault write -f auth/approle/role/payments_service/secret-id -format=json | sed -E -n 's/.*"secret_id": "([^"]*).*/\1/p' > secret_id
Why is this error happening?
I found that the usual reason that this happens because the secret ID file wasn't generated correctly in the first place. See this Github thread for example. Unfortunately, in my case, the file was generated. The file secret_id referenced in auth_config.hcl contained the secret ID.
In my case, the problem was that after I generated the file, secret_id, I executed the command vault write -f auth/approle/role/payments_service/secret-id a second time. This new command didn't write over the original file with a new secret ID. The consequence of this new command was that it respawned a new secret ID which invalidated the previous secret ID which was written to the secret_id file.
My solution was to rerun the command that wrote the secret ID to the file, secret_id, and then immediately run the Vault Agent. Problem solved.
My case was because the app (kes) was trying to use http, instead of https, to connect to vault, while the tls was enabled both in vault and the app (kes). Once it was updated, the app could connect to vault without any issue
Error: failed to connect to Vault: Error making API request.
URL: PUT http://vault.vault:8200/v1/auth/approle/login
Code: 400. Raw Message:
Client sent an HTTP request to an HTTPS server.
Authenticating to Hashicorp Vault 'http://vault.vault:8200'
I have been building and creating templates for google dataflow for over a year now. I never had a problem creating templates and uploading them to gcs with the options.setTemplateLocation(templatePath); call. Since today, when creating the Pipeline with Pipeline.create(options); and running the java-program in eclipse, I get following exception:
Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
at mypackage.PipelineCreation.getTemplatePipeline(PipelineCreation.java:34)
at myotherpackage.Main.main(Main.java:51)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 5 more
Caused by: java.lang.RuntimeException: Unable to verify that GCS bucket gs://my-projects-staging-bucket exists.
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:92)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:61)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:228)
... 10 more
Caused by: com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error" : "invalid_grant",
"error_description" : "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:207)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize(ChainingHttpRequestInitializer.java:52)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call(ResilientOperation.java:166)
at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66)
at org.apache.beam.sdk.util.GcsUtil.getBucket(GcsUtil.java:505)
at org.apache.beam.sdk.util.GcsUtil.bucketAccessible(GcsUtil.java:492)
at org.apache.beam.sdk.util.GcsUtil.bucketAccessible(GcsUtil.java:457)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:88)
... 12 more
I was logged-in today with another account into gcloud but logged in again with the account associated with the project as "Owner" with gcloud auth login.
I also restarted Eclipse but the same error keeps occuring. Also when trying to run the pipeline locally, I get another error but also with the "invalid_grant" "bad request" content. Restarting the laptop also had no effect.
My pom defines the google-cloud-dataflow-java-sdk-all with version 2.2.0 and upgrading to 2.5.0 had no effect.
I am able to copy data to the bucket with gsutil from commandline. But when running the java-program from command-line with mvn compile exec:java -Dexec.mainClass=mypackage.Main i still get the same errors.
My function to create a templatePipeline looks like the following:
public static Pipeline getTemplatePipeline(String jobName, String templatePath){
DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
options.setProject("my-project-id");
options.setRunner(DataflowRunner.class);
options.setStagingLocation("gs://my-projects-staging-bucket/binaries");
options.setTempLocation("gs://my-projects-staging-bucket/binaries/tmp");
options.setGcpTempLocation("gs://my-projects-staging-bucket/binaries/tmp");
options.setZone("europe-west3-a");
options.setWorkerMachineType("n1-standard-2");
options.setJobName(jobName);
options.setMaxNumWorkers(2);
options.setDiskSizeGb(40);
options.setTemplateLocation(templatePath);
return Pipeline.create(options);
}
Any help is highly appreciated.
You don't have to use service account and still you can use gcloud, you should use the following command and login with your account:
gcloud auth application-default login
I found the solution in the quickstart docs.
It seems like the gcloud auth is no longer used and you have to use a service account. So like in the docs I created a service account with role "project/owner" and downloaded it's json file to $path.
Then on my Mac i used export GOOGLE_APPLICATION_CREDENTIALS="$path" and within the same session used the command mentioned in the question to compile and execute the java-program.
Suddenly I am getting an SSL error message when I am trying to access a dashDb from an auto-configured liberty server, from somewhere deep in the DB2 driver. I have verified in the deployed files that the default keystore is auto-configured into the liberty server.
What is happening here?
java.security.cert.CertPathValidatorException: The certificate issued by CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US is not trusted; internal cause is:
[ERROR ] CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=*.services.dal.bluemix.net, O=International Business Machines Corporation, L=Armonk, ST=New York, C=US was sent from the target host. The signer might need to be added to local trust store /home/vcap/app/wlp/usr/servers/BluemixServer/resources/security/key.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is
[err] java.sql.SQLNonTransientException: [jcc][t4][2030][11211][4.19.49] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream. Error location: Reply.fill() - socketInputStream.read (-1). Message: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499
[err] at com.ibm.db2.jcc.am.kd.a(Unknown Source)
There was a change to dashDB last Friday which enhanced security requirements for cipher specs of applications accessing dashDB. If your application was working before last week and is not now, you may need to update your cipher.
Please refer to IBM technical report via this link
We can connect to dashDB with one of the following Liberty for java buildpacks. Please try redeploying your application and make sure that database URI has :sslConnection=true at the end.
Build packs
buildpack_liberty-for-java_v3.8-20170308-1507.zip (newest)
buildpack_liberty-for-java_v3.4.1-20161030-2241.zip (oldest)