Kerberos authentication (GSSAPI) in Apache Kafka 6.0 uses Pre windows 2000 names format - apache-kafka

I trying to make Apache Kafka protected using SASL_SSL and GSSAPI mecanism. Everythnig is workin properly apart from the fact that the Authentication names used by Kafka are the "pre-windows 2000" formatted names instead of the "standard" new ones.
For instance, I declare a new kafka broker in our Active Directory (I forgot to say that it's a Windows 10 version...):
User logon Name: kafka/kafka1.myfqdn.com#MYFQDN.COM
User logon name (pre-Windows 2000): FAKE_USER1
When I login into kafka using this user keytab, I have this into the logs:
[2020-11-21 17:05:50,168] INFO Successfully authenticated client: authenticationID=FAKE_USER1#MYFQDN.COM; authorizationID=kafka/kafka1.myfqdn.com#MYFQDN.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2020-11-21 17:09:50,909] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2020-11-21 17:12:00,672] INFO Successfully authenticated client: authenticationID=FAKE_USER1#MYFQDN.COM; authorizationID=kafka/kafka1.myfqdn.com#MYFQDN.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2020-11-21 17:12:00,772] INFO Successfully authenticated client: authenticationID=FAKE_USER1#MYFQDN.COM; authorizationID=kafka/kafka1.myfqdn.com#MYFQDN.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2020-11-21 17:12:00,799] DEBUG No acl found for resource ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL), authorized = false (kafka.authorizer.logger)
[2020-11-21 17:12:00,799] INFO Principal = User:FAKE_USER1 is Denied Operation = DescribeConfigs from host = xxx.xxx.xxx.xxx on resource = Cluster:LITERAL:kafka-cluster for request = DescribeConfigs with resourceRefCount = 1 (kafka.authorizer.logger)
Of course, the Denied at the end is normal because my rules expect to extract "kafka" from the kafka/kafka1.myfqdn.com#MYFQDN.COM user.
Could you tell me what I do not do properly ?

Related

"SchemaRegistryException: Failed to get Kafka cluster ID" for LOCAL setup

I'm downloaded the .tz (I am on MAC) for confluent version 7.0.0 from the official confluent site and was following the setup for LOCAL (1 node) and Kafka/ZooKeeper are starting fine, but the Schema Registry keeps failing (Note, I am behind a corporate VPN)
The exception message in the SchemaRegistry logs is:
[2021-11-04 00:34:22,492] INFO Logging initialized #1403ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2021-11-04 00:34:22,543] INFO Initial capacity 128, increased by 64, maximum capacity 2147483647. (io.confluent.rest.ApplicationServer)
[2021-11-04 00:34:22,614] INFO Adding listener: http://0.0.0.0:8081 (io.confluent.rest.ApplicationServer)
[2021-11-04 00:35:23,007] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)
io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryException: Failed to get Kafka cluster ID
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.kafkaClusterId(KafkaSchemaRegistry.java:1488)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:166)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:71)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:90)
at io.confluent.rest.Application.configureHandler(Application.java:271)
at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:245)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.kafkaClusterId(KafkaSchemaRegistry.java:1486)
... 7 more
My schema-registry.properties file has bootstrap URL set to
kafkastore.bootstrap.servers=PLAINTEXT://localhost:9092
I saw some posts saying its the SchemaRegistry unable to connect to the KafkaCluster URL because of the localhost address potentially. I am fairly new to Kafka and basically just need this local setup to run a git repo that is utilizing some Topics/Kafka so my questions...
How can I fix this (I am behind a corporate VPN but I figured this shouldn't affect this)
Do I even need the SchemaRegistry?
I ended up just going with the Docker local setup inside, and the only change I had to make to the docker compose YAML was to change the schema-registry port (I changed it to 8082 or 8084, don't remember exactly but just an unused port that is not being used by some other Confluent service listed in the docker-compose.yaml) and my local setup is working fine now

Azure DevOps Services - self hosted agents have gone Offline

When starting up the windows service for the agent following error is logged in the event log:
Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.
(Level: Error; Source: VstsAgentService)
The agents are configured using a PAT.
The windows services are run as Network Service.
Configured according to: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops
The agents have been running successfully for weeks but s omething has broken our agents. Of course it will afterwards be connected to some type of change. Thanks in advance for any troubleshooting hints.
Extractions from Agent_20200421-125954-utc.log.
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2020-04-21 12:59:58Z WARN VisualStudioServices] Authentication failed with status code 401.
...
[2020-04-21 12:59:58Z ERR VisualStudioServices] POST request to https://dev.azure.com/{organization}/_apis/distributedtask/pools/1/sessions is not authorized. Details: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR MessageListener] Catch exception during create session.
[2020-04-21 12:59:58Z ERR MessageListener] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z INFO MessageListener] Non-retriable exception: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR Terminal] WRITE ERROR: Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.
Something has broken our agents.
Since completely know nothing to what are the changes and how's the broken of agent now, just give some possible suggestion and you can have a try.
Go D:\agent5, and open .credentials with text.
Make sure its content is like below:
{
"scheme": "OAuth",
"data": {
"clientId": "xxxxxx", # This are generated automatically by
"authorizationUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token",
"oauthEndpointUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token"
}
}
Then save it, and try to re-run agent again.
If this way still could not make your agent available, or the .credentials file has contained the contents I mentioned above, now I guess the broken should caused on .credentials_rsaparams file.
.credentials_rsaparams is a file which has a RSA private key for OAuth token exchange between agent and server. The server hold the public key, and the server will send a new token to agent encrypted by the public key every 50 mins.
It is the file which generated by server automatically, which means you can not modified it manually since you don't know what you should input.
At present, you'd better re-configure this agent by following this steps:
1) Run Command line/Powershell line by using Run as Administrator, then cd D:\agent5.
2) Run .\config remove. And do agent configuration steps after the remove succeed.

SQL Server 2012 service not starting automatically

I have multiple SQL Server instance and I want to start service automatically for each when system starts. So I choose services state to automatically but it does not start automatically.
Here is the error log of one of the SQL Server instances
2018-07-12 17:55:30.96 Server SQL Server detected 1 sockets with 2 cores per socket and 4 logical processors per socket, 4 total logical processors; using 4 logical processors based on SQL Server licensing. This is an informational message; no user action is required.
2018-07-12 17:55:30.96 Server SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2018-07-12 17:55:30.97 Server Detected 16155 MB of RAM. This is an informational message; no user action is required.
2018-07-12 17:55:30.97 Server Using conventional memory in the memory manager.
2018-07-12 17:55:47.27 Server This instance of SQL Server last reported using a process ID of 4400 at 12/07/2018 5:26:22 PM (local) 12/07/2018 9:26:22 AM (UTC). This is an informational message only; no user action is required.
2018-07-12 17:55:47.54 Server Node configuration: node 0: CPU mask: 0x000000000000000f:0 Active CPU mask: 0x000000000000000f:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2018-07-12 17:55:47.56 Server Using dynamic lock allocation. Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node. This is an informational message only. No user action is required.
2018-07-12 17:55:47.59 Server Software Usage Metrics is disabled.
2018-07-12 17:55:48.57 spid1s Starting up database 'master'.
2018-07-12 17:55:49.53 Server CLR version v4.0.30319 loaded.
2018-07-12 17:55:53.62 Server Common language runtime (CLR) functionality initialized using CLR version v4.0.30319 from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\.
2018-07-12 17:56:09.76 spid1s SQL Server Audit is starting the audits. This is an informational message. No user action is required.
2018-07-12 17:56:09.81 spid1s SQL Server Audit has started the audits. This is an informational message. No user action is required.
2018-07-12 17:56:13.96 spid1s SQL Trace ID 1 was started by login "sa".
2018-07-12 17:56:14.92 spid1s Server name is 'NB-1608-001\SQLEXPRESS1'. This is an informational message only. No user action is required.
2018-07-12 17:56:16.22 spid12s A self-generated certificate was successfully loaded for encryption.
2018-07-12 17:56:16.28 spid12s Server is listening on [ 'any' <ipv6> 51270].
2018-07-12 17:56:16.28 spid12s Server is listening on [ 'any' <ipv4> 51270].
2018-07-12 17:56:16.33 spid12s Server local connection provider is ready to accept connection on [ \\.\pipe\SQLLocal\SQLEXPRESS1 ].
2018-07-12 17:56:16.33 spid12s Server named pipe provider is ready to accept connection on [ \\.\pipe\MSSQL$SQLEXPRESS1\sql\query ].
2018-07-12 17:56:16.33 spid12s Dedicated administrator connection support was not started because it is disabled on this edition of SQL Server. If you want to use a dedicated administrator connection, restart SQL Server using the trace flag 7806. This is an informational message only. No user action is required.
2018-07-12 17:56:16.36 spid12s SQL Server is now ready for client connections. This is an informational message; no user action is required.
2018-07-12 17:56:16.36 Server SQL Server is attempting to register a Service Principal Name (SPN) for the SQL Server service. Kerberos authentication will not be possible until a SPN is registered for the SQL Server service. This is an informational message. No user action is required.
2018-07-12 17:56:16.36 Server The SQL Server Network Interface library could not register the Service Principal Name (SPN) [ MSSQLSvc/NB-1608-001:SQLEXPRESS1 ] for the SQL Server service. Windows return code: 0xffffffff, state: 53. Failure to register a SPN might cause integrated authentication to use NTLM instead of Kerberos. This is an informational message. Further action is only required if Kerberos authentication is required by authentication policies and if the SPN has not been manually registered.
2018-07-12 17:56:16.36 Server The SQL Server Network Interface library could not register the Service Principal Name (SPN) [ MSSQLSvc/NB-1608-001:51270 ] for the SQL Server service. Windows return code: 0xffffffff, state: 53. Failure to register a SPN might cause integrated authentication to use NTLM instead of Kerberos. This is an informational message. Further action is only required if Kerberos authentication is required by authentication policies and if the SPN has not been manually registered.
2018-07-12 17:56:18.34 Logon Error: 18456, Severity: 14, State: 38.
2018-07-12 17:56:18.34 Logon Login failed for user 'NT AUTHORITY\SYSTEM'. Reason: Failed to open the explicitly specified database 'ReportServer$SQLEXPRESS1'. [CLIENT: 192.168.0.100]
2018-07-12 17:56:25.44 Logon Error: 18456, Severity: 14, State: 38.
2018-07-12 17:56:25.44 Logon Login failed for user 'NT AUTHORITY\SYSTEM'. Reason: Failed to open the explicitly specified database 'ReportServer$SQLEXPRESS1'. [CLIENT: 192.168.0.100]
2018-07-12 17:56:31.17 spid13s Error: 9954, Severity: 16, State: 1.
2018-07-12 17:56:31.17 spid13s SQL Server failed to communicate with filter daemon launch service (Windows error: Windows Error: hr = 0x8007006d(failed to retrieve text for this error)). Full-Text filter daemon process failed to start. Full-text search functionality will not be available.
2018-07-12 17:56:31.57 spid1s Starting up database 'msdb'.
First I changed to Service state to Start Delay but it does not work.
Second I changed to Log on properties and set to computer logon user's account but it does not work.
I am not sure why this error occurs and I cant find any useful resource.

kerberos authentication in Kudu for spark2 job

I am trying to put some data in kudu, but the worker cannot find the kerberos token, so I am not able to put some data into the kudu database.
here you can see my spark2-submit statement
spark2-submit --master yarn "spark.yarn.maxAppAttempts=1" --conf "spark.authenticate=true" --deploy-mode cluster --properties-file ./custom.properties --keytab /home/stream_ingest/stream_ingest.keytab --principal xxx#xxx --class app.test.Task
the exception is the following:
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
ERROR: exception occured during an insert to kudu, sending record ConsumerRecord(topic = custom, partition = 0, offset = 120, CreateTime = 1528444250889, checksum = 3907376269, serialized key size = -1, serialized value size = 1067790, key = null, value = [B#1623bffa) to error topic custom
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx:7051,xxx:7051,xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:352)
at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:222)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:246)
at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:365)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:354)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:246)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:300)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:221)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:60)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:132)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:128)
at org.apache.kudu.client.Connection.cleanup(Connection.java:715)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:460)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.client.Connection.handleUpstream(Connection.java:244)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at org.apache.kudu.shaded.org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
It looks like that the kudu client does not find the keytab file. In the kudu documentation they mentioned, that you only have to specify the keytab and principal parameter.
If you execute a openTable on the KuduClient in the driver, everything works fine.

what does these ejabberd info log messages mean?

can anyone tell me what does these log messages mean? is any session been terminated ,why?
2016-01-20 15:48:24.651 [info] <0.477.0>#ejabberd_listener:accept:333 (#Port<0.16235>) Accepted connection 192.16.35.6:1432 -> 28.4.5.2
2016-01-20 15:48:27.497 [info] <0.1411.0>#ejabberd_c2s:wait_for_feature_request:740 ({socket_state,p1_tls,{tlssock,#Port<0.16235>,#Port<0.16236>},<0.1410.0>}) Accepted authentication for 14512843168518 by ejabberd_auth_odbc from 103.233.119.62
2016-01-20 15:48:27.903 [info] <0.1411.0>#ejabberd_c2s:wait_for_session:1106 ({socket_state,p1_tls,{tlssock,#Port<0.16235>,#Port<0.16236>},<0.1410.0>}) Opened session for 14512843168518#cndivneofveofv/androidjc1PGFLG
2016-01-20 15:48:27.906 [info] <0.1355.0>#ejabberd_c2s:terminate:1768 ({socket_state,p1_tls,{tlssock,#Port<0.16227>,#Port<0.16228>},<0.1354.0>}) Replaced session for 14512843168518#cedefjwojffj/androidjc1PGFLG
Yes, the user 14512843168518#devchat.drooly.co had an open session with resource androidjc1PGFLG, and this session was terminated and replaced by a new session with the same resource.
This is a feature of XMPP: if a user's client has lost its connection to the XMPP server, but the server hasn't detected it yet, the client can force the server to terminate the previous connection by connecting again and specifying the same resource.