After upgrade Flink 1.10 to Flink 1.11, the log4j configuration is no longer working.
my previous configuration was using a library with an adapter that requires log4j 1.x and is no longer compatible with Flink 1.11
according to the new configuration, the flink-conf.yaml should look like this
log4j-console.properties: |+
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.console.ref = ConsoleAppender
rootLogger.appenderRef.rolling.ref = RollingFileAppender
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
my current configuration using log4j1 looks something similar to this
log4j-console.properties: |+
log4j.rootLogger=INFO,myappender,console
log4j.appender.myappender=com.company.log4j.MyAppender
log4j.appender.myappender.endpoints=http://
is there a way to tell Flink 1.11 to use log4j1 in the flink-conf.yaml file?
As far as I know, flink-conf.yaml does not contain log4j-console.properties section and this is a separate file. What you have specified I suppose is a part of flink-configuration-configmap.yaml cluster resource definition.
According to the flink Configuring Log4j1 Section, in order to use log4j1, you need to:
remove the log4j-core, log4j-slf4j-impl and log4j-1.2-api jars from the lib directory,
add the log4j, slf4j-log4j12 and log4j-to-slf4j jars to the lib directory,
After upgrading the Flink-1.10.2 to Flink-1.11.3 I came across the same issue in the Kubernetes and DCOS(Mesos) Flink cluster. Then to cross-verify I downloaded the Flink's binaries flink-1.11.3-bin-scala_2.12.tgz in local and tested the loggers and found it working without any change.
Flink 1.11 switched from Log4j1 to Log4j2
Then I have followed the steps mentioned in Flink's official documents to use Flink with Log4j1.
Remove the log4j-core, log4j-slf4j-impl and log4j-1.2-api jars from the Flink's lib directory.
Add the log4j, slf4j-log4j12, and log4j-to-slf4j jars to the Flink's lib directory.
Restarted the Kubernetes and DCOS(Mesos) Flink cluster and verified the loggers and found it working.
Related
I have a spark structured streaming job in scala, reading from kafka and writing to S3 as hudi tables. Now I am trying to move this job to spark operator on EKS.
When I give the option in the yaml file.
spark.jars.packages: org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,org.apache.hudi:hudi-spark3.1-bundle_2.12:0.11.1
But still I get the error at both the driver and executor
java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition .
How to add the package org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2, so it works.
Edit: Seems it is an existing issue fixed only in yet to be released version spark 3.4. Based on the suggestions here and here I had to bake all the jars (spark-sql-kafka-0-10_2.12-3.1.2 and its dependencies and also hudi jar) into the spark image. Then it worked.
Currently I am using Kafka SpoolDir connector in standalone mode. After adding the required configurations to the properties file, I start the connector using
kafka/bin/connect-standalone.sh connect-standalone.properties file-source.properties
Is there any way to start the connector(stadalone/ distributed) using a java code only, in the way we can write consumer and producer java codes?
ConnectStandalone is the Java class that this command starts, but Connect Framework is not meant to be ran as an embedded service
You can see the source code here that starts the server and parses the config file
The problem is simple: i want to print all topics from apache kafka after installing kafka module on karaf. I need to get properties from cfg file which is located in jbossfuse/etc and create a KafkaConsumer object. I want to implement BundleActivator to be able to start method in the moment of installation module.
The question is: how can i get properties from the config file?
I found some solution here: some solution, they said " you can use ConfigAdimn service from OSGi spec. ". How can i use it? All examples with the code are welcome
Karaf uses Felix-FileInstall to read config files: http://felix.apache.org/documentation/subprojects/apache-felix-file-install.html
So if there is a file named kafka.cfg, it will pick it up and register a config with the ConfigAdmin-Service under the pid 'kafka'.
You can fetch the ConfigAdmin-Service and fetch the config using an Activator and read that config from there, but I strongly recommend to use DeclarativeServices or Blueprint instead to interact with the OSGi-Framework, both support injection of configuration if it is available.
Because otherwise you have to deal yourself with the following topics:
there is no ConfigAdmin (yet), maybe because your bundle starts earlier)
the ConfigAdmin changes (for example due to a package refresh or update)
the configuration is not yet registered (because felix has not read it yet)
the configuration gets updated (for example somone changes the file)
I have the following configuration:
Spring-integration-kafka 1.3.1.RELEASE
I have a custom kafka-sink and a custom kafka-source
The configuration I want to have:
I'd like to still using Spring-integration-kafka 1.3.1.RELEASE with my custom kafka-sink.
I'm changing my kafka-source logic to use Spring-integration-kafka-2.1.0.RELEASE. I noticed the way to implement a consumer/producer is way different to prior versions of Spring-integration-kafka.
My question is: could I face some compatibily issues?
I'm using Rabbit.
You should be ok then; it would probably work with the newer kafka jars in the source's /lib directory since each module is loaded in its own classloader so there should be no clashes with the xd/lib jars.
However, you might have to remove the old kafka jars from the xd/lib directory (which is why I asked about the message bus).
I am trying to execute a sample producer consumer application using Apache Kafka. I downloaded it from https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka-0.10.0.0-src.tgz . Then I started following the steps given in http://www.javaworld.com/article/3060078/big-data/big-data-messaging-with-kafka-part-1.html.
When I tried to run bin/zookeeper-server-start.sh config/zookeeper.properties, I am getting Error: Could not find or load main class config.zookeeper.properties I googled about the issue but didn't get any useful information on this. Can anyone help me to continue?
You've downloaded the source package. Download the binary package of Kafka and do testing.
You have to download the binary version from the official Kafka web site.
Assuming you have the correct binary version check to see that you do not already have CLASSPATH defined in your environment. If you do and the defined CLASSPATH has a space in it (e.g.C:\Program Files\<>) then neither zookeeper or kafka will start.
To solve this either delete your existing CLASSPATH or modify the startup script that builds the zookeeper and kafka CLASSPATH values, putting your CLASSPATH entry in double quotes before the path is built