I am trying to use a kafka connect JDBC sink connector with avro. But when i try to start the connector i keep getting the two following errors :
java.lang.NoClassDefFoundError: io/confluent/connect/avro/AvroConverterConfig
and
java.lang.NoClassDefFoundError: io/confluent/common/config/AbstractConfig.
I am a little but confused here because my plugin path is /app/confluent-5.0.0/share/java
confluent-5.0.0/share/java
And we can see the jar corresponding to the missing class is there.
Anyone has an idea about what is causing this missing class ?
Thanks a lot !
basically i have just pulled the binaries for confluent-5.0.0 from confluent, i have a custom script getting the env variable and setting them to the worker variables like that :
if [ "$1" = 'bash' ]; then
exec "$#"
else
if [ -z "$KAFKA_CONNECT_BOOTSTRAP_SERVERS" ]; then
export KAFKA_CONNECT_BOOTSTRAP_SERVERS="kafka:9092"
fi
if [ -z "$KAFKA_CONNECT_GROUP_ID" ]; then
export KAFKA_CONNECT_GROUP_ID="connect-cluster"
fi
and then calling at the end :
exec $SCRIPTS/connect_distributed.sh /tmp/temp.properties
Related
I am using Dataproc with 1 job on 1 cluster.
I would like to start my job as soon as the cluster is created. I found that the best way to achieve this is to submit a job using an initialization script like below.
function submit_job() {
echo "Submitting job..."
gcloud dataproc jobs submit pyspark ...
}
export -f submit_job
function check_running() {
echo "checking..."
gcloud dataproc clusters list --region='asia-northeast1' --filter='clusterName = {{ cluster_name }}' |
tail -n 1 |
while read name platform worker_count preemptive_worker_count status others
do
if [ "$status" = "RUNNING" ]; then
return 0
fi
done
}
export -f check_running
function after_initialization() {
local role
role=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${role}" == 'Master' ]]; then
echo "monitoring the cluster..."
while true; do
if check_running; then
submit_job
break
fi
sleep 5
done
fi
}
export -f after_initialization
echo "start monitoring..."
bash -c after_initialization & disown -h
is it possible? When I ran this on Dataproc, a job is not submitted...
Thank you!
Consider to use Dataproc Workflow, it is designed for workflows of multi-steps, creating cluster, submitting job, deleting cluster. It is better than init actions, because it is a first class feature of Dataproc, there will be a Dataproc job resource, and you can view the history.
Please consider to use cloud composer - then you can write a single script that creates the cluster, runs the job and terminates the cluster.
I found a way.
Put a shell script named await_cluster_and_run_command.sh on GCS. Then, add the following codes to the initialization script.
gsutil cp gs://...../await_cluster_and_run_command.sh /usr/local/bin/
chmod 750 /usr/local/bin/await_cluster_and_run_command.sh
nohup /usr/local/bin/await_cluster_and_run_command.sh &>>/var/log/master-post-init.log &
reference: https://github.com/GoogleCloudDataproc/initialization-actions/blob/master/post-init/master-post-init.sh
I have been going through the documentation of OrientDB about Stress tester. But I couldn't find the script for it in my setup's bin folder.
the script for the stress test was present in version older than 3.x
You can take stresstester.sh from, for example, OrientDB 2.2 and adapt it for your version.
Here is /orientdb/bin/stresstester.sh from 2.2.37:
#!/usr/bin/env bash
# resolve links - $0 may be a softlink
PRG="$0"
while [ -h "$PRG" ]; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`/"$link"
fi
done
# Get standard environment variables
PRGDIR=`dirname "$PRG"`
# Only set ORIENTDB_HOME if not already set
[ -f "$ORIENTDB_HOME"/lib/orientdb-tools-2.2.37.jar ] || ORIENTDB_HOME=`cd "$PRGDIR/.." ; pwd`
export ORIENTDB_HOME
# Set JavaHome if it exists
if [ -f "${JAVA_HOME}/bin/java" ]; then
JAVA=${JAVA_HOME}/bin/java
else
JAVA=java
fi
export JAVA
exec "$JAVA" -XX:MaxDirectMemorySize=512g -cp "$ORIENTDB_HOME/lib/*" \
com.orientechnologies.orient.stresstest.OStressTester $*
In my case I only had to change orientdb-tools-2.2.37.jar to orientdb-tools-3.0.33.jar in order to run this script with OrientDB 3.0.33 I have installed.
I am trying to import tables from RDBMS with a JSON config file which stated the edge relationship. But when I am trying to run the command, it throws a NullPointerException.
Command -
oteleporter.bat -jdriver mysql -jurl jdbc:mysql://localhost:3306/hive_catalog_test_db -juser root -jpasswd root -ourl plocal:C:\Users\ME\desktop\OrientDB\orientdb-community-importers-2.2.27\databases\HiveCatalog_D02_TeleTest -s naive -nr java -v 1 -include "tbls,columns_v2" -conf "C:\Users\ME\desktop\Catalog_POC\Inheritance_and_edge_loaders\migration-config.json"
Exception -
Exception in thread "main" java.lang.NullPointerException
at com.orientechnologies.teleporter.context.OTeleporterContext.printExceptionMessage(OTeleporterContext.java:152)
at com.orientechnologies.teleporter.util.OMigrationConfigManager.loadMigrationConfigFromFile(OMigrationConfigManager.java:64)
at com.orientechnologies.teleporter.main.OTeleporter.execute(OTeleporter.java:250)
at com.orientechnologies.teleporter.main.OTeleporter.main(OTeleporter.java:219)
However, when I am running the command without the config attribute, it is executing fine.
Sample JSON -
{
"edges": [
{
"HasColumns": {
"mapping": {
"fromTable": "Tbls",
"fromColumns": ["tbId"],
"toTable": "ColumnsV2",
"toColumns": ["cdId"],
"direction": "direct"
}
}
}
]
}
There's something wrong with your json config file of course. Maybe the path is wrong?
By the way, you can enjoy a new visual tool in order to build your custom configuration. You can find it in the server management area of Studio.
Here is the doc link:
https://orientdb.com/docs/2.2/Studio-Teleporter.html
Good luck!
I am trying to add a user to WildFly server but get an error:
./add-user.sh: 1: eval: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java/bin/java: not found
I suppose I need to change the JAVA_HOME location in a .conf file but have done so in the .conf files of the wildfly/bin directory. How can I solve this issue?
The add-user.sh script doesn't source any configuration file, so you have to provide the Java binary through your environment.
You can do this in multiple ways :
through a JAVA variable pointing to the java executable :
export JAVA=/path/to/jdk_install/bin/java
./add-user.sh [...]
through a JAVA_HOME variable pointing to the Java installation directory :
export JAVA_HOME=/path/to/jdk_install/
./add-user.sh [...]
by including the Java's installation bin directory to your PATH :
export PATH="/path/to/jdk_install/bin:$PATH"
./add-user.sh [...]
Note that these may vary depending on the WildFly or JBoss EAP version ; I gathered these from reading the script of a wildfly-8.1.0.Final installation.
If you're not sure this applies to your own version and if you have even the most basic understanding of shell scripts you should check your own : they're only a few dozens lines long and are well commented.
Your java path is wrong.
You can observe that in below line there is extra '/bin/java'. You should remove that.
./add-user.sh: 1: eval: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java/bin/java: not found
To do so, in your adduser.sh/.bat file replace
# Setup the JVM
if [ "x$JAVA" = "x" ]; then
if [ "x$JAVA_HOME" != "x" ]; then
JAVA="$JAVA_HOME/bin/java"
else
JAVA="java"
fi
fi
To
# Setup the JVM
if [ "x$JAVA" = "x" ]; then
if [ "x$JAVA_HOME" != "x" ]; then
JAVA="$JAVA_HOME"
else
JAVA="java"
fi
fi
This does the magic for me.
I'm trying to install Spark on my local machine. I have been following this guide. I have installed JDK-7 (also have JDK-8) and Scala 2.11.7. A problem occurs when I try to use sbt to build Spark 1.4.1. I get the following exception.
NOTE: The sbt/sbt script has been relocated to build/sbt.
Please update references to point to the new location.
Invoking 'build/sbt assembly' now ...
Attempting to fetch sbt
Launching sbt from build/sbt-launch-0.13.7.jar
Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar
I have searched for a solution to this problem. I have found a nice guide https://stackoverflow.com/a/31597283/2771315 which uses a pre-built version. Other than using the pre-built version, is there a way to install Spark using sbt? Further, is there a reason as to why the Invalid or corrupt jarfile error occurs?
I met the same problem. I have fixed it now.
This probably because sbt-launch-0.13.7.jar has a unsuccessful download, although you can see the file is exist, but it's not correct file. The file is about 1.2MB in size. If less than that, you can get into the build/ , use "vim sbt-launch-0.13.7.jar" or other tools to open sbt-launch-0.13.7.jar file.
If the file have the content like this:
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
It implys that sbt-launch-0.13.7.jar is not downloaded.
Then open sbt-launch-lib.bash in the same directory,check the line 41 and 42, it gives two urls. Open it to check if they work well.
If url1 doesn't work，download the sbt-launch.jar manually(you can use url2, it may works,or you can download from sbt official website), put it in the same directory, rename it to sbt-launch-0.13.7.jar, then you shoud comment lines in relation to the downloading(may be between line 47 and 68)， avoid the script download it again. Like this:
acquire_sbt_jar () {
SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
URL1=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
JAR=build/sbt-launch-${SBT_VERSION}.jar
sbt_jar=$JAR
# if [[ ! -f "$sbt_jar" ]]; then
# # Download sbt launch jar if it hasn't been downloaded yet
# if [ ! -f "${JAR}" ]; then
# # Download
# printf "Attempting to fetch sbt\n"
# JAR_DL="${JAR}.part"
# if [ $(command -v curl) ]; then
# (curl --silent ${URL1} > "${JAR_DL}" || curl --silent ${URL2} > "${JAR_DL}") && mv "${JAR_DL}" "${JAR}"
# elif [ $(command -v wget) ]; then
# (wget --quiet ${URL1} -O "${JAR_DL}" || wget --quiet ${URL2} -O "${JAR_DL}") && mv "${JAR_DL}" "${JAR}"
# else
# printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
# exit -1
# fi
# fi
# if [ ! -f "${JAR}" ]; then
# # We failed to download
# printf "Our attempt to download sbt locally to ${JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
# exit -1
# fi
# printf "Launching sbt from ${JAR}\n"
# fi
}
Then use "build/sbt assembly" to build the spark again.
Hope you will succeed.
If I didn't express clearly, the following links may be helpful.
https://www.mail-archive.com/user#spark.apache.org/msg34358.html
Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.5.jar the answer by prabeesh
https://groups.google.com/forum/#!topic/predictionio-user/fllCh8n-0d4
Download the sbt-launch.jar file manually (you can use url2, it may work, or you can download from the sbt official website), put it in the same directory, rename it to sbt-launch-0.13.7.jar, then run the sbt/sbt assembly command.