I am running SCDF with Docker Compose in Ubuntu 20.04 on WSL. I can create and deploy streams using starter sources/sinks without issue but when trying to deploy a custom Spring Cloud Stream app the following happens and I am not sure how to resolve it:
dataflow:>app register --name order-source --type source --uri docker://*******/****:order-producer
Successfully registered application 'source:order-source'
dataflow:>stream create ordering-stream --definition "order-source | log"
Created new stream 'ordering-stream'
dataflow:>stream deploy ordering-stream
Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: SkipperException
SkipperException
org.springframework.cloud.dataflow.rest.client.DataFlowClientException: SkipperException
at org.springframework.cloud.dataflow.rest.client.VndErrorResponseErrorHandler.handleError(VndErrorResponseErrorHandler.java:65)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:672)
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:416)
at org.springframework.cloud.dataflow.rest.client.StreamTemplate.deploy(StreamTemplate.java:141)
at org.springframework.cloud.dataflow.shell.command.StreamCommands.deployStream(StreamCommands.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
at java.lang.Thread.run(Thread.java:748)
This skipper server exception includes this:
Caused by: java.io.IOException: Cannot run program "docker" (in directory "/tmp/1627185411996/ordering-stream.order-source-v1"): error=2, No such file or directory
I can include the verbose server output if that would be helpful.
I was able to resolve the problem by referencing the application's jar file locally with Uri = file:///root/scdf/order-service-0.0.1-SNAPSHOT.jar. The docker-compose.yml file also needs to be inside the same directory as the jar file (~/scdf in this case). I'm still unsure why the Docker Hub Uri reference fails since docker is working for SCDF, all of the other services within Docker Compose, and elsewhere.
Related
I have tested default existing jdbc source connector. It is working fine for DB. I want to create another oracle jdbc source connector for different DB. For that I have created new properties file with different DB details. Then, I ran the below command.
bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties source-quickstart-sqlite1.properties
The Error:
[2020-06-20 09:43:46,751] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:130)
org.apache.kafka.connect.errors.ConnectException: Unable to initialize REST server
at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:217)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:87)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8083
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:215)
... 1 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
... 8 more
I have tried usuing rest.port:8089. It is also not working.
There is already an application running on you computer, who is attached (binded) to the port 8083. Probably, it's another Kafka Connect instance. Stop it before executing you command.
To find the app listening the port, execute lsof -i :8083 and you'll get the PID. You can then simply kill it, assuming it's all about local development of Konnect applications.
I tested my ApachedMQ image on Ubuntu and it works fine with Docker.
On Openshift deploying the same image does not work.
It should work out of the box, just like Ubuntu, but it always complains about missing directories in the pod, or lack of permissions on directories in the pod. Below you will see errors regarding the former.
I am using the FREE edition of Openshift online v3.x
I am only using the console to deploy from the image.
Below just a sample of the log messages when the pod is ramping up for the deployment.
java.io.FileNotFoundException: /opt/activemq/data/activemq.log (Permission denied)
Caused by: java.io.IOException: Failed to create directory '/opt/activemq/data/kahadb'
Any help would be much appreciated, as this image should IMO, be deployable from the console without any YAML and oc CLI necessary.
Some log trace information:
INFO: Loading '/opt/activemq/bin/env'
INFO: Using java '/docker-java-home/jre/bin/java'
INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
INFO: Creating pidfile /opt/activemq/data/activemq.pid
bin/activemq: 484: bin/activemq: cannot create /opt/activemq/data/activemq.pid: Permission denied
Java Runtime: Oracle Corporation 1.8.0_141 /usr/lib/jvm/java-8-openjdk-amd64/jre
Heap sizes: current=62976k free=58714k max=932352k
JVM args: -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/activemq/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/activemq/tmp -Dactivemq.classpath=/opt/activemq/conf:/opt/activemq/../lib/: -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data
Extensions classpath:
[/opt/activemq/lib,/opt/activemq/lib/camel,/opt/activemq/lib/optional,/opt/activemq/lib/web,/opt/activemq/lib/extra]
ACTIVEMQ_HOME: /opt/activemq
ACTIVEMQ_BASE: /opt/activemq
ACTIVEMQ_CONF: /opt/activemq/conf
ACTIVEMQ_DATA: /opt/activemq/data
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /opt/activemq/data/activemq.log (Permission denied)
...
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /opt/activemq/data/audit.log (Permission denied)
...
Loading message broker from: xbean:activemq.xml
INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#5bcab519: startup date [Fri Mar 16 21:10:18 UTC 2018]; root of context hierarchy
WARN | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: java.io.IOException: Failed to create directory '/opt/activemq/data/kahadb'
ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: java.io.IOException: Failed to create directory '/opt/activemq/data/kahadb'
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: java.io.IOException: Failed to create directory '/opt/activemq/data/kahadb'
SOLVED proof::
In order for the image to successfully be deployed, the following have to be done on the cluster:
oc adm policy add-scc-to-user anyuid -z default -n myproject --as system:admin
When deploying set the ACTIVEMQ_DATA environment variable to /tmp
I will continue to find other alternatives including a rebuild of the image, creating a pod with the requisite permissions for the $APACHEMQ_DATA directory.
Below proof of the route working on the playground instance.
It is o.k. to paste it as the playground will expire in the next 10 minutes.
I corrected the image.
The docker file can be found here on my fork:
https://github.com/beezerbt/docker-activemq/tree/master/5.14.5-openshift-v3.9.14
The image for the above can be found here:
https://hub.docker.com/r/beezer/openshift-ready-mq/tags/
Tag::[5.14.5-3.9.14]
Finally, since the image has an ephemeral persistence, a volume can be mapped to /app within the openshift cluster quite easily; however do remember that you must set the volume mount point as something OTHER than /app and then further down in the volume definition on Openshift console, specify where in the container it should be mapped to...which is then /app.
I have created a activemq image which can be deployed in Openshift
https://hub.docker.com/r/rangareddyv/activemq-openshift
Created an image with latest version of activemq and amzn-corretto-jdk-11 as bas image because openjdk has lot of CVE's and alpine does not support the jdk-11
Spring Boot has a handy feature that it will embed an init.d starup script into an executable jar if you configure the maven plugin to do so:
http://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html#deployment-install
So I am "installing" a Spring Boot app (executable fat jar) as a service in Centos 6.6 using the above method.
So far so good. I create a link to my jar and set the permissions:
sudo ln -s /path/to/myapp.jar /etc/init.d/my-service
sudo chmod 0755 /etc/init.d/my-service
And I can successfully start the application:
sudo service my-service start
However, I want the app to come up on startup, so I use the chkconfig utility:
sudo chkconfig --add my-service
sudo chkconfig my-service on
No errors with those commands, but when I reboot Centos the service does not auto-start.
running:
sudo service my-service status
shows:
Not running
running:
chkconfig --list my-service
shows:
my-service 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Everything looks good, but it's not starting. At this point I can manually start the service with "sudo service my-service start" and it comes back up, but it's not auto-starting on boot.
Any ideas?
Thanks
Problem solved. Turned out to be a path issue and my fault. The path where the application lived was a mounted directory that was not available at boot time.
I have the similar issue, I got the following error message when I run service myapp start
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/springframework/boot/loader/JarLauncher : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.springframework.boot.loader.JarLauncher. Program will exit.
However I could make it work via directly run /etc/init.d/app start.
Finally I found the problem is the default jdk is 1.6 although the java is 1.8 in the current user. Then I changed the symbolic and it worked.
I tried looking for a Jenkins plugin (like AWS codeDeploy) so that I could deploy my application to a Kubernetes cluster. So far, I have been successful at pushing it to a Docker registry and adding some command line build steps to deploy to Kubernetes.
Looking at the CloudBees announcement this seems possible
Installing the Kubernetes plugin gave me errors...I can attach a screenshot if that helps ...
Also it seems like this plugin allows you to run slaves in Docker containers not deploy your own app.
After looking at this video
, it seemed I could accomplish this
using the withKubernetes workflow stage...
However adding that line to my workflow script gives me the following error
java.lang.NoSuchMethodError: No such DSL method withKubernetes found among [archive, bat, build, catchError, checkout, dir, dockerFingerprintFrom, dockerFingerprintRun, echo, error, fileExists, git, input, load, mail, node, parallel, publishHTML, pwd, readFile, retry, sh, sleep, stage, stash, step, svn, timeout, tool, unarchive, unstash, waitUntil, withDockerContainer, withDockerRegistry, withDockerServer, withEnv, wrap, writeFile, ws]
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:107)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:112)
at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:75)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:15)
at WorkflowScript.run(WorkflowScript:17)
at Unknown.Unknown(Unknown)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79)
at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:40)
at com.cloudbees.groovy.cps.Next.step(Next.java:58)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:271)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The Jenkins Kubernetes plugin is (so far) only to run slaves dynamically in a Kubernetes cluster
There's not a lot about deploying from Jenkins to Kubernetes, maybe this post Continuous Delivery Pipelines with Fabric8 and Jenkins on OpenShift helps
After I deleted a Google Cloud Storage directory through the Google Cloud Console, (the directory was generated by early Spark (ver 1.3.1) job), when re-run the job, it always fail and seemed the directory was still there to the job; I cannot find the directory with gsutil.
Is this a bug, or anything I missed? Thanks!
The error I got:
java.lang.RuntimeException: path gs://<my_bucket>/job_dir1/output_1.parquet already exists.
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:112)
at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:240)
at org.apache.spark.sql.DataFrame.save(DataFrame.scala:1196)
at org.apache.spark.sql.DataFrame.saveAsParquetFile(DataFrame.scala:995)
at com.xxx.Job1$.execute(Job1.scala:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
It appears you might be running into a known bug with the NFS list-consistency cache: https://github.com/GoogleCloudPlatform/bigdata-interop/issues/5
It was fixed in the latest release, and if you upgrade by deploying a new cluster with bdutil-1.3.1 (announced here: https://groups.google.com/forum/#!topic/gcp-hadoop-announce/vstNuV0LpDc) the problem should be fixed. If you need to upgrade in-place, you can try to download the latest gcs-connector-1.4.1 jarfile onto your master and worker nodes under /home/hadoop/hadoop-install/lib/gcs-connector-*.jar and then rebooting the Spark daemons:
sudo sudo -u hadoop /home/hadoop/spark-install/sbin/stop-all.sh
sudo sudo -u hadoop /home/hadoop/spark-install/sbin/start-all.sh