I have a a beam pipeline that reads from PubSub and write to BigQuery after applying some transformation. The pipeline fails consistently with a NPE. I am using beam SDK version 0.6.0. Any Idea on what I could be doing wrong? I am trying to run the pipeline with a DirectRunner.
java.lang.NullPointerException
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubReader.ackBatch(PubsubUnboundedSource.java:640)
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubCheckpoint.finalizeCheckpoint(PubsubUnboundedSource.java:313)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.getReader(UnboundedReadEvaluatorFactory.java:174)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:127)
at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This problem exists because of a Bug (BEAM-1656) in the DirectRunner and a precondition within PubsubCheckpoint. The bug in the DirectRunner was fixed in pull request 2237, which is merged into the Github master branch, but after 0.6.0 release.
Updating to the 0.7.0 nightly build or building from github HEAD will solve this problem when using the DirectRunner.
To update to the current nightly build you will have to add the following repositories to your project's pom.xml. The earliest version of the beam-runners-direct-java module containing the fix is 0.7.0-20170316.070901-9, but not all modules are built with this specific version so you may have to either specify individually compatible versions or use 0.7.0-SNAPSHOT
<repositories>
<repository>
<id>apache.snapshots</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
Related
Relatively unfamiliar with Maven.
Im trying to set up a shared project in my laptop.
I've heard there's apparently a security issue with the Maven Repo, so, in the pom.xml, I added this to the <repositories>
<repository>
<id>central</id>
<name>Central Repository</name>
<url>https://repo.maven.apache.org/maven2</url>
<layout>default</layout>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
But, on top of still getting a 501 error, this is what Im seeing:
Failed to look for file: http://repo.maven.apache.org/maven2/http://repo.maven.apache.org/maven2/org/codehaus/mojo/gwt-maven-plugin/2.4.0/gwt-maven-plugin-2.4.0.pom.
Why is it appending to the URL instead of replacing it?
Failure to transfer org.codehaus.mojo:gwt-maven-plugin:pom:2.4.0 from http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced. Original error: Could not transfer artifact org.codehaus.mojo:gwt-maven-plugin:pom:2.4.0 from/to central (http://repo.maven.apache.org/maven2): Failed to look for file: http://repo.maven.apache.org/maven2/http://repo.maven.apache.org/maven2/org/codehaus/mojo/gwt-maven-plugin/2.4.0/gwt-maven-plugin-2.4.0.pom.
And this is the follow-up message.
Citrus 2.7.7 is not compatible with cucumber 3.0.2 or later and when I check the sample from GitHub, they are using 2.8.0-SNAPSHOT with cucumber 3.0.2. So when is the 2.8.0 release version will be out? For me, I am able to get the 2.8.0-SNAPSHOT without any issue. But for some of my colleague they are not able to get it even with this.
<repositories>
<repository>
<id>consol-labs-snapshots</id>
<url>http://labs.consol.de/maven/snapshots-repository/</url>
<snapshots>
<enabled>true</enabled>
<updatePolicy>interval:10080</updatePolicy>
</snapshots>
</repository>
</repositories>
#Sven Hettwer,
There is no error message, it just says not able to find the artifact!
We're planning to release 2.8.0 no later than by the end of the year. Stay tuned on twitter or on http://citrusframework.org/.
The maven config seems to be okay so far. Could you provide some information about the error message?
Edit:
Have your tried the -U flag for maven? E.g. mvn clean install -U to force updates on snapshots?
BR,
Sven
My current try:
Eclipse 4.4.2 Luna with scala-ide plugin
Rebuild Flink for scala 2.11 (by default maven downloads official Flink build which is for scala 2.10. This brings some issues with the IDE, such as the dreaded macros incompatibility
But working like that is somewhat cumbersome as you need :
To rebuild Flink for scala 2.11 everytime you update to latest source
code.
have correct maven settings so that your project dependencies are not overwritten with Flink official build (i.e. overwriting your scala 2.11 Flink build with scala 2.10 build) when building your project in Eclipse.
From Flink 0.10, the Flink community will provide Flink for Scala 2.11. Currently, you can use Flink for Scala 2.11 (0.10-SNAPSHOT) with Apache Snapshot Maven Repository.
You can add the repository using following maven settings:
<repositories>
<repository>
<id>apache.snapshots</id>
<name>Apache Snapshot Repository</name>
<url>http://repository.apache.org/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
Running IntelliJ solved my issues.
I post this as an answer as Matthias provided the answer in a comment. Thanks Matthias.
This is not a question but a simple maven dependency solution of oltu openid connect that I like to share. I found it might be helpful to other as there's not too many resource about it on the web.
I was having a problem with oltu openid connect dependency in maven (pom). Somehow I found its groupid, artifact, and its version. But whenever I insert the dependency in pom, eclipse notifies me with missing artifact. The problem seemed like this:
The problem was as above mentioned that the dependency was not available in maven central repository. As I searched through web, I found those dependencies are hosted in other repository which is http://repository.idega.com/maven2/. After that, the solution was just to include this repository in your pom like shown below:
Try using the Apache Snapshot Repository:
<repositories>
<repository>
<id>apache.snapshots</id>
<name>Apache Snapshot Repository</name>
<url>https://repository.apache.org/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
</repository>
</repositories>
I am trying to build the sample application for spring batch 2.1.6. (ie. spring-batch-2.1.6.RELEASE/samples/spring-batch-samples) using maven but am getting this error for a missing plugin:
[ERROR] Plugin
com.springsource.bundlor:com.springsource.bundlor.maven:1.0.0.RELEASE
or one of its dependencies could not be resolved: Failure to find
com.springsource.bundlor:com.springsource.bundlor.maven:jar:1.0.0.RELEASE
in http://repo1.maven.org/maven2 was cached in the local repository,
resolution will not be reattempted until the update interval of
central has elapsed or updates are forced ->
Is there another repository I can set up to get this plugin? I am a bit suprised to be getting this errror as this is the latest realease version of spring batch.
Here is the repository section from the pom as it came in the download:
<repositories>
<repository>
<id>com.springsource.repository.bundles.external</id>
<name>SpringSource Enterprise Bundle Repository - SpringSource Bundle External</name>
<url>http://repository.springsource.com/maven/bundles/external</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
The project's parent pom has a bootstrap profile which contains the necessary repository definitions. Build the project with the command mvn test -P bootstrap and it will download the dependencies.
P.S. This is explained in the readme's instructions that how to build Spring Batch. It would be good if they would also tell how to do it in the instructions for using the samples - maybe you could file a bug report?
i am using maven3 and was able to solve this problem by adding this to my pom:
<pluginRepositories>
<pluginRepository>
<id>plugin.repo.maven.central2</id>
<url>http://objectstyle.org/maven2/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
The Spring batch Admin sample build is broken for sure. At least as of today.
The -Pbootstrap doesn't help and adding the repos in that profile in my local settings doens't help either.
When I attempted to build spring-batch-admin-sample, I found that I was missing dependencies of spring batch!
So I went to build that. There I found that I was missing org.neo4j:neo4j-cypher-dsl-1.9.M04 and also gemfire.7.0.1.jar. Adding repos doesn't help because of the maven2/3 incompatibility issues.
So a sure way to fix this is to go to each repo, download the missing dep and mvn install-file them.
So get the neo4j one here:
http://m2.neo4j.org/content/repositories/releases/org/neo4j/neo4j-cypher-dsl/1.9.M04/
Get the gemfire one from here:
https://repo.springsource.org/gemstone-release-cache/com/gemstone/gemfire/gemfire/7.0.1/
And then I ran into a foundrylogic.vpp dependency that I found here:
http://objectstyle.org/maven2/foundrylogic/vpp/vpp/2.2.1/
Don't forget to get the corresponding poms also to keep it clean and get all the transitives, if any.
Use the mvn install-file plugin described here to get all three deps to your local repo.
http://maven.apache.org/plugins/maven-install-plugin/usage.html
Now spring-batch should build clean.
Now if you go to build spring-batch-admin it will still fail because it depends on spring-batch-core-2.2.3.BUILD-SNAPSHOT whereas what we just built was version 3.0.0.BUILD-SNAPSHOT.
So go to spring-batch-admin-parent's pom and modify the pom like so:
<!-- <spring.batch.version>2.2.3.BUILD-SNAPSHOT</spring.batch.version> -->
<spring.batch.version>3.0.0.BUILD-SNAPSHOT</spring.batch.version>
And then step back to spring-batch-admin and mvn clean install should build the sample.
Hope this helps someone!
I installed maven 3.2.1 and it works.