Osmosis throws strange Java error - postgresql

I have been attempting to import a north-america-latest.osm.pbf (from Geofabrik) into a Postgres database for some time. After reviewing the wiki detailed usage page thoroughly, I set the database to include all necessary tables (pgSnapshot) via the included sql scripts. I also made sure that osmosis was functioning as intended by running a smaller file (Antarctica) through, and I got the results I expected. However, when I attempt to do the same process with the north america file, I get an error that is dissimilar to others that have been reported on the web. I am trying to get this data onto a server, uploads to my local seem to be fine.
Here's my code (via command prompt) :
C:\Users\eddie\Desktop>osmosis --read-pbf-fast north-america-latest.osm.pbf --log-progress interval=3000 --write-pgsql nodeLocationStoreType="TempFile" host=1*.8*.*.*0* database=osm postgresSchema=osm_updates user=eddie password=***
Here is the error message I get:
SEVERE: Thread for task 1-read-pbf-fast failed
org.springframework.dao.EmptyResultDataAccessException: Incorrect result
size: expected 1, actual 0
at org.springframework.dao.support.DataAccessUtils.requiredSingleResult(DataAccessUtils.java:71)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:495)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:500)
at org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateDBVersion(SchemaVersionValidator.java:64)
at org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateVersion(SchemaVersionValidator.java:47)
at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetLoader.run(CopyFilesetLoader.java:77)
at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlCopyWriter.complete(PostgreSqlCopyWriter.java:117)
at org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.complete(EntityProgressLogger.java:82)
at org.openstreetmap.osmosis.pbf2.v0_6.PbfReader.run(PbfReader.java:96)
at java.lang.Thread.run(Unknown Source)
Jul 19, 2018 8:28:24 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
I am running osmosis .46, Postgres/PostGis 10/2.4 on Windows 10 with 12GB of RAM and 2 Intel 2.4GHz processors.
UPDATE: the error now occurs even when I run smaller files. Additionally, osmosis behaves as if it is processing a larger file (reaches node 4614331685 for Antarctica) as seen through the progress logger messages. An upload of OSM data for Canada to my local went through without any issues, so the problem probably has to do with the server I am trying to connect to. If anyone has any clues as to how to decipher the error message though, I'd like to hear them!

I got osmosis to work by taking #mmd 's advice of turning off the schema validation. Even though I ran the pgsnapshot scripts and had been successful putting data there before, something about doing all of north america seemed to throw it off. I'll update this answer after subsequent database updates.

Related

The first element has not yet passed through in 5000 milliseconds

I am new to lagom scala. Created a simple crud to learn the key concept of it. First I compiled my code using sbt clean compile and it successfully compiled the code.
After that I tried to run the project using sbt clean runAll command. Sometimes service started without any problem. But sometimes I found a message saying that
java.util.concurrent.TimeoutException: The first element has not yet passed through in 5000 milliseconds.
at akka.stream.impl.Timers$Initial$$anon$1.onTimer(Timers.scala:62)
at akka.stream.stage.TimerGraphStageLogic.onInternalTimer(GraphStage.scala:1601)
at akka.stream.stage.TimerGraphStageLogic.$anonfun$getTimerAsyncCallback$1(GraphStage.scala:1590)
at akka.stream.stage.TimerGraphStageLogic.$anonfun$getTimerAsyncCallback$1$adapted(GraphStage.scala:1590)
at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:466)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:497)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:599)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:768)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:783)
at akka.actor.Actor.aroundReceive(Actor.scala:534)
at akka.actor.Actor.aroundReceive$(Actor.scala:532)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:690)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:573)
at akka.actor.ActorCell.invoke(ActorCell.scala:543)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:269)
at akka.dispatch.Mailbox.run(Mailbox.scala:230)
at akka.dispatch.Mailbox.exec(Mailbox.scala:242)
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Can anyone please explain me the solution? Thanks in advance
I think that is one of the offset-timeouts, either for the read side or for the Kafka client.
If you are running everything on one machine and it is struggling, getting things running, setting up connections etc. could potentially take more time, so try tweaking those timeouts to higher values and see if it helps.

Error trying to execute feature files in Katalon Studio v6.3.3 - NoClassDefFoundError

Every time I attempt to execute a feature file the below error is thrown. It doesn't matter if it's a new feature file or an existing one. My colleagues are able to execute the exact same tests on their computers. This only occurs when trying to run feature file tests. I'm able to execute other types of tests with Katalon studio. I've tried versions 6.3.2 and 6.3.3 and received the same results. Has anyone seen this before? Based on the error I think it's failing prior to even getting to execute the code within the feature files but I'm at a loss.
09-09-2019 02:56:05 PM Verification
Elapsed time: 1.000s
Verification FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Keyword runFeatureFile was failed (Root cause: java.lang.NoClassDefFoundError: Integer
at cucumber.runtime.java.MethodScanner.scan(MethodScanner.java:40)
at cucumber.runtime.java.JavaBackend.loadGlue(JavaBackend.java:82)
at cucumber.runner.Runner.<init>(Runner.java:36)
at cucumber.runtime.Runtime.<init>(Runtime.java:65)
at cucumber.runtime.Runtime.<init>(Runtime.java:46)
at cucumber.runtime.Runtime.<init>(Runtime.java:42)
at cucumber.api.cli.Main.run(Main.java:34)
at cucumber.api.cli.Main$run.call(Unknown Source)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy:76)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
at com.kms.katalon.core.keyword.internal.KeywordMain$runKeyword.call(Unknown Source)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:46)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile$0.callStatic(Unknown Source)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:101)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile.call(Unknown Source)
at WSVerification1568055365381.run(WSVerification1568055365381:2)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.WSVerificationExecutor.runScript(WSVerificationExecutor.java:146)
at com.kms.katalon.core.main.WSVerificationExecutor.doExecute(WSVerificationExecutor.java:140)
at com.kms.katalon.core.main.WSVerificationExecutor.processExecutionPhase(WSVerificationExecutor.java:123)
at com.kms.katalon.core.main.WSVerificationExecutor.accessMainPhase(WSVerificationExecutor.java:115)
at com.kms.katalon.core.main.WSVerificationExecutor.execute(WSVerificationExecutor.java:103)
at com.kms.katalon.core.main.TestCaseMain.runFeatureFile(TestCaseMain.java:144)
at com.kms.katalon.core.main.TestCaseMain$runFeatureFile$0.call(Unknown Source)
at TempTempCase1568055363653.run(TempTempCase1568055363653.groovy:21)
Caused by: java.lang.ClassNotFoundException: Integer
... 27 more
)
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy:50)
at com.kms.katalon.core.keyword.internal.KeywordMain.stepFailed(KeywordMain.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:70)
at com.kms.katalon.core.keyword.internal.KeywordMain$runKeyword.call(Unknown Source)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:46)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile$0.callStatic(Unknown Source)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:101)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile.call(Unknown Source)
at WSVerification1568055365381.run(WSVerification1568055365381:2)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.WSVerificationExecutor.runScript(WSVerificationExecutor.java:146)
at com.kms.katalon.core.main.WSVerificationExecutor.doExecute(WSVerificationExecutor.java:140)
at com.kms.katalon.core.main.WSVerificationExecutor.processExecutionPhase(WSVerificationExecutor.java:123)
at com.kms.katalon.core.main.WSVerificationExecutor.accessMainPhase(WSVerificationExecutor.java:115)
at com.kms.katalon.core.main.WSVerificationExecutor.execute(WSVerificationExecutor.java:103)
at com.kms.katalon.core.main.TestCaseMain.runFeatureFile(TestCaseMain.java:144)
at com.kms.katalon.core.main.TestCaseMain$runFeatureFile$0.call(Unknown Source)
at TempTempCase1568055363653.run(TempTempCase1568055363653.groovy:21)
Caused by: java.lang.NoClassDefFoundError: Integer
at cucumber.runtime.java.MethodScanner.scan(MethodScanner.java:40)
at cucumber.runtime.java.JavaBackend.loadGlue(JavaBackend.java:82)
at cucumber.runner.Runner.<init>(Runner.java:36)
at cucumber.runtime.Runtime.<init>(Runtime.java:65)
at cucumber.runtime.Runtime.<init>(Runtime.java:46)
at cucumber.runtime.Runtime.<init>(Runtime.java:42)
at cucumber.api.cli.Main.run(Main.java:34)
at cucumber.api.cli.Main$run.call(Unknown Source)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy:76)
at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
... 16 more
Caused by: java.lang.ClassNotFoundException: Integer
... 27 more
Well, this isn't an answer as to what happened but I deleted the local source code I was using and pulled it back down into a separate location and now I'm able to run the tests. Maybe something got corrupted in the original clone from the repo. Anyway, all is well now.

My Android Thing Raspberry Pi is throwing a javax.net.ssl.SSLException when sending request to Firebase Storage

I'm trying to connect my Raspberry Pi using Android Thing to Firebase Storage. I keep getting the attached exception. The Storage Rules are completely public no auth needed. Has anyone else received this error and found a workaround?
javax.net.ssl.SSLException: Write error: ssl=0x95683f00: I/O error during system call, Broken pipe
at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:824)
at com.android.okhttp.okio.Okio$1.write(Okio.java:76)
at com.android.okhttp.okio.AsyncTimeout$1.write(AsyncTimeout.java:155)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:176)
at com.android.okhttp.okio.RealBufferedSink.writeUtf8(RealBufferedSink.java:58)
at com.android.okhttp.internal.http.HttpConnection.writeRequest(HttpConnection.java:172)
at com.android.okhttp.internal.http.HttpTransport.writeRequestHeaders(HttpTransport.java:76)
at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:265)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:457)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:405)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:521)
at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java)
at brv.b(:com.google.android.gms.DynamiteModulesC:1386)
at brv.a(:com.google.android.gms.DynamiteModulesC:269)
at brp.onTransact(:com.google.android.gms.DynamiteModulesC:53)
at android.os.Binder.transact(Binder.java:499)
at com.google.android.gms.internal.zzbrc$zza$zza.zzjN(Unknown Source)
at com.google.android.gms.internal.zzbrf.zza(Unknown Source)
at com.google.android.gms.internal.zzbqw.zza(Unknown Source)
at com.google.android.gms.internal.zzbqw.zzd(Unknown Source)
at com.google.firebase.storage.UploadTask.zzb(Unknown Source)
at com.google.firebase.storage.UploadTask.zzaba(Unknown Source)
at com.google.firebase.storage.UploadTask.run(Unknown Source)
at com.google.firebase.storage.StorageTask$8.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
at java.lang.Thread.run(Thread.java:761)
A few things to check:
1) Make sure that you have network (you likely already do since you're trying to reach Firebase) e.g. 'ifconfig'
2) Make sure that DNS is working e.g. 'ping ...' something
3) Check system time to see if it's synced to current rather than 1970 e.g. 'date'; you can manually set it current as a workaround
If for any of the above reasons that time is not set properly, your SSL cert probably won't work.
See working code in the Doorbell sample here: https://github.com/androidthings/doorbell

MongoDB java driver - massive number of parked threads

In a long run I am seeing massive number of threads getting piled up by MongoDB Java driver (v3.0.3). All these threads are server monitoring threads, all parked waiting:
cluster-ClusterId{value='562233d1b26c940820028340', description='null'}-192.168.0.2:27017
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:237)
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:218)
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:167)
java.lang.Thread.run(Unknown Source)
Right now there are about 250 of them. I don't think that many threads are needed to monitor a connection to a single database host. I am obviously doing something wrong.., but as far as I can tell we didn't do any setting changes when moved from driver v2 to v3. Could be a bug in driver? Any ideas?
This issue has been fixed in 3.2.2.
https://jira.mongodb.org/browse/JAVA-2074

Testing Camunda REST API performance with SoapUI Pro - java.net.SocketTimeoutException: Read timed out

I'm running a load test on camunda BPM engine's REST API using SoapUI Pro 5.1.2 and I get about 10-30% of failures with the following exception.
Mon Mar 30 11:53:48 PDT 2015:ERROR:java.net.SocketTimeoutException: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:110)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:264)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at com.eviware.soapui.impl.wsdl.support.http.HttpClientSupport$SoapUIHttpRequestExecutor.doReceiveResponse(HttpClientSupport.java:147)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:633)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:454)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at com.eviware.soapui.impl.wsdl.support.http.HttpClientSupport$Helper.execute(HttpClientSupport.java:233)
at com.eviware.soapui.impl.wsdl.support.http.HttpClientSupport.execute(HttpClientSupport.java:323)
at com.eviware.soapui.impl.wsdl.submit.transports.http.HttpClientRequestTransport.submitRequest(HttpClientRequestTransport.java:290)
at com.eviware.soapui.impl.wsdl.submit.transports.http.HttpClientRequestTransport.sendRequest(HttpClientRequestTransport.java:220)
at com.eviware.soapui.impl.wsdl.WsdlSubmit.run(WsdlSubmit.java:119)
at com.eviware.soapui.impl.wsdl.WsdlSubmit.submitRequest(WsdlSubmit.java:80)
at com.eviware.soapui.impl.rest.RestRequest.submit(RestRequest.java:192)
at com.eviware.soapui.impl.wsdl.teststeps.RestTestRequestStep.run(RestTestRequestStep.java:793)
at com.eviware.soapui.impl.wsdl.support.AbstractTestCaseRunner.runTestStep(AbstractTestCaseRunner.java:213)
at com.eviware.soapui.impl.wsdl.testcase.WsdlTestCaseRunner.runCurrentTestStep(WsdlTestCaseRunner.java:47)
at com.eviware.soapui.impl.wsdl.support.AbstractTestCaseRunner.internalRun(AbstractTestCaseRunner.java:139)
at com.eviware.soapui.impl.wsdl.support.AbstractTestCaseRunner.internalRun(AbstractTestCaseRunner.java:47)
at com.eviware.soapui.impl.wsdl.support.AbstractTestRunner.run(AbstractTestRunner.java:129)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTestRunner$InternalTestCaseRunner.run(WsdlLoadTestRunner.java:490)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Here's the load test scenario:
In my test case, I have a series of REST calls being done in sequence.
Load test attributes - Threads: 100, Startegy: Burst, Burst Delay: 10s, Burst Duration: 5s, Limit 1000 Total Runs.
I've tried setting the 'Socket Timeout' option of SoapUI to 300000 i.e 300 seconds and I see this is the only option I found on forums but I still get these errors. Also, as I mentioned, not all 1000 runs are failures, only 10-30 % of them fail randomly with this exception.
Sometimes, the load test succeeds with limiting the total runs to under 500 and even then, I get some of these errors and are occasional.
From the above trace, I understand the issue is at the client (SoupUI) side (or is my understanding not right?) and wanted to know how to get around this one?
If your process is synchronous the REST Call which starts the process will block until the process is completed. Which means if the process takes longer than your configured socket timeout the connection will be closed as no further data is expected on the socket. A solution would be to set the start event of your process to async so the REST call will immediately return. But I don't know if this fits your benchmark.
I've earlier set the socket timeout in the global settings of SoapUI but I found in a SmartBear forum that we have separate socket timeout option for the test case. It worked when I've set the timeout for the test case.