java.lang.Process hang when recursively called - mcafee

My program recursivly calls the McAfee OnDemad Virusscanner for a specific path. In a loop I fill the path with several files, scan it, remove the files, fill it and so on.
After a while the program suddenly hangs and I've no clue for the reason.
Calendar cl = Calendar.getInstance();
String cmd="uvscan -v /var/tmp/McAfee/scanFiles/"
try {
long start=cl.getTimeInMillis();
process = Runtime.getRuntime().exec(cmd, null);
Worker worker = new Worker(process);
worker.start();
try {
LOGGER.debug("Virusscan timeout set to {}ms", timeout);
worker.join(timeout);
if (worker.exit != null)
workerRC=worker.exit;
else {
cl = Calendar.getInstance();
long waitTime=cl.getTimeInMillis()-start;
throw new TimeoutException("Virusscan timeout after " + waitTime + "ms");
}
// else
// throw new TimeoutException();
} catch(InterruptedException ex) {
worker.interrupt();
Thread.currentThread().interrupt();
cl = Calendar.getInstance();
long waitTime=cl.getTimeInMillis()-start;
throw new TimeoutException("Virusscan timeout after " + waitTime + "ms");
}
File scanLog = new File(ConfigReader.getScanLog());
if (!ConfigReader.mustKeepScanLog()) scanLog.deleteOnExit();
LOGGER.debug("ScanLog is: " + scanLog.getPath() + " - RC=" + workerRC);
BufferedWriter bw = new BufferedWriter(new FileWriter(scanLog));
BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream()));
// read the output from the command
while ((buffer = stdInput.readLine()) != null) {
inspectScanlog(buffer, workerRC);
bw.write(buffer+"\n");
LOGGER.debug("Scan STDOUT: " + buffer);
}
BufferedReader stdErr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
while ((buffer = stdErr.readLine()) != null) {
LOGGER.debug("Scan STDERR: " + buffer);
}
bw.close();
cl = Calendar.getInstance();
long waitTime=cl.getTimeInMillis()-start;
LOGGER.info("Virusscan took " + waitTime + "ms");
if (workerRC != 0){
int cc=12;
ConfigReader.setMaxCC(cc);
LOGGER.error("RC={} - Virusscan ended with error. CMD=",cc, cmd);
if (workerRC==13) {
LOGGER.error("RC={} - The scanner found one or more viruses or hostile objects - such as a Trojan-horse program, joke program, or test file.", cc);
}
}
}
catch (Exception e) {
int cc=12;
ConfigReader.setMaxCC(cc);
LOGGER.error("RC={} - {}",cc, e.getMessage());
e.printStackTrace();
} finally {
if (process!=null) {
process.destroy();
}
}
It looks like a hang of the complete program; even the timeout (240000ms) doesn't match. The GC-log seems kind of interrupted:
258.070: [Full GC [PSYoungGen: 13456K->7362K(85120K)] [PSOldGen: 59271K->60735K(81728K)] 72727K->68098K(166848K) [PSPermGen: 16282K->16282K(30400K)], 0.3019290 secs] [Times: user=0.31 sys=0.00, real=0.31 secs]
264.208: [GC [PSYoungGen: 65466K->8773K(92224K)] 126202K->69581K(173952K), 0.0528360 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
265.445: [GC [PSYoungGen: 81774K->9133K(92096K)] 142582K->70061K(173824K), 0.0205430 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
266.916: [GC [PSYoungGen: 82515K->9698K(100096K)] 143443K->70658K(181824K), 0.0189050 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
267.817: [GC [PSYoungGen: 92311K->1436K(100480K)] 153271K->70986K(182208K), 0.0210400 secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
274.208: [GC [PSYoungGen: 84072K->672K(112256K)] 153622K->71297K(193984K), 0.0029610 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
275.769: [GC [PSYoungGen: 94880K->500K(112832K)] 165505K->71732K(194560K), 0.0022440 secs]
The uvscan-command has been called 277.609 seconds after program-start.
Could anyone suggest how to get the reason for the hang?
Thanks in advance,
Ulrich

Related

Android 11 - intent action SEND - limit to email clients only [duplicate]

I'm trying to send an e-mail from my Android App. With the click on a button, gmail should open and show a new email with my previously defined recipient, subject and email body. So far I've tried sending the Intent.ACTION_VIEW as well as Intent.ACTION_SENDTO. Both show my draft with the recipient only. Both subject and message are being opressed. Weird thing is when using the emulator, it works just fine. Also was trying to lock at the android errorlog. Seems like i don't have permission. Is it really a permission problem or could it be something else?
I'd really appreciate any help
cheers
Here is my code:
sending email via ACTION_VIEW
Intent intent = new Intent(Intent.ACTION_VIEW, Uri.parse("mailto:" + to));
intent.putExtra(intent.EXTRA_SUBJECT, subject);
intent.putExtra(intent.EXTRA_TEXT, message);
mainActivity.startActivity(intent);
sending email via ACTION_SENDTO
Intent email = new Intent(Intent.ACTION_SENDTO);
email.setType("message/rfc822");
email.putExtra(Intent.EXTRA_EMAIL, new String[]{to});
email.putExtra(Intent.EXTRA_SUBJECT, subject);
email.putExtra(Intent.EXTRA_TEXT, message);
mainActivity.startActivity(Intent.createChooser(email, "Choose an Email client :"));
error message from logcat
2019-12-13 01:30:35.172 29268-29268/? E//system/bin/webview_zygote32: failed to make and chown /acct/uid_99044: Permission denied
2019-12-13 01:30:35.172 29268-29268/? E/Zygote: createProcessGroup(99044, 0) failed: Permission denied
2019-12-13 01:30:35.206 29289-29289/? E/asset: setgid: Operation not permitted
2019-12-13 01:30:35.226 29296-29296/? E/asset: setgid: Operation not permitted
2019-12-13 01:30:35.355 29268-29268/? E/Typeface: Error mapping font file /system/fonts/NotoSansKhmer-Regular.ttf
2019-12-13 01:30:35.356 29268-29268/? E/Typeface: Error mapping font file /system/fonts/NotoSansKhmer-Bold.ttf
2019-12-13 01:30:35.356 29268-29268/? E/Minikin: Could not get cmap table size!
2019-12-13 01:30:35.356 29268-29268/? E/Typeface: Unable to load Family: null:und-Khmr
2019-12-13 01:30:35.484 29268-29268/? E/Typeface: Error mapping font file /system/fonts/LGAka_Light.ttf
2019-12-13 01:30:35.484 29268-29268/? E/Minikin: Could not get cmap table size!
2019-12-13 01:30:35.484 29268-29268/? E/Typeface: Unable to load Family: lg-lgaka:null
2019-12-13 01:30:35.816 29342-29342/? E//system/bin/webview_zygote32: failed to make and chown /acct/uid_99045: Permission denied
2019-12-13 01:30:35.816 29342-29342/? E/Zygote: createProcessGroup(99045, 0) failed: Permission denied
2019-12-13 01:30:35.842 29354-29354/? E/asset: setgid: Operation not permitted
2019-12-13 01:30:35.864 29367-29367/? E/asset: setgid: Operation not permitted
2019-12-13 01:30:36.139 29342-29342/? E/Typeface: Error mapping font file /system/fonts/NotoSansKhmer-Regular.ttf
2019-12-13 01:30:36.139 29342-29342/? E/Typeface: Error mapping font file /system/fonts/NotoSansKhmer-Bold.ttf
2019-12-13 01:30:36.139 29342-29342/? E/Minikin: Could not get cmap table size!
2019-12-13 01:30:36.139 29342-29342/? E/Typeface: Unable to load Family: null:und-Khmr
2019-12-13 01:30:36.362 29342-29342/? E/Typeface: Error mapping font file /system/fonts/LGAka_Light.ttf
2019-12-13 01:30:36.362 29342-29342/? E/Minikin: Could not get cmap table size!
2019-12-13 01:30:36.362 29342-29342/? E/Typeface: Unable to load Family: lg-lgaka:null
2019-12-13 01:30:36.523 4349-4359/? E/GBMv2: FPS Scaler: EXP
2019-12-13 01:30:36.602 29342-29342/? E/WebViewFactory: can't load with relro file; address space not reserved
2019-12-13 01:30:37.058 29220-29220/? E/Gmail: Gmail:EditWebView JS Console: b/119949571:draft.editor.onLoad; source: file:///android_asset/draft_editor_gmail_compiled.js at 89
2019-12-13 01:30:37.146 29220-29220/? E/Gmail: Gmail:EditWebView JS Console: b/119949571:draft.editor.onLoad is finished; source: file:///android_asset/draft_editor_gmail_compiled.js at 90
I think we had the same issue. Android API 29 introduced some improvements about sending data to other apps. See more details here: Sending simple data to other apps
Here is the solution that works for me.
Intent selectorIntent = new Intent(Intent.ACTION_SENDTO);
selectorIntent.setData(Uri.parse("mailto:"));
final Intent emailIntent = new Intent(Intent.ACTION_SEND);
emailIntent.putExtra(Intent.EXTRA_EMAIL, new String[]{"address#mail.com"});
emailIntent.putExtra(Intent.EXTRA_SUBJECT, "The subject");
emailIntent.putExtra(Intent.EXTRA_TEXT, "The email body");
emailIntent.setSelector( selectorIntent );
activity.startActivity(Intent.createChooser(emailIntent, "Send email..."));
In few words, with this you are asking for the Android standard app chooser and, in addition, you specify that you want to send an email. So, as result, email clients will appear only.
If user has one email client installed only, the intent will redirect to it instantly.
Hope this helps you too.
try out this code, it worked for me.
Intent intent = new Intent(Intent.ACTION_SENDTO);
intent.setData(Uri.parse("mailto:")); // only email apps should handle this
intent.putExtra(Intent.EXTRA_EMAIL, new String[]{email});
intent.putExtra(Intent.EXTRA_SUBJECT, "Subject here");
intent.putExtra(Intent.EXTRA_TEXT,"Body Here");
if (intent.resolveActivity(getPackageManager()) != null) {
startActivity(intent);
}
also add intent filter in android manifest.
<activity ...>
<intent-filter>
<action android:name="android.intent.action.SENDTO" />
<data android:scheme="mailto" />
<category android:name="android.intent.category.DEFAULT" />
</intent-filter>
In order to get it to work on Samsung and Pixel devices, we had to add the parameters on both the url and the extras
val email = "xxxx#xxxx.com"
val subject = "xxxx"
val body = "xxxx"
val selectorIntent = Intent(Intent.ACTION_SENDTO)
val urlString = "mailto:" + Uri.encode(email) + "?subject=" + Uri.encode(subject) + "&body=" + Uri.encode(body)
selectorIntent.data = Uri.parse(urlString)
val emailIntent = Intent(Intent.ACTION_SEND)
emailIntent.putExtra(Intent.EXTRA_EMAIL, arrayOf(email))
emailIntent.putExtra(Intent.EXTRA_SUBJECT, subject)
emailIntent.putExtra(Intent.EXTRA_TEXT, body)
emailIntent.selector = selectorIntent
startActivity(Intent.createChooser(emailIntent, "Send email"))
Our old code for emails stopped working some days ago.
It was the following:
public static void shareTextToEmail(Context context, String[] email, String subject, String text)
Intent emailIntent = new Intent(Intent.ACTION_SENDTO, Uri.parse("mailto:" + TextUtils.join(",", email)));
emailIntent.putExtra(Intent.EXTRA_EMAIL, email);
emailIntent.putExtra(Intent.EXTRA_SUBJECT, subject);
emailIntent.putExtra(Intent.EXTRA_TEXT, text);
try {
context.startActivity(Intent.createChooser(emailIntent, context.getString(R.string.share_email_title)));
} catch (android.content.ActivityNotFoundException e) {
Toast.makeText(context, context.getString(R.string.share_no_intent_handler_found), Toast.LENGTH_SHORT).show();
}
}
I've adopted it according to the Zak.Antonio answer:
public static void shareTextToEmail(Context context, String[] email, String subject, String text)
Intent selectorIntent = new Intent(Intent.ACTION_SENDTO);
selectorIntent.setData(Uri.parse("mailto:"));
final Intent emailIntent = new Intent(Intent.ACTION_SEND);
emailIntent.putExtra(Intent.EXTRA_EMAIL, email);
emailIntent.putExtra(Intent.EXTRA_SUBJECT, subject);
emailIntent.putExtra(Intent.EXTRA_TEXT, text);
emailIntent.setSelector(selectorIntent);
try {
context.startActivity(Intent.createChooser(emailIntent, context.getString(R.string.share_email_title)));
} catch (android.content.ActivityNotFoundException e) {
Toast.makeText(context, context.getString(R.string.share_no_intent_handler_found), Toast.LENGTH_SHORT).show();
}
}
The key points are:
Replace Intent.ACTION_SENDTO with Intent.ACTION_SEND in emailIntent
Move Intent.ACTION_SENDTO to a selectorIntent
Do not put emails in intent data, put them only in extras at Intent.EXTRA_EMAIL
If your app targets Android 11 (API level 30) or higher then we require to add :
<queries> element in the app's manifest file (Because of
Package visibility changes in Android 11)
Set the queries in the Manifest file like this:
<manifest package="com.example.app">
...
<!-- Package visibility -->
<queries>
<!-- Mail -->
<intent>
<action android:name="android.intent.action.SENDTO" />
<data android:scheme="mailto" />
<category android:name="android.intent.category.DEFAULT" />
</intent>
</queries>
...
</manifest>
Kotlin method to open Email apps :
fun composeEmail(recipient: String, subject: String, body: String) {
val selectorIntent = Intent(Intent.ACTION_SENDTO).apply{
data = Uri.parse("mailto:") // only email apps should handle this
}
val emailIntent = Intent(Intent.ACTION_SEND).apply {
putExtra(Intent.EXTRA_EMAIL, arrayOf(recipient))
putExtra(Intent.EXTRA_SUBJECT, subject)
putExtra(Intent.EXTRA_TEXT, body)
selector = selectorIntent
}
if (emailIntent.resolveActivity(packageManager) != null) {
startActivity(emailIntent)
}
}
This answer worked for me using parsing using Uri
Intent emailIntent = new Intent(Intent.ACTION_SENDTO, Uri.parse("mailto:?subject=" + subject + "&to=" + to));
context.startActivity(emailIntent);
try this code
val emailIntent = Intent(Intent.ACTION_SEND)
emailIntent.setType("text/plain")
emailIntent.putExtra(Intent.EXTRA_EMAIL, arrayOf("jon#example.com"))
emailIntent.putExtra(Intent.EXTRA_SUBJECT, "Email subject")
emailIntent.putExtra(Intent.EXTRA_TEXT, "Email message text")
val packageManager = packageManager
val activities = packageManager.queryIntentActivities(emailIntent, 0)
val isIntentSafe = activities.size > 0
if (isIntentSafe) {
startActivity(emailIntent);
}else{
Log.d("MainActivty","Email App not installed");
}
Neither adding Extras to the SENDTO intent nor using a selectorIntent seemed to work on a Samsung running Android 10. Uri encoding the email address, subject, and body seemed to work best across multiple devices running Android 10 and below. The EmailIntentLibrary was a big help in ironing out the finer points of getting the URI encoding to work for complex body content.
val email = Uri.encode("xxxx#xxxx.com")
val subject = Uri.encode("xxxx")
val body = Uri.encode("some body. one two & more. \n new line \n &%>?")
val uri = "mailto:$email?subject=$subject&body=$body"
val intent = Intent(Intent.ACTION_SENDTO)
intent.type = "text/plain"
intent.data = Uri.parse(uri)
startActivity(intent)
If you're ok with having quite a few options in the chooser, the example below from the Android Development Guide does work (the ACTION_SENDTO example meant to filter those options did not work for me)
fun composeEmail(addresses: Array<String>, subject: String, attachment: Uri) {
val intent = Intent(Intent.ACTION_SEND).apply {
type = "*/*"
putExtra(Intent.EXTRA_EMAIL, addresses)
putExtra(Intent.EXTRA_SUBJECT, subject)
putExtra(Intent.EXTRA_STREAM, attachment)
}
if (intent.resolveActivity(packageManager) != null) {
startActivity(intent)
}
}
I had issues with the other solutions, here is what worked for me:
val selectorIntent = Intent(Intent.ACTION_SENDTO).apply {
data = Uri.parse("mailto:")
}
val emailIntent = Intent(Intent.ACTION_SEND).apply {
selector = selectorIntent
putExtra(
Intent.EXTRA_EMAIL,
arrayOf(resources.getString(R.string.support_email_address))
)
putExtra(
Intent.EXTRA_SUBJECT,
resources.getString(R.string.support_email_subject)
)
putExtra(
Intent.EXTRA_TEXT,
resources.getString(R.string.support_email_body)
)
putExtra(
Intent.EXTRA_STREAM,
latestLogFileUri
)
}
emailIntent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION)
if (selectorIntent.resolveActivity(packageManager) != null) {
Handler(Looper.getMainLooper()).postDelayed({
startActivity(
Intent.createChooser(
emailIntent,
resources.getString(R.string.support_email_chooser_title)
)
)
}, 2000)
}
and don't forget to put this into your AndroidManifest.xml:
<queries>
<intent>
<action android:name="android.intent.action.SENDTO" />
<data android:scheme="mailto" />
</intent>
<intent>
<action android:name="android.intent.action.SEND" />
<data android:mimeType="*/*" />
</intent>
</queries>

pyspark is hanging while attempting to make URL requests in parallel

I have an rdd of URLs and I want to make url requests in parallel. I am running get_menus with an rdd of url strings. Pyspark hangs when trying to collect(), take(), etc. on items.
def format_json(response, slug):
clean_response = str(response)[:-1]
clean_response += ", 'slug': '" + slug + "'}\n" # ', \"store-slug\": \"' + slug +'\"}'
clean_response = clean_response.replace("None", "null").replace("True", 'true').replace("False", "false")
return clean_response
def get_all(url, slug):
response = requests.get(url)
response = json.loads(response).get('data', {})
clean_response = format_json(response, slug)
return clean_response
def get_items(rdd_of_urls):
get_items_udf = udf(lambda x: get_all(x), StringType())
items = rdd_of_urls.map(get_items_udf)
return items
I see many GC allocation errors but don't know how to interpret them:
2019-08-21T18:40:04.666+0000: [GC (Allocation Failure) 2019-08-21T18:40:04.666+0000: [ParNew: 34369K->1837K(36864K), 0.0029907 secs] 127948K->95416K(186848K), 0.0030414 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-08-21T18:40:04.951+0000: [GC (Allocation Failure) 2019-08-21T18:40:04.951+0000: [ParNew: 34605K->2576K(36864K), 0.0036551 secs] 128184K->96155K(186848K), 0.0037126 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]

Standalone Spark Task Hangs When Inserting Into DB

I have a standalone spark 1.4.1 job that runs on a Red Hat box I submit via spark-submit that sometimes hangs during insertion of data from an RDD. I have auto-commit on the connection turned off and commit the transactions in batches of insertions. What the logs show me before it hangs:
16/03/25 14:00:05 INFO Executor: Finished task 3.0 in stage 138.0 (TID 915). 1847 bytes result sent to driver
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=0 lim=1847 cap=1
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=0 lim=1847 cap=1847
16/03/25 14:00:05 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_138, runningTasks: 1
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.118 ms) AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=621 li
16/03/25 14:00:05 INFO TaskSetManager: Finished task 3.0 in stage 138.0 (TID 915) in 7407 ms on localhost (23/24)
16/03/25 14:00:05 TRACE DAGScheduler: Checking for newly runnable parent stages
16/03/25 14:00:05 TRACE DAGScheduler: running: Set(ResultStage 138)
16/03/25 14:00:05 TRACE DAGScheduler: waiting: Set()
16/03/25 14:00:05 TRACE DAGScheduler: failed: Set()
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(driver, local
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(driver, localhos
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.099 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(dri
And then it just repeats the last 3 lines with this intermittently:
16/03/25 14:01:04 TRACE HeartbeatReceiver: Checking for hosts with no recent heartbeats in HeartbeatReceiver.
The kicker is that I can't take a look at the web UI due to some firewall issues on these machines. What I noticed is that this issue was more prevalent when I was inserting with batches of 1000 than with 100. This is the scala code that looks to be the culprit.
//records should have up to INSERT_BATCH_SIZE entries
private def insertStuff(records: Seq[(String, (String, Stuff1, Stuff2, Stuff3))]) {
if (!records.isEmpty) {
//get statement used for insertion (instantiated in an array of statements)
val stmt = stuffInsertArray(//stuff)
logger.info("Starting insertions on stuff" + table + " for " + time + " with " + records.length + " records")
try {
records.foreach(record => {
//get vals from record
...
//perform sanity checks
if (//validate stuff)
{
//log stuff because it didn't validate
}
else
{
stmt.setInt(1, //stuff)
stmt.setLong(2, //stuff)
...
stmt.addBatch()
}
})
//check if connection is still valid
if (!connInsert.isValid(VALIDATE_CONNECTION_TIMEOUT))
{
logger.error("Insertion connection is not valid while inserting stuff.")
throw new RuntimeException(s"Insertion connection not valid while inserting stuff.")
}
logger.debug("Stuff insertion executing batch...")
stmt.executeBatch()
logger.debug("Stuff insertion execution complete. Committing...")
//commit insert batch. Either INSERT_BATCH_SIZE insertions planned or the last batch to be done
insertCommit() //this does the commit and resets some counters
logger.debug("stuff insertion commit complete.")
} catch {
case e: Exception => throw new RuntimeException(s"insertStuff exception ${e.getMessage}")
}
}
}
And here's how it gets called:
//stuffData is an RDD
stuffData.foreachPartition(recordIt => {
//new instance of the object of whose member function we're currently in
val obj = new Obj(clusterInfo)
recordIt.grouped(INSERT_BATCH_SIZE).foreach(records => obj.insertStuff(records))
})
All the extra logging and connection checking I put in just to isolate the issue but since I write for every batch of insertions, the logs get convoluted. If I serialize the insertions, the issue still persists. Any idea why the last task (out of 24) doesn't finish? Thanks.

java.io.IOException: Wrong index checksum, store was not closed properly and could be corrupted

I encountered the exception after restarting Jetty, which contains producer-consumer implementation via MapDB's Queue. I haven't call "DBMaker.transactionDisable()", but why did I still get the above exceptions?
Before I restarted Jetty, I found the consumer seems blocked. By jstack command, I got lots of following logs:
"qtp1730704097-270" #270 prio=5 os_prio=0 tid=0x00007f7b60006800
nid=0xb44f waiting on condition [0x00007f7986ae9000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000054047c420> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
Here are some code fragment:
// Consumer code:
public void run() {
try {
while (!stopped) {
if (System.currentTimeMillis() - lastCheckTime > 60000) {
localDb.commit();
localDb.compact();
lastCheckTime = System.currentTimeMillis();
}
...
for (...) {
queue.poll();
}
...
localDb.commit();
}
localDb.commit();
localDb.compact();
} catch (InterruptedException ie) {
LOG.error("", ie);
}
}
// Others code:
...
DB localDb = DBMaker.newFileDB(new File(path)).closeOnJvmShutdown().make();
localDb.catPut(QUEUE_NAME + ".useLocks", false);
queue = localDb.getQueue(QUEUE_NAME);
...
public void interrupt() {
stopped = true;
localDb.close();
}
java -version: java version "1.8.0_11"
Thanks!

Async Spring Batch job fails to process file

I'm trying to process a file, and upload it into a database, using spring batch, right after uploading it. However the job completes right after it's started, and I'm not too sure of the exact reason. I think it's not doing what it should, in the tasklet.execute. Below is the DEBUG output:
22:25:09.823 [http-nio-127.0.0.1-8080-exec-2] DEBUG o.s.b.c.c.a.SimpleBatchConfiguration$ReferenceTargetSource - Initializing lazy target object
22:25:09.912 [SimpleAsyncTaskExecutor-1] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [FlowJob: [name=moneyTransactionImport]] launched with the following parameters: [{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}]
22:25:09.912 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.job.AbstractJob - Job execution starting: JobExecution: id=95, version=0, startTime=null, endTime=null, lastUpdated=Tue Sep 16 22:25:09 BST 2014, status=STARTING, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=52, version=0, Job=[transactionImport]], jobParameters=[{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}]
22:25:09.971 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Resuming state=transactionImport.step with status=UNKNOWN
22:25:09.972 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Handling state=transactionImport.step
22:25:10.018 [SimpleAsyncTaskExecutor-1] INFO o.s.batch.core.job.SimpleStepHandler - Executing step: [step]
22:25:10.019 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.step.AbstractStep - Executing: id=93
22:25:10.072 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.scope.StepScope - Creating object in scope=step, name=scopedTarget.reader
22:25:10.117 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.scope.StepScope - Registered destruction callback in scope=step, name=scopedTarget.reader
22:25:10.136 [SimpleAsyncTaskExecutor-1] WARN o.s.b.item.file.FlatFileItemReader - Input resource does not exist class path resource [C:/Users/test/AppData/Local/Temp/tomcat.1435325122308787143.8080/uploads/test.csv]
22:25:10.180 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Starting repeat context.
22:25:10.181 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat operation about to start at count=1
22:25:10.181 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.s.c.StepContextRepeatCallback - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext#5d85b879
22:25:10.181 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.s.c.StepContextRepeatCallback - Chunk execution starting: queue size=0
22:25:12.333 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Starting repeat context.
22:25:12.333 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat operation about to start at count=1
22:25:12.334 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat is complete according to policy and result value.
22:25:12.334 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.s.item.ChunkOrientedTasklet - Inputs not busy, ended: true
22:25:12.334 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.core.step.tasklet.TaskletStep - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
22:25:12.337 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.core.step.tasklet.TaskletStep - Saving step execution before commit: StepExecution: id=93, version=1, name=step, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
22:25:12.358 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat is complete according to policy and result value.
22:25:12.358 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.step.AbstractStep - Step execution success: id=93
22:25:12.419 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.step.AbstractStep - Step execution complete: StepExecution: id=93, version=3, name=step, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0
22:25:12.442 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Completed state=transactionImport.step with status=COMPLETED
22:25:12.443 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Handling state=transactionImport.COMPLETED
22:25:12.443 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Completed state=transactionImport.COMPLETED with status=COMPLETED
22:25:12.445 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.job.AbstractJob - Job execution complete: JobExecution: id=95, version=1, startTime=Tue Sep 16 22:25:09 BST 2014, endTime=null, lastUpdated=Tue Sep 16 22:25:09 BST 2014, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=52, version=0, Job=[transactionImport]], jobParameters=[{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}]
22:25:12.466 [SimpleAsyncTaskExecutor-1] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [FlowJob: [name=transactionImport]] completed with the following parameters: [{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}] and the following status: [COMPLETED]
My config is as follows:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Inject
private TransactionRepository transactionRepository;
#Inject
private JobRepository jobRepository;
#Bean
#StepScope
public FlatFileItemReader<MoneyTransaction> reader(#Value("#{jobParameters[targetFile]}") String file) {
FlatFileItemReader<MoneyTransaction> reader = new FlatFileItemReader<>();
reader.setResource(new ClassPathResource(file));
reader.setLineMapper(new DefaultLineMapper<MoneyTransaction>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(new String[]{"Number", "Date", "Account", "Payee", "Cleared", "Amount", "Category", "Subcategory", "Memo"});
}
}
);
setFieldSetMapper(new BeanWrapperFieldSetMapper<MoneyTransaction>() {
{
setTargetType(MoneyTransaction.class);
}
});
}
}
);
reader.setStrict(false);
reader.setLinesToSkip(1);
return reader;
}
#Bean
public ItemProcessor<MoneyTransaction, Transaction> processor() {
return new TransactionProcessor();
}
#Bean
public RepositoryItemWriter writer() {
RepositoryItemWriter writer = new RepositoryItemWriter();
writer.setRepository(transactionRepository);
writer.setMethodName("save");
return writer;
}
#Bean
public Step step(StepBuilderFactory stepBuilderFactory, ItemReader<MoneyTransaction> reader,
ItemWriter<Transaction> writer, ItemProcessor<MoneyTransaction, Transaction> processor) {
return stepBuilderFactory.get("step")
.<MoneyTransaction, Transaction>chunk(100)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
#Bean
public SimpleAsyncTaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor();
executor.setConcurrencyLimit(1);
return executor;
}
#Bean
public SimpleJobLauncher jobLauncher() {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
jobLauncher.setTaskExecutor(taskExecutor());
return jobLauncher;
}
}
And I save the file, and start processing in the following way:
public JobExecution processFile(String name, MultipartFile file) {
if (!file.isEmpty()) {
try {
byte[] bytes = file.getBytes();
String rootPath = System.getProperty("catalina.home");
File uploadDirectory = new File(rootPath.concat(File.separator).concat("uploads"));
if (!uploadDirectory.exists()) {
uploadDirectory.mkdirs();
}
File uploadFile = new File(uploadDirectory.getAbsolutePath() + File.separator + file.getOriginalFilename());
BufferedOutputStream stream =
new BufferedOutputStream(new FileOutputStream(uploadFile));
stream.write(bytes);
stream.close();
return startImportJob(uploadFile, "transactionImport");
} catch (Exception e) {
logger.error(String.format("Error processing file '%s'.", name), e);
throw new MoneyException(e);
}
} else {
throw new MoneyException("There was no file to process.");
}
}
/**
* #param file
*/
private JobExecution startImportJob(File file, String jobName) {
logger.debug(String.format("Starting job to import file '%s'.", file));
try {
Job job = jobs.get(jobName).incrementer(new MoneyRunIdIncrementer()).flow(step).end().build();
return jobLauncher.run(job, new JobParametersBuilder().addString("targetFile", file.getAbsolutePath()).toJobParameters());
} catch (JobExecutionAlreadyRunningException e) {
logger.error(String.format("Job for processing file '%s' is already running.", file), e);
throw new MoneyException(e);
} catch (JobParametersInvalidException e) {
logger.error(String.format("Invalid parameters for processing of file '%s'.", file), e);
throw new MoneyException(e);
} catch (JobRestartException e) {
logger.error(String.format("Error restarting job, for processing file '%s'.", file), e);
throw new MoneyException(e);
} catch (JobInstanceAlreadyCompleteException e) {
logger.error(String.format("Job to process file '%s' has already completed.", file), e);
throw new MoneyException(e);
}
}
I'm kind of stumped at the minute, and any help would be greatly received.
Thanks.
Found the issue. The problem was with the type of resource ClassPathResource(file), combined with the fact that I was setting the strict property to false.
reader.setResource(new ClassPathResource(file));
I should have used
reader.setResource(new FileSystemResource(file));
Which makes complete sense, as I wasn't uploading the file as a class path resource.