Apache Camel File process is resulting in TypeConversion Error - scala

I am using akka-camel to process files. My initial tests were working great, however when I started passing in actual xml files it is puking with type conversions.
Here is my consumer (very simple, but puking at msg.bodyAs[String]
class FileConsumer extends Consumer {
def endpointUri = "file:/data/input/actor"
val processor = context.actorOf(Props[Processor], "processor")
def receive = {
case msg: CamelMessage => {
println("Parent...received %s" format msg)
processor ! msg.bodyAs[String]
}
}
}
Error:
[ERROR] [04/27/2015 12:10:48.617] [ArdisSystem-akka.actor.default-dispatcher-5] [akka://ArdisSystem/user/$a] Error during type conversion from type: org.apache.camel.converter.stream.FileInputStreamCache to the required type: java.lang.String with value org.apache.camel.converter.stream.FileInputStreamCache#4611b35a due java.io.FileNotFoundException: /var/folders/dh/zfqvn9gn7cl6h63d3400y4zxp3xtzf/T/camel-tmp-807558/cos2920459202139947606.tmp (No such file or directory)
org.apache.camel.TypeConversionException: Error during type conversion from type: org.apache.camel.converter.stream.FileInputStreamCache to the required type: java.lang.String with value org.apache.camel.converter.stream.FileInputStreamCache#4611b35a due java.io.FileNotFoundException: /var/folders/dh/zfqvn9gn7cl6h63d3400y4zxp3xtzf/T/camel-tmp-807558/cos2920459202139947606.tmp (No such file or directory)
I am wondering if it has something to do with the actual contents of the xml. They are not big at all (roughly 70kb). I doubt I will be able to provide an actual example of the XML itself. Just baffled as to why something so small and being converted to a string is having issues. Other dummy example xml files have worked fine.
EDIT:
One of the suggestions I had was to enable StreamCache, which I did. However, it still doesn't seem to be working. As Ankush commented, the error is confusing. I am not sure if it actually is a Stream issue or if it really is a conversion problem.
http://camel.apache.org/stream-caching.html
Added the below
camel.context.setStreamCaching(true)

I was finally able to figure out the problem. The issue was not bad data, but the size of the files. To account for this, you need to add addtional settings to the camel context.
http://camel.apache.org/stream-caching.html
The settings I used are below. I will need to further research if I should just turn off the streamcache, but this is a start.
camel.context.getProperties.put(CachedOutputStream.THRESHOLD, "750000");
or turn off streamcache
camel.context.setStreamCaching(false)
Hope this helps someone else.

we were having same issue commenting the streamCaching() helped
from(IEricssonConstant.ROUTE_USAGE_DATA_INDIVIDUAL_PROCSESS)
//.streamCaching()
.split(new ZipSplitter()) .stopOnException()
.streaming()
.unmarshal().csv()
.process(new UsageDataCSVRequestProcessor())

Related

How can I use a Session attribute saved from a Check in an SSE request in Gatling?

See working Java code at bottom.
I'm trying to create an SSE connection testing scenario that uses a client ID that is the response to a previous REST POST call. Here's the Scala code:
package computerdatabase
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class ssetest extends Simulation {
val registration = scenario("registration")
.exec(http("endpoint1")
.post("http://localhost:7070/api/register")
.body(RawFileBody("./Stocks.json")).asJson
.check(status is 201)
.check(jsonPath("$.id").saveAs("clientId")))
setUp(scn.inject(constantUsersPerSec(200).during(180))).assertions(global.failedRequests.count.is(0))
val sseConnection = scenario("ServerSentEvents")
.exec { session =>
sse("GET messages")
.connect("/api/sse/" + session("clientId").as[String])
.await(1)(
sse.checkMessage("ConnectionCheck").matching(substring("connection established"))
.check(bodyString.saveAs("InitialMessage"))
)
}
}
I would like to use the client ID saved into the Session in the first scenario, to configure the SSE endpoint to connect to in the second one. Presently, when running the scenario, I'm getting:
19:35:50.086 [ERROR] i.g.c.ZincCompiler$ - /home/dingo/Downloads/gatling-charts-highcharts-bundle-3.7.6/user-files/simulations/computerdatabase/SseSimulation.scala:21:15: type mismatch;
found : io.gatling.http.request.builder.sse.SseConnectRequestBuilder
required: io.gatling.commons.validation.Validation[io.gatling.core.session.Session]
.await(1)(
^
19:35:50.135 [ERROR] i.g.c.ZincCompiler$ - one error found
19:35:50.145 [ERROR] i.g.c.ZincCompiler$ - Compilation crashed
xsbt.InterfaceCompileFailed: null
at xsbt.CachedCompiler0.handleErrors(CompilerBridge.scala:183)
at xsbt.CachedCompiler0.run(CompilerBridge.scala:172)
at xsbt.CachedCompiler0.run(CompilerBridge.scala:134)
at xsbt.CompilerBridge.run(CompilerBridge.scala:39)
at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:91)
at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$7(MixedAnalyzingCompiler.scala:192)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at sbt.internal.inc.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:247)
at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4(MixedAnalyzingCompiler.scala:182)
at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4$adapted(MixedAnalyzingCompiler.scala:163)
at sbt.internal.inc.JarUtils$.withPreviousJar(JarUtils.scala:239)
at sbt.internal.inc.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:163)
at sbt.internal.inc.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:210)
at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1(IncrementalCompilerImpl.scala:528)
at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1$adapted(IncrementalCompilerImpl.scala:528)
at sbt.internal.inc.Incremental$.$anonfun$apply$5(Incremental.scala:177)
at sbt.internal.inc.Incremental$.$anonfun$apply$5$adapted(Incremental.scala:175)
at sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:461)
at sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:116)
at sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
at sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
at sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:263)
at sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:416)
at sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:503)
at sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:403)
at sbt.internal.inc.Incremental$.apply(Incremental.scala:169)
at sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:528)
at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:482)
at sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
... 5 common frames omitted
Wrapped by: sbt.internal.inc.CompileFailed: Compilation failed
at sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:335)
at sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:420)
at sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
at io.gatling.compiler.ZincCompiler$.doCompile(ZincCompiler.scala:218)
at io.gatling.compiler.ZincCompiler$.main(ZincCompiler.scala:226)
at io.gatling.compiler.ZincCompiler.main(ZincCompiler.scala)
How can I fix this?
Disclaimer: I don't know Scala.
Actions taken:
Created a new folder structure for Gatling in my project
Put the simulation class in ./src/gatling/java/computerdatabase
Put Stocks.json in ./src/gatling/resources/Stocks.json and made the file path in the code absolute
Added the Gatling Gradle plugin to build.gradle
Checked this example Simulation for inspiration
Fiddled around a bit with the code, reading about Session function syntax
package computerdatabase;
import io.gatling.javaapi.core.ScenarioBuilder;
import io.gatling.javaapi.core.Simulation;
import io.gatling.javaapi.http.HttpProtocolBuilder;
import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;
public class SseSimulation extends Simulation {
HttpProtocolBuilder httpProtocol = http
.baseUrl("http://localhost:7070");
ScenarioBuilder scn = scenario("Scenario Name")
.exec(http("client id registration")
.post("/api/register/")
.body(RawFileBody("Stocks.json")).asJson()
.check(status().is(201))
.check(jsonPath("$.id").saveAs("clientId")))
.exec(
sse("open channel")
.connect(session -> "/api/sse/" + session.getString("clientId"))
);
{
setUp(scn.injectOpen(constantUsersPerSec(200).during(1)).protocols(httpProtocol)).assertions(global().failedRequests().count().is(0L));
}
}
Disclaimer: I don't know Scala.
Then switch to Java. Gatling 3.7 (that you're using) supports it as well and both the official documentation and the official online courses explain it very well.
Lots of things wrong:
the code you've provided can't possibly compile. There's nothing named scn in there. Please fix it so one can help you.
your compile error has nothing to do with the piece of code you've provided. The compiler complains about an error in SseSimulation while you've provided a class named ssetest. Again, please first fix your code sample
RawFileBody("./Stocks.json") is wrong. You shouldn't use a relative path. Let the ClassLoader resolve your files. This Stocks.json file should go in user-files/resources and you should write RawFileBody("Stocks.json")
Checks save data into the Session of the virtual user that executed the request. You can't use different scenarios because they would execute different sets of virtual users. You have to perform register and SSE connect in the same scenario.
I recommend that you should invest some time in reading the documentation and checking the online courses, you will ultimately save a lot of time.

How to provide an explicit error/failure message in the Scala fastparse library?

I'm using Li Haoyi's FastParse library. I have several situations where I'd like to provide explicit failure messages.
For example:
def courseRE[p: P]: P[Regex] =
P(CharIn("ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.|*+[]()-^$").repX(1).!).map { re =>
try { re.r }
catch { case e => failure(s"Ill-formed regular expression: ${re}.") }
}
But there is (apparently) no failure function.
Any suggested work-arounds? Simply throwing an exception doesn't give any context information.
I haven't yet found a good solution. I don't like the solution proposed by #user2963757 because it loses all the information from the parser about what it was looking for, where it was looking, etc.
This is raised a number of times in the FastParse issues list on GitHub. See issue 213, issue 187, issue 243, and pull request 244. There are a few vague suggestions but as far as I can tell the pull request hasn't been acted on (as of 2023-02-09).
The best I've found so far is defining this in an accessible location:
// Fail with a message. See https://github.com/com-lihaoyi/fastparse/issues/213
// The message shows up as "Expected ..."; phrase it appropriately.
private def Fail[T](expected: String)(implicit ctx: P[_]): P[T] = {
val res = ctx.freshFailure()
if (ctx.verboseFailures) ctx.setMsg(ctx.index, () => expected)
res
}
To use it:
P(CharIn("ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.|*+[]()-^$").repX(1).!).flatMap(re =>
Try(re.r)
.map(Pass(_))
.getOrElse(Fail("<Well-formed regular expression>"))
)
Trying to parse "^CS1[1345" yields
Expected <Well-formed regular expression>:1:10, found ""
Notice that the failure message has to be stated in terms of what was expected, not the actual problem. The actual error message thrown by the exception usually doesn't work well in this situation. I'm also not getting the fragment that it found.
Unfortunately, even this message is usually unavailable. For example, parsing a larger piece of my input results in
Expected (courseSpecDef | minUnits | xOf | courseSpec):1:14, found "^CS1[1345 "
I'd like to be able to surface the more exact error of "Unclosed character class" but seemingly can't.
By the way, I looked in the documentation, source code, and the sample parsers (PythonParse and ScalaParse) for examples of the use of the Fail parser. Can't find any. The only one is the one shown in the documentation, which doesn't compose with another parser.
If anyone has a better solution, I'd still love to hear it.

Stop huge error output from testing-library

I love testing-library, have used it a lot in a React project, and I'm trying to use it in an Angular project now - but I've always struggled with the enormous error output, including the HTML text of the render. Not only is this not usually helpful (I couldn't find an element, here's the HTML where it isn't); but it gets truncated, often before the interesting line if you're running in debug mode.
I simply added it as a library alongside the standard Angular Karma+Jasmine setup.
I'm sure you could say the components I'm testing are too large if the HTML output causes my console window to spool for ages, but I have a lot of integration tests in Protractor, and they are SO SLOW :(.
I would say the best solution would be to use the configure method and pass a custom function for getElementError which does what you want.
You can read about configuration here: https://testing-library.com/docs/dom-testing-library/api-configuration
An example of this might look like:
configure({
getElementError: (message: string, container) => {
const error = new Error(message);
error.name = 'TestingLibraryElementError';
error.stack = null;
return error;
},
});
You can then put this in any single test file or use Jest's setupFiles or setupFilesAfterEnv config options to have it run globally.
I am assuming you running jest with rtl in your project.
I personally wouldn't turn it off as it's there to help us, but everyone has a way so if you have your reasons, then fair enough.
1. If you want to disable errors for a specific test, you can mock the console.error.
it('disable error example', () => {
const errorObject = console.error; //store the state of the object
console.error = jest.fn(); // mock the object
// code
//assertion (expect)
console.error = errorObject; // assign it back so you can use it in the next test
});
2. If you want to silence it for all the test, you could use the jest --silent CLI option. Check the docs
The above might even disable the DOM printing that is done by rtl, I am not sure as I haven't tried this, but if you look at the docs I linked, it says
"Prevent tests from printing messages through the console."
Now you almost certainly have everything disabled except the DOM recommendations if the above doesn't work. On that case you might look into react-testing-library's source code and find out what is used for those print statements. Is it a console.log? is it a console.warn? When you got that, just mock it out like option 1 above.
UPDATE
After some digging, I found out that all testing-library DOM printing is built on prettyDOM();
While prettyDOM() can't be disabled you can limit the number of lines to 0, and that would just give you the error message and three dots ... below the message.
Here is an example printout, I messed around with:
TestingLibraryElementError: Unable to find an element with the text: Hello ther. This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible.
...
All you need to do is to pass in an environment variable before executing your test suite, so for example with an npm script it would look like:
DEBUG_PRINT_LIMIT=0 npm run test
Here is the doc
UPDATE 2:
As per the OP's FR on github this can also be achieved without injecting in a global variable to limit the PrettyDOM line output (in case if it's used elsewhere). The getElementError config option need to be changed:
dom-testing-library/src/config.js
// called when getBy* queries fail. (message, container) => Error
getElementError(message, container) {
const error = new Error(
[message, prettyDOM(container)].filter(Boolean).join('\n\n'),
)
error.name = 'TestingLibraryElementError'
return error
},
The callstack can also be removed
You can change how the message is built by setting the DOM testing library message building function with config. In my Angular project I added this to test.js:
configure({
getElementError: (message: string, container) => {
const error = new Error(message);
error.name = 'TestingLibraryElementError';
error.stack = null;
return error;
},
});
This was answered here: https://github.com/testing-library/dom-testing-library/issues/773 by https://github.com/wyze.

Proper format for log4j2.xml RollingFile configuration

I am getting the following exception in my glassfish 4 application that uses log4j2:
SEVERE: ERROR StatusLogger Invalid URL C:/glassfish4/glassfish/domains/domain1/config/log4j2.xml java.net.MalformedURLException: Unknown protocol: c
I have the following section in my log4j2.xml:
<RollingFile name="RollingFile" fileName="C:/glassfish4/glassfish/domains/domain1/logs/ucsvc.log"
filePattern="C:/glassfish4/glassfish/domains/domain1/logs/$${date:yyyy-MM}/ucsvc-%d{MM-dd-yyyy}-%i.log">
I understand that if it's looking for a URL, then "C:/glassfish4/..." is not the correct format.
However, the rolling file part actually works: I see a log file and the rolled log files where I expect them.
If I change to a URL (e.g. file:///C/glassfish4/...) that doesn't work at all.
So should I ignore the exception? (everything seems to be working ok). Or can someone explain the correct format for this section of the configuration?
I have not yet fully determined why it is that the config file works for me as well as the OP, but, I can confirm that changing the path reference to a file:// url solves the problem (ie: gets rid of the error/warning/irritant).
In my IntelliJ Run/Debug configurations, for VM options, I have:
-Dlog4j.configurationFile=file://C:\dev\path\to\log4j2.xml
I can confirm that '\' are translated to '/' so, no worries there.
EDIT:
Okay, the whole thing works because they (the apache guys) try really hard to load the configuration and they do, in fact, load from the file as specified via the c:\... notation. They just throw up a rather misleading exception before continuing to try.
In ConfigurationFactory::getConfiguration:
**source = getInputFromURI(FileUtils.getCorrectedFilePathUri(config));**
} catch (Exception ex) {
// Ignore the error and try as a String.
}
if (source == null) {
final ClassLoader loader = this.getClass().getClassLoader();
**source = getInputFromString(config, loader);**
The first bolded line tries to load from a URL and fails, throwing the exception. The code then continues, pops into getInputFromString:
try {
final URL url = new URL(config);
return new ConfigurationSource(url.openStream(), FileUtils.fileFromURI(url.toURI()));
} catch (final Exception ex) {
final ConfigurationSource source = getInputFromResource(config, loader);
if (source == null) {
try {
**final File file = new File(config);
return new ConfigurationSource(new FileInputStream(file), file);**
Where it tries to load the config again, fails and falls into the catch, tries again, fails and finally succeeds on the bolded lines (dealing with a File).
Okay, the code lines I wanted in emphasize with bold are actually just wrapped in **; guess the site doesn't permit nested tags? Anyway, y'all get the meaning.
It's all a bit of a mess to read, but that's why it works even though you get that nasty-looking (and wholly misleading) exception.
Thanks Jon, i was searching all over.. this helped!
This is on Intellij 13, Tomcat 7.0.56
-Dlog4j.configurationFile=file://C:\Surendra\workspace\cmdb\resources\elasticityLogging.xml
The problem is not the contents of your log4j2.xml file.
The problem is that log4j2 cannot locate your log4j2.xml config file. If you look carefully at the error, the URL that is reported as invalid is C:/glassfish4/glassfish/domains/domain1/config/log4j2.xml: the config file.
I'm not sure why this is. Are you specifying the location of the config file via the system property -Dlog4j.configurationFile=path/to/log4j2.xml?
Still, if the application and logging works then perhaps there is no problem. Strange though. You can get more details about the log4j configuration by configuring <Configuration status="trace"> at the top of your log4j2.xml file. This will print log4j initialization details to the console.

GetExportedValues<MyType> returns nothing, I can see the parts

I have a strange MEF problem, I tested this in a test project and it all seems to work pretty well but for some reason not working in the real project
This is the exporting code
public void RegisterComponents()
{
_registrationBuilder = new RegistrationBuilder();
_registrationBuilder
.ForTypesDerivedFrom(typeof(MyType))
.SetCreationPolicy(CreationPolicy.NonShared)
.Export();
var catalog = new AggregateCatalog();
catalog.Catalogs.Add(new AssemblyCatalog(typeof(MyType).Assembly, _registrationBuilder));
var directoryCatalog = new DirectoryCatalog(PathToMyTypeDerived, _registrationBuilder);
catalog.Catalogs.Add(directoryCatalog);
_compositionContainer = new CompositionContainer(catalog);
_compositionContainer.ComposeParts();
var exports = _compositionContainer.GetExportedValues<MyType>();
Console.WriteLine("{0} exports in AppDomain {1}", exports.Count(), AppDomain.CurrentDomain.FriendlyName);
}
exports count is 0 :( Any ideas why?
IN the log file I have many of this
System.ComponentModel.Composition Information: 6 : The ComposablePartDefinition 'SomeOthertype' was ignored because it contains no exports.
Though I would think this is ok because I wasn' interested in exporting 'someOtherType'
UPDATE: I found this link but after debuging over it I am not wiser but maybe I m not following up properly.
Thanks for any pointers
Cheers
I just had the same problem and this article helped me a lot.
It describes different reasons why a resolve can fail. One of the more important ones is that the dependency of a dependency of the type you want to resolve is not registered.
What helped me a lot was the the trace output that gets written to the Output window when you debug your application. It describes exactly the reasons why a type couldn't be resolved.
Even with this output. you might need to dig a little bit, because I only got one level deep.
Example:
I wanted to resolve type A and I got a message like this:
System.ComponentModel.Composition Warning: 1 : The ComposablePartDefinition 'Namespace.A' has been rejected. The composition remains unchanged. The changes were rejected because of the following error(s): The composition produced multiple composition errors, with 1 root causes. The root causes are provided below. Review the CompositionException.Errors property for more detailed information.
1) No exports were found that match the constraint:
ContractName Namespace.IB
RequiredTypeIdentity Namespace.IB
Resulting in: Cannot set import 'Namespace.A..ctor (Parameter="b", ContractName="namespace.IB")' on part 'Namespace A'.
Element: Namespace.A..ctor (Parameter="b", ContractName="Namespace.IB") --> Namespace.A --> AssemblyCatalog (Assembly="assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=...")
But I clearly saw a part for Namespace.IB. So, in the debugger, I tried to resolve that one. And I got another trace output. This time it told me that my implementation of Namespace.IB couldn't be resolved because for one of its imports there was a missing export, so basically the same message as above, just with different types. And this time, I didn't find a part for that missing import. Now I knew, which type was the real problem and figure out, why no registration happened for it.