Task not serializable : Spark - scala

My spark job is throwing Task not serializable at runtime. Can anyone tell me if what i am doing wrong here?
#Component("loader")
#Slf4j
public class LoaderSpark implements SparkJob {
private static final int MAX_VERSIONS = 1;
private final AppProperties props;
public LoaderSpark(
final AppProperties props
) {
this.props = props;
}
#Override
public void run(SparkSession spark, final String... args) throws IOException {
HBaseUtil hBaseUtil = new HBaseUtil(props);
byte[][] prefixes = new byte[][]{toBytes("document"),
toBytes("dataSource"),
toBytes("hold:")};
Filter filter = new MultipleColumnPrefixFilter(prefixes);
Scan scan = new Scan();
scan.addFamily(toBytes("data"));
scan.setCaching(100000);
scan.setMaxVersions(MAX_VERSIONS);
scan.setFilter(filter);
JavaRDD<TestMethod> mapFileJavaRDD
= hBaseUtil.createScanRdd(spark, "TestTable", scan).mapPartitions(tuple -> {
return StreamUtils.asStream(tuple)
.map(this::extractResult)
.filter(Objects::nonNull)
.iterator();
});
Dataset<TestMethod> testDataset = spark.createDataset(mapFileJavaRDD.rdd(), bean(TestMethod.class));
testDataset.limit(100);
}
private TestMethod extractResult(Tuple2<ImmutableBytesWritable, Result> resultTuple) {
TestMethod.TestMethodBuilder testBuilder = TestMethod.builder();
Result result;
result = resultTuple._2();
CdfParser cdfParser = new CdfParser();
List<String> holdingId = new ArrayList<>();
testBuilder.dataSource(Bytes.toString(result.getValue(Bytes.toBytes("data"),
Bytes.toBytes("dataSource"))));
testBuilder.document(cdfParser.fromXml(result.getValue(Bytes.toBytes("data"),
Bytes.toBytes("document"))));
NavigableMap<byte[], byte[]> familyMap = result.getFamilyMap(Bytes.toBytes("data"));
for (byte[] bQunitifer : familyMap.keySet()) {
if (Bytes.toString(bQunitifer).contains("hold:")) {
LOG.info(Bytes.toString(bQunitifer));
holdingId.add(Bytes.toString(bQunitifer));
}
}
testBuilder.holding(holdingId);
return testBuilder.build();
}
}
HERE is the stacktrace:
2020-04-29 12:48:59,837 INFO [Thread-4]o.a.s.d.y.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.oclc.googlelinks.spark.SpringSparkJob.main(SpringSparkJob.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:694)
Caused by: org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:403)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:393)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:798)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:797)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:797)
at org.apache.spark.api.java.JavaRDDLike$class.mapPartitions(JavaRDDLike.scala:155)
at org.apache.spark.api.java.AbstractJavaRDDLike.mapPartitions(JavaRDDLike.scala:45)
at org.oclc.googlelinks.spark.job.LoaderSpark.run(LoaderSpark.java:79)
at org.oclc.googlelinks.spark.SpringSparkJob.run(SpringSparkJob.java:79)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784)
... 8 more

try adding getter and setters for props
public void setProps(AppProperties props) {
this.props = props;
}
public AppProperties getProps() {
return props;
}

Just make the function extractResult static . In order to call a static method, you don’t need to serialize the class, you need the declaring class to be reachable by the classloader (and it is the case, as the jar archives can be shared among driver and workers).
Thanks to https://www.nicolaferraro.me/2016/02/22/using-non-serializable-objects-in-apache-spark/

Related

Class not found when trying to return List from consumeEvent in Quarkus

When I try to return a Uni with a typed java.util.List in Quarkus in dev mode, i get a ClassNotFound exception. I have read about Quarkus using different class loaders in different profiles, but I don't see that I do anything fancy.
Here's the sender
#Query("offers")
public Uni<List<OfferResponse>> getOffers(#PathParam("category") Integer categoryId) {
OfferRequest event = new OfferRequest();
event.setCategoryId(categoryId);
Uni<List<OfferResponse>> offers = bus.<List<OfferResponse>>request(OfferRequest.ADDRESS, event).onItem().transform(Message::body);
return offers;
}
And here's the consumer
#ConsumeEvent(OfferRequest.ADDRESS)
public Uni<List<OfferResponse>> onOfferQuery(OfferRequest request) {
List<KelkooOffer> offers = getOffers(request.getCategoryId());
List<OfferResponse> responses = new ArrayList<OfferResponse>();
for (KelkooOffer offer : offers) {
responses.add(offer.getEventResponse());
}
return Uni.createFrom().item(responses);
}
The bean I'm trying to return is just a POJO
and the error message
2021-08-18 11:11:16,186 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile dev): java.lang.ClassNotFoundException: java.util.List<se.bryderi.events.OfferResponse>
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:414)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:405)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:414)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:405)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at io.quarkus.deployment.steps.VertxProcessor$build609260703.deploy_0(VertxProcessor$build609260703.zig:142)
at io.quarkus.deployment.steps.VertxProcessor$build609260703.deploy(VertxProcessor$build609260703.zig:40)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:784)
at io.quarkus.runtime.Application.start(Application.java:101)
at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:101)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:66)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:42)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:119)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:98)
at java.base/java.lang.Thread.run(Thread.java:829)
I get the same result if I run the dev profile or if I run the packaged fast-jar.
Happy for any hint that will point me in the right direction
There is a limitation of the Vert.x EventBus which prevents to encode/decode lists or sets directly.
You could create a wrapper for your OfferResponse list:
import java.util.AbstractList;
import java.util.List;
public class OfferResponseList extends AbstractList<OfferResponse> {
private List<OfferResponse> wrapped;
private OfferResponseList(List<OfferResponse> wrapped) {
this.wrapped = wrapped;
}
public static OfferResponseList wrap(List<OfferResponse> list) {
return new OfferResponseList(list);
}
#Override
public OfferResponse get(int index) {
return wrapped.get(index);
}
#Override
public int size() {
return wrapped.size();
}
}
Then transform your consumer to:
#ConsumeEvent(OfferRequest.ADDRESS)
public Uni<OfferResponseList> onOfferQuery(OfferRequest request) {
List<KelkooOffer> offers = getOffers(request.getCategoryId());
List<OfferResponse> responses = new ArrayList<OfferResponse>();
for (KelkooOffer offer : offers) {
responses.add(offer.getEventResponse());
}
// Notice the call to OfferResponseList.wrap here
return Uni.createFrom().item(OfferResponseList.wrap(responses));
}
That's it. Quarkus will register a codec for OfferResponseList automatically.
On the client side, you do not need to change your code:
#Query("offers")
public Uni<List<OfferResponse>> getOffers(#PathParam("category") Integer categoryId) {
OfferRequest event = new OfferRequest();
event.setCategoryId(categoryId);
// You can use Uni<List<OfferResponse>> or Uni<OfferResponseList>
Uni<List<OfferResponse>> offers = bus.<List<OfferResponse>>request(OfferRequest.ADDRESS, event).onItem().transform(Message::body);
return offers;
}

Usage of timer and side input on ParDo in Apache Beam

I'm trying to write a ParDo, which will use both Timer and Side Input, but it crashes when I try to run it with beam-runners-direct-java with IllegalArgumentException on a line https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java#L167, because there are actually two inputs to ParDo (main PCollection and side input), while only one is expected.
Is there some way to workaround this? Is this a bug in Beam?
Here's the code snippet that reproduces that behaviour:
public class TestCrashesForTimerAndSideInput {
#Rule
public final transient TestPipeline p = TestPipeline.create();
#RequiredArgsConstructor
private static class DoFnWithTimer extends DoFn<KV<String, String>, String> {
private final PCollectionView<Map<String, String>> sideInput;
#TimerId("t")
private final TimerSpec tSpec = TimerSpecs.timer(TimeDomain.PROCESSING_TIME);
#ProcessElement
public void processElement(ProcessContext c, #TimerId("t") Timer t) {
KV<String, String> element = c.element();
c.output(element.getKey() + c.sideInput(sideInput).get(element));
t.offset(Duration.standardSeconds(1)).setRelative();
}
#OnTimer("t")
public void onTimerFire(OnTimerContext x) {
x.output("Timer fired");
}
}
#Test
public void testCrashesForTimerAndSideInput() {
ImmutableMap<String, String> sideData = ImmutableMap.<String, String>builder().
put("x", "X").
put("y", "Y").
build();
PCollectionView<Map<String, String>> sideInput =
p.apply(Create.of(sideData)).apply(View.asMap());
TestStream<String> testStream = TestStream.create(StringUtf8Coder.of()).
addElements("x").
advanceProcessingTime(Duration.standardSeconds(1)).
addElements("y").
advanceProcessingTime(Duration.standardSeconds(1)).
advanceWatermarkToInfinity();
PCollection<String> result = p.
apply(testStream).
apply(MapElements.into(kvs(strings(), strings())).via(v -> KV.of(v, v))).
apply(ParDo.of(new DoFnWithTimer(sideInput)).withSideInputs(sideInput));
PAssert.that(result).containsInAnyOrder("X", "Y", "Timer fired");
p.run();
}
}
and the exception:
java.lang.IllegalArgumentException: expected one element but was: <ParDo(DoFnWithTimer)/ParMultiDo(DoFnWithTimer)/To KeyedWorkItem/ParMultiDo(ToKeyedWorkItem).output [PCollection], View.AsMap/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization).output [PCollection]>
at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:322)
at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
at org.apache.beam.runners.direct.QuiescenceDriver.fireTimers(QuiescenceDriver.java:167)
at org.apache.beam.runners.direct.QuiescenceDriver.drive(QuiescenceDriver.java:110)
at org.apache.beam.runners.direct.ExecutorServiceParallelExecutor$2.run(ExecutorServiceParallelExecutor.java:170)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Use JMockit test the RESTful API,RestTemplate,error missing invocation

I am learning to use the JMockit to test my Restful API.The missing invocation error is abusing me.
The Restful API:
#RestController
public class AgentInfoRest {
#Autowired
AgentInfoRepository agentInfoRepository;
private Logger logger = LoggerFactory.getLogger(AgentInfoRest.class);
#RequestMapping("/allagentInfo")
RestResponse getAllAgentInfo()
{
RestResponse restResponse = new RestResponse();
logger.info("getAllAgentInfo has no request parameter");
Collection<AgentInfo> result = agentInfoRepository.getAllAgents();
if(result.isEmpty())
{
restResponse.setRetCode(-1);
restResponse.setRetMsg("Request success,no such record in db");
}
restResponse.setResult(result);
return restResponse;
}
}
The Test Class:
public class AgentInfoRestTest {
String host = "http://127.0.0.1:8080";
#Test
public void getAllAgentInfoTest(#Mocked final AgentInfoRepository agentInfoRepository) {
RestTemplate restTemplate = new RestTemplate();
RestResponse restResponse = new RestResponse();
AgentInfoRest agentInfoRest = new AgentInfoRest();
new StrictExpectations() {
{
agentInfoRepository.getAllAgents(); result = null; times = 1;
}
};
String url = "/allagentInfo";
ResponseEntity<RestResponse> result = restTemplate.getForEntity(host + url, RestResponse.class);
new FullVerifications() {
{
}
};
}
}
The Error:
mockit.internal.MissingInvocation: Missing invocation of:
com.egoo.dao.repository.freelink.AgentInfoRepository#getAllAgents()
on mock instance: com.egoo.dao.repository.freelink.$Impl_AgentInfoRepository#50a7bc6e
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: Missing invocation
at com.egoo.dao.rest.freelink.AgentInfoRestTest$1.(AgentInfoRestTest.java:33)
at com.egoo.dao.rest.freelink.AgentInfoRestTest.getAllAgentInfoTest(AgentInfoRestTest.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
... 7 more

Unit testing Spring Boot MultiPartFile PDF Upload Rest Controller using MockMvc

Good day Pals,
I am new to Springboot and having problems testing my PDF uploader spring controller endpoint.
The rest endpoint works when tested via postman, but not in my unit test.
I am using a spring boot application with the following key components and problem is:
org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
at org.springframework.web.method.annotation.RequestParamMethodArgumentResolver.handleMissingValue(RequestParamMethodArgumentResolver.java:251)
2016-06-09 19:03:03,359 4198 [main] DEBUG o.s.t.c.w.ServletTestExecutionListener - Resetting RequestContextHolder for test context [DefaultTestContext#646be2c3 testClass = PDFUploadControllerTest, testInstance = x.y.z.rest.controller.PDFUploadControllerTest#2b44d6d0, testMethod = testHandleFileUpload#PDFUploadControllerTest, testException = java.lang.AssertionError: Status expected:<200> but was:<400>, mergedContextConfiguration = [WebMergedContextConfiguration#797badd3 testClass = PDFUploadControllerTest, locations = '{}', classes = '{class x.y.z.rest.PresentationConfiguration, class x.y.z.rest.controller.MockDomainConfiguration}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]].
java.lang.AssertionError: Status
Expected :200
Actual :400
Please, see some code and stacktrace below ... I have been banging my head on the wall all day, debugging using all online references but NO luck ....
Your help will be most appreciated. Thanks
Application
#EnableAutoConfiguration()
public class Application {
public static void main(String[] args) {
SpringApplication.run(
new Object[]{
DomainConfiguration.class,
PresentationConfiguration.class
},
args);
}
}
DomainConfiguration
#Configuration
#EnableAutoConfiguration()
#ComponentScan(basePackages = {
"x.y.z.rest.domain",
"x.y.z.rest.service",
"x.y.z.rest.util",
"x.y.z.rest.repository"
})
public class DomainConfiguration {
#Autowired(required = true)
public void configeJackson(ObjectMapper jackson2ObjectMapper) {
jackson2ObjectMapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false);
}
}
PresentationConfiguration
#Configuration
#ComponentScan(basePackages = {"x.y.z.rest.controller"})
public class PresentationConfiguration {
}
## RestController
#RestController
public class PDFUploadController {
#Autowired
public void setUploadService(UploadService uploadService) {
this.uploadService = uploadService;
}
private UploadService uploadService;
#RequestMapping(value="/upload", method=RequestMethod.POST)
#ResponseStatus(HttpStatus.CREATED)
public #ResponseBody String handleFileUpload(
#RequestParam("file") MultipartFile file,#RequestParam String a,#RequestParam String b,#RequestParam String c, #RequestParam String d,#RequestParam String e) throws Exception{
java.io.InputStream inputStream =null;
if (!file.isEmpty() && checkContentType(file.getContentType())) {
try {
DBObject dbObject = new BasicDBObject();
//populate DBObject
inputStream = file.getInputStream();
uploadService.uploadFile(inputStream,file.getOriginalFilename(),file.getContentType(),dbObject);
return "success";
} catch (Exception e) {
return "You failed to upload " + file.getOriginalFilename() + " => " + e.getMessage();
}
finally {
if(inputStream!=null) {
inputStream.close();
}
}
} else {
return "You failed to upload " + file.getOriginalFilename() + " as it is an invalid file.";
}
}
}
##### Unit Test Class for Controller
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = {
PresentationConfiguration.class,
MockDomainConfiguration.class})
#WebAppConfiguration
public class PDFUploadControllerTest {
#Autowired
private WebApplicationContext webApplicationContext;
#Autowired
private UploadService uploadService;
private MockMvc mockMvc;
#Autowired
private ObjectMapper mapper;
private RestDocumentationResultHandler document;
#Before
public void setUp() {
mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext)
.build();
}
#After
public void resetMocks() {
reset(uploadService);
}
#Test
public void testHandleFileUpload() throws Exception {
FileInputStream fileInputStream = null;
MockMultipartFile mockMultipartFile = null;
try {
File file = new File("//Users//olatom//Desktop//testFile4Upload.pdf");
// create FileInputStream object
fileInputStream = new FileInputStream(file);
System.out.println("# File input stream for PDF : " + fileInputStream);
byte fileContent[] = new byte[(int) file.length()];
// Reads up to certain bytes of data from this input stream into an array of bytes.
fileInputStream.read(fileContent);
//create string from byte array
String pdfContent = new String(fileContent);
//mockMultipartFile = new MockMultipartFile("upload", file.getName(), "multipart/form-data", fileInputStream);
mockMultipartFile = new MockMultipartFile("file", fileInputStream);
HashMap<String, String> contentTypeParams = new HashMap<String, String>();
contentTypeParams.put("boundary", "265001916915724");
MediaType mediaType = new MediaType("multipart", "form-data", contentTypeParams);
//mockMvc.perform(fileUpload("/upload")
// .file(mockMultipartFile)
// .param("a", "1234").param("b", "PX1234").param("c", "100").param("d", "120")
// .contentType(MediaType.MULTIPART_FORM_DATA))
// .andExpect(status().isOk());
// mockMvc.perform(fileUpload("/upload")).andExpect(status().isOk());
//mockMvc.perform(fileUpload("/upload").file(mockMultipartFile)).andExpect(status().isOk());
mockMvc.perform(
fileUpload("/upload")
.content(mockMultipartFile.getBytes())
.param("a", "1234").param("b", "PX1234").param("c", "100").param("d", "120")
.contentType(mediaType)
)
.andExpect(status().isOk());
} catch (FileNotFoundException e) {
System.out.println("File not found" + e);
} catch (IOException ioe) {
System.out.println("IO Exception while reading file " + ioe);
} catch (Exception exc) {
} finally {
// close the streams using close method
try {
if (fileInputStream != null) {
fileInputStream.close();
}
} catch (IOException ioe) {
System.out.println("Error while closing stream: " + ioe);
}
}
}
}
####### MockDomainConfiguration #####
/**
* Create Mockito mocks for the service classes.
* For this to work the #EnableAutoConfiguration annotation below also has to exclude JPA Autoconfiguration.
*/
#Configuration
#EnableAutoConfiguration()
public class MockDomainConfiguration {
#Bean
public UploadService mockUploadService() {
return mock(UploadService.class);
}
}
## MY ERROR
When I run the junit test in Intellij I get the following error:
2016-06-09 19:03:03,331 4170 [main] DEBUG o.s.t.w.s.TestDispatcherServlet - DispatcherServlet with name '' processing POST request for [/upload]
2016-06-09 19:03:03,342 4181 [main] DEBUG o.s.b.a.e.m.EndpointHandlerMapping - Looking up handler method for path /upload
2016-06-09 19:03:03,343 4182 [main] DEBUG o.s.b.a.e.m.EndpointHandlerMapping - Did not find handler method for [/upload]
2016-06-09 19:03:03,343 4182 [main] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping - Looking up handler method for path /upload
2016-06-09 19:03:03,344 4183 [main] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping - Returning handler method [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
2016-06-09 19:03:03,344 4183 [main] DEBUG o.s.b.f.s.DefaultListableBeanFactory - Returning cached instance of singleton bean 'PDFUploadController'
2016-06-09 19:03:03,352 4191 [main] DEBUG o.s.w.s.m.m.a.ServletInvocableHandlerMethod - Error resolving argument [0] [type=org.springframework.web.multipart.MultipartFile]
HandlerMethod details:
Controller [x.y.z.rest.controller.PDFUploadController]
Method [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
at org.springframework.web.method.annotation.RequestParamMethodArgumentResolver.handleMissingValue(RequestParamMethodArgumentResolver.java:251)
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:96)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:99)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at org.springframework.test.web.servlet.TestDispatcherServlet.service(TestDispatcherServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.springframework.mock.web.MockFilterChain$ServletFilterProxy.doFilter(MockFilterChain.java:167)
at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:155)
at x.y.z.rest.controller.PDFUploadControllerTest.testHandleFileUpload(PDFUploadControllerTest.java:140)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
2016-06-09 19:03:03,353 4192 [main] DEBUG o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver - Resolving exception from handler [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]: org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
2016-06-09 19:03:03,354 4193 [main] DEBUG o.s.w.s.m.a.ResponseStatusExceptionResolver - Resolving exception from handler [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]: org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
2016-06-09 19:03:03,354 4193 [main] DEBUG o.s.w.s.m.s.DefaultHandlerExceptionResolver - Resolving exception from handler [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]: org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
2016-06-09 19:03:03,355 4194 [main] DEBUG o.s.t.w.s.TestDispatcherServlet - Null ModelAndView returned to DispatcherServlet with name '': assuming HandlerAdapter completed request handling
2016-06-09 19:03:03,355 4194 [main] DEBUG o.s.t.w.s.TestDispatcherServlet - Successfully completed request
2016-06-09 19:03:03,359 4198 [main] DEBUG o.s.t.c.s.AbstractDirtiesContextTestExecutionListener - After test method: context [DefaultTestContext#646be2c3 testClass = PDFUploadControllerTest, testInstance = x.y.z.rest.controller.PDFUploadControllerTest#2b44d6d0, testMethod = testHandleFileUpload#PDFUploadControllerTest, testException = java.lang.AssertionError: Status expected:<200> but was:<400>, mergedContextConfiguration = [WebMergedContextConfiguration#797badd3 testClass = PDFUploadControllerTest, locations = '{}', classes = '{class x.y.z.rest.PresentationConfiguration, class x.y.z.rest.controller.MockDomainConfiguration}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]], class annotated with #DirtiesContext [false] with mode [null], method annotated with #DirtiesContext [false] with mode [null].
2016-06-09 19:03:03,359 4198 [main] DEBUG o.s.t.c.w.ServletTestExecutionListener - Resetting RequestContextHolder for test context [DefaultTestContext#646be2c3 testClass = PDFUploadControllerTest, testInstance = x.y.z.rest.controller.PDFUploadControllerTest#2b44d6d0, testMethod = testHandleFileUpload#PDFUploadControllerTest, testException = java.lang.AssertionError: Status expected:<200> but was:<400>, mergedContextConfiguration = [WebMergedContextConfiguration#797badd3 testClass = PDFUploadControllerTest, locations = '{}', classes = '{class x.y.z.rest.PresentationConfiguration, class x.y.z.rest.controller.MockDomainConfiguration}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]].
java.lang.AssertionError: Status
Expected :200
Actual :400
<Click to see difference>
at org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:60)
at org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:89)
at org.springframework.test.web.servlet.result.StatusResultMatchers$10.match(StatusResultMatchers.java:655)
at org.springframework.test.web.servlet.MockMvc$1.andExpect(MockMvc.java:171)
at x.y.z.rest.controller.PDFloadControllerTest.testHandleFileUpload(PDFUploadControllerTest.java:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Your help will be most appreciated.

JBoss-A-MQ encountered an UnknownHostException in JMS messaging application

I have a sample JMS messaging application as shown in below snippet. When I executing the program (specifically when starting the connection) I get an UnknowHostException.
The reason for the exception is clientid property get's null.
Application:
public class MessagingTestApp {
private static final String MY_JMS_CONNECTION_FACTORY_NAME = "myJmsFactory";
private static final String EXCHANGE_NAME = "topicExchange";
private static final String JNDI_PROPERTIES_FILE_NAME = "jndi.properties";
private static final String COULD_NOT_LOAD_JNDI_PROPERTIES_MESSAGE = "Could not load JNDI properties....";
private static final boolean NON_TRANSACTED = false;
private static final Logger LOG = Logger.getLogger(MessagingTestApp.class);
public MessagingTestApp() {}
public static void main(String[] args) {
MessagingTestApp messagingTestApp = new MessagingTestApp();
messagingTestApp.runTest();
}
private void runTest() {
try {
Properties properties = new Properties();
properties.load(loadPropertiesFile());
Context context = new InitialContext(properties);
ConnectionFactory connectionFactory = (ConnectionFactory) context.lookup(MY_JMS_CONNECTION_FACTORY_NAME);
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(NON_TRANSACTED, Session.AUTO_ACKNOWLEDGE);
Destination destination = (Destination) context.lookup(EXCHANGE_NAME);
MessageProducer messageProducer = session.createProducer(destination);
MessageConsumer messageConsumer = session.createConsumer(destination);
TextMessage message = session.createTextMessage("Hello JMS!");
messageProducer.send(message);
message = (TextMessage) messageConsumer.receive();
System.out.println(message.getText());
connection.close();
context.close();
} catch (Exception e) {
LOG.error(e);
e.printStackTrace();
}
}
private InputStream loadPropertiesFile() {
Thread currentThread = Thread.currentThread();
ClassLoader contextClassLoader = currentThread.getContextClassLoader();
InputStream propertiesStream = contextClassLoader.getResourceAsStream(JNDI_PROPERTIES_FILE_NAME);
if (propertiesStream != null) {
return propertiesStream;
} else {
System.out.println(COULD_NOT_LOAD_JNDI_PROPERTIES_MESSAGE);
return null;
}
}
}
JNDI properties file:
java.naming.factory.initial = org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory
java.naming.provider.url = src/main/resources/jndi.properties
connectionfactory.myJmsFactory = amqp://admin:admin#clientid/test?
brokerlist = 'tcp://localhost:5672'
destination.topicExchange = amq.topic
Stack-trace:
javax.jms.JMSException: java.net.UnknownHostException: clientid
at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.connect(ConnectionImpl.java:112)
at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.start(ConnectionImpl.java:266)
at com.adc.efg.MessagingTestApp.runTest(MessagingTestApp.java:39)
at com.adc.efg.MessagingTestApp.main(MessagingTestApp.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.apache.qpid.amqp_1_0.client.ConnectionException: java.net.UnknownHostException: clientid
at org.apache.qpid.amqp_1_0.client.Connection.<init>(Connection.java:271)
at org.apache.qpid.amqp_1_0.client.Connection.<init>(Connection.java:135)
at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.connect(ConnectionImpl.java:105)
... 8 more
Caused by: java.net.UnknownHostException: clientid
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at org.apache.qpid.amqp_1_0.client.Connection.<init>(Connection.java:159)
... 10 more
I'm new to both JBoss-AMQ and JMS. Much appreciated if anyone can point me out where did I go wrong.
Problem was in JNDI properties file. It should be like below,
java.naming.factory.initial = org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory
java.naming.provider.url = src/main/resources/jndi.properties
connectionfactory.myJmsFactory = amqp://admin:admin#localhost/test?
brokerlist = 'tcp://localhost:5672'
destination.topicExchange = amq.topic