My spark job is throwing Task not serializable at runtime. Can anyone tell me if what i am doing wrong here?
#Component("loader")
#Slf4j
public class LoaderSpark implements SparkJob {
private static final int MAX_VERSIONS = 1;
private final AppProperties props;
public LoaderSpark(
final AppProperties props
) {
this.props = props;
}
#Override
public void run(SparkSession spark, final String... args) throws IOException {
HBaseUtil hBaseUtil = new HBaseUtil(props);
byte[][] prefixes = new byte[][]{toBytes("document"),
toBytes("dataSource"),
toBytes("hold:")};
Filter filter = new MultipleColumnPrefixFilter(prefixes);
Scan scan = new Scan();
scan.addFamily(toBytes("data"));
scan.setCaching(100000);
scan.setMaxVersions(MAX_VERSIONS);
scan.setFilter(filter);
JavaRDD<TestMethod> mapFileJavaRDD
= hBaseUtil.createScanRdd(spark, "TestTable", scan).mapPartitions(tuple -> {
return StreamUtils.asStream(tuple)
.map(this::extractResult)
.filter(Objects::nonNull)
.iterator();
});
Dataset<TestMethod> testDataset = spark.createDataset(mapFileJavaRDD.rdd(), bean(TestMethod.class));
testDataset.limit(100);
}
private TestMethod extractResult(Tuple2<ImmutableBytesWritable, Result> resultTuple) {
TestMethod.TestMethodBuilder testBuilder = TestMethod.builder();
Result result;
result = resultTuple._2();
CdfParser cdfParser = new CdfParser();
List<String> holdingId = new ArrayList<>();
testBuilder.dataSource(Bytes.toString(result.getValue(Bytes.toBytes("data"),
Bytes.toBytes("dataSource"))));
testBuilder.document(cdfParser.fromXml(result.getValue(Bytes.toBytes("data"),
Bytes.toBytes("document"))));
NavigableMap<byte[], byte[]> familyMap = result.getFamilyMap(Bytes.toBytes("data"));
for (byte[] bQunitifer : familyMap.keySet()) {
if (Bytes.toString(bQunitifer).contains("hold:")) {
LOG.info(Bytes.toString(bQunitifer));
holdingId.add(Bytes.toString(bQunitifer));
}
}
testBuilder.holding(holdingId);
return testBuilder.build();
}
}
HERE is the stacktrace:
2020-04-29 12:48:59,837 INFO [Thread-4]o.a.s.d.y.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.oclc.googlelinks.spark.SpringSparkJob.main(SpringSparkJob.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:694)
Caused by: org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:403)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:393)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:798)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:797)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:797)
at org.apache.spark.api.java.JavaRDDLike$class.mapPartitions(JavaRDDLike.scala:155)
at org.apache.spark.api.java.AbstractJavaRDDLike.mapPartitions(JavaRDDLike.scala:45)
at org.oclc.googlelinks.spark.job.LoaderSpark.run(LoaderSpark.java:79)
at org.oclc.googlelinks.spark.SpringSparkJob.run(SpringSparkJob.java:79)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784)
... 8 more
try adding getter and setters for props
public void setProps(AppProperties props) {
this.props = props;
}
public AppProperties getProps() {
return props;
}
Just make the function extractResult static . In order to call a static method, you don’t need to serialize the class, you need the declaring class to be reachable by the classloader (and it is the case, as the jar archives can be shared among driver and workers).
Thanks to https://www.nicolaferraro.me/2016/02/22/using-non-serializable-objects-in-apache-spark/
I have installed Hadoop 2.6 in centos7 and it's running fine. But when I run a jar exported from Eclipse, it gives the following error:
[root#myspark ~]# hadoop jar fengcount.jar intput output1
17/05/26 21:24:51 INFO client.RMProxy: Connecting to ResourceManager
at myspark/192.168.44.100:8032 17/05/26 21:24:53 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1495765615548_0004 Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://myspark:54310/user/root/intput
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:302)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:319)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1297)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1294)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1315)
at hdfs.hadoop_hdfs.fengcount.main(fengcount.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
The file input/test1.txt actually exists:
[root#myspark ~]# hdfs dfs -ls -R
drwxr-xr-x - root supergroup 0 2017-05-26 21:02 input
-rw-r--r-- 1 root supergroup 16 2017-05-24 01:57 input/test1.txt
My code:
package hdfs.hadoop_hdfs;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class fengcount {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
// TODO Auto-generated method stub
Configuration conf=new Configuration();
String[] otherargs=new GenericOptionsParser(conf,args).getRemainingArgs();
if (otherargs.length!=2) {
System.err.println("Usage:fengcount<int><out>");
System.exit(2);
}
#SuppressWarnings("deprecation")
Job job=new Job(conf, "fengcount");
job.setJarByClass(fengcount.class);
job.setMapperClass(TokerizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherargs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherargs[1]));
System.exit(job.waitForCompletion(true)?0:1);
}
// mapper class
public static class TokerizerMapper extends Mapper<Object, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
#Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
// TODO Auto-generated method stub
System.out.println("key=" + key.toString());
System.out.println("value=" + value.toString());
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
//reduce process
public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();
#Override
public void reduce(Text key, Iterable<IntWritable> values,
Reducer<Text, IntWritable, Text, IntWritable>.Context context)
throws IOException, InterruptedException {
// TODO Auto-generated method stub
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
//mapreduce process
}
From the error log, I could see hdfs://myspark:54310/user/root/intput, I suspect its incorrect. I guess the path ends with input.
Good luck!
I am learning to use the JMockit to test my Restful API.The missing invocation error is abusing me.
The Restful API:
#RestController
public class AgentInfoRest {
#Autowired
AgentInfoRepository agentInfoRepository;
private Logger logger = LoggerFactory.getLogger(AgentInfoRest.class);
#RequestMapping("/allagentInfo")
RestResponse getAllAgentInfo()
{
RestResponse restResponse = new RestResponse();
logger.info("getAllAgentInfo has no request parameter");
Collection<AgentInfo> result = agentInfoRepository.getAllAgents();
if(result.isEmpty())
{
restResponse.setRetCode(-1);
restResponse.setRetMsg("Request success,no such record in db");
}
restResponse.setResult(result);
return restResponse;
}
}
The Test Class:
public class AgentInfoRestTest {
String host = "http://127.0.0.1:8080";
#Test
public void getAllAgentInfoTest(#Mocked final AgentInfoRepository agentInfoRepository) {
RestTemplate restTemplate = new RestTemplate();
RestResponse restResponse = new RestResponse();
AgentInfoRest agentInfoRest = new AgentInfoRest();
new StrictExpectations() {
{
agentInfoRepository.getAllAgents(); result = null; times = 1;
}
};
String url = "/allagentInfo";
ResponseEntity<RestResponse> result = restTemplate.getForEntity(host + url, RestResponse.class);
new FullVerifications() {
{
}
};
}
}
The Error:
mockit.internal.MissingInvocation: Missing invocation of:
com.egoo.dao.repository.freelink.AgentInfoRepository#getAllAgents()
on mock instance: com.egoo.dao.repository.freelink.$Impl_AgentInfoRepository#50a7bc6e
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: Missing invocation
at com.egoo.dao.rest.freelink.AgentInfoRestTest$1.(AgentInfoRestTest.java:33)
at com.egoo.dao.rest.freelink.AgentInfoRestTest.getAllAgentInfoTest(AgentInfoRestTest.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
... 7 more
Good day Pals,
I am new to Springboot and having problems testing my PDF uploader spring controller endpoint.
The rest endpoint works when tested via postman, but not in my unit test.
I am using a spring boot application with the following key components and problem is:
org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
at org.springframework.web.method.annotation.RequestParamMethodArgumentResolver.handleMissingValue(RequestParamMethodArgumentResolver.java:251)
2016-06-09 19:03:03,359 4198 [main] DEBUG o.s.t.c.w.ServletTestExecutionListener - Resetting RequestContextHolder for test context [DefaultTestContext#646be2c3 testClass = PDFUploadControllerTest, testInstance = x.y.z.rest.controller.PDFUploadControllerTest#2b44d6d0, testMethod = testHandleFileUpload#PDFUploadControllerTest, testException = java.lang.AssertionError: Status expected:<200> but was:<400>, mergedContextConfiguration = [WebMergedContextConfiguration#797badd3 testClass = PDFUploadControllerTest, locations = '{}', classes = '{class x.y.z.rest.PresentationConfiguration, class x.y.z.rest.controller.MockDomainConfiguration}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]].
java.lang.AssertionError: Status
Expected :200
Actual :400
Please, see some code and stacktrace below ... I have been banging my head on the wall all day, debugging using all online references but NO luck ....
Your help will be most appreciated. Thanks
Application
#EnableAutoConfiguration()
public class Application {
public static void main(String[] args) {
SpringApplication.run(
new Object[]{
DomainConfiguration.class,
PresentationConfiguration.class
},
args);
}
}
DomainConfiguration
#Configuration
#EnableAutoConfiguration()
#ComponentScan(basePackages = {
"x.y.z.rest.domain",
"x.y.z.rest.service",
"x.y.z.rest.util",
"x.y.z.rest.repository"
})
public class DomainConfiguration {
#Autowired(required = true)
public void configeJackson(ObjectMapper jackson2ObjectMapper) {
jackson2ObjectMapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false);
}
}
PresentationConfiguration
#Configuration
#ComponentScan(basePackages = {"x.y.z.rest.controller"})
public class PresentationConfiguration {
}
## RestController
#RestController
public class PDFUploadController {
#Autowired
public void setUploadService(UploadService uploadService) {
this.uploadService = uploadService;
}
private UploadService uploadService;
#RequestMapping(value="/upload", method=RequestMethod.POST)
#ResponseStatus(HttpStatus.CREATED)
public #ResponseBody String handleFileUpload(
#RequestParam("file") MultipartFile file,#RequestParam String a,#RequestParam String b,#RequestParam String c, #RequestParam String d,#RequestParam String e) throws Exception{
java.io.InputStream inputStream =null;
if (!file.isEmpty() && checkContentType(file.getContentType())) {
try {
DBObject dbObject = new BasicDBObject();
//populate DBObject
inputStream = file.getInputStream();
uploadService.uploadFile(inputStream,file.getOriginalFilename(),file.getContentType(),dbObject);
return "success";
} catch (Exception e) {
return "You failed to upload " + file.getOriginalFilename() + " => " + e.getMessage();
}
finally {
if(inputStream!=null) {
inputStream.close();
}
}
} else {
return "You failed to upload " + file.getOriginalFilename() + " as it is an invalid file.";
}
}
}
##### Unit Test Class for Controller
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = {
PresentationConfiguration.class,
MockDomainConfiguration.class})
#WebAppConfiguration
public class PDFUploadControllerTest {
#Autowired
private WebApplicationContext webApplicationContext;
#Autowired
private UploadService uploadService;
private MockMvc mockMvc;
#Autowired
private ObjectMapper mapper;
private RestDocumentationResultHandler document;
#Before
public void setUp() {
mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext)
.build();
}
#After
public void resetMocks() {
reset(uploadService);
}
#Test
public void testHandleFileUpload() throws Exception {
FileInputStream fileInputStream = null;
MockMultipartFile mockMultipartFile = null;
try {
File file = new File("//Users//olatom//Desktop//testFile4Upload.pdf");
// create FileInputStream object
fileInputStream = new FileInputStream(file);
System.out.println("# File input stream for PDF : " + fileInputStream);
byte fileContent[] = new byte[(int) file.length()];
// Reads up to certain bytes of data from this input stream into an array of bytes.
fileInputStream.read(fileContent);
//create string from byte array
String pdfContent = new String(fileContent);
//mockMultipartFile = new MockMultipartFile("upload", file.getName(), "multipart/form-data", fileInputStream);
mockMultipartFile = new MockMultipartFile("file", fileInputStream);
HashMap<String, String> contentTypeParams = new HashMap<String, String>();
contentTypeParams.put("boundary", "265001916915724");
MediaType mediaType = new MediaType("multipart", "form-data", contentTypeParams);
//mockMvc.perform(fileUpload("/upload")
// .file(mockMultipartFile)
// .param("a", "1234").param("b", "PX1234").param("c", "100").param("d", "120")
// .contentType(MediaType.MULTIPART_FORM_DATA))
// .andExpect(status().isOk());
// mockMvc.perform(fileUpload("/upload")).andExpect(status().isOk());
//mockMvc.perform(fileUpload("/upload").file(mockMultipartFile)).andExpect(status().isOk());
mockMvc.perform(
fileUpload("/upload")
.content(mockMultipartFile.getBytes())
.param("a", "1234").param("b", "PX1234").param("c", "100").param("d", "120")
.contentType(mediaType)
)
.andExpect(status().isOk());
} catch (FileNotFoundException e) {
System.out.println("File not found" + e);
} catch (IOException ioe) {
System.out.println("IO Exception while reading file " + ioe);
} catch (Exception exc) {
} finally {
// close the streams using close method
try {
if (fileInputStream != null) {
fileInputStream.close();
}
} catch (IOException ioe) {
System.out.println("Error while closing stream: " + ioe);
}
}
}
}
####### MockDomainConfiguration #####
/**
* Create Mockito mocks for the service classes.
* For this to work the #EnableAutoConfiguration annotation below also has to exclude JPA Autoconfiguration.
*/
#Configuration
#EnableAutoConfiguration()
public class MockDomainConfiguration {
#Bean
public UploadService mockUploadService() {
return mock(UploadService.class);
}
}
## MY ERROR
When I run the junit test in Intellij I get the following error:
2016-06-09 19:03:03,331 4170 [main] DEBUG o.s.t.w.s.TestDispatcherServlet - DispatcherServlet with name '' processing POST request for [/upload]
2016-06-09 19:03:03,342 4181 [main] DEBUG o.s.b.a.e.m.EndpointHandlerMapping - Looking up handler method for path /upload
2016-06-09 19:03:03,343 4182 [main] DEBUG o.s.b.a.e.m.EndpointHandlerMapping - Did not find handler method for [/upload]
2016-06-09 19:03:03,343 4182 [main] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping - Looking up handler method for path /upload
2016-06-09 19:03:03,344 4183 [main] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping - Returning handler method [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
2016-06-09 19:03:03,344 4183 [main] DEBUG o.s.b.f.s.DefaultListableBeanFactory - Returning cached instance of singleton bean 'PDFUploadController'
2016-06-09 19:03:03,352 4191 [main] DEBUG o.s.w.s.m.m.a.ServletInvocableHandlerMethod - Error resolving argument [0] [type=org.springframework.web.multipart.MultipartFile]
HandlerMethod details:
Controller [x.y.z.rest.controller.PDFUploadController]
Method [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
at org.springframework.web.method.annotation.RequestParamMethodArgumentResolver.handleMissingValue(RequestParamMethodArgumentResolver.java:251)
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:96)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:99)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at org.springframework.test.web.servlet.TestDispatcherServlet.service(TestDispatcherServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.springframework.mock.web.MockFilterChain$ServletFilterProxy.doFilter(MockFilterChain.java:167)
at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:155)
at x.y.z.rest.controller.PDFUploadControllerTest.testHandleFileUpload(PDFUploadControllerTest.java:140)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
2016-06-09 19:03:03,353 4192 [main] DEBUG o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver - Resolving exception from handler [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]: org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
2016-06-09 19:03:03,354 4193 [main] DEBUG o.s.w.s.m.a.ResponseStatusExceptionResolver - Resolving exception from handler [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]: org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
2016-06-09 19:03:03,354 4193 [main] DEBUG o.s.w.s.m.s.DefaultHandlerExceptionResolver - Resolving exception from handler [public java.lang.String x.y.z.rest.controller.PDFUploadController.handleFileUpload(org.springframework.web.multipart.MultipartFile,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]: org.springframework.web.bind.MissingServletRequestParameterException: Required MultipartFile parameter 'file' is not present
2016-06-09 19:03:03,355 4194 [main] DEBUG o.s.t.w.s.TestDispatcherServlet - Null ModelAndView returned to DispatcherServlet with name '': assuming HandlerAdapter completed request handling
2016-06-09 19:03:03,355 4194 [main] DEBUG o.s.t.w.s.TestDispatcherServlet - Successfully completed request
2016-06-09 19:03:03,359 4198 [main] DEBUG o.s.t.c.s.AbstractDirtiesContextTestExecutionListener - After test method: context [DefaultTestContext#646be2c3 testClass = PDFUploadControllerTest, testInstance = x.y.z.rest.controller.PDFUploadControllerTest#2b44d6d0, testMethod = testHandleFileUpload#PDFUploadControllerTest, testException = java.lang.AssertionError: Status expected:<200> but was:<400>, mergedContextConfiguration = [WebMergedContextConfiguration#797badd3 testClass = PDFUploadControllerTest, locations = '{}', classes = '{class x.y.z.rest.PresentationConfiguration, class x.y.z.rest.controller.MockDomainConfiguration}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]], class annotated with #DirtiesContext [false] with mode [null], method annotated with #DirtiesContext [false] with mode [null].
2016-06-09 19:03:03,359 4198 [main] DEBUG o.s.t.c.w.ServletTestExecutionListener - Resetting RequestContextHolder for test context [DefaultTestContext#646be2c3 testClass = PDFUploadControllerTest, testInstance = x.y.z.rest.controller.PDFUploadControllerTest#2b44d6d0, testMethod = testHandleFileUpload#PDFUploadControllerTest, testException = java.lang.AssertionError: Status expected:<200> but was:<400>, mergedContextConfiguration = [WebMergedContextConfiguration#797badd3 testClass = PDFUploadControllerTest, locations = '{}', classes = '{class x.y.z.rest.PresentationConfiguration, class x.y.z.rest.controller.MockDomainConfiguration}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]].
java.lang.AssertionError: Status
Expected :200
Actual :400
<Click to see difference>
at org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:60)
at org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:89)
at org.springframework.test.web.servlet.result.StatusResultMatchers$10.match(StatusResultMatchers.java:655)
at org.springframework.test.web.servlet.MockMvc$1.andExpect(MockMvc.java:171)
at x.y.z.rest.controller.PDFloadControllerTest.testHandleFileUpload(PDFUploadControllerTest.java:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Your help will be most appreciated.
I have a sample JMS messaging application as shown in below snippet. When I executing the program (specifically when starting the connection) I get an UnknowHostException.
The reason for the exception is clientid property get's null.
Application:
public class MessagingTestApp {
private static final String MY_JMS_CONNECTION_FACTORY_NAME = "myJmsFactory";
private static final String EXCHANGE_NAME = "topicExchange";
private static final String JNDI_PROPERTIES_FILE_NAME = "jndi.properties";
private static final String COULD_NOT_LOAD_JNDI_PROPERTIES_MESSAGE = "Could not load JNDI properties....";
private static final boolean NON_TRANSACTED = false;
private static final Logger LOG = Logger.getLogger(MessagingTestApp.class);
public MessagingTestApp() {}
public static void main(String[] args) {
MessagingTestApp messagingTestApp = new MessagingTestApp();
messagingTestApp.runTest();
}
private void runTest() {
try {
Properties properties = new Properties();
properties.load(loadPropertiesFile());
Context context = new InitialContext(properties);
ConnectionFactory connectionFactory = (ConnectionFactory) context.lookup(MY_JMS_CONNECTION_FACTORY_NAME);
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(NON_TRANSACTED, Session.AUTO_ACKNOWLEDGE);
Destination destination = (Destination) context.lookup(EXCHANGE_NAME);
MessageProducer messageProducer = session.createProducer(destination);
MessageConsumer messageConsumer = session.createConsumer(destination);
TextMessage message = session.createTextMessage("Hello JMS!");
messageProducer.send(message);
message = (TextMessage) messageConsumer.receive();
System.out.println(message.getText());
connection.close();
context.close();
} catch (Exception e) {
LOG.error(e);
e.printStackTrace();
}
}
private InputStream loadPropertiesFile() {
Thread currentThread = Thread.currentThread();
ClassLoader contextClassLoader = currentThread.getContextClassLoader();
InputStream propertiesStream = contextClassLoader.getResourceAsStream(JNDI_PROPERTIES_FILE_NAME);
if (propertiesStream != null) {
return propertiesStream;
} else {
System.out.println(COULD_NOT_LOAD_JNDI_PROPERTIES_MESSAGE);
return null;
}
}
}
JNDI properties file:
java.naming.factory.initial = org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory
java.naming.provider.url = src/main/resources/jndi.properties
connectionfactory.myJmsFactory = amqp://admin:admin#clientid/test?
brokerlist = 'tcp://localhost:5672'
destination.topicExchange = amq.topic
Stack-trace:
javax.jms.JMSException: java.net.UnknownHostException: clientid
at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.connect(ConnectionImpl.java:112)
at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.start(ConnectionImpl.java:266)
at com.adc.efg.MessagingTestApp.runTest(MessagingTestApp.java:39)
at com.adc.efg.MessagingTestApp.main(MessagingTestApp.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.apache.qpid.amqp_1_0.client.ConnectionException: java.net.UnknownHostException: clientid
at org.apache.qpid.amqp_1_0.client.Connection.<init>(Connection.java:271)
at org.apache.qpid.amqp_1_0.client.Connection.<init>(Connection.java:135)
at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.connect(ConnectionImpl.java:105)
... 8 more
Caused by: java.net.UnknownHostException: clientid
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at org.apache.qpid.amqp_1_0.client.Connection.<init>(Connection.java:159)
... 10 more
I'm new to both JBoss-AMQ and JMS. Much appreciated if anyone can point me out where did I go wrong.
Problem was in JNDI properties file. It should be like below,
java.naming.factory.initial = org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory
java.naming.provider.url = src/main/resources/jndi.properties
connectionfactory.myJmsFactory = amqp://admin:admin#localhost/test?
brokerlist = 'tcp://localhost:5672'
destination.topicExchange = amq.topic