I wrote something like this to read file from resource directory:
val filePath = MyClass.getClass.getResource("/myFile.csv")
val file = filePath.getFile
println(file)
CSVReader.open(file)
and the result I got was something like this:
file:/path/to/project/my_module/src/main/resources/my_module-assembly-0.1.jar!/myFile.csv
Exception in thread "main" java.io.FileNotFoundException: file:/path/to/project/my_module/src/main/resources/my_module-assembly-0.1.jar!/myFile.csv (No such file or directory)
Whereas, if I run the same code in IDE(Intellij), no issues and the path printed to console is:
/path/to/project/my_module/target/scala-2.11/classes/myFile.csv
FYI, its a multi build project with a couple of modules and I build the jars using sbt assembly
This is more related to Java or the JVM itself than to Scala or SBT.
There is a difference when running your application from the IDE vs the command line (or outside the IDE). The method getClass.getResource(...) attempts to find the resource URL in the current classpath, and that's the key difference.
If you look at the URL itself, you will find that in the first case you have a my_module-assembly-0.1.jar! bit in it, meaning that URL is actually pointing towards the contents of the JAR, not to a file accessible from the file system.
From inside your IDE your class path will include the actual files on disk, from the source folders, because the IDE assumes that there is not any JAR file itself. So when obtaining the URL given by getClass.getResource(...) you have an URL that does not have the my_module-assembly-0.1.jar! bit in it.
Since you want to read the contents of the file, you may want to do a getClass.getResourceAsStream(...). That will give you an InputStream that you can use to read the contents regardless you are in the IDE or anywhere else.
Your CSVReader class may have a method that allows it read the data from an InputStream or a Reader or something similar.
EDIT: As pointed out by Luis Miguel Mejia Suarez, a more Scala idiomatic way of reading files from your class path is using the Source.fromResource method. This will return a BufferedSource that then can be used to read the file contents.
Related
So if one was to want to use Google protocol buffers in Matlab and using a Windows computer what would be the best way to do that since Matlab is not in the list of supported languages?
I haven't seen an answer on this and I thought the solution was a bit obscure so I am going to post a how to for matlab_out using the protoc.exe
A how to for google protocol buffer matlab out, this is using resources from the internet I will also include a zip file containing all this already done.
Unzip protobuf-‘version#’.zip (looks like: protobuf-#.#.#)
Open file protobuf-#.#.# -> src
Choose your favorite editor (Notepad++ is good) and change Makefile.am (do not include quotes)
a. Under “nobase_include_HEADERS =”
and below “$(GZHEADERS)”
add the line “farsounder/protobuf/compiler/matlab/matlab_generator.h \” (Note the back slash)
b. Under “libprotoc_la_LIBADD = $(PTHREAD_LIBS) libprotobuf.la”
add lines
“farsounder/protobuf/compiler/matlab/matlab_generator.cc \”
“farsounder/protobuf/compiler/matlab/matlab_plugin.cc \”
c. Save the file and exit out
While still in the src directory go into ->google ->protobuf->compiler and change main.cc
a. Under “#include <google/protobuf/compiler/java/java_generator.h>”
add the line “#include <farsounder/protobuf/compiler/matlab/matlab_generator.h>”
b. In main function add the lines
“// Proto2 Matlab
farsounder::protobuf::compiler::matlab::MatlabGenerator matlab_generator;
cli.RegisterGenerator("--matlab_out", &matlab_generator,
"Generate Matlab M files.");”
c. Save the file and exit out
Unzip protobuf-matlab
Open file protobuf-matlab ->src and copy the farsounder directory to protobuf-#.#.# -> src directory
Go back to protobuf-#.#.# directory and into -> vsprojects and open protobuf.sln in Visual Studio, I believe any version should work
There should be a popup that wants to convert the solution file to a more updated version, go ahead and do that
If you do not see solutions explorer open it up using ctrl+alt+L
Open a new windows explorer and go into protobuf-#.#.# ->src->farsounder->protobuf->compiler->matlab, now in visual studio using the solutions explorer expand the libprotoc->Header Files, now in windows explorer copy the matlab_generator.h file and paste it into the Header Files directory
Still in Solutions explorer go into the Source Files directory and copy in the matlab_generator.cc
Contract libprotoc and right click on protoc and click on properties
Under Configuration Properties->Linker->General, Edit Enable Incremental Linking to be No
Under Configuration Properties->Linker->Input, Edit Additional Dependencies
a. Add the lines “Release\libprotobuf.lib” and underneath “Release\libprotoc.lib”
b. Click OK on the Additional Dependencies window and OK on the protoc Property Page
Change the build type to Release
Right-click on libprotobuf and select build, once completed right-click on libprotoc and select build
Once completed right-click on protoc and select build, it should provide you with an protoc.exe under protobuf-#.#.# ->vsprojects->Release, this will now allow you to create matlab .m files by using the matlab_out command
Help to find protobuf-#.#.# and the protobuf-matlab:
Follow these links:
https://code.google.com/p/protobuf-matlab/source/browse/
download the zip file
https://github.com/google/protobuf/releases
download the source code
Extra help, for using protoc.exe
Copy and paste the created protoc.exe to where your .proto file is
Run cmd and change the directory to where your protoc.exe and .proto file is
Run this command (lets pretend the name of my .proto file is “afunprotofile”) “protoc –matlab_out=./ -I./ afunprotofile.proto”
The easiest way is to use Java
The FarSounder code was nice, but it is quite old and unmaintained. The easiest way to produce Matlab compatible code is to just use the Java version of Protobuf. This solution should work on any platform that supports Matlab and Java.
The Steps
Create a .proto file
Process the file with the protoc compiler and output Java source
Using IntelliJ or other tool produce a JAR file that includes the dependencies
Add the JAR file to the Matlab classpath. edit('classpath.txt')
Restart Matlab
The protobuf runtime dependencies
I include these in a single JAR file output of the Protobuf and the two runtime libraries.
protobuf-java-3.3.0.jar
protobuf-java-util-3.3.0.jar
A sample
I wrote a simple Java wrapper class to hide the MyProtobuf.Builder return type from Matlab that I added to the JAR file
public class MyProtobufWrapperWrapper {
public static MyProtobuf.Builder newBuilder()
{
return MyProtobuf.newBuilder();
}
}
In Matlab
p = com.cameronpalmer.MyProtobufWrapper.newBuilder();
p.setIdentifier(java.util.UUID.randomUUID().toString());
p.setTimestampMilliseconds(timestamp);
p.setAngleRadians(0);
p.addAllChannelSamples(channel_vector);
planeWaveBuilt = p.build();
byteArray = planeWaveBuilt.toByteArray();
As Cameron Lowell Palmer's answer suggests, the way to go is via Java.
Is I lost a couple of hours today on this problem, I would like to elaborate some more. I started with Cameron's answer but I had to do a couple of more steps. Essentially, I had to do all of the following:
Compile proto message for java, e.g. protoc --java_out=./ your_file.proto
Remember to install protobuf java runtime, e.g. in ubuntu: sudo apt install libprotobuf-java
Locate the protobuf java runtime on your system, e.g.: /usr/share/java/protobuf-3.6.1.jar; its path will be used later on. The name should always follow the pattern protobuf-version.jar or protobuf-java-version.jar, therefore locate protobuf- | grep jar$ should reduce the search space for you.
Add a short java file to the same directory, where your protobuf java files went (./x/y/z/MyProto.java). Use your package path instead of x/y/z. If you did not declare java package explicitly in the proto file, then protoc just used your filename as package name. Either way, you can probably check where the protoc generated files went by yourself. Contents of MyProto.java are listed below. Just replace YourProtoFileName and YourMessageName with your stuff. Note that this step is not optional, as this will not generate a simple helper class. For the life of me, Matlab would not let me use inner classes directly (in java, YourMessageName is the inner class of YourProtoFileName). But with the above helper, it was quite happy to generate them for me. Note that if you have more than one message defined in your proto file, you might need to expose more than one builder in this way. And if you only need to read protobuffers, then you might need to export just YourMessageName and not Builder.
package x.y.z;
import x.y.z.YourProtoFileName; // if you do not know it, do `ls x.y.z/*.java`
public class MyProto {
public static YourProtoFileName.YourMessageName.Builder newBuilder() {
return YourProtoFileName.YourMessageName.newBuilder();
}
}
Check Matlab's version of java; in Matlab write version -java. In my case it was 1.8, while the default java installed on my system (java -version) is 11. I had to manually select java 1.8 for the next step, otherwise the whole thing did not work. Even worse, Matlab only produced a very nondescript error "No class x.y.z.YourProtoFileName." Thanks Matlab! You might need to install proper version first (sudo apt install openjdk-8-jdk) and then use update-java-alternatives or just locate javac for the appropriate java version on your system.
Compile both your java file and the one generated by protoc. Remember to point classpath at the jar file or protobuffer runtime that you found above. And remember to use your javac path. In a single command: /usr/lib/jvm/java-8-openjdk-amd64/bin/javac x.y.z/*.java -cp /usr/share/java/protobuf-3.6.1.jar. This will generate class files in ./x/y/z/.
Optional: pack the class files in a jar for easier distribution: jar cvf ./YourProtoFilename.jar x/y/z/*.class. Note that this command line works for me, since I've put all classes, including MyProto, in the same package. You might need to adapt it to your needs.
Start Matlab.
% make Matlab aware of your new classes
javaaddpath('./')
% tell Matlab where protobuf dependancy lives (use the path from step 3)
javaaddpath('/usr/share/java/protobuf-3.6.1.jar')
% test if the classes were found
methods('x.y.z.YourProtoFilename.YourMessageName')
% if methods are listed then you are good to go
% use the helper form step 4
b = x.y.z.MyProto.newBuilder();
% now you have a builder you can use to build your protobuf message
See Google's protobuf java primer to go on from here, as using java in Matlab is fairly straight forward: just write java statements. As long as they are simple statements :)
The same procedure works on octave too. With a bit different syntax for java inside octave. And octave was less picky about the java version in my case. YMMV
#WPFUser,
We followed below steps to build protoc for matlab.
1) git clone https://github.com/protocolbuffers/protobuf.git to protobuf-main folder
2) git submodule update --init --recursive
3) git clone https://github.com/farsounder/protobuf-matlab.git - protobuf_matlab folder
4) Copied src folder from protobuf_matlab repo to protobuf-main/src
5) Updated cmake/libprotoc.cmake to include matlab_generator.cc and matlab_generator.h files
6) Updated cmake/extract_includes.bat.in to include matlab_generator.h file
7) \src\google\protobuf\compiler\main.cc to include matlab_generator
I am trying to run a my MapReduce job by building a jar from eclipse , but while trying to execute the job , I am getting "Not a valid Jar" error.
I have tried to follow the link Not a valid Jar but that didnt help.
Can anyone please give me the instructions on how to build the jar from eclipse, for it to run on Hadoop.
I am aware of the process of building the Jar file from eclipse,however I am not sure, do I have to take any special care for building a jar file, so that it runs on Hadoop.
When you submit the command, make certain you have the following things on the line to do the command:
When you indicate the jar, make certain you are directing to the jar properly. It may be easiest to be certain by using the absolute path. To get the absolute path, if you navigate to the place where the jar is, then run 'readlink -f ' command to get the absolute path. So for you, not just hist.jar, but maybe /home/akash_user/jars/hist.jar or wherever it is on your system. If you are using Eclipse, it may be saving it somewhere funny, so make sure that is not the problem. The jar cannot be run from HDFS storage. must run from local storage.
When you name your main class, in your example Histogram, you must use the fully qualified name of the class, with the package, the project, and the class. So, usually, if the program/project is named Histogram, and there is a HistogramDriver, HistogramMapper, HistogramReducer, and your main() is in HistogramDriver, you need to type Histogram.HistogramDriver to get the program running. (Unless you made your jar runnable, which requires extra stuff at the beginning, making .mdf and things.)
Make sure that the jar you are submitting (hist.jar) is in the current directory from where you are submitting the 'hadoop jar' command.
If the issue is still persisting, please tell the Java, Hadoop and Linux version you are using.
You should not keep the jar file in HDFS when executing the MapReduce job. Make sure Jar is available in the local path. Input path and output directory should be the path from HDFS.
I am trying to access a file in Scala using io.Source.fromfile.
I have specified the full path, but i am still getting a no such directory or file error.
This is a general version of what my code looks like:
val lines = io.Source.fromFile("~/top/next/source/resources/desiredFile.txt").getLines()
I'm running Ubuntu if that makes any difference.
It probably because you are using tilde sign, use full absolute path.
If you want to avoid hard coding your home directory, you can get it from environment variables:
val home = System.getProperty("user.home")
val s = Source.fromFile(s"${home}/.....").getLines()
The compiler was assuming it should start in the project folder I was already in, so when I specified the directories above that folder, it tried to find them all as a sub-directory of my root project folder.
This would obviously cause an error.
I now have:
val lines = io.Source.fromFile("source/resources/desiredFile.txt")
which is working properly
i am to copy a file in scala But getting FileNotFound error, The assets folder is in the same directory where is src:
val src = new File("/assets/public/images/default/male.jpg")
val dest = new File("/assets/public/images/profile/male1.jpg")
new FileOutputStream(dest) getChannel() transferFrom(
new FileInputStream(src) getChannel, 0, Long.MaxValue )
In your code you are trying to copy the file using FileOutputStream, which requires a valid path to the existing file else it'll throw FileNotFoundException. (see the doc here)
val src = new File("/assets/public/images/default/male.jpg")
val dest = new File("/assets/public/images/profile/male1.jpg")
new FileOutputStream(dest) //dest should exist
Nevertheless, Play has its own utility to copy files. Here is the link.
import play.api.libs.Files
Files.copyFile(src, dest, true, true)
println(dest.getAbsolutePath()) // filepath of copied file
Also, since the files get copied to the working directory, you might not be able to see the new file in the folder structure of your favorite IDE.
Aside, you may get the path for public assets by using routes
val srcPath = routes.Assets.at("public/images/default/male.jpg").url
A general advise
When copying files on Java, use the FileUtils.copy(...) from the Apache commons project.
For your specific problem
You get "File not found" if the file cannot be found by the running process. This could be because your file is indeed not there or because the process is lacking rights to see the file.
Your wording is a little ambiguous, it seems you meant to give paths relative to the current working directory. That means that this code should be executed from a directory which contains the assets directory. If this is so, then you have made a mistake and given absolute paths to your file objects, not relative ones. All you have to do is remove the initial forward slash from those paths and it should work.
As it is, you are telling Scala/Java to look in the root directory for assets.
I read that the servlets map the current location based on the url. Clicking a button from my Home.jsp page directs me to my servlet, ExcelUploader. The URL when said button is clicked is
http://localhost:8080/ServletExample/ExcelUploader
I'm trying to open an excel file located in the same folder as my JSP. so that means I have to move one folder up relative to the url above. I have this in my servlet:
InputStream inp = new FileInputStream("../OpenMe.xls");
However I'm still getting a
java.io.FileNotFoundException: ..\OpenMe.xls (The system cannot find the file specified)
This is how my project is setup:
The FileInputStream operates on the local disk file system relative to the working directory and knows absolutely nothing about the fact that it's invoked from a Java EE web application. Any relative path you pass to it is relative to the folder which was been opened at the moment the command to start the server is executed. This is often the server's own installation folder, but in case of an IDE this can also be project's own root folder. This variable is not controllable from inside your Java code. You should not be relying on that.
You've stored the file as a resource of the public webcontent. So it's available as a webcontent resource by ServletContext#getResourceAsStream() which returns an InputStream. If you have absolutely a legitimate reason to invoke the servlet by its URL instead of just using the file's own URL http://localhost:8080/ServletExample/OpenMe.xls, then you should be getting at as follows:
InputStream input = getServletContext().getResourceAsStream("/OpenMe.xls");
// ...
If your intent is indeed to restrict the file's access to by the servlet only, you might want to consider to move the file into the /WEB-INF folder, so that the enduser can never open it directly by entering the file's own URL. You only need to change the resource path accordingly.
InputStream input = getServletContext().getResourceAsStream("/WEB-INF/OpenMe.xls");
// ...
You should not be using getRealPath() as suggested by the other answer. This won't work when the servletcontainer is configured to expand the WAR file into memory instead of into local disk file system, which is often the case in 3rd party hosts. It would return null then.
See also:
getResourceAsStream() vs FileInputStream
Paths for files that live in the webtree have to be "translated" using getRealPath before they are usable, like this:
File excelFile = new File(getServletContext().getRealPath("/OpenMe.xls"));
While you're at it, using the default package isn't a good idea, create a package for your files.