In spark installing README,I write command './bin/spark-shell',
I got some hint follwing:
Found multiple Spark assembly jars in /opt/spark-0.9.0-incubating-bin-hadoop1/assembly/target/scala-2.10:
spark-assembly-0.9.0-incubating-hadoop1.0.4.jar
spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar
Please remove all but one jar.
I ever try to remove one of them ,but I fail to success.
If somebody can deal with it ,please tell me.
thanks.
all you need to do is just cd to the assembly/target/scala-2.10 dir, and use the sbt to compile again.
this is caused by the incompatible between different compile method, you may used the maven and sbt both?
The problem is that the JARs are built into the assembly JARs when you run
sbt/sbt clean assembly
What I did is:
sbt/sbt clean package
This will only give you the small JARs. The next steps is to update the CLASSPATH in the
bin/compute-classpath.sh script manually, appending all the JARs.
With :
sbt/sbt assembly
We can't introduce our own Hadoop patch since it will always pull from Maven repo, unless we
hijack the repository path, or do a 'mvn install' locally. This is more of a hack I think.
Related
is is possible to execute such command in sbt to download all jar dependencies (needed for assembly task without build actual target jar? I would like to fullfill local cache (I mean ~/.sbt, ~/.m2 and so on directories). Is there exist such command or way to do it?
The sbt update task does this. This is listed in the sbt Command Line Reference, under Project-level Tasks.
I have a Intellij Scala project, that depends on another lib, which in turn depends on some other libs (gson, log4j...), but somehow Intellij won't add those transitive dependencies to my project, so when I run my main method, jvm always complains no class found. I also tried sbt update and sbt update-classifier, they won't resolve down the dependency path as well. How can I have sbt resolve all levels of dependencies?
If all works in SBT command lines but not in Idea then remove the .idea folder and reload the SBT project in Idea.
I should warn that you will loose the project configs (such as run configs) by doing that.
I finally figure out what was wrong, it was the sbt-s3-resolver plugin I was using couldn't resolve maven style dependencies, I used an another plugin which supports maven libs and it worked. But I still couldn't figure out why it worked before.
I am trying to use scala to access Amazon's DynamoDB and found this great package on github https://github.com/piotrga/async-dynamo
so I downloaded the code as a zip file , unzipped it and then did "sbt clean test" and getting the following error
error sbt.ResolveException: unresolved dependency: asyncdynamo#async-dynamo;1.6.0: not found
Questions : is this the correct way to generate a jar file that I can include in my Scala program or is there a better way?
thanks in advance.
EDIT:
just for the benefit of others, the SCALA SBT documentation provides lots of information regarding the build process.
Instead of generating a jar file, you can just run 'sbt publish-local' and then include the lines for the managed dependency in the other project.
Sbt/ivy will see you have the artifact that way you don't need to add the jar to the other project which is much cleaner.
Then for example if you need to update the other project you don't need to replace the jar again - just publish-local again and clean and run your other project!
You are not the only one to have problems with this it seems, see github issues page:
https://github.com/piotrga/async-dynamo/issues
The command 'sbt clean test' will run the tests sbt detects. If you want a .jar file you could use 'sbt clean package', which produces a .jar in the target/ folder.
I cloned the repo and was able to run sbt package after changing release.sbt a bit. I had to change the 'publishTo'-variable as it seemed to depend on the repository creators local environment variable, so I just commented it away.
I did not get the dependency problem, so I suppose it is correctly declared. The tests it tries to run do fail though, but sbt package compiles produces the .jar just fine.
EDIT: As Matthias Schlaipfer pointed out in the comments, the more elegant way(and much easier) would just be to add this as an depency in your build.sbt. From the readme, this is what you need to add:
resolvers += "piotrga" at
"https://github.com/piotrga/piotrga.github.com/tree/master/maven-repo"
libraryDependencies += "asyncdynamo" % "async-dynamo" % "1.6"
I have SBT and Scala installed and things work well.
Now I have an IntellIJ IDEA project, which works otherwise OK except that dependencies which are downloaded into /Users/MyName/.ivy2/cache/ are not understood.
I tried setting the cache path as a source folder:
but still things don't work out:
As you can see, org.eclipse is in red color meaning it can't find it. It must be inside that .jar file I showed on the left.
I just want to update/download dependencies using SBT, and then get the compilation to work in IntelliJ IDEA with those dependencies.
Use the sbt-idea plugin for SBT. Then you can run "gen-idea" from the sbt command line and it will generate the metadata files used by IDEA. These files contain the locations of jars as SBT uses them so that IDEA will know where the dependencies are, just like SBT.
Since most IDEs are only able to import Maven projects, I'd like to generate a POM.xml from an SBT managed project, is there a better way to do it?
Did you try make-pom from sbt?
It generates basic POM for the current project at the ./target.
To customize generation, you can override pomExtra, pomIncludeRepository and pomPostProcess at the project definition.
There is a very direct way provided by SBT. You can use the below command where your SBT file exixts:
sbt makePom
This will generate the .pom file in the target folder you can search that and rename to pom.xml and keep that file in the location and run mvn clean compile install to get full out of it.
Vasil's answer is correct, but for Eclipse and IDEA you can generate IDE metadata more directly using plugins. For IDEA https://github.com/mpeltonen/sbt-idea, and for Eclipse https://github.com/musk/SbtEclipsify .