I am writing a code to perform kernel K-Means (aka https://en.wikipedia.org/wiki/K-means_clustering, but with a trick). I need to generate data, and as a first simple generator I tried to implement a Gaussian Mixture Model. Here are my code:
package p02kmeans
import breeze.linalg._
import breeze.stats.distributions._
/**
* First data generation is simple, gaussian mixture model.
*/
object Data {
class GaussianClassParam (
val mean: Double,
val sd: Double)
/**
* #param proportion marginal probability for each label
* #param param param[j][k] returns the GaussianClassParam for the k class of the j variable
* #param nObs number of observations to be generated
* #result DenseMatrix_ij where i is the observation index and j is the variable number
*/
def gaussianMixture(
proportion: DenseVector[Double],
param: Vector[Vector[GaussianClassParam]],
nObs: Int)
: DenseMatrix[Double] = {
val nVar = param.size
val multiSampler = Multinomial(proportion) // sampler for the latent class
val varSamplerVec = param.map(v => v.map(c => Gaussian(c.mean, c.sd)))
val zi = DenseVector.fill[Int](nObs)(multiSampler.sample)
val data = DenseMatrix.tabulate[Double](nObs, nVar)((i, j) => varSamplerVec(j)(zi(i)).sample)
return data
}
}
When I try to compile my code (I use Scala-Ide and sbt eclipse on Windows 10) I get 2 errors:
Error in Scala compiler: assertion failed: List(method apply$mcI$sp, method apply$mcI$sp)
SBT builder crashed while compiling. The error message is 'assertion failed: List(method apply$mcI$sp, method apply$mcI$sp)'. Check Error Log for details.
The error is triggered by the line:
val data = DenseMatrix.tabulate[Double](nObs, nVar)((i, j) => varSamplerVec(j)(zi(i)).sample)
And disappear with:
val data = DenseMatrix.tabulate[Double](nObs, nVar)((i, j) => 12.0)
Could you help me debug this ?
My sbt configuration:
name := "Sernel"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scalanlp" %% "breeze" % "0.13.1",
"org.scalanlp" %% "breeze-natives" % "0.13.1",
"org.scalanlp" %% "breeze-viz" % "0.13.1"
)
I have the same errors on my OSX setup.
If you want to test the whole package (as, if you want to reproduce the error), the code is available on Github: https://github.com/vkubicki/sernel, and I am available to provide directions :).
It seems like it's a compiler bug (I suppose in scala macroses as Breeze is using those). You could try to perform total clean in the project (maybe even including .ivy2 folder - this could be a difference between your MacOS and Windows setup) and also update your scala to 2.11.11 (or maybe even 2.12.x)
However, similar issue with Scala 2.11.6 (and something tells me it's inherited in subsequent versions of Scala) wasn't fixed: https://issues.scala-lang.org/browse/SI-9284
So probably, you'll have to repeatedly perform cleaning sometimes or maybe try some other NumPy analogs like: scalala, Nd4j/Ndjs.
It could also help to try another IDE (IDEA/Atom) or try to use "bare" SBT as Eclipse is probably interfering by calling Scala's compiler front-end.
Related
I need to build different versions of a project that depend on different versions of specific libraries. These library versions are compatible on a source level. So I want to build com.company:myproject-mylib-1.0_2.12:1.0.0 that depends on com.company:mylib:1.0 and com.company:myproject-mylib-2.0_2.12:1.0.0 that depends on com.company:mylib:2.0. One of the ways I can think of is using environment variable to store library version, is there a more elegant way?
Well, one of these you have to set explicitly, then the other one can be calculated:
libraryDependencies += "com.company" % "mylib" % (if (name.value.endsWith("1.0")) "1.0" else "2.0")`
Then during build you could do something like:
sbt> ;compile;publish // <- e.g. for v 1.0
sbt> set name := "myproject-mylib-2.0"
sbt> ;compile;publish // <- e.g. for v 2.0
If you want it more principled you could define a new key:
val isNew = settingKey[Boolean]("Are we building a new version of a library")
Global / isNew := false
name := if (isNew.value) "myproject-mylib-2.0" else "myproject-mylib-1.0"
libraryDependencies += "com.company" % "mylib" % (if (isNew.value) "2.0" else "1.0")`
which you would build like
sbt> ;compile;publish // <- for v 1.0
sbt> set isNew := true
sbt> ;compile;publish // <- for v 2.0
or from command line
sbt compile publish "set isNew := true" compile publish
In fact, I don't see any of the xMapped functions in the new release of the librarymanagement repository.
I get this error:
build.sbt:84: error: value fullMapped is not a member of object sbt.librarymanagement.CrossVersion
"org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.fullMapped{
for the following usage:
,addCompilerPlugin( // For circe generic:
"org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.fullMapped{
_ => scalaVersionSelect
}
)
CrossVersion.fullMapped wasn't kept in its sbt 0.13 form because sbt 1.x wanted to be able to serialize its key types (and CrossVersion is a transitive part of that object graph).
It was replaced by CrossVersion.fullWith, which instead of taking a general String => String function takes a String prefix and a String suffix to prepend/append to the Scala binary version.
But given you're discarding the input to use scalaVersionSelect then you don't even need CrossVersion.fullWith and can just use CrossVersion.constant, as in:
addCompilerPlugin(
"org.scalamacros" % "paradise" % "2.1.1"
cross CrossVersion.constant(scalaVersionSelect)
)
I am trying dynamically change reference to the jars dependency I am using in my project, depending on the platform (Windows or Linux)
So, its a very trivial scenario,
How can I implement this simple check in the build.sbt ?
Potential approach is to pattern match on System.getProperty("os.name") within a custom defined setting like so
val configureDependencyByPlatform = settingKey[ModuleID]("Dynamically change reference to the jars dependency depending on the platform")
configureDependencyByPlatform := {
System.getProperty("os.name").toLowerCase match {
case mac if mac.contains("mac") => "org.example" %% "somelib-mac" % "1.0.0"
case win if win.contains("win") => "org.example" %% "somelib-win" % "1.0.0"
case linux if linux.contains("linux") => "org.example" %% "somelib-linux" % "1.0.0"
case osName => throw new RuntimeException(s"Unknown operating system $osName")
}
}
and then add the evaluated setting to libraryDependencies as follows
libraryDependencies ++= Seq(
configureDependencyByPlatform.value,
"org.scalatest" %% "scalatest" % "3.0.5",
...
)
I have just spent hours on this. I'm trying to make a pretty complicated query in PostgreSQL through Slick in Scala, but Slick will NEVER take any of my parameters into account. If I try something as simple as:
def get(location: String) = {
val query = sql"select * from cities_v where name = $location limit 10"
println(query.toString)
}
The output will be:
SQLActionBuilder(Vector(select * from cities_v where name = ? limit 10),<function2>)
If instead, I try the literal insert:
def get(location: String) = {
val query = sql"select * from cities_v where name = #$location limit 10"
println(query.toString)
}
The output will be:
SQLActionBuilder(Vector(select * from cities_v where name = , ville, limit 10),<function2>
Slick will ALWAYS add commas around any literal argument, no matter where it is placed, even if it's the only argument in the query, as in:
sql"#$q"
Now, what I'd like is make more complex queries, with calculations and function calls within Postgres. But I can't get anywhere given that Slick won't let me use any of my variables.
I tried setting all sorts of implicits, some that extend GetResult some that extend StatementParameters, Slick seems to ignore all them and either replaces my arguments with ? or surrounds them with commas, thereby rendering all of my queries invalid. I would like to add that the #$ is not great because it does not provide sanitization. I'd like to stick with Slick because it's asynchronous, but I'm not sure where to go from here.
These are my version numbers, according to build.sbt:
"postgresql" % "postgresql" % "9.1-901-1.jdbc4",
"com.typesafe.slick" % "slick_2.12" % "3.2.3",
What am I doing wrong?
There is nothing wrong with that ? appears in you result statements. It is just parametrized query, and each ? represents placeholder for future value.
Just run you queries against any database and check results.
I'm trying to implement the answer to this question: https://stackoverflow.com/questions/3704647/can-you-recommend-a-charting-library-for-scala/3704974#3704974
I've downloaded and compiled Scalala from the git hub and placed the scalala_2.8.1-1.0.0.RC2-SNAPSHOT.jar in my lib folder (I'm using SBT to do my build). Here's the code:
import scalala.library.Plotting
object ScalalaTest extends Application
{
val x = Plotting.linspace(0,1);
}
I'm getting the following error:
[error] /src/main/scala/ScalalaTest.scala:6: value linspace is not a member of object scalala.library.Plotting
[error] val x = Plotting.linspace(0,1);
[error] ^
[error] one error found
It looks like my scala compiler recognizes the scalala package but doesn't recognize members of Plotting (I've tried others besides linspace). This is strange because according to the Scalala API, linspace is a member of Plotting.
That used to work and was nice and elegant - it seems the current way is:
val x = DenseVector.range(0,100) / 100.0;
plot.hold = true
plot(x, x :^ 2)
plot(x, x :^ 3, '.')
xlabel("x axis")
ylabel("y axis")
saveas("lines.png")
This needs includes:
import scalala.tensor.dense.DenseVector
import scalala.library.Plotting._
The SBT dependencies are:
val scalaToolsSnapshots = "Scala Tools Snapshots" at "http://scala-tools.org/repo-snapshots/"
val ScalaNLPMaven2 = "ScalaNLP Maven2" at "http://repo.scalanlp.org/repo/"
val ondex = "ondex" at "http://ondex.rothamsted.bbsrc.ac.uk/nexus/content/groups/public/"
val scalala = "org.scalala" %% "scalala" % "1.0.0.RC2-SNAPSHOT"
linspace seems to be a member of the trait Plotting, not of the companion object. So you'll have to create an instance of Plotting (or anything with Plotting) in order to access that method.