How to define withSources and/or withJavadoc in mill ivyDeps - scala

If you want to load module sources and/or javadocs you write following sbt:
lazy val joda_timeV = "1.2.3"
lazy val scalatagsV = "1.2.3"
lazy val upickleV = "1.2.4"
lazy val autowireV = "1.2.5"
lazy val scalarxV = "1.2.6"
libraryDependencies ++= Seq(
"joda-time" % "joda-time" % joda_timeV withJavadoc(),
"com.lihaoyi" %%% "scalatags" % scalatagsV withSources() withJavadoc(),
"com.lihaoyi" %% "upickle" % upickleV withSources() withJavadoc(),
"com.lihaoyi" %%% "autowire" % autowireV withJavadoc(),
"com.lihaoyi" %%% "scalarx" % scalarxV withSources(),
"org.scalatestplus.play" %% "scalatestplus-play" % scalatestplus_playV % "test" withSources() withJavadoc()
),
In mill you say
override def ivyDeps = Agg(
ivy"joda-time:joda-time:${joda_timeV}",
ivy"com.lihaoyi:::scalatags:${scalatagsV}",
ivy"com.lihaoyi::upickle:${upickleV}",
ivy"com.lihaoyi:::autowire:${autowireV}",
ivy"com.lihaoyi:::scalarx:${scalarxV}"
)
but how can you add withJavadoc() or withSources() or withSources() withJavadoc() in to mill build.sc?
There is function .withConfiguration(String) but no scaladoc how to use it.
Is it possible to define that a module is available only in test (like org.scalatestplus.play in the previous code) or should I create separate ivyDeps for testing module?

Regarding your first question, I assume, your are interested in good IDE support, e.g. completion and jump-to the sources of your dependencies.
Mill already supports IDE integration. It comes with a project generator for IntelliJ IDEA (mill mill.scalalib.GenIdea/idea), which automatically downloads the sources for you. Alternatively, you can use the new BSP Support (Build Server Protocol) which should in combination with the Metals Language Server (https://scalameta.org/metals/) provide a nice editing experience in various IDEs and Editors. Unfortunately, at the time of this writing, Mills built-in BSP server isn't as robust as its IDEA generator, but there is even another alternative, the Bloop contrib module. All these methods should provide decent code navigation through dependencies and completion.
And to your second question:
Is it possible to define that a module is available only in test (like org.scalatestplus.play in the previous code) or should I create separate ivyDeps for testing module?
Test dependencies are declared it the test modules (which are technically regular modules too).
// build.sc
// ...
object yourplaymodule extends PlayModule {
override def ivyDeps = Agg(
ivy"joda-time:joda-time:${joda_timeV}",
ivy"com.lihaoyi:::scalatags:${scalatagsV}",
ivy"com.lihaoyi::upickle:${upickleV}",
ivy"com.lihaoyi:::autowire:${autowireV}",
ivy"com.lihaoyi:::scalarx:${scalarxV}"
)
// ...
object test extends PlayTests {
override def ivyDeps = Agg(
ivy"org.scalatestplus.play::scalatestplus-play:${scalatestplus_playV}"
)
}
}
Edit 2021-09-16: Added the answer to the first question.

Related

reduceByKey/aggregateByKey alternative for a DStream[Class] Spark Streaming

There's already a similar question here, but it is using Maven, and I'm using sbt. Moreover none of the solutions there worked for me
I'm using Spark 2.4.0, Scala 2.11.12 and IntelliJ IDEA 2019.1
My build.sbt looks like:
libraryDependencies ++= Seq(
"com.groupon.sparklint" %% "sparklint-spark212" % "1.0.12" excludeAll ExclusionRule(organization = "org.apache.spark"),
"org.apache.spark" %% "spark-core" % "2.4.0",
"org.apache.spark" %% "spark-sql" % "2.4.0",
"org.apache.spark" %% "spark-streaming" % "2.4.0",
"org.apache.spark" %% "spark-streaming-kafka" % "1.6.2",
"com.datastax.spark" %% "spark-cassandra-connector" % "2.4.0",
"com.typesafe.slick" %% "slick" % "3.3.0",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.typesafe.slick" %% "slick-hikaricp" % "3.3.0",
"com.typesafe.slick" %% "slick-extensions" % "3.0.0"
)
Edit all over:
I will be receiving a stream of data from Kafka, which will be sent to the Spark Streaming context using:
val rawWeatherStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)
From this, I want to create a stream of RawWeatherData objects. A sample output from the stream would look like:
(null,725030:14732,2008,12,31,11,0.6,-6.7,1001.7,80,6.2,8,0.0,0.0)
Things look all good, except that I need to remove the first null value to create the stream of RawWeatherData objects as the constructor cannot accept the first null value, but can accept all other values from the stream.
Just for clarity sakes, here's what RawWeatherData looks like (I cannot edit this):
case class RawWeatherData(
wsid: String,
year: Int,
month: Int,
day: Int,
hour: Int,
temperature: Double,
dewpoint: Double,
pressure: Double,
windDirection: Int,
windSpeed: Double,
skyCondition: Int,
skyConditionText: String,
oneHourPrecip: Double,
sixHourPrecip: Double) extends WeatherModel
To achieve that purpose, I send my stream into a function, which returns me the desired stream of RawWeatherData objects:
def ingestStream(rawWeatherStream: InputDStream[(String, String)]): DStream[RawWeatherData] = {
rawWeatherStream.map(_._2.split(",")).map(RawWeatherData(_))
}
Now I am looking to insert this stream into a MySQL/DB2 database. From this RawWeatherData object (725030:14732,2008,12,31,11,0.6,-6.7,1001.7,80,6.2,8,0.0,0.0), the left highlighted bold part is the primary key, and the right bold part is the value that has to be reduced/aggregated.
So essentially I want my DStream to have key-value pairs of ([725030:14732,2008,12,31] , <summed up values for the key>)
So after ingestStream, I try to perform this:
parsedWeatherStream.map { weather =>
(weather.wsid, weather.year, weather.month, weather.day, weather.oneHourPrecip)
}.saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
After the end of map, I try to write .reduceByKey(), but when I try that, the error says Cannot resolve symbolreduceByKey`. I'm not sure why this is happening as the function is available in the spark documentation.
PS. Right now weather.oneHourPrecip is set to counter in cassandra, so cassandra will automatically aggregate the value for me. But this will not be possible in other databases like DB2, hence I wanted an apt replacement, like reduceByKey in spark. Is there any way to proceed with such a case?
Type of your stream is DStream[RawWeatherData] and reduceByKey is available only on streams of type DStream[(K,V)], which is a stream of tuples consisting of key and value.
What you wanted to do is probably to use mapValues instead of map:
val parsedWeatherStream: DStream[(String, RawWeatherData)] = rawWeatherStream
.mapValues(_.split(","))
.mapValues(RawWeatherData)
As you can see by type of parsedWeatherStream from the snippet above, if you'd use mapValues, you'd not discard your keys and you could use reduceByKey.

Building a KNN classifier with multiple classes that identifies different audio

I am in the process of building a KNN algorithm with the purpose of identifying and categorizing common engine problems in Matlab. I have been using the builds from the book 'An Introduction To audio Analysis: A Matlab Approach' which contains exactly what I need to continue my experiment. The problem is that when I identify where the audio samples are kept on my computer, the system claims that it gives out the message "audio sample path is not valid!" Below is the original algorithm the authors of the book have supplied
function kNN_model_add_class(modelName, className, classPath, ...
listOfStatistics, stWin, stStep, mtWin, mtStep)
%
% function kNN_model_add_class(modelName, className, classPath, ...
% listOfStatistics, stWin, stStep, mtWin, mtStep)
%
% This function adds an audio class to the kNN classification model
%
% ARGUMENTS;
% - modelName: the filename of the model (mat file)
% - className: the name of the audio class to be added to the model
% - classPath: the path of the directory where the audio segments of the
% new class are stored
% - listOfStatistics: list of mid-term statistics (cell array)
% - stWin, stStep: short-term window size and step
% - mtWin, mtStep: mid-term window size and step
%
% Example:
% kNN_model_add_class('modelSpeech.mat', 'speech', './Music/', ...
% {'mean','std',}, 0.050, 0.025, 2.0, 1.0);
%
if ~exist(classPath,'dir')
error('Audio sample path is not valid!');
else
classPath = [classPath filesep];
end
% check if the model elaready exists:
fp = fopen(modelName, 'r');
if fp>0 % check if file already exists
load(modelName);
end
% Feature extraction:
D = dir([classPath '*.wav']);
F = [];
for (i=1:length(D)) % for each wav file in the given path:
curFileName = [classPath D(i).name];
FileNamesTemp{i} = curFileName;
% mid-term feature extraction for each wav file:
midFeatures = featureExtractionFile(curFileName, ...
stWin, stStep, mtWin, mtStep, listOfStatistics);
% long-term averaging:
longFeatures = mean(midFeatures,2);
F = [F longFeatures];
end
% save the model:
Statistics = listOfStatistics;
fp = fopen(modelName, 'r');
if fp<0 % model does not exist --> generate
ClassNames{1} = className;
Features{1} = F;
FileNames{1} = FileNamesTemp;
save(modelName, 'ClassNames', 'Features', ...
'Statistics', 'stWin', 'stStep', 'mtWin', 'mtStep', 'FileNames');
else
load(modelName);
ClassNames{end+1} = className;
Features{end+1} = F;
FileNames{end+1} = FileNamesTemp;
save(modelName, 'ClassNames', 'Features', ...
'Statistics', 'stWin', 'stStep', 'mtWin', 'mtStep', 'FileNames');
end
Here is the way that I have implemented it into my own project.
%Knn algorithm training
%path to folder containing the audio segements
strDir ='/Users/itsolutions/Documents/MATLAB/Wav_engine_edits ';
%mid-term statistics to be used:
Statistics = {'mean','median','std','stdbymean','max','min'};
%short-term and mid-term, processing windon length and step:
stWin = 0.040; stStep = 0.040;
mtWin = 2.0; mtStep = 1.0;
%perform feature extraction
kNN_model_add_class ('model8.mat','connection rod noise', [ strDir '/Connection_rod_edits/'],Statistics, stWin, stStep, mtWin, mtStep);
kNN_model_add_class ('model8.mat','detonation noise', [strDir ' /Detonation_noise_edits/'],Statistics, stWin, stSteo, mtWin, mtStep)
%kNN_model_add_class ('model8.mat','Engine bearing noise' [strDir '/Engine Bearing Noise edits/'],Statistics, stWin, stStep, mtWin, mtStep);
Any ideas? I have looked into the objects present in the function knn_model_add_class() and have found that it is still using wavread, which in the 2016b version of matlab in invalid syntax now.
Could anyone give me a hand? Seems like I am going in circles and it is something really obvious that i can't see.
Regards
M.Brown
Edit:
Using the process of elimination I have found that the error comes from this piece of code
if ~exist(classPath,'dir')
error('Audio sample path is not valid!');
else
classPath = [classPath filesep];
end
The code cannot seem to find the audio samples in question. I have contacted the author of the book, but until then does anybody know why the matlab file path won't recognize the wav files?

Scalatest Playframework must contain List[String]

I am using playframework 2.4.x, and this libraries
"org.scalatest" %% "scalatest" % "2.2.1" % "test"
"org.scalatestplus" %% "play" % "1.4.0-M3" % "test"
I want to check if therea are some strings in a List I build on the test, this is the code
val userTeams = validateAndGet((teamsUserResponse.json \ "teams").asOpt[List[TeamUser]]).map( x => x.teamKey )
userTeams must contain ("team1", "team2")
But I am getting this error
List("team1", "team2") did not contain element (team1,team2)
If you write ("team1", "team2") then you're actualy creating a tuple of two strings which is from perspective of the ScalaTest matcher a single element.
Based on documentation you have to use allOf:
userTeams must contain allOf ("team1", "team2")

MCC compiler "Internal Error: Could not determine class of method"

I have Matlab script that I am trying to compile as an executable that will run on a headless server as part of a large batch process. The script calls functions and classes written by several non-programmers (scientists) over the span of more than a decade, and I am having a difficult time getting it to compile.
The script will run in a Matlab instance, but MCC gives me an error that I do not quite understand:
Compiler version: 5.1 (R2014a)
Dependency analysis by REQUIREMENTS.
Error using matlab.depfun.internal.MatlabSymbol/proxy (line 405)
Internal1 Error: Could not determine class of method
"/home/me/MyCode/#tsd/Data.m". Number of classes checked:
17.
#tsd/tsd.m looks like this:
function tsa = tsd(t, Data)
% tsd Creates tsd object (tsd = class of timestamped arrays of data)
%
% tsa = tsd(t, Data)
%
% INPUTS:
% t - list of timestamps - must be sequential, but don't have to be continuous
% Data - data, possibly an array. If data is n-dimensional, then time should be the FIRST axis.
% OUTPUTS:
% tsa - a tsd object
%
% Completely compatible with ctsd.
%
% Methods
% tsd/Range - Timestamps used
% tsd/Data - Returns the data component
% tsd/DT - Returns the DT value (mean diff(timestamps))
% tsd/StartTime - First timestamp
% tsd/EndTime - Last timestamp
% tsd/Restrict - Keep data within a certain range
% tsd/CheckTS - Makes sure that a set of tsd & ctsd objects have identical start and end times
% tsd/cat - Concatenate ctsd and tsd objects
% tsd/Mask - Make all non-mask values NaN
%
% ADR 1998, version L4.0, last modified '98 by ADR
% RELEASED as part of MClust 2.0
% See standard disclaimer in ../Contents.m
if nargin == 0
tsa.t = NaN;
tsa.data = NaN;
tsa = class(tsa, 'tsd');
return
end
if nargin < 2
error('tsd constructor must be called with T, Data');
end
tsa.t = t;
tsa.data = Data;
tsa = class(tsa, 'tsd');
and the Data.m file:
function v = Data(tsa, ix)
% tsd/Data Retrieves data from tsd
%
% d = Data(tsa)
% d = Data(tsa, ix)
%
% INPUTS:
% tsa - tsd object
% ix - alignment list (timestamps)
% OUTPUTS:
% v - the tsa.Data
%
% if called with alignment list, returns those tsa.Data(ix)
% if called without, returns complete tsa.Data
%
% ADR 1998, version L4.1, last modified '98 by ADR
% RELEASED as part of MClust 2.0
% See standard disclaimer in ../Contents.m
switch nargin
case 2
f = findAlignment(tsa, ix);
v = SelectAlongFirstDimension(tsa.data,f);
case 1
v = tsa.data;
otherwise
error('Unknown number of input arguments');
end
So the script calling the tsd function runs just find in the Matlab session, but the compiler throws the error described above. This is my first time working with Matlab and I am totally stumped. There is another class with a method named "Data", but that shouldn't cause this problem, could it?
I was having the exact same problem and was able to come up with a workaround.
I was facing this problem when I was compiling the MATLAB library using the command
mcc -W cpplib:LibName -T link:lib main.m -a './' -d 'dll_destintaion_dir'
LibName: DLL name
main.m: MATLAB entry point
dll_destintaion_dir: destination where DLL created will be stored
The error was gone when I compiled using this command instead
mcc -W cpplib:LibName -T link:lib main.m -a 'full_path_of_project_folder/*'
-d 'dll_destintaion_dir'
Note: This method does not add folders recursively to the archive. So subfolders have to be added explicitly using another -a flag in the command shown above.

Play Framework 2.1 upgrade, now having to cast everything for views?

I just migrated across to Play Framework 2.1-RC1 from 2.0 on an existing project and for some reason I'm now having to cast everything to Scala classes from Java classes when I render the views. (Obviously I'm using Play in a Java project rather than a Scala project)
Below is an example error...
render(java.lang.String,scala.collection.immutable.List<models.User>) in views.html.list cannot be applied to (java.lang.String,java.util.List<models.User>)
And the top line of my view...
#(message: String, users : List[models.User])
From this I surmise that for some reason classes aren't being automatically cast from java.util.List to the scala equivalent. I'm a Java guy, not a Scala guy at this stage so I may be doing something stupid.
Example code that calls render...
public static Result list() {
List<User> users = MorphiaManager.getDatastore().find(User.class).asList();
System.out.println("about to try to display list of " + users.size() + " users");
return ok(list.render("Welcome", msgs));
}
Build.scala below
import sbt._
import Keys._
import PlayProject._
object ApplicationBuild extends Build {
val appName = "blah-worker"
val appVersion = "1.0-SNAPSHOT"
val appDependencies = Seq(
// Play framework dependencies
javaCore, javaJdbc, javaEbean,
// Add your project dependencies here,
"org.apache.camel" % "camel-core" % "2.10.0",
"org.apache.camel" % "camel-jms" % "2.10.0",
"org.apache.camel" % "camel-mail" % "2.10.0",
"org.apache.camel" % "camel-jackson" % "2.10.0",
"org.apache.camel" % "camel-gson" % "2.10.0",
"org.apache.activemq" % "activemq-core" % "5.6.0",
"org.apache.activemq" % "activemq-camel" % "5.6.0",
"org.apache.activemq" % "activemq-pool" % "5.6.0",
"com.google.code.morphia" % "morphia" % "0.99.1-SNAPSHOT",
"com.google.code.morphia" % "morphia-logging-slf4j" % "0.99",
"cglib" % "cglib-nodep" % "[2.1_3,)",
"com.thoughtworks.proxytoys" % "proxytoys" % "1.0",
"org.apache.james" % "apache-mime4j" % "0.7.2",
("org.jclouds" % "jclouds-allblobstore" % "1.5.0-beta.4").exclude("com.google.guava", "guava").exclude("org.reflections", "reflections"),
("org.reflections" % "reflections" % "0.9.7").exclude("com.google.guava", "guava").exclude("javassist", "javassist")
)
val main = play.Project(appName, appVersion, appDependencies).settings(
// Add your own project settings here
resolvers += "Morphia repo" at "http://morphia.googlecode.com/svn/mavenrepo/",
resolvers += "CodeHaus for ProxyToys" at "http://repository.codehaus.org/",
checksums := Nil
)
}
Are you missing the new 'javaCore' dependency? It is required for Java projects using Play 2.1. Look here for migration details:
https://github.com/playframework/Play20/wiki/Migration
Figured it out, I hadn't updated one of the imports on Build.scala.
Specifically...
import PlayProject._
should be updated to...
import play.Project._
which is also detailed in the migration guide (but I missed it): https://github.com/playframework/Play20/wiki/Migration
I'm not sure if it will fix you problem but can you try adding this import:
import scala.collection.JavaConversions.*;