I've read other threads on SO about iterating through collections from config files in Scala but they always assume the type in question is either a ConfigList or an ObjectList. In my case it is a more complex structure and I could not figure out how to access its elements.
In my config file I need to have a group of tuples without being too finicky on the collection type. For instance, I'm open to use a List of tuples or a Map[String, List], etc. Like the following (from application.conf):
myprogr {
groupsOfSomeStuff
{
group_1 -> {
name = "name1",
url = "url1",
something = "whatever"
},
...,
group_n -> {
name = "namen",
url = "urln",
something = "whatever"
}
}
}
At the moment with the conf file above, I can only print the whole groupsOfSomeStuff but I can not access any of its individual elements:
var conf = ConfigFactory.load()
println(conf.getObject("myprogr.groupsOfSomeStuff"))
which returns:
SimpleConfigObject({"group_1 ->":{"something":"whatever","name":"name1","url":"url1"}, ..., "group_n ->":{"something":"whatever","name":"namen","url":"urln"})
If I try to print conf.getObjectList or conf.getConfList I get an error at run time cause what gets extracted from the conf file is not a list but an object. The same happens if I substitute the "->" in the conf file with ":" or with "=" (since as I wrote, I'm open to different types of collections).
If I try to assign conf.getObject("myprogr.groupsOfSomeStuff") to a var of type SimpleConfigObject (with the intention of iterate through the elements of its "value" Map attribute), I get a compile-time error "SimpleConfigObject is not accessible from this position".
How can I iterate through the group_1, ..., group_n elements and individually access the name, url and something parts of each entry's value?
Thanks a million in advance! ;)
object TestConfig extends App {
import scala.collection.JavaConverters._
case class Foo(name: String, url: String, something: String)
val config = ConfigFactory.parseResources("test.conf")
val path = "myprogr.groupsOfSomeStuff"
val fooList: List[Foo] = config.getObject(path).keySet().asScala.map { key =>
val member = config.getObject(s"$path.$key").toConfig
Foo(member.getString("name"), member.getString("url"), member.getString("something"))
}.toList
println(fooList)
}
It should print List(Foo(name1,url1,whatever), Foo(namen,urln,whatever))
I hope this is what you are trying to do.
Related
I am parsing an XML document and store its data in various other structured document formats. In this XML document, the elements reference other elements, such as:
<myCar id="12" name="Porsche XYZ" ...>
<connected refId="3" />
</myCar>
...
<myCar id="3" name="Audi XYZ" ...>
...
</myCar>
Here, refId maps to id. When creating the myCar instance with the id 12, I cannot reference to myCar with id 3, because it has not yet been parsed.
Obviously, the easy solution would be to parse the document twice (instantiate references in the second run, after all elements have been parsed and created). However, for performance reasons I only want to parse the document once. Thus, I thought I could just store the relevant reference data in a case class and build a list of instances that is passed from one method to another, in order to process it after having parsed the entire document.
However, my problem is that the logic for creating the references varies to a great extent. So, I cannot use something like this:
case class Ref (a: String, b: String)
val refs: List[Ref] = List.empty
// 1. fill the list with references during parsing
// 2. after parsing the document, process all references in the list
I think what I need is to move all my reference creation logic to separate methods, and then when parsing the document maintain a list with "pointers" to these methods including the appropriate parameters. In this way, I could just iterate through the list after parsing the entire document, and call every method with the correct parameters.
How can this be achieved?
I'm not 100% sure what you're asking but I think you are asking for a way to link the connected cars to the base case class such that after parsing the XML you have a list of cars and for any given car you can access the refId attribute (materialized as a full car object) from the connected tag.
Here is a simple approach:
given:
val xml = <root>
<myCar id="12" name="Porsche XYZ">
<connected refId="3" />
</myCar>
<myCar id="3" name="Audi XYZ">
</myCar>
</root>
First we'll make a case class to model a myCar:
case class Car(
id: String,
name: String,
connectedId: Option[String]
)
Then we parse the XML into Car instances. I'm going to parse it into a Map[String, Car] where the key is Car.id:
val result = (xml \ "myCar").foldLeft(Map.empty[String, Car]) {
case (acc, next) =>
val id: String = (next \# "id")
val name = (next \# "name")
val connectionStr = (next \ "connected" \# "refId")
val connection = Option.unless(connectionStr.isEmpty)(connectionStr)
val car = Car(
id,
name,
connection
)
acc + (id -> car)
}
Next we need a way to turn connectedId into an actual car. I did this by adding a method to Car changing the case class to:
case class Car(
id: String,
name: String,
connectedId: Option[String]
) {
def getConnected(cars: Map[String, Car]): Option[Car] = {
connectedId.flatMap { id =>
cars.get(id)
}
}
}
This method (getConnected) takes the Map produced in the previous step.
Get the list of cars with:
result.values // Iterable(Car(12,Porsche XYZ,Some(3)), Car(3,Audi XYZ,None))
Get the connected car for the first car in the list:
result.values.head.getConnected(result) // Some(Car(3,Audi XYZ,None))
If you want to "fill in" the connected cars add a field to hold the connected car (pass None in the initial foldLeft above):
case class Car(
id: String,
name: String,
connectedId: Option[String],
connected: Option[Car],
) {
def getConnected(cars: Map[String, Car]): Option[Car] = {
connectedId.flatMap { id =>
cars.get(id)
}
}
}
Then just map over the list, adding the connections:
result.values
.map { car =>
val connectedCar = car.connectedId.flatMap { id =>
car.getConnected(result)
}
car.copy(connected = connectedCar)
}
This produces:
List(Car(12,Porsche XYZ,Some(3),Some(Car(3,Audi XYZ,None,None))), Car(3,Audi XYZ,None,None))
This does not recursively fill in the connected cars. You'd have to switch to either make this recursive somehow or use a var in Car to track the connected car and modify references instead of using .copy to accomplish that. I haven't thought about this too much though.
Full working code here: https://scastie.scala-lang.org/YuhNdszQROKTaNExMchaCg
def getListOfImageNames(dir: String): List[String] = {
val names = new File(dir)
names.listFiles.filter(_.isFile)
.map(_.getName).toList
}
def getListOfImages(dir: String): List[String] = {
val files = new File(dir)
files.listFiles.filter(_.isFile)
.filter(_.getName.endsWith(".png"))
.filter(_.getName.endsWith(".jpg"))
.map(_.getPath).toList
}
I have a directory where I have different photos, small size, large size and I have already managed to write methods which: one of them only pulls out the names of the photos and the other the photos. How can I now combine them into a map, for example, then calculate their resolution using the method, if the photo is larger than, for example, 500x500, add a prefix to the name and save it in the X folder. Do you have any ideas? I'm not experienced in Scala, but I like the language very much.
As I got you need to get map of image name to image path. You can achieve it like so:
def getImagesMap(dirPath: String): Map[String, String] = {
val directory = new File(dirPath)
directory.listFiles.collect{
case file if file.isFile &&
(file.getName.endsWith(".png") ||
file.getName.endsWith(".jpg")) =>
file.getName -> file.getPath
}.toMap
}
here I use collect function. It's like combination of map and filter functions. Inside collect a pattern matching expression. If file matches pattern matching it will evaluate pair creation: file name to file path. Otherwise file just will be filtered. After I use toMap for conversion Array[(String, String)] to Map[String, String]. You can read more about collect here.
My Query is, read input from a file and convert data lines of the file to List[Map[Int,String]] using scala. Here I give a dataset as the input. My code is,
def id3(attrs: Attributes,
examples: List[Example],
label: Symbol
) : Node = {
level = level+1
// if all the examples have the same label, return a new node with that label
if(examples.forall( x => x(label) == examples(0)(label))){
new Leaf(examples(0)(label))
} else {
for(a <- attrs.keySet-label){ //except label, take all attrs
("Information gain for %s is %f".format(a,
informationGain(a,attrs,examples,label)))
}
// find the best splitting attribute - this is an argmax on a function over the list
var bestAttr:Symbol = argmax(attrs.keySet-label, (x:Symbol) =>
informationGain(x,attrs,examples,label))
// now we produce a new branch, which splits on that node, and recurse down the nodes.
var branch = new Branch(bestAttr)
for(v <- attrs(bestAttr)){
val subset = examples.filter(x=> x(bestAttr)==v)
if(subset.size == 0){
// println(levstr+"Tiny subset!")
// zero subset, we replace with a leaf labelled with the most common label in
// the examples
val m = examples.map(_(label))
val mostCommonLabel = m.toSet.map((x:Symbol) => (x,m.count(_==x))).maxBy(_._2)._1
branch.add(v,new Leaf(mostCommonLabel))
}
else {
// println(levstr+"Branch on %s=%s!".format(bestAttr,v))
branch.add(v,id3(attrs,subset,label))
}
}
level = level-1
branch
}
}
}
object samplet {
def main(args: Array[String]){
var attrs: sample.Attributes = Map()
attrs += ('0 -> Set('abc,'nbv,'zxc))
attrs += ('1 -> Set('def,'ftr,'tyh))
attrs += ('2 -> Set('ghi,'azxc))
attrs += ('3 -> Set('jkl,'fds))
attrs += ('4 -> Set('mno,'nbh))
val examples: List[sample.Example] = List(
Map(
'0 -> 'abc,
'1 -> 'def,
'2 -> 'ghi,
'3 'jkl,
'4 -> 'mno
),
........................
)
// obviously we can't use the label as an attribute, that would be silly!
val label = 'play
println(sample.try(attrs,examples,label).getStr(0))
}
}
But How I change this code to - accepting input from a .csv file?
I suggest you use Java's io / nio standard library to read your CSV file. I think there is no relevant drawback in doing so.
But the first question we need to answer is where to read the file in the code? The parsed input seems to replace the value of examples. This fact also hints us what type the parsed CSV input must have, namely List[Map[Symbol, Symbol]]. So let us declare a new class
class InputFromCsvLoader(charset: Charset = Charset.defaultCharset()) {
def getInput(file: Path): List[Map[Symbol, Symbol]] = ???
}
Note that the Charset is only needed if we must distinguish between differently encoded CSV-files.
Okay, so how do we implement the method? It should do the following:
Create an appropriate input reader
Read all lines
Split each line at the comma-separator
Transform each substring into the symbol it represents
Build a map from from the list of symbols, using the attributes as key
Create and return the list of maps
Or expressed in code:
class InputFromCsvLoader(charset: Charset = Charset.defaultCharset()) {
val Attributes = List('outlook, 'temperature, 'humidity, 'wind, 'play)
val Separator = ","
/** Get the desired input from the CSV file. Does not perform any checks, i.e., there are no guarantees on what happens if the input is malformed. */
def getInput(file: Path): List[Map[Symbol, Symbol]] = {
val reader = Files.newBufferedReader(file, charset)
/* Read the whole file and discard the first line */
inputWithHeader(reader).tail
}
/** Reads all lines in the CSV file using [[java.io.BufferedReader]] There are many ways to do this and this is probably not the prettiest. */
private def inputWithHeader(reader: BufferedReader): List[Map[Symbol, Symbol]] = {
(JavaConversions.asScalaIterator(reader.lines().iterator()) foldLeft Nil.asInstanceOf[List[Map[Symbol, Symbol]]]){
(accumulator, nextLine) =>
parseLine(nextLine) :: accumulator
}.reverse
}
/** Parse an entry. Does not verify the input: If there are less attributes than columns or vice versa, zip creates a list of the size of the shorter list */
private def parseLine(line: String): Map[Symbol, Symbol] = (Attributes zip (line split Separator map parseSymbol)).toMap
/** Create a symbol from a String... we could also check whether the string represents a valid symbol */
private def parseSymbol(symbolAsString: String): Symbol = Symbol(symbolAsString)
}
Caveat: Expecting only valid input, we are certain that the individual symbol representations do not contain the comma-separation character. If this cannot be assumed, then the code as is would fail to split certain valid input strings.
To use this new code, we could change the main-method as follows:
def main(args: Array[String]){
val csvInputFile: Option[Path] = args.headOption map (p => Paths get p)
val examples = (csvInputFile map new InputFromCsvLoader().getInput).getOrElse(exampleInput)
// ... your code
Here, examples uses the value exampleInput, which is the current, hardcoded value of examples if no input argument is specified.
Important: In the code, all error handling has been omitted for convenience. In most cases, errors can occur when reading from files and user input must be considered invalid, so sadly, error handling at the boundaries of your program is usally not optional.
Side-notes:
Try not to use null in your code. Returning Option[T] is a better option than returning null, because it makes "nullness" explicit and provides static safety thanks to the type-system.
The return-keyword is not required in Scala, as the last value of a method is always returned. You can still use the keyword if you find the code more readable or if you want to break in the middle of your method (which is usually a bad idea).
Prefer val over var, because immutable values are much easier to understand than mutable values.
The code will fail with the provided CSV string, because it contains the symbols TRUE and FALSE which are not legal according to your programs logic (they should be true and false instead).
Add all information to your error-messages. Your error message only tells me what that a value for the attribute 'wind is bad, but it does not tell me what the actual value is.
Read a csv file ,
val datalines = Source.fromFile(filepath).getLines()
So this datalines contains all the lines from the csv file.
Next, convert each line into a Map[Int,String]
val datamap = datalines.map{ line =>
line.split(",").zipWithIndex.map{ case (word, idx) => idx -> word}.toMap
}
Here, we split each line with ",". Then construct a map with key as column number and value as each word after the split.
Next, If we want List[Map[Int,String]],
val datamap = datalines.map{ line =>
line.split(",").zipWithIndex.map{ case (word, idx) => idx -> word}.toMap
}.toList
I'm using the Specs2 JSONMatcher to validate that a GET request is being correctly converted from its internal representation (there are some manipulations we do before generating the JSON). What I need to do is, make sure that an element in the JSON array matches the corresponding object from our repository.
What I've tried:
val response = response.entity.asString // Spray's way of getting the JSON response
repository.all.map { obj =>
resp must */ ("id" -> obj.id)
resp must */ ("state" -> generateState(obj)
}
The problem is that the */ matcher just finds that "state": "whatever" (assuming generateState returns "whatever") exists somewhere in the JSON document, not necessarily in the same one matched by the ID
I tried using the indices but the repository.all method doesn't always return the elements in the same order, so there's no way of matching by index.
What I'd like to do is, iterate over the elements of the JSON array and match each one separately. Say an /## operator (or something) that takes matchers for each element:
resp /## { elem =>
val id = elem("id")
val obj = repository.lookup(id)
elem /("state" -> generateState(obj))
}
Does anyone have a way to do this or something similar?
Probably the easiest thing to do for now (until a serious refactoring of JsonMatchers) is to do some parsing and recursively use a JsonMatchers in a Matcher[String]:
"""{'db' : { 'id' : '1', 'state' : 'ok_1'} }""" must /("db" -> stateIsOk)
// a string matcher for the json string in 'db'
def stateIsOk: Matcher[String] = { json: String =>
// you need to provide a way to access the 'id' field
// then you can go on using a json matcher for the state
findId(json) must beSome { id: String =>
val obj = repository.lookup(id)
json must /("state" -> generate(obj))
}
}
// I am using my own parse function here
def findId(json: String): Option[String] =
parse(json).flatMap { a =>
findDeep("id", a).collect { case JSONArray(List(v)) => v.toString }
}
// dummy system
def generate(id: String) = "ok_"+id
case object repository {
def lookup(id: String) = id
}
What I did in the end is use responseAs[JArray], JArray#arr and JObject#values to convert the JSON structures into Lists and Maps, and then used the standard List and Map matchers. Much more flexible.
I have in a Play2 Template:
files: Option[List[(String, reactivemongo.api.gridfs.ReadFile[reactivemongo.bson.BSONValue])]]
I would like to display the files in a drop down form Select, but only if the files are there!
How would I do this?
As Julien Lafont said, you should consider transforming this list before giving it to your template. For example if you just want a list of read files (on which you can call filename and id):
val fileList = files.toList.flatten.map(_._2) // fileList is a List[ReadFile[BSONValue]]]
And if you want just want to get filenames and their ids (given that their ids are BSONObjectIDs), you may write this:
val fileList = files.toList.flatten.map { file =>
val id = file._2.id match {
case oid: BSONObjectID => oid.stringify
case id => id.toString
}
id -> file._2.filename
}
// fileList is a List[(String, String)] where the first element of the tuple is a string version of the id and the second is the name of the file.