Unique constraint for repeated values - forms

I am trying to define a form in play! 2.0.4 with the following properties and constraints:
The form handles repeated values (lets conveniently assume that these values are of type number). So this will get us to something like this:
"numbers" -> list(number)
Each number must be unique, i.e. it must be unique with regard to all the other numbers submitted and it must be unique to the numbers that are already existent in the database (this can be checked via some function check(num: Int): Boolean).
The form error should be specific to the number, that is not unique. I don't want a general form error saying "There is duplicate number".
What would be the best way to go?

The trick here is to define a custom Constraint sort of like this example. The custom Constraint can then be used on the Mapping[T] to verify a field in the form with the verifying method.
The custom Constraint contains the logic to return a ValidationResult that is either Valid or Invalid. An error message can be passed to an Invalid result which is where you can specify what is duplicated or exists in the database.
See Play for Scala for a section on custom validation.
- Create the Constraint
//Make this lazy to prevent java.lang.ExceptionInInitializerError at runtime.
lazy val uniqueNumbersConstraint = Constraint[String](Some("Unique numbers constraint"), "")(checkNumbers)
//"Business Logic".
//Important part here is that the function returns a ValidationResult and complies with the signature for Constraint. i.e. f: (T) => ValidationResult
//Return Valid if n in numbers is not in database and there are no duplicates.
//Otherwise return Invalid and an error message showing what numbers are in the database or duplicated.
def checkNumbers(numbers: String):ValidationResult = {
val splitNums = numbers.split(" ").toList.map(_.toInt)
val dbnums = splitNums.partition(database.contains(_))
if(dbnums._1.isEmpty && uniquesAndDuplicates(splitNums)._2.isEmpty){
Valid
}else{
val duplicates = uniquesAndDuplicates(dbnums._2)._2
val error = "Database contains: " + dbnums._1 + ", duplicated values: " + duplicates
Invalid(error)
}
}
- Validate Form Using Custom Constraint
val helloForm = Form(
tuple(
"numbers" -> nonEmptyText.verifying(uniqueNumbersConstraint)
))
- Utilities
//Return unique values on left side and duplicate values on right side
def uniquesAndDuplicates(numbers: List[Int]):Tuple2[List[Int], List[Int]] = {
numbers.partition(i => numbers.indexOf (i) == numbers.lastIndexOf(i))
}
def checkNum(num: Int) = {
database.contains(num)
}
val database = List(5,6,7)
- etc
Note I defined numbers as a String in the form. When I defined it as list(number) it kept evaluating to List(). I think that is a binding issue. It's a fairly simple change to use List(1,2,3) instead of "1 2 3" if using list(number) works.
- Samples

How about something like:
def validateUnique(input: List[Int]): ValidationResult = {
// Assuming check return true if the input num doesn't exist yet
def check(num: Int): Boolean = num % 2 == 0
val unique = input.toSet
val dbDuplicates = unique.filterNot(check)
val formDuplicates = input.diff(unique.toSeq)
val duplicates = (dbDuplicates ++ formDuplicates).toList
duplicates match {
case List() => Valid
case _ => Invalid("Duplicates: " + duplicates.mkString(", "))
}
}
val uniqueConstraint = Constraint[List[Int]](validateUnique(_))
And then you can just use the new constraint with:
mapping(
...,
"ints" -> list(number).verifying(uniqueConstraint)
...

Related

Passing result of one DBIO into another

I'm new to Slick and I am trying to rewrite the following two queries to work in one transaction. My goal is to
1. check if elements exists
2. return existing element or create it handling autoincrement from MySQL
The two functions are:
def createEmail(email: String): DBIO[Email] = {
// We create a projection of just the email column, since we're not inserting a value for the id column
(emails.map(p => p.email)
returning emails.map(_.id)
into ((email, id) => Email(id, email))
) += email
}
def findEmail(email: String): DBIO[Option[Email]] =
emails.filter(_.email === email).result.headOption
How can I safely chain them, ie. to run first check for existence, return if object already exists and if it does not exist then create it and return the new element in one transaction?
You could use a for comprehension:
def findOrCreate(email: String) = {
(for {
found <- findEmail(email)
em <- found match {
case Some(e) => DBIO.successful(e)
case None => createEmail(email)
}
} yield em).transactionally
}
val result = db.run(findOrCreate("batman#gotham.gov"))
// Future[Email]
With a little help of cats library:
def findOrCreate(email: String): DBIO[Email] = {
OptionT(findEmail(email)).getOrElseF(createEmail(email)).transactionally
}

Right way to run multiple validation rules on an input value in Scala ?

What is the right way to validate an input value in Scala ? Suppose I have a password to validate with the following rules
Password should not be empty
Password length should be greater than 8 characters
There can be more validation rules. But the goal is run all the rules and return a list of Errors (List[String]).
The following are the two ways I can think of. But is there a better way?
The code
import scala.collection.mutable.ListBuffer
//Validate password using mutable list (Feels like Java style coding)
//Uses mutable list but seems a bit simpler
def validatePasswordMutableList(password: String): List[String] = {
val errors = ListBuffer.empty[String]
if (password == "")
errors += "Is empty."
if (password.length < 8)
errors += "Not long enough."
errors.toList
}
validatePasswordMutableList("") foreach println
//Validate password using Immutable list
//But has to run filter and map on the error list to get desired result.
def validatePasswordImmutableList(password: String): List[String] = {
def isEmpty = if (password == "") Some("Is empty.") else None
def isLengthValid = if (password.length < 8) Some("Not long enough.") else None
val errors = isEmpty :: isLengthValid :: Nil
errors.filter(_.isDefined).map(_.get)
}
validatePasswordImmutableList("") foreach println
One approach you should consider is to process an unlimited select of dynamic rules, passed as a list (or sequence) of functions. This way, you can recycle some or all of those rules and easily update new ones in the future.
It would be much simpler to use functions that just return Boolean, however the code below has the advantage of returning all password violations that happened in a similar way most website and applications do these days.
object ValidateExample extends App {
def applyFilters(password:String, rules:Seq[Function[String,Option[String]]]):Seq[String] = {
//You could use a while loop here if you had many rules, and wanted to stop early
val temp = rules.flatMap{f=> f(password) } //seq of items that converted to Some(string) only
temp
}
val rule1:(String)=>Option[String] = { s=>
if (s.length<8) Some("Error in password length") else None
}
//same idea different declaration syntax
val rule2:Function[String,Option[String]] = { s=>
if (s.contains("X")) Some("Error cant contain capitol X") else None
}
//example of a partial function, lifted below
val rule3:PartialFunction[String,String]={case s:String if s.isEmpty =>"Password can't be empty"}
val test1 = applyFilters("helloX",Seq(rule1,rule2,rule3.lift))
println(test1)
val test2 = applyFilters("",Seq(rule1,rule3.lift))
println(test2)
//example passed
val passed = test2.isEmpty
}
Output:
List(Error in password length, Error cant contain capitol X)
List(Error in password length, Password can't be empty)

How to read input from a file and convert data lines of the file to List[Map[Int,String]] using scala?

My Query is, read input from a file and convert data lines of the file to List[Map[Int,String]] using scala. Here I give a dataset as the input. My code is,
def id3(attrs: Attributes,
examples: List[Example],
label: Symbol
) : Node = {
level = level+1
// if all the examples have the same label, return a new node with that label
if(examples.forall( x => x(label) == examples(0)(label))){
new Leaf(examples(0)(label))
} else {
for(a <- attrs.keySet-label){ //except label, take all attrs
("Information gain for %s is %f".format(a,
informationGain(a,attrs,examples,label)))
}
// find the best splitting attribute - this is an argmax on a function over the list
var bestAttr:Symbol = argmax(attrs.keySet-label, (x:Symbol) =>
informationGain(x,attrs,examples,label))
// now we produce a new branch, which splits on that node, and recurse down the nodes.
var branch = new Branch(bestAttr)
for(v <- attrs(bestAttr)){
val subset = examples.filter(x=> x(bestAttr)==v)
if(subset.size == 0){
// println(levstr+"Tiny subset!")
// zero subset, we replace with a leaf labelled with the most common label in
// the examples
val m = examples.map(_(label))
val mostCommonLabel = m.toSet.map((x:Symbol) => (x,m.count(_==x))).maxBy(_._2)._1
branch.add(v,new Leaf(mostCommonLabel))
}
else {
// println(levstr+"Branch on %s=%s!".format(bestAttr,v))
branch.add(v,id3(attrs,subset,label))
}
}
level = level-1
branch
}
}
}
object samplet {
def main(args: Array[String]){
var attrs: sample.Attributes = Map()
attrs += ('0 -> Set('abc,'nbv,'zxc))
attrs += ('1 -> Set('def,'ftr,'tyh))
attrs += ('2 -> Set('ghi,'azxc))
attrs += ('3 -> Set('jkl,'fds))
attrs += ('4 -> Set('mno,'nbh))
val examples: List[sample.Example] = List(
Map(
'0 -> 'abc,
'1 -> 'def,
'2 -> 'ghi,
'3 'jkl,
'4 -> 'mno
),
........................
)
// obviously we can't use the label as an attribute, that would be silly!
val label = 'play
println(sample.try(attrs,examples,label).getStr(0))
}
}
But How I change this code to - accepting input from a .csv file?
I suggest you use Java's io / nio standard library to read your CSV file. I think there is no relevant drawback in doing so.
But the first question we need to answer is where to read the file in the code? The parsed input seems to replace the value of examples. This fact also hints us what type the parsed CSV input must have, namely List[Map[Symbol, Symbol]]. So let us declare a new class
class InputFromCsvLoader(charset: Charset = Charset.defaultCharset()) {
def getInput(file: Path): List[Map[Symbol, Symbol]] = ???
}
Note that the Charset is only needed if we must distinguish between differently encoded CSV-files.
Okay, so how do we implement the method? It should do the following:
Create an appropriate input reader
Read all lines
Split each line at the comma-separator
Transform each substring into the symbol it represents
Build a map from from the list of symbols, using the attributes as key
Create and return the list of maps
Or expressed in code:
class InputFromCsvLoader(charset: Charset = Charset.defaultCharset()) {
val Attributes = List('outlook, 'temperature, 'humidity, 'wind, 'play)
val Separator = ","
/** Get the desired input from the CSV file. Does not perform any checks, i.e., there are no guarantees on what happens if the input is malformed. */
def getInput(file: Path): List[Map[Symbol, Symbol]] = {
val reader = Files.newBufferedReader(file, charset)
/* Read the whole file and discard the first line */
inputWithHeader(reader).tail
}
/** Reads all lines in the CSV file using [[java.io.BufferedReader]] There are many ways to do this and this is probably not the prettiest. */
private def inputWithHeader(reader: BufferedReader): List[Map[Symbol, Symbol]] = {
(JavaConversions.asScalaIterator(reader.lines().iterator()) foldLeft Nil.asInstanceOf[List[Map[Symbol, Symbol]]]){
(accumulator, nextLine) =>
parseLine(nextLine) :: accumulator
}.reverse
}
/** Parse an entry. Does not verify the input: If there are less attributes than columns or vice versa, zip creates a list of the size of the shorter list */
private def parseLine(line: String): Map[Symbol, Symbol] = (Attributes zip (line split Separator map parseSymbol)).toMap
/** Create a symbol from a String... we could also check whether the string represents a valid symbol */
private def parseSymbol(symbolAsString: String): Symbol = Symbol(symbolAsString)
}
Caveat: Expecting only valid input, we are certain that the individual symbol representations do not contain the comma-separation character. If this cannot be assumed, then the code as is would fail to split certain valid input strings.
To use this new code, we could change the main-method as follows:
def main(args: Array[String]){
val csvInputFile: Option[Path] = args.headOption map (p => Paths get p)
val examples = (csvInputFile map new InputFromCsvLoader().getInput).getOrElse(exampleInput)
// ... your code
Here, examples uses the value exampleInput, which is the current, hardcoded value of examples if no input argument is specified.
Important: In the code, all error handling has been omitted for convenience. In most cases, errors can occur when reading from files and user input must be considered invalid, so sadly, error handling at the boundaries of your program is usally not optional.
Side-notes:
Try not to use null in your code. Returning Option[T] is a better option than returning null, because it makes "nullness" explicit and provides static safety thanks to the type-system.
The return-keyword is not required in Scala, as the last value of a method is always returned. You can still use the keyword if you find the code more readable or if you want to break in the middle of your method (which is usually a bad idea).
Prefer val over var, because immutable values are much easier to understand than mutable values.
The code will fail with the provided CSV string, because it contains the symbols TRUE and FALSE which are not legal according to your programs logic (they should be true and false instead).
Add all information to your error-messages. Your error message only tells me what that a value for the attribute 'wind is bad, but it does not tell me what the actual value is.
Read a csv file ,
val datalines = Source.fromFile(filepath).getLines()
So this datalines contains all the lines from the csv file.
Next, convert each line into a Map[Int,String]
val datamap = datalines.map{ line =>
line.split(",").zipWithIndex.map{ case (word, idx) => idx -> word}.toMap
}
Here, we split each line with ",". Then construct a map with key as column number and value as each word after the split.
Next, If we want List[Map[Int,String]],
val datamap = datalines.map{ line =>
line.split(",").zipWithIndex.map{ case (word, idx) => idx -> word}.toMap
}.toList

InsertAll only new records with Slick

[PSQLException: ERROR: duplicate key value violates unique constraint
"dictionary_word_idx" Detail: Key (word)=(odirane) already exists.]
I have unique index preventing any duplications. I wonder how to InsertAll an Array with thousands elements but only the new ones? I'm using Slick 1.0.1 and Postgresql 9.1
Edit:
I'm trying the following:
def run = {
val source = scala.io.Source.fromFile("/home/user/dev/txt/test1.txt")
val lines = source.mkString
source.close()
val words = lines.split("[^\\p{Ll}]").distinct
database withTransaction {
val q = for {
w <- words.toList
row <- Dictionary if row.word != w
} yield w
Dictionary.autoInc.insertAll(q: _*)
}
words.length
}
but t dosent compile:
polymorphic expression cannot be instantiated to expected type;
[error] found : [G, T]scala.slick.lifted.Query[G,T]
[error] required: scala.collection.GenTraversableOnce[?] [error]
row <- Dictionary if row.word != w
Edit 2:
case class Word(id: Option[Long], word:String)
object Dictionary extends Table[Word]("dictionary") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def word = column[String]("word")
def * = id.? ~ word <> (Word, Word.unapply _)
def dictionary_word_idx = index("dictionary_word_idx", word, unique = true)
def autoInc = word returning id
}
Another alternative is to write raw SQL. Postgres doesn't have a default way to on duplicate ignore, but you can emulate it in a few different ways, shown here https://dba.stackexchange.com/questions/30499/optimal-way-to-ignore-duplicate-inserts
Combine that with http://slick.typesafe.com/doc/1.0.0-RC2/sql.html
Edit:
Here's an example
def insert(c: String) =
(Q.u + """INSERT INTO dictionary
(word)
SELECT""" +? c +
"""WHERE
NOT EXISTS (
SELECT word FROM dictionary WHERE word = """ +? c + ")"
).execute
val words = lines.split("[^\\p{Ll}]")
words.foreach(insert)
Is that what you mean by "at once"? I think that's going to be the most performant way of doing this without being crazy.
If it's too slow for you, there's another suggestion of creating a temporary table without the unique constraint, copy your current table into the temp table, insert the new words into the temp table, and then select distinct out of that table. That's shown here: https://stackoverflow.com/a/4070385/375874
But I think that's WAY overkill. Unless you have some crazy requirements or something.
Conceptually:
def insertAll[T](items: Seq[T]): Seq[Either[(T, Exception), (T, Int)]] = items.map { i =>
try {
// Perform an insert supposing returns and int representing the PK on the table
val pk = …
Right(i, pk)
} catch {
case e: Exception => Left(i, e)
}
}
You perform each insert operation and then, based on the result, you return a Left or Right object that keep tracks of the end result and give you a detailed context to interpret the operation.
EDIT
Let's suppose that your DAO object looks like:
object Dictionary extends Table[Word]("dictionary") {
// ...
}
where Word is your object model and moreover you have provided the nuts and bolts (as I can deduce from your pasted code) it should be (where words is a Seq[Word]):
words.map { w =>
try {
Right(w, Dictionary.autoInc.insert(w))
} catch {
case e: Exception => Left(w, e)
}
}
What you get is a sequence of Either that encapsulates the outcome for further processing.
Considerations
The solution provided by me attempts optimistically to perform the operation against the DB without requiring to pre-filter the list based on the state of the DB.
In general pre-filtering is problematic in an heavily multiuser application provided you can't assume that nobody added a word in your pre-filtered list after you've performed the filter.
State more simply: uniqueness constraint is a robust feature provided by DBMS which is better to exploit than to reinvent.
The solution you edited above is a no-solution because you still need to face possibly PK violation exception.

Play framework 2.0: Create validation error message using attribute value

Using scalaforms for the play framework, say that i have form such as:
case class User(name: String, emails: List[String])
val userForm = Form(
mapping(
"name" -> text,
"emails" -> list(text).verifying("Emails are duplicated",x => SomeFunctionThatHandlesDuplicateEmails(x))
)(User.apply, User.unapply)
)
Where SomeFunctionThatHandlesDuplicateEmails is a function that returns false (thus, making the field invalid) if any of the emails received in the form is already in database.
Now then, my question is:
Is there a way to use the value of the validated field to create the error message? I would like to tell the user which emails in particular were duplicated, not just tell them "Emails are duplicated" as shown above.
verifying() takes a series of Constraint[T].
You can see examples of Constraints implemented here.
Note that the validation function in each receives the value to be validated e.g. "o" in the "min" constraint repeated below:
def min(minValue: Int): Constraint[Int] = Constraint[Int]("constraint.min", minValue) { o =>
if (o >= minValue) Valid else Invalid(ValidationError("error.min", minValue))
}
This could just as easily be:
def min(minValue: Int): Constraint[Int] = Constraint[Int]("constraint.min", minValue) { o =>
if (o >= minValue) Valid else Invalid(ValidationError("error.min", minValue, o))
}
which would make "o" available to the error message formatter as {1} (minValue is {0}).