I am having trouble using a left outer join in Slick. I'll start with some code :
val articles = (for {
(article, lecture) <- ArticleDAO leftJoin LectureDAO on (_.id === _.idArticle) if (article.flux === idFlux)
} yield (article, lecture.isStarred.?)).groupBy(_._1.guid).map {
case (guid, rows) => rows.first
}
PS : The ArticleDAO & LectureDAO are the objects extending Table in opposition to Article & Lecture which are simple case classes.
This is the error, I am getting when compiling the part above :
Don't know how to unpack (models.Article, Option[Boolean]) to T and pack to G
I don't really understand this error. I know it has something to do with transformation, composition of queries but I have no idea how to change/fix it. Could some shed some light on this?
The fix is explained here: https://groups.google.com/forum/#!topic/scalaquery/bIFH6be99B0 . .first is not a query operation at the moment, use .min instead.
Related
I am refactoring a scala http4s application to remove some pesky side effects causing my app to block. I'm replacing .unsafeRunSync with cats.effect.IO. The problem is as follows:
I have 2 lists: alreadyAccessible: IO[List[Page]] and pages: List[Page]
I need to filter out the pages that are not contained in alreadyAccessible.
Then map over the resulting list to "grant Access" in the database to these pages. (e.g. call another method that hits the database and returns an IO[Page].
val addable: List[Page] = pages.filter(p => !alreadyAccessible.contains(p))
val added: List[Page] = addable.map((p: Page) => {
pageModel.grantAccess(roleLst.head.id, p.id) match {
case Right(p) => p
}
})
This is close to what I want; However, it does not work because filter requires a function that returns a Boolean but alreadyAccessible is of type IO[List[Page]] which precludes you from removing anything from the IO monad. I understand you can't remove data from the IO so maybe transform it:
val added: List[IO[Page]] = for(page <- pages) {
val granted = alreadyAccessible.flatMap((aa: List[Page]) => {
if (!aa.contains(page))
pageModel.grantAccess(roleLst.head.id, page.id) match { case Right(p) => p }
else null
})
} yield granted
this unfortunately does not work with the following error:
Error:(62, 7) ';' expected but 'yield' found.
} yield granted
I think because I am somehow mistreating the for comprehension syntax, I just don't understand why I cannot do what I'm doing.
I know there must be a straight forward solution to such a problem, so any input or advice is greatly appreciates. Thank you for your time in reading this!
granted is going to be an IO[List[Page]]. There's no particular point in having IO inside anything else unless you truly are going to treat the actions like values and reorder them/filter them etc.
val granted: IO[List[Page]] = for {
How do you compute it? Well, the first step is to execute alreadyAccessible to get the actual list. In fact, alreadyAccessible is misnamed. It is not the list of accessible pages; it is an action that gets the list of accessible pages. I would recommend you rename it getAlreadyAccessible.
alreadyAccessible <- getAlreadyAccessible
Then you filter pages with it
val required = pages.filterNot(alreadyAccessible.contains)
Now, I cannot decipher what you're doing to these pages. I'm just going to assume you have some kind of function grantAccess: Page => IO[Page]. If you map this function over required, you will get a List[IO[Page]], which is not desirable. Instead, we should traverse with grantAccess, which will produce a IO[List[Page]] that executes each IO[Page] and then assembles all the results into a List[Page].
granted <- required.traverse(grantAccess)
And we're done
} yield granted
I have spark streaming job. I am using Cassandra as datastore.
I have stream which is need to be joined with cassandra table.
I am using spark-cassandra-connector, there is great method joinWithCassandraTable which is as far as I can understand implementing inner join with cassandra table
val source: DStream[...] = ...
source.foreachRDD { rdd =>
rdd.joinWithCassandraTable( "keyspace", "table" ).map{ ...
}
}
So the question is how can I implement left outer join with cassandra table?
Thanks in advance
This is currently not supported, but there is a ticket to introduce the functionality. Please vote on it if you would like it introduced in the future.
https://datastax-oss.atlassian.net/browse/SPARKC-181
A workaround is suggested in the ticket
As RussS mentioned such feature is not available in spark-cassandra-connector driver yet. So as workaround I propose the following code snippet.
rdd.foreachPartition { partition =>
CassandraConnector(rdd.context.getConf).withSessionDo { session =>
for (
leftSide <- partition;
rightSide <- {
val rs = session.execute(s"""SELECT * FROM "keyspace".table where id = "${leftSide._2}"""")
val iterator = new PrefetchingResultSetIterator(rs, 100)
if (iterator.isEmpty) Seq(None)
else iterator.map(r => Some(r.getString(1)))
}
) yield (leftSide, rightSide)
}
}
I'm refactoring some scala code to teach my coworkers about for-comprehensions, and I've got a line like:
for {
// ...
result <- components.collectFirst({ case section if section.startsWith(DESIRED_SUBSTRING) => section.substring(section.indexOf(DELIM) + 1).trim() == "true" })
} yield result
That's a bit long.
At first, I wished I could just skip the result <- ... followed by the immediate yield, as I can in Haskell, but then I noticed the processing going on inside collectFirst.
So I thought it'd be much easier to read as I should better do this as
for {
// ...
section <- components.filter(_.startsWith(DESIRED_SUBSTRING)).headOption
} yield section.substring(section.indexOf(DELIM) + 1).trim() == "true"
Which works, but it is less efficient, since filter has to process all the elements. I'd like to be able to use a lazy filter:
components.withFilter(_.startsWith(DESIRED_SUBSTRING)).headOption
But FilterMonadic doesn't seem to support headOption, and I can't figure out a way to derive it from the operations it does support. I'm sure there's a way with flatMap and some bf, but I'm too unfamiliar with the scala ecosystem at the moment.
If I want to stick with standard library tricks, am I stuck with
for {
// ...
section <- components.collectFirst({ case section if section.startsWith(DESIRED_SUBSTRING) => section })
} yield section.substring(section.indexOf(DELIM) + 1).trim() == "true"
Or is there something better I can use?
If you use components.find(_.startsWith(DESIRED_SUBSTRING)) that will give you an Option with the first element that meets the condition. Then, you can just map over it with any subsequent processing you need.
for (String stock : allStocks) {
Quote quote = getQuote(...);
if (null == quoteLast) {
continue;
}
Price price = quote.getPrice();
if (null == price) {
continue;
}
}
I don't necessarily need a line by line translation, but I'm looking for the "Scala way" to handle this type of problem.
You don't need continue or breakable or anything like that in cases like this: Options and for comprehensions do the trick very nicely,
val stocksWithPrices =
for {
stock <- allStocks
quote <- Option(getQuote(...))
price <- Option(quote.getPrice())
} yield (stock, quote, price);
Generally you try to avoid those situations to begin with by filtering before you even start:
val goodStocks = allStocks.view.
map(stock => (stock, stock.getQuote)).filter(_._2 != null).
map { case (stock, quote) => (stock,quote, quote.getPrice) }.filter(_._3 != null)
(this example showing how you'd carry along partial results if you need them). I've used a view so that results will be computed as-needed, instead of creating a bunch of new collections at each step.
Actually, you'd probably have the quotes and such return options--look around on StackOverflow for examples of how to use those instead of null return values.
But, anyway, if that sort of thing doesn't work so well (e.g. because you are generating too many intermediate results that you need to keep, or you are relying on updating mutable variables and you want to keep the evaluation pattern simple so you know what's happening when) and you can't conceive of the problem in a different, possibly more robust way, then you can
import scala.util.control.Breaks._
for (stock <- allStocks) {
breakable {
val quote = getQuote(...)
if (quoteLast eq null) break;
...
}
}
The breakable construct specifies where breaks should take you to. If you put breakable outside a for loop, it works like a standard Java-style break. If you put it inside, it acts like continue.
Of course, if you have a very small number of conditions, you don't need the continue at all; just use the else of the if-statement.
Your control structure here can be mapped very idiomatically into the following for loop, and your code demonstrates the kind of filtering that Scala's for loop was designed for.
for {stock <- allStocks.view
quote = getQuote(...)
if quoteLast != null
price = quote.getPrice
if null != price
}{
// whatever comes after all of the null tests
}
By the way, Scala will automatically desugar this into the code from Rex Kerr's solution
val goodStocks = allStocks.view.
map(stock => (stock, stock.getQuote)).filter(_._2 != null).
map { case (stock, quote) => (stock,quote, quote.getPrice) }.filter(_._3 != null)
This solution probably doesn't work in general for all different kinds of more complex flows that might use continue, but it does address a lot of common ones.
If the focus is really on the continue and not on the null handling, just define an inner method (the null handling part is a different idiom in scala):
def handleStock(stock: String): Unit {
val quote = getQuote(...)
if (null == quoteLast) {
return
}
val price = quote.getPrice();
if (null == price) {
return
}
}
for (stock <- allStocks) {
handleStock(stock)
}
The simplest way is to embed the skipped-over code in an if with reversed-sense to what you have.
See http://www.scala-lang.org/node/257
I'll give some C-style "bracket" pseudo-code to show what I'd like to express in another way:
for (int i = 0; i < n; i++) {
if (i == 3 || i == 5 || i == 982) {
assertTrue( isCromulent(i) );
} else {
assertFalse( isCromulent(i) );
}
}
The for loop is not very important, that is not the point of my question: I'd like to know how I could rewrite what is inside the loop using Scala.
My goal is not to have the shortest code possible: it's because I'd like to understand what kind of manipulation can be done on method names (?) in Scala.
Can you do something like the following in Scala (following is still some kind of pseudo-code, not Scala code):
assert((i==3 || i==5 || i==982)?True:False)(isCromulent(i))
Or even something like this:
assertTrue( ((i==3 || i==5 || i==982) ? : ! ) isCromulent(i) )
Basically I'd like to know if the result of the test (i==3 || i==5 || i==982) can be used to dispatch between two methods or to add a "not" before an expression.
I don't know if it makes sense so please be kind (look my profile) :)
While pelotom's solution is much better for this case, you can also do this (which is a bit closer to what you asked originally):
(if (i==3||i==5||i==982) assertTrue else assertFalse)(isCromulent(i))
Constructing names dynamically can be done via reflection, but this certainly won't be concise.
assertTrue(isCromulent(i) == (i==3||i==5||i==982))
Within the Scala type system, it isn't possible to dynamically create a method name based on a condition.
But it isn't at all necessary in this case.
val condition = i == 3 || i == 5 || i == 982
assertEquals(condition, isCromulent(i))
I hope nobody minds this response, which is an aside rather than a direct answer.
I found the question and the answers so far very interesting and spent a while looking for a pattern matching based alternative.
The following is an attempt to generalise on this (very specific) category of testing:
class MatchSet(s: Set[Int]) {def unapply(i: Int) = s.contains(i)}
object MatchSet {def apply(s: Int*) = new MatchSet(Set(s:_*))}
val cromulentSet = MatchSet(3, 5, 982)
0 until n foreach {
case i # cromulentSet() => assertTrue(isCromulent(i))
case i => assertFalse(isCromulent(i))
}
The idea is to create ranges of values contained in MatchSet instances rather than use explicit matches.