I am trying to use .UnionAll method in go-pg library for golang.
var model []Customer
q0 := db.Model(&model).Where("name = ?", name0).Limit(4)
q1 := db.Model(&model).Where("name = ?", name1).Limit(3)
var result []Customer
if err := q0.UnionAll(q1).Limit(1).Select(&result); !as.NoError(err) {
return
}
This code produces the query:
(SELECT "customer"."id", "customer"."name" FROM customers AS "customer" WHERE (name = 'customer 1deificatory zonoid reprepare alacrify serenissime')
LIMIT 1)
UNION ALL
(SELECT "customer"."id", "customer"."name" FROM customers AS "customer" WHERE (name = 'customer 2fordless ferroboron radiability dandizette smutch'
LIMIT 3)
)
But I expect it to be:
(SELECT "customer"."id", "customer"."name" FROM customers AS "customer" WHERE (name = 'customer 1deificatory zonoid reprepare alacrify serenissime')
LIMIT 4)
UNION ALL
(SELECT "customer"."id", "customer"."name" FROM customers AS "customer" WHERE (name = 'customer 2fordless ferroboron radiability dandizette smutch')
LIMIT 3)
LIMIT 1
So how to use go-pg to get raw SQL query which I expect?
The thing is I can't apply the Limit 1 expression to the whole query for some reason.
Also, my Limit 4 is not applied correctly to the first union all member.
My whole code is here.
I believe this query does the trick for you.
if err := db.Model().With("union_q", q0.UnionAll(q1)).Table("union_q").Limit(1).Select(&result); !as.NoError(err) {
return
}
The query isn't exactly what you wanted to be and as far as I know, the current version of go-pg can't generate that query. But this query does exactly what you wanted.
I need to get value from previous or next row while Im iterating through RDD[Row]
(10,1,string1)
(11,1,string2)
(21,1,string3)
(22,1,string4)
I need to sum strings for rows where difference between 1st value is not higher than 3. 2nd value is ID. So the result should be:
(1, string1string2)
(1, string3string4)
I tried use groupBy, reduce, partitioning but still I can't achieve what I want.
I'm trying to make something like this(I know it's not proper way):
rows.groupBy(row => {
row(1)
}).map(rowList => {
rowList.reduce((acc, next) => {
diff = next(0) - acc(0)
if(diff <= 3){
val strings = acc(2) + next(2)
(acc(1), strings)
}else{
//create new group to aggregatre strings
(acc(1), acc(2))
}
})
})
I wonder if my idea is proper to solve this problem.
Looking for help!
I think you can use sqlContext to Solve your problem by using lag function
Create RDD:
val rdd = sc.parallelize(List(
(10, 1, "string1"),
(11, 1, "string2"),
(21, 1, "string3"),
(22, 1, "string4"))
)
Create DataFrame:
val df = rdd.map(rec => (rec._1.toInt, rec._2.toInt, rec._3.toInt)).toDF("a", "b", "c")
Register your Dataframe:
df.registerTempTable("df")
Query the result:
val res = sqlContext.sql("""
SELECT CASE WHEN l < 3 THEN ROW_NUMBER() OVER (ORDER BY b) - 1
ELSE ROW_NUMBER() OVER (ORDER BY b)
END m, b, c
FROM (
SELECT b,
(a - CASE WHEN lag(a, 1) OVER (ORDER BY a) is not null
THEN lag(a, 1) OVER (ORDER BY a)
ELSE 0
END) l, c
FROM df) A
""")
Show the Results:
res.show
I Hope this will Help.
I am currently facing a weird problem.
Whenever a user types something into the search bar that starts with an 's', the request crashes.
What you see next is a sample sql code generated by the search engine I programmed for this project.
SELECT Profiles.ProfileID,Profiles.Nickname,Profiles.Email,Profiles.Status,Profiles.Role,Profiles.Credits, Profiles.Language,Profiles.Created,Profiles.Modified,Profiles.Cover,Profiles.Prename, Profiles.Lastname,Profiles.BirthDate,Profiles.Country,Profiles.City,Profiles.Phone,Profiles.Website, Profiles.Description, Profiles.Affair,Scores.AvgScore, coalesce(Scores.NumScore, 0) AS NumScore, coalesce(Scores.NumScorer, 0) AS NumScorer, (
(SELECT count(*)
FROM Likes
JOIN Comments using(CommentID)
WHERE Comments.ProfileID = Profiles.ProfileID)) NumLikes, (
(SELECT count(*)
FROM Likes
JOIN Comments using(CommentID)
WHERE Comments.ProfileID = Profiles.ProfileID) /
(SELECT coalesce(nullif(count(*), 0), 1)
FROM Comments
WHERE Comments.ProfileID = Profiles.ProfileID)) AvgLikes, Movies.MovieID, Movies.Caption, Movies.Description, Movies.Language, Movies.Country, Movies.City, Movies.Kind, Movies.Integration,
(SELECT cast(least(25 + 5.000000 * round((75 * ((0.500000 * SIZE/1024.0/1024.0 * 0.001250) + (0.500000 * Duration/60.0 * 0.050000))) / 5.000000), 100) AS signed int)
FROM Streams
WHERE MovieID = Movies.MovieID
AND Tag = "main"
AND ENCODING = "mp4") AS ChargeMain,
(SELECT cast(least(25 + 10.000000 * round((75 * ((0.200000 * SIZE/1024.0/1024.0 * 0.001000) + (0.800000 * Duration/60.0 * 0.016667))) / 10.000000), 100) AS signed int)
FROM Streams
WHERE MovieID = Movies.MovieID
AND Tag = "notes"
AND ENCODING = "mp4") AS ChargeNotes,
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "main") AS MainViews,
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "notes") AS NotesViews,
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "trailer") AS TrailerViews,
(SELECT coalesce(greatest(
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "trailer"),
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "main")), 0)) AS MaxMainTrailerViews,
(SELECT avg(Score)
FROM Scores
WHERE Scores.MovieID = Movies.MovieID) AS Score,
(SELECT coalesce(group_concat(cast(Score AS signed int)), "")
FROM Scores
WHERE Scores.MovieID = Movies.MovieID) AS Scores, Movies.Cover, Movies.Locked, Movies.Created, Movies.Modified,
(SELECT coalesce(group_concat(name separator ','),"")
FROM Tags
JOIN TagLinks using(TagID)
WHERE TagLinks.MovieID = Movies.MovieID
ORDER BY name ASC) AS Tags,
(SELECT count(*)
FROM Purchases
WHERE MovieID = Movies.MovieID
AND ProfileID = %s
AND TYPE = "main") AS PurchasedMain,
(SELECT count(*)
FROM Purchases
WHERE MovieID = Movies.MovieID
AND ProfileID = %s
AND TYPE = "notes") AS PurchasedNotes,
(SELECT count(*)
FROM Watchlist
WHERE MovieID = Movies.MovieID
AND ProfileID = %s) AS Watchlist,
(SELECT count(*)
FROM Scores
WHERE MovieID = Movies.MovieID
AND ProfileID = %s) AS Rated,
(SELECT count(*)
FROM Comments
WHERE MovieID = Movies.MovieID
AND Deleted IS NULL) AS Comments,
(SELECT sum(Duration)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Tag IN ("main",
"notes")
AND Streams.ENCODING = "mp4") AS Runtime,
(SELECT cast(count(*) AS signed int)
FROM Movies
JOIN Profiles ON Profiles.ProfileID = Movies.ProfileID
WHERE ((Movies.Locked = 0
AND
(SELECT count(*)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Status <> "ready") = 0
AND Profiles.Status = "active")
OR (%s = 1)
OR (Movies.ProfileID = %s))
AS Movies,
(SELECT cast(ceil(count(*) / %s) AS signed int)
FROM Movies
JOIN Profiles using(ProfileID)
WHERE ((Movies.Locked = 0
AND
(SELECT count(*)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Status <> "ready") = 0
AND Profiles.Status = "active")
OR (%s = 1)
OR (Movies.ProfileID = %s))
AS Pages
FROM Movies
JOIN Profiles using(ProfileID)
LEFT JOIN
(SELECT Movies.ProfileID AS ProfileID,
avg(Scores.Score) AS AvgScore,
count(*) AS NumScore,
count(DISTINCT Scores.ProfileID) AS NumScorer
FROM Scores
JOIN Movies using(MovieID)
GROUP BY Movies.ProfileID) AS Scores using(ProfileID)
WHERE ((Movies.Locked = 0
AND
(SELECT count(*)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Status <> "ready") = 0
AND Profiles.Status = "active")
OR (%s = 1)
OR (Movies.ProfileID = %s))
ORDER BY Score DESC LIMIT %s,
%s
After countless hours of investigating and comparing possible user inputs with the generated sql code I finally nailed the problem down to some really strange behaviour of the JDBC driver which I consider a serious bug - yet I am not sure:
I spent another few hours trying to reproduce the problem with as less sql code as possible and ended up with the following:
SQL("""select * from Movies where "s" like "%s" and MovieID = {a} """)
.on('a -> 1).as(scalar[Long]*)
[SQLException: Parameter index out of range (1 > number of parameters, which is 0).]
SQL("""select * from Movies where "s" like "%samuel" and MovieID = {a} """)
.on('a -> 1).as(scalar[Long]*)
[SQLException: Parameter index out of range (1 > number of parameters, which is 0).]
SQL("""select * from Movies where "s" like "%flower" and MovieID = {a} """)
.on('a -> 1).as(scalar[Long]*)
[OK]
SQL("""select * from Movies where "s" like "%samuel" and MovieID = 1 """)
.on('a -> 1).as(scalar[Long]*)
[OK]
SQL("""select * from Movies where "s" like "%s" and MovieID = "{a}" """)
.on('a -> 1).as(scalar[Long]*)
[OK]
SQL("""select * from Movies where MovieID = {a} and "s" like "%s" """)
.on('a -> 1).as(scalar[Long]*)
[OK]
I believe to see a pattern here:
Under the exact condition that there is a %s sequence (quoted or unquoted) anywhere in a sql code, followed by a non quoted named parameter with arbitrary name and arbitrary distance
to the %s sequence, jdbc (or anorm) crashes. The crash seems to occur in JDBC, however its also possible that Anorm submits invalid values to JDBC.
Do you guys have any suggestions?
I think I found an enduring solution for the problem meanwhile. Since my sql generator needs to stay very flexible I somehow need a way to pass along sql fragments with their corresponding parameters without evaluating them right away. Instead the generator must be able to assemble and compose various sql fragments into bigger fragments at any time - just as he does now - but now with the acompanying, not yet evaluated parameters. I came up with this prototype:
DB.withConnection("betterdating") { implicit connection =>
case class SqlFragment(Fragment: String, Args: NamedParameter*)
val aa = SqlFragment("select MovieID from Movies")
val bb = SqlFragment("join Profiles using(ProfileID)")
val cc = SqlFragment("where Caption like \"%{a}\" and MovieID = {b}", 'a -> "s", 'b -> 5)
// combine all fragments
val v1 = SQL(Seq(aa, bb, cc).map(_.Fragment).mkString(" "))
.on((aa.Args ++ bb.Args ++ cc.Args): _*)
// better solution
val v2 = Seq(aa, bb, cc).unzip(frag => (frag.Fragment, frag.Args)) match {
case (frags, args) => SQL(frags.mkString(" ")).on(args.flatten: _*)
}
// works
println(v1.as(scalar[Long].singleOpt))
println(v2.as(scalar[Long].singleOpt))
}
It seems to work great! :-)
I then rewrote the last part of the freetext filter as follow:
// finally transform the expression
// list a single sql fragment
expressions.zipWithIndex.map { case (expr, index) =>
s"""
(concat(Movies.Caption, " ", Movies.Description, " ", Movies.Kind, " ", Profiles.Nickname, " ",
(select coalesce(group_concat(Tags.Name), "") from Tags join TagLinks using (TagID)
where TagLinks.MovieID = Movies.MovieID)) like "%{expr$index}%"))
""" -> (s"expr$index" -> expr)
}.unzip match { case (frags, args) => SqlFragment(frags.mkString(" and "), args.flatten: _*)
What do you think?
This is how it is being implemented right now:
/**
* This private helper method transforms a content filter string into an sql expression
* for searching within movies, owners and kinds and tags.
* #author Samuel Lörtscher
*/
private def contentFilterToSql(value: String) = {
// trim and clean and the parametric value from any possible anomalies
// (those include strange spacing and non closed quotes)
val cleaned = value.trim match {
case trimmed if trimmed.count(_ == '"') % 2 != 0 =>
if (trimmed.last == '"') trimmed.dropRight(1).trim
else trimmed + '"'
case trimmed =>
trimmed
};
// transform the cleaned value into a list of expressions
// (words between quotes are considered being one expression)
// empty expressions between quotes are being removed
// expressions will contain no quotes as they are being stripped during evaluation -
// thus counter measures for sql injection should be obsolete
// (we put an empty space at the end because it makes the lexer algorithm much
// more efficient as it will not need to check for end of file in every iteration)
val expressions = (cleaned + " ").foldLeft((List[String](), "", false)) { case ((list, expr, quoted), char) =>
// perform the lexer operation for the current character
if (char == ' ' && !quoted) (expr :: list, "", false)
else if (char == '"') (expr :: list, "", !quoted)
else (list, expr + char, quoted)
}._1.filter(_.nonEmpty).map(_.trim)
// finally transform the expression
// list into a variable length sql condition statement
expressions.map { expr =>
s"""
(concat(Movies.Caption, " ", Movies.Description, " ", Movies.Kind, " ", Profiles.Nickname, " ",
(select coalesce(group_concat(Tags.Name), "")
from Tags join TagLinks using (TagID) where TagLinks.MovieID = Movies.MovieID)) like "%$expr%")
"""
}.mkString(" and ")
}
Since the number of search expressions is variable, I cannot use Anorm arguments here. :-/
I found a simple solution now, but I am not exactly happy being forced to apply such crappy hacks.
Since putting a %s character sequence seems to trigger the bug, I was looking for possibilities to submit the same semantical outcome without directly passing this character sequence. I finally ended up replacing like "%$expr%" by like concat("%", "$expr%"). Since concat is being evaluated by the MySql Server engine BEFORE "like", he will put the original pattern back together before processing it by "like" - and without the sequence %s ever being transmitted through the anorm, jdbc data processors.
// finally transform the expression
// list into a variable length sql condition statement
// (freaking concat("%", "$expr%")) is required due to a freaking bug in either anorm or JDBC
// which results into a crash when %s is anyway submitted)
expressions.map { expr =>
s"""
(concat(Movies.Caption, " ", Movies.Description, " ", Movies.Kind, " ", Profiles.Nickname, " ",
(select coalesce(group_concat(Tags.Name), "")
from Tags join TagLinks using (TagID) where TagLinks.MovieID = Movies.MovieID)) like concat("%", "$expr%"))
"""
}.mkString(" and ")
I'm trying to build a list of DataTables based on DataRelations in a DataSet, where the tables returned are only included by their relationships with each other, knowing each end of the chain in advance. Such that my DataSet has 7 Tables. The relationships look like this:
Table1 -> Table2 -> Table3 -> Table4 -> Table5
-> Table6 -> Table7
So given Table1 and Table7, I want to return Tables 1, 2, 3, 6, 7
My code so far traverses all the relations and returns all Tables so in the example it returns Table4 and Table5 as well. I've passed in the first, last as arguments, and know that I'm not yet using the last one yet, I am still trying to think how to go about it, and is where I need the help.
type DataItem =
| T of DataTable
| R of DataRelation list
let GetRelatedTables (first, last) =
let rec flat_rec dt acc =
match dt with
| T(dt) ->
let rels = [ for r in dt.ParentRelations do yield r ]
dt :: flat_rec(R(rels)) acc
| R(h::t) ->
flat_rec(R(t)) acc # flat_rec(T(h.ParentTable)) acc
| R([]) -> []
flat_rec first []
I think something like this would do it (although I haven't tested it). It returns DataTable list option because, in theory, a path between two tables might not exist.
let findPathBetweenTables (table1 : DataTable) (table2 : DataTable) =
let visited = System.Collections.Generic.HashSet() //check for circular references
let rec search path =
let table = List.head path
if not (visited.Add(table)) then None
elif table = table2 then Some(List.rev path)
else
table.ChildRelations
|> Seq.cast<DataRelation>
|> Seq.tryPick (fun rel -> search (rel.ChildTable::path))
search [table1]
Here it is my Sql query:
Update Accounting.ACNT.Book
Set [No] = Case When A.IsAccType10 = 1 Then [No] - 5000000000 Else [No] + 4000000000 End,
FullNo = Stuff(Stuff(FullNo, Len(FullNo) - 10, 2, '05'), 1, Len(FullNo) - 12, '????')
From #Temp A,
[Master].dbo.ParentIDMap B
Where A.BookID = Book.ID And
B.AccType = Cast(SubString(Book.OriginalAccNo, 2, 1) as Int) - 100 And
B.BranchID = Book.BranchID
Can I use a select statement instead of '?????' as last parameter of Stuff function?
The query I need to replace with is:
Select ParentFullNo
From [Master].dbo.ParentIDMap A
Where A.BranchID = Book.BranchID
Yes, just put the query in parentheses ().