I want to get all the tables names from a sql query in Spark using Scala.
Lets say user sends a SQL query which looks like:
select * from table_1 as a left join table_2 as b on a.id=b.id
I would like to get all tables list like table_1 and table_2.
Is regex the only option ?
Thanks a lot #Swapnil Chougule for the answer. That inspired me to offer an idiomatic way of collecting all the tables in a structured query.
scala> spark.version
res0: String = 2.3.1
def getTables(query: String): Seq[String] = {
val logicalPlan = spark.sessionState.sqlParser.parsePlan(query)
import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
logicalPlan.collect { case r: UnresolvedRelation => r.tableName }
}
val query = "select * from table_1 as a left join table_2 as b on a.id=b.id"
scala> getTables(query).foreach(println)
table_1
table_2
Hope it will help you
Parse the given query using spark sql parser (spark internally does same). You can get sqlParser from session's state. It will give Logical plan of query. Iterate over logical plan of query & check whether it is instance of UnresolvedRelation (leaf logical operator to represent a table reference in a logical query plan that has yet to be resolved) & get table from it.
def getTables(query: String) : Seq[String] ={
val logical : LogicalPlan = localsparkSession.sessionState.sqlParser.parsePlan(query)
val tables = scala.collection.mutable.LinkedHashSet.empty[String]
var i = 0
while (true) {
if (logical(i) == null) {
return tables.toSeq
} else if (logical(i).isInstanceOf[UnresolvedRelation]) {
val tableIdentifier = logical(i).asInstanceOf[UnresolvedRelation].tableIdentifier
tables += tableIdentifier.unquotedString.toLowerCase
}
i = i + 1
}
tables.toSeq
}
I had some complicated sql queries with nested queries and iterated on #Jacek Laskowski's answer to get this
def getTables(spark: SparkSession, query: String): Seq[String] = {
val logicalPlan = spark.sessionState.sqlParser.parsePlan(query)
var tables = new ListBuffer[String]()
var i: Int = 0
while (logicalPlan(i) != null) {
logicalPlan(i) match {
case t: UnresolvedRelation => tables += t.tableName
case _ =>
}
i += 1
}
tables.toList
}
def __sqlparse2table(self, query):
'''
#description: get table name from table
'''
plan = self.spark._jsparkSession.sessionState().sqlParser().parsePlan(query)
plan_string = plan.toString().replace('`.`', '.')
unr = re.findall(r"UnresolvedRelation `(.*?)`", plan_string)
cte = re.findall(r"CTE \[(.*?)\]", plan.toString())
cte = [tt.strip() for tt in cte[0].split(',')] if cte else cte
schema = set()
tables = set()
for table_name in unr:
if table_name not in cte:
schema.update([table_name.split('.')[0]])
tables.update([table_name])
return schema, tables
Since you need to list all the columns names listed in table1 and table2, what you can do is to show tables in db.table_name in your hive db.
val tbl_column1 = sqlContext.sql("show tables in table1");
val tbl_column2 = sqlContext.sql("show tables in table2");
You will get list of columns in both the table.
tbl_column1.show
name
id
data
unix did the trick, grep 'INTO\|FROM\|JOIN' .sql | sed -r 's/.?(FROM|INTO|JOIN)\s?([^ ])./\2/g' | sort -u
grep 'overwrite table' .txt | sed -r 's/.?(overwrite table)\s?([^ ])./\2/g' | sort -u
I am currently facing a weird problem.
Whenever a user types something into the search bar that starts with an 's', the request crashes.
What you see next is a sample sql code generated by the search engine I programmed for this project.
SELECT Profiles.ProfileID,Profiles.Nickname,Profiles.Email,Profiles.Status,Profiles.Role,Profiles.Credits, Profiles.Language,Profiles.Created,Profiles.Modified,Profiles.Cover,Profiles.Prename, Profiles.Lastname,Profiles.BirthDate,Profiles.Country,Profiles.City,Profiles.Phone,Profiles.Website, Profiles.Description, Profiles.Affair,Scores.AvgScore, coalesce(Scores.NumScore, 0) AS NumScore, coalesce(Scores.NumScorer, 0) AS NumScorer, (
(SELECT count(*)
FROM Likes
JOIN Comments using(CommentID)
WHERE Comments.ProfileID = Profiles.ProfileID)) NumLikes, (
(SELECT count(*)
FROM Likes
JOIN Comments using(CommentID)
WHERE Comments.ProfileID = Profiles.ProfileID) /
(SELECT coalesce(nullif(count(*), 0), 1)
FROM Comments
WHERE Comments.ProfileID = Profiles.ProfileID)) AvgLikes, Movies.MovieID, Movies.Caption, Movies.Description, Movies.Language, Movies.Country, Movies.City, Movies.Kind, Movies.Integration,
(SELECT cast(least(25 + 5.000000 * round((75 * ((0.500000 * SIZE/1024.0/1024.0 * 0.001250) + (0.500000 * Duration/60.0 * 0.050000))) / 5.000000), 100) AS signed int)
FROM Streams
WHERE MovieID = Movies.MovieID
AND Tag = "main"
AND ENCODING = "mp4") AS ChargeMain,
(SELECT cast(least(25 + 10.000000 * round((75 * ((0.200000 * SIZE/1024.0/1024.0 * 0.001000) + (0.800000 * Duration/60.0 * 0.016667))) / 10.000000), 100) AS signed int)
FROM Streams
WHERE MovieID = Movies.MovieID
AND Tag = "notes"
AND ENCODING = "mp4") AS ChargeNotes,
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "main") AS MainViews,
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "notes") AS NotesViews,
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "trailer") AS TrailerViews,
(SELECT coalesce(greatest(
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "trailer"),
(SELECT coalesce(count(*), 0)
FROM Views
WHERE Views.MovieID = Movies.MovieID
AND Tag = "main")), 0)) AS MaxMainTrailerViews,
(SELECT avg(Score)
FROM Scores
WHERE Scores.MovieID = Movies.MovieID) AS Score,
(SELECT coalesce(group_concat(cast(Score AS signed int)), "")
FROM Scores
WHERE Scores.MovieID = Movies.MovieID) AS Scores, Movies.Cover, Movies.Locked, Movies.Created, Movies.Modified,
(SELECT coalesce(group_concat(name separator ','),"")
FROM Tags
JOIN TagLinks using(TagID)
WHERE TagLinks.MovieID = Movies.MovieID
ORDER BY name ASC) AS Tags,
(SELECT count(*)
FROM Purchases
WHERE MovieID = Movies.MovieID
AND ProfileID = %s
AND TYPE = "main") AS PurchasedMain,
(SELECT count(*)
FROM Purchases
WHERE MovieID = Movies.MovieID
AND ProfileID = %s
AND TYPE = "notes") AS PurchasedNotes,
(SELECT count(*)
FROM Watchlist
WHERE MovieID = Movies.MovieID
AND ProfileID = %s) AS Watchlist,
(SELECT count(*)
FROM Scores
WHERE MovieID = Movies.MovieID
AND ProfileID = %s) AS Rated,
(SELECT count(*)
FROM Comments
WHERE MovieID = Movies.MovieID
AND Deleted IS NULL) AS Comments,
(SELECT sum(Duration)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Tag IN ("main",
"notes")
AND Streams.ENCODING = "mp4") AS Runtime,
(SELECT cast(count(*) AS signed int)
FROM Movies
JOIN Profiles ON Profiles.ProfileID = Movies.ProfileID
WHERE ((Movies.Locked = 0
AND
(SELECT count(*)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Status <> "ready") = 0
AND Profiles.Status = "active")
OR (%s = 1)
OR (Movies.ProfileID = %s))
AS Movies,
(SELECT cast(ceil(count(*) / %s) AS signed int)
FROM Movies
JOIN Profiles using(ProfileID)
WHERE ((Movies.Locked = 0
AND
(SELECT count(*)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Status <> "ready") = 0
AND Profiles.Status = "active")
OR (%s = 1)
OR (Movies.ProfileID = %s))
AS Pages
FROM Movies
JOIN Profiles using(ProfileID)
LEFT JOIN
(SELECT Movies.ProfileID AS ProfileID,
avg(Scores.Score) AS AvgScore,
count(*) AS NumScore,
count(DISTINCT Scores.ProfileID) AS NumScorer
FROM Scores
JOIN Movies using(MovieID)
GROUP BY Movies.ProfileID) AS Scores using(ProfileID)
WHERE ((Movies.Locked = 0
AND
(SELECT count(*)
FROM Streams
WHERE Streams.MovieID = Movies.MovieID
AND Streams.Status <> "ready") = 0
AND Profiles.Status = "active")
OR (%s = 1)
OR (Movies.ProfileID = %s))
ORDER BY Score DESC LIMIT %s,
%s
After countless hours of investigating and comparing possible user inputs with the generated sql code I finally nailed the problem down to some really strange behaviour of the JDBC driver which I consider a serious bug - yet I am not sure:
I spent another few hours trying to reproduce the problem with as less sql code as possible and ended up with the following:
SQL("""select * from Movies where "s" like "%s" and MovieID = {a} """)
.on('a -> 1).as(scalar[Long]*)
[SQLException: Parameter index out of range (1 > number of parameters, which is 0).]
SQL("""select * from Movies where "s" like "%samuel" and MovieID = {a} """)
.on('a -> 1).as(scalar[Long]*)
[SQLException: Parameter index out of range (1 > number of parameters, which is 0).]
SQL("""select * from Movies where "s" like "%flower" and MovieID = {a} """)
.on('a -> 1).as(scalar[Long]*)
[OK]
SQL("""select * from Movies where "s" like "%samuel" and MovieID = 1 """)
.on('a -> 1).as(scalar[Long]*)
[OK]
SQL("""select * from Movies where "s" like "%s" and MovieID = "{a}" """)
.on('a -> 1).as(scalar[Long]*)
[OK]
SQL("""select * from Movies where MovieID = {a} and "s" like "%s" """)
.on('a -> 1).as(scalar[Long]*)
[OK]
I believe to see a pattern here:
Under the exact condition that there is a %s sequence (quoted or unquoted) anywhere in a sql code, followed by a non quoted named parameter with arbitrary name and arbitrary distance
to the %s sequence, jdbc (or anorm) crashes. The crash seems to occur in JDBC, however its also possible that Anorm submits invalid values to JDBC.
Do you guys have any suggestions?
I think I found an enduring solution for the problem meanwhile. Since my sql generator needs to stay very flexible I somehow need a way to pass along sql fragments with their corresponding parameters without evaluating them right away. Instead the generator must be able to assemble and compose various sql fragments into bigger fragments at any time - just as he does now - but now with the acompanying, not yet evaluated parameters. I came up with this prototype:
DB.withConnection("betterdating") { implicit connection =>
case class SqlFragment(Fragment: String, Args: NamedParameter*)
val aa = SqlFragment("select MovieID from Movies")
val bb = SqlFragment("join Profiles using(ProfileID)")
val cc = SqlFragment("where Caption like \"%{a}\" and MovieID = {b}", 'a -> "s", 'b -> 5)
// combine all fragments
val v1 = SQL(Seq(aa, bb, cc).map(_.Fragment).mkString(" "))
.on((aa.Args ++ bb.Args ++ cc.Args): _*)
// better solution
val v2 = Seq(aa, bb, cc).unzip(frag => (frag.Fragment, frag.Args)) match {
case (frags, args) => SQL(frags.mkString(" ")).on(args.flatten: _*)
}
// works
println(v1.as(scalar[Long].singleOpt))
println(v2.as(scalar[Long].singleOpt))
}
It seems to work great! :-)
I then rewrote the last part of the freetext filter as follow:
// finally transform the expression
// list a single sql fragment
expressions.zipWithIndex.map { case (expr, index) =>
s"""
(concat(Movies.Caption, " ", Movies.Description, " ", Movies.Kind, " ", Profiles.Nickname, " ",
(select coalesce(group_concat(Tags.Name), "") from Tags join TagLinks using (TagID)
where TagLinks.MovieID = Movies.MovieID)) like "%{expr$index}%"))
""" -> (s"expr$index" -> expr)
}.unzip match { case (frags, args) => SqlFragment(frags.mkString(" and "), args.flatten: _*)
What do you think?
This is how it is being implemented right now:
/**
* This private helper method transforms a content filter string into an sql expression
* for searching within movies, owners and kinds and tags.
* #author Samuel Lörtscher
*/
private def contentFilterToSql(value: String) = {
// trim and clean and the parametric value from any possible anomalies
// (those include strange spacing and non closed quotes)
val cleaned = value.trim match {
case trimmed if trimmed.count(_ == '"') % 2 != 0 =>
if (trimmed.last == '"') trimmed.dropRight(1).trim
else trimmed + '"'
case trimmed =>
trimmed
};
// transform the cleaned value into a list of expressions
// (words between quotes are considered being one expression)
// empty expressions between quotes are being removed
// expressions will contain no quotes as they are being stripped during evaluation -
// thus counter measures for sql injection should be obsolete
// (we put an empty space at the end because it makes the lexer algorithm much
// more efficient as it will not need to check for end of file in every iteration)
val expressions = (cleaned + " ").foldLeft((List[String](), "", false)) { case ((list, expr, quoted), char) =>
// perform the lexer operation for the current character
if (char == ' ' && !quoted) (expr :: list, "", false)
else if (char == '"') (expr :: list, "", !quoted)
else (list, expr + char, quoted)
}._1.filter(_.nonEmpty).map(_.trim)
// finally transform the expression
// list into a variable length sql condition statement
expressions.map { expr =>
s"""
(concat(Movies.Caption, " ", Movies.Description, " ", Movies.Kind, " ", Profiles.Nickname, " ",
(select coalesce(group_concat(Tags.Name), "")
from Tags join TagLinks using (TagID) where TagLinks.MovieID = Movies.MovieID)) like "%$expr%")
"""
}.mkString(" and ")
}
Since the number of search expressions is variable, I cannot use Anorm arguments here. :-/
I found a simple solution now, but I am not exactly happy being forced to apply such crappy hacks.
Since putting a %s character sequence seems to trigger the bug, I was looking for possibilities to submit the same semantical outcome without directly passing this character sequence. I finally ended up replacing like "%$expr%" by like concat("%", "$expr%"). Since concat is being evaluated by the MySql Server engine BEFORE "like", he will put the original pattern back together before processing it by "like" - and without the sequence %s ever being transmitted through the anorm, jdbc data processors.
// finally transform the expression
// list into a variable length sql condition statement
// (freaking concat("%", "$expr%")) is required due to a freaking bug in either anorm or JDBC
// which results into a crash when %s is anyway submitted)
expressions.map { expr =>
s"""
(concat(Movies.Caption, " ", Movies.Description, " ", Movies.Kind, " ", Profiles.Nickname, " ",
(select coalesce(group_concat(Tags.Name), "")
from Tags join TagLinks using (TagID) where TagLinks.MovieID = Movies.MovieID)) like concat("%", "$expr%"))
"""
}.mkString(" and ")
I'm learning Play Scala with Slick and run into a problem. The query generated by Slick throws an error when using drop and take (works fine without drop and take, but I need pagination).
val categories = TableQuery[Categories]
def list(page: Int = 0, pageSize: Int = 10, orderBy: Int = 1)(implicit s: Session): Page[Category] = {
val offset = pageSize * page
val totalRows = count
/* Error here */
val result = categories.drop(offset).take(pageSize).list
/* This works fine */
/* val result = categories.list */
Page(result, page, offset, totalRows)
}
The query generated and error stacktrace:
[JdbcSQLException: Syntax error in SQL statement "SELECT X2.""id"", X2.""name"", X2.""description"" FROM (SELECT X3.""id"" AS ""id"", X3.""name"" AS ""name"", X3.""description"" AS ""description"" FROM ""core_category"" X3 FETCH[*] NEXT 10 ROW ONLY) X2 "; expected "RIGHT, LEFT, FULL, INNER, JOIN, CROSS, NATURAL, ,, WHERE, GROUP, HAVING, UNION, MINUS, EXCEPT, INTERSECT, ORDER, LIMIT, FOR, )"; SQL statement:
select x2."id", x2."name", x2."description" from (select x3."id" as "id", x3."name" as "name", x3."description" as "description" from "core_category" x3 fetch next 10 row only) x2 [42001-175]]
Any ideas how could I implement pagination without getting this error?
I was importing JdbcDriver instead of personalized drivers (H2Driver in my case) to get the correct dialect.
I receive such an array:
[1,2,3,4,5];
And I need to implement a query that has many ORs in WHERE clause. I take values from my array. It looks like this:
SELECT foo, bar FROM tbl WHERE (a.bar = 1 OR a.bar = 2 OR a.bar = 3 ... and so on)
How may I create such a WHERE part in node-mysql? Or how to pass it parameters?
I know this is fairly old, but I've had success, at least with recent versions of node-mysql using this format:
var titles = ['title 1', 'title 2'];
var sql = 'SELECT t.id ' +
' FROM topics t ' +
' WHERE t.title IN (?)';
var params = [titles];
This translates pretty nicely to:
SELECT t.id FROM topics t WHERE t.title IN (\'title 1\', \'title 2\')
Much cleaner, I think.
simple thing to do is to use the join() of the array
var values = [1, 2, 3, 4, 5];
var query = 'SELECT foo, bar FROM tbl WHERE (a.bar = ';
query = query + values.join(' OR a.bar = ') + ')';
That will generate the query you're looking for
I want to force slick to create queries like
select max(price) from coffees where ...
But slick's documentation doesn't help
val q = Coffees.map(_.price) //this is query Query[Coffees.type, ...]
val q1 = q.min // this is Column[Option[Double]]
val q2 = q.max
val q3 = q.sum
val q4 = q.avg
Because those q1-q4 aren't queries, I can't get the results but can use them inside other queries.
This statement
for {
coffee <- Coffees
} yield coffee.price.max
generates right query but is deprecated (generates warning: " method max in class ColumnExtensionMethods is deprecated: Use Query.max instead").
How to generate such query without warnings?
Another issue is to aggregate with group by:
"select name, max(price) from coffees group by name"
Tried to solve it with
for {
coffee <- Coffees
} yield (coffee.name, coffee.price.max)).groupBy(x => x._1)
which generates
select x2.x3, x2.x3, x2.x4 from (select x5."COF_NAME" as x3, max(x5."PRICE") as x4 from "coffees" x5) x2 group by x2.x3
which causes obvious db error
column "x5.COF_NAME" must appear in the GROUP BY clause or be used in an aggregate function
How to generate such query?
As far as I can tell is the first one simply
Query(Coffees.map(_.price).max).first
And the second one
val maxQuery = Coffees
.groupBy { _.name }
.map { case (name, c) =>
name -> c.map(_.price).max
}
maxQuery.list
or
val maxQuery = for {
(name, c) <- Coffees groupBy (_.name)
} yield name -> c.map(_.price).max
maxQuery.list