JSQL Parser - Info on parsing functions - postgresql

Can JSQLParser differentiate between View/Function/Table in a SELECT query ?
If for example, executing a function in the following ways:
select * from public.new(10);
select public.new(10);
Is it possible for JSQL to figure out that it is executing a function and return that information?

The answer is: it depends.
JSqlParser is only a parser and does not have information about the database schema. In some databases parameterless functions are allowed to be called without parenthesis, e.g. select NOW (hope that is indeed a function ;)). In this case NOW would be accepted as a column name.
But JSqlParser supports parameterized functions, e.g. select testfunc(param1). In this case it will be accepted as a function.
Syntactically the usage of view and table is identical and JSqlParser is not able to differ between those. The view name would be accepted as a table name.
To get a differentiation:
first you would let JSqlParser parse your statement
extract all column names, table names, function names (a good start here is the TableNameFinder utility of JSqlParser)
get the final type you need to check it with your database schema
So here is a little example for point 1 and 2:
Statement statement = CCJSqlParserUtil.parse("select myfunc(5), now from public.new(10), mytable");
TablesNamesFinder tablesNamesFinder = new TablesNamesFinder() {
#Override
public void visit(Column tableColumn) {
System.out.println("column = " + tableColumn);
}
#Override
public void visit(Function function) {
System.out.println("function = " + function.getName());
super.visit(function);
}
#Override
public void visit(Table tableName) {
System.out.println("table = " + tableName.getFullyQualifiedName());
super.visit(tableName);
}
#Override
public void visit(TableFunction valuesList) {
System.out.println("table function = " + valuesList.getFunction().getName());
super.visit(valuesList);
}
};
System.out.println("all extracted tables=" + tablesNamesFinder.getTableList(statement));
and the result is:
function = myfunc
column = now
table function = public.new
table = mytable
all extracted tables=[mytable]

Related

dynamically choose columns in select with R2DB2

I have been trying to pass the columns I want to select in but out of the box it appears it is not possible. I have tried things like
#Query("SELECT :columns FROM USERS u WHERE u.LOCALE = :locale AND u.id IN (:ids)")
Flux<Users> retrieveExportData(#Param("columns") String columns,
#Param("locale") String locale,
#Param("ids") String[] ids);
and with
private final R2dbcEntityTemplate template;
I tried to create my own query and but that was not working because it has to be of type Criteria and that was just creating a complexity that just was not worth it.
It would be nice if I could add the columns like
criteriaList.add(
Criteria.where("LOCALE").is(locale)
);
Criteria criteria = Criteria.from(criteriaList);
and execute it like
Flux<Users> users = this.template.select(User.class)
.matching(Query.query(criteria))
.all();
or just calling the repository like in my first example.
Has anyone been able to do this successfully?
----- update 1 -----
I tried doing like so:
import org.springframework.r2dbc.core.DatabaseClient;
DatabaseClient databaseClient = DatabaseClient.create(connectionFactory);
String sql = "SELECT " + columns + " FROM USERS u WHERE u.LOCALE=" + locale + " AND u.id IN (" + ids + ")";
return databaseClient.sql(sql)
.fetch()
.all().cast(User.class);
but Since Spring 4.3.6.RELEASE, LinkedCaseInsensitiveMap doesn't extend LinkedHashMap and HashMap, but only implements Map interface.
This results in a
Cannot cast org.springframework.util.LinkedCaseInsensitiveMap to User at java.base/java.lang.Class.cast
error.
I then tried the jooq approach suggested in the answers but it just produces syntax errors. Example
private final DSLContext ctx = DSL.using(connectionFactory);
private final Users users = ctx.newRecord(Users.USERS); <-- USERS not found
#Query("SELECT :columns FROM USERS u WHERE u.LOCALE = :locale AND u.id IN (:ids)")
public Flux<Users> retrieveExportData(
List<Field<?>> columns,
String locale,
String[] ids
) {
return Flux.from(ctx
.select(columns)
.from("USERS")
.where(users.LOCALE.eq(locale)) <--- LOCALE not found
.and(users.ID.in(ids)) <--- ID not found
).map(r -> r.into(Users.class)); <---- into not found
}
the library look promising. I will try to get it working.
You cannot replace a bind parameter (:columns) by syntactic elements like this, other than actual bind values. For this type of dynamic SQL, you'll have to resort to some sort of query building mechanism.
Perhaps look at jOOQ, which has R2DBC support? Your implementation would then look like this:
#Query("SELECT :columns FROM USERS u WHERE u.LOCALE = :locale AND u.id IN (:ids)")
public Flux<Users> retrieveExportData(
List<Field<?>> columns,
String locale,
String[] ids
) {
return Flux.from(ctx
.select(columns)
.from(USERS)
.where(USERS.LOCALE.eq(locale))
.and(USERS.ID.in(ids))
).map(r -> r.into(Users.class));
}
Disclaimer: I work for the company behind jOOQ.
you can write dynamic SQL like so.
import org.springframework.r2dbc.core.DatabaseClient;
DatabaseClient databaseClient = DatabaseClient.create(connectionFactory);
String sql = "SELECT " + columns + " FROM USERS u WHERE u.LOCALE=" + locale + " AND u.id IN (" + ids + ")";
return databaseClient.sql(sql)
.fetch()
.all().cast(User.class);
and you will need this dependency
implementation "org.springframework.boot:spring-boot-starter-data-r2dbc:2.6.2"

How to assure the return StringList will be ordered : Scala

I am using Scala 2.11.8
I am trying to read queries from my Property File. Each Query Set has multiple parts (explained below)
And i have certain sequence in which these queries must execute.
Code:
import com.typesafe.config.ConfigFactory
object ReadProperty {
def main(args : Array[String]): Unit = {
val queryRead = ConfigFactory.load("testqueries.properties").getConfig("select").getStringList("caseInc").toArray()
val localRead = ConfigFactory.load("testqueries.properties").getConfig("select").getStringList("caseLocal").toArray.toSet
queryRead.foreach(println)
localRead.foreach(println)
}
}
PropertyFile Content :
select.caseInc.2 = Select emp_salary, emp_dept_id from employees
select.caseLocal.1 = select one
select.caseLocal.3 = select three
select.caseRemote.2 = Select e1.emp_name, d1.dept_name, e1.salary from emp_1 e1 join dept_1 d1 on(e1.emp_dept_id = d1.dept_id)
select.caseRemote.1 = Select * from departments
select.caseInc.1 = Select emp_id, emp_name from employees
select.caseLocal.2 = select two
select.caseLocal.4 = select four
Output:
Select emp_id, emp_name from employees
Select emp_salary, emp_dept_id from employees
select one
select two
select three
select four
As we can see in output, The result is Sorted . In the property if you see i have tried numbering the queries in the sequence it should run.(passing the caseInc, caseLocal as arguments).
With getStringList() i am always getting the Sorted List on the basis of the sequence number i am providing.
Even when i tried using toArray() & toArray().toSet i am getting sorted output.
So far its Good
But how to be sure that it will always return in Sorted Order which i have provided in the property file. I am confused because somehow i am not able to find the API which says that the returned List will be Sorted.
I think you can rely on this fact. Looking into the code of DefaultTransformer you can see following piece of logic:
} else if (requested == ConfigValueType.LIST && value.valueType() == ConfigValueType.OBJECT) {
// attempt to convert an array-like (numeric indices) object to a
// list. This would be used with .properties syntax for example:
// -Dfoo.0=bar -Dfoo.1=baz
// To ensure we still throw type errors for objects treated
// as lists in most cases, we'll refuse to convert if the object
// does not contain any numeric keys. This means we don't allow
// empty objects here though :-/
AbstractConfigObject o = (AbstractConfigObject) value;
Map<Integer, AbstractConfigValue> values = new HashMap<Integer, AbstractConfigValue>();
for (String key : o.keySet()) {
int i;
try {
i = Integer.parseInt(key, 10);
if (i < 0)
continue;
values.put(i, o.get(key));
} catch (NumberFormatException e) {
continue;
}
}
if (!values.isEmpty()) {
ArrayList<Map.Entry<Integer, AbstractConfigValue>> entryList = new ArrayList<Map.Entry<Integer, AbstractConfigValue>>(
values.entrySet());
// sort by numeric index
Collections.sort(entryList,
new Comparator<Map.Entry<Integer, AbstractConfigValue>>() {
#Override
public int compare(Map.Entry<Integer, AbstractConfigValue> a,
Map.Entry<Integer, AbstractConfigValue> b) {
return Integer.compare(a.getKey(), b.getKey());
}
});
// drop the indices (we allow gaps in the indices, for better or
// worse)
ArrayList<AbstractConfigValue> list = new ArrayList<AbstractConfigValue>();
for (Map.Entry<Integer, AbstractConfigValue> entry : entryList) {
list.add(entry.getValue());
}
return new SimpleConfigList(value.origin(), list);
}
}
Note how keys are parsed as integer values and then sorted using Integer.compare

can i use custom lambda method in entity framework?

i have some methods like:
public static string ToOtherFormat (this string inp)
{
// some code to change inp
return inp;
}
and in my select i want to have code like this:
var DetailMembers = db.TB_Members
.Where(x=> x.FName == obj.ToOtherFormat())
.Select( x=> new { name = (x.FName.ToOtherFormat() + " " + x.LName) , x.ActCode });
i try and just have error. is it possible?
thanks!
i receive this error in simple convert to integer
LINQ to Entities does not recognize the method 'Int32 ToInt32(System.String)' method, and this method cannot be translated into a store expression.
with this code
.Where(x => x.MemberID == Convert.ToInt32(Hmemid.Hash_two_Decrypt())
Looks like you are querying against the database. Your current query will get translated into SQL query and since SQL doesn't recognize your function that is why you get error.
You may get the data from the tables using a query without that function and then later do the formatting on the result set.
i found it on use .AsEnumerable() method like:
var DetailMembers = db.TB_Members.AsEnumerable()
.Where(x=> x.FName == obj.ToOtherFormat())
.Select( x=> new { name = (x.FName.ToOtherFormat() + " " + x.LName) , x.ActCode });

FunctionImport in entity framework 4 issue

I'm using entity framework 4.
I have a stored procedure that just updates one value in my table, namely the application state ID. So I created a stored procedure that looks like this:
ALTER PROCEDURE [dbo].[UpdateApplicationState]
(
#ApplicationID INT,
#ApplicationStateID INT
)
AS
BEGIN
UPDATE
[Application]
SET
ApplicationStateID = #ApplicationStateID
WHERE
ApplicationID = #ApplicationID;
END
I created a function import called UpdateApplicationState. I had initially set its return type to null, but then it wasn't created in the context. So I changed its return type to int. Now it was created in the context. Is it wise to return something from my stored procedure?
Here is my method in my ApplicationRepository class:
public void UpdateApplicationState(int applicationID, int applicationStateID)
{
var result = context.UpdateApplicationState(applicationID, applicationStateID);
}
Here is my calling code to this method in my view:
applicationRepository.UpdateApplicationState(id, newApplicationStateID);
When I run it then I get the following error:
The data reader returned by the store
data provider does not have enough
columns for the query requested.
Any idea/advise on what I can do to get this to work?
Thanks
To get POCO to work with function imports that return null, you can customize the .Context.tt file like this.
Find the "Function Imports" named region (the section that starts with region.Begin("Function Imports"); and ends with region.End();) in the .Context.tt file and replace that whole section with the following:
region.Begin("Function Imports");
foreach (EdmFunction edmFunction in container.FunctionImports)
{
var parameters = FunctionImportParameter.Create(edmFunction.Parameters, code, ef);
string paramList = String.Join(", ", parameters.Select(p => p.FunctionParameterType + " " + p.FunctionParameterName).ToArray());
var isReturnTypeVoid = edmFunction.ReturnParameter == null;
string returnTypeElement = String.Empty;
if (!isReturnTypeVoid)
returnTypeElement = code.Escape(ef.GetElementType(edmFunction.ReturnParameter.TypeUsage));
#>
<# if (isReturnTypeVoid) { #>
<#=Accessibility.ForMethod(edmFunction)#> void <#=code.Escape(edmFunction)#>(<#=paramList#>)
<# } else { #>
<#=Accessibility.ForMethod(edmFunction)#> ObjectResult<<#=returnTypeElement#>> <#=code.Escape(edmFunction)#>(<#=paramList#>)
<# } #>
{
<#
foreach (var parameter in parameters)
{
if (!parameter.NeedsLocalVariable)
{
continue;
}
#>
ObjectParameter <#=parameter.LocalVariableName#>;
if (<#=parameter.IsNullableOfT ? parameter.FunctionParameterName + ".HasValue" : parameter.FunctionParameterName + " != null"#>)
{
<#=parameter.LocalVariableName#> = new ObjectParameter("<#=parameter.EsqlParameterName#>", <#=parameter.FunctionParameterName#>);
}
else
{
<#=parameter.LocalVariableName#> = new ObjectParameter("<#=parameter.EsqlParameterName#>", typeof(<#=parameter.RawClrTypeName#>));
}
<#
}
#>
<# if (isReturnTypeVoid) { #>
base.ExecuteFunction("<#=edmFunction.Name#>"<#=code.StringBefore(", ", String.Join(", ", parameters.Select(p => p.ExecuteParameterName).ToArray()))#>);
<# } else { #>
return base.ExecuteFunction<<#=returnTypeElement#>>("<#=edmFunction.Name#>"<#=code.StringBefore(", ", String.Join(", ", parameters.Select(p => p.ExecuteParameterName).ToArray()))#>);
<# } #>
}
<#
}
region.End();
What I'm doing here is instead of ignoring all function imports that return null, I'm creating a method that returns null. I hope this is helpful.
It is because you do not actually returning anything from your stored procedure. Add a line like below to your SP (SELECT ##ROWCOUNT), and it will be executing properly.
BEGIN
...
SELECT ##ROWCOUNT
END
While this solution will address your issue and actually returns the number of effected rows by your SP, I am not clear on why this is an issue for you:
I had initially set its return type to null, but then it wasn't created in the context.
When doing a Function Import, you can select "None" as return type and it will generate a new method on your ObjectContext with a return type of int. This method basically executes a stored procedure that is defined in the data source; discards any results returned from the function; and returns the number of rows affected by the execution.
EDIT: Why a Function without return value is ignored in a POCO Scenario:
Drilling into ObjectContext T4 template file coming with ADO.NET C# POCO Entity Generator reveals why you cannot see your Function in your ObjectContext class: Simply it's ignored! They escape to the next iteration in the foreach loop that generates the functions.
The workaround for this is to change the T4 template to actually generate a method for Functions without return type or just returning something based on the first solution.
region.Begin("Function Imports");
foreach (EdmFunction edmFunction in container.FunctionImports)
{
var parameters = FunctionImportParameter.Create(edmFunction.Parameters, code, ef);
string paramList = String.Join(", ", parameters.Select(p => p.FunctionParameterType + " " + p.FunctionParameterName).ToArray());
// Here is why a Function without return value is ignored:
if (edmFunction.ReturnParameter == null)
{
continue;
}
string returnTypeElement = code.Escape(ef.GetElementType(edmFunction.ReturnParameter.TypeUsage));
...

using the TSqlParser

I'm attempting to parse SQL using the TSql100Parser provided by microsoft. Right now I'm having a little trouble using it the way it seems to be intended to be used. Also, the lack of documentation doesn't help. (example: http://msdn.microsoft.com/en-us/library/microsoft.data.schema.scriptdom.sql.tsql100parser.aspx )
When I run a simple SELECT statement through the parser it returns a collection of TSqlStatements which contains a SELECT statement.
Trouble is, the TSqlSelect statement doesn't contain attributes such as a WHERE clause, even though the clause is implemented as a class. http://msdn.microsoft.com/en-us/library/microsoft.data.schema.scriptdom.sql.whereclause.aspx
The parser does recognise the WHERE clause as such, looking at the token stream.
So, my question is, am I using the parser correctly? Right now the token stream seems to be the most useful feature of the parser...
My Test project:
public static void Main(string[] args)
{
var parser = new TSql100Parser(false);
IList<ParseError> Errors;
IScriptFragment result = parser.Parse(
new StringReader("Select col from T1 where 1 = 1 group by 1;" +
"select col2 from T2;" +
"select col1 from tbl1 where id in (select id from tbl);"),
out Errors);
var Script = result as TSqlScript;
foreach (var ts in Script.Batches)
{
Console.WriteLine("new batch");
foreach (var st in ts.Statements)
{
IterateStatement(st);
}
}
}
static void IterateStatement(TSqlStatement statement)
{
Console.WriteLine("New Statement");
if (statement is SelectStatement)
{
PrintStatement(sstmnt);
}
}
Yes, you are using the parser correctly.
As Damien_The_Unbeliever points out, within the SelectStatement there is a QueryExpression property which will be a QuerySpecification object for your third select statement (with the WHERE clause).
This represents the 'real' SELECT bit of the query (whereas the outer SelectStatement object you are looking at has just got the 'WITH' clause (for CTEs), 'FOR' clause (for XML), 'ORDER BY' and other bits)
The QuerySpecification object is the object with the FromClauses, WhereClause, GroupByClause etc.
So you can get to your WHERE Clause by using:
((QuerySpecification)((SelectStatement)statement).QueryExpression).WhereClause
which has a SearchCondition property etc. etc.
Quick glance around would indicate that it contains a QueryExpression, which could be a QuerySpecification, which does have the Where clause attached to it.
if someone lands here and wants to know how to get the whole elements of a select statement the following code explain that:
QuerySpecification spec = (QuerySpecification)(((SelectStatement)st).QueryExpression);
StringBuilder sb = new StringBuilder();
sb.AppendLine("Select Elements");
foreach (var elm in spec.SelectElements)
sb.Append(((Identifier)((Column)((SelectColumn)elm).Expression).Identifiers[0]).Value);
sb.AppendLine();
sb.AppendLine("From Elements");
foreach (var elm in spec.FromClauses)
sb.Append(((SchemaObjectTableSource)elm).SchemaObject.BaseIdentifier.Value);
sb.AppendLine();
sb.AppendLine("Where Elements");
BinaryExpression binaryexp = (BinaryExpression)spec.WhereClause.SearchCondition;
sb.Append("operator is " + binaryexp.BinaryExpressionType);
if (binaryexp.FirstExpression is Column)
sb.Append(" First exp is " + ((Identifier)((Column)binaryexp.FirstExpression).Identifiers[0]).Value);
if (binaryexp.SecondExpression is Literal)
sb.Append(" Second exp is " + ((Literal)binaryexp.SecondExpression).Value);
I had to split a SELECT statement into pieces. My goal was to COUNT how many record a query will return. My first solution was to build a sub query such as
SELECT COUNT(*) FROM (select id, name from T where cat='A' order by id) as QUERY
The problem was that in this case the order clause raises the error "The ORDER BY clause is not valid in views, inline functions, derived tables, sub-queries, and common table expressions, unless TOP or FOR XML is also specified"
So I built a parser that split a SELECT statment into fragments using the TSql100Parser class.
using Microsoft.Data.Schema.ScriptDom.Sql;
using Microsoft.Data.Schema.ScriptDom;
using System.IO;
...
public class SelectParser
{
public string Parse(string sqlSelect, out string fields, out string from, out string groupby, out string where, out string having, out string orderby)
{
TSql100Parser parser = new TSql100Parser(false);
TextReader rd = new StringReader(sqlSelect);
IList<ParseError> errors;
var fragments = parser.Parse(rd, out errors);
fields = string.Empty;
from = string.Empty;
groupby = string.Empty;
where = string.Empty;
orderby = string.Empty;
having = string.Empty;
if (errors.Count > 0)
{
var retMessage = string.Empty;
foreach (var error in errors)
{
retMessage += error.Identifier + " - " + error.Message + " - position: " + error.Offset + "; ";
}
return retMessage;
}
try
{
// Extract the query assuming it is a SelectStatement
var query = ((fragments as TSqlScript).Batches[0].Statements[0] as SelectStatement).QueryExpression;
// Constructs the From clause with the optional joins
from = (query as QuerySpecification).FromClauses[0].GetString();
// Extract the where clause
where = (query as QuerySpecification).WhereClause.GetString();
// Get the field list
var fieldList = new List<string>();
foreach (var f in (query as QuerySpecification).SelectElements)
fieldList.Add((f as SelectColumn).GetString());
fields = string.Join(", ", fieldList.ToArray());
// Get The group by clause
groupby = (query as QuerySpecification).GroupByClause.GetString();
// Get the having clause of the query
having = (query as QuerySpecification).HavingClause.GetString();
// Get the order by clause
orderby = ((fragments as TSqlScript).Batches[0].Statements[0] as SelectStatement).OrderByClause.GetString();
}
catch (Exception ex)
{
return ex.ToString();
}
return string.Empty;
}
}
public static class Extension
{
/// <summary>
/// Get a string representing the SQL source fragment
/// </summary>
/// <param name="statement">The SQL Statement to get the string from, can be any derived class</param>
/// <returns>The SQL that represents the object</returns>
public static string GetString(this TSqlFragment statement)
{
string s = string.Empty;
if (statement == null) return string.Empty;
for (int i = statement.FirstTokenIndex; i <= statement.LastTokenIndex; i++)
{
s += statement.ScriptTokenStream[i].Text;
}
return s;
}
}
And to use this class simply:
string fields, from, groupby, where, having, orderby;
SelectParser selectParser = new SelectParser();
var retMessage = selectParser.Parse("SELECT * FROM T where cat='A' Order by Id desc",
out fields, out from, out groupby, out where, out having, out orderby);