SiddhiQL query Error : mismatched input 'group' - group-by

I am using a simple SiddhiQL query to get number of records having same timestamp till minute entry and these timestamps. The query is:
from inputStream
select time:dateFormat(ts,'yyyy-MM-dd HH:mm') as formatedTs, count(formatedTs)
group by formatedTs
insert into outputStream;
It gives me the error mismatched input 'group' expecting {'*', '+', '-', '/', '%', '<', '<=', '>', '>=', '==', '!=', AS, OR, AND, IN}. What's wrong with group by clause in this context?

Reason for the error:
This particular error is coming due to the missing AS, following count(formatedTs)
(This is indicated in the error message as well.
mismatched input 'group' expecting {'*', '+', '-', '/', '%', '<',
'<=', '>', '>=', '==', '!=', AS, OR, AND, IN}
Query Compiler thinks group is in wrong place due to the missing AS)
Correction needed:
So the select statement, needs to be corrected as below:
select time:dateFormat(ts,'yyyy-MM-dd HH:mm') as formatedTs, count(formatedTs) as tsCount
Further correction which you might need:
Also, does the inputStream has an attribute named formatedTs? If not, after fixing the select statement, you'll get another error like below:
Cannot find attribute type as 'formatedTs' does not exist in 'inputStream'
because the attribute for which you are taking the count should exist in the inputStream
If that is the case, following query (which should compile successfully) might help you:
from inputStream
select time:dateFormat(ts,'yyyy-MM-dd HH:mm') as formatedTs, count(ts) as countTs
group by ts
insert into outputStream;
Update
Updating the query since your requirement is to group by the formatted timestamp:
from inputStream
select time:dateFormat(ts,'yyyy-MM-dd HH:mm') as formatedTs
insert into innerStream;
from innerStream
select formatedTs, count(formatedTs) as countTs
group by formatedTs
insert into outStream;

Related

Copy into snowflake table from raw data file using Perl DBI

There's not much info out there for perl dbi and snowflake so I'll give this a shot. I have a raw file, of which the headers are contained in line 1. This exact 'copy into' command works from the snowflake gui. I'm not sure if I can just take this exact command and put it into a perl prepare and execute.
COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA$FILENAME,'/',4) as SEAT_ID,
$1:auction_id_64 as AUCTION_ID_64,
DATEADD(S,\$1:date_time,'1970-01-01') as DATE_TIME,
$1:user_tz_offset as USER_TZ_OFFSET,
$1:creative_width as CREATIVE_WIDTH,
$1:creative_height as CREATIVE_HEIGHT,
$1:media_type as MEDIA_TYPE,
$1:fold_position as FOLD_POSITION,
$1:event_type as EVENT_TYPE
FROM #DBTABLE.lnd.S3_STAGE_READY/pr/data/standard/data_dt=20200825/00/STANDARD_FILE.gz.parquet)
pattern = '.*.parquet' file_format = (TYPE = 'PARQUET' SNAPPY_COMPRESSION = TRUE)
ON_ERROR = 'SKIP_FILE_10%'
my $SQL = "COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA\$FILENAME,'/',4) as SEAT_ID,
\$1:auction_id_64 as AUCTION_ID_64,
DATEADD(S,\$1:date_time,'1970-01-01') as DATE_TIME,
\$1:user_tz_offset as USER_TZ_OFFSET,
\$1:creative_width as CREATIVE_WIDTH,
\$1:creative_height as CREATIVE_HEIGHT,
\$1:media_type as MEDIA_TYPE,
\$1:fold_position as FOLD_POSITION,
\$1:event_type as EVENT_TYPE
FROM \#DBTABLE.lnd.S3_STAGE_READY/pr/data/standard/data_dt=20200825/00/STANDARD_FILE.gz.parquet)
pattern = '.*.parquet' file_format = (TYPE = 'PARQUET' SNAPPY_COMPRESSION = TRUE)
ON_ERROR = 'SKIP_FILE_10%'";
my $sth = $dbh->prepare($sql);
$sth->execute;
In looking at the output from snowflake I see this error
syntax error line 3 at position 4 unexpected '?'.
syntax error line 4 at position 13 unexpected '?'.
COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA$FILENAME,'/',4) as SEAT_ID,
$1? as AUCTION_ID_64,
DATEADD(S,$1?,'1970-01-01') as DATE_TIME,
$1? as USER_TZ_OFFSET,
$1? as CREATIVE_WIDTH,
$1? as CREATIVE_HEIGHT,
$1? as MEDIA_TYPE
Do I need to create bind variables for each of the columns? I usually pull in the data from the file and put them into variables but this is different as I can't read the raw file first, it has to come directly from the copy into command.
Any help would be appreciated.
It was interpreting the : as a bind variable value, rather than a value in a variant. I used the bracket notation, instead like the following:
my $SQL = "COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA\$FILENAME,'/',4) as SEAT_ID,
\$1['auction_id_64'] as AUCTION_ID_64,
DATEADD(S,\$1['date_time,'1970-01-01') as DATE_TIME,
\$1['user_tz_offset'] as USER_TZ_OFFSET,
\$1:creative_width'] as CREATIVE_WIDTH,
etc...
That worked

DB2 SQL Error: SQLCODE=-302 while executing prepared statement

I have a SQL query which takes user inputs hence security flaw is present.
The existing query is:
SELECT BUS_NM, STR_ADDR_1, CITY_NM, STATE_CD, POSTAL_CD, COUNTRY_CD,
BUS_PHONE_NB,PEG_ACCOUNT_ID, GDN_ALERT_ID, GBIN, GDN_MON_REF_NB,
ALERT_DT, ALERT_TYPE, ALERT_DESC,ALERT_PRIORITY
FROM ( SELECT A.BUS_NM, AE.STR_ADDR_1, A.CITY_NM, A.STATE_CD, A.POSTAL_CD,
CC.COUNTRY_CD, A.BUS_PHONE_NB, A.PEG_ACCOUNT_ID, 'I' ||
LPAD(INTL_ALERT_DTL_ID, 9,'0') GDN_ALERT_ID,
LPAD(IA.GBIN, 9,'0') GBIN, IA.GDN_MON_REF_NB,
DATE(IAD.ALERT_TS) ALERT_DT,
XMLCAST(XMLQUERY('$A/alertTypeConfig/biqCode/text()' passing
IAC.INTL_ALERT_TYPE_CONFIG as "A") AS CHAR(4)) ALERT_TYPE,
, ROW_NUMBER() OVER () AS "RN"
FROM ACCOUNT A, Other tables
WHERE IA.GDN_MON_REF_NB = '100'
AND A.PEG_ACCOUNT_ID = IAAR.PEG_ACCOUNT_ID
AND CC.COUNTRY_CD = A.COUNTRY_ISO3_CD
ORDER BY IA.INTL_ALERT_ID ASC )
WHERE ALERT_TYPE IN (" +TriggerType+ ");
I changed it to accept TriggerType from setString like:
SELECT BUS_NM, STR_ADDR_1, CITY_NM, STATE_CD, POSTAL_CD, COUNTRY_CD,
BUS_PHONE_NB,PEG_ACCOUNT_ID, GDN_ALERT_ID, GBIN, GDN_MON_REF_NB,
ALERT_DT, ALERT_TYPE, ALERT_DESC,ALERT_PRIORITY
FROM ( SELECT A.BUS_NM, AE.STR_ADDR_1, A.CITY_NM, A.STATE_CD, A.POSTAL_CD,
CC.COUNTRY_CD, A.BUS_PHONE_NB, A.PEG_ACCOUNT_ID,
'I' || LPAD(INTL_ALERT_DTL_ID, 9,'0') GDN_ALERT_ID,
LPAD(IA.GBIN, 9,'0') GBIN, IA.GDN_MON_REF_NB,
DATE(IAD.ALERT_TS) ALERT_DT,
XMLCAST(XMLQUERY('$A/alertTypeConfig/biqCode/text()' passing
IAC.INTL_ALERT_TYPE_CONFIG as "A") AS CHAR(4)) ALERT_TYPE,
ROW_NUMBER() OVER () AS "RN"
FROM ACCOUNT A, other tables
WHERE IA.GDN_MON_REF_NB = '100'
AND A.PEG_ACCOUNT_ID = IAAR.PEG_ACCOUNT_ID
AND CC.COUNTRY_CD = A.COUNTRY_ISO3_CD
ORDER BY IA.INTL_ALERT_ID ASC )
WHERE ALERT_TYPE IN (?);
Setting trigger type as below:
if (StringUtils.isNotBlank(request.getTriggerType())) {
preparedStatement.setString(1, triggerType != null ? triggerType.toString() : "");
}
Getting error as
Caused by: com.ibm.db2.jcc.am.SqlDataException: DB2 SQL Error: SQLCODE=-302, SQLSTATE=22001, SQLERRMC=null, DRIVER=4.19.26
The -302 SQLCODE indicates a conversion error of some sort.
SQLSTATE 22001 narrows that down a bit by telling us that you are trying to force a big string into a small variable. Given the limited information in your question, I am guessing it is the XMLCAST that is the culprit.
DB2 won't jam 30 pounds of crap into a 4 pound bag so to speak, it gives you an error. Maybe giving XML some extra room in the cast might be a help. If you need to make sure it ends up being only 4 characters long, you could explicitly do a LEFT(XMLCAST( ... AS VARCHAR(64)), 4). That way the XMLCAST has the space it needs, but you cut it back to fit your variable on the fetch.
The other thing could be that the variable being passed to the parameter marker is too long. DB2 will guess the type and length based on the length of ALERT_TYPE. Note that you can only pass a single value through a parameter marker. If you pass a comma separated list, it will not behave as expected (unless you expect ALERT_TYPE to also contain a comma separated list). If you are getting the comma separated list from a table, you can use a sub-select instead.
Wrong IN predicate use with a parameter.
Do not expect that IN ('AAAA, M250, ABCD') (as you try to do passing a comma-separated string as a single parameter) works as IN ('AAAA', 'M250', 'ABCD') (as you need). These predicates are not equivalent.
You need some "string tokenizer", if you want to pass such a comma-separated string like below.
select t.*
from
(
select XMLCAST(XMLQUERY('$A/alertTypeConfig/biqCode/text()' passing IAC.INTL_ALERT_TYPE_CONFIG as "A") AS CHAR(4)) ALERT_TYPE
from table(values xmlparse(document '<alertTypeConfig><biqCode>M250, really big code</biqCode></alertTypeConfig>')) IAC(INTL_ALERT_TYPE_CONFIG)
) t
--WHERE ALERT_TYPE IN ('AAAA, M250, ABCD')
join xmltable('for $id in tokenize($s, ",\s?") return <i>{string($id)}</i>'
passing cast('AAA, M250 , ABCD' as varchar(200)) as "s"
columns token varchar(200) path '.') x on x.token=t.ALERT_TYPE
;
Run the statement as is. Then you may uncomment the string with WHERE clause and comment out the rest to see what you try to do.
P.S.:
The error you get is probably because you don't specify the data type of the parameter (you don't use something like IN (cast(? as varchar(xxx))), and db2 compiler assumes that its length is equal to the length of the ALERT_TYPE expression (4 bytes).

Hive FAILED: Parse Exception line 3:39 mismatched input

Im trying to make a query that will check if there is any row which has a salary that is 10000 higher than the salary for that department but when I try to run it I get this Error:
FAILED: ParseException line 3:39 mismatched input 'SELECT' expecting ) near '''' in expression specification
this is the query Im using
set AVERAGES ='SELECT ROLE, AVG(AnnualSalary) From Salaries GROUP BY ROLE';
SELECT ROLE, AVG(AnnualSalary) FROM Salaries
GROUP BY ROLE, AnnualSalary HAVING AnnualSalary > ('${hiveconf:AVERAGES}' + 10000);
Currently Hive does not support storing the query result into variable.
You can use window function to achieve this.
select * from
( select *,
avg(AnnualSalary) over(partition by ROLE) role_avg
from
Salaries
) a
where
AnnualSalary > role_avg+10000

group by error with postgres and pomm orm

I want to execute the following SQL query :
SELECT date, COUNT(id_customers)
FROM event
WHERE event_id = 3
GROUP BY date
When I try this query in my database, it works perfectly. But in my code I get an error which I can't resolve.
I use symfony2 with the orm pomm. It's Postgresql.
Here is my code :
$sql = "SELECT e.date, COUNT(id_customers) FROM event e WHERE event_id = $* GROUP BY e.date";
return $this->query($sql, [$eventId])->extract();
Here is the error :
request.CRITICAL: Uncaught PHP Exception InvalidArgumentException:
"No such field 'id'. Existing fields are {date, count}"
at /home/vagrant/sourcefiles/vendor/pomm-project/model-manager/sources/lib/Model/FlexibleEntity/FlexibleContainer.php line 64
{"exception":" [object] (InvalidArgumentException(code: 0): No such field 'id'.
Existing fields are {date, count}
at /home/vagrant/sourcefiles/vendor/pomm-project/model-manager/sources/lib/Model/FlexibleEntity/FlexibleContainer.php:64)"} []
So I tried to had the id in my select, by I get this error :
request.CRITICAL: Uncaught PHP Exception
PommProject\Foundation\Exception\SqlException: "
SQL error state '42803' [ERROR] ==== ERROR: column "e.id" must appear in the GROUP BY clause or be used in an aggregate function LINE 1: SELECT e.id, e.date, COUNT(id_customers) FROM event e WHERE ... ^
==== «PREPARE === SELECT e.id, e.date, COUNT(id_customers) FROM event e WHERE event_id = $1 GROUP BY e.date ===»." at /home/vagrant/sourcefiles/vendor/pomm-project/foundation/sources/lib/Session/Connection.php line 327 {"exception":"[object] (PommProject\Foundation\Exception\SqlException(code: 0): \nSQL error state '42803' [ERROR]\n====\nERROR: column \"e.id\" must appear in the GROUP BY clause or be used in an aggregate function\nLINE 1: SELECT e.id, e.date, COUNT(id_customers) FROM event e WHERE ...\n ^\n\n====\n«PREPARE ===\nSELECT e.id, e.date, COUNT(id_customers) FROM event e WHERE event_id = $1 GROUP BY e.date\n ===». at /home/vagrant/sourcefiles/vendor/pomm-project/foundation/sources/lib/Session/Connection.php:327)"} []
The only thing that works is when I had the id in the group by, but this is not the result I want.
Someone can explain me why this is working in the database and not in the php ?
this is because you are fetching flexible entities without their primary key. There is an identity mapper behind the scene that ensure fetching twice the same entity will return the same instance.
In this case, you do not need to fetch entities (hence the extract after the query). So you can just use the QueryManager pooler to return converted arrays like the following:
$sql = "SELECT e.date, COUNT(id_customers) FROM event e WHERE event_id = $* GROUP BY e.date";
// Return an iterator that fetches converted arrays on demand:
return $this
->getSession()
->getQueryManager()
->query($sql, [$eventId])
;
i think its because the alias,
try this
$sql = "SELECT e.date, COUNT(e.id_customers) FROM event e WHERE event_id = $* GROUP BY e.date";
return $this->query($sql, [$eventId])->extract();
This is exactly how GROUP BY works in PostgreSQL:
When GROUP BY is present, it is not valid for the SELECT list
expressions to refer to ungrouped columns except within aggregate
functions, since there would be more than one possible value to return
for an ungrouped column.
It means that each field in your query either must be present in GROUP BY statement or handled by any of the aggregation functions. This is one of the differences between GROUP BY in MySQL and PostreSQL.
In other words you can add id at GROUP BY statement and do not worry about it ;)

Using date in ref cusor with pl/sql

I have a var: acc_date with type date.
It takes its value from a cursor and when I insert its value to logger table as:
insert into logger values(1,acc_date);
the out put when a select it from logger is
1 01-JAN-10
but when i use it to compare with another Date value in another cursor as
OPEN c_get_date_id
for 'SELECT Date_D.DATEKEY from Date_D where Date_D.DATEVALUE='||acc_date;
EXIT WHEN c_get_date_id%NOTFOUND;
FETCH c_get_date_id
INTO date_id;
insert into logger values (1,'Now with date_id'||date_id);
CLOSE c_get_date_id;
an error occurs:
Error report:
ORA-00904: "JAN": invalid identifier
ORA-06512: at "HW.FILLFACT", line 82
ORA-06512: at line 1
00904. 00000 - "%s: invalid identifier"
*Cause:
*Action:
strong text
You need at least add some quotes around the date:
....' where Date_D.DATEVALUE='''||acc_date||'''';
Double apostrophes within a string will be concatenated to a single apostrophe, so that the expression becomes
where Date_D.DATEVALUE='....';
In order to make the thing more foolprof, I'd also add a specific to_date:
.... ' where Date_D.DATEVALUE=to_date(''' || acc_date || ', ''dd-mon-yy'')';
At the moment your dynamic query is being interpreted as:
SELECT Date_D.DATEKEY from Date_D where Date_D.DATEVALUE=01-JAN-10
The error is because string representation of the date isn't being quoted, so it's seeing JAN as an identifier - and nothing matches that name. You could enclose the date value in quotes:
open c_get_date_id
for 'SELECT Date_D.DATEKEY from Date_D where Date_D.DATEVALUE='''||acc_date||'''';
But you're treating the date as a string, and forcing conversion of all your table values to strings to be compared, using your session's NLS_DATE_FORMAT. It would be better to compare it as a date (although this somewhat assumes all your values have the time portion set to midnight):
open c_get_date_id
for select date_d.datekey from date_d where date_d.datevalue = acc_date;
Your exit is in the wrong place though, and you aren't looping, so maybe you want:
open c_get_date_id
for select date_d.datekey from date_d where date_d.datevalue = acc_date;
loop
fetch c_get_date_id into date_id;
exit when c_get_date_id%notfound;
insert into logger values (1, 'Now with date_id'||date_id);
end loop;
close c_get_date_id;
If you only have one value in the first place though, you probably don't want a loop or cursor at all, and could do a simple select ... into instead:
select date_d.datekey into date_id from date_d
where date_d.datevalue = acc_date;
insert into logger values (1, 'Now with date_id'||date_id);
Though of course that would error if you had no matching date in your table, or more than one, and you'd need to deal with that - but then I guess you'd want to anyway.