I need help on achieving GROUP BY functionality for a dynamic table. Here I have just created a sample piece of code that is similar to my production code.
I have a database table that has more than 166 fields. Based on Factor_Type, the internal table will be created with selective fields according to the config table. Now I need to upload excel data into this dynamic internal table.
Here for example, below is the sample structure with few fields which looks similar to my DB table,
TYPES: BEGIN OF ty_table,
factor_type TYPE char30,
field1 TYPE char30,
field2 TYPE char30,
field3 TYPE char30,
field4 TYPE char30,
percentage_type1 TYPE i,
percentage_type2 TYPE i,
END OF ty_table.
Now when I upload the data from excel, based on FACTOR TYPE, I will be creating a dynamic internal table <FS_TABLE>.
My LOOP AT GROUP logic,
" get the key field type to local variable
DATA(lv_factor_type) = <fs_factor_type>.
LOOP AT <fs_table> ASSIGNING FIELD-SYMBOL(<fs_line>)
GROUP BY SWITCH string( lv_factor_type WHEN 'TYPE1'
THEN | FIELD1 = <FS_LINE>-FIELD1 |
WHEN 'TYPE2' THEN | FIELD2 = <FS_LINE>-FIELD2 FIELD3 = <FS_LINE>-FIELD3 | )
ASCENDING ASSIGNING FIELD-SYMBOL(<fs_group>).
LOOP AT GROUP <fs_group> ASSIGNING FIELD-SYMBOL(<fs_mem>).
<fs_table_temp> = VALUE #( BASE <fs_table_temp> ( <fs_mem> ) ).
ENDLOOP.
DATA(lv_total) = REDUCE i( INIT value TYPE i
FOR <fs_val> IN <fs_table_temp>
NEXT value = value + SWITCH #( lv_factor_type
WHEN 'TYPE1' THEN (`<fs_val>-percentage_type1`)
WHEN 'TYPE2' THEN (`<fs_val>-percentage_type2`) ) ) .
IF lv_total > 100.
" Append that to the export table with the FACTOR TYPE AND TOTAL
ENDIF.
ENDLOOP.
Here,
The group by clause should differ based on the FACTOR_TYPE. I have to make it dynamic since I have more than 50+ combinations of grouping based on Factor_Type.
In the Reduce statement, I can't mention the fields PERCENTAGE_TYPE1 and PERCENTAGE_TYPE2 since <FS_VAL> is a dynamic structure.
Is there any way to achieve these?
Related
I have a model called Testing. The field called alias is a JSON field (a list really) and has values such as ["a", "b"] or ["d", "e"] and so on.
class Testing(db.Model):
__tablename__ = 'testing'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(25))
alias = db.Column(JSON)
def __init__(self, name, alias):
self.name = name
self.alias = alias
In my flask view I grab a url parameter that I want to use to filter Testing to get all Testing objects in which the parameter value is in the alias json list. So for example if url_param_value="a" I want all the Testing objects where "a" is in alias. So the alias value of ["a", "b"] would be a hit in this example.
Here is my approach but its not working and I assume it has to do with seralization.
Testing.query.filter(Testing.alias.contains(url_param_val)).all()
I am getting the below error
NotImplementedError: Operator 'contains' is not supported on this expression
The name field is a JSON type, not an array type. JSON columns don't have a contains method, even if you happen to be storing array data (how would the database know?)
In Postgres, you can use json_array_elements to expand a JSON array to a set of JSON values; this will return one row per element:
select id, json_array_elements(alias) as val from testing;
id | val
---------+--------------------
1 | "a"
2 | "b"
You can use that as a subquery to select records that contain a matching value:
select t.id, t.name, t.alias, cast(q.val as varchar)
from testing t, (
select id, json_array_elements(alias) as val
from testing
) q
where q.id=t.id and cast(q.val as varchar) = '"a"';
In SQLAlchemy syntax:
subq = session.query(
Testing.id,
func.json_array_elements(Testing.alias).label("val")
).subquery()
q = session.query(Testing).filter(
cast(subq.c.val, sa.Unicode) == '"a"',
subq.c.id == Testing.id)
Warning: this is going to be very inefficient for large tables; you're probably better off fixing the types to match your data, and then creating appropriate indexes.
I have a details table with adeet column defined as jsonb[]
a sample value stored in adeet column is as below image
Sample data stored in DB :
I want to return the rows which satisfies id=26088 i.e row 1 and 3
I have tried array operations and json operations but it does'nt work as required. Any pointers
Obviously the type of the column adeet is not of type JSON/JSONB, but maybe VARCHAR and we should fix the format so as to convert into a JSONB type. I used replace() and r/ltrim() funcitons for this conversion, and preferred to derive an array in order to use jsonb_array_elements() function :
WITH t(jobid,adeet) AS
(
SELECT jobid, replace(replace(replace(adeet,'\',''),'"{','{'),'}"','}')
FROM tab
), t2 AS
(
SELECT jobid, ('['||rtrim(ltrim(adeet,'{'), '}')||']')::jsonb as adeet
FROM t
)
SELECT t.*
FROM t2 t
CROSS JOIN jsonb_array_elements(adeet) j
WHERE (j.value ->> 'id')::int = 26088
Demo
You want to combine JSONB's <# operator with the generic-array ANY construct.
select * from foobar where '{"id":26088}' <# ANY (adeet);
I have a SQL table that stores data in Json format. I am using sample data below to understand the issue. Each document type has its own JSON structure.
DocumentID DocumentTypeID Status JsonData
----------------------------------------------------------------------------
1 2 Active {"FirstName":"Foo","LastName":"Bar","States":"[OK]"}
2 2 Active {"FirstName":"James","LastName":"Smith","States":"[TX,NY]"}
3 3 Active {"Make":"Ford","Model":"Focus","Year":"[2020]"}
4 3 Active {"Make":"Tesla","Model":"X","Year":"[2012,2015,2019]"}
then I have another JSON that needs to use in Where condition
#Condition = '{"FirstName": "James",LastName:"Smith","States":[TX]}'
I will also have DocumentTypeID as parameter
So in normal sql if i hard-code the property names then SQL would look something like
SELECT * FROM Documents d
WHERE
d.DocumentTypeID = #DocumentTypeID AND
JSON_VALUE(d.JsonData,'$.FirstName') = JSON_VALUE(#Condition,'$.FirstName') AND
JSON_VALUE(d.JsonData,'$.LastName') = JSON_VALUE(#Condition,'$.LastName') AND
JSON_QUERY(d.JsonData,'$.States') = JSON_QUERY(#Condition,'$.States') -- This line is wrong. I have
-- to check if one array is
-- subset of another array
Given
The property names in JsonData column and Condition will exactly match for a given DocumentTypeID.
I already have another SQL table that stores DocumentType and its Properties. If it helps, I can store json path for each property that can be used in above query to dynamically construct where condition
DocumentTypeID PropertyName JsonPath DataType
---------------------------------------------------------------------------------
2 FirstName $.FirstName String
2 LastName $.LastName String
2 States $.States Array
3 Make $.Make String
3 Model $.Model String
3 Year $.Year Array
ISSUE
For each document type the #condition will have different JSON structure. How do i construct dynamic where condition? Is this even possible in SQL?
I am using C#.NET so i was thinking of constructing SQL query in C# and just execute SQL Query. But before i go that route i want to check if its possible to do this in TSQL
Unfortunately, JSON support was only added to SQL Server in 2016 version, and still have room for improvement. Working with JSON data that contains arrays is quite cumbersome, involving OPENJSON to get the data, and another OPENJSON to get the array data.
An SQL based solution to this is possible - but a I wrote - cumbersome.
First, create and populate sample table (Please save us this step in your future questions):
DECLARE #Documents AS TABLE (
[DocumentID] int,
[DocumentTypeID] int,
[Status] varchar(6),
[JsonData] varchar(100)
);
INSERT INTO #Documents ([DocumentID], [DocumentTypeID], [Status], [JsonData]) VALUES
(1, 2, 'Active', '{"FirstName":"Foo","LastName":"Bar","States":["OK"]}'),
(2, 2, 'Active', '{"FirstName":"James","LastName":"Smith","States":["TX","NY"]}'),
(2, 2, 'Active', '{"FirstName":"James","LastName":"Smith","States":["OK", "NY"]}'),
(2, 2, 'Active', '{"FirstName":"James","LastName":"Smith","States":["OH", "OK"]}'),
(3, 3, 'Active', '{"Make":"Ford","Model":"Focus","Year":[2020]}'),
(4, 3, 'Active', '{"Make":"Tesla","Model":"X","Year":[2012,2015,2019]}');
Note I've added a couple of rows to the sample data, to verify the condition is working properly.
Also, as a side Note - some of the JSON data in the question was improperly formatted - I've had to fix that.
Then, declare the search parameters (Note: I still think sending a JSON string as a search condition is potentially risky):
DECLARE #DocumentTypeID int = 2,
#Condition varchar(100) = '{"FirstName": "James","LastName":"Smith","States":["TX", "OH"]}';
(Note: I've added another state - again to make sure the condition works as it should.)
Then, I've used a common table expression with openjson and cross apply to convert the json condition to tabular data, and joined that cte to the table:
WITH CTE AS
(
SELECT FirstName, LastName, [State]
FROM OPENJSON(#Condition)
WITH (
FirstName varchar(10) '$.FirstName',
LastName varchar(10) '$.LastName',
States nvarchar(max) '$.States' AS JSON
)
CROSS APPLY OPENJSON(States)
WITH (
[State] varchar(2) '$'
)
)
SELECT [DocumentID], [DocumentTypeID], [Status], [JsonData]
FROM #Documents
CROSS APPLY
OPENJSON([JsonData])
WITH(
-- Since we already have to use OPENJSON, no point of also using JSON_VALUE...
FirstName varchar(10) '$.FirstName',
LastName varchar(10) '$.LastName',
States nvarchar(max) '$.States' AS JSON
) As JD
CROSS APPLY OPENJSON(States)
WITH(
[State] varchar(2) '$'
) As JDS
JOIN CTE
ON JD.FirstName = CTE.FirstName
AND JD.LastName = CTE.LastName
AND JDS.[State] = CTE.[State]
WHERE DocumentTypeID = #DocumentTypeID
Results:
DocumentID DocumentTypeID Status JsonData
2 2 Active {"FirstName":"James","LastName":"Smith","States":["TX","NY"]}
2 2 Active {"FirstName":"James","LastName":"Smith","States":["OH", "OK"]}
I have the following case statement to prepare as a dynamic as shown below:
Example:
I have the case statement:
case cola
when cola between '2001-01-01' and '2001-01-05' then 'G1'
when cola between '2001-01-10' and '2001-01-15' then 'G2'
when cola between '2001-01-20' and '2001-01-25' then 'G3'
when cola between '2001-02-01' and '2001-02-05' then 'G4'
when cola between '2001-02-10' and '2001-02-15' then 'G5'
else ''
end
Note: Now I want to create dynamic case statement because of the values dates and name passing as a parameter and it may change.
Declare
dates varchar = '2001-01-01to2001-01-05,2001-01-10to2001-01-15,
2001-01-20to2001-01-25,2001-02-01to2001-02-05,
2001-02-10to2001-02-15';
names varchar = 'G1,G2,G3,G4,G5';
The values in the variables may change as per the requirements, it will be dynamic. So the case statement should be dynamic without using loop.
You may not need any function for this, just join to a mapping data-set:
with cola_map(low, high, value) as (
values(date '2001-01-01', date '2001-01-05', 'G1'),
('2001-01-10', '2001-01-15', 'G2'),
('2001-01-20', '2001-01-25', 'G3'),
('2001-02-01', '2001-02-05', 'G4'),
('2001-02-10', '2001-02-15', 'G5')
-- you can include as many rows, as you want
)
select table_name.*,
coalesce(cola_map.value, '') -- else branch from case expression
from table_name
left join cola_map on table_name.cola between cola_map.low and cola_map.high
If your date ranges could collide, you can use DISTINCT ON or GROUP BY to avoid row duplication.
Note: you can use a simple sub-select too, I used a CTE, because it's more readable.
Edit: passing these data (as a single parameter) can be achieved by passing a multi-dimensional array (or an array of row-values, but that requires you to have a distinct, predefined composite type).
Passing arrays as parameters can depend on the actual client (& driver) you use, but in general, you can use the array's input representation:
-- sql
with cola_map(low, high, value) as (
select d[1]::date, d[2]::date, d[3]
from unnest(?::text[][]) d
)
select table_name.*,
coalesce(cola_map.value, '') -- else branch from case expression
from table_name
left join cola_map on table_name.cola between cola_map.low and cola_map.high
// client pseudo code
query = db.prepare(sql);
query.bind(1, "{{2001-01-10,2001-01-15,G2},{2001-01-20,2001-01-25,G3}}");
query.execute();
Passing each chunk of data separately is also possible with some clients (or with some abstractions), but this is highly depends on your driver/orm/etc. you use.
I've got two tables that I'd like to combine into a view. The first table contains the structure:
Template Table
componentID | title
======================
1000 | blue
1001 | red
1002 | orange
The second table contains the actual data that will be stored, and the columns reference the ID of the first table:
Data Table
id | field1000 | field1001 | field1002
======================================
1 | navy | ruby | vermilion
2 | midnight | crimson | amber
What I'd like to get as a result in a view:
Combined Table/View?
id | blue | red | orange
=================================
1 | navy | ruby | vermilion
2 | midnight | crimson | amber
Is this possible? I've been trying to get it to work with pivot tables, but I'm getting hung up on how to use the titles as the columns for the data.
Ok, I went a bit overboard with this one but this will do what you want. This procedure will combine all fields with the proper data table columns, and does not need to know nor care how many columns there are in the data tables.
It does not use cursors, but due to the possibility of many template tables, it does use Dynamic SQL to generate the Select statement for the final return.
Only caveat is it's not a View, it's a stored procedure, because it allows to pass the variable for the data table you want to ultimately select from.
The assumptions:
The template table is static
There is one template table for all data tables
All fields in any data tables must be unique *
All data tables have a PK/identity field with the word 'id' in it that must be ignored
All fields in the data tables have a corresponding title in the template table
All fields in the data table are prefixed with the word 'field' and all of the reference ID's in the template table correspond to those field names with 'field' removed, based on your example
*- It can of course be improved by modifying the template table schema to also have a field for the data table that the field title belongs to, for example, which would remove this assumption #3.
The process:
First we need a mapping of the field names, reference IDs, and column titles. We do this with a table variable and get our info from syscolumns. Then, we update our temp table to get the titles from the TemplateTable table.
Then, we need to build a dynamic Select list from the DataTable (which is a parameter in the SP and therefore requires some dynamic SQL to execute). My preferred method of doing this is by having a bit column in my source table that I can update, something like 'IsCompleted', and then using a regular While loop to get through each row. Inside the While loop, all we do is grab the current "TitleReference" from our temporary table variable, and append to the select list the real field name from syscolumns (from first step above).
Finally, we execute the dynamic SQL statement which has a Select, and when this is inside a stored procedure that is executed, the result is returned as the result of the stored procedure.
The Full Working Code
Create Procedure usp_CombineTables
(
#DataTableName varchar(50)
)
As
-- Test
-- Exec usp_CombineTables 'DataTable'
-- Set up our variables
Declare #DataTableIdFieldName varchar(50), -- The ID field of the data table, dynamic
#IsCompleted bit, -- Used by While loop to know when to exit
#CurrentTitleReference int, -- Used in While loop as the ID from TemplateTable that relates to the real data field name and the desired title
#CurrentDataFieldName varchar(50), -- Used in While loop for the current actual field name in the data table
#CurrentTitle varchar(50), -- Used in While loop for the desired field name in the resulting table of the stored proc
#DynamicSelectQuery varchar(2000) -- Stores the SQL query that is dynamically built and executed for the final result; can increase value if needed
-- Use table variable to correlate the datatable columns, titles, and references
Declare #TitleReferences Table (
TitleReference int,
DataTableColumnName varchar(50),
Title varchar(50),
Completed bit default 0
)
-- Get the info from syscolumns about our datatable; assumes that all of the field names are prefixed with the word 'field' which needs to be removed
Insert Into #TitleReferences (
TitleReference,
DataTableColumnName
)
Select
Replace(name, 'field', '') As TitleReference,
name As DataTableColumnName
From syscolumns
Where id = OBJECT_ID(#DataTableName)
And name Not Like '%id%' -- assumes DataTable will always have a PK with 'id' in it, need to ignore/remove
-- Get the titles -- assumes only one template table for all data tables; all data fields accross tables must be unique
Update #TitleReferences
Set Title = t.Title From TemplateTable As t
Where TitleReference = t.ComponentID
-- Get the ID field of the data table
Set #DataTableIdFieldName = (
Select name From syscolumns
Where id = OBJECT_ID(#DataTableName)
And name Like '%id%')
-- Build a dynamic SQL query to select from the datatable with the right column names
Set #DynamicSelectQuery = 'Select ' + #DataTableIdFieldName + ', ' -- start with the ID
Set #IsCompleted = 0
While (#IsCompleted = 0)
Begin
-- Retrieve the field name and title from the current row based on title reference
Set #CurrentTitleReference = (Select Top 1 TitleReference From #TitleReferences Where Completed = 0)
Set #CurrentDataFieldName = (Select DataTableColumnName From #TitleReferences Where TitleReference = #CurrentTitleReference)
Set #CurrentTitle = (Select Title From #TitleReferences Where TitleReference = #CurrentTitleReference)
-- Append the next select field to the dynamic query
Set #DynamicSelectQuery = #DynamicSelectQuery +
#CurrentDataFieldName + ' As ' + QuoteName(#CurrentTitle)
-- Set up to move past current record in next iteration
Update #TitleReferences Set Completed = 1 Where TitleReference = #CurrentTitleReference
-- Exit loop or add comma for next field
If (Select Count(Completed) From #TitleReferences Where Completed = 0) = 0
Begin
Set #IsCompleted = 1
End
Else
Begin
-- Add comma to select field for next column
Set #DynamicSelectQuery = #DynamicSelectQuery + ','
End
End
-- Now the column list is built, just add the table and exec
Set #DynamicSelectQuery = #DynamicSelectQuery +
' From ' + #DataTableName
Exec(#DynamicSelectQuery)
The Result
Hope this helps, it was fun writing it!
something on these lines
DECLARE #f0 VARCHAR(50)=(SELECT title FROM template WHERE componentID=1000)
DECLARE #f1 VARCHAR(50)=(SELECT title FROM template WHERE componentID=1001)
DECLARE #f2 VARCHAR(50)=(SELECT title FROM template WHERE componentID=1002)
#sql='SELECT field1000 AS ' + quotename(#f0) + ' field1001 AS ' + quotename(#f1) + ' field1002 AS ' + quotename(#f2) + ' FROM data'
exec sp_executesql #sql