As per the official documentation, it depicts as though we can insert into multiple tables from a task. Which sounds inaccurate since
Once consumed the offsets of the stream are reset
It is possible to execute only one SQL statement from a task
am I missing something here? I want to be able to insert into 2 tables reading out of a stream through the task.
You can do this with a multi-table insert:
https://docs.snowflake.com/en/sql-reference/sql/insert-multi-table.html
You can do this. Multi-table inserts are one way, but there is another.
The pointer in the stream is only advanced at the end of a transactions. Therefore, you can enclose multiple DML statements that read from the stream in a single transaction. Unfortunately, tasks can only execute a single SQL statements, so you will have to embed your queries in a stored procedure.
Hope this helps.
Related
I have a CICS program, which will read a DB2 table to obtain the rules based of the field name. Let's say my record type is AA and this type will have at least 20 rules that I need to do loop in DB2 tables. Like wise I have few record types and many more rules tied to each type.
I get data from MQ and for each record type I call separate CICS program. So when I have to process high load, DB2 rules table is getting held by so many program and this causing performance issue.
I want to get away from DB2 and load this rules in CICS Container and maintain periodically. But I'm not sure if this will work. I don't want to use or create VSAM's. I'm looking for some kind of storage I could use and maintain in CICS.
My question is. If I create a pipeline and container will I able to access them by multiple program at a same time and will data stored rules stay in Container after successful get?
Before reading further, please understand that DB2 solves all the sharing and locking problems very efficiently. I've never encountered a problem with too many transactions trying to read a DB2 table concurrently. Updating, yes; a mix of updates and reads, yes; just reading, no.
So, in order to implement your own caching of a DB2 table inside CICS you need a data store. As #BruceMartin indicates, a TS queue is an option, I would say that given your other constraints it is your only option.
In order to automate this you must create a trigger on your DB2 table that fires after INSERT, UPDATE, or DELETE. The trigger must cause the TS queue to be repopulated. The repopulation mechanism could be EXCI or MQ, as the code performing the repopulation must execute within CICS.
During the repopulation, all transactions reading the TS queue must wait for the repopulation to complete. This can be done with the CICS ENQ API, with a caveat. In order to prevent all these transactions from single-threading through their TS queue read due to always ENQing, I suggest using two TS queues, one holds the DB2 data and the other is a "trigger" TS queue. The contents of the trigger TS queue are not significant, you can store a timestamp, or "Hello, World", or "ABC" it doesn't matter.
A normal transaction attempts a read of the trigger TS queue. If the read is unsuccessful the transaction simply reads the TS queue with the DB2 data. But if the read is successful then repopulation is in progress and the transaction ENQs on a resource (call it XYZ). On return from the ENQ, DEQ and read the TS queue with the DB2 data.
During repopulation, a program executed by the trigger on the DB2 table executes in CICS. First ENQing on resource XYZ, then creating the trigger TS queue, then deleting the TS queue with the DB2 data, then creating the TS queue and populating it with the new DB2 data, deleting the trigger TS queue, finally DEQing resource XYZ. I would strongly suggest using a multi-row SELECT to obtain the DB2 data as it is significantly more efficient than the traditional OPEN CURSOR, FETCH, CLOSE CURSOR method.
I have a pipeline that ingests data from Kusto, does some simple transformation, and flows the data to SQL. It will be run once per day, and needs to clear the sink tables in SQL. I thought this would be straightforward (and probably is) but I can't figure out how to do it. Thanks for any assistance!
As #wBob said, if you are using Copy activity in ADF, we can enter TRUNCATE TABLE <your-table-name> at Pre-copy script. It will execute the T-SQL script here before copying.
You have to write a stored procedure on prior to transformation, which can delete your staging data.
Stored procedure->do transformation
I've a procedure in Oracle PL/SQL which fetches transactional data based on certain condition, then performs some logical calculations. I used cursor to store the SQL and then I used FETCH (cursor) BULK COLLECT INTO (table type variable) LIMIT 10000, iterated over this table variable to perform calculation and ultimately storing the value in a DB table. Once 10000 rows have been processed, query will be executed to fetch next set of records,
This helped me limiting number of times SQL is executed via cursor and limiting the number of records loaded into memory.
I am trying to migrate this code to plpgsql. How can I achieve this functionality in plpgsql?
You cannot achieve this functionality in PostgreSQL.
I wrote an extension https://github.com/okbob/dbms_sql . It can be used for reduce of necessary work related to migration from Oracle to Postgres.
But you don't need this feature in Postgres. Although PL/pgSQL is similar to PL/SQL, the architecture is very different - and bulk collect operations are not necessary.
I need to insert a Big amount of data(Some Millions) and I need to perform it Quickly.
I read about Bulk insert via ODBC on .NET and JAVA But I need to perform it directly on the Database.
I also read about Batch Insert but What I have tried have not seemed to work
Batch Insert, Example
I'm executing a INSERT SELECT but it's taking something like 0,360s per row, this is very slow and I need to perform some improvements here.
I would really appreciate some guidance here with examples and documentation if possible.
DATABASE: SYBASE ASE 15.7
Expanding on some of the comments ...
blocking, slow disk IO, and any other 'wait' events (ie, anything other than actual insert/update activity) can be ascertained from the master..monProcessWaits table (where SPID = spid_of_your_insert_update_process) [see the P&T manual for Monitoring Tables (aka MDA tables)]
master..monProcessObject and master..monProcessStatement will show logical/physical IOs for currently running queries [again, see P&T manual for MDA tables]
master..monSysStatement will show logical/physical IOs for recently completed queries [again, see P&T manual for MDA tables]
for UPDATE statements you'll want to take a look at the query plan to see if you're suffering from a poor join order; also of key importance ... direct (fast/good) updates vs deferred (slow/bad) updates; deferred updates can occur for many reasons ... some fixable, some not ... updating indexed columns, poor join order, updates that cause page splits and/or row forwardings
RI (PK/FK) constraints can be viewed with sp_helpconstraint table_name; query plans will also show the under-the-covers joins required when performing RI (PK/FK) validations during inserts/updates/deletes
triggers are a bit harder to locate (an official sp_helptrigger doesn't show up until ASE 16); check the sysobjects.[ins|upd|del]trig where name = your_table - these represent the object id(s) of any insert/update/delete triggers on the table; also check sysobjects records where type = 'TR' and deltrig = object_id(your_table) - provides support for additional insert/update/delete triggers (don't recall at moment if this is just ASE 16+)
if triggers are being fired, need to review the associated query plans to make sure the inserted and deleted tables (if referenced) are driving any queries where these pseudo tables are joined with permanent tables
There are likely some areas I'm forgetting (off the top of my head) ... key take away is that there could be many reasons for 'slow' DML statements.
One (relatively) quick way to find out if RI (PK/FK) constraints or triggers are at play ...
set showplan on
go
insert/update/delete statements
go
Then review the resulting query plan(s); if you see references to any tables other than the ones explicitly listed in the insert/update/delete statements then you're likely dealing with RI constraints and/or triggers.
I want to make a loop with one INSERT query for each fetched row from a previous query with prepared statements. In a more visual way:
MAKE QUERY
While Fetching
- Make new query
I can't close my statement since I need it to keep working...
The problem is that when we use prepared statements, we have to fetch all data before doing a new prepare(). So, how could I do this? Maybe without statements, but it's not a nice solution.
You are going to kill your DB (and if you have a DBA your DBA is going to kill you) if you try to do this. The problem is that what you want to do is send one inser request per line to the database. You have to create and dispose of all these commands over and over again for each line of data. That is expensive.
If you are hard-set on doing it, Nothing prevents you from creating a second prepared statement (with a different name of course) within a loop which reads from the first, but I highly advise against it. At the very least, buffer your incoming data and insert a few hundred rows at a time.