My requirement is to do a batch update to a table. I was able to do a Batch insert but could not find a way to do the batch update in Slick3. Any sample or link to a document will be very help full. Tried to search the web could not find a solution. Using Slick 3 on Postgresql.
Why don't you just add whatever data you need to perform to a seq and perform the query. That will do the batching for you. Link here . This applies to all operations not only insert.
Look at this SO question to fall back on standard JDBC and do the update batching.
Also there are some items the team is working on. See here.
Related
I am evaluating Talend and I have little experience using the tool. My question is regarding audit. Does Talend offer a way to run with an audit option that extracts the source data, transforms it and then compares the transformed data to the existing target data? I would like a report that tells me which records would have triggered an update if the audit option had not prevented it.
I know that ETL's are triggered on change but I would want to run this audit without regard to changed timestamps. That is, a query would specify what source records to audit.
I have done a search of the talend documentation but nothing jumped out at me.
Thank you.
EDIT
Looking outside of Talend, I found where Tim Mitchell wrote:
While I have not yet found an auditing framework with enough
customization to suit me, I have learned that it is possible to audit
ETL processes in a customized way without reinventing the wheel every
time. I keep on hand a common (but never set in stone) set of table
definitions and supporting logic to shorten the path to the following
auditing objectives:
Simple aggregate auditing of row counts, financial data, and other key metrics
Documentation templates for source-to-target mappings and transformation logic
Lightweight auditing of batch-level changes
Full auditing of every change made in the transformation process
So I guess that auditing is on me.
I am currently creating a Form on access however when I go into form view I am unable to update any of the current records or add new records would anyone have any idea why I cant edit or update my records? or how I go about fixing this?
My Form is currently linked to a query, and incase it matters I also have inlcuded a search function in my form.
Any help/advice would be greatly appreciated.
Thanks
Paula
It means that the query you use is not updateable. There are a lot of limitations for updateable queries design, for instance you cannot use aggregaing in query, joins should have unique keys etc. Try to redesign your query. As a workaround you can copy the data from your query to temporary table, edit the data in this table and then copy data back to main table(s)
Your query is probably not updateable. You can check this by simply opening the query directly, and trying to edit/add data.
The most common reason is a JOIN on non-indexed columns.
For more reasons see: Dealing with Non-Updateable Microsoft Access Queries
or Allen Browne: Why is my query read-only?
I use MS data access application block for interaction with database and I saw its performance is good. When I like to add 100 or more records then I send those 100 records in xml format to a stored procedure and from there I do a bulk insert. Now I have to use Entity Framework. I haven't ever used EF before so I am not familiar with EF and how it works.
In another forum I asked a question like "How Entity Framework works in case of batch insert and update data" and got answer
From my experience, EF does not support batch insert or batch update.
What it does is that it will issue an individual insert or update statement, but it will wrap all of them in a transaction if you add all of your changes to the dbcontect before calling SaveChanges().
Is it true that EF can not handle batch insert/update? In case of batch insert/update EF inserts data in loop? If there are 100 records which we need to commit at once then EF can not do it?
If it is not right then please guide me how one should write code as a result EF can do batch insert/update. Also tell me the trick how to see what kind of SQL it will generate.
If possible please guide me with sample code for batch insert/update with EF. also tell me which version of EF support true batch operation. Thanks
Yes EF is not a Bulk load, Update tool.
You can of course put a a few K entries and commit (SaveChanges)
But when you have serious volumes of speed is critical, use SQL.
see Batch update/delete EF5 as an example on the topic
I need to insert a table from a master table having 2 billion records . Insert needs to satisfy some conditons and also in the some columns to be calculated and then it has to be inserted.
I am having 2 options but I dont know which to follow to improve performance.
1 option
Create a cursor by filtering from master table with the conditons. and get one by one record for caluclation and then last insertion to the child table
2 option
insert first using into conditon and then calculation using update statement.
Please Assist.
Having a cursor to get data, perform calculation, and then insert into the database will be time consuming. My guess is that since it involves data connections and I/O for each retrieval and insertion (for both the databases )
Databases are usually better with bulk operations, so it will definitely give you better performance if you use Option 2. Option 2 is better for troubleshooting also ( as the process is cleanly separated - step1: download, step2: calculate) than Option 1 where in case of an error in the middle of the process, you'll be forced to redo all the steps again.
Opening a cursor and inserting records one by one might have serious performance issues at the volumes on the order of a Billion . Especially if you have a weak network between your Database tier and App tier . The fastest way to do this could be to use Db2 export utility to download data , let the program manipulate the data from the file and later load the file back to the child table . Apart from the file based option you can also consider the following approaches
1) Write an SQL stored procedure (No need to ship the data out of the database to make changes )
2) If you using Java/JDBC use Batch Update feature to update multiple records at the same time
3) If you using a tool like Informatica, turn on the bulk load feature in informatica
Also see the IBM DW article on imporving insert performance . The article is a little bit older but concepts are still valid . http://www.ibm.com/developerworks/data/library/tips/dm-0403wilkins/
I have a question regarding a semi-constant update in a database. In short it is regarding a checkout function on a web page, which each time the checkout function is evoked it do five steps.
I want to try to optimize this function and have my eye on a step where I update a table each time the checkout is performed. I take the information retrieved from the shopping cart and then update the table in question.
I do have some indexes on the table, the gain from those are greater than leaving them so this is a cost I’m willing to take.
Now, my question is. Could it in some way regarding to performance be better to not update the table instantly but collect every checkout items and save them in some way (maybe in a file) and then at a specific time (or several times) at day take this file and then update the table with the new information.
Then I started thinking about if there was a possibility to use some sort of Bulk Update to take a file, hashmap, array (or?) and then update it.
And I’m using IBM DB2 version 9.7
Mestika
You will lose the ability to do transactions, or to recover from failure after a step midway, so I would avoid using this approach. You could try using prepared statements, or batch updates offered by JDBC 2.0 where multiple statements are submitted to the DB as a single unit.