I need to bulk load from csv to db2 database, i tried with tFileInputDelimited but it took 7 hours for 199million rows, can someone explain how to use it?
Is the template that I use correct?
job testing
Related
I have a job which runs everyday and based on column 'modifyts' pulls the records from db2 as delta: modifyts>current_date-1.
Recently we found out that some of the data is getting missed and not loaded to our netezza target table.
Is there a way we can get the missing data?
As of now, we are planning to load for past 3 days with condition modifyts>current_date-3 but it could cause data error in other process. Is there an efficient way to achieve this?
Any suggestions will greatly help. thanks in advance
I need to process millions of records coming from MongoDb and put a ETL pipeline to insert that data into a PostgreSQL database. However, in all the methods I've tried, I keep getting the out memory heap space exception. Here's what I've already tried -
Tried connecting to MongoDB using tMongoDBInput and put a tMap to process the records and output them using a connection to PostgreSQL. tMap could not handle it.
Tried to load the data into a JSON file and then read from the file to PostgreSQL. Data got loaded into JSON file but from there on got the same memory exception.
Tried increasing the RAM for the job in the settings and tried the above two methods again, still no change.
I specifically wanted to know if there's any way to stream this data or process it in batches to counter the memory issue.
Also, I know that there are some components dealing with BulkDataLoad. Could anyone please confirm whether it would be helpful here since I want to process the records before inserting and if yes, point me to the right kind of documentation to get that set up.
Thanks in advance!
As you already tried all the possibilities the only way that I can see to do this requirement is breaking done the job into multiple sub-jobs or going with incremental load based on key columns or date columns, Considering this as a one-time activity for now.
Please let me know if it helps.
I a developing a POC to import some data from MongoDB to MS SQL Server 2012. For e g. there are 5 collections n Mongo which need to be loaded to 5 tables in MS SQL Server tables. The data needs to be dumped as is.
What is the best approach to achieve this ? Wil any ETL Tool (SSIS) suffice or shall we have to write code in Node jS? Can anyone point me in the right direction.
Please advise.
Thanks in advance.
I use ZappySys' SSIS PowerPack, which includes MongoDB Source and Destination connections.
well my problem is, how could i copy a database with talend from postgresql to sap hana without needing to write a job for every table ?
The reason for this is, because it could take some long time to prepare all those jobs, while taking in consideration, having at least 200 tables, which at least have 30 columns.
I tried tTransferDatabase plugin, but i can't success to transfer it to sap hana, it gives me an error that it can't copy schema (while it successfully worked copying it to other database in postgresql), and i am sure that the schemas names are right.
here is the error:
Exception in component tTransferDatabase_1
java.lang.NullPointerException
at org.apache.ddlutils.PlatformFactory.createNewPlatformInstance(PlatformFactory.java:86)
at org.apache.ddlutils.PlatformFactory.createNewPlatformInstance(PlatformFactory.java:124)
at com.devjpcb.transferdatabase.TransferDatabase.getPlatformDestine(TransferDatabase.java:179)
at com.devjpcb.transferdatabase.TransferDatabase.copySchemaToDatabase(TransferDatabase.java:249)
at local_project.aaasa_0_1.aaasa.tTransferDatabase_1Process(aaasa.java:836)
at local_project.aaasa_0_1.aaasa.runJobInTOS(aaasa.java:1130)
at local_project.aaasa_0_1.aaasa.main(aaasa.java:951)
Is there maybe a chance to do sth like .. for each table in connection, table guess schema, copy columns from table to other side of tmap, run ?
Any advice would be helpful ;), Thank you !
With some work, you could use the example job created by rbaldwin on Talend Exchange; note that it starts with files, not a database. But you could easily create a job that loops through all your database tables and does an extract to file, to then use as the starting point.
Another option is Bekwam's solution
I am learning oracle myself with help of internet...
Now, for some scenario I need thousands of records which should be available in my table.
It is not possible to create thousands of records manually...
Is there any tools or any other way to do the same in ORACLE 10g...
As I said I am a novice to Oracle I need some advices from you SOF professionals....
Thanks in advance...
This database has a JDBC driver. Download Eclipse, add this driver to the path and write ten lines of code to insert as much blabla as required, here tutorial. Even if you have never programmed Java before and would not try again, easy enough to do.