I am trying to find out how to do dynamic creation of tables in MySQL using Talend.
In other ETL tools such as Pentaho they have a specific component called "metadata" to do this.
So my use case is the following:
1) Create database manually in MySQL
2) Use Talend to read CSV Header info, and use this as the fields of a table to be created in MySQL using Talend.
I have searched and could not find anything for this specific feature online.
[Note : using Talend Open Studio for Big Data Version: 6.4.1]
UPDATE:
I have made progress on this, but running into issues trying to generate a primary key on the MySQL output using the NumericSequence function.
My data does not have natively a primary key, but wish to include one:
I have a screenshot of the tMap attached here:
Also I am getting a compile/build error. It seems it does not like the primary key generation it seems.
See attached image.
And here also is the tMySQLOutput settings for Primary Key:
More Work Done:
I have changed the name of the Primary Key in the tMySQL component to match the name of the tMap output area component, i get the same compile error.
I will attach this error here:
To create dynamically MySQL tables in Talend, you can use one of the "Action on table" options in tMysqlOutput components (in your case, "Create table...").
https://help.talend.com/reader/4I8tDQGtrOPDl5MXAS3Q~w/aDNKleHXlevILu9pnbCoNg
Don't forget to define correctly PK fields for further inserts, updates, deletes...
Then, if necessary, through your favorite Database Tool (MySQL Workbench, DBeaver or other), you can retrieve the DDL (and DML) script(s).
I hope this answers to your problem/question.
Related
I have one already existing Talend Open Studio tMySQLInput component with some sql code inside it, in order to retrieve some joined columns linked to a tMySQLOuput component (pointing to an already existing MySQL table) with few records.
QUESTION:
Will the "tMySQLInput" component overwrite the already existing table data that the tMySQLOutput component relates to? I mean is there an option to check in the tMySQLInput our output in order to say, overwrite each time this job is executed ?
Thank you all.
Yes, there is an option where in tMySQLOutput where you can specify what action you want to do to your table. Follow following steps:
Go to component tab of tMySQLOutput, it will open the basic settings of this component.
If you will look closer you will find Action on table. This is the action which you can perform on the table which is pointed by tMySQLOutput. It has options as Default, Drop and Create Table etc.
Then you have Action on data. These are the options which you can perform on the data like Insert, Update etc.
In your case I suppose you can choose Action on Table as Default and Action on Data as Insert. Default action would not do anything on the table and Insert option would insert the records at the end of table. But in case of Insert if you will have duplicate rows then job would stop the moment it will find any duplicate row.
I am following the standard steps to create date dimension in SSAS here-
The generate schema option fails! this is the message from schema generation wizard
create failed for table myusername.dimdate
I think the problem is that the table should be created for a different schema. If below is the selected option for schema generation -
Subject Area
Data source view: ABC PROJ
**Schema: ABC_PROJ(WXY1230)**
Data will be preserved
Time table will be populated
Shouldn't the Dimdate table be created ABC_PROF(WXY1230).Dimdate and not under myusername.DimDate?
Any idea what could be missing? I have been struggling with this for hours now.
Thanks for any help.
I think that is an unfortunate bug in the SSAS generated date dimension. When you generate the dimension you are using your Windows credentials and SSAS uses the default schema for the user context it was provided. I believe your user has a default schema of myusername on that database, which is what is causing the date dimesion to be created there.
If you create the dimension, you have the option to generate the schema now or generate it later. If you choose to generate it later and go through the Schema Generation Wizard, you eventually see a screen that shows the Owning Schema.
. There is no mechanism to change it even though you can see it (At least not in SSDT. I don't have BIDS on any of my machines). You are not the only person to run into this issue.
So you have a few options as a workaround:
Use a different account that has a default schema that is the correct schema where you want to place your date table.
Create the table under your default user and populate it. Then alter the table in SQL Server to move it to the correct schema. Then update your DSV.
Don't use SSAS to generate your date dimension. There are plenty of scripts out there that create great date dimensions. Here's an example. This means you will need to create the dimension and build the hierarchy yourself, but it's not that much work compared to the time you've spent trying to fix this issue.
I'm using Filemaker Pro 12 and I was wondering if there is a way of creating a template for tables. There are a number of fields I'm placing in my tables that are identical utility-fields like modification time-stamp, active/inactive flags, etc. I was hoping there was a way that I could define the skeleton of each table somehow instead of having to manually add these identical fields every time.
If you are using the Advanced version, you can copy&paste fields among tables/files.
Using the regular version, you can import records from your "default" table and specify [New Table...] as the target table. This will recreate the source table's structure in the target file. The source table does not have to contain any records for this to work.
To expand a little bit on michael-hor257k's answer, if you're using FileMaker Pro Advanced, a good practice is to create a "Default" table that has your core utility fields. When you want to make a new table in Manage Database, instead:
Highlight the Default table,
Copy & Paste the table, then
Rename the new table.
I have a database with about 50 something tables. I would like to run the report "Data Dictionary" on each table.
Ideally, I would like them all to be in one report, for example, in PGAdminIII, if I right click select "Tables" I will get a report of all the 'objects' and under each one a data dictionary report.
Is there an automatic way of doing this, or an plugin that I can install to postgres? Or is there something analogous to this?
If I understand correctly, you're referring to the ability to right mouse click on a table in PgAdminIII and select Reports > Data Dictionary report?
I'm not aware of any way to do that from PgAdminIII. You could look into using a different tool such as SchemaSpy. Another option (as alluded to by #kgrittn) is to use psql \d with the \H flag to generate html output. My solution (since SchemaSpy didn't do what I needed, and I needed the same output for both Postgres and Oracle) was to roll my own using perl, DBD::Pg and Template::Toolkit.
Update: Added GitHub link.
I wrote a fairly simple Postgres data dictionary generator in Python that spans all schemas and tables within a specified database. If it doesn't have exactly what you want it would be fairly easy to modify.
https://github.com/kylejmcintyre/pypostgreports
I'm sure this is something very simple, but for the life of me can't
find the correct keywords on google.
Basically I've updated a couple models since my last deployment. Dev is set up with the jpa.ddl default setting of create-drop. Now I read that prod isn't supposed to run with jpa.ddl=update, so does that mean I have to manually script a schema change? I couldn't find any documentation saying the correct way. I am also using playapps, so the database is set up there. I set up ssl, so I should have sql access via command line. However, I was having difficulty figuring out the syntax for modifying the db. I'm so used to a gui environment such as phpMySQL or microsoft's sql server. The errors specifically I'm getting are the following two (when running the application after uploading to prod).
Unsuccessful: alter table PhotoSlide add index FK57E3FABF5C905145 (aPhoto_id), add constraint FK57E3FABF5C905145 foreign key (aPhoto_id) references StorePhoto (id)
Cannot add or update a child row: a foreign key constraint fails (play/#sql-2e29_32, CONSTRAINT FK57E3FABF5C905145 FOREIGN KEY (aPhoto_id) REFERENCES StorePhoto (id))
you can achieve that by using the migration module.More details you can find #
http://www.playframework.org/modules/migrate
The flow will be like the below:
You push new code to prod --> you run migrations --> restart the server
More documentation at :
https://github.com/dcardon/play-migrate/blob/master/documentation/manual/home.textile
With the latest version play supports migrations :
http://www.playframework.org/documentation/1.2.4/evolutions