i have a big job defined in talend studio. It works
well. But now I have to make another job with only
some changes in prefixes in sqlrequests as destination
tables.
My plan was to export the job and edit the job in my
emacs editor and use the easy way to find and replace
the prefix and reimport this job in talend with another
name. I tried it. But talend studio shows me a dialog
with invalid job message.
Is it possible in general to edit jobfiles (jobname.item and jobname.properties) utside of talend studio and reimport it?
Or are there issues and mechanism in talend which prevent this
way like protect jobfiles with hashes and so on?
thanks
It is possible to edit an item file, but risky.
I advise you to duplicate your big job in Talend Studio, copy the content of the item file from your big job and paste it into the new one with the correction.
Related
I just installend Talend Open Studio for Big Data version 6.4.1, and I created a job with tHiveConnection component. It has some missing modules that I have to install. Is anybody have a link with description of what is the best way to do it? Should I install them one by one (there is quite a lot of them)?
I just started working on a new project. We are building a new application from scratch. Team started with a brand new schema. I wanted to automate the database build process, so I started looking for the options. Flyway seems to be a good one. I have been playing around a bit and found some limitations of the tool. Perhaps, someone will be able to help.
We have the following directory structure for SQL files:
SQL
-- DDL
-- DML
-- PACKAGES
We are doing agile development, so file names are based on the sprint number. The file naming convention we are using is:
Sprint#_script#_userstory#_description
For example:
S1_01_US123_CreateNewTable.sql
S1_02_US123_AddConstraint.sql
Next sprint:
S2_01_US456_AddColumn.sql
And so on...
I setup the JDBC parameter and I am able to connect. I tested basic things like: clean, repair, info and migrate with couple of test scripts and that worked like a charm. I started to run into issues when I tried deploying all the scripts. Issues like:
- It didn't like single underscore.
- It didn't like the file names starting with S1_01_*, rest of the file name is different and they are in different folders.
I have the following questions:
Can I build using Flyway without having to rename the files?
How can I get it to deploy in this order:
DDLs
DMLs
Packages (everytime I deploy). And we have a separate header and body files, so deploy header first as well.
Can I change the structure of schema_version table?
Can I do selective clean? Like flag some of the objects to not to be dropped?
My main concern is running DDLs before everything else. If I can accomplish that, then I can start using Flyway and learn as I go.
Thanks in advance.
Harbinder
Can I build using Flyway without having to rename the files?
Maybe. Experiment with the flyway.sqlMigrationSeparator property. Try "_US" which will break after the script number. You'll also need to set flyway.sqlMigrationPrefix=S.
How can I get it to deploy in this order: DDLs, DMLs, Packages (everytime I deploy). And we have a separate header and body files, so deploy header first as well.
Specify multiple locations (separated with comma) and ensure the version numbering ordering makes sense as if these files were all in the same directory. If running from the command line, turn on debug with -X to see how flyway collects the migrations.
Additionally, if possible, you should consider renaming your packages as a Repeatable migration (default:R) so that you just need to change the contents of the file for flyway to pick it up.
Can I change the structure of schema_version table?
No. This is managed by flyway.
Can I do selective clean? Like flag some of the objects to not to be dropped?
No. In this situation it might be best to set flyway.cleanDisabled=true to stop accidental mistakes. There are callbacks before and after clean if you wish to do extra cleaning but I don't think you can restrict clean itself without delving into the code.
Good luck!
This question already has an answer here:
SSIS/SQL Automation
(1 answer)
Closed 6 years ago.
My first question on Stack overflow.
Here's my problem: I'm wanting to be able to execute a SSIS packages (on it's own) when a file has been placed in a folder. Preferably without the Visual Studio/BIDS application open. I also want it to reset itself after the package has run successfully so that it begins monitoring the folder again for the next day's file drop to repeat the process. After hours of research I haven't found any information on a complete "Lights-out" (No human interaction necessary) solution.
Here's what I have so far: I have a WMI component in my control flow monitoring the folder for the file, it works, but I still have to manually go into the project and then run it so that it will start the monitoring process and then execute the data flow upon success. Data Flow consists of Excel source data, uploading to SQL database.
Please keep in mind, I am new to SSIS, teaching myself and it's only been about a week. I apologize if this is a beginner type of question.
Any ideas out there on a Lights-Out solution? I'm all ears.
constantly monitoring / running
you may use 'For Loop Container'.
add your file processing components into 'For Loop Container', let the for loop container run constantly.
avoiding duplicate processing
after the source file processed, you may use a 'File System Task' to 'Rename' your source file so that it'll not be processed again.
You can use file watcher task in ssis .
http://www.sqlis.com/post/file-watcher-task.aspx
or you can use WMI Event watcher task in your ssis package (which can trigger child package)
https://msdn.microsoft.com/en-us/library/ms141130.aspx
I have experienced this one, I had try to save the rdl and it happends that it deploys all the reports in the project instead of saving it. Does anyone experienced it?How to resolved this kind of issue?
Start here: http://msdn.microsoft.com/en-us/library/ms173782(v=sql.105).aspx
Ctrl-S while the focus is in the report design surface will save the RDL - I couldn't get it to do anything when I put the focus in Solution Explorer.
Are you running any sort of keystroke manager which might be trapping the hot key and sending a deploy command? I don't even think there's a shortcut key for executing deploy - I suppose a framework might deploy a macro which could automate deployment.
I've seen similar questions and answers but found them not really what I want.
I have a large solution with several web projects in it. All the web projects share some common code so they are all part of the same solution (there is a common project that is referenced by all of them).
I've been running on Team Foundation Server 2008 and we are upgrading to TFS 2010 to be ready for our move to visual studio 2010 and .net 4.0.
I've got the build working with a 2010 build by creating a new build because we only had one customization I'm trying to reproduce.
Once the build is done I have the files dumped at
\\SERVER\tfsdump\devel2010 build\devel2010 build_20101008.3\_PublishedWebsites
by using the dump folder option but the build location is
C:\Builds\1\Web Applications\devel2010 build\Binaries\_PublishedWebsites
All I want to do is after the build is finished I want to copy _PublishedWebsites\SiteA to \ServerA\ShareA
and copy _PublishedWebsites\SiteB to \ServerB\ShareB
Because there is only one build agent all I need is to call a batch file to run the copy but for easy of use for my developers I need this batch file to be called after every build. Is this possible or is there a better solution?
To make things harder the site is still a 2008 project and it is requested that it stays that way until we can fully test it under VS 2010.
Use a CopyDirectory build activity, set Source property equal to BuildDetail.DropLocation + "\_PublishedWebsites\SiteA" and set Destination property to your desired location. I have put this activity as the last activity in build process and it works properly.
Taking a cue from Afshar's answer, for people directly editing XAML template file, create a new sequence after copy to the Drop location is successful:
DisplayName="Something" Source="[BuildDetail.DropLocation + "_PublishedWebsites\SiteA " ]" />
The quotes inside should be written as " and semicolon.