Informatica Workflow Scheduling with Autosys.
I am trying to understand more about the Informatica Workflow Scheduling with Autosys.
Assume I have an Informatica workflow wf_test and a UNIX script say test.sh with pmcmd command to run this workflow. Also, I wrote a JIL
(test.jil) for Autosys to schedule my test.sh. at daily 10:00 PM.
How exactly Autosys kick-off workflow wf_test at the specified schedule?
Can anyone shed some light about the communication between Autosys and Informatica?
Do we need to have both Informatica and Autosys server installed on the same server?
Is there any agent or service needs be present in-between Autosys and Informatica to happen this possible?
Additionally, can we directly give informatica details to Autosys without any script?
Many Thanks
aks
How exactly Autosys kick-off workflow wf_test at the specified schedule?
Autosys is a scheduling tool. An autosys job keep checking every 5 seconds, if any job is scheduled to run, based on the jil. When the time comes and the condition satisfied, it will run the given command on the given host. It could be a pmcmd command or any shell script.
Can anyone shed some light about the communication between Autosys and Informatica?
The communication should be between Autosys Server and the server where Informatica is installed. Read this article. Additionally check if your autosys engineering team on steps to implement the same in your project/environment.
Do we need to have both Informatica and Autosys server installed on the same server?
Definately not. It should be separated. But the connectivity should be established.
Is there any agent or service needs be present in-between Autosys and Informatica to happen this possible?
Yes, Read the article given in point 2.
Additionally, can we directly give informatica details to Autosys without any script?
Yes. You can mentioned the whole pmcmd command.
As Autosys is scheduling tool , it will trigger command at specified time mentioned in the Job jil , the important part here is , we also mention the machine name where we want to execute that particular command.
So to answer your question, Autosys and Informatica can be on different servers , provided Autosys agent is configured on Informatica server and the Informatica machine/server details are configured in Autosys.(its like creating a machine on Autosys similiar to creating Global variable or a Job)
As we are running our workflows through shell scripts using pmcmd command , and not to mention Autosys and Informatica are on different servers, there might be way you can directly call Workflows from Autosys but that will make things complicated when you're working at large scale calling 1000s of workflows, Instead having a generic script to call pmcmd which can utilised by multiple workflows seems an easier option.
All Autosys does is "run a command at a specified time" in this case. It's completely unaware of Informatica. It doesn't need to be on the same server as there simply is no communication between them.
All it needs, is the access to the test.sh script, wherever it is. And this, in turn, needs to be able to run the pmcmd utility. So in most basic setup, the Informatica >client< with the pmcmd could be on the same server with Autosys. Informatica Server just needs to be reachable to pmcmd.
I would suggest you to schedule the jobs using the in-built scheduler service,available from 10.x version. You don't have to even write a pmcmd command to trigger the workflow.
Related
We have a cloud full of self-hosted Azure Agents running on custom AMIs. In some cases, I have some cleanup operations which I'd really like to do either before or after a job runs on the machine, but I don't want the developer waiting for the job to wait either at the beginning or the end of the job (which holds up other stages).
What I'd really like is to have the Azure Agent itself say "after this job finishes, I will run a set of custom scripts that will prepare for the next job, and I won't accept more work until that set of scripts is done".
In a pinch, maybe just a "cooldown" setting would work -- wait 30 seconds before accepting another job. (Then at least a job could trigger some background work before finishing.)
Has anyone had experience with this, or knows of a workable solution?
I suggest three solutions
Create another pipeline to run the clean up tasks on agents - you can also add demand for specific agents (See here https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) by Agent.Name -equals [Your Agent Name]. You can set frequency to minutes, hours, as you like using cron pattern. As while this pipeline will be running and taking up the agent, the agent being cleaned will not be available for other jobs. Do note that you can trigger this pipeline from another pipeline, but if both are using the same agents - they can just get deadlocked.
Create a template containing scripts tasks having all clean up logic and use it at the end of every job (which you have discounted).
Rather than using static VM's for agent hosting, use Azure scaleset for Self hosted agents - everytime agents are scaled down they are gone and when scaled up they start fresh. This saves a lot of money in terms of sitting agents not doing anything when no one is working. We use this option and went away from static agents. We have also used packer to create the VM image/vhd overnight to update it with patches, softwares required, and docker images cached.
ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops
For those discovering this question, there's a much better way: run your self-hosted agent with the --once flag, documented here.
You'll need to wrap it in your own bash script, but something like this works:
while :
do
echo "Performing pre-job setup..."
echo "Waiting for job..."
./run.sh --once
echo "Cleaning up..."
sleep 2
done
Another option would be to use a ScaleSet VM setup which preps a new agent for each run and discards the VM when the run is done. It can prepare new VMs in the background while the job is running.
And I suspect you could implement your own IMaintenanceProvirer..
https://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/Maintenance/MaintenanceJobExtension.cs#L53
I am trying to schedule datastage job using datastage director client 11.7.
But facing issue:
Error adding to schedule:sh:/usr/bin/at:permission denied
Make sure that the datastage engine user is able to schedule cron and at jobs. cron is used to schedule recurring jobs wheras at is used to schedule "one-shots". In some environments, these services are not activated by default anymore and must be enabled.
Looks like dsadm is unable to perform scheduling.
DataStage uses Linux internal scheduling utility such as cron/at to schedule the jobs, check whether dsadm has the privilege to do it.
I had developed a Job in Talend and built the job and automated to run the Windows Batch file from the below build
On the Execution of the Job Start Windows Batch file it will invoke the dimtableinsert job and then after it finishes it will invoke fact_dim_combine it is taking just minutes to run in the Talend Open Studio but when I invoke the batch file via the Task Scheduler it is taking hours for the process to finish
Time Taken
Manual -- 5 Minutes
Automation -- 4 hours (on invoking Windows batch file)
Can someone please tell me what is wrong with this Automation Process
The reason of the delay in the execution would be a latency issue. Talend might be installed in the same server where database instance is installed. And so whenever you execute the job in Talend, it will complete as expected. But the scheduler might be installed in the other server, when you call the job through scheduler, it would take some time to insert the data.
Make sure you scheduler and database instance is on the same server
Execute the job directly in the windows terminal and check if you have same issue
The easiest way to know what is taking so much time is to add some logs to your job.
First, add some tWarn at the start and finish of each of the subjobs (dimtableinsert and fact_dim_combine) to know which one is the longest.
Then add more logs before/after the components inside the jobs.
This way you should have a better idea of what is responsible for the slowdown (DB access, writing of some files, etc ...)
What is the best way of scheduling a PowerShell script in Azure? Should I create a VM and schedule it via a task scheduler. Or is there any better way?
I have a PowerShell script that I extracts data from audit log and reports some information. Thank you.
You should use Azure Automation. Its easy to use and you can run jobs for 500 minutes for free (every month).
I created a simple process by Applicaiton Lab interface in Bluemix Workload Scheduler. I ran my process, but the step didn't proceed and remained in queued status.
How can I proceed the step?
I executed the process by "Run now". The process doesn't have triggers
The step remains "Queued status".
The Process information
There is only one step. The step is "ping www.ibm.com"
Process doesn't have trigger. It is an on-demand process.
There might be a problem with the agent as I can successfully run a simple workload process without any issues. If you are using the Workload Automation Agent that is created for you then you will need to open a support ticket to have the Workload team look at that agent.
reviewing your question I think that a process submitted to the Workload Scheduler service should be a process which will complete: a ping command like the one you are trying to submit, will never complete if not 'killed' using CTRL+C (or called with [-c count] option)