Hi i've been trying to execute a custom activity in ADF which receives csv file from the container (A) after further transformation on the data set, transformed DF stored into another csv file in a same container (A).
I've written the transformation logic in python and have it stored in the same container (A).
Error raises here, when i execute the pipeline it returns an error *can't find the specified file *
Nothing wrong in the connections, Is anything wrong in batch Account or pools!!
can anyone tell me where to place the python script..!!!
Install azure batch explorer and make sure to choose proper configuration for virtual machine (dsvm-windows) which will ensure python is already in place in the virtual machine where your code is being run.
This video explains the steps
https://youtu.be/_3_eiHX3RKE
Related
I am working at something where i have to create a shell script to sync databases between source and target. the target is always localhost:<db_name> (db_name can change). The source db can change.
I am using pgsync to the sync between target and source. The way i am structuring my script is to start a shell script which gives option which db to sync. based on selection, build the source url and pass it to pgsync for from attribute.
I have been trying to find pgsync documentation and couldn't find a way how can i pass it the from value. is it possible
I have a notebook instance with a notebook file. I use the instance's startup script to run this notebook file using papermill.
I want the notebook file to be run only when I remotely start the instance, and not from google cloud console.
I'd like to know if one of these is possible, or if there's another solution:
1 - The script will detect that the instance was started from the dashboard.
2 - I will remove the startups script and use another script that can be run by a remote command.
3 - The shutdown script will remove the startup script.
Script definition is kept under /var/run/google.startup.script for the scenarios mentioned:
Seems complicated detect if the instance was started from the dashboard.
Is feasible to delete the startup script mentioned earlier, and run your own script remotely.
Delete the script.
Before to delete the scrip I suggest you to make a backup of it in case of any issue arises for delete it, also keep in mind that AI Platform Notebooks is a managed service, any admin configuration could cause potential issues in your instance, be careful when deleting or modifying the startup script.
My advise will be actually run a notebook via a Scheduler system, I posted here the multiple options: GCP run a prediction of a model every day
I have a tableau prep flow which uses python script. When i run the flow from tableau prep, it works.
However, i am using command line alongside task scheduler to automate the flow. When running the automated flow; i am encountering the error below:
I have already configured the connection to tabpy in the tableau prep itself. Previously this used to work. But i am now getting this error. Should the TabPy configuration be added to the json file used for credentials? if so, in what format should it be?
Got it to work. Just had to include the following command in the json file:
I have to think this is a solved issue but I am just not getting it to work. So I have come to you StackOverflow with this issue:
I have a windows server 2016 machine running in amazon ec2. I have a machine.ps1 script in a config directory.
I create an image of the box. (I have tried with checking noreboot and unchecking it)
When I create a new instance of the image I want it to run machine.ps1 at launch to set the computer name and then set routes and some config settings for the box. The goal is to do this without logging into the box.
I have read and tried:
Running Powershell scripts at Start up
and used this to ensure user data was getting passed in:
EC2 Powershell Launch Tools
I have tried setting up a scheduled task that runs the machine.ps1 on start up (It just hangs)
I see the initializeInstance.ps1 on start up task and have tried to even coop that replacing the line to run userdata with the line to run my script. Nothing.
If I log into the box and run machine.ps1, it will restart the computer and set the computer name and then I need to run it once more to set routes. This works manually. I just need to find a way to do it automagically.
I want to launch these instances from powershell not with launch configurations and auto scale.
You can use User data
Whenever you deploy a new server, workstation or virtual machine there is nearly always a requirement to make final changes to the system before it’s ready for use. Typically this is normally done with a post-deployment script that might be triggered manually on start-up or it might be a final step in a Configuration Manager task sequence or if you using Azure you may use the Custom Script Extension. So how do you achieve similar functionality using EC2 instances in Amazon Web Services (AWS)? If you’ve created your own Amazon Machine Image (AMI) you can set the script to run from the Runonce registry key, but then can be a cumbersome approach particularly if you want to make changes to the script and it’s been embedded into the image. AWS offers a much more dynamic method of injecting a script to run upon start-up through a feature called user data.
Please refer following link for ther same:
Poershell User data
Windows typically won't let a powershell script call another powershell script unless it is being run as Administrator. It is a weird 'safety' feature. But it is perfectly okay to load the ps1 files and use any functions inside them.
The UserData script is typically run as "system". You would THINK that would pass muster. But it fails...
The SOLUTION: Make ALL of your scripts into powershell functions instead.
In your machine.ps1 - wrap the contents with function syntax
function MyDescriptiveName { <original script contents> }
Then in UserData - use the functions like this
# To use a relative path
Set-Location -Path <my location>
# Load script file into process memory
. <full-or-relpath>/machine.ps1
# Call function
MyDescriptiveName <params-if-applicable>
If the function needs to call other functions (aka scripts), you'll need to make those scripts into functions and load the script file into process memory in UserData also.
I have found this great script which backs up SQL Azure database to BLOB.
I want to run many different variations of this script - e.g. DB1 goes to Customer1Blob, DB2 goes to Customer2Blob.
I have looked at Scheduler Job Collections. However I can only see options (Action settings) for HTTP(S)/ Storage Queue / Service Bus.
Is it possible to run a specific .ps1 script (with commands) scheduled?
You can definitely run a Powershell script as a WebJob. If you want to run a script on a schedule, you can add a settings.job file containing a chron expression with your webjob. The docs for doing so are here.
For this type of automation tasks, I prefer to use the Azure Automation service. You can create runbooks using powershell and then schedule this with the use of the Azure scheduler. You can have it run "on azure" so you do not need to use compute power that you pay for (rather you pay by the minute the job runs) or you can configure it to run with a hybrid worker.
For more information, please see the documentation
When exporting from SQL DB or from SQL Server, make sure you are exporting from a quiescent database. Exporting from a database with active transactions can result in data integrity issues - data being added to various tables while they are also being exported.