I am new to Azure environment. i have written some code using .net core that starts Azure pipeline using Azure Data factory. The status of the pipeline run status when trying from my local is always success. I deployed the code in the azure environment. When try to start the pipe line from azure server always the status is queued. what is queued status and what i have to do with it. can some one please help. do i need to change any settings in azure so that the pipeline run will be success
AuthenticationContext context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(applicationId, authenticationKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new Microsoft.Azure.Management.DataFactory.DataFactoryManagementClient(cred) { SubscriptionId = subscriptionId };
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName).Result.Body;
string RunId = runResponse.RunId;
PipelineRun pipelineRun;
while (true)
{
pipelineRun = client.PipelineRuns.Get(resourceGroup, dataFactoryName, runResponse.RunId);
if (pipelineRun.Status == "InProgress")
System.Threading.Thread.Sleep(15000);
else
break;
}
You should try to look into the diagnotics logs of the IR , it does have valuable info which should help .
Navigate to IR -> diagnotics log or open event viewer -> Application and service logs ->
Related
I am trying to automate the process of azure databricks creation along with cluster, notebook and job and to run the notebook, by terraform. The code that I have, creates the above resources. But I am not able to run the notebook or the job by terraform.
provider "databricks" {
azure_workspace_resource_id = azurerm_databricks_workspace.example.id
}
resource "azurerm_resource_group" "example" {
name = "example-resources"
location = "West Europe"
}
resource "azurerm_databricks_workspace" "example" {
name = "databricks-test"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "standard"
}
resource "databricks_cluster" "shared_autoscaling" {
cluster_name = "Autoscaling-Cluster"
spark_version = "10.5.x-scala2.12"
node_type_id = "Standard_DS3_v2"
autotermination_minutes = 15
autoscale {
min_workers = 1
max_workers = 3
}
library {
maven {
coordinates = "com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.18"
}
}
library {
maven {
coordinates = "com.microsoft.azure:azure-event-hubs-reactive_2.12:0.1.0"
}
}
}
resource "databricks_notebook" "notebook" {
path = "/IOT_notebook"
language = "PYTHON"
format = "SOURCE"
source = "./IOT_notebook.dbc"
}
resource "databricks_job" "myjob5" {
name = "Featurization2"
timeout_seconds = 3600
max_retries = 1
max_concurrent_runs = 1
existing_cluster_id = databricks_cluster.shared_autoscaling.id
notebook_task {
notebook_path = databricks_notebook.notebook.path
//base_parameters = var.resource_group_name
}
}
I am not able to find any code related to running the notebook or the job, neither in terraform nor in powershell. I want to know if it is possible to do it through a script or not.
Any suggestions would be appreciated.
The triggering of the job isn't a responsibility of the Terraform - it takes care that all objects are created & their state is matching to what is declared, but triggering is an activity, not the final state. So although technically you can trigger the job from terraform using the local resource + for, example, curl, it's better to use something else to actually trigger that job.
You can trigger job multiple ways using the job ID returned by Terraform:
Using the REST API - just use something like curl
Using the databricks-cli: databricks jobs run-now --job-id <job-id>
for Powershell, there is a DatabricksPS package that includes the Start-DatabricksJob function that simplifies execution of existing jobs
So, I can get the build details, but it does not contain any info on the build jobs. E.g. Each build job has run on a build agent - how can I get this piece using REST Api?
We are talking about a vNext build, not XAML.
You can find all tasks and jobs in timeline records: Timeline - Get. You can paste into browser this template to check results for a specific build:
https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline
I use Microsoft.TeamFoundationServer.Client package and this is example for it:
static void PrintTimeLine(string TeamProjectName, int BuildId)
{
var timeline = BuildClient.GetBuildTimelineAsync(TeamProjectName, BuildId).Result;
if (timeline.Records.Count > 0)
{
Console.WriteLine("Task Name-----------------------------Start Time---Finish Time---Result");
foreach(var record in timeline.Records)
if (record.RecordType == "Task")
Console.WriteLine("{0, -35} | {1, -10} | {2, -10} | {3}",
(record.Name.Length < 35) ? record.Name : record.Name.Substring(0, 35),
(record.StartTime.HasValue) ? record.StartTime.Value.ToLongTimeString() : "",
(record.FinishTime.HasValue) ? record.FinishTime.Value.ToLongTimeString() : "",
(record.Result.HasValue) ? record.Result.Value.ToString() : "");
}
}
https://github.com/ashamrai/TFRestApi/blob/master/19.TFRestApiAppQueueBuild/TFRestApiApp/Program.cs
https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId} will let you know the Agent used under the object queue and there it shows the agent queue (91) number and the pool id (8)
"queue":{
"id":91,
"name":"MotBuild-Default",
"pool":{
"id":8,
"name":"MotBuild-Default"
}
Use
https://dev.azure.com/{org}/_apis/distributedtask/pools/{pool_id}?api-version=5.0-preview.1 or https://dev.azure.com/{org}/{project}/_apis/distributedtask/queues/{queue_id} will return the pool.
So now using https://dev.azure.com/{org}/_apis/distributedtask/pools/{pool_id}/agents will return a LIST of agents under the Agent Pools
Now that I've explained all that let's try to tie everything together.
1) Use https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId} and find the Queue and Pool IDs.
2) use https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline and find the record of type Job and the property workerName which will return NAME of the Agent used.
3) Query the Agents with https://dev.azure.com/{org}/_apis/distributedtask/pools/{pool_id}/agents and find the agent id by filtering the name from the name found in step #2 above.
4) Finally query https://dev.azure.com/{org}/_apis/distributedtask/pools/{pool_id}/agents/{agent_id} will return a high-level info of the agent, not much info.
This next api is undocumented
5) To get the detailed capabilities query https://dev.azure.com/{org}/_apis/distributedtask/pools/{pool_id}/agents/{agent_id}?includeCapabilities=true which a huge result set will be returned!! I think this is what you want.
Read more about the APIs at:
Pools
Queues
Agents
I have a custom VHD file with me. I am able to create Pool with my custom image through portal. But i want to try the same with .Net SDK. But it is throwing error "Operation returned an invalid status code 'Forbidden".
I am referring this link Azure Batch
I am able to create Pool from MarketPlace images from same code
Below is my code
ImageReference imageReference = new ImageReference("/subscriptions/XXXXXXXXXXXXXXX/resourceGroups/RG-OneGolden/providers/Microsoft.Compute/images/OMGoldenImage");
VirtualMachineConfiguration virtualMachineConfiguration =
new VirtualMachineConfiguration(
imageReference: imageReference,
nodeAgentSkuId: "batch.node.windows amd64");
try
{
CloudPool pool = batchClient.PoolOperations.CreatePool(
poolId: PoolId,
targetDedicatedComputeNodes: PoolNodeCount,
virtualMachineSize: PoolVMSize,
virtualMachineConfiguration: virtualMachineConfiguration);
pool.Commit();
}
catch (BatchException be)
{
// Accept the specific error code PoolExists as that is expected if the pool already exists
if (be.RequestInformation?.BatchError?.Code == BatchErrorCodeStrings.PoolExists)
{
Console.WriteLine("The pool {0} already existed when we tried to create it", PoolId);
}
else
{
throw; // Any other exception is unexpected
}
}
You need to ensure you have met the prerequisites for custom images in Azure Batch:
The ARM Image is in the same subscription and region as the Batch account.
You are using Azure Active Directory to authenticate with the Batch service.
I want to enable running few jobs remotely on various Windows Nodes on some random interval. I can use PS remoting to logon to a remote computer and create a task there to run it under some specific user account.
In order for PS Remoting to work, I understand that I don't need to be an admin from the following article however I am unable to understand what is the minimum privilege needed to schedule a task on that remote machine for this script user.
I don't want to use an Admin account as this can be a security risk that I open up on those machines.
Can some one please help me out. I am using a C# application to initiate the remote powershell connection to that computer as below.
private void StartCollection(string triggerId, ActivationData parameters)
{
string scriptUserName = parameters.Params["PSUser"];
string scriptUserPassword = parameters.Params["PSUserPassword"];
string remotecomputerName = parameters.Params["ComputerName"];
string scheme = parameters.Params["PSScheme"];
string port = parameters.Params["PSPort"];
var agentFile = Path.Combine(parameters.AssemblyPath, parameters.FileName);
List<string> allParams = new List<string>();
allParams.Add(String.Format("-trigger {0} -collector {1} ", triggerId, parameters.CollectorId));
allParams.Add(String.Format("-monitored_by {0} -monitored_at {1}", parameters.Monitoring.Name, parameters.Monitoring.InterfacePoint));
foreach (var param in parameters.Params)
{
if (!ConfigurationKeys.Contains(param.Key))
{
allParams.Add(string.Format("-{0} {1}", param.Key, param.Value));
}
}
var cmdLineParams = string.Join(" ", allParams.ToArray());
// Cred to execute PS on the remote computer.
SecureString securePassword = scriptUserPassword.ToSecureString();
PSCredential scriptCred = new PSCredential(scriptUserName, securePassword);
var remoteUriStr = String.Format("{0}://{1}:{2}/wsman", scheme, remotecomputerName, port);
var remoteComputer = new Uri(remoteUriStr);
var connection = String.IsNullOrEmpty(scriptUserName) ? new WSManConnectionInfo(remoteComputer) : new WSManConnectionInfo(remoteComputer, null, scriptCred);
//connection.AuthenticationMechanism = AuthenticationMechanism.Credssp;
var runspace = RunspaceFactory.CreateRunspace(connection);
runspace.Open();
using (var powershell = PowerShell.Create())
{
powershell.Runspace = runspace;
//Cred to run the agent under: Job credential.
SecureString jobPass = parameters.UserCred.Item2.ToSecureString();
string jonUser = parameters.UserCred.Item1;
PSCredential jobCred = new PSCredential(jonUser, jobPass);
var scriptfile = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Start-Program.ps1");
Command command = new Command(scriptfile);
command.Parameters.Add("ComputerName", remotecomputerName);
command.Parameters.Add("StartTime", "22:50");
command.Parameters.Add("Program", agentFile);
command.Parameters.Add("Parameters", cmdLineParams);
command.Parameters.Add("TaskCredential", jobCred);
powershell.Commands.AddCommand(command);
var results = powershell.Invoke();
runspace.Close();
Console.WriteLine("Outputing the PS1 Result: {0}: " ,parameters.CollectorId);
foreach (var obj in results.Where(o => o != null))
{
Console.WriteLine("\t" + obj);
}
if (powershell.Streams.Error.Count > 0)
{
var errors = from err in powershell.Streams.Error select err.ErrorDetails.Message;
throw new Exception(String.Join(Environment.NewLine, errors.ToArray()));
}
}
}
Once you set the PS Remoting permissions, the simple way to give them Task Scheduling access would be to make your remote users a member of the local "Backup Operators" group on the target server. This would give them the necessary permissions to schedule and manage tasks on the server but would also give them indirect read access to files on that server.
If that's still too much access, then I've seen references to granting control of the C:\Windows\Tasks folder on the target server through ACLs but I've never tried that myself.
From c# code I call schtasks to delete some scheduled tasks. I make the first call and I get this error returned:
ERROR: Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed....
Here's the code that runs the process:
Process stProc = new Process();
stProc.StartInfo.UseShellExecute = false;
stProc.StartInfo.FileName = "SCHTASKS.exe";
stProc.StartInfo.RedirectStandardError = true;
stProc.StartInfo.RedirectStandardOutput = true;
stProc.StartInfo.CreateNoWindow = true;
stProc.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
stProc.StartInfo.Arguments = args;
stProc.Start();
stProc.BeginOutputReadLine();
stProc.BeginErrorReadLine();
stProc.OutputDataReceived += new DataReceivedEventHandler(p_OutputDataReceived);
stProc.ErrorDataReceived += new DataReceivedEventHandler(p_ErrorDataReceived);
stProc.WaitForExit();
stProc.Close();
stProc.Dispose();
My arguments are correct as per: http://msdn.microsoft.com/en-us/library/bb736357(v=vs.85).aspx
Just to make sure that my connection is not hanging around, I create a new process to kill it before every delete call:
StartProcess(args);
Process x = new Process();
x.StartInfo.FileName = "cmd";
x.StartInfo.Arguments = #" net use \\servername\ipc$ /delete";
x.StartInfo.CreateNoWindow = true;
x.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
x.Start();
Not sure what's up here. Might it be someone else that's using this machine and so my calls can't get through?
Any ideas appreciated!
Thanks!!
The answer is this: http://support.microsoft.com/kb/938120
I used the IP address of the machine.