Creating Azure Stream Analytics Job through Powershell - powershell

I would like to create a stream analytics job using only Powershell. I know that the command to do this is: New-AzureRMStreamAnalyticsInput. However it requires a JSON file with job details. I found a documentation provided by Microsoft where there is a small template of such a JSON file (check Create paragraph). However it's not enough for me.
I want to create an input from blob storage hence my JSON looks like this:
{
"properties":{
"type":"stream",
"datasource":{
"type":"Microsoft.Storage/Blob",
"properties":{
"accountName":"abc",
"accountKey":"###",
"container":"appinsights",
"pathPattern":"test-blob_2324jklj/PageViews/{date}/{time}",
"dateFormat":"YYYY-MM-DD",
"timeFormat":"HH"
}
}
}
}
After saving it and passing as an argument in New-AzureRMStreamAnalyticsInput I receive following error: New-AzureRMStreamAnalyticsInput : Stream analytics input name cannot be null. I think that my JSON file is not correct.
Do you have any templates of json files containing stream analytics job details or can you just tell me how to correctly set up a job through powershell?

A simple way of getting your template right is to manually create an input from Portal and then run PowerShell command Get-AzureRmStreamAnalyticsInput to get the JSON payload.
From your example, it seems you missed the input name. Try something like below:
{
"Name": "BlobInput1",
"Properties": {
... ...
}
}

Related

Send log with customDimensions to Application Insights using Powershell

I want to send logs to my Application Insights using PowerShell script and I want it to contain customDimensions value.
I tried something like this:
$client = [Microsoft.ApplicationInsights.TelemetryClient]::new()
$client.InstrumentationKey = "xxxx"
$client.TrackTrace("MY CUSTOM LOG", "Information", #{foo="bar"})
$Client.Flush()
but it gives me an error:
Cannot find an overload for "TrackTrace" and the argument count: "3".
Ideally I want the code above to work so that I can read the log like this:
I know how to do it using Python and opencensus-ext-azure lib but now I also need Powershell as well.

How to send dataset in Web Activity? ADF

I want to publish data into a Service Bus, from my Storage Account.
I already tried send a simple body and it works fine. But i dont know how should set a data set.
Web Activity Setting
When i run this activity into a pipeline, this send
{
"myMessage": "Sample",
"datasets": [{
"name": "MyDataset",
"properties": {
...
}
}],
"linkedServices": [{
"name": "MyStorageLinkedService1",
"properties": {
...
}
}]
}
and i want send data from the file in dataset. Anyone know how i should set web activity?
You can achieve that by using "Copy Activity".
Here is a quick demo that I made :
I used JsonPlaceHolder API , I want to modify the array and add a custom value by doing a PUT request.
check it out here : https://jsonplaceholder.typicode.com/guide/
please read carefully "Updating a resource"
Here is a Json that I want to modify , I added it as a Dataset in ADF.
The main idea is to set the Dataset as a source and the sink is a REST API method so we are sending the Dataset as an input to the POST request in Copy activity.
Copy activity:
Source:
Sink:
You can read more about it here:
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#dataset-properties
Here is the output of the Copy Activity:

What is the appropriate way to build JSON within a Data Factory Pipeline

In my earlier post, SQL Server complains about invalid json, I was advised to use an 'appropriate methods' for building a json string, which is to be inserted into a SQL Server table for logging purposes. In the earlier post, I was using string concatenation to build a json string.
What is the appropriate tools/functions to build json within a Data Factory pipeline? I've looked into the json() and string() functions, but they would still rely on concatenation.
Clarification: I'm trying to generate a logging message that looks like this: Right now I'm using string concatenation to generate the logging json. Is there a better, more elegant (but lightweight) way to generate the json data?
{ "EventType": "DataFactoryPipelineRunActivity",
"DataFactoryName":"fa603ea7-f1bd-48c0-a690-73b92d12176c",
"DataFactoryPipelineName":"Import Blob Storage Account Key CSV file into generic SQL table using Data Flow Activity Logging to Target SQL Server",
"DataFactoryPipelineActivityName":"Copy Generic CSV Source to Generic SQL Sink",
"DataFactoryPipelineActivityOutput":"{runStatus:{computeAcquisitionDuration:316446,dsl: source() ~> ReadFromCSVInBlobStorage ReadFromCSVInBlobStorage derive() ~> EnrichWithDataFactoryMetadata EnrichWithDataFactoryMetadata sink() ~> WriteToTargetSqlTable,profile:{ReadFromCSVInBlobStorage:{computed:[],lineage:{},dropped:0,drifted:1,newer:1,total:1,updated:0},EnrichWithDataFactoryMetadata:{computed:[],lineage:{},dropped:0,drifted:1,newer:6,total:7,updated:0},WriteToTargetSqlTable:{computed:[],lineage:{__DataFactoryPipelineName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__DataFactoryPipelineName]}]},__DataFactoryPipelineRunId:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__DataFactoryPipelineRunId]}]},id:{mapped:true,from:[{source:ReadFromCSVInBlobStorage,columns:[id]}]},__InsertDateTimeUTC:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__InsertDateTimeUTC]}]},__DataFactoryName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__DataFactoryName]}]},__FileName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__FileName]}]},__StorageAccountName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__StorageAccountName]}]}},dropped:0,drifted:1,newer:0,total:7,updated:7}},metrics:{WriteToTargetSqlTable:{rowsWritten:4,sinkProcessingTime:1436,sources:{ReadFromCSVInBlobStorage:{rowsRead:4}},stages:[{stage:3,partitionTimes:[621],bytesWritten:0,bytesRead:24,streams:{WriteToTargetSqlTable:{type:sink,count:4,partitionCounts:[4],cached:false},EnrichWithDataFactoryMetadata:{type:derive,count:4,partitionCounts:[4],cached:false},ReadFromCSVInBlobStorage:{type:source,count:4,partitionCounts:[4],cached:false}},target:WriteToTargetSqlTable,time:811}]}}},effectiveIntegrationRuntime:DefaultIntegrationRuntime (East US)}",
"DataFactoryPipelineRunID":"63759585-4acb-48af-8536-ae953efdbbb0",
"DataFactoryPipelineTriggerName":"Manual",
"DataFactoryPipelineTriggerType":"Manual",
"DataFactoryPipelineTriggerTime":"2019-11-05T15:27:44.1568581Z",
"Parameters":{
"StorageAccountName":"fa603ea7",
"FileName":"0030_SourceData1.csv",
"TargetSQLServerName":"5a128a64-659d-4481-9440-4f377e30358c.database.windows.net",
"TargetSQLDatabaseName":"TargetDatabase",
"TargetSQLUsername":"demoadmin"
},
"InterimValues":{
"SchemaName":"utils",
"TableName":"vw_0030_SourceData1.csv-2019-11-05T15:27:57.643"
}
}
You can using Data Flow, it help you build the JSON string within pipeline in Data Factory.
Here's the Data Flow tutorial: Mapping data flow JSON handling.
It can help you:
Creating JSON structures in Derived Column
Source format options
Hope this helps.

How to send a list as parameter in databricks notebook task?

I am using Databricks Resi API to create a job with notebook_task in an existing cluster and getting the job_id in return.
Then I am calling the run-now api to trigger the job.
In this step, I want to send a list as argument via the notebook_params, which throws an error saying "Expected non-array for field value".
Is there any way I can send a list as an argument to the job?
I have tried sending the list argument in base_params as well with same error.
user_json={
"name": job_name,
"existing_cluster_id": cluster_id,
"notebook_task": {
"notebook_path": notebook_path
},
"email_notifications":{
"on_failure":[email_id]
},
"max_retries": 0,
"timeout_seconds": 3600
}
response=requests.post('https://<databricks_uri>/2.0/jobs/create',headers=head,json=user_json,timeout=5, verify=False)
job_id=response.json()['job_id']
json_job={"job_id":job_id,"notebook_params":{"name":"john doe","my_list":my_list}}
response = requests.post('https://<databricks_uri>/2.0/jobs/run-now', headers=head, json=json_job, timeout=200, verify=False)
Not found any native solution yet, but my solution was to pass the list as a string and parse it back out on the other side:
json_job={"job_id":job_id,
"notebook_params":{
"name":"john doe",
"my_list":"spam,eggs"
}
}
Then in databricks:
my_list=dbutils.widgets.get("my_list")
my_list=my_list.split(",")
With appropriate care around special characters or e.g. conversion to numeric types.
If the objects in the list are more substantial, then sending them as a file to dbfs using the CLI or API before running the job may be another option to explore.
Hi may be I'm bit late but found a better solution.
Step 1:
Use JSON.stringyfy() in the console of any browser to convert your value(object, array, JSON etc) into string
Ex:
Now use this value in the body of URL
In Databricks notebook convert string to JSON using python json module.
Hope this helps

How can I pass context params using talend api?

I'm trying to automate talend job executions using the Talend API but I'm getting an error when I try to pass the context params using the api.
The json I'm encoding to 64 is the following:
JSON='{ "actionName":"runTask", "authPass": "TalendPass", "authUser": "name#example.com", "jvmParams": [ "-Xmx256m" , "-Xms64m" ], "contextParams": ["host_mysql_db01": "failed", "database_analytics": "testing.it"],"mode": "synchronous", "taskId": 43}'
Error message:
{"error":"Expected a ',' or ']' at character 172","returnCode":2}
I found another stackoverflow issue Add context parameters to Talend job in Tac via API without actually running it but he doesn't say how he pass it and I cannot reply with a comment asking how he did it
The real talend api call is:
wget -O file http://localhost:8080/org.talend.administrator/metaServlet?$JSON_ENCODED
Can I get some help?
Actually, the json your are passing to the metaservlet is not valid json. You can check it with an online validator like http://jsonlint.com.
You are specifying the contextParams attribute as an array, but that syntax is not valid in json. An array can contain either a list of values (like jvmParams) or objects (which can themselves contain arrays). Here's an example.
Moreover, according to Talend reference, the attribute should be called "context" and must be an object instead of an array, like so:
"context":{"varname1": "varvalue", "varname2": "varvalue2"}