I struggle with very basic powershell scripting.
I have an object "ids" (I cant tell exactly what type it is) that translates to json like this
[{"id": "3ccbfe7a-e381-ed11-81ad-000d3aba6139","RowKey": "204640","PartitionKey": "ppm"},
{"id": "7339255d-e381-ed11-81ad-000d3aba6139","RowKey": "205269","PartitionKey": "ppm"}]
I simply want to get the "ids" by "rowkey". e.g.
$ids["204640"] shall resolve to "3ccbfe7a-e381-ed11-81ad-000d3aba6139".
It is obvious that this should be very easy - but I fail ... My understanding is, that it is a list or array of objects with named properties. I assume some simple casting magic is necessary...
(my debugging skills are very limited because I am a Powershell newbie and I do the scripting in the azure portal, trying to write an Azure Function using a table binding. actually "ids" comes from that binding.)
It is unclear if your Json string has already been converted into objects or not, for that you can use ConvertFrom-Json. Then you can use Group-Object -AsHashtable grouping on RowKey property:
$string = '
[
{"id": "3ccbfe7a-e381-ed11-81ad-000d3aba6139","RowKey": "204640","PartitionKey": "ppm"},
{"id": "7339255d-e381-ed11-81ad-000d3aba6139","RowKey": "205269","PartitionKey": "ppm"}
]
'
$hash = (ConvertFrom-Json $string) | Group-Object RowKey -AsHashTable
$hash['205269'].id # => 7339255d-e381-ed11-81ad-000d3aba6139
Related
I am exposing JSONPaths for advanced queries in my application, and sometimes it would be really convenient to chain expressions as I'd do for example with jq. Example:
{
"foo": [
{
"bar": "bar",
"from": 10,
"to": 20
}
]
}
$.foo[*] ? (#.bar == "bar") | #.to - #.from
Now I see that the | operator as used above is not implemented, and in pure SQL I could store the result of the first part in a table / variable. However in application I pass the JSONPath to jsonb_path_query and it would be convenient if I didn't have to chain the invocations manually but just let PostgreSQL handle it all in one pre-baked statement. Is there anything that would let me do that?
EDIT: I could do that using e.g.
SELECT jsonb_path_query((
SELECT jsonb_path_query('{ ... }', '$.foo[*] ? (#.bar == "bar")')
), '$.to - $.from');
but that would mean that I'd need to split the JSONPath query (user input) - exactly what I don't want to do in the app.
I have below JSON
{
"id": " https://xxx.vault.azure.net/secrets/xxx ",
"attributes": {
"enabled": true,
"nbf": 1632075242,
"created": 1632075247,
"updated": 1632075247,
"recoveryLevel": "Recoverable+Purgeable"
},
"tags": {}
}
The above JSON is the output of a web activity and I am using this output into a ForEach activity. The above output when goes to ForEach activity as input, all the values are coming with escape characters.
{
{"id":" https://xxx.vault.azure.net/secrets/xxx ","attributes":{"enabled":true,"nbf":1632075242,"created":1632075247,"updated":1632075247,"recoveryLevel":"Recoverable+Purgeable"},"tags":{}}
From this JSON, I am trying to get only xxx value from the id attribute. How can I do this in Dynamic expression.
Any help is much appreciated.
Thanks
Use the built-in functions lastIndexOf (to find the last occurence of backslash), length (to determine the length of a string), add (to add numbers), sub (to subtract numbers) and substring to do this. For example:
#substring(item().id,add(lastIndexOf(item().id,'/'),1),sub(length(item().id),add(lastIndexOf(item().id,'/'),1)))
I'm using Azure data factory to retrieve data and copy into a database... the Source looks like this:
{
"GroupIds": [
"4ee1a-0856-4618-4c3c77302b",
"21259-0ce1-4a30-2a499965d9",
"b2209-4dda-4e2f-029384e4ad",
"63ac6-fcbc-8f7e-36fdc5e4f9",
"821c9-aa73-4a94-3fc0bd2338"
],
"Id": "w5a19-a493-bfd4-0a0c8djc05",
"Name": "Test Item",
"Description": "test item description",
"Notes": null,
"ExternalId": null,
"ExpiryDate": null,
"ActiveStatus": 0,
"TagIds": [
"784083-4c77-b8fb-0135046c",
"86de96-44c1-a497-0a308607",
"7565aa-437f-af36-8f9306c9",
"d5d841-1762-8c14-d8420da2",
"bac054-2b6e-a19b-ef5b0b0c"
],
"ResourceIds": []
}
In my ADF pipeline, I am trying to get the count of GroupIds and store that in a database column (along with the associated Id from the JSON above).
Is there some kind of syntax I can use to tell ADF that I just want the count of GroupIds or is this going to require some kind of recursive loop activity?
You can use the length function in Azure Data Factory (ADF) to check the length of json arrays:
length(json(variables('varSource')).GroupIds)
If you are loading the data to a SQL database then you could use OPENJSON, a simple example:
DECLARE #json NVARCHAR(MAX) = '{
"GroupIds": [
"4ee1a-0856-4618-4c3c77302b",
"21259-0ce1-4a30-2a499965d9",
"b2209-4dda-4e2f-029384e4ad",
"63ac6-fcbc-8f7e-36fdc5e4f9",
"821c9-aa73-4a94-3fc0bd2338"
],
"Id": "w5a19-a493-bfd4-0a0c8djc05",
"Name": "Test Item",
"Description": "test item description",
"Notes": null,
"ExternalId": null,
"ExpiryDate": null,
"ActiveStatus": 0,
"TagIds": [
"784083-4c77-b8fb-0135046c",
"86de96-44c1-a497-0a308607",
"7565aa-437f-af36-8f9306c9",
"d5d841-1762-8c14-d8420da2",
"bac054-2b6e-a19b-ef5b0b0c"
],
"ResourceIds": []
}'
SELECT *
FROM OPENJSON( #json, '$.GroupIds' )
SELECT COUNT(*) countOfGroupIds
FROM OPENJSON( #json, '$.GroupIds' );
My results:
If your data is stored in a table the code is similar. Make sense?
Another funky way to approach it if you really need the count in-line, is to convert the JSON to XML using the built-in functions and then run some XPath on it. It's not as complicated as it sounds and would allow you to get the result inside the pipeline.
The Data Factory XML function converts JSON to XML, but that JSON must have a single root property. We can fix up the json with concat and a single line of code. In this example I'm using a Set Variable activity, where varSource is your original JSON:
#concat('{"root":', variables('varSource'), '}')
Next, we can just apply the XPath with another simple expression:
#string(xpath(xml(json(variables('varIntermed1'))), 'count(/root/GroupIds)'))
My results:
Easy huh. It's a shame there isn't more built-in support for JPath unless I'm missing something, although you can use limited JPath in the Copy activity.
You can use Data flow activity in the Azure data factory pipeline to get the count.
Step1:
Connect the Source to JSON dataset, and in Source options under JSON settings, select single document.
In the source preview, you can see there are 5 GroupIDs per ID.
Step2:
Use flatten transformation to deformalize the values into rows for GroupIDs.
Select GroupIDs array in Unroll by and Unroll root.
Step3:
Use Aggregate transformation, to get the count of GroupIDs group by ID.
Under Group by: Select a column from the drop-down for your aggregation.
Under Aggregate: You can build the expression to get the count of the column (GroupIDs).
Aggregate Data preview:
Step4: Connect the output to Sink transformation to load the final output to database.
I need to create a singleton complex array. Background: I am trying to automate Let's Encrypt DNS challenge with GoDaddy. But the question is just about PS objects.
Suppose I have the following JSON object, which is my target notation
[
{
"data": "string",
"port": 0,
"priority": 0,
"protocol": "string",
"service": "string",
"ttl": 0,
"weight": 0
}
]
I try to initialize the variable via Powershell as follows
$goDaddyDnsBody=#(
#{
"data"= $dnsChallenge;
"port"= 1;
"priority"= 0;
"ttl"= 600;
"weight"= 0
}
)
The above code is supposed to create an array containing only one item
But when I try to Write-Host ($goDaddyDnsBody |ConvertTo-Json) I get only the first element, not the whole array
{
"weight": 0,
"priority": 0,
"data": ".................",
"ttl": 600,
"port": 1
}
The question is obvious: how can I make sure, without necessarily using string manipulation, that the object created is an array?
Great work with the self answer!
I wanted to point out two things:
the -AsArray parameter to ConvertTo-Json was not always available. It's in the current supported versions of PowerShell that are cross-platform. However, it is not available in Windows PowerShell (versions 5.1 and earlier) which many are still using.
The original issue stems from using the pipeline with an array that contains a single object. Because the pipeline is "unrolling" the array and passing each object into the cmdlet, with a single object array the cmdlet only ever sees the single object. For more on that, see mklement0's excellent and detailed explanation in another answer.
You can instead produce an array by passing the array directly to the parameter:
ConvertTo-Json -InputObject #(#{a=1})
Result:
[
{
"a": 1
}
]
Also worth noting that with the newer -AsArray option you don't need the wrapping array at all, in either the direct parameter or pipeline invocations:
ConvertTo-Json -InputObject #{a=1} -AsArray
#{a=1}|ConvertTo-Json -AsArray
I found the solution myself.
The problem is in with the serialization, not with the initialization of the singleton array. I just needed to add the following -AsArray switch
Write-Host ($goDaddyDnsBody |ConvertTo-Json -AsArray)
Sharing for posterity
I am trying to use a Lookup Activity to return a row count. I am able to do this, but once I do, I would like to run an If Statement against it and if the count returns more than 20MIL in rows, I want to execute an additional pipeline for further table manipulation. The issue, however, is that I can not compare the returned value to a static integer. Below is the current Dynamic Expression I have for this If Statement:
#greater(int(activity('COUNT_RL_WK_GRBY_LOOKUP').output),20000000)
and when fired, the following error is returned:
{
"errorCode": "InvalidTemplate",
"message": "The function 'int' was invoked with a parameter that is not valid. The value cannot be converted to the target type",
"failureType": "UserError",
"target": "If Condition1",
"details": ""
}
Is it possible to convert this returned value to an integer in order to make the comparison? If not, is there a possible work around in order to achieve my desired result?
Looks like the issue is with your dynamic expression. Please correct your dynamic expression similar to below and retry.
If firstRowOnly is set to true : #greater(int(activity('COUNT_RL_WK_GRBY_LOOKUP').output.firstRow.propertyname),20000000)
If firstRowOnly is set to false : #greater(int(activity('COUNT_RL_WK_GRBY_LOOKUP').output.value[zero based index].propertyname),20000000)
The lookup result is returned in the output section of the activity run result.
When firstRowOnly is set to true (default), the output format is as shown in the following code. The lookup result is under a fixed firstRow key. To use the result in subsequent activity, use the pattern of #{activity('MyLookupActivity').output.firstRow.TableName}.
Sample Output JSON code is as follows:
{
"firstRow":
{
"Id": "1",
"TableName" : "Table1"
}
}
When firstRowOnly is set to false, the output format is as shown in the following code. A count field indicates how many records are returned. Detailed values are displayed under a fixed value array. In such a case, the Lookup activity is followed by a Foreach activity. You pass the value array to the ForEach activity items field by using the pattern of #activity('MyLookupActivity').output.value. To access elements in the value array, use the following syntax: #{activity('lookupActivity').output.value[zero based index].propertyname}. An example is #{activity('lookupActivity').output.value[0].tablename}.
Sample Output JSON Code is as follows:
{
"count": "2",
"value": [
{
"Id": "1",
"TableName" : "Table1"
},
{
"Id": "2",
"TableName" : "Table2"
}
]
}
Hope this helps.
Do this - when you run the debugger look at the output from your lookup. It will give a json string including the alias for the result of your query. If it's not firstrow set then you get a table. But for first you'll get output then firstRow and then your alias. So that's what you specify.
For example...if you put alias of your count as Row_Cnt then...
#greater(activity('COUNT_RL_WK_GRBY_LOOKUP').output.firstRow.Row_Cnt,20000000)
You don't need the int function. You were trying to do that (just like I was!) because it was complaining about datatype. That's because you were returning a bunch of json text as the output instead of the value you were after. Totally makes sense after you realize how it works. But it is NOT intuitively obvious because it's coming back with data but its string stuff from json, not the value you're after. And functions like equals are just happy with that. It's not until you try to do something like greater where it looks for numeric value that it chokes.