Azure Functions (PowerShell): How to use table input binding? - powershell

I have a very simple input binding in my powershell function.
# Input bindings are passed in via param block.
param($Request, $table, $TriggerMetadata)
I am kind of a beginner in Powershell, though I had plenty of experience in other languages back then.
I do understand the Azure Table Datamodel in general
Can someone please explain the structure of the binding in terms of Powershell? Is it a Hashtable, Dictionary, array of entities, of properties? What (exactly) is an entity or a property? (again in terms of Powershell-Objects)
Actually I simply want to lookup an entity that has only one property by PartitionKey and RowKey in the most simple way!
$value = $table['mykey']
Thanks alot!

What (exactly) is an entity or a property? (again in terms of Powershell-Objects)
As far as I know,
An Entity in the Azure Table Storage is a set of properties such as database row (Eg, key-value pair).
Every entity has a Composite Primary Key (combination of Partition Key and Row Key) and also a timestamp.
Partition is a column in every entity.
Azure Table Storage does not enforce any schema structure of all the entities that are collectively making up the table.
Every entity will have different types of partitions/columns.
Partition Key:
A Partition key is used to partition a table to support load balancing. It's used to specify which partition an object belongs to.
Row Key: A Row Key is used to uniquely identify an entity (record) in a given partition.
And I believe you have to work upon the entity that has one property (1 Partition-Row Key) which is the format of:
$inputCloudTable = $storageTable.CloudTable
Here is the MS Doc that provides example of Working on the Entities as a PowerShell Objects.

I solved it with plain old Powershell - for debugging purposes I used "ConvertTo-JSON" to inspect the binding object.
I also added a binding that refines to the respective partion upfront in the function.json by simply adding the partitionkey property to the binding. It filters the entities accordingly upfront.
Then it is straightforward to work with the binding (when you're able to use Powershell properly):
#select entity by property (here RowKey is used as primary Key) using "Where-Object" (alias "?")
$entity = $binding | ?{ $_.RowKey -eq $id }
#get an arbitrary property from the azure table entity
$property = $entity.someproperty
So as expected, no AzTable module / cmdlet neded !

You can add PartitionKey and RowKey to the binding input so that you can filter without having to return all results.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-table-input?tabs=in-process%2Ctable-api&pivots=programming-language-powershell
{
"name": "PersonEntity",
"type": "table",
"tableName": "Person",
"partitionKey": "Test",
"rowKey": "{queueTrigger}",
"connection": "MyStorageConnectionAppSetting",
"direction": "in"
}
There is also an oData filter option within the binding which isn't included in the documentation examples.
{
"name": "PersonEntity",
"type": "table",
"tableName": "Person",
"filter": "(age ge '{myNum}')",
"connection": "MyStorageConnectionAppSetting",
"direction": "in"
}
You also don't need to use the queue binding if you need HTTP input.
{
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "Request",
"methods": [
"get"
],
"route": "route/{FirstName}"
},
{
"name": "PersonEntity",
"type": "table",
"tableName": "Person",
"partitionKey": "{FirstName}",
"connection": "MyStorageConnectionAppSetting",
"direction": "in"
},
{
"type": "http",
"direction": "out",
"name": "Response"
}
]
}

Related

Return Only Most Recent Record From Related Entity in OData Query

I am trying to create an OData Query to return Bugs from Azure DevOps for a PowerBI report, but I am not getting the results I am looking for, as one of the Related Entities that I am trying to expand returns multiple results.
My base Query looks like this (simplified & removing custom fields)
https://analytics.dev.azure.com/[organization]/[project]/_odata/v3.0-preview/WorkItems?$select=WorkItemId,WorkItemType,Title,State,LeadTimeDays&$filter=WorkItemType eq 'bug'&$expand=Teams($select=TeamName,AnalyticsUpdatedDate)
Some records return multiple Team Names in the JSON Response
"value": [
{
"WorkItemId": 16547,
"LeadTimeDays": 173.0639004,
"Title": "test",
"WorkItemType": "Bug",
"State": "Closed",
"Severity": "3 - Medium",
"Teams": [
{
"TeamName": "Team1",
"AnalyticsUpdatedDate": "2019-09-17T01:48:46.5433333Z"
},
{
"TeamName": "Team2",
"AnalyticsUpdatedDate": "2019-12-03T16:52:39.9466667Z"
}
]
}
]
I can't tell why these records have multiple values for this Entity, but I only need the most recent (Team 2 in the example above). Is it possible to return only the most recent record for the Related Teams Entity? I've tried using orderby and top on the expand clause and other places in the query to no effect. If I can't do it in the OData query, then I can accomplish it in Power BI after expanding the Table.
I found how to solve this. I needed semicolons between the clauses within the Expand clause.
https://analytics.dev.azure.com/[organization]/[projet]_odata/v3.0-preview/WorkItems?$select=WorkItemId,WorkItemType,Title,State,LeadTimeDays&$filter=WorkItemType eq 'bug'&$expand=Teams($select=TeamName,AnalyticsUpdatedDate;$orderby=AnalyticsUpdatedDate desc;$top=1)

How to query nested fields in MongoDB using Presto

I'm setting up a Presto cluster which I'd like to use to query a MongoDB instance. Data in my Mongo instance has the following structure:
{
_id: <value>
somefield: <value>
otherfield: <value>
nesting_1: {
nested_field_1_1: <value>
nested_field_1_2: <value>
...
}
nesting_2: {
nesting_2_1: {
nested_field_2_1_1: <value>
nested_field_2_1_2: <value>
...
}
nesting_2_2: {
nested_field_2_2_1: <value>
nested_field_2_2_2: <value>
...
}
}
}
Just by plugging it, Presto correctly identifies and creates columns for the values in the top level (e.g. somefield, otherfield) and in the first nesting level -- that is, it creates a column for nesting_1, and its content is a row(nested_field_1_1 <type>, nested_field_1_2 <type>, ...), and I can query table.nesting1.nested_field_1_1.
However, fields with an extra nesting layer (e.g. nesting_2 and everything within it) are missing from the table schema. Presto's documentation for the MongoDB connector does mention that:
At startup, this connector tries guessing fields’ types, but it might not be correct for your collection. In that case, you need to modify it manually. CREATE TABLE and CREATE TABLE AS SELECT will create an entry for you.
While that seems to explain my use case, it's not very clear on how to "modify it manually" -- a CREATE TABLE statement doesn't seem appropriate, as the table is already there. The documentation also has a section on how to declare fields and their types, but it's also not very clear on how to deal with multiple nesting levels.
My question is: how do I setup Presto's MongoDB connector so that I can query fields in the third nesting layer?
Answers can assume that:
all nested fields' names are known;
there are only 3 layers;
there is no need to preserve the layered table layout (i.e. I don't mind if my resulting Presto table has all nested fields as unique columns like somefield, rather than one field with rows like nesting_1 in the above example);
extra points if the solution doesn't require me to explicitly declare the names and types of all columns in the third layer, as I have over 1500 of them -- but this is not a hard requirement.
On mongodb.properties, the property mongodb.schema-collection can be used to describe the schema of your MongoDB collections. As described in the documentation, this property is optional and the default is _schema.
it's not very clear on how to "modify it manually" -- a CREATE TABLE statement doesn't seem appropriate, as the table is already there.
It is supposed to be created and populated automatically but what I've noticed is that it is populated until some queries are executed, and it only generates the schema for the collections that are queried.
However, there is a open bug, some fields/columns are not automatically picked up.
Also, once an entry for a collection is created/populated it won't be updated automatically, any update needs to be done manually (if the collection start to have new fields they won't be detected automatically).
To manually update the schema, the field column is just another entry in the fields array, as mentioned in the doc, it has three parts :
name Name of the column in the Presto table, it needs to match with the name of the collection field.
type Presto type of the column. Here are the available types, the ROW type can be used for nested properties.
hidden Hides the column from DESCRIBE <table name> and SELECT *. Defaults to false.
My question is: how do I setup Presto's MongoDB connector so that I can query fields in the third nesting layer?
The schema definition for a MongoDB collection like the one you posted will be containing something like:
...
"fields": [
{
"name": "_id",
"type": "ObjectId",
"hidden": true
},
{
"name": "somefield",
"type": "varchar",
"hidden": false
},
{
"name": "otherfield",
"type": "varchar",
"hidden": false
},
{
"name": "nesting_1",
"type": "row(nested_field_1_1 varchar, nested_field_1_2 bigint)",
"hidden": false
},
{
"name": "nesting_2",
"type": "row(nesting_2_1 row(nested_field_2_1_1 varchar, nested_field_2_1_2 varchar),nesting_2_2 row(nested_field_2_2_1 varchar, nested_field_2_2_2 varchar))",
"hidden": false
}
]
...
It can be queried using . over the columns, like:
SELECT nesting_2.nesting_2_1.nested_field_2_1_1 FROM table;
If the mongo collection being queried does not have a fixed schema, indicated in the _schema collection, Presto is not able to infer the document structure.
If you prefer,the option is to explicitly declare the schema in the connector configuration, using field mongodb.schema-collection, as described in the documentation. You can set it to a different mongo collection which stores the same values, and create this collection directly.
Nested fields can be declared using the ROW data type, which is also described in the docs and behaves like what would be a struct or dictionary in other programming languages.
You can create a collection in mongodb, for example "presto_schema" in your database and insert sample schema like this
db.presto_schema.insertOne({
"table" : "your_collection",
"fields" : [
{
"name" : "_id",
"type" : "ObjectId",
"hidden" : true
},
{
"name" : "last_name",
"type" : "varchar",
"hidden" : false
},
{
"name" : "id",
"type" : "varchar",
"hidden" : false
}
]})
In your presto mongodb.properties, add the property like this:
mongodb.schema-collection=presto_schema
From now, presto will use "presto_schema" instead of your default "_schema" to query.

Is it possible to query by array content?

Using the FIWARE PointOfInterest data model I would like to filter by POI category which is an array. For instance
http://130.206.118.244:1027/v2/entities?type=PointOfInterest&options=keyValues&attrs=name,category&limit=100&q=category=="311"
having as entity instances something like
{
"id": "Museum-f85a8c66d617c23d33847f8110341a29",
"type": "PointOfInterest",
"name": "The Giant Squid Centre",
"category":
[
"311"
]
},
{
"id": "Museum-611f228f42c7fbfa4bd58bad94455055",
"type": "PointOfInterest",
"name": "Museo Extremeño e Iberoamericano de Arte Contemporáneo",
"category":
[
"311"
]
},
Looking to the NGSIv2 specification it seems that works in the way you mention:
Single element, e.g. temperature==40. For an entity to match, it must contain the target property (temperature) and the target property value must be the query value (40) (or include the value, in case the target property value is an array).
I mean, in particular the part that says:
...or include the value, in case the target property value is an array.

Is it possible to run a query using Orion where the searching criteria is given by attributes value?

As I create entities in an Orion server, I can search by ID, as flat or using regular expressions:
http://<localhost>:1026/v1/queryContext
Content:
{
"entities": [
{
"type": "Sensor",
"isPattern": "true",
"id": "sensor_1.*"
}
],
"attributes": ["temperature","humidity"]
}
In the above example I'd get all objects of type "Sensor" whose ID starts with "sensor_1", and their attributes "temperature" and "humidity". I wonder if there is any way that would allow me to search by specific attribute value, for example to get those sensors whose humidity is over "60.2", or this selection must be done over the retrieved data queried by ID.
Not in the current Orion version (0.19.0) but it will be possible in the future. Have a look to the attribute::<name> filter with the = operator in this document.

mongodb-php: "key" side value for nested querying of find() function doesnot work

I want to retrive record which are matching to booking's client id & want to show it to client. I am doing the following:
$mongoDb = $mongoDb->selectCollection('booking');
$bookingInfo = $mongoDb->find(array("client.id" => $_SESSION['client_id']));
My mongo database record looks like this:
"paymentDue": "",
"client": {
"contacts": [
{
"name": "loy furison",
"email": "loy#hotmail.com"
}
],
"id": "5492abba64363df013000029",
"name": "Birdfire"
},
want to fire the query with key value as client.id in find function. But this query doesnt work..whats the issue
I got a little logic that is different by key name only. If i find it with client.name then i shows me records & there i need to insert these in json object & then through foreach loop each record if i retrive & compare then it works...got it but the expected doesnt work why?????...didnt get:-!