data factory - check if result of pipeline task contains a specific string or value? - azure-data-factory

I have a web task in a pipeline.
The result of the api call can be one of 3:
An item exists in a database.
an item does not exist in a database.
the body sent to the call was invalid.
If the record exists, I want to take further action.
As the resulting JSON from the API call is entirely different when an entity exists vs does not exists, how can I check for if a specific value is returned, if there is a chance it wont be returned at all?
Below is the output from a successful call. I'd like to check for the "Tag" attribute, but also cover a case where this does not exist.
How is this done using a dynamic content expression, seeing as there is no exists function?
{
"Data": [
{
"Data": {
"RowNumber": 0,
"Tag": "GLT-GM-45",
"GMStatus": 0,
"CustomValues": [
{
"CSLabel": "Asset Status",
"CSType": 0,
"CSValue": "Assigned"
},
{
"CSLabel": "Usage Status",
"CSType": 0,
"CSValue": "Permanent"
}
],
"AttachmentsToAdd": [],
"AttachmentsToDelete": []
},
"Messages": [
{
"ResultCode": 0,
"Message": "Success",
"HttpStatusCode": 200,
"FieldName": "GLT-GM-45"
}
],
"HasError": false,
"HasHttpError": false,
"HasMessage": true,
"HasSuccessWithMoreDataRemaining": false
}
],
"Messages": [
{
"ResultCode": 0,
"Message": "Success.",
"HttpStatusCode": 200,
"FieldName": ""
}
],
"TotalRecordsLongCount": 1,
"HasSuccessWithMoreDataRemaining": false,
"HasError": false,
"HasMessage": true,
"HasHttpError": false,
"ADFWebActivityResponseHeaders": {
"Connection": "keep-alive",
"Pragma": "no-cache",
"WaspResult": "WaspResult",
"X-UA-Compatible": "IE=edge;IE=edge",
"Cache-Control": "no-store, no-cache",
"Date": "Tue, 02 Aug 2022 11:58:31 GMT",
"Server": "Microsoft-IIS/10.0;Microsoft-IIS/8.5",
"X-AspNet-Version": "4.0.30319",
"X-Powered-By": "ASP.NET;ASP.NET",
"Content-Length": "2364",
"Content-Type": "application/json",
"Expires": "-1"
},
"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (North Europe)",
"executionDuration": 0,
"durationInQueue": {
"integrationRuntimeQueue": 1
},
"billingReference": {
"activityType": "ExternalActivity",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.016666666666666666,
"unit": "Hours"
}
]
}
}

Wouldn't you be able to use an IF Condition that uses a contains function? Obviously I do not have a way to recreate your output, but I did the following to simulate something.
Added an If Condition to a pipeline
Added dynamic content that uses a contains function. The first part was a string containing part of your output from above (copy/paste). The part behind the comma (i.e. the search string) being 'Tag'
Added False activity causing the component to fail
Added a True activity to set a variable to be equal to "Found"!.
The full If Condition:
#contains('{
"Data": [
{
"Data": {
"RowNumber": 0,
"Tag": "GLT-GM-45",
"GMStatus": 0
}
}', 'Tag')
Seeing the 'Tag' should be found in the string, it produces the result of setting the variable:
Changing the search word to NotATag which does not exist in the string produces this instead:
My guess would be that you can substitute the hard-coded string with something like the following:
#contains(activity('Web1').output, 'Tag')
Obviously, change 'Web1' to the name of your web component.

Might not be an ideal way but you can do something like below to check the existence of Tag attribute in output (the reason i am trying to check the string size is the pipeline exp expected the return type to be boolean, hence using condition to achieve it), you can map the other actions based on If activity status. If the property doesn't exist if condition fails with error.
Condition
Mapping

Related

Create new DateTime in Microsoft RulesEngine rule

I'm looking at the Microsoft RulesEngine. This appears to have what I'm looking for but I'm having difficulties getting some of my rules to execute. I have a need to dynamically set/check date times for the customer's rules. I can do this in code and pass it into the ruleset, but I want to eliminate the need to recompile my app to send input values should they decide to change their rules. Having everything in my json file is the optimal solution here. I can then modify the rules, resend a json and call it done.
Here's the scenario: If a person has a license and they are renewing between certain dates then they should get a specific cost. Yes, it's a bit contrived but it's what I need to meet the acceptance criteria. The following doesn't work. Any recommendations/pointers? Thank you in advance!
Here's the sample:
[
{
"WorkflowName": "WithinRenewalPeriod",
"GlobalParams": [
{
"Name": "Last_December",
"Expression": "new DateTime(DateTime.Now.Year - 1 , 12, 1)"
},
{
"Name": "Current_March",
"Expression": "new DateTime(DateTime.Now.Year, 3, 1)"
},
{
"Name": "Current_July",
"Expression": "new DateTime(DateTime.Now.Year, 7, 10)"
}
],
"Rules": [
{
"RuleName": "PriorLicense_Between_Dec1_Mar1",
"Enabled": true,
"ErrorType": 0,
"RuleExpressionType": 0,
"Expression": "person.HistoricalLicenseList.Count() > 0 AND (DateTime.Now >= Last_December AND DateTime.Now <= Current_March)",
"SuccessEvent": "1"
},
{
"RuleName": "HasPriorLicense_Between_Mar2_July10",
"Enabled": true,
"ErrorType": 0,
"RuleExpressionType": 0,
"Expression": "person.HistoricalLicenseList.Count() > 0 AND (DateTime.Now >= Current_March AND DateTime.Now <= Current_July)",
"SuccessEvent": "3000"
}
]
}
]
I figured it out. I was assuming that you would need to "new" the DateTime. This wasn't the case. I ended up removing the "new" keyword and was able to get the GlobalParams to work:
This is wrong:
This is the correct way to instantiate "new" DateTime in GlobalParams:
The end results generated as Inputs:

Can not create new layer (featuretype) in GeoServer using REST API

So I just used 2 working days trying to figure this out. We are automatic rendering process for maps. All the data is given in SQL base and my job is to write "wrapper" so we can implement this in our in-house framework. I managed all but one needed requests.
That request is POST featuretype since this is a way of creating a layer that can later be rendered.
I have all requests saved in postman for pre-testing on example data given by geoserver itself. I can't even get response with status code 201 and always get 500 internal server error. This status is described as possible syntax error in sytax. But I actually just copied and pasted exampled and used geoserver provided data.
This is the requst: http://127.0.0.1:8080/geoserver/rest/workspaces/tiger/datastores/nyc/featuretypes
and its body:
{
"name": "poi",
"nativeName": "poi",
"namespace": {
"name": "tiger",
"href": "http://localhost:8080/geoserver/rest/namespaces/tiger.json"
},
"title": "Manhattan (NY) points of interest",
"abstract": "Points of interest in New York, New York (on Manhattan). One of the attributes contains the name of a file with a picture of the point of interest.",
"keywords": {
"string": [
"poi",
"Manhattan",
"DS_poi",
"points_of_interest",
"sampleKeyword\\#language=ab\\;",
"area of effect\\#language=bg\\;\\#vocabulary=technical\\;",
"Привет\\#language=ru\\;\\#vocabulary=friendly\\;"
]
},
"metadataLinks": {
"metadataLink": [
{
"type": "text/plain",
"metadataType": "FGDC",
"content": "www.google.com"
}
]
},
"dataLinks": {
"org.geoserver.catalog.impl.DataLinkInfoImpl": [
{
"type": "text/plain",
"content": "http://www.google.com"
}
]
},
"nativeCRS": "GEOGCS[\"WGS 84\", \n DATUM[\"World Geodetic System 1984\", \n SPHEROID[\"WGS 84\", 6378137.0, 298.257223563, AUTHORITY[\"EPSG\",\"7030\"]], \n AUTHORITY[\"EPSG\",\"6326\"]], \n PRIMEM[\"Greenwich\", 0.0, AUTHORITY[\"EPSG\",\"8901\"]], \n UNIT[\"degree\", 0.017453292519943295], \n AXIS[\"Geodetic longitude\", EAST], \n AXIS[\"Geodetic latitude\", NORTH], \n AUTHORITY[\"EPSG\",\"4326\"]]",
"srs": "EPSG:4326",
"nativeBoundingBox": {
"minx": -74.0118315772888,
"maxx": -74.00153046439813,
"miny": 40.70754683896324,
"maxy": 40.719885123828675,
"crs": "EPSG:4326"
},
"latLonBoundingBox": {
"minx": -74.0118315772888,
"maxx": -74.00857344353275,
"miny": 40.70754683896324,
"maxy": 40.711945649065406,
"crs": "EPSG:4326"
},
"projectionPolicy": "REPROJECT_TO_DECLARED",
"enabled": true,
"metadata": {
"entry": [
{
"#key": "kml.regionateStrategy",
"$": "external-sorting"
},
{
"#key": "kml.regionateFeatureLimit",
"$": "15"
},
{
"#key": "cacheAgeMax",
"$": "3000"
},
{
"#key": "cachingEnabled",
"$": "true"
},
{
"#key": "kml.regionateAttribute",
"$": "NAME"
},
{
"#key": "indexingEnabled",
"$": "false"
},
{
"#key": "dirName",
"$": "DS_poi_poi"
}
]
},
"store": {
"#class": "dataStore",
"name": "tiger:nyc",
"href": "http://localhost:8080/geoserver/rest/workspaces/tiger/datastores/nyc.json"
},
"cqlFilter": "INCLUDE",
"maxFeatures": 100,
"numDecimals": 6,
"responseSRS": {
"string": [
4326
]
},
"overridingServiceSRS": true,
"skipNumberMatched": true,
"circularArcPresent": true,
"linearizationTolerance": 10,
"attributes": {
"attribute": [
{
"name": "the_geom",
"minOccurs": 0,
"maxOccurs": 1,
"nillable": true,
"binding": "com.vividsolutions.jts.geom.Point"
},
{},
{},
{}
]
}
}
So it is example case and I can't get any useful response from the server. I get the code 500 with body name (the first item in json). Similarly I get same code with body FeatureTypeInfo when trying with xml body(first tag).
I already tried the request in new instance of geoserver in Docker (changed the port) and still no success.
I check if datastore, workspace is available and that layer "poi" doesn't yet exists.
Here are also some logs of request (similar for xml body):
2018-08-03 07:35:02,198 ERROR [geoserver.rest] -
com.thoughtworks.xstream.mapper.CannotResolveClassException: name at
com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79)
at .....
Does anyone know the solution to this and got it working. I am using GeoServer 2.13.1
So i was still looking for the answer and using this post (https://gis.stackexchange.com/questions/12970/create-a-layer-in-geoserver-using-rest) got to the right content to POST featureType and hence creating a layer in GeoServer.
The documentation is off in REST API docs.
Using above link I found out that when using JSON there is a missing insertion in JSON. For API to work here we need to add:
{featureType:
name: "...",
nativeName: "...",
.
.
.}
So that it doesn't start with "name" attribute but it is contained in "featureType".
I didn't try that for XML also but I guess it could be similar.
Hope this helps someone out there struggling like I did.
Blaz is correct here, you need an outer object of FeatureType and then an inner object with your config. So;
{
"featureType": {
"name": "layer",
"nativeName": "poi",
"your config": "stuff"
}
I find though that using a post request I get very little if any response and its not obvious if the layer creation worked. But you can call http://IP:8080/geoserver/rest/layers.json to check if your new layer is there.
It costs me a lot of time to create FeatureTypes using REST API. Use Json like this really works:
{
"featureType": {
"name": "layer",
"nativeName": "poi"
"otherProperties...":"values..."
}
And use Json below to create Workspace:
{
"workspace": {
"name": "test_workspace"
}
}
The REST API is out of date now. That's disappointing. Is there anyone knows how to get the lastest REST API document?

How to get rid of the json attributes "variant", "annotations", from the response

I have started using the dropwizard to develop a REST server. The Issue as long resource-method return an EntityType (say Enrolment), the out put is as expected but I have decided to send custom status codes using the below code
Response.status(Response.Status.PRECONDITION_FAILED)
.entity(Entity.json(new enrolment, AdapterResponseStatus.FAILURE)))
.build();
Everything is fine but the out put now contains a few more extra extra attributes as shown below.
{
"entity":
{
"id": 1267,
"courseId": "5798890",
"userName": "user#abc.com",
"tenantId": "tenant1",
"status": "approved",
"link": "/enrollments/null"
},
"variant":
{
"language": null,
"mediaType":
{
"type": "application",
"subtype": "json",
"parameters":
{
},
"wildcardType": false,
"wildcardSubtype": false
},
"encoding": null,
"languageString": null
},
"annotations":
[
],
"mediaType":
{
"type": "application",
"subtype": "json",
"parameters":
{
},
"wildcardType": false,
"wildcardSubtype": false
},
"language": null,
"encoding": null
}
I was expecting "entity" property alone but was getting other attributes. Is there any to get rid of them as no one is going to consuming them?
Those tags appear even if I replace my entity object (enrolment) with an empty string.
If you look at the signature of the ResponseBuilder's entity method, it takes the object directly; unlike Jersey client which requires a special Entity object that coincidentally has annotations and variants fields within.
Change your code to this:
Response.status(Response.Status.PRECONDITION_FAILED)
.entity(new Enrolment())
.build();

Getting specific headers along with other body data in Users.thread: get

You can very easily select what headers you want in a Users.thread: get-request, like so:
format = metadata
metadataHeaders = From
GET https://www.googleapis.com/gmail/v1/users/me/threads/14eaffaf5e3e8242?metadataHeaders=From&format=metadata&key={YOUR_API_KEY}
Response:
{
"id": "14eaffaf5e3e8242",
"historyId": "510358",
"messages": [
{
"id": "14eaffaf5e3e8242",
"threadId": "14eaffaf5e3e8242",
"labelIds": [
"SENT",
"INBOX",
"IMPORTANT"
],
"snippet": "Wow Emil!",
"historyId": "510292",
"internalDate": "1437471536000",
"payload": {
"mimeType": "multipart/mixed",
"headers": [
{
"name": "From", // I just got the header I asked for.
"value": "Emil Tholin <emtholin#gmail.com>"
}
]
},
"sizeEstimate": 9260
}, ...
}
You can also get certain parts of the body very easily. E.g. here I ask for the attachmentIds of all the attachments in the message body:
field = messages/id,messages/payload/parts/body/attachmentId
GET https://www.googleapis.com/gmail/v1/users/me/threads/14eaffaf5e3e8242?fields=messages%2Fid%2Cmessages%2Fpayload%2Fparts%2Fbody%2FattachmentId&key={YOUR_API_KEY}
Response:
{
"messages": [
{
"id": "14eaffaf5e3e8242",
"payload": {
"parts": [
{
"body": { // This message had an attachment.
"attachmentId": "ANGjdJ_0lphTo48BO0xBT_YOSo3tYah23hzpjyATe3GwfziK0I6401P_8-ZYoGuCQPHhpPP0-S_pjL68WIEZzQ0tu72RcIOE4UY3kA4u8PjXPf3Cm5PxVJjmH9N0hm0fFX31RYo8bfZQ6l7bDbYbnCSZbckG7g8enGaKMPbBzIEEC4HXr_YghOYWSfrXKXiFLnxWN4LfsFk3IXUN2tVvMe_0xMhDDfBlqYPnXHr2PhPghq7bQojNxiH4YziIqaKmwiU4xqVfygbae-K-_Q2blyz0EgI4OXjMzwz56Q5e1w"
}
}
]
}
},
{
"id": "14eaffb277b61cd0" // This message had no attachment.
}, ...
]
}
As you can see in the first request, no part of the body is retrieved when asking for specific metadata headers. Individual fields are also hard to pick out in the fields-parameter, since headers are not key-value pairs, but objects on the form { "name": <HEADER_NAME>, "value": <HEADER_VALUE> }.
Is there any way to combine these two requests? I would like to get all the relevant metadata about a message at the same time as getting how many attachments there are in the message.
From Users.threads.get.
"metadata": Returns email headers with message metadata such as identifiers and labels.
The fields parameter is only used to limit the amount of data returned. Since metadata mode is already a limited response you can't use fields to get data outside of that subset. Your best option would be to use your second example with fields and then filter the metadata values locally.

GoodData "Create Report Definition" API Call giving 500 Internal Server Error

I'm trying to create a report definition using the GoodData REST API. I use the following endpoint to invoke the rest call.
"/gdc/md/{project-id}/obj"
When i try to invoke the API call with the following dataset in which the projectId and the userId are valid, it gives me the error with the response code 500.
{
"reportDefinition": {
"content": {
"filters": [],
"format": "grid",
"grid": {
"rows": [],
"columns": [
"metricGroup"
],
"sort": {
"columns": [],
"rows": []
},
"columnWidths": [],
"metrics": [
{
"uri": "/gdc/md/qy48iv4flikdlcwpwioizuip74wt8nb5/obj/63f3cecd2a8d3ce2ec9378381c8f39e3",
"alias": ""
}
]
}
},
"meta": {
"title": "Sample report definition",
"summary": "This is a sample report",
"tags": "",
"deprecated": 0,
"category": "samplecategory"
}
}
}
{
"error": {
"message": "Internal server error. Please fill in bug report with request_id='lp78FL5S1IPMqB2n'"
}
}
I'm certain that the user project_id and the user_id are valid. Is this an error in the API?
Thank you in advance.
Apart from the metrics URI that looks weird (hash instead of numeric ID), I was able to dig in our logs an error that says: "Category is not equal to tag structure".
In your example you have its value set to "samplecategory". "category" property defines what type of object are you creating. If you are creating a report definition it should have value of "reportDefinition".
Last time I worked with GoodData API, metrics had numeric IDs. That seems most likely to be the culprit. Where did you get "/gdc/md/qy48iv4flikdlcwpwioizuip74wt8nb5/obj/63f3cecd2a8d3ce2ec9378381c8f39e3" from, especially the "63f3cecd2a8d3ce2ec9378381c8f39e3" part?