Unable to get the output of a web activity into a sql-table using azure-data-factory.
This is what i have done and where im getting stuck (Step 3).
Steps:
1.Get a token from a API-call
Get the results from API Call using token from step 1
and tkae this results in a successfull query that provides me with 'JSON'
Take the result from previous activity 'JSON' and put in in a azure sql database table.
azure-datafactory - web activites
Can you not get this done using a copy activity ? You should configure the source as Web activity and sink as SQL. I was playing with this http://dummy.restapiexample.com/api/v1/employees and we needed to introduce the structure . This is what I did and it work .
"source": {
"type": "RestSource",
"httpRequestTimeout": "00:01:40",
"requestInterval": "00.00:00:00.010",
"structure": [
{
"id": "id"
},
{
"employee_salary": "employee_salary"
}
]
},
"sink": {
"type": "SqlServerSink",
"writeBatchSize": 10000
},
"enableStaging": false
},
You can read more . https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping
Related
I am retrieving my azure cosmosdb/mongodb document from a custom trigger to azure functions.. But my objectId seems to be encrypted.. How to get the correct objectid..
for example ObjectId("5df88e60d588f00c32a3c9ce") is coming as ]øŽ`Õˆð2£ÉÎ
or ObjectId("5df88f92d588f00c32a3c9d1") is coming as ]ø’Õˆð2£ÉÑ
Is there a way to retrieve objectid in nodejs/python or any script if i give ]ø’Õˆð2£ÉÑ as input.
This is my function.json used in the azure function
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "cosmosDBTrigger",
"name": "documents",
"direction": "in",
"leaseCollectionName": "leases1",
"connectionStringSetting": "devcosmosdb_DOCUMENTDB",
"databaseName": "devcosmosdb",
"collectionName": "newCollection",
"createLeaseCollectionIfNotExists": "true"
}
]
}
This is my nodejs code..
module.exports = async function (context, documents) {
if (!!documents && documents.length > 0) {
context.log('Document Id: ', documents[0].id);
context.log(documents[0]);
}
}
This is my output and this is where i am not getting the objectid properly..
2020-06-16T17:16:38Z [Information] Executing 'Functions.changeTrigger' (Reason='New changes on collection newCollection at 2020-06-16T17:16:38.2618864Z', Id=adc9556a-133f-4e85-b533-5574283a5a7d)
2020-06-16T17:16:38Z [Information] Document Id: NWRmODhkZGRkNTg4ZjAwYzMyYTNjOWNj
2020-06-16T17:16:38Z [Information] {
id: 'NWRmODhkZGRkNTg4ZjAwYzMyYTNjOWNj',
_rid: 'KEcnAO163B4EAAAAAAAAAA==',
_self: 'dbs/KEcnAA==/colls/KEcnAO163B4=/docs/KEcnAO163B4EAAAAAAAAAA==/',
_ts: 1592327797,
_etag: '"0000c1d2-0000-0300-0000-5ee8fe750000"',
'$t': 3,
'$v': {
_id: { '$t': 7, '$v': ']øÝÕð\f2£ÉÌ' },
name: { '$t': 2, '$v': 'myname' },
email: { '$t': 2, '$v': 'my email' },
},
_lsn: 537
}
Please go to Azure portal to check the content of your document. I have done a test on my side, it just works fine.
Here is the document I used to test.
{
"id": "testid1",
"test1":"testvalue1",
"test2":{
"test21":"test21value",
"objectId":"5df88f92d588f00c32a3c9d1"
}
}
After clicking save button, the function will be triggered.
Here is my testing code.
def main(documents: func.DocumentList) -> str:
if documents:
logging.info('Document id: %s', documents[0]['id'])
logging.info('%s',documents[0].to_json())
The output is as below.
Update:
Currently only SQL API base is supported in Azure function cosmosdb trigger. You can also find the feature under Settings part.
The URI should be something like https://testbowman.documents.azure.com:443/
If you create a mongodb api cosmosdb account, you won't find the 'add to function' feature. And the URI should be something like https://tonycosmosdb.mongo.cosmos.azure.com:443/
I'm experiencing a MashupValueException while creating a Web Table Linked Service and I wonder if someone here have come across the same issue.
The config of the Linked Service is as follows:
{
"name": "WikipediaLS",
"properties": {
"annotations": [],
"type": "Web",
"typeProperties": {
"url": "https://www.the-numbers.com/",
"authenticationType": "Anonymous"
},
"connectVia": {
"referenceName": "MySelfHostedIR",
"type": "IntegrationRuntimeReference"
}
}
}
The error happens after filling the info on the form and clicking on "Test Connection". Then the following error pops up:
Connection failed
'Type=Microsoft.Data.Mashup.MashupValueException,Message=The supplied URL must be a valid 'http:' or 'https:' URL.,Source=Microsoft.Data.Mashup.ProviderCommon,' Activity ID: c8d924b9-cc81-49c1-85c8-70abfa85c859.
I have tested the extraction of the web table using Office 2016 and it works all right.
The aim is to extract the index 0 table from "https://www.the-numbers.com/movie/budgets/all".
This is what you are facing:
You can try to use below url, it can connect to the web successfully on my side:
https://www.the-numbers.com/movie/budgets/all
The website you give seems not stable. Sometimes you cannot visit the database.
When I invoke the skill with 'okay Google, ask {skillname} to {utterance}' I get the response 'The agent returned an empty tts" and the conversation closes. It doesn't even seem to hit my backend - I've tried hosting the backend code on both a local server and on AWS and the same issue happens in both cases. There's no log of the request being made either on Stackdriver on Google or on CloudWatch on AWS. The only response I get is the following in the debug tab:
{
"response": "The agent returned an empty TTS.",
"expectUserResponse": false,
"conversationToken": "EosDS2o4d0...",
"audioResponse": "",
"ssmlMarkList": [],
"debugInfo": {
"sharedDebugInfoList": [
{
"name": "Response Validation",
"debugInfo": "The agent returned an empty TTS.",
"subDebugEntryList": []
}
],
"conversationBuilderExecutionEventsList": []
},
"visualResponse": {
"visualElementsList": [
{
"displayText": {
"content": "The agent returned an empty TTS."
}
}
],
"suggestionsList": [],
"agentLogoUrl": ""
},
"clientError": 0,
"is3pResponse": true,
"clientOperationList": [],
"projectName": "",
"renderedHtml": ""
}
Nothing in any of the other tabs. When I try invoking the phrases in the same way on my phone, I just see a loading symbol and again nothing in the logs to say it's even hitting the backend.
Weirdly, this doesn't happen when I say 'okay Google, talk to {skillname} {utterance}' - this works absolutely fine.
All the intents work perfectly well when I've opened my action first, and then invoke them. I've built this using Jovo and published on Alexa also and the problem is only happening on Google.
I was trying to test Transactions via Playground, Composer-Rest-server and Finally POSTMAN.
The Transactions worked perfectly on Playground (Figure 1 and 2) and also on the Composer-Rest-Server (Figure 3 and 4). But then I tried the same using POSTMAN, for which I am getting an Error ( Status Code 422 )(Figure 5).
I'm using the basic-sample-network. And the Transaction just updates the value of the asset.
Figure 1 - Executing Sample Transaction On Playground
Figure 2 - Updated Value of Asset #3952
Figure 3 - Executing the Transaction on the composer Rest Server
Figure 4 - Updated Value of Asset #3952 after the Rest Server Transaction
Figure 5 - Error While Using POSTMAN
The Error
"error": {
"statusCode": 422,
"name": "ValidationError",
"message": "The `org_example_basic_SampleTransaction` instance is not valid. Details: `asset` can't be blank (value: undefined); `newValue` can't be blank (value: undefined).",
"details": {
"context": "org_example_basic_SampleTransaction",
"codes": {
"asset": [
"presence"
],
"newValue": [
"presence"
]
},
"messages": {
"asset": [
"can't be blank"
],
"newValue": [
"can't be blank"
]
}
},
"stack": "ValidationError: The `org_example_basic_SampleTransaction` instance is not valid. Details: `asset` can't be blank (value: undefined); `newValue` can't be blank (value: undefined).\n at ...
What is the reason for this Error and How can I solve it?
You have to post data in JSON format.
Using PowerShell, how can I query Team Foundation Server 2017 Update 2 (on-premises) to get the code coverage percentage metric from the latest completed gated check-in?
I've not found a clear API call to use in the MS reference documentation. In the web interface, I can see in the dashboard for a given completed build the percentage value and a link to download the entire Visual Studio coverage file. I don't want the file, though. I just want to make an API call and get the percent coverage value for the last successful build of a given definition.
Unfortunately, the docs for VSTS's REST API are down right now, but this should get you started (pulled from Google's Cache).
This endpoint handles everything related to Tests and Code Coverage.
https://{instance}/DefaultCollection/{project}/_apis/test/codeCoverage?api-version={version}[&buildId={int}&flags={int}]
Provide values for the following and then run this Invoke-RestMethod to get the data back.
$Instance = 'fabrikam-fiber-inc.visualstudio.com' #your URL here
$ProjectName = #YourProjectNameHere
$buildID = #YourBuildIDHere
$version = '2.0-preview'
Invoke-RestMethod -uri https://$Instance/DefaultCollection/$ProjectName/_apis/test/codeCoverage?api-version=$version
Here's a sample response you can get back:
Status code: 200
{
"value": [
{
"configuration": {
"id": 51,
"flavor": "Debug",
"platform": "Any CPU",
"uri": "vstfs:///Build/Build/363",
"project": {}
},
"state": "0",
"lastError": "",
"modules": [
{
"blockCount": 2,
"blockData": "Aw==",
"name": "fabrikamunittests.dll",
"signature": "c27c5315-b4ec-3748-9751-2a20280c37d5",
"signatureAge": 1,
"statistics": {
"blocksCovered": 2,
"linesCovered": 4
},
"functions": []
}
],
"codeCoverageFileUrl": "..."
}
],
"count": 1
}
It looks like blocksCovered and livesCovered are probably the closest you'll get from the API. Let me know if you need some help or get stuck. Eventually, the docs will be back online at this URL.