Azure Data Factory LOOKUP possibilities - azure-data-factory

I'm trying to add a lookup activity that will look up a series of values(companyIds) and insert the values into here
["/apiCore/api/countries","/apiCore/api/Metrics/MyLookup"]
At present my configuration looks like the following:
I was wondering if it was possible to add Lookup activity to insert the values as follows:
And then enter a parameter like #activity('MyLookup').output.value to:
["/apiCore/api/countries","/apiCore/api/Metrics/MyLookup"] so it change to:
["/apiCore/api/countries","/apiCore/api/Metrics/** #activity('MyLookup').output.value**"]
Can someone let me know if the above would work? If not, do you have any suggestions?
I got the answer to my suggestion with the following error:
{
"errorCode": "2200",
"message": "ErrorCode=HttpRequestFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with client error, status code 400 BadRequest, please check your activity settings. If you configured a baseUrl that includes path, please make sure it ends with '/'.\nRequest URL: https://pm2.preqinsolutions.com/apiCore/api/Metrics/#activity('MyLookup').output.value.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (400) Bad Request.,Source=System,'",
"failureType": "UserError",
"target": "dynamoCompanies",
"details": []
}
You will notice the error is with:
#activity('MyLookup').output.value
here:
https://pm2.preqinsolutions.com/apiCore/api/Metrics/#activity('MyLookup').output.value.,
Any suggestions will be very much welcomed
Updated question

Using string interpolation, you can build the URL by adding the output of look up activity. Instead of using the URL as https://pm2.preqinsolutions.com/apiCore/api/Metrics/#activity('MyLookup').output.value, you can try the following:
I have taken a sample lookup which gives the following output:
Now, I have used a variable to build the URL value. I have taken the value /apiCore/api/Metrics/ in a parameter called relativeURL.
https://pm2.preqinsolutions.com#{pipeline().parameters.relativeUrl}#{activity('Lookup1').output.value}
This generates the required URL. When I call it in web activity, you can see the URL that is being passed in debug input (I don't have access to the activity fails).

Related

Parameterize Parameter in API Body in dataflow

below is the image of the dynamic body in Web activity wherein I am leveraging a pipeline parameter
With the above set of configuration and values, the web activity is successfully getting executed.
Now for the same REST API, I am trying to use it as source in dataflow :
with the same aspect of configuration and it is failing with below error :
Dataflow param:
Error:
ailure to read most recent page request: DF-REST_001 - Error response from server: Some({"error":{"code":"DatasetExecuteQueriesError","pbi.error":{"code":"DatasetExecuteQueriesError","parameters":{},"details":[{"code":"DetailsMessage","detail":{"type":1,"value":"The JSON DDL request failed with the following error: Invalid JavaScript property identifier character: }. Path '', line 1, position 7.."}},{"code":"AnalysisServicesErrorCode","detail":{"type":1,"value":"3239182519"}}]}}}), Status code: 400. Please check your request url and body.
Can someone please help what I am doing Wrong?
To use dataflow parameter within the expression builder, we have to concatenate the parameter with our data. Look at the following demonstration.
I am using derived column to show how to use parameters within expression builder. Using the dynamic content same as yours, I get the following output:
'{"queries":[{"query":"{$query}"}],"serializerSettings":{"includeNulls":true}}'
String interpolation does not work in the dataflow expression builder. To get it correctly, you can concatenate the parameter using + to get desired result.
'{"queries":[{"query":"'+$query+'"}],"serializerSettings":{"includeNulls":true}}'
Using the following content in expression builder also works (using concat() function).
concat('{"queries":[{"query":"',$query,'"}],"serializerSettings":{"includeNulls":true}}')

core_user_create_user and moodle webservice setup not working

I have done everything needed to setup webservices on my moodle 3.11 instance, including roles/capabilities/user. However sending a test request always gives {
"exception": "dml_missing_record_exception",
"errorcode": "invalidrecord",
"message": "Can't find data record in database table external_functions."
}
The URL to access it is of the format https:///moodle/webservice/rest/server.php?wsfunction=core_user_create_user&service=mymoodleusermanage&moodlewsrestformat=json&users[0][username]=ABC&users[0][firstname]=VPTest&users[0][lastname]=None&users[0][email]=mail#xxx.com&users[0][password]=xxxxx&users[0][auth]=manual&wstoken=xxxxxxxxxxxxxx
The service parameter is correctly set to the shortname of the service. Does the service have to be defined anywhere additionally apart from Site Administration->Server->Web Services->External Services->Custom Services
Thanks for any help that can be given
The answer is very simple - you are trying to call a non-existent webservice function (hence the error message about being unable to find the database record for the function in the external_functions database table).
If you look in the Moodle code: https://github.com/moodle/moodle/blob/master/lib/db/services.php#L1717 you will see that the function is called core_user_create_users - with an "s" at the end of it.
If you add that extra "s" into the URL parameters you are using, then it should work.
https:///moodle/webservice/rest/server.php?wsfunction=core_user_create_user&service=mymoodleusermanage&moodlewsrestformat=json&users[0][username]=ABC&users[0][firstname]=VPTest&users[0][lastname]=None&users[0][email]=mail#xxx.com&users[0][password]=xxxxx&users[0][auth]=manual&wstoken=xxxxxxxxxxxxxx
you must change username all character small letter [username]=ABC like this [username]=abc and add s wsfunction=core_user_create_users

RestTemplate considering a resource to be a parameter

My request goes like this... https://......./results/#codeResults
When i hit this get request with postman, i get the desired output. (Please note that #codeResults is used as it is in the request and is not to be replaced by any value)
But when I try to hit this using exchange method of RestTemplate it says -
Status Code : 400
message: invalid resultId: #codeResults
As an alternative approach, I tried to pass #codeResults as a path param using URIComponentBuilder as well but still I am getting the same issue.
Please provide any pointers on this.

Invalid_request_parameter (create and sending envelopes)

I'm trying to use a service of DocuSign API in an abap project. I want to send a document to a specific email so it can be signed. But im getting the following error:
"errorCode": "INVALID_REQUEST_PARAMETER",## "message": "The request contained at least one invalid parameter. Query parameter 'from_date' must be set to a valid DateTime, or 'envelope_ids' or 'transaction_ids' must be specified.
I tried the following:
CALL METHOD cl_http_client=>create_by_url
EXPORTING
url = l_url (https://demo.docusign.net/restapi/v2/accounts/XXXXXX')
proxy_host = co_proxy_host
proxy_service = co_proxy_service
IMPORTING
client = lo_http_client
lo_http_client->request->set_method( method = 'POST').
CALL METHOD lo_http_client->request->set_header_field
EXPORTING
name = 'Accept'
value = 'application/json'.
CALL METHOD lo_http_client->request->set_header_field
EXPORTING
name = 'X-DocuSign-Authentication'
value = get_auth_header( ). (json auth header)
CALL METHOD lo_http_client->request->set_cdata
EXPORTING
data = create_body( ).
This is my body:
CONCATENATE
`{`
`"emailSubject": "DocuSign REST API Quickstart Sample",`
`"emailBlurb": "Shows how to create and send an envelope from a document.",`
`"recipients": {`
`"signers": [{`
`"email": "test#email",`
`"name": "test",`
`"recipientId": "1",`
`"routingOrder": "1"`
`}]`
`},`
`"documents": [{`
`"documentId": "1",`
`"name": "test.pdf",`
`"documentBase64":` `"` l_encoded_doc `"`
`}],`
`"status": "sent"`
`}` INTO re_data.
The api request to get the Baseurl is working fine. (I know the error is quite specific what the problem is, but i cant find any sources on the docusign api documentation that one of the mentioned parameters should be added to the request)
Thank you in regards
The error message seems to indicate that you're Posting to an endpoint that requires certain query string parameters -- but you're not specifying them as expected in the query string. I'd suggest you check the DocuSign API documentation for the operation you are using, to determine what query string parameters it requires, and then ensure that you're including those parameters in your request URL.
If you can't figure this out using the documentation, then I'd suggest that you update your post to clarify exactly what URL (endpoint) you are using for the request, including any querystring parameters you're specifying in the URL. You can put fake values for things like Account ID, of course -- we just need to see the endpoint you are calling, and what qs params you're sending.
To create an envelope, use
https://demo.docusign.net/restapi/v2/accounts/XXXXXX/envelopes
instead of
https://demo.docusign.net/restapi/v2/accounts/XXXXXX
Thank you for all the answers, i found the mistake. Creating the request wasn´t the problem. I was using the wrong "sending"-method -_-.
now its working :)
lo_rest_client->post( EXPORTING io_entity = lo_request_entity ).

Google Fit REST API "Unable to fetch DataSource for Dataset: xyz"

I'm testing out a few things in the OAuth 2.0 Playground and trying to get data in and out of Google Fit using their REST API
I have done this previously with success, I just didn't write down what I did.. now I've come back to make it a proper thing and can't get it working again.
I have access to Google Fit datasources via the dashboard. I can get a list of the dataSources that exist from:
https://www.googleapis.com/fitness/v1/users/me/dataSources
And that is successful. I have also created my own stream which has a single floating point weight value on it called
raw:com.google.weight:b6ac18c0:dten.sync
It already has data in it, I put it there last time I used it. I can select all that data by requesting a GET on the following
https://www.googleapis.com/fitness/v1/users/me/dataSources/raw:com.google.weight:b6ac18c0:dten.sync/datasets/0-1432193482000000000
It returns me all the data points I entered last time as JSON
I then try to PATCH the data adding my own data to the folliwng URL
https://www.googleapis.com/fitness/v1/users/me/dataSources/raw:com.google.weight:b6ac18c0:dten.sync/datasets/1432193482000000000-1432193482000000000
With this as a the request body
{
"minStartTimeNs": "1421912895000000000",
"maxEndTimeNs": "1432193482000000000",
"dataSourceId": "raw:com.google.weight:b6ac18c0:dten.sync",
"point": [
{
"startTimeNanos": "1421912895000000000",
"modifiedTimeMillis": "1421912895000",
"endTimeNanos": "1421912895000000000",
"value": [
{
"fPVal": 89.1
}
],
"dataTypeName": "com.google.weight"
}
]
}
But I get back
{
"error": {
"code": 400,
"message": "Unable to fetch DataSource for Dataset: raw:com.google.weight:b6ac18c0:dten.sync",
"errors": [
{
"domain": "global",
"message": "Unable to fetch DataSource for Dataset: raw:com.google.weight:b6ac18c0:dten.sync",
"reason": "invalidArgument"
}
]
}
}
I can't find any one referencing a similar anywhere soo I'm here
Also note if I miss spell my source it tells me off because they don't match the URL, if i include an empty list of data points I get the same error. I'm quite lost so I'm throwing it out there to see if anyone knows what that means
Thanks in advance
edit: i tried changing the hex code for my project's integer code and got an error about untrusted source. so i tried making a new test data source which works as expected. Slightly annoyed but guess I'll just start over..
OK I was stupid and didn't set up my own credentials in the OAuth settings in top right of the dashboard as it said to here. I forgot that bit -_- now I can access my own stream again and it shows my integer project id in the stream id not the hex one
https://developers.google.com/fit/rest/v1/get-started
Now I get invalid argument, but.. whatever >_<
edit 2:
invalid argument was because I have fPVal instead of fpVal and modifiedTimeMillis mills is not supposed to be submitted, obviously