How to access execution_date in SimpleHttpOperator in Airflow - rest

I am trying automate a call I make using a REST API with the SimpleHttpOperator.
Here is an example of the call I can make and is working great.
curl -u username:password
-H "Content-Type: application/json"
-X POST
-d '{"job_id":10,"date":{"year":"2016","month":"10"}}'
https://MY_INSTANCE_NAME.cloud.databricks.com/api/2.0/jobs/run-now
Now, formatting this to be generated by the SimpleHttpOperator looks like this and also works great:
t2 = SimpleHttpOperator(
task_id=TASK_ID,
http_conn_id=CONN_ID,
method='POST',
endpoint='api/2.0/jobs/run-now',
headers={"Content-Type": "application/json"},
data=json.dumps({"job_id":10, "date": { "year": "2016" , "month": "10" }}),
dag=dag
)
I now am trying to access execution_date, a variable set by Airflow which lets an Operator know the time it was suppose to be called. This normally is accessible using a jinga template for BashOperator or PythonOperator but not SimpleHttpOperator:
e.g. """echo {{execution_date}}"""
This variable is not available to the SimpleHttpOperator and it is very much needed. If you know any work arounds or a way to access it within the SimpleHttpOperator, please let me know.
Thanks.

The date field in SimpleHttpOperator accepts jinja template:
t2 = SimpleHttpOperator(
task_id=TASK_ID,
http_conn_id=CONN_ID,
method='POST',
endpoint='api/2.0/jobs/run-now',
headers={"Content-Type": "application/json"},
data="""{"job_id":"10", "date": { "year": {{code}}, "month": {{code}}{{'}}'}}""",
dag=dag
)

Related

Receiving 404 when trying to edit a datasource despite being able to retrieve it's details

I have a datasource created in Grafana and attempting to update it to refresh the bearer token for auth access.
However, I'm receiving a 404 Not Found error from the grafana api when making a request to localhost:3000/api/datasources/uid/:uid with a uid just received from the datasources/name api - attempting to update as per the documentation https://grafana.com/docs/grafana/latest/developers/http_api/data_source/#update-an-existing-data-source
I'm using the grafana opensource docker container with the Infinity plugin.
docker run -d -p 3000:3000 --name=grafana -e "GF_INSTALL_PLUGINS=yesoreyeram-infinity-datasource" grafana/grafana-oss
I'm able to create a datasource via the api, just can't update an existing one.
My code is:-
grafana_api_token = '<my api token>'
new_access_token = '<my new bearer token>'
my_data_source = 'my_data_source'
grafana_header = {"authorization": f"Bearer {grafana_api_token}", "content-type":"application/json;charset=UTF-8"}
grafana_datasource_url = f"http://localhost:3000/api/datasources/name/{my_data_source}"
firebolt_datasource_resp = get(url=grafana_datasource_url, headers=grafana_header)
full_datasource = loads(firebolt_datasource_resp.content.decode("utf-8"))
datasource_uid = full_datasource["uid"]
update_token_url = f"http://localhost:3000/api/datasources/uid/{datasource_uid}"
new_data = {"id": full_datasource["id"],
"uid": full_datasource["uid"],
"orgId": full_datasource["orgId"],
"name": "new_data_source",
"type": full_datasource["type"],
"access": full_datasource["access"],
"url": full_datasource["url"],
"user": full_datasource["user"],
"database": full_datasource["database"],
"basicAuth": full_datasource["basicAuth"],
"basicAuthUser": full_datasource["basicAuthUser"],
"withCredentials": full_datasource["withCredentials"],
"isDefault": full_datasource["isDefault"],
"jsonData": full_datasource["jsonData"],
"secureJsonData": {
"bearerToken": new_access_token
}
}
update_bearer_token_resp = post(url=update_token_url, data=dumps(new_data), headers=grafana_header)
Oh, oh, oh, idiot mode. Using post rather than put. Doh.

How do you run the server to use/test the JSON format?

The Reference Manuak at https://github.com/jruizgit/rules/blob/master/docs/json/reference.md says that I can assert facts like this:
curl -H "content-type: application/json" -X POST -d '{"subject": "Tweety", "predicate": "eats", "object": "worms"}' http://localhost:5000/animal/facts
But how I run the server at port 5000? Does durable_rules comes with a built in HTTP Server?
thanks for asking the question. The JSON doc is outdated. durable_rules V2, no longer starts an http server (which gives you the flexibility to choose how you want to host your rulesets). You can provide a JSON document as follows:
from durable.lang import *
def callback(c):
print('risk7 fraud detected')
get_host().set_rulesets({ 'risk7': {
'suspect': {
'run': callback,
'all': [
{'first': {'t': 'purchase'}},
{'second': {'$neq': {'location': {'first': 'location'}}}}
],
}
}})
post('risk7', {'t': 'purchase', 'location': 'US'});
post('risk7', {'t': 'purchase', 'location': 'CA'});

How to add the column to Google Sheets using API and provide the name and type of the column in the same call?

So, what I could achieve using the Google Sheets API is being able to create a new column using the following curl based call
curl -v \
-H 'Authorization: Bearer ya29.GlxUB9K_96tyQFyQ64eaYOeImtJt32213zjosf6LW1Inv6MOqQCCodA7CycvL5EFKIpeX4dVEebS4rUl24U1J7euhMjqBZq0QEU7ZK1B64THQXNwBpDvTzoUT9hTRg' \
-H 'Content-Type: application/json' \
-d '{
"requests": [
{
"insertDimension": {
"range": {
"sheetId": 2052094881,
"dimension": "COLUMNS",
"startIndex": 0,
"endIndex": 1
}
}
}
],
}' \
https://sheets.googleapis.com/v4/spreadsheets/1mHrPXQILuprO4NdqTgrVKlGazvvzgCFqIphGdsmptD8:batchUpdate
While this call is useful, it does not completely help. This is because the reason I wanted to add a column was to give a name and type (or format) the column values. But, as per this API, this is what see as an output
Is there a way to create and add name and type to the column in a single API call?
Thanks a lot!

What is the format of the JSON for a Jenkins REST buildWithParameters to override the default parameters values

I am able to build a Jenkins job with its parameters' default values by sending a POST call to
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters
and I can override the default parameters "product", "suites" and "markers by sending to this URL:
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters?product=ALL&suites=ALL&markers=ALL
But I saw examples were the parameters can be override by sending a JSON body with new values. I am trying to do that by sending the following json bodies. Neither of them works for me.
{
'product': 'ALL',
'suites': 'ALL',
'markers': 'ALL'
}
and
{
"parameter": [
{
"name": "product",
"value": "ALL"
},
{
"name": "suites",
"value": "ALL"
},
{
"name": "markers",
"value": "ALL"
}
]
}
What JSON to send if I want to override the values of parameters "product", "suites" & "markers"?
I'll leave the original question as is and elaborate here on the various API calls to trigger parameterized builds. These are the calls options that I used.
Additional documentation: https://wiki.jenkins.io/display/JENKINS/Remote+access+API
The job contains 3 parameters named: product, suites, markers
Send the parameters as URL query parameters to /buildWithParameters:
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters?product=ALL&suites=ALL&markers=ALL
Send the parameters as JSON data\payload to /build:
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/build
The JSON data\payload is not sent as the call's json_body (which is what confused me), but rater in the data payload as:
json:'{
"parameter": [
{"name":"product", "value":"123"},
{"name":"suites", "value":"high"},
{"name":"markers", "value":"Hello"}
]
}'
And here are the CURL commands for each of the above calls:
curl -X POST -H "Jenkins-Crumb:2e11fc9...0ed4883a14a" http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/build --user "raameeil:228366f31...f655eb82058ad12d" --form json='{"parameter": [{"name":"product", "value":"123"}, {"name":"suites", "value":"high"}, {"name":"markers", "value":"Hello"}]}'
curl -X POST \
'http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters?product=234&suites=333&markers=555' \
-H 'authorization: Basic c2hsb21pb...ODRlNjU1ZWI4MjAyOGFkMTJk' \
-H 'cache-control: no-cache' \
-H 'jenkins-crumb: 0bed4c7...9031c735a' \
-H 'postman-token: 0fb2ef51-...-...-...-6430e9263c3b'
What to send to Python's requests
In order to send the above calls in Python you will need to pass:
headers = jenkins-crumb
auth = tuple of your (user_name, user_auth_token)
data = dictionary type { 'json' : json string of {"parameter":[....]} }
curl -v POST http://user:token#host:port/job/my-job/build --data-urlencode json='{"parameter": [{"name":"xx", "value":"xxx"}]}
or use Python request:
import requests
import json
url = " http://user:token#host:port/job/my-job/build "
pyload = {"parameter": [
{"name":"xx", "value":"xxx"},
]}
data = {'json': json.dumps(pyload)}
rep = requests.post(url, data)

How to enable Javascript in Druid

I have been using Druid for the past week and wanted to enable javascript for some postAggregations.
I think I followed the outlined steps and updated the common.runtime.properties file in ../con f/druid/_common/ to include druid.javascript.enabled=true. I then stopped the current processes and re-ran the Quickstart procedures, but it still says that JavaScript is disabled:
{
"error" : "Unknown exception",
"errorMessage" : "Instantiation of [simple type, class io.druid.query.aggregation.post.JavaScriptPostAggregator] value failed: JavaScript is disabled. (through reference chain: java.util.ArrayList[0])",
"errorClass" : "com.fasterxml.jackson.databind.JsonMappingException",
"host" : null
}
I am currently running it in the 'Quickstart' configuration - single local machine. Any pointers? Thanks!
JavaScript Query For druid Aggregation. Save the file as .body and hit the curl request.
This is a sample query for Average value.
curl -X POST "http://localhost:8082/druid/v2/?pretty" \ -H
'content-type: application/json' -d #query.body
{
"queryType":"groupBy",
"dataSource":"whirldata",
"granularity":"all",
"dimensions":[],
"aggregations":[{"name":"rows","type":"count","fieldName":"rows"},
{"name":"TargetDOS","type":"doubleSum","fieldName":"Target DOS"}],"postAggregations":[
{
"type": "javascript",
"name": "Target DOS Average",
"fieldNames": ["TargetDOS", "rows"],
"function": "function(TargetDOS, rows) { return Math.abs(TargetDOS) / rows; }"
}], "intervals":[ "2006-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z" ]}
The part you are missing is likely that the quickstart reads configs from conf-quickstart rather than conf. So try editing conf-quickstart/druid/_common/common.runtime.properties.