IBM Cloud Code Engine: How to overwrite environment variable in cron-invoked job run? - ibm-cloud

When working with jobs in IBM Cloud Code Engine, I would submit a jobrun for the actual invocation. For the jobrun I can specify environment variables to be passed into the runtime environment (--env FOO=BAR).
How can I do the same when using a cron subscription to trigger the job, i.e., to set FOO=BAR?

I believe the correct flag to pass to the CLI is the --ext or --extension.
% ibmcloud ce subscription cron create --
NAME:
create - Create a cron event subscription.
USAGE:
create --name CRON_SOURCE_NAME --destination DESTINATION_REF [options...]
OPTIONS:
--destination, -d value Required. The name of the resource that will receive events.
--name, -n value Required. The name of the cron event subscription. Use a name that is unique within the project.
--content-type, --ct value The media type of the 'data' or 'data-base64' option. Examples include 'application/json',
'application/x-www-form-urlencoded', 'text/html', and 'text/plain'.
--data, --da value The data to send to the destination.
--data-base64, --db value The base64-encoded data to send to the destination.
--destination-type, --dt value The type of the 'destination'. Valid values are 'app' and 'job'. (default: "app")
--extension, --ext value Set CloudEvents extensions to send to the destination. Must be in NAME=VALUE format. This option can be
specified multiple times. (accepts multiple inputs)
For example:
% ibmcloud ce subscription cron create -d application-5c -n sample-cron-sub --ext FOO=BAR
Creating cron event subscription 'sample-cron-sub'...
Run 'ibmcloud ce subscription cron get -n sample-cron-sub' to check the cron event subscription status.
% ibmcloud ce subscription cron get -n sample-cron-sub
Getting cron event subscription 'sample-cron-sub'...
OK
Name: sample-cron-sub
ID: xxxx
Project Name: susan-project
Project ID: xxxx
Age: 3m16s
Created: 2022-06-06T11:17:17-07:00
Destination Type: app
Destination: application-5c
Schedule: * * * * *
Time Zone: UTC
Ready: true
CloudEvents Extensions:
Name Value
FOO BAR
Events:
Type Reason Age Source Messages
Normal PingSourceSkipped 3m17s pingsource-controller PingSource is not ready
Normal PingSourceDeploymentUpdated 3m17s (x2 over 3m17s) pingsource-controller PingSource adapter deployment updated
Normal PingSourceSynchronized 3m17s pingsource-controller PingSource adapter is synchronized
After looking at this a bit more it appears that the name=value pairs you pass in to an event subscription prepend the string 'CE_' to the name.
Therefore, to allow for this in the running job, you would need to prepend the environment variable in the job with the CE_. For example:
When I create the jobdefinition I add the environment variable like this:
CE_FOO=BAR
Then, when I create the event subscription, for the --ext flag, I use the original suggestion: --ext FOO=BAR
I believe since the FOO variable in the event subscription automatically gets the prepended CE_ to the FOO variable it should work.
Please let me know if this does not work or I misunderstood you.

Related

How to wait for tekton pipelinRun conditions

I have the following code within a gitlab pipeline which results in some kind of race condition:
kubectl apply -f pipelineRun.yaml
tkn pipelinerun logs -f pipeline-run
The tkn command immediately exits, since the pipelineRun object is not yet created. There is one very nice solution for this problem:
kubectl apply -f pipelineRun.yaml
kubectl wait --for=condition=Running --timeout=60s pipelinerun/pipeline-run
tkn pipelinerun logs -f pipeline-run
Unfortunately this is not working as expected, since Running seems to be no valid condition for a pipelineRun object. So my question is: what are the valid conditions of a pipelineRun object?
I didn't search too far and wide, but it looks like they only have two condition types imported from the knative.dev project?
https://github.com/tektoncd/pipeline/blob/main/vendor/knative.dev/pkg/apis/condition_types.go#L32
The link above is for the imported condition types from the pipeline source code of which it looks like Tekton only uses "Ready" and "Succeeded".
const (
// ConditionReady specifies that the resource is ready.
// For long-running resources.
ConditionReady ConditionType = "Ready"
// ConditionSucceeded specifies that the resource has finished.
// For resource which run to completion.
ConditionSucceeded ConditionType = "Succeeded"
)
But there may be other imports of this nature elsewhere in the project.
Tekton TaskRuns and PipelineRun only use a condition of type Succeeded.
Example:
conditions:
- lastTransitionTime: "2020-05-04T02:19:14Z"
message: "Tasks Completed: 4, Skipped: 0"
reason: Succeeded
status: "True"
type: Succeeded
The different status and messages available for the Succeeded condition are available in the documentation:
TaskRun: https://tekton.dev/docs/pipelines/taskruns/#monitoring-execution-status
PipelineRun: https://tekton.dev/docs/pipelines/pipelineruns/#monitoring-execution-status
As a side note, there is an activity timeout available in the API. That timeout is not surfaced to the CLI options though. You could create a tkn feature request for that.

How to wait until a job is done or a file is updated in airflow

I am trying to use apache-airflow, with google cloud-composer, to shedule batch processing that result in the training of a model with google ai platform. I failed to use airflow operators as I explain in this question unable to specify master_type in MLEngineTrainingOperator
Using the command line I managed to launch a job successfully.
So now my issue is to integrate this command in airflow.
Using BashOperator I can train the model but I need to wait for the job to be completed before creating a version and setting it as the default. This DAG create a version before the job is done
bash_command_train = "gcloud ai-platform jobs submit training training_job_name " \
"--packages=gs://path/to/the/package.tar.gz " \
"--python-version=3.5 --region=europe-west1 --runtime-version=1.14" \
" --module-name=trainer.train --scale-tier=CUSTOM --master-machine-type=n1-highmem-16"
bash_train_operator = BashOperator(task_id='train_with_bash_command',
bash_command=bash_command_train,
dag=dag,)
...
create_version_op = MLEngineVersionOperator(
task_id='create_version',
project_id=PROJECT,
model_name=MODEL_NAME,
version={
'name': version_name,
'deploymentUri': export_uri,
'runtimeVersion': RUNTIME_VERSION,
'pythonVersion': '3.5',
'framework': 'SCIKIT_LEARN',
},
operation='create')
set_version_default_op = MLEngineVersionOperator(
task_id='set_version_as_default',
project_id=PROJECT,
model_name=MODEL_NAME,
version={'name': version_name},
operation='set_default')
# Ordering the tasks
bash_train_operator >> create_version_op >> set_version_default_op
The training result in updating of a file in Gcloud storage. So I am looking for an operator or a sensor that will wait until this file is updated, I noticed GoogleCloudStorageObjectUpdatedSensor, but I dont know how to make it retry until this file is updated.
An other solution would be to check for the job to be completed, but I can't find how too.
Any help would be greatly appreciated.
The Google Cloud documentation for the --stream-logs flag:
"Block until job completion and stream the logs while the job runs."
Add this flag to bash_command_train and I think it should solve your problem. The command should only release once the job is finished, then Airflow will mark it as success. It will also let you monitor your training job's logs in Airflow.

Visualize Jobber tasks on ELK (via Filebeat)

A Jobber Docker container (running periodic tasks) outputs on stdout, which is captured by Filebeat (with Docker containers autodiscovery flag on) and then sent to Logstash (within an ELK stack) or to Elasticsearch directly.
Now on Kibana, the document looks as such:
#timestamp Jan 20, 2020 # 20:15:07.752
...
agent.type filebeat
container.image.name jobber_jobber
...
message {
"job": {
"command":"curl http://my.service/run","name":"myperiodictask",
"status":"Good",
"time":"0 */5 * * * *"
},
"startTime":1579540500,
"stdout":"{\"startDate\":\"2020-01-20T16:35:00.000Z\",\"endDate\":\"2020-01-20T17:00:00.000Z\",\"zipped\":true,\"size\":3397}",
"succeeded":true,
"user":"jobberuser",
"version":"1.4"
}
...
Note: above 'message' field is a simple string reflecting a json object; above displayed format is for clearer readability.
My goal is to be able to request Elastic on the message fields, so I can filter by Jobber tasks for instance.
How can I make that happen ?
I know Filebeat uses plugins and the container tags to apply this or that filter: are there any for Jobber? If not, how to do this?
Even better would be to be able to exploit the fields of the Jobber task result (under the 'stdout' field)! Could you please direct me to ways to implement that?
Filebeat provides processors to handle such tasks.
Below's a configuration to handle the needs "Decode the json in the 'message' field", "Decode the json in the 'stdout' within" (both using the decode_json_fields processor), and other Jobber-related needs.
Note that given example filter the events going through Filebeat by a 'custom-tag' label given to the Docker container hosting the Jobber process. The docker.container.labels.custom-tag: jobber condition should be replaced according to your usecase.
filebeat.yml:
processors:
# === Jobber events processing ===
- if:
equals:
docker.container.labels.custom-tag: jobber
then:
# Drop Jobber events which are not job results
- drop_event:
when:
not:
regexp:
message: "{.*"
# Json-decode event's message part
- decode_json_fields:
when:
regexp:
message: "{.*"
fields: ["message"]
target: "jobbertask"
# Json-decode message's stdout part
- decode_json_fields:
when:
has_fields: ["jobbertask.stdout"]
fields: ["jobbertask.stdout"]
target: "jobbertask.result"
# Drop event's decoded fields
- drop_fields:
fields: ["message"]
- drop_fields:
when:
has_fields: ["jobbertask.stdout"]
fields: ["jobbertask.stdout"]
The decoded fields are placed in the "jobbertask" field. This is to avoid index-mapping collision on the root fields. Feel free to replace "jobbertask" by any other field name, keeping care of mapping collisions.
In my case, this works whether Filebeat addresses the events to Logstash or to Elasticsearch directly.

How to stop kubernetes from reporting to usage.projectcalico.org?

I found that my kubernetes cluster was sending reports to usage.projectcalico.org, how can this be disabled and how exactly is it using usage.projectcalico.org?
Felix is the Calico component that sends usage information.
Felix can be configured to disable the usage ping.
Set the FELIX_USAGEREPORTINGENABLED environment variable can be to "false" (needs to be a string in yaml land!) in the calico-node DaemonSet
Set the UsageReportingEnabled field in the FelixConfiguration resource to false. This could be in etcd or in the Kubernetes API depending on what store you use. Both modifiable with calicoctl.
calicoctl patch felixConfiguration default \
--patch='{"spec": {"UsageReportingEnabled": false}}'
If you happen to be using kubespray, modifying this setting is a little harder as these variables are not exposed to Ansible, other than by manually modifying templates or yaml.
According to the source code:
# Disable Usage Reporting to usage.projectcalico.org
# We want to avoid polluting analytics data with unit test noise
curl_etcd("calico/v1/config/UsageReportingEnabled",
options=["-XPUT -d value=False"], ip=ip)
And here is the definition of curl_etcd
def curl_etcd(path, options=None, recursive=True, ip=None):
"""
Perform a curl to etcd, returning JSON decoded response.
:param path: The key path to query
:param options: Additional options to include in the curl
:param recursive: Whether we want recursive query or not
:return: The JSON decoded response.
"""
if options is None:
options = []
if ETCD_SCHEME == "https":
# Etcd is running with SSL/TLS, require key/certificates
rc = check_output(
"curl --cacert %s --cert %s --key %s "
"-sL https://%s:2379/v2/keys/%s?recursive=%s %s"
% (ETCD_CA, ETCD_CERT, ETCD_KEY, ETCD_HOSTNAME_SSL,
path, str(recursive).lower(), " ".join(options)),
shell=True)
else:
rc = check_output(
"curl -sL http://%s:2379/v2/keys/%s?recursive=%s %s"
% (ip, path, str(recursive).lower(), " ".join(options)),
shell=True)
return json.loads(rc.strip())

How to create a global variable for session and use it across test cases ( RobotFramework,REST API )

Can someone let me know how to create a global session variable and that session variable should be used across keywords/testcases . I have tried the following way and getting error as ${itoc_session} not found.
*** Settings ***
Library RequestsLibrary
Library Collections
*** Keywords ***
Create session for server
#{auth}= Create List admin admin123
Create Session httpbin https://host141.swlab.net:71/ auth=#{auth}
Set Global Variable ${itoc_session} httpbin
Get Policy for Server
${resp}= Get Request ${itoc_session} uri=/policies/
Log ${resp.status_code}
:FOR ${item} in #{resp.json()}
\ Log ${item}
\ ${get_policy_id}= Get Variable Value ${item['id']}
\ ${get_policy_name}= Get Variable Value ${item['name']}
\ Log ${get_policy_id},${get_policy_name}
Set Global Variable ${policy_id} ${get_policy_id}
Get Policy with policyID
${res}= Get Request ${itoc_session} uri = /policies/${policy_id}
Log ${res}
Log ${res.json()['name']} , ${res.json()['extReference']}
*** Test case ***
get_policy_id
Get Policy for Server
get_policy_details
Get Policy with policyID
You never call Create session for server, so the variable never gets set.