Telegraf & InfluxDB: how to convert PROCSTAT's pid from field to tag? - grafana

Summary: I am using telegraf to get procstat into InfluxDB. I want to convert the pid from an integer field to a TAG so that I can do group by on it in Influx.
Details:
After a lot of searching I found the following on some site but it seems to be doing the opposite (converts tag into a field). I am not sure how to deduce the opposite conversion syntax from it:
[processors]
[[processors.converter]]
namepass = [ "procstat",]
[processors.converter.tags]
string = [ "cmdline",]
I'm using Influx 1.7.9

The correct processor configuration to convert pid as tag is as below.
[processors]
[[processors.converter]]
namepass = [ "procstat"]
[processors.converter.fields]
tag = [ "pid"]
Please refer the documentation of converter processor plugin
https://github.com/influxdata/telegraf/tree/master/plugins/processors/converter
In the latest version of telegraf pid can be stored as tag by specifying it in the input plugin configuration. A converter processor is not needed here.
Mention pid_tag = true in the configuration. However be aware of the performance impact of having pid as a tag when processes are short lived.
P.S: You should try to upgrade your telegraf version to 1.14.5. There is a performance improvement fix for procstat plugin in this version.
Plugin configuration reference https://github.com/influxdata/telegraf/tree/master/plugins/inputs/procstat
Sample config.
# Monitor process cpu and memory usage
[[inputs.procstat]]
## PID file to monitor process
pid_file = "/var/run/nginx.pid"
## executable name (ie, pgrep <exe>)
# exe = "nginx"
## pattern as argument for pgrep (ie, pgrep -f <pattern>)
# pattern = "nginx"
## user as argument for pgrep (ie, pgrep -u <user>)
# user = "nginx"
## Systemd unit name
# systemd_unit = "nginx.service"
## CGroup name or path
# cgroup = "systemd/system.slice/nginx.service"
## Windows service name
# win_service = ""
## override for process_name
## This is optional; default is sourced from /proc/<pid>/status
# process_name = "bar"
## Field name prefix
# prefix = ""
## When true add the full cmdline as a tag.
# cmdline_tag = false
## Add the PID as a tag instead of as a field. When collecting multiple
## processes with otherwise matching tags this setting should be enabled to
## ensure each process has a unique identity.
##
## Enabling this option may result in a large number of series, especially
## when processes have a short lifetime.
# pid_tag = false

Related

containerd multiline logs parsing with fluentbit

After shifting from Docker to containerd as docker engine used by our kubernetes, we are not able to show the multiline logs in a proper way by our visualization app (Grafana) as some details prepended to the container/pod logs by the containerd itself (i.e. timestamp, stream & log severity to be specific it is appending something like the following and as shown in the below sample: 2022-07-25T06:43:17.20958947Z stdout F  ) which make some confusion for the developers and the application owners.
I am showing here a dummy sample of the logs generated by the application and how it got printed in the nodes of kuberenetes'nodes after containerd prepended the mentioned details.
The following logs generated by the application (kubectl logs ):
2022-07-25T06:43:17,309ESC[0;39m dummy-[txtThreadPool-2] ESC[39mDEBUGESC[0;39m
  ESC[36mcom.pkg.sample.ComponentESC[0;39m - Process message meta {
  timestamp: 1658731397308720468
  version {
      major: 1
      minor: 0
      patch: 0
  }
}
when I check the logs in the filesystem (/var/log/container/ABCXYZ.log) :
2022-07-25T06:43:17.20958947Z stdout F 2022-07-25T06:43:17,309ESC[0;39m dummy-[txtThreadPool-2]
ESC[39mDEBUGESC[0;39m
ESC[36mcom.pkg.sample.ComponentESC[0;39m - Process message meta {
2022-07-25T06:43:17.20958947Z stdout F timestamp: 1658731449723010774
2022-07-25T06:43:17.209593379Z stdout F version {
2022-07-25T06:43:17.209595933Z stdout F major: 14
2022-07-25T06:43:17.209598466Z stdout F minor: 0
2022-07-25T06:43:17.209600712Z stdout F patch: 0
2022-07-25T06:43:17.209602926Z stdout F }
2022-07-25T06:43:17.209605099Z stdout F }
I am able to parse the multiline logs with fluentbit but the problem is I am not able to remove the details injected by containerd ( >> 2022-07-25T06:43:17.209605099Z stdout F .......). So is there anyway to configure containerd to not prepend these details somehow in the logs and print them as they are generated from the application/container ?
On the other hand is there any plugin to remove such details from fluentbit side .. as per the existing plugins none of them can manipulate or change the logs (which is logical  as the log agent should not do any change on the logs).
Thanks in advance.
This is the workaround I followed to show the multiline log lines in Grafana by applying extra fluentbit filters and multiline parser.
1- First I receive the stream by tail input which parse it by a multiline parser (multilineKubeParser).
2- Then another filter will intercept the stream to do further processing by a regex parser (kubeParser).
3- After that another filter will remove the details added by the containerd by a lua parser ().
fluent-bit.conf: |-
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Flush 1
Daemon Off
Log_Level warn
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
multiline.Parser multilineKubeParser
Exclude_Path /var/log/containers/*_ABC-logging_*.log
DB /run/fluent-bit/flb_kube.db
Mem_Buf_Limit 5MB
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Merge_Parser kubeParser
K8S-Logging.Parser Off
K8S-Logging.Exclude On
[FILTER]
Name lua
Match kube.*
call remove_dummy
Script filters.lua
[Output]
Name grafana-loki
Match kube.*
Url http://loki:3100/api/prom/push
TenantID ""
BatchWait 1
BatchSize 1048576
Labels {job="fluent-bit"}
RemoveKeys kubernetes
AutoKubernetesLabels false
LabelMapPath /fluent-bit/etc/labelmap.json
LineFormat json
LogLevel warn
labelmap.json: |-
{
"kubernetes": {
"container_name": "container",
"host": "node",
"labels": {
"app": "app",
"release": "release"
},
"namespace_name": "namespace",
"pod_name": "instance"
},
"stream": "stream"
}
parsers.conf: |-
[PARSER]
Name kubeParser
Format regex
Regex /^([^ ]*).* (?<timeStamp>[^a].*) ([^ ].*)\[(?<requestId>[^\]]*)\] (?<severity>[^ ]*) (?<message>[^ ].*)$/
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
Time_Offset +0200
[MULTILINE_PARSER]
name multilineKubeParser
type regex
flush_timeout 1000
rule "start_state" "/[^ ]* stdout .\s+\W*\w+\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d\d\d.*$/" "cont"
rule "cont" "/[^ ]* stdout .\s+(?!\W+\w+\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d\d\d).*$/" "cont"
filters.lua: |-
function remove_dummy(tag, timestamp, record)
new_log=string.gsub(record["log"],"%d+-%d+-%d+T%d+:%d+:%d+.%d+Z%sstdout%sF%s","")
new_record=record
new_record["log"]=new_log
return 2, timestamp, new_record
end
As I mentioned this is a workaround till I can find any other/better solution.
From the configuration options for containerd, it appears there's no way to configure logging in any way. You can see the config doc here.
Also, I checked at the logging code inside containerd and it appears this is prepended to the logs as they are redirected from the stdout of the container. You can see that the testcase here checks for appropriate fields by "splitting" the log line received. It checks for a tag and a stream entry prepended to the content of the log. I suppose that's the way logs are processed in containerd.
The best thing to do would be to open an issue in the project with your design requirement and perhaps the team can develop configurable stdout redirection for you.
This might help. They use a custom regex to capture the message log and the rest of the info in the logs and then use lift in the nest filter to flatten the json.
https://github.com/microsoft/fluentbit-containerd-cri-o-json-log

Airflow DAGS are running but the tasks are not running/queuing - Error sending Celery task:Timeout

We have Airflow 1.10.3 working with Celery 4.1.1 and Redis as the message broker.
When we bring up the webserver the scheduled DAGs go into running state indefinitely and We cannot see any active tasks in the Flower UI.
In the logs(airflow-start-up-logs) we get the following error :(Error sending Celery task:Timeout)
{"timestamp":"2020-11-11T09:45:58.326682", "hostname":"", "process":"scheduler", "name":"airflow.executors.celery_executor.CeleryExecutor", "level":"ERROR", "message":"Error sending Celery task:Timeout, PID: 16001\nCelery Task ID: ('tutorial', 'print_date', datetime.datetime(2020, 11, 9, 0, 0, tzinfo=<Timezone [UTC]>), 1)\nTraceback (most recent call last):\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/utils\/functional.py\", line 42, in __call__\n return self.__value__\nAttributeError: 'ChannelPromise' object has no attribute '__value__'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/transport\/virtual\/base.py\", line 921, in create_channel\n return self._avail_channels.pop()\nIndexError: pop from empty list\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/airflow\/executors\/celery_executor.py\", line 118, in send_task_to_executor\n result = task.apply_async(args=[command], queue=queue)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/celery\/app\/task.py\", line 535, in apply_async\n **options\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/celery\/app\/base.py\", line 745, in send_task\n amqp.send_task_message(P, name, message, **options)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/celery\/app\/amqp.py\", line 552, in send_task_message\n **properties\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/messaging.py\", line 181, in publish\n exchange_name, declare,\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/connection.py\", line 518, in _ensured\n return fun(*args, **kwargs)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/messaging.py\", line 187, in _publish\n channel = self.channel\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/messaging.py\", line 209, in _get_channel\n channel = self._channel = channel()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/utils\/functional.py\", line 44, in __call__\n value = self.__value__ = self.__contract__()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/messaging.py\", line 224, in <lambda>\n channel = ChannelPromise(lambda: connection.default_channel)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/connection.py\", line 866, in default_channel\n self.ensure_connection(**conn_opts)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/connection.py\", line 430, in ensure_connection\n callback, timeout=timeout)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/utils\/functional.py\", line 343, in retry_over_time\n return fun(*args, **kwargs)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/connection.py\", line 283, in connect\n return self.connection\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/connection.py\", line 837, in connection\n self._connection = self._establish_connection()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/connection.py\", line 792, in _establish_connection\n conn = self.transport.establish_connection()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/transport\/virtual\/base.py\", line 941, in establish_connection\n self._avail_channels.append(self.create_channel(self))\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/transport\/virtual\/base.py\", line 923, in create_channel\n channel = self.Channel(connection)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/kombu\/transport\/redis.py\", line 521, in __init__\n self.client.ping()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/redis\/client.py\", line 1351, in ping\n return self.execute_command('PING')\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/redis\/client.py\", line 875, in execute_command\n conn = self.connection or pool.get_connection(command_name, **options)\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/redis\/connection.py\", line 1185, in get_connection\n connection.connect()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/redis\/connection.py\", line 552, in connect\n sock = self._connect()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/redis\/connection.py\", line 845, in _connect\n sock = super(SSLConnection, self)._connect()\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/redis\/connection.py\", line 579, in _connect\n socket.SOCK_STREAM):\n File \"\/usr\/lib64\/python3.7\/socket.py\", line 748, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n File \"\/usr\/local\/lib\/python3.7\/site-packages\/airflow\/utils\/timeout.py\", line 43, in handle_timeout\n raise AirflowTaskTimeout(self.error_message)\nairflow.exceptions.AirflowTaskTimeout: Timeout, PID: 16001\n\n"}
Config File
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /home/ec2-user/airflow
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /home/ec2-user/airflow/dags
# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = /var/log/airflow
# Logging level
logging_level = DEBUG
fab_logging_level = WARN
# Logging class
# Specify the class that will specify the logging configuration
# This class has to be on the python classpath
# logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
logging_config_class = log_config.CUSTOM_LOGGING_CONFIG
# Log format
# we need to escape the curly braces by adding an additional curly brace
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
# Log filename format
# we need to escape the curly braces by adding an additional curly brace
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
# Hostname by providing a path to a callable, which will resolve the hostname
hostname_callable = socket:getfqdn
# Default timezone in case supplied date times are naive
# can be utc (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)
default_timezone = utc
# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
# must supply a remote location URL (starting with either 's3://...' or
# 'gs://...') and an Airflow connection id that provides access to the storage
# location.
remote_base_log_folder =
remote_log_conn_id =
# Use server-side encryption for logs stored in S3
encrypt_s3_logs = False
# DEPRECATED option for remote log storage, use remote_base_log_folder instead!
s3_log_folder =
# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = CeleryExecutor
broker_url = redis://***************************************:6379/0
# The SqlAlchemy connection string to the metadata database.
# SqlAlchemy supports many different database engine, more information
# their website
sql_alchemy_conn = mysql://***************************************:3306/airflow
# The SqlAlchemy pool size is the maximum number of database connections
# in the pool.
sql_alchemy_pool_size = 5
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
# not apply to sqlite.
sql_alchemy_pool_recycle = 2000
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
# The number of task instances allowed to run concurrently by the scheduler
dag_concurrency = 6
# Are DAGs paused by default at creation
dags_are_paused_at_creation = False
# When not using pools, tasks are run in the "default pool",
# whose size is guided by this config element
non_pooled_task_slot_count = 128
# The maximum number of active DAG runs per DAG
max_active_runs_per_dag = 16
# Whether to load the examples that ship with Airflow. It's good to
# get started, but you probably want to set this to False in a production
# environment
load_examples = False
# Where your Airflow plugins are stored
plugins_folder = /home/ec2-user/airflow/plugins
# Secret key to save connection passwords in the db
fernet_key =
# Whether to disable pickling dags
donot_pickle = False
# How long before timing out a python file import while filling the DagBag
dagbag_import_timeout = 30
# The class to use for running task instances in a subprocess
task_runner = BashTaskRunner
# If set, tasks without a `run_as_user` argument will be run with this user
# Can be used to de-elevate a sudo user running Airflow when executing tasks
default_impersonation =
# What security module to use (for example kerberos):
security =
# Turn unit test mode on (overwrites many configuration options with test
# values at runtime)
unit_test_mode = False
# full path of dag_processor_manager logfile
dag_processor_manager_log_location = /var/log/airflow/dag_processor_manager/dag_processor_manager.log
# Name of handler to read task instance logs.
# Default to use task handler.
task_log_reader = task
# Whether to enable pickling for xcom (note that this is insecure and allows for
# RCE exploits). This will be deprecated in Airflow 2.0 (be forced to False).
enable_xcom_pickling = True
# When a task is killed forcefully, this is the amount of time in seconds that
# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
killed_task_cleanup_time = 60
# Whether to override params with dag_run.conf. If you pass some key-value pairs through `airflow backfill -c` or
# `airflow trigger_dag -c`, the key-value pairs will override the existing ones in params.
dag_run_conf_overrides_params = False
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
##endpoint_url = http://localhost:8080
endpoint_url = 10.136.119.91
[api]
# How to authenticate users of the API
#auth_backend = airflow.api.auth.backend.default
[lineage]
# what lineage backend to use
#backend =
[atlas]
sasl_enabled = False
host =
port = 21000
username =
password =
[operators]
# The default owner assigned to each new operator, unless
# provided explicitly or passed via `default_args`
default_owner = Airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
[webserver]
# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
#base_url =
# The ip specified when starting the web server
web_server_host = 0.0.0.0
# The port on which to run the web server
web_server_port = 8080
# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_cert = /etc/ssl/certs/airflow-selfsigned.crt
web_server_ssl_key = /etc/ssl/private/airflow-selfsigned.key
# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
web_server_master_timeout = 1200
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 1200
# Number of workers to refresh at a time. When set to 0, worker refresh is
# disabled. When nonzero, airflow periodically refreshes webserver workers by
# bringing up new ones and killing old ones.
worker_refresh_batch_size = 1
# Number of seconds to wait before refreshing a batch of workers.
worker_refresh_interval = 30
# Secret key used to run your flask app
secret_key = temporary_key
# Number of workers to run the Gunicorn web server
workers = 4
# The worker class gunicorn should use. Choices include
# sync (default), eventlet, gevent
worker_class = sync
# Log files for the gunicorn webserver. '-' means log to stderr.
access_logfile = /var/log/airflow/gunicorn-access.log
error_logfile = /var/log/airflow/gunicorn-error.log
# Expose the configuration file in the web server
expose_config = False
# Set to true to turn on authentication:
# http://pythonhosted.org/airflow/security.html#web-authentication
authenticate = True
auth_backend = airflow.contrib.auth.backends.password_auth
# Filter the list of dags by owner name (requires authentication to be enabled)
filter_by_owner = False
# Filtering mode. Choices include user (default) and ldapgroup.
# Ldap group filtering requires using the ldap backend
#
# Note that the ldap server needs the "memberOf" overlay to be set up
# in order to user the ldapgroup mode.
owner_mode = user
# Default DAG orientation. Valid values are:
# LR (Left->Right), TB (Top->Bottom), RL (Right->Left), BT (Bottom->Top)
dag_orientation = LR
# Puts the webserver in demonstration mode; blurs the names of Operators for
# privacy.
demo_mode = False
# The amount of time (in secs) webserver will wait for initial handshake
# while fetching logs from other worker machine
log_fetch_timeout_sec = 5
# By default, the webserver shows paused DAGs. Flip this to hide paused
# DAGs by default
hide_paused_dags_by_default = False
# Consistent page size across all listing views in the UI
page_size = 100
# Use FAB-based webserver with RBAC feature
rbac = True
# Define the color of navigation bar
navbar_color = #007A87
# Default dagrun to show in UI
default_dag_run_display_number = 25
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
smtp_port = 25
smtp_mail_from =
[celery]
# This section only applies if you are using the CeleryExecutor in
# [core] section above
# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor
# The concurrency that will be used when starting workers with the
# "airflow worker" command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
worker_concurrency = 16
# When you start an airflow worker, airflow starts a tiny web server
# subprocess to serve the workers local log files to the airflow main
# web server, who then builds pages and sends them to users. This defines
# the port on which the logs are served. It needs to be unused, and open
# visible from the main web server to connect into the workers.
worker_log_server_port = 8793
# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
# a sqlalchemy database. Refer to the Celery documentation for more
# information.
broker_url = redis://***************************************:6379/0
celery_result_backend = db+mysql://***************************************:3306/airflow
# Another key Celery setting
result_backend = db+mysql://***************************************:3306/airflow
# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
# it `airflow flower`. This defines the IP that Celery Flower runs on
flower_host = 0.0.0.0
# This defines the port that Celery Flower runs on
flower_port = 8443
# Default queue that tasks get assigned to and that worker listen on.
default_queue = default
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
# In case of using SSL
ssl_active = True
ssl_key = /etc/ssl/private/airflow-selfsigned.key
ssl_cert = /etc/ssl/certs/airflow-selfsigned.crt
ssl_cacert =
[celery_broker_transport_options]
# This section is for specifying options which can be passed to the
# underlying celery broker transport. See:
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
# The visibility timeout defines the number of seconds to wait for the worker
# to acknowledge the task before the message is redelivered to another worker.
# Make sure to increase the visibility timeout to match the time of the longest
# ETA you're planning to use.
#
# visibility_timeout is only supported for Redis and SQS celery brokers.
# See:
# http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-broker_transport_options
#
#visibility_timeout = 21600
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
# The scheduler constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5
# after how much time should the scheduler terminate in seconds
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
# How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
min_file_parsing_loop_time = 1
dag_dir_list_interval = 300
# How often should stats be printed to the logs
print_stats_interval = 30
child_process_log_directory = /var/log/airflow/scheduler
# Local task jobs periodically heartbeat to the DB. If the job has
# not heartbeat in this many seconds, the scheduler will mark the
# associated task instance as failed and will re-schedule the task.
scheduler_zombie_task_threshold = 300
# Turn off scheduler catchup by setting this to False.
# Default behavior is unchanged and
# Command Line Backfills still work, but the scheduler
# will not do scheduler catchup if this is False,
# however it can be set on a per DAG basis in the
# DAG definition (catchup)
catchup_by_default = True
# This changes the batch size of queries in the scheduling main loop.
# If this is too high, SQL query performance may be impacted by one
# or more of the following:
# - reversion to full table scan
# - complexity of query predicate
# - excessive locking
#
# Additionally, you may hit the maximum allowable query length for your db.
#
# Set this to 0 for no limit (not advised)
max_tis_per_query = 512
# Statsd (https://github.com/etsy/statsd) integration settings
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
# The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run. However airflow will never
# use more threads than the amount of cpu cores available.
max_threads = 4
authenticate = False
[mesos]
# Mesos master address which MesosExecutor will connect to.
master = localhost:5050
# The framework name which Airflow scheduler will register itself as on mesos
framework_name = Airflow
# Number of cpu cores required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_cpu = 1
# Memory in MB required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_memory = 256
# Enable framework checkpointing for mesos
# See http://mesos.apache.org/documentation/latest/slave-recovery/
checkpoint = False
# Failover timeout in milliseconds.
# When checkpointing is enabled and this option is set, Mesos waits
# until the configured timeout for
# the MesosExecutor framework to re-register after a failover. Mesos
# shuts down running tasks if the
# MesosExecutor framework fails to re-register within this timeframe.
# failover_timeout = 604800
# Enable framework authentication for mesos
# See http://mesos.apache.org/documentation/latest/configuration/
authenticate = False
# Mesos credentials, if authentication is enabled
# default_principal = admin
# default_secret = admin
[admin]
# UI to hide sensitive variable fields when set to True
hide_sensitive_variable_fields = True
Could you please help
1-Usually your celery error logs will be available in your scheduler logs, so its better to check it there.
if you are not running it as daemon or background process, you can see it in detail in your terminal that what exactly is the problem.
2-As I gave it(your configs) a quick look, it looks so much like a default config file or a config that is almost match to the install guides online ==> so there must be no big issue with it.
3- Your error is not clear but i believe if you've just shipped your dag and its a sudden error after that there's a high chance you are facing your broker permission errors or any related errors to that as celery will communicate with it. So I will provide a common solution to a common problem hope it help the community, as you are far past the problem as 3 months past :)
rabbitmqctl set_permissions -p /myvhost guest ".*" ".*" ".*"
guest = your user( what you have provided as broker(in this case, RabbitMQ) user)
/myvhost = for you it might be just slash or /
Good luck.

Programmatically run a job in Kubernetes and get the output from the container command?

Looking at a client library for the Kubernetes API, I couldn't find an option to retrieve the output returned from a command execution inside the container. If we take the following python code example, can I get the actual output returned from the container start command?
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Creates, updates, and deletes a job object.
"""
from os import path
import yaml
from kubernetes import client, config
JOB_NAME = "pi"
def create_job_object():
# Configureate Pod template container
container = client.V1Container(
name="pi",
image="perl",
command=["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"])
# Create and configurate a spec section
template = client.V1PodTemplateSpec(
metadata=client.V1ObjectMeta(labels={"app": "pi"}),
spec=client.V1PodSpec(restart_policy="Never", containers=[container]))
# Create the specification of deployment
spec = client.V1JobSpec(
template=template,
backoff_limit=4)
# Instantiate the job object
job = client.V1Job(
api_version="batch/v1",
kind="Job",
metadata=client.V1ObjectMeta(name=JOB_NAME),
spec=spec)
return job
def create_job(api_instance, job):
api_response = api_instance.create_namespaced_job(
body=job,
namespace="default")
print("Job created. status='%s'" % str(api_response.status))
def update_job(api_instance, job):
# Update container image
job.spec.template.spec.containers[0].image = "perl"
api_response = api_instance.patch_namespaced_job(
name=JOB_NAME,
namespace="default",
body=job)
print("Job updated. status='%s'" % str(api_response.status))
def delete_job(api_instance):
api_response = api_instance.delete_namespaced_job(
name=JOB_NAME,
namespace="default",
body=client.V1DeleteOptions(
propagation_policy='Foreground',
grace_period_seconds=5))
print("Job deleted. status='%s'" % str(api_response.status))
def main():
# Configs can be set in Configuration class directly or using helper
# utility. If no argument provided, the config will be loaded from
# default location.
config.load_kube_config()
batch_v1 = client.BatchV1Api()
# Create a job object with client-python API. The job we
# created is same as the `pi-job.yaml` in the /examples folder.
job = create_job_object()
create_job(batch_v1, job)
update_job(batch_v1, job)
delete_job(batch_v1)
if __name__ == '__main__':
main()
I've only attached an example code from the Python client library but I've got a specific task in which I need the actual output after the job has been completed. I need that output in the API response though, because I will need to access that programatically.
So basically:
Run job in Python
Job finishes
Get output from container inside job (output from starting command)
This code will print the log (stdout output) after the job has completed:
from kubernetes import client, config
JOB_NAMESPACE = "default"
JOB_NAME = "pi"
def main():
config.load_kube_config()
batch_v1 = client.BatchV1Api()
job_def = batch_v1.read_namespaced_job(name=JOB_NAME, namespace=JOB_NAMESPACE)
controllerUid = job_def.metadata.labels["controller-uid"]
core_v1 = client.CoreV1Api()
pod_label_selector = "controller-uid=" + controllerUid
pods_list = core_v1.list_namespaced_pod(namespace=JOB_NAMESPACE, label_selector=pod_label_selector, timeout_seconds=10)
# Notice that:
# - there are more parameters to limit size, lines, and more - see
# https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#read_namespaced_pod_log
# - the logs of the 1st pod are returned (similar to `kubectl logs job/<job-name>`)
# - assumes 1 container in the pod
pod_name = pods_list.items[0].metadata.name
try:
# For whatever reason the response returns only the first few characters unless
# the call is for `_return_http_data_only=True, _preload_content=False`
pod_log_response = core_v1.read_namespaced_pod_log(name=pod_name, namespace=JOB_NAMESPACE, _return_http_data_only=True, _preload_content=False)
pod_log = pod_log_response.data.decode("utf-8")
print(pod_log)
except client.rest.ApiException as e:
print("Exception when calling CoreV1Api->read_namespaced_pod_log: %s\n" % e)
if __name__ == '__main__':
main()

snmpget : Unknown user name

I am trying to install net-snmp from scratch to make snmpv3 to work on my computer.
I did install net-snmp and create the user, but when I want to make snmpget it reject me with snmpget: Unknown user name
To install net-snmp I followed the official guide
I did install the packages libperl-dev, snmp-mibs-downloader and snmp too using sudo apt-get install
Here is my /usr/local/share/snmp configuration where you can find the particular line rouser neutg
###############################################################################
#
# EXAMPLE.conf:
# An example configuration file for configuring the Net-SNMP agent ('snmpd')
# See the 'snmpd.conf(5)' man page for details
#
# Some entries are deliberately commented out, and will need to be explicitly activated
#
###############################################################################
#
# AGENT BEHAVIOUR
#
# Listen for connections from the local system only
# agentAddress udp:127.0.0.1:161
# Listen for connections on all interfaces (both IPv4 *and* IPv6)
agentAddress udp:161,udp6:[::1]:161
###############################################################################
#
# SNMPv3 AUTHENTICATION
#
# Note that these particular settings don't actually belong here.
# They should be copied to the file /var/lib/snmp/snmpd.conf
# and the passwords changed, before being uncommented in that file *only*.
# Then restart the agent
# createUser authOnlyUser MD5 "remember to change this password"
# createUser authPrivUser SHA "remember to change this one too" DES
# createUser internalUser MD5 "this is only ever used internally, but still change the password"
# If you also change the usernames (which might be sensible),
# then remember to update the other occurances in this example config file to match.
###############################################################################
#
# ACCESS CONTROL
#
# system + hrSystem groups only
view systemonly included .1.3.6.1.2.1.1
view systemonly included .1.3.6.1.2.1.25.1
# Full access from the local host
#rocommunity public localhost
# Default access to basic system info
rocommunity public default -V systemonly
# rocommunity6 is for IPv6
rocommunity6 public default -V systemonly
# Full access from an example network
# Adjust this network address to match your local
# settings, change the community string,
# and check the 'agentAddress' setting above
#rocommunity secret 10.0.0.0/16
# Full read-only access for SNMPv3
rouser authOnlyUser
# Full write access for encrypted requests
# Remember to activate the 'createUser' lines above
#rwuser authPrivUser priv
# It's no longer typically necessary to use the full 'com2sec/group/access' configuration
# r[ow]user and r[ow]community, together with suitable views, should cover most requirements
###############################################################################
#
# SYSTEM INFORMATION
#
# Note that setting these values here, results in the corresponding MIB objects being 'read-only'
# See snmpd.conf(5) for more details
sysLocation Sitting on the Dock of the Bay
sysContact Me <me#example.org>
# Application + End-to-End layers
sysServices 72
#
# Process Monitoring
#
# At least one 'mountd' process
proc mountd
# No more than 4 'ntalkd' processes - 0 is OK
proc ntalkd 4
# At least one 'sendmail' process, but no more than 10
proc sendmail 10 1
# Walk the UCD-SNMP-MIB::prTable to see the resulting output
# Note that this table will be empty if there are no "proc" entries in the snmpd.conf file
#
# Disk Monitoring
#
# 10MBs required on root disk, 5% free on /var, 10% free on all other disks
disk / 10000
disk /var 5%
includeAllDisks 10%
# Walk the UCD-SNMP-MIB::dskTable to see the resulting output
# Note that this table will be empty if there are no "disk" entries in the snmpd.conf file
#
# System Load
#
# Unacceptable 1-, 5-, and 15-minute load averages
load 12 10 5
# Walk the UCD-SNMP-MIB::laTable to see the resulting output
# Note that this table *will* be populated, even without a "load" entry in the snmpd.conf file
###############################################################################
#
# ACTIVE MONITORING
#
# send SNMPv1 traps
trapsink localhost public
# send SNMPv2c traps
#trap2sink localhost public
# send SNMPv2c INFORMs
#informsink localhost public
# Note that you typically only want *one* of these three lines
# Uncommenting two (or all three) will result in multiple copies of each notification.
#
# Event MIB - automatically generate alerts
#
# Remember to activate the 'createUser' lines above
iquerySecName internalUser
rouser internalUser
# generate traps on UCD error conditions
defaultMonitors yes
# generate traps on linkUp/Down
linkUpDownNotifications yes
###############################################################################
#
# EXTENDING THE AGENT
#
#
# Arbitrary extension commands
#
extend test1 /bin/echo Hello, world!
extend-sh test2 echo Hello, world! ; echo Hi there ; exit 35
#extend-sh test3 /bin/sh /tmp/shtest
# Note that this last entry requires the script '/tmp/shtest' to be created first,
# containing the same three shell commands, before the line is uncommented
# Walk the NET-SNMP-EXTEND-MIB tables (nsExtendConfigTable, nsExtendOutput1Table
# and nsExtendOutput2Table) to see the resulting output
# Note that the "extend" directive supercedes the previous "exec" and "sh" directives
# However, walking the UCD-SNMP-MIB::extTable should still returns the same output,
# as well as the fuller results in the above tables.
#
# "Pass-through" MIB extension command
#
#pass .1.3.6.1.4.1.8072.2.255 /bin/sh PREFIX/local/passtest
#pass .1.3.6.1.4.1.8072.2.255 /usr/bin/perl PREFIX/local/passtest.pl
# Note that this requires one of the two 'passtest' scripts to be installed first,
# before the appropriate line is uncommented.
# These scripts can be found in the 'local' directory of the source distribution,
# and are not installed automatically.
# Walk the NET-SNMP-PASS-MIB::netSnmpPassExamples subtree to see the resulting output
#
# AgentX Sub-agents
#
# Run as an AgentX master agent
master agentx
# Listen for network connections (from localhost)
# rather than the default named socket /var/agentx/master
#agentXSocket tcp:localhost:705
rouser neutg
Here is my persistant configuration file /var/net-snmp/snmpd.conf
createUser neutg SHA "password" AES passphrase
The command I run is :
snmpget -u neutg -A password -a SHA -X 'passphrase'
-x AES -l authPriv localhost -v 3 1.3.6.1.2.1.1
I don't understand why it do not take in count my user. (I did restart the snmpd after entering the user - multiple times!)
The version of net-snmp I use :
Thanks in advance :)
After many research I've found what the problem is.
snmpd was not taking in count my configuration files. I saw it using the command :
snmpd -Dread_config -H 2>&1 | grep "Reading" | sort -u
Which tells you which configurations files are loaded by snmpd.
You can see it as well looking at the configuration file /var/lib/snmp/snmpd.conf. When snmpd handle your users it creates special lines in the file. It looks like :
usmUser 1 3 0x80001f888074336938f74f7c5a00000000 "neutg" "neutg" NULL .1.3.6.1.6.3.10.1.1.3 0xf965e4ab0f35eebb3f0e3b30\
6bc0797c025821c5 .1.3.6.1.6.3.10.1.2.4 0xe277044beccd9991d70144c4c8f4b672 0x
usmUser 1 3 0x80001f888074336938f74f7c5a00000000 "myuser" "myuser" NULL .1.3.6.1.6.3.10.1.1.2 0x2223c2d00758353b7c3076\
236be02152 .1.3.6.1.6.3.10.1.2.2 0x2223c2d00758353b7c3076236be02152 0x
setserialno 1424757026
So if you do not see any usmUser it's probably that your badly added your users.
The soluce
sudo /usr/local/sbin/snmpd -c /var/net-snmp/snmpd.conf -c /usr/local/share/snmp/snmpd.conf

ORA-12505, TNS:listener does not currently know of SID given in connect descriptor using oracle sql developer

i have gone through the forum to look for a solution for hours on end, i haven't encountered anyone with the same predicament as i have. I have installed Oracle 12c 64 bit on windows 8.1 64 bit system. When i try to make a new connection to the database, using oracle sql developer, run into this error, being a newbie to oracle database, i did try to run lsnrctl start i got
LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 18-OCT-2015 19:00
:45
Copyright (c) 1991, 2014, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1539)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
64-bit Windows Error: 61: Unknown error
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521_1)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
64-bit Windows Error: 2: No such file or directory
I tried to make sense of it but i coudn't.
Here is my listener.ora
# copyright (c) 1997 by the Oracle Corporation
#
# NAME
# listener.ora
# FUNCTION
# Network Listener startup parameter file example
# NOTES
# This file contains all the parameters for listener.ora,
# and could be used to configure the listener by uncommenting
# and changing values. Multiple listeners can be configured
# in one listener.ora, so listener.ora parameters take the form
# of SID_LIST_<lsnr>, where <lsnr> is the name of the listener
# this parameter refers to. All parameters and values are
# case-insensitive.
# <lsnr>
# This parameter specifies both the name of the listener, and
# it listening address(es). Other parameters for this listener
# us this name in place of <lsnr>. When not specified,
# the name for <lsnr> defaults to "LISTENER", with the default
# address value as shown below.
#
# LISTENER =
# (ADDRESS_LIST=
# (ADDRESS=(PROTOCOL=tcp)(HOST=localhost(PORT=1521))
# (ADDRESS=(PROTOCOL=ipc)(KEY=PNPKEY)))
# SID_LIST_<lsnr>
# List of services the listener knows about and can connect
# clients to. There is no default. See the Net8 Administrator's
# Guide for more information.
#
# SID_LIST_LISTENER=
# (SID_LIST=
# (SID_DESC=
#BEQUEATH CONFIG
# (GLOBAL_DBNAME=salesdb.mycompany)
# (SID_NAME=sid1)
# (ORACLE_HOME=/private/app/oracle/product/8.0.3)
# #PRESPAWN CONFIG
# (PRESPAWN_MAX=20)
# (PRESPAWN_LIST=
# (PRESPAWN_DESC=(PROTOCOL=tcp)(POOL_SIZE=2)(TIMEOUT=1))
# )
# )
# )
# PASSWORDS_<lsnr>
# Specifies a password to authenticate stopping the listener.
# Both encrypted and plain-text values can be set. Encrypted passwords
# can be set and stored using lsnrctl.
# LSNRCTL> change_password
# Will prompt for old and new passwords, and use encryption both
# to match the old password and to set the new one.
# LSNRCTL> set password
# Will prompt for the new password, for authentication with
# the listener. The password must be set before running the next
# command.
# LSNRCTL> save_config
# Will save the changed password to listener.ora. These last two
# steps are not necessary if SAVE_CONFIG_ON_STOP_<lsnr> is ON.
# See below.
#
# Default: NONE
#
# PASSWORDS_LISTENER = 20A22647832FB454 # "foobar"
# SAVE_CONFIG_ON_STOP_<lsnr>
# Tells the listener to save configuration changes to listener.ora when
# it shuts down. Changed parameter values will be written to the file,
# while preserving formatting and comments.
# Default: OFF
# Values: ON/OFF
#
# SAVE_CONFIG_ON_STOP_LISTENER = ON
# USE_PLUG_AND_PLAY_<lsnr>
# Tells the listener to contact an Onames server and register itself
# and its services with Onames.
# Values: ON/OFF
# Default: OFF
#
# USE_PLUG_AND_PLAY_LISTENER = ON
# LOG_FILE_<lsnr>
# Sets the name of the listener's log file. The .log extension
# is added automatically.
# Default=<lsnr>
#
# LOG_FILE_LISTENER = lsnr
# LOG_DIRECTORY_<lsnr>
# Sets the directory for the listener's log file.
# Default: <oracle_home>/network/log
#
# LOG_DIRECTORY_LISTENER = /private/app/oracle/product/8.0.3/network/log
# TRACE_LEVEL_<lsnr>
# Specifies desired tracing level.
# Default: OFF
# Values: OFF/USER/ADMIN/SUPPORT/0-16
#
# TRACE_LEVEL_LISTENER = SUPPORT
# TRACE_FILE_<lsnr>
# Sets the name of the listener's trace file. The .trc extension
# is added automatically.
# Default: <lsnr>
#
# TRACE_FILE_LISTENER = lsnr
# TRACE_DIRECTORY_<lsnr>
# Sets the directory for the listener's trace file.
# Default: <oracle_home>/network/trace
#
# TRACE_DIRECTORY_LISTENER=/private/app/oracle/product/8.0.3/network/trace
# CONNECT_TIMEOUT_<lsnr>
# Sets the number of seconds that the listener waits to get a
# valid database query after it has been started.
# Default: 10
#
# CONNECT_TIMEOUT_LISTENER=10
and my tnsnames.ora
# This file contains the syntax information for
# the entries to be put in any tnsnames.ora file
# The entries in this file are need based.
# There are no defaults for entries in this file
# that Sqlnet/Net3 use that need to be overridden
#
# Typically you could have two tnsnames.ora files
# in the system, one that is set for the entire system
# and is called the system tnsnames.ora file, and a
# second file that is used by each user locally so that
# he can override the definitions dictated by the system
# tnsnames.ora file.
# The entries in tnsnames.ora are an alternative to using
# the names server with the onames adapter.
# They are a collection of aliases for the addresses that
# the listener(s) is(are) listening for a database or
# several databases.
# The following is the general syntax for any entry in
# a tnsnames.ora file. There could be several such entries
# tailored to the user's needs.
<alias>= [ (DESCRIPTION_LIST = # Optional depending on whether u have
# one or more descriptions
# If there is just one description, unnecessary ]
(DESCRIPTION=
[ (SDU=2048) ] # Optional, defaults to 2048
# Can take values between 512 and 32K
[ (ADDRESS_LIST= # Optional depending on whether u have
# one or more addresses
# If there is just one address, unnecessary ]
(ADDRESS=
[ (COMMUNITY=<community_name>) ]
(PROTOCOL=tcp)
(HOST=<hostname>)
(PORT=<portnumber (1521 is a standard port used)>)
)
[ (ADDRESS=
(PROTOCOL=ipc)
(KEY=<ipckey (PNPKEY is a standard key used)>)
)
]
[ (ADDRESS=
[ (COMMUNITY=<community_name>) ]
(PROTOCOL=decnet)
(NODE=<nodename>)
(OBJECT=<objectname>)
)
]
... # More addresses
[ ) ] # Optional depending on whether ADDRESS_LIST is used or not
[ (CONNECT_DATA=
(SID=<oracle_sid>)
[ (GLOBAL_NAME=<global_database_name>) ]
)
]
[ (SOURCE_ROUTE=yes) ]
)
(DESCRIPTION=
[ (SDU=2048) ] # Optional, defaults to 2048
# Can take values between 512 and 32K
[ (ADDRESS_LIST= ] # Optional depending on whether u have more
# than one address or not
# If there is just one address, unnecessary
(ADDRESS
[ (COMMUNITY=<community_name>) ]
(PROTOCOL=tcp)
(HOST=<hostname>)
(PORT=<portnumber (1521 is a standard port used)>)
)
[ (ADDRESS=
(PROTOCOL=ipc)
(KEY=<ipckey (PNPKEY is a standard key used)>)
)
]
... # More addresses
[ ) ] # Optional depending on whether ADDRESS_LIST
# is being used
[ (CONNECT_DATA=
(SID=<oracle_sid>)
[ (GLOBAL_NAME=<global_database_name>) ]
)
]
[ (SOURCE_ROUTE=yes) ]
)
[ (CONNECT_DATA=
(SID=<oracle_sid>)
[ (GLOBAL_NAME=<global_database_name>) ]
)
]
... # More descriptions
[ ) ] # Optional depending on whether DESCRIPTION_LIST is used or not
am sorry for the long post, i just dint want to exclude anything out, is there a config that am missing?