Bitbake server does not start - yocto

I am having some trouble with building the Yocto project, hope I can find some help. Is there any way to fix the following issue? Let me know if you need any more information. Thanks.
My goal
I am building the default image from this guide : https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html
This build takes up a lot of space, so I want to build it on a network drive.
Current situation
I am able to finish the build normally if I am using a normal folder.
If I use the the shared drive that is mounted in the system, the build never starts. The error looks like this :
$ bitbake core-image-sato
NOTE: Bitbake server didn't start within 5 seconds, waiting for 90
ERROR: Error parsing configuration files
Traceback (most recent call last):
File "/mnt/NetworkShare/yocto/poky/bitbake/lib/bb/persist_data.py", line 45, in SQLTable.wrap_func(*args=('CREATE TABLE IF NOT EXISTS BB_URI_HEADREVS(key TEXT PRIMARY KEY NOT NULL, value TEXT);',), **kwargs={}):
if self.connection is None and reconnect:
> self.reconnect()
File "/mnt/NetworkShare/yocto/poky/bitbake/lib/bb/persist_data.py", line 105, in SQLTable.reconnect():
self.connection.text_factory = str
> self._setup_database()
File "/mnt/NetworkShare/yocto/poky/bitbake/lib/bb/persist_data.py", line 50, in SQLTable.wrap_func(*args=(), **kwargs={}):
try:
> return f(self, *args, **kwargs)
except sqlite3.OperationalError as exc:
File "/mnt/NetworkShare/yocto/poky/bitbake/lib/bb/persist_data.py", line 79, in SQLTable.wrap_func(*args=(), **kwargs={}):
with contextlib.closing(self.connection.cursor()) as cursor:
> return f(self, cursor, *args, **kwargs)
return wrap_func
File "/mnt/NetworkShare/yocto/poky/bitbake/lib/bb/persist_data.py", line 93, in SQLTable._setup_database(cursor=<sqlite3.Cursor object at 0x7f3d59c5dab0>):
def _setup_database(self, cursor):
> cursor.execute("pragma synchronous = off;")
# Enable WAL and keep the autocheckpoint length small (the default is
sqlite3.OperationalError: disk I/O error
Details
The /etc/fstab line to mount the drive is :
NetworkShare /mnt/NetworkShare 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0
The host is Ubuntu server 20.04 running in a VM inside UnRAID. I don't think the VM is the issue (It's possible that I am very wrong) because I get the same error if I mount an external share on my own computer (OpenSuse Tumbleweed) and try to build in it.
The bitbake-cookerdaemon.log :
1221 13:38:18.293775 --- Starting bitbake server pid 1221 at 2022-01-19 13:38:18.293689 ---
1221 13:38:18.333537 Started bitbake server pid 1221
1221 13:38:18.339125 Entering server connection loop
1221 13:38:18.340399 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
1221 13:38:18.341382 Processing Client
1221 13:38:18.342099 Connecting Client
1221 13:38:18.343689 Running command ['setFeatures', [2]]
1221 13:38:18.344805 Command Completed
1221 13:38:18.346085 Running command ['updateConfig', {'abort': True, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'SHELL': '/bin/bash', 'PWD': '/mnt/NetworkShare/yocto/poky/build', 'LOGNAME': 'metics', 'HOME': '/home/metics', 'BBPATH': '/mnt/NetworkShare/yocto/poky/build', 'BB_ENV_EXTRAWHITE': 'ALL_PROXY BBPATH_EXTRA BB_LOGCONFIG BB_NO_NETWORK BB_NUMBER_THREADS BB_SETSCENE_ENFORCE BB_SRCREV_POLICY DISTRO FTPS_PROXY FTP_PROXY GIT_PROXY_COMMAND HTTPS_PROXY HTTP_PROXY MACHINE NO_PROXY PARALLEL_MAKE SCREENDIR SDKMACHINE SOCKS5_PASSWD SOCKS5_USER SSH_AGENT_PID SSH_AUTH_SOCK STAMPS_DIR TCLIBC TCMODE all_proxy ftp_proxy ftps_proxy http_proxy https_proxy no_proxy ', 'USER': 'metics', 'PATH': '/mnt/NetworkShare/yocto/poky/scripts:/mnt/NetworkShare/yocto/poky/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'LC_ALL': 'en_US.UTF-8', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#51:co#110:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d#:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Km=\\E[M:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:#1=\\E[1~:kH=\\E[4~:#7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:', 'WINDOW': '0', 'XDG_SESSION_TYPE': 'tty', 'MOTD_SHOWN': 'pam', 'LANG': 'en_US.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'SSH_CONNECTION': '10.0.0.12 60522 10.0.0.19 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'PYTHONPATH': '/mnt/NetworkShare/yocto/poky/bitbake/lib:', 'TERM': 'screen.xterm-256color', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'SHLVL': '2', 'XDG_SESSION_ID': '1', 'XDG_RUNTIME_DIR': '/run/user/1000', 'SSH_CLIENT': '10.0.0.12 60522 22', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/var/lib/snapd/desktop', 'STY': '1116.pts-0.ubuntuserver', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'BUILDDIR': '/mnt/NetworkShare/yocto/poky/build', 'SSH_TTY': '/dev/pts/0', 'OLDPWD': '/mnt/NetworkShare/yocto/poky', '_': '/mnt/NetworkShare/yocto/poky/bitbake/bin/bitbake'}, ['/mnt/NetworkShare/yocto/poky/bitbake/bin/bitbake', 'core-image-sato']]
1221 13:38:33.830099 Command Completed
1221 13:38:33.831731 Processing Client
1221 13:38:33.832344 Disconnecting Client
1221 13:38:33.833129 No timeout, exiting.
1221 13:38:33.933875 Exiting
1221 13:38:33.942717 Original lockfile contents: ['1221\n']
1221 13:38:33.954461 Exiting as we could obtain the lock
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/mnt/NetworkShare/yocto/poky/build/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'>

It means your hard disk is full. You should delete some files before re-running job to create a new image.

Related

Couldn't resolve module/action 'k8s_exec' on Ansible Playbook

I am trying to make a reboot to a pod on a Rancher Cluster through Ansible and I am having this error:
ERROR! couldn't resolve module/action 'k8s_exec'. This often indicates a misspelling, missing collection, or incorrect module path.
The error appears to be in '/home/ansible/ansible/GetKubectlPods': line 27, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
-
name: Reboot Machine
^ here
I don't know why I am getting an error since I am also using k8s_info and he is working be fine.
Here is the playbook I am running:
---
- hosts: localhost
connection: local
remote_user: root
vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'
tasks:
-
name: Get the pods in the specific namespace
k8s_info:
kubeconfig: '/etc/ansible/RCCloudConfig'
kind: Pod
namespace: redmine
register: pod_list
-
name: Print pod names
debug:
msg: "pod_list: {{ pod_list | json_query('resources[*].status.podIP') }} "
- set_fact:
pod_names: "{{pod_list|json_query('resources[*].metadata.name')}}"
-
name: Reboot Machine
k8s_exec:
kubeconfig: '/etc/ansible/RCCloudConfig'
namespace: redmine
pod: redmine_quick-testing-6c57cc5d65-lwkww #pod name
command: reboot
Ansible Version:
ansible 2.9.9
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
In case I edit my playbook and add the community.kubernetes collection, I get the following error:
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/__init__.py:12: DeprecationWarning: The package kubernetes.client.apis is renamed and deprecated, use kubernetes.client.api instead (please note that the trailing s was removed).\n DeprecationWarning\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 296, in websocket_call\n client = WSClient(configuration, get_websocket_url(url), headers, capture_all)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 94, in __init__\n self.sock.connect(url, header=header)\n File \"/usr/local/lib/python3.6/site-packages/websocket/_core.py\", line 226, in connect\n self.handshake_response = handshake(self.sock, *addrs, **options)\n File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 80, in handshake\n status, resp = _get_resp_headers(sock)\n File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 165, in _get_resp_headers\n raise WebSocketBadStatusException(\"Handshake status %d %s\", status, status_message, resp_headers)\nwebsocket._exceptions.WebSocketBadStatusException: Handshake status 404 Not Found\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1592530746.4685414-46123-84159696554463/AnsiballZ_k8s_exec.py\", line 102, in <module>\n _ansiballz_main()\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1592530746.4685414-46123-84159696554463/AnsiballZ_k8s_exec.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1592530746.4685414-46123-84159696554463/AnsiballZ_k8s_exec.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.community.kubernetes.plugins.modules.k8s_exec', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_k8s_exec_payload_q6oom26c/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 148, in <module>\n File \"/tmp/ansible_k8s_exec_payload_q6oom26c/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 135, in main\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 35, in stream\n return func(*args, **kwargs)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py\", line 841, in connect_get_namespaced_pod_exec\n (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py\", line 941, in connect_get_namespaced_pod_exec_with_http_info\n collection_formats=collection_formats)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 345, in call_api\n _preload_content, _request_timeout)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 176, in __call_api\n _request_timeout=_request_timeout)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 30, in _intercept_request_call\n return ws_client.websocket_call(config, *args, **kwargs)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 302, in websocket_call\n raise ApiException(status=0, reason=str(e))\nkubernetes.client.rest.ApiException: (0)\nReason: Handshake status 404 Not Found\n\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
Solution:
Find your k8s_exec.py file and edit your import settings from Kubernetes.apis to Kubernetes.api
Thank you in advance!

Airflow scheduler fails to start with kubernetes executor

I am using using https://github.com/helm/charts/tree/master/stable/airflow helm chart and building v1.10.8 puckle/docker-airflow image with kubernetes installed on it and using that image in the helm chart,
But I keep getting
File "/usr/local/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1140, in initdb
db.initdb(settings.RBAC)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 332, in initdb
dagbag = models.DagBag()
File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 95, in __init__
executor = get_default_executor()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
DEFAULT_EXECUTOR = _get_executor(executor_name)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 87, in _get_executor
return KubernetesExecutor()
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 702, in __init__
self.kube_config = KubeConfig()
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 283, in __init__
self.kube_client_request_args = json.loads(kube_client_request_args)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
In my scheduler, also as various sources advise,
I tried setting :
AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: {"_request_timeout" : [60,60] }
in my helm values. that also didn't work any one have any ideas what am I missing?
Here's my values.yaml
airflow:
image:
repository: airflow-docker-local
tag: 1.10.8
executor: Kubernetes
service:
type: LoadBalancer
config:
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: airflow-docker-local
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: 1.10.8
AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: Never
AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: airflow
AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM: airflow
AIRFLOW__KUBERNETES__NAMESPACE: airflow
AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: {"_request_timeout" : [60,60] }
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://postgres:airflow#airflow-postgresql:5432/airflow
persistence:
enabled: true
existingClaim: ''
workers:
enabled: false
postgresql:
enabled: true
redis:
enabled: false
EDIT :
Various attempts to set environment variable in helm values.yaml didn't work, after that I added (pay attention to double and single quotes)
ENV AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS='{"_request_timeout" : [60,60] }'
to Dockerfile here : https://github.com/puckel/docker-airflow/blob/1.10.9/Dockerfile#L19
after that my airflow-scheduler pod starts but then I keep getting following error on my scheduler pod.
Process KubernetesJobWatcher-9: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 313,
in recv_into return self.connection.recv_into(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py",
line 1840, in recv_into self._raise_ssl_error(self._ssl, result) File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py",
line 1646, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError
For the helm value, the template uses a loop that places the airflow.config map into double quotes ". This means any " in a value needs to be escaped for the output templated YAML to be valid.
airflow:
config:
AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: '{\"_request_timeout\":60}'
That deploys and runs (but I haven't completed an end to end test)
According to this github issue, the python scheduler SSL timeout may not be a problem as the watcher starts again after the 60 second connection timeout.

Unable to run airflow scheduler

I have recently installed airflow on an AWS server by using this guide for ubuntu 16.04. After a painful and successful install started the webserver. I tried a sample dag as follows
from airflow.operators.python_operator import PythonOperator
from airflow.operators.dummy_operator import DummyOperator
from datetime import timedelta
from airflow import DAG
import airflow
# DEFAULT ARGS
default_args = {
'owner': 'airflow',
'start_date': airflow.utils.dates.days_ago(2),
'depends_on_past': False}
dag = DAG('init_run', default_args=default_args, description='DAG SAMPLE',
schedule_interval='#daily')
def print_something():
print("HELLO AIRFLOW!")
with dag:
task_1 = PythonOperator(task_id='do_it', python_callable=print_something)
task_2 = DummyOperator(task_id='dummy')
task_1 << task_2
But when i open the UI the tasks in the dag are still in "No Status" no matter how many times i trigger manually or refresh the page.
Later i found out that airflow scheduler is not running and shows the following error:
{celery_executor.py:228} ERROR - Error sending Celery task:No module named 'MySQLdb'
Celery Task ID: ('init_run', 'dummy', datetime.datetime(2019, 5, 30, 18, 0, 24, 902499, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 118, in send_task_to_executor
result = task.apply_async(args=[command], queue=queue)
File "/usr/local/lib/python3.7/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 728, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/local/lib/python3.7/site-packages/celery/app/amqp.py", line 552, in send_task_message
**properties
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 510, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 194, in _publish
[maybe_declare(entity) for entity in declare]
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 194, in <listcomp>
[maybe_declare(entity) for entity in declare]
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 102, in maybe_declare
return maybe_declare(entity, self.channel, retry, **retry_policy)
File "/usr/local/lib/python3.7/site-packages/kombu/common.py", line 121, in maybe_declare
return _maybe_declare(entity, channel)
File "/usr/local/lib/python3.7/site-packages/kombu/common.py", line 145, in _maybe_declare
entity.declare(channel=channel)
File "/usr/local/lib/python3.7/site-packages/kombu/entity.py", line 608, in declare
self._create_queue(nowait=nowait, channel=channel)
File "/usr/local/lib/python3.7/site-packages/kombu/entity.py", line 617, in _create_queue
self.queue_declare(nowait=nowait, passive=False, channel=channel)
File "/usr/local/lib/python3.7/site-packages/kombu/entity.py", line 652, in queue_declare
nowait=nowait,
File "/usr/local/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 531, in queue_declare
self._new_queue(queue, **kwargs)
File "/usr/local/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 82, in _new_queue
self._get_or_create(queue)
File "/usr/local/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 70, in _get_or_create
obj = self.session.query(self.queue_cls) \
File "/usr/local/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 65, in session
_, Session = self._open()
File "/usr/local/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 56, in _open
engine = self._engine_from_config()
File "/usr/local/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 51, in _engine_from_config
return create_engine(conninfo.hostname, **transport_options)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 443, in create_engine
return strategy.create(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 104, in dbapi
return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'
Here is the setting in the config file (airflow.cfg):
sql_alchemy_conn = postgresql+psycopg2://airflow#localhost:5432/airflow
broker_url = sqla+mysql://airflow:airflow#localhost:3306/airflow
result_backend = db+postgresql://airflow:airflow#localhost/airflow
I been struggling with this issue for two days now, Please help
In your airflow.cfg, there should also be a config option for celery_result_backend. Are you able to let us know what this value is set to? If it is not present in your config, set it to the same value as the result_backend
i.e:
celery_result_backend = db+postgresql://airflow:airflow#localhost/airflow
And then restart the airflow stack to ensure the configuration changes apply.
(I wanted to leave this as a comment but don't have enough rep to do so)
I think the example you are following didnt told you to install mysql and it seems you are using it in broker URL.
you can install mysql and than configure it. (for python 3.5+)
pip install mysqlclient
Alternatively, for a quick fix. You can also use rabbit MQ(Rabbitmq is a message broker, that you will require to rerun airflow dags with celery) guest user login
and then your broker_url will be
broker_url = amqp://guest:guest#localhost:5672//
if not already installed, Rabbitmq can be installed with following command.
sudo apt install rabbitmq-server
Change configuration NODE_IP_ADDRESS=0.0.0.0 in configuration file located at
/etc/rabbitmq/rabbitmq-env.conf
start RabbitMQ service
sudo service rabbitmq-server start

Kubectl connecting to Azure ACS cluster

I have deployed a cluster to Azure successfully. Now trying to set up kubectl to work with it.
Running:
az acs kubernetes get-credentials --resource-group=group --name=cluster
Results in:
scp: .kube/config: No such file or directory
Traceback (most recent call last):
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/main.py", line 36, in main
cmd_result = APPLICATION.execute(args)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/application.py", line 212, in execute
result = expanded_arg.func(params)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 377, in __call__
return self.handler(*args, **kwargs)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 626, in _execute_command
reraise(*sys.exc_info())
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 603, in _execute_command
result = op(client, **kwargs) if client else op(**kwargs)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 814, in k8s_get_credentials
_k8s_get_credentials_internal(name, acs_info, path, ssh_key_file)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 835, in _k8s_get_credentials_internal
'.kube/config', path_candidate, key_filename=ssh_key_file)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/command_modules/acs/acs_client.py", line 64, in secure_copy
scp.get(src, dest)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/scp.py", line 198, in get
self._recv_all()
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/scp.py", line 348, in _recv_all
raise SCPException(asunicode(msg[1:]))
SCPException: scp: .kube/config: No such file or directory
File ~/.kube/config exists, and I can ssh to the master node of the cluster.
Any idea what on can be wrong with what I am doing?
UPDATE.
Tried running the command per one of the comments here:
az acs kubernetes get-credentials --resource-group=group --name=cluster -f /Users/me/.kube/config --debug
And received the following info:
Password for private key:
paramiko.transport : starting thread (client mode): 0x9659890L
paramiko.transport : Local version/idstring: SSH-2.0-paramiko_2.3.1
paramiko.transport : Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)
paramiko.transport : kex algos:[u'curve25519-sha256#libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] server key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'chacha20-poly1305#openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm#openssh.com', u'aes256-gcm#openssh.com'] server encrypt:[u'chacha20-poly1305#openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm#openssh.com', u'aes256-gcm#openssh.com'] client mac:[u'umac-64-etm#openssh.com', u'umac-128-etm#openssh.com', u'hmac-sha2-256-etm#openssh.com', u'hmac-sha2-512-etm#openssh.com', u'hmac-sha1-etm#openssh.com', u'umac-64#openssh.com', u'umac-128#openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'umac-64-etm#openssh.com', u'umac-128-etm#openssh.com', u'hmac-sha2-256-etm#openssh.com', u'hmac-sha2-512-etm#openssh.com', u'hmac-sha1-etm#openssh.com', u'umac-64#openssh.com', u'umac-128#openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'zlib#openssh.com'] server compress:[u'none', u'zlib#openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
paramiko.transport : Kex agreed: ecdh-sha2-nistp256
paramiko.transport : HostKey agreed: ecdsa-sha2-nistp256
paramiko.transport : Cipher agreed: aes128-ctr
paramiko.transport : MAC agreed: hmac-sha2-256
paramiko.transport : Compression agreed: none
paramiko.transport : kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
paramiko.transport : Switch to new keys ...
paramiko.transport : Trying SSH key 765f702f5fff7fe30260d53b5bbb57eb
paramiko.transport : userauth is OK
paramiko.transport : Authentication (publickey) successful!
paramiko.transport : [chan 0] Max packet in: 32768 bytes
paramiko.transport : Received global request "hostkeys-00#openssh.com"
paramiko.transport : Rejecting "hostkeys-00#openssh.com" global request from server.
paramiko.transport : [chan 0] Max packet out: 32768 bytes
paramiko.transport : Secsh channel 0 opened.
paramiko.transport : [chan 0] Sesch channel 0 request ok
paramiko.transport : [chan 0] EOF received (0)
paramiko.transport : [chan 0] EOF sent (0)
scp: .kube/config: No such file or directory
Traceback (most recent call last):
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/main.py", line 36, in main
cmd_result = APPLICATION.execute(args)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/application.py", line 212, in execute
result = expanded_arg.func(params)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 377, in __call__
return self.handler(*args, **kwargs)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 626, in _execute_command
reraise(*sys.exc_info())
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 603, in _execute_command
result = op(client, **kwargs) if client else op(**kwargs)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 814, in k8s_get_credentials
_k8s_get_credentials_internal(name, acs_info, path, ssh_key_file)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 835, in _k8s_get_credentials_internal
'.kube/config', path_candidate, key_filename=ssh_key_file)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/azure/cli/command_modules/acs/acs_client.py", line 64, in secure_copy
scp.get(src, dest)
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/scp.py", line 198, in get
self._recv_all()
File "/Users/me/lib/azure-cli/lib/python2.7/site-packages/scp.py", line 348, in _recv_all
raise SCPException(asunicode(msg[1:]))
SCPException: scp: .kube/config: No such file or directory
Version of azure cli is 2.0.18. Would appreciate advice.
Try Using --file -f flag to specify the absolute path of the "config" file in .kube directory and you can use --debug to debug further.
az acs kubernetes get-credentials --resource-group=group --name=cluster -f /.../.../.kube/config --debug
In my case, .kube/config file wasn't created on the master. There was an error during the attempt to create .kube/config file, which I have noticed by ssh-ing to my master node and looking into /var/log/azure directory. The error was related to my service principal password. With a new service principal, new clusters work just fine.

How to open firewall port with ansible firewalld task on Centos 7

I have a task in my ansible-playbook script to open TCP port on a remote machine. but when I run my ansible playbook it throws an error. But when i run "firewall-cmd --permanent --zone=public --add-port=1234/tcp" and "firewalld-cmd --reload" I can see port is added in public zone.
Environment
Ansible local: OS x El Capitan
Ansible remote: AWS Centos 7 minimum version
Ansible version: 2.1.1.0
Remote python version: 2.7.5
My task
- name: open management console port
firewalld: port=1234/tcp zone=public permanent=true state=enabled immediate=yes
The error I am getting
fatal: [X.X.X.X]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_MojhHQ/ansible_module_firewalld.py\", line 605, in <module>\r\n main()\r\n File \"/tmp/ansible_MojhHQ/ansible_module_firewalld.py\", line 456, in main\r\n is_enabled = get_port_enabled_permanent(zone, [port, protocol])\r\n File \"/tmp/ansible_MojhHQ/ansible_module_firewalld.py\", line 170, in get_port_enabled_permanent\r\n fw_zone = fw.config().getZoneByName(zone)\r\n File \"<string>\", line 2, in getZoneByName\r\n File \"/usr/lib/python2.7/site-packages/slip/dbus/polkit.py\", line 103, in _enable_proxy\r\n return func(*p, **k)\r\n File \"<string>\", line 2, in getZoneByName\r\n File \"/usr/lib/python2.7/site-packages/firewall/client.py\", line 52, in handle_exceptions\r\n return func(*args, **kwargs)\r\n File \"/usr/lib/python2.7/site-packages/firewall/client.py\", line 1505, in getZoneByName\r\n path = dbus_to_python(self.fw_config.getZoneByName(name))\r\n File \"/usr/lib64/python2.7/site-packages/dbus/proxies.py\", line 70, in __call__\r\n return self._proxy_method(*args, **keywords)\r\n File \"/usr/lib/python2.7/site-packages/slip/dbus/proxies.py\", line 50, in __call__\r\n return dbus.proxies._ProxyMethod.__call__(self, *args, **kwargs)\r\n File \"/usr/lib64/python2.7/site-packages/dbus/proxies.py\", line 145, in __call__\r\n **keywords)\r\n File \"/usr/lib64/python2.7/site-packages/dbus/connection.py\", line 651, in call_blocking\r\n message, timeout)\r\ndbus.exceptions.DBusException: org.fedoraproject.slip.dbus.service.PolKit.NotAuthorizedException.org.fedoraproject.FirewallD1.config: \r\n", "msg": "MODULE FAILURE", "parsed": false}
- name: Install firewalld
yum:
name: firewalld
state: latest
notify:
- start firewalld
- name: start firewalld
service:
name: firewalld
state: started
enabled: yes
become: yes
- name: enable 1234
firewalld:
zone: public
port: 1234/tcp
permanent: true
state: enabled
become: yes
Do it this way . It will work
dbus.exceptions.DBusException: org.fedoraproject.slip.dbus.service.PolKit.NotAuthorizedException.org.fedoraproject.FirewallD1.config indicates there's some sort of permissions error. The task probably needs to elevate its privileges with become: yes.
See the become documentation for more details.