Airflow server is not running in daemon mode - daemon

Airflow webserver is still running in the foreground with this command "airflow webserver -D". I am not sure if I am missing something here? Although, airflow scheduler -D is working properly

remove airflow-webserver.err and airflow-webserver-monitor.pid from AIRFLOW_HOME directory.
helped me

Related

How to run systemctl in a pod

Getting access denied error while running the systemctl command in a pod.
Whenever try to start any service, for example, MySQL or tomcat server in a pod, it gives access denied error.
Is there any way by which I can run systemctl within a pod.
This is a problem related to Docker, not Kubernetes.
According to the page Run multiple services in a container in docker docs:
It is generally recommended that you separate areas of concern by
using one service per container
However if you really want to use a process manager, you can try supervisord, which allows you to use supervisorctl commands, similar to systemctl. The page above explains how to do that:
Here is an example Dockerfile using this approach, that assumes the
pre-written supervisord.conf, my_first_process, and my_second_process
files all exist in the same directory as your Dockerfile.
FROM ubuntu:latest
RUN apt-get update && apt-get install -y supervisor
RUN mkdir -p /var/log/supervisor
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
COPY my_first_process my_first_process
COPY my_second_process my_second_process
CMD ["/usr/bin/supervisord"]
That's a rather short question. The 'systemctl' command does try to talk to the systemd daemon which is not running in a pod by default (it could however). Running multiple services is yet another question about service management. It both cases it could help to use a tool like the docker-systemctl-replacement overwriting /usr/bin/systemctl and registering it as the init-CMD of the container.

systemd service inside kubernetes is unable to get the env

I setup a Centos systemd service but I'm not able to read the kubernetes env variables. If I run the bash inside the pod I'm able to see env (such as _UI_SERVICE_PORT_TCP_443=443, KUBERNETES_PORT_443_TCP_ADDR=10.202.0.1 or container=docker) but not when I execute a bash script as a service inside the container.
I also tried Type=forking and ExecStart=/bin/bash believing the executed bash will inherit the kubernetes env but it's clean.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/ LANG=en_US.UTF-8 SHLVL=1
_=/bin/printenv
[Unit] Description= script after boot on k8s After=e.service
[Service] Type=forking ExecStart=#BINDIR#/virtual_service.py
Your problem seems to be related to the handling of environment variables in services. From my understanding, the env vars are stripped when running as a service, so you won't have access to what bash sees when your process runs as a service..
This answer provides a good description and some workarounds.
Hope this helps!
I found the answer for this.
/proc/1/environ contains the environment and i managed to read the env while i'm running as a service.
hope this will help someone in the future.

Command Line Option to Activate Airflow DAGs

We have a continuous integration pipeline which automatically deploys our Airflow DAGs to the Airflow server. When a new version of a DAG is deployed, its status is OFF by default. We would like to turn it ON as part of the tasks executed by the deployment process.
Is there a command line option in Airflow which allows to turn ON a DAG?
Thank you
Ok it seems I did not look carefully enough. The answer is just here in Airflow Documentation
You can turn OFF a DAG with the following command:
$ airflow pause <dag_id>
You can turn ON a DAG with the following command:
$ airflow unpause <dag_id>
Update:
The command airflow unpause has been removed.
You should now use
$ airflow dags unpause <dag_id>
instead of
$ airflow unpause <dag_id>
When you say new version, I am assuming you change the DAG_ID, have you consider to update the airflow.cfg to
dags_are_paused_at_creation = False?

How to run kafka rest proxy on windows

How to run kafka rest proxy on windows.
I downloaded confluent-2.0.1-2.11.7.tar.gz
in windows folder i cannot see kafka-rest-start.
Windows isn't currently a supported platform. However, it should work fine if you adapt the script. Even just running java io.confluent.kafkarest.KafkaRestMain with the appropriate classpath should work.
Here's the example of the command they are actually executing in the end of the bash script:
java -Xmx256M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dlog4j.configuration=file:C:/Dev/kafka/confluent-4.0.0/etc/kafka-rest/log4j.properties -cp .;C:/Dev/kafka/confluent-4.0.0/target/kafka-rest-*-development/share/java/kafka-rest/*;C:/Dev/kafka/confluent-4.0.0/share/java/confluent-common/*;C:/Dev/kafka/confluent-4.0.0/share/java/rest-utils/*;C:/Dev/kafka/confluent-4.0.0/share/java/kafka-rest/* io.confluent.kafkarest.KafkaRestMain C:/Dev/kafka/confluent-4.0.0/etc/kafka-rest/kafka-rest.properties
Make sure you change paths to yours if you want to try it out.
Perhaps, this answer will help anybody who is new to Kafka and stumble upon this situation, like me :).
I was looking for an answer to the very same question a week ago, came across the official suggestion to run jar files(in this path confluent-x.x.x\share\java\kafka-rest) in windows and was NOT successful in doing so.
Always ran into this error no main attribute found with or without specifying the proper classpath and io.confluent.kafkarest.KafkaRestMain.
I even tried running the shell scripts packaged for Linux distribution using [babun]: http://babun.github.io/, but that resulted in the error like Error: Could not find or load main class io.confluent.kafkarest.KafkaRestMain .
Eventually, docker image built with zookeeper, kafka, schema-registry, kafka-rest worked like a charm.
Here is the official page with info about the image name, further reference to it's doc: https://hub.docker.com/r/confluentinc/cp-kafka-rest/
Upon pulling this image, a new virtual machine gets created with four more images inside it(one for each service like zookeeper, Kafka, schem-registry and Kafka-rest). Running the images runs a separate Docker container.
This guide should get you started quickly:
http://docs.confluent.io/current/cp-docker-images/docs/quickstart.html
And finally, if you would like to expose the kafka REST proxy server running as a Docker container to outside network(like windows machine which is part of the separate network than these containers) just mention the Docker host IP(find it by hitting docker-machine ip <hostname>) in KAFKA_REST_LISTENERS and expose the port with -p option.
Like this:
docker run -d \
--net=host \
--name=kafka-rest \
-p 8082:8082 \
-e KAFKA_REST_ZOOKEEPER_CONNECT=localhost:32181 \
-e KAFKA_REST_LISTENERS=http://192.168.99.100:8082 \
-e KAFKA_REST_SCHEMA_REGISTRY_URL=http://localhost:8081 \
-e KAFKA_REST_HOST_NAME=localhost \
confluentinc/cp-kafka-rest:3.2.1
If everything is OK, you will be able to access REST proxy at this url http://<Docker_host_IP>:8082 from the windows machine.
I was able to run the command that #lexler mentioned above, but outside of cygwin. (directly with the windows command prompt.)

docker build does not sustain processes

So this might be my Dockerfile:
FROM ubuntu:latest
RUN apt-get -y update && apt-get install -y mysql-server-5.6
RUN service mysql start
RUN service mysql status
It throws an error during the build that MySQL is not running, even though the previous command finished with a success. The deamons seem not to be able to be running between different commands in the Dockerfile.
This is an artificial example, but in my real Dockerfile I have lines which configure the database and they need to have a deamon running in the backgroud. The only solution to go around this that I found is to run:
RUN service mysql start && ./database_configure1.sh
RUN service mysql start && ./do_something_else_with_db.sh
and so on
But this is probably not the way to do it. Is there any better way to go about this?
Each RUN command within your Dockerfile runs within a different container, so here's the actual sequence of events:
service mysql start starts MySQL.
Then the container is stopped (MySQL is stopped).
Then a snapshot is taken.
Then a new container is launched using that snapshot.
service mysql status is run in the new container.
Of course, mysql isn't actually running in the latter container, so that fails.
So, instead, you need to do everything in a single build step. Usually, you'll want to do this by running a shell script within your container.
Here goes.
Your directory tree should look like this:
Dockerfile
do_stuff_with_mysql.sh
Then, in your Dockerfile, do:
ADD do_stuff_with_mysql.sh /
RUN chmod 755 /do_stuff_with_mysql.sh
RUN do_stuff_with_mysql.sh
And, in do_stuff_with_mysql.sh, you should have something that looks like this:
#!/bin/bash
set -o errexit
set -o nounset
service mysql start
./database_configure1.sh
./do_something_else_with_db.sh
service mysql stop
# you should loop on `service mysql status` to confirm MySQL is done shutting down