I'm trying to setup Kube's cluster with 2 master node 10.0.11.108 and 10.0.11.97 (Ubuntu) with config "ai ai" in cluster/ubuntu/config-default.sh file.
When I run
KUBERNETES_PROVIDER=ubuntu ./kube-up.sh,
it run deployscript in node 10.0.11.97 twice. And become error:
[sudo] password to copy files and start node: cp: cannot create
regular file ‘/opt/bin/etcd’: Text file busy cp: cannot create regular
file ‘/opt/bin/kube-apiserver’: Text file busy cp: cannot create
regular file ‘/opt/bin/kube-controller-manager’: Text file busy cp:
cannot create regular file ‘/opt/bin/kube-scheduler’: Text file busy
start: Job is already running: etcd
I setup Kube's cluster with 1 master, run twice, encounter the same error message,
I modified utuntu/util.sh, and get 'Cluster validation succeeded'.
First add -f for all cp commands, like this: cp ~/kube/default/* /etc/default/ ===> cp -f ~/kube/default/* /etc/default/
Then rerun ./kube-up.sh, you will encounter "start: Job is already running: etcd"
Then modified utuntu/util.sh, modify all 'service XXX start' to 'service XXX restart' , and run kube-up.sh, You'll get the success message.
Related
I am running a Celery Executor and I'm trying to run some python script in the KubernetesPodOperator. Below are examples of what I have tried that didn't work. What am I doing wrong?
Running sctipt
org_node = KubernetesPodOperator(
namespace='default',
image="python",
cmds=["python", "somescript.py" "-c"],
arguments=["print('HELLO')"],
labels={"foo": "bar"},
image_pull_policy="Always",
name=task,
task_id=task,
is_delete_operator_pod=False,
get_logs=True,
dag=dag
)
Running function load_users_into_table()
def load_users_into_table(postgres_hook, schema, path):
gdf = read_csv(path)
gdf.to_sql('users', con=postgres_hook.get_sqlalchemy_engine(), schema=schema)
org_node = KubernetesPodOperator(
namespace='default',
image="python",
cmds=["python", "somescript.py" "-c"],
arguments=[load_users_into_table],
labels={"foo": "bar"},
image_pull_policy="Always",
name=task,
task_id=task,
is_delete_operator_pod=False,
get_logs=True,
dag=dag
)
The script somescript.py must be in Docker image.
Step-1: let's create a image https://docs.docker.com/develop/develop-images/dockerfile_best-practices/.
FROM python:3.8
# copy requirement.txt from local to container
COPY requirements.txt requirements.txt
# install dependencies into container (geopandas, sqlalchemy)
RUN pip install -r requirements.txt
# copy the python script from local to container
COPY somescript.py somescript.py
ENTRYPOINT [ "python", "somescript.py"]
Step-2: Build and push the image into public Docker repository https://hub.docker.com.
NB: kubernetes_pod_operator looks for image from public docker repo
# build image
docker build -t my-python-img:latest .
# test if your image works perfectly
docker run my-python-img:latest
# push image.
docker tag my-python-img username/my-python-img
docker push username/my-python-img
docker pull username/my-python-img
step-3: Lest's create k8s task.
p = KubernetesPodOperator(
namespace='default',
image='username/my-python-img:latest',
labels={'dag-id': dag.dag_id},
name='airflow-my-image-pod',
task_id='load-users',
in_cluster=False, #False: local, True: cluster
cluster_context='microk8s',
config_file='/usr/local/airflow/include/.kube/config',
is_delete_operator_pod=True,
get_logs=True,
dag=dag
)
If you don't understand where configuration file comes from, look here: https://www.astronomer.io/docs/cloud/stable/develop/kubepodoperator-local.
Finally: I want to mention something important when working with databases (credentials). Kubernetes offers the use secret to secure sensitive information. https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html
KubernetesPodOperator launches a Kubernetes pod that runs a container as specified in the operator's arguments.
First Example
In the first example, the following happens:
KubernetesPodOperator instructs K8s to lunch a pod and prepare to run a container in it using the python image (the image parameter) from hub.docker.com (the default image registry)
ENTRYPOINT of the python image is replaced by ["python", "somescript.py" "-c"] (the cmd parameter)
CMD of the python image is replaced by ["print('HELLO')"] (the arguments parameter)
...
The container is run
So, the complete command that is run in the container is
python somescript.py -c print('HELLO')
Obviously, the official Python image from Docker Hub does not have somescript.py in its working directory. Even if did, it probably would have been not the one that you wrote. That is why the command fails with something like:
python: can't open file 'somescrit.py': [Errno 2] No such file or directory
Second Example
In the second example, pretty much the same happens as in the first example, but the command that is run in the container (again based on the cmd and arguments parameters) is
python somescript.py -c None
(None is the string representation of the load_users_into_table()'s return value)
This command fails, because of the same reasons as in the first example.
How It Could be Done (a Sketch)
You could build a Docker image with somescript.py and all its dependencies. Push the image to an image registry. Specify the image, ENTRYPOINT, and CMD in the corresponding parameters of KubernetesPodOperator.
In the following Dockerfile I'm trying to copy a jar file from a location on the host into the container, but seems Docker does not like it as I guess I'm missing something. Here is my Dockerfile:
FROM anapsix/alpine-java:jdk8
MAINTAINER joesan
ENV SBT_VERSION 0.13.15
ENV CHECKSUM 18b106d09b2874f2a538c6e1f6b20c565885b2a8051428bd6d630fb92c1c0f96
ENV APP_NAME my-app
ENV PROJECT_HOME /opt/apps
RUN mkdir -p $PROJECT_HOME/$APP_NAME
# Copy the jar file
COPY ./target/scala-*/my-app-*.jar $PROJECT_HOME/$APP_NAME
# Copy the database file
COPY .my-db.mv.db $PROJECT_HOME/$APP_NAME
# Run the application
CMD ["$PROJECT_HOME/$APP_NAME java -Denv=dev -jar my-app-*.jar"]
In my build pipeline, I could see the following error message:
Step 8/10 : COPY ./target/scala-*/my-app-*.jar $PROJECT_HOME/$APP_NAME
COPY failed: no source files were specified
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 4a240742a379 Less than a second ago 171MB
anapsix/alpine-java jdk8 ed55c27d366d 3 years ago 171MB
Error response from daemon: No such image: [secure]
Pushing image [secure] to repository hub.docker.com
The push refers to repository [docker.io/[secure]/my-app]
An image does not exist locally with the tag: [secure]/my-app
What is that I'm missing and how could I debug this? I mean I could add some echo statements to print out the path, but I'm not sure why I face this error!
This is probably because the target folder is not in "./" folder. which can be because it's ignored by .dockerignore file or the build context is not pointing to the parent folder of the target folder.
In case you are not familiar with build context, it's explained here
I am trying to create a TPM-based unlock script using tpm2-tools with instructions from Tevora Secure boot tpm2. I have set up the key, loaded it with cryptsetup luksAddKey secret.bin, then tested it using tpm2_unlock -c 0x81000000 --auth pci:sha1:0,2,3,7 and returns the value of secret.bin. For extra measures, to make sure it works, I loaded secret.bin into "/etc/crypttab", ran # update-initramfs -u -k all, and rebooted. Upon reboot, the system unlocked.
I copied over the following code into "/etc/initramfs-tools/hooks/tpm2"
#!/bin/sh -e
if [ "$1" = "prereqs" ]; then exit 0; fi
. /usr/share/initramfs-tools/hook-functions
copy_exec /usr/local/bin/tpm2_unseal
copy_exec /usr/local/lib/libtss2-tcti-device.so
I appended my etc/crypttab from cryptname UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks to cryptname UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks,keyscript=/usr/local/bin/passphrase-from-tpm
I rewrote the following script because the tpm2-tools command was outdated, edited in the new command, and stored it in /usr/local/bin/passphrase-from-tpm:
#!/bin/sh
set -e
echo "Unlocking via TPM" >&2
export TPM2TOOLS_TCTI="device:/dev/tpm0"
/usr/local/bin/tpm2_unseal -c 0x81000000 --auth pcr:sha1:0,2,3,7
if [ $? -eq 0 ]; then
exit
fi
/lib/cryptsetup/askpass "Unlocking the disk fallback $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)\nEnter passphrase: "
I ran # update-initramfs -u -k all then rebooted. In reboot, I get the following error: /lib/cryptsetup/scripts/passphrase-from-tpm: line 5: /usr/local/bin/tpm2_unseal: not found
I have tried many times to edit passphrase-from-tpm unsuccessfully, including:
Moving both passphrase-from-tpm into "/boot/efi/EFI/BOOT/" and referencing crypttab to that file
Modifying passphrase-from-tpm to use a relative file path to tpm_unseal
Before I figured out how to create a backup linux boot using:
objcopy \
--add-section .osrel=/etc/os-release --change-section-vma .osrel=0x20000 \
--add-section .cmdline=cmdline.txt --change-section-vma .cmdline=0x30000 \
--add-section .linux="/boot/vmlinuz" --change-section-vma .linux=0x40000 \
--add-section .initrd="/boot/initrd.img" --change-section-vma .initrd=0x3000000 \
/usr/lib/systemd/boot/efi/linuxx64.efi.stub /boot/EFI/BOOT/BOOT_RECX64.EFI
I would be locked out of the system completely because of the error and had to reinstall Ubuntu about 40 times. I have suffered a lot and want to quit but I am too stubborn to throw in the flag.
just copy tpm2_unseal to /usr/local/bin/
I'm trying to make a working setup following basically those instructions, plus a few others I have found. While not working 100% yet, check that both /etc/initramfs-tools/hooks/tpm2 and /usr/local/bin/passphrase-from-tpm are marked executable (sudo chmod ug+x $filename).
After making the initramfs, you can run the following to ensure that the TPM related files are actually in the image. Replace the path in the filename by whatever update-initramfs said it was generating:
$ lsinitramfs /boot/initrd.img-5.0.0-37-generic | egrep "(tpm|libtss)"
lib/cryptsetup/scripts/passphrase-from-tpm
lib/modules/5.0.0-37-generic/kernel/crypto/asymmetric_keys/tpm_key_parser.ko
lib/modules/5.0.0-37-generic/kernel/crypto/asymmetric_keys/asym_tpm.ko
lib/udev/rules.d/tpm-udev.rules
usr/local/lib/libtss2-sys.so.0
usr/local/lib/libtss2-mu.so.0
usr/local/lib/libtss2-sys.so.0.0.0
usr/local/lib/libtss2-tcti-device.so
usr/local/lib/libtss2-tcti-device.so.0
usr/local/lib/libtss2-tcti-device.so.0.0.0
usr/local/lib/libtss2-mu.so.0.0.0
usr/local/bin/tpm2_unseal
Additionally, I have modified /usr/local/bin/passphrase-from-tpm to the following:
#!/bin/sh
TPM_DEVICE=/dev/tpm0
TPM_REGISTER=0x81000001
TPM_SEAL_POLICY=sha256:0,2,4,7
export TPM2TOOLS_TCTI="device:$TPM_DEVICE"
if [ "$CRYPTTAB_TRIED" -eq 0 ]; then
echo "Unlocking via TPM" >&2
/usr/local/bin/tpm2_unseal -H $TPM_REGISTER -L $TPM_SEAL_POLICY
UNSEAL_STATUS=$?
echo "Unseal status $UNSEAL_STATUS" >&2
if [ $UNSEAL_STATUS -eq 0 ]; then
exit
fi
else
echo "TPM unlocking previously failed for $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)" >&2
/lib/cryptsetup/askpass "Enter passphrase for $CRYPTTAB_SOURCE ($CRYPTTAB_NAME): "
fi
Note that the command line options to tpm2_unseal are for the 3.x versions of tpm2-tools. If you're using another version, you might need to update the options.
I pulled out various bits into variables at the top of the file. Modify TPM_REGISTER and TPM_SEAL_POLICY to match how you created the TPM object. set -e was removed since if any command failed, the whole script would exit, preventing the askpass fallback from ever running if tpm2_unseal failed.
Additionally, I noticed that if the script fails for some reason, systemd will attempt to run it again. If the secret in the TPM doesn't match the LUKS key, this will render the system unbootable, since the unseal succeeds, but unlocking fails, and systemd will run the script again.
Looking at the man page for crypttab, I discovered that one of the environment variables provided to the keyscript is CRYPTTAB_TRIED which is the number of tries it has attempted to unlock the volume. If CRYPTTAB_TRIED is 0, it'll attempt to use the TPM, as shown by this test (Running as non-root, so accessing the TPM device fails):
$ export CRYPTTAB_SOURCE=some_device
$ export CRYPTTAB_NAME=some_device_name
$ export CRYPTTAB_TRIED=0
$ ./passphrase-from-tpm
Unlocking via TPM
ERROR:tcti:src/tss2-tcti/tcti-device.c:440:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Permission denied
ERROR: tcti init allocation routine failed for library: "device" options: "/dev/tpm0"
ERROR: Could not load tcti, got: "device"
Unseal status 1
When it tries running the script again, CRYPTTAB_TRIED will be greater than 0, making it display the password prompt instead:
$ export CRYPTTAB_TRIED=1
$ ./passphrase-from-tpm
TPM unlocking previously failed for some_device (some_device_name)
Enter passphrase for some_device (some_device_name):
Hopefully this is still of use to you, and helpful to anyone else trying to get the house of cards that is disk encryption with a TPM on Linux working.
I was new to concourse, and set up the environment in my centos7.6 like below.
$ wget https://concourse-ci.org/docker-compose.yml
$ docker-compose up -d
Then login by `fly --target example login --team-name main --concourse-url http://192.168.77.140:8080/ -u test -p test`
I can see below.
[root#centostest ~]# fly targets
name url team expiry
example http://192.168.77.140:8080 main Sun, 16 Jun 2019 02:23:48 UTC
I used below yaml.xml named with 2.yaml
---
resources:
- name: my-git-repo
type: git
source:
uri: https://github.com/ruanbekker/concourse-test
branch: basic-helloworld
jobs:
- name: hello-world-job
public: true
plan:
- get: my-git-repo
- task: task_print-hello-world
file: my-git-repo/ci/task-hello-world.yml
Then I run below commands step by step.
fly -t example sp -c 2.yaml -p pipeline-01
fly -t example up -p pipeline-01
fly -t example tj -j pipeline-01/hello-world-job --watch
But i just hang on there , no useful response like below.
[root#centostest ~]# fly -t example tj -j pipeline-01/hello-world-job --watch
started pipeline-01/hello-world-job #3
Theoretically， it should print something like below.
Cloning into '/tmp/build/get'...
Fetching HEAD
292c84b change task name
initializing
running echo hello world
hello world
succeeded
Where I did wrong? thanks.
welcome to Concourse!
One thing that can be confusing when starting with Concourse is understanding when Concourse detects that the pipeline has changed and what happens if the pipeline is one file or multiple files.
Your pipeline (as the majority of real-world pipelines) is "nested": main pipeline file 2.yaml refers to a task file named my-git-repo/ci/task-hello-world.yml
What sets Concourse apart from other CI systems is that:
the main pipeline file (2.yaml) can reside everywhere, also in a different repository.
Due to 1, Concourse is unable to detect a change to the main pipeline file, you have to tell Concourse that the file has changed, either with fly set-pipeline or with automatic means such as the concourse-pipeline-resource.
So the following errors happen often:
Changing the main pipeline file, committing and pushing, and expecting Concourse to pick up the change. Missing: you have to do fly set-pipeline
Once doing fly set-pipeline becomes second nature, you can stumble upon the opposite error: Change both the main pipeline file and the nested task file, not pushing, doing set-pipeline. In this case, the only changes picked up by Concourse will be the ones to the main pipeline file, not to the task file. Missing: commit and push.
From the description of your problem, I have the feeling that it is a mixture of the gotchas I mentioned.
I need to run a script on a target OS built by Yocto.
This script needs to be ran as part of the install and thus must be ran only once (either after the entire OS install or on the first boot). It cannot be ran on the host system, as it depends on the hardware IO which exists only on the target.
An additional, minor, constraint is that the rootfs is mounted as read only, but I guess that can be avoided by having the script re-mount as rw and again remount as r after the execution or something along those lines.
Any help is appreciated.
I ended up doing what shibley had written. Here's a detailed howto:
Create a new layer
Put the desired layer wherever your other layers are. Mine are in stuff directory, next to the build directory.
Make the following files/directories:
meta_mylayer
├── conf
│ └── layer.conf
└── recipes-core
└── mylayer-initscript
├── initscript.bb
└── files
├── initscript.service
└── initscript.sh
meta_mylayer is the name of your new layer.
Let's define the layer in conf/layer.conf and tell it where to search for the recipes:
BBPATH .= ":${LAYERDIR}"
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
BBFILE_COLLECTIONS += "meta-mylayer"
BBFILE_PATTERN_meta-mylayer := "^${LAYERDIR}/"
BBFILE_PRIORITY_meta-mylayer = "99"
The recipes are defined by the name of the .bb file. This layer only has one recipe, named initscript.
initscript.bb contains the recipe information. The following recipe will add our initscript service and put the actual install script, initscript.sh, into /usr/sbin/
SUMMARY = "Initial boot script"
DESCRIPTION = "Script to do any first boot init, started as a systemd service which removes itself once finished"
LICENSE = "CLOSED"
PR = "r3"
SRC_URI = " \
file://initscript.sh \
file://initscript.service \
"
do_compile () {
}
do_install () {
install -d ${D}/${sbindir}
install -m 0755 ${WORKDIR}/initscript.sh ${D}/${sbindir}
install -d ${D}${systemd_unitdir}/system/
install -m 0644 ${WORKDIR}/initscript.service ${D}${systemd_unitdir}/system
}
NATIVE_SYSTEMD_SUPPORT = "1"
SYSTEMD_PACKAGES = "${PN}"
SYSTEMD_SERVICE_${PN} = "initscript.service"
inherit allarch systemd
install -d will create any directories needed for the specified path, while install -m 0644 will copy the specified file with 644 permissions. ${D} is the destination directory, by default it's ${WORKDIR}/image
Create the systemd service definition
I won't go into much details about how systemd works, but will rather paste the service definition:
[Unit]
Description=start initscript upon first boot
[Service]
Type=simple
ExecStart=/bin/sh -c 'sleep 5 ; /usr/sbin/initscript.sh'
Do note the script location at /usr/sbin/ - that's where it will be copied by the last line of our do_install function above.
Lastly, our initscript.sh script itself:
#!/bin/sh
logger "starting initscript"
# do some work here. Mount rootfs as rw if needed.
logger "initscript work done"
#job done, remove it from systemd services
systemctl disable initscript.service
logger "initscript disabled"
Register the layer
We need to register our new layer, so that bitbake knows it's there.
Edit the build/conf/bblayers.conf file and add the following line to the BASELAYERS variable:
${TOPDIR}/../stuff/meta-mylayer \
Now that the bitbake recognizes our layer, we need to add our recipe to the image.
Edit the build/conf/local.conf and add the initscript recipe to the IMAGE_INSTALL_append variable. Here's how it looks like when added next to the python.
IMAGE_INSTALL_append = " python initscript"
Run the build
Run the build like you usually do. For example:
bitbake angstrom-lxde-image
After you install the build and boot for the first time, your initscript.sh will be executed.
The basic approach is to write a systemd service. The service can be enabled by default as defined in the yocto recipe systemd configuration. The script or application evoked by the service will disable the service when the script/application completes - ie. systemctl disable foo. Therefore, the service will not run in future boots.
As you mentioned, the rootfs will require mounting as rw for this to work.
Thanks, this helped out. I needed to add
[Install]
WantedBy=multi-user.target
to the initscript.service to get it working
A simple solution is to use a package post/install script that stops itself running at rootfs time (exit 1 if $D is set). This will result in it running at first boot. Yes, the script will need to remount the root fs.
Besides, I don't know how to address the issue that rootfs is mounted as read-only, you can use pkg_postinst_ontarget_${PN}
Add this to one of your recipes:
pkg_postinst_ontarget_${PN}() {
#!/bin/bash
// bash script you want to run
echo Post Install Script Test > /dev/ttyS1
}
${PN} will be replaced with the package name the recipes corresponds to.
The script will be run only once, on the first boot on the target machine as a post-install script of the package.