Run Daemonset
kubectl create -f test-daemon.yaml --validate=false
Error
Error from server: error when creating "test-daemon.yaml": the server could not find the requested resource (post daemonsets.extensions)
Config
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=network-online.target etcd2.service generate-serviceaccount-key.service
After=network-online.target etcd2.service generate-serviceaccount-key.service
[Service]
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/bin/mkdir -p /opt/bin
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-apiserver -z /opt/bin/kube-apiserver https://storage.googleapis.com/kubernetes-release/release/v1.0.1/bin/linux/amd64/kube-apiserver
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-apiserver
ExecStartPre=/opt/bin/wupiao 127.0.0.1:2379/v2/machines
ExecStart=/opt/bin/kube-apiserver \
--service_account_key_file=/opt/bin/kube-serviceaccount.key \
--service_account_lookup=false \
--admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \
--runtime_config=api/v1,extensions/v1beta1=true,extensions/v1beta1/daemonsets=true \
--allow_privileged=true \
--insecure_bind_address=0.0.0.0 \
--insecure_port=3001 \
--kubelet_https=true \
--secure_port=6443 \
--service-cluster-ip-range=10.100.0.0/16 \
--etcd_servers=http://127.0.0.1:2379 \
--public_address_override=${COREOS_PRIVATE_IPV4} \
--logtostderr=true
Restart=always
RestartSec=10
Added config
--runtime_config=api/v1,extensions/v1beta1=true,extensions/v1beta1/daemonsets=true
ReplicationController
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
app: test
name: test
spec:
template:
metadata:
labels:
app: test
spec:
containers:
name: test
image: 192.168.1.3:4000/test
ports:
- containerPort: 80
Try removing the schema cache: rm -rf /tmp/kubectl.schema
Related
I am trying to set up a load test for an endpoint. This is what I have followed so far:
Dockerfile
FROM python:3.8
# Add the external tasks directory into /tasks
WORKDIR /src
ADD requirements.txt .
RUN pip install --no-cache-dir --upgrade locust==2.10.1
ADD run.sh .
ADD load_test.py .
ADD load_test.conf .
# Expose the required Locust ports
EXPOSE 5557 5558 8089
# Set script to be executable
RUN chmod 755 run.sh
# Start Locust using LOCUS_OPTS environment variable
CMD ["bash", "run.sh"]
# Modified from:
# https://github.com/scrollocks/locust-loadtesting/blob/master/locust/docker/Dockerfile
run.sh
#!/bin/bash
LOCUST="locust"
LOCUS_OPTS="--config=load_test.conf"
LOCUST_MODE=${LOCUST_MODE:-standalone}
if [[ "$LOCUST_MODE" = "master" ]]; then
LOCUS_OPTS="$LOCUS_OPTS --master"
elif [[ "$LOCUST_MODE" = "worker" ]]; then
LOCUS_OPTS="$LOCUS_OPTS --worker --master-host=$LOCUST_MASTER_HOST"
fi
echo "${LOCUST} ${LOCUS_OPTS}"
$LOCUST $LOCUS_OPTS
# Copied from
# https://github.com/scrollocks/locust-loadtesting/blob/master/locust/docker/locust/run.sh
This is how I have written the load test locust script:
import json
from locust import HttpUser, constant, task
class CategorizationUser(HttpUser):
wait_time = constant(1)
#task
def predict(self):
payload = json.dumps(
{
"text": "Face and Body Paint washable Rubies Halloween item 91#385"
}
)
_ = self.client.post("/predict", data=payload)
I am invoking that with a configuration:
locustfile = load_test.py
headless = false
users = 1000
spawn-rate = 1
run-time = 5m
host = IP
html = locust_report.html
So, after building and pushing the Docker image and creating a k8s cluster on GKE, I am deploying it. This is how the deployment.yaml looks like:
# Copied from
# https://github.com/scrollocks/locust-loadtesting/blob/master/locust/kubernetes/templates/deployment.yaml
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: locust-master-deployment
labels:
name: locust
role: master
spec:
replicas: 1
selector:
matchLabels:
name: locust
role: master
template:
metadata:
labels:
name: locust
role: master
spec:
containers:
- name: locust
image: gcr.io/PROJECT_ID/IMAGE_URI
imagePullPolicy: Always
env:
- name: LOCUST_MODE
value: master
- name: LOCUST_LOG_LEVEL
value: DEBUG
ports:
- name: loc-master-web
containerPort: 8089
protocol: TCP
- name: loc-master-p1
containerPort: 5557
protocol: TCP
- name: loc-master-p2
containerPort: 5558
protocol: TCP
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: locust-worker-deployment
labels:
name: locust
role: worker
spec:
replicas: 2
selector:
matchLabels:
name: locust
role: worker
template:
metadata:
labels:
name: locust
role: worker
spec:
containers:
- name: locust
image: gcr.io/PROJECT_ID/IMAGE_URI
imagePullPolicy: Always
env:
- name: LOCUST_MODE
value: worker
- name: LOCUST_MASTER
value: locust-master-service
- name: LOCUST_LOG_LEVEL
value: DEBUG
After deployment, I am exposing the required ports like so:
kubectl expose pod locust-master-deployment-f9d4c4f59-8q6wk \
--type NodePort \
--port 5558 \
--target-port 5558 \
--name locust-5558
kubectl expose pod locust-master-deployment-f9d4c4f59-8q6wk \
--type NodePort \
--port 5557 \
--target-port 5557 \
--name locust-5557
kubectl expose pod locust-master-deployment-f9d4c4f59-8q6wk \
--type LoadBalancer \
--port 80 \
--target-port 8089 \
--name locust-web
The cluster and the nodes provision successfully. But the moment I hit the IP of locust-web, I am getting:
Any suggestions on how to resolve the bug?
Since you are exposing your pods and you are trying to access to them outside the cluster (your web application), you have to port-forward them or add an Ingress in order to access to your locust pods.
My first approach will be trying to ping or send requests to your locust pods with a simple port-forward.
More infos about the port forwarding here.
Probably the environment variables set by k8s is colliding with locust’s (LOCUST_WEB_PORT specifically). Change your setup so that no containers are named ”locust”.
See https://github.com/locustio/locust/issues/1226 for a similar issue.
I have a job st as post-install in my helm charts which is looks like
apiVersion: batch/v1
kind: Job
metadata:
name: job-hook-postinstall
annotations:
"helm.sh/hook": "post-install"
spec:
template:
spec:
containers:
- name: post-install
image: {{ .Values.image.initContainer.curl }}
imagePullPolicy: IfNotPresent
command:
- '/bin/sh'
- '-c'
- >
PAYLOAD='{"password":"password"}';
curl -X PUT "http://server:{{ .Values.app.Server.service.port }}/users/userA" -H "Authorization: Basic {{ .Values.app.livenessProbe.Server.authorization }}" -H 'Content-Type: application/json' --data-raw $PAYLOAD &&
curl -X PUT "http://server:{{ .Values.app.Server.service.port }}/users/userB" -H "Authorization: Basic {{ .Values.app.livenessProbe.Server.authorization }}" -H 'Content-Type: application/json' --data-raw $PAYLOAD
restartPolicy: OnFailure
terminationGracePeriodSeconds: 0
backoffLimit: 20
completions: 1
which I expect will be executed only during helm install but is getting executed during helm upgrade as well.
Other problem is that it will error out on Error: failed post-install: job failed: BackoffLimitExceeded when I have bumped it to 20
Any thoughts on both problems? Can I do some kind of a loop to check if the connection to the server endpoint is present then execute curls to prevent at least second problem? Will below work?
...
command:
- '/bin/sh'
- '-c'
- >
for i in $(seq 1 300); do nc -zvw1 server {{ .Values.app.Server.service.port }} && exit 0 || sleep 3;
done;
PAYLOAD='{"password":"password"}';'
....
or to execute that job after specific pod deployment?
I would like to specify dependent environment variables on a Cloud Run service.
If the environment variables have been defined in a .env file it would look like this
DATABASE_NAME=my-database
DATABASE_USER=root
DATABASE_PASSWORD=P4SSw0rd!
DATABASE_PORT=5432
DATABASE_HOST="/socket/my-database-socket"
DATABASE_URL="user=${DATABASE_USER} password=${DATABASE_PASSWORD} dbname=${DATABASE_NAME} host=${DATABASE_HOST}"
In this example, DATABASE_URL depends on every other environment variables.
To deploy the service I run the following command:
gcloud run deploy my-service \
--image gcr.io/my-project/my-image:latest \
--region europe-west1 \
--port 80 \
--platform managed \
--allow-unauthenticated \
--set-env-vars 'DATABASE_NAME=my-database' \
--set-env-vars 'DATABASE_USER=root' \
--set-env-vars 'DATABASE_PASSWORD=P4SSw0rd!' \
--set-env-vars 'DATABASE_PORT=5432' \
--set-env-vars 'DATABASE_HOST="/socket/my-database-socket"' \
--set-env-vars 'DATABASE_URL="user=$(DATABASE_USER) password=$(DATABASE_PASSWORD) dbname=$(DATABASE_NAME) host=$(DATABASE_HOST)"'
Here is the created YAML definition of the service (some values are omitted)
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: my-service
spec:
template:
metadata:
name: ...
spec:
containerConcurrency: 80
timeoutSeconds: 300
containers:
- image: ...
ports:
- name: http1
containerPort: 80
env:
- name: DATABASE_NAME
value: my-database
- name: DATABASE_USER
value: root
- name: DATABASE_PASSWORD
value: P4SSw0rd!
- name: DATABASE_HOST
value: /socket/my-database-socket
- name: DATABASE_URL
value: user=$(DATABASE_USER) password=$(DATABASE_PASSWORD) dbname=$(DATABASE_NAME) host=$(DATABASE_HOST)
The problem is that when the service is running, the env vars in DATABASE_URL seem not interpolated.
I read that Kubernetes supports dependent env vars but I can't figure out how to make this run in Cloud Run.
I am wondering if it is supported in Cloud Run in the end.
It's likely this may work in Knative open source (which uses Kubernetes to execute pods) but not on Google Cloud Run (fully hosted), which runs on a proprietary execution engine.
I have three Azure Pipeline agents built on Ubuntu 18.04 images and deployed to a Kubernetes cluster. Agents are running the latest version, 2.182.1, but this problem also happened using 2.181.0.
Executing build pipelines individually works just fine. Build completes successfully every time. But whenever a second pipeline starts while another pipeline is already running, it fails - every time - on the "Checkout" job with the following error:
The working folder U:\azp\agent\_work\1\s is already in use by the workspace ws_1_34;Project Collection Build Service (myaccount) on computer linux-agent-deployment-78bfb76d.
These are three separate and distinct agents running as separate containers. Why would a job from one container be impacting a job running on a different container? Concurrent builds work all day long on my non-container Windows servers.
The container agents are deployed as a standard Kubernetes "deployment" object:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: linux-agent
name: linux-agent-deployment
namespace: pipelines
annotations:
kubernetes.io/change-cause: "update agent image to 20210304 - change from OpenJDK to Oracle Java JDK 11"
spec:
replicas: 3
revisionHistoryLimit: 3
selector:
matchLabels:
app: linux-agent
strategy:
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: linux-agent
spec:
serviceAccountName: sa-aws-azp-pipelineagent
containers:
- name: linux-agent
image: 999999999999.dkr.ecr.us-east-2.amazonaws.com/mgmt/my-linux-agent:20210304
imagePullPolicy: IfNotPresent
env:
- name: AZP_URL
value: https://dev.azure.com/myaccount
- name: AZP_POOL
value: EKS-Linux
- name: AZP_TOKEN
valueFrom:
secretKeyRef:
name: azure-devops
key: agent-token
My build agent containers are pretty straightforward...
FROM ubuntu:18.04
ENV ACCEPT_EULA=y
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN ln -fs /usr/share/zoneinfo/America/Chicago /etc/localtime
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
apt-transport-https \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl4 \
libicu60 \
libunwind8 \
netcat \
dnsutils \
wget \
zip \
unzip \
telnet \
ftp \
file \
time \
tzdata \
build-essential \
libc6 \
libgcc1 \
libgssapi-krb5-2 \
liblttng-ust0 \
libssl1.0 \
libstdc++6 \
zlib1g \
apt-utils \
bison \
brotli \
bzip2 \
dbus \
dpkg \
fakeroot \
flex \
gnupg2 \
iproute2 \
lib32z1 \
libc++-dev \
libc++abi-dev \
libgbm-dev \
libgconf-2-4 \
libgtk-3-0 \
libsecret-1-dev \
libsqlite3-dev \
libxkbfile-dev \
libxss1 \
locales \
m4 \
openssh-client \
parallel \
patchelf \
pkg-config \
rpm \
rsync \
shellcheck \
sqlite3 \
ssh \
sudo \
texinfo \
tk \
upx \
xorriso \
xvfb \
xz-utils \
zstd \
zsync \
software-properties-common
### REQUIRED APPLICATIONS
# Amazon Web Services - CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& sudo ./aws/install
# MS SQL Tools (ONE-TIME SETUP OF MICROSOFT REPOSITORY INCLUDED)
RUN curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add - \
&& curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list | sudo tee /etc/apt/sources.list.d/msprod.list \
&& sudo apt-get update && sudo ACCEPT_EULA=Y apt-get install -y mssql-tools unixodbc-dev
# Powershell Global Tool (https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7.1)
RUN sudo apt-get install -y powershell
# .NET Core SDKs (https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu)
# see also (https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/) "Packages"
# SDKs Included: 2.1, 2.2, 3.0, 3.1, 5.0
RUN sudo apt-get install -y dotnet-host \
aspnetcore-store-2.0.0 \
aspnetcore-store-2.0.3 \
aspnetcore-store-2.0.5 \
aspnetcore-store-2.0.6 \
aspnetcore-store-2.0.7 \
aspnetcore-store-2.0.8 \
aspnetcore-store-2.0.9 \
dotnet-hostfxr-2.0.7 \
dotnet-hostfxr-2.0.9 \
dotnet-hostfxr-2.1 \
dotnet-hostfxr-2.2 \
dotnet-hostfxr-3.0 \
dotnet-hostfxr-3.1 \
dotnet-hostfxr-5.0 \
dotnet-runtime-deps-2.1 \
dotnet-runtime-deps-2.2 \
dotnet-runtime-deps-3.0 \
dotnet-runtime-deps-3.1 \
dotnet-runtime-deps-5.0 \
dotnet-targeting-pack-3.0 \
dotnet-targeting-pack-3.1 \
dotnet-targeting-pack-5.0 \
netstandard-targeting-pack-2.1 \
aspnetcore-targeting-pack-3.0 \
aspnetcore-targeting-pack-3.1 \
aspnetcore-targeting-pack-5.0 \
dotnet-runtime-2.1 \
dotnet-runtime-2.2 \
dotnet-runtime-3.0 \
dotnet-runtime-3.1 \
dotnet-runtime-5.0 \
aspnetcore-runtime-2.1 \
aspnetcore-runtime-2.2 \
aspnetcore-runtime-3.0 \
aspnetcore-runtime-3.1 \
aspnetcore-runtime-5.0 \
dotnet-sdk-2.1 \
dotnet-sdk-2.2 \
dotnet-sdk-3.0 \
dotnet-sdk-3.1 \
dotnet-sdk-5.0
# Initialize dotnet
RUN dotnet help
RUN dotnet --info
# Node.js (https://github.com/nodesource/distributions/blob/master/README.md)
RUN curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash - \
&& sudo apt-get install -y nodejs \
&& node --version \
&& npm --version
# Java JDK 11
COPY JDK/ /var/cache/oracle-jdk11-installer-local/
RUN add-apt-repository -y ppa:linuxuprising/java && \
apt-get update && \
echo oracle-java11-installer shared/accepted-oracle-license-v1-2 select true | sudo /usr/bin/debconf-set-selections && \
apt-get install -y oracle-java11-installer-local
ENV JAVA_HOME=/usr/lib/jvm/java-11-oracle \
JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
# Clean package cache
RUN rm -rf /var/lib/apt/lists/* \
&& rm -rf /etc/apt/sources.list.d/*
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh"]
What am I doing wrong?
Solution has been found. Here's how I resolved this for anyone coming across this post:
I discovered a helm chart for Azure Pipeline agents - emberstack/docker-azure-pipelines-agent - and after poking around in the contents, discovered what was staring me in the face the last couple of days: "StatefulSets"
Simple, easy to test, and working well so far. I refactored my k8s manifest as a StatefulSet object and the agents are up and able to run builds concurrently. Still more testing to do, but looking very positive at this point.
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: linux-agent
name: linux-pipeline-agent
namespace: pipelines
annotations:
kubernetes.io/change-cause: "Init 20210304 - Oracle Java JDK 11"
spec:
podManagementPolicy: Parallel
replicas: 3
revisionHistoryLimit: 3
selector:
matchLabels:
app: linux-agent
serviceName: agent-svc
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: linux-agent
spec:
serviceAccountName: sa-aws-azp-pipelineagent
containers:
- name: linux-agent
image: 999999999999.dkr.ecr.us-east-2.amazonaws.com/mgmt/my-linux-agent:20210304
imagePullPolicy: IfNotPresent
env:
- name: AZP_AGENT_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: AZP_URL
value: https://dev.azure.com/myaccount
- name: AZP_POOL
value: EKS-Linux
- name: AZP_TOKEN
valueFrom:
secretKeyRef:
name: azure-devops
key: agent-token
Let's say I have a Pod with 2 containers: App and Database. I want to run a Pod that executes a command in App and then terminates.
I have set up my App container to run that command, and then it succesully runs and terminates which is great. But now my Database container is still running, so the Pod is not marked as complete.
How can I get the Pod to be marked as complete when the App container is completed?
You can make a call to the Kubernetes API server to accomplish this. Consider the following example:
---
apiVersion: v1
kind: Pod
metadata:
name: multi-container-completion
spec:
containers:
- name: long-running-process
image: fbgrecojr/office-hours:so-47848488
command: ["sleep", "1000"]
- name: short-running-process
image: fbgrecojr/office-hours:so-47848488
command: ["sleep", "1"]
lifecycle:
preStop:
exec:
command: ["/pre-stop.sh"]
pre-stop.sh
#!/bin/bash
curl \
-X DELETE \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
https://kubernetes.default.svc.cluster.local/api/v1/namespaces/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/pods/$HOSTNAME
Dockerfile for fbgrecojr/office-hours:so-47848488
FROM centos:latest
COPY pre-stop.sh /
RUN chmod +x /pre-stop.sh
NOTE: I was not able to properly test this because preStop hooks do not seem to be working for my local Minikube setup. In case this issue is not localized to me, the corresponding issue can be tracked here.