So I got locked out of my Kubernetes instance, presumably due to a cert expiration. It was created with k3sup, by someone with a lot more Kubernetes experience than me.
To dig into the issue, I used AWS session manager to connect to the instance. When I ran sudo kubectl get pods -A from within the instance, I got the same error as I got from my local machine:
error: You must be logged in to the server (Unauthorized)
I then ran sudo systemctl restart k3s to restart the kubernetes, and which supposedly rotates the certs. Now kubectl commands work from within the container, which is great, but still not from my local machine.
If this did rotate the cert as I assume, I think I need the new public key for my local ~/.kube/config. Where do I get this?
I found the updated certs in the kube config at /etc/rancher/k3s/k3s.yaml
Background: Something happened yesterday which led to usage of kubectl commands rendering my macbook unusable ( network/internet breaking and freezing all applications).
I fixed it by using
sriramkumar#Srirams-MacBook-Pro ~ % brew link --overwrite kubernetes-cli
Linking /usr/local/Cellar/kubernetes-cli/1.25.0... 224 symlinks created.
sriramkumar#Srirams-MacBook-Pro ~ % ls -al /usr/local/bin/kubectl
lrwxr-xr-x 1 sriramkumar admin 43 8 Sep 23:21 /usr/local/bin/kubectl -> ../Cellar/kubernetes-cli/1.25.0/bin/kubectl
And today I discovered it is helm that is broken, so eg. when I use `helm ls -a`
It is doing the exact same thing ( network/internet breaking and freezing all applications)
The error I'm getting is
sriramkumar#Srirams-MacBook-Pro ~ % helm ls -a
Error: Kubernetes cluster unreachable: Get "https://something.ibm.com:25628/version?timeout=32s": dial tcp: i/o timeout
I have uninstalled helm, reinstalled set KUBECONFIG=/etc/rancher/k3s/k3s.yaml ( from the internet) but it's not solving the issue. Any help is much appreciated
I have a kubernetes Cronjob that performs some backup jobs, and the backup files needs to be uploaded to a bucket. The pod have the service account credentials mounted inside the pod at /var/run/secrets/kubernetes.io/serviceaccount, but how can I instruct gsutil to use the credentials in /var/run/secrets/kubernetes.io/serviceaccount?
lrwxrwxrwx 1 root root 12 Oct 8 20:56 token -> ..data/token
lrwxrwxrwx 1 root root 16 Oct 8 20:56 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 13 Oct 8 20:56 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 31 Oct 8 20:56 ..data -> ..2018_10_08_20_56_04.686748281
drwxr-xr-x 2 root root 100 Oct 8 20:56 ..2018_10_08_20_56_04.686748281
drwxrwxrwt 3 root root 140 Oct 8 20:56 .
drwxr-xr-x 3 root root 4096 Oct 8 20:57 ..
The short answer is that the token there is not in a format that gsutil knows how to use, so you can't use it. You'll need a JSON keyfile, as mentioned in the tutorial here (except that you won't be able to use the GOOGLE_APPLICATION_CREDENTIALS environment variable):
https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform
Rather than reading from the GOOGLE_APPLICATION_CREDENTIALS environment variable, Gsutil uses Boto configuration files to load credentials. The common places that it knows to look for these Boto config files are /etc/boto.cfg and $HOME/.boto. Note that the latter value changes depending on the user running the command ($HOME expands to different values for different users); since cron jobs usually run as a different user than the one who set up the config file, I wouldn't recommend relying on this path.
So, on your pod, you'll need to first create a Boto config file that references the keyfile:
# This option is only necessary if you're running an installation of
# gsutil that came bundled with gcloud. It tells gcloud that you'll be
# managing credentials manually via your own Boto config files.
$ gcloud config set pass_credentials_to_gsutil False
# Set up your boto file at /path/to/my/boto.cfg - the setup will prompt
# you to supply the /path/to/your/keyfile.json. Alternatively, to avoid
# interactive setup prompts, you could set up this config file beforehand
# and copy it to the pod.
$ gsutil config -e -o '/path/to/my/boto.cfg'
And finally, whenever you run gsutil, you need to tell it where to find that Boto config file which references your JSON keyfile (and also make sure that the user running the command has permission to read both the Boto config file and the JSON keyfile). If you wrote your Boto config file to one of the well-known paths I mentioned above, gsutil will attempt to find it automatically; if not, you can tell gsutil where to find the Boto config file by exporting the BOTO_CONFIG environment variable in the commands you supply for your cron job:
export BOTO_CONFIG=/path/to/my/boto.cfg; /path/to/gsutil cp <src> <dst>
Edit:
Note that GCE VM images come with a pre-populated file at /etc/boto.cfg. This config file tells gsutil to load a plugin that allows gsutil to contact the GCE metadata server and fetch auth tokens (corresponding to the default robot service account for that VM) that way. If your pod is able to read the host VM's /etc/boto.cfg file, you're able to contact the GCE metadata server, and you're fine with operations being performed by the VM's default service account, this solution should work out-of-the-box.
Note that your Kubernetes Service Account is different from your Google Cloud Storage service account.
gsutil uses the boto config so you can mount a Kubernetes secret under /etc/boto.cfg or ~/.boto
You can authenticate with GCP using a token or a service account. You can generate a token using gsutil config -f you can generate service account credentials using gsutil config -e. It will generate a ~/.boto file and then you can mount that as Kubernetes secret on your pods.
More information here.
We currently have a docker registry setup, that has security. Normally, in order to access it, from a developer's perspective, I have to do a long with the docker login --username=someuser --password=somepassword --email user#domain.com https://docker-registry.domain.com.
However, since I am currently trying to do an automatized deployment of a docker container in the cloud, one of the operations, which is the docker pull command, fails because the login was not performed (it works if I add the login in the template, but that's bad).
I was suggested to use the certificate to allow the pull from being done (.crt file). I tried installing the certificate using the steps explained here: https://www.linode.com/docs/security/ssl/ssl-apache2-centos
But it does not seem to work, I still have to do a manual login in order to be able to perform my docker pull from the registry.
Is there a way I can replace the login command by the use of the certificate?
As I see, it's wrong URL for SSL authentication between docker server and private registry server.
You can follow this:
Running a domain registry
While running on localhost has its uses, most people want their registry to be more widely available. To do so, the Docker engine requires you to secure it using TLS, which is conceptually very similar to configuring your web server with SSL.
Get a certificate
Assuming that you own the domain myregistrydomain.com, and that its DNS record points to the host where you are running your registry, you first need to get a certificate from a CA.
Create a certs directory:
mkdir -p certs
Then move and/or rename your crt file to: certs/domain.crt, and your key file to: certs/domain.key.
Make sure you stopped your registry from the previous steps, then start your registry again with TLS enabled:
docker run -d -p 5000:5000 --restart=always --name registry \
-v `pwd`/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2
You should now be able to access your registry from another docker host:
docker pull ubuntu
docker tag ubuntu myregistrydomain.com:5000/ubuntu
docker push myregistrydomain.com:5000/ubuntu
docker pull myregistrydomain.com:5000/ubuntu
Gotcha
A certificate issuer may supply you with an intermediate certificate. In this case, you must combine your certificate with the intermediate's to form a certificate bundle. You can do this using the cat command:
cat domain.crt intermediate-certificates.pem > certs/domain.crt
There seem to be many questions on StackOverflow about this but unfortunately nothing has worked for me.
I'm getting a 502 bad gateway on nginx, and the following on the logs: connect() to ...myproject.sock failed (13: Permission denied) while connecting to upstream
I'm running wsgi and nginx on ubuntu, and I've been following this guide from Digital Ocean. I apparently configured wsgi correctly since uwsgi -s myproject.sock --http 0.0.0.0:8000 --module app --callable app worked, but I keep getting the nginx permission denied error and I have no idea why:
After coming across this question and this other one, I changed the .ini file and added the chown-socket, chmod-socket, uid and gid parameters (also tried just setting the first two, either or, and a couple of different permission settings --and even the most permissive didn't work).
This one seemed promising, but I don't believe selinux is installed on my Ubuntu (running sudo apt-get remove selinux gives "Package 'selinux' is not installed, so not removed" and find / -name "selinux" doesn't show anything). Just in case, though, I tried what this post recommended as well. Uninstalling apparmor (sudo apt-get install apparmor) didn't work either.
Every time I make a change, I run sudo service nginx restart, but I only see the 502 Gateway Error (and the permission denied error when I read the logs).
This is is my nginx configuration file:
server {
listen 80;
server_name 104.131.110.156;
location / {
include uwsgi_params;
uwsgi_pass unix:/home/user/myproject/web_server/myproject.sock;
}
}
.conf file:
description "uWSGI server instance configured to serve myproject"
start on runlevel [2345]
stop on runlevel [!2345]
setuid user
setgid www-data
env PATH=/root/.virtualenvs/my-env/bin
chdir /home/user/myproject/web_server
exec uwsgi --ini /home/user/myproject/web_server/myproject.ini
.ini file:
[uwsgi]
module = wsgi
master = true
processes = 5
socket = /home/user/myproject/web_server/myproject.sock
chown-socket=www-data:www-data
chmod-socket = 664
uid = www-data
gid = www-data
vacuum = true
die-on-term = true
(If it helps, these are the specs of my Digital Ocean machine: Linux 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux)
Please let me know if there's anything I can do, and thank you very much.
After following all the advice in this thread I was still getting permission errors. The finally missing piece was to correct the nginx user in the /etc/nginx/nginx.conf file:
# old: user nginx;
user www-data;
I also followed that tutorial and ran into the same issue. After quite a bit of trial and error, the following steps allowed me to run uWSGI and nginx successfully:
My nginx.config file:
server {
listen 80;
server_name localhost;
location / { try_files #yourapplication; }
location #yourapplication; {
include uwsgi_params;
uwsgi_pass unix:/PATH_TO_PROJECT/PROJECT.sock;
}
}
My .ini file wasn't working very well, so I decided to take advantage of uWSGI's extensive arguments that are available. Here's what I used:
uwsgi -s /PATH_TO_PROJECT/PROJECT.sock -w wsgi:app -H /PATH_TO_PROJECT/venv --http-processes=4 --chmod-socket=666 --master &
Where:
-s /PATH_TO_PROJECT/PROJECT.sock = the location of my .sock file
-w wsgi:app = the location of my wsgi.py file and app being the name of my Flask object
-H /PATH_TO_PROJECT/venv = the location of my virtual environment
--http-processes=4 = the number of http processes for uWSGI to create
--chmod-socket=666 = the permissions to set on the socket
--master = allow uWSGI to run with its master process manager
& = run uWSGI in the background
To summarize what others have said to solve permission denied error in nginx (which you can look into /var/log/nginx/error.log is usually due to the following:
you are writing .sock file at a place nginx does not have permission
SELinux is causing the problem
To solve 1: First, don't write .sock file at /tmp as suggested here server fault answer because different services see different /tmp in fedora. You can write at some place such as ~/myproject/mysocket.sock. The nginx user must have access to our application directory in order to access the socket file there. By default, CentOS locks down each user's home directory very restrictively, so we will add the nginx user to our user's group so that we can then open up the minimum permissions necessary to grant access.
You can add the nginx user to your user group with the following command. Substitute your own username for the user in the command:
sudo usermod -a -G $USER nginx
Now, we can give our user group execute permissions on our home directory. This will allow the Nginx process to enter and access content within:
chmod 710 /path/to/project/dir
If the permission denied error is still there:
then the hack sudo setenforce 0 will do the trick.
The path: unix:/PATH_TO_PROJECT/PROJECT.sock should be placed in /tmp this fixed my problem.
(13: Permission denied)
This indicates that Nginx was unable to connect to the uWSGI socket because of permissions problems. Usually, this happens when the socket is being created in a restricted environment or if the permissions were wrong. While the uWSGI process is able to create the socket file, Nginx is unable to access it.
This can happen if there are limited permissions at any point between the root directory (/) the socket file. We can see the permissions and ownership values of the socket file and each of its parent directories by passing the absolute path to our socket file to the namei command:
namei -nom /PATH_TO_YOUR_SOCKET_FILE/YOUR_SOCKET.sock
The output should be similar to this (Your case might have different folder name)
f: /run/uwsgi/firstsite.sock
drwxr-xr-x root root /
drwxr-xr-x root root run
drwxr-xr-x sammy www-data uwsgi
srw-rw---- sammy www-data firstsite.sock
The output displays the permissions of each of the directory components. By looking at the permissions (first column), owner (second column) and group owner (third column), we can figure out what type of access is allowed to the socket file.
In the above example, each of the directories leading up to the socket file have world read and execute permissions (the permissions column for the directories end with r-x instead of ---). The www-data group has group ownership over the socket itself. With these settings, Nginx process should be able to access the socket successfully.
If any of the directories leading up to the socket are not owned by the www-data group or do not have world read and execute permission, Nginx will not be able to access the socket. Usually, this means that the configuration files have a mistake.
So you fix this issue but giving all the upper folder the permission using this command:
chmod 755 directory_name
I know it is late, but to help others overcome the issue faster, I have posted this answer. Hope it helps, good luck.
This may happen if user www-data may not have the permission to create new
socket in the given path, so use root user.
Relpace user www-data to user root in nginx.conf;
ex: #nginx.conf
#user www-data;
user root;
worker_processes auto;
pid /var/run/nginx.pid;
.............
If you have tested all the permissions and it is still not working then maybe SELinux is enabled, this will cause the same behaviour.
Run getenforce and if the result is Enforcing then that will not help.
Quick fix is to disable it, setenforce 0 but a restart is required.
2 things helped me
I had correct configuration in nginx I also was seeing /tmp/wsgi.sock with my two eyes in the folder but there was still permission denied or directory not exists:
In file /lib/systemd/system/nginx.service set PrivateTmp=false and restart nginx (do not forget systemctl daemon reload to refresh config)
Run command setenforce 0
Bonus material:
/usr/local/bin/uwsgi --chdir /home/biohazard/myproject -s /tmp/wsgi.sock -w api:app --chmod-socket=777 --master --thunder-lock --http-processes=2 where in api:app api stands for /home/biohazard/myproject/api.py
And
location = /api { rewrite ^ /api/; }
location /api { try_files $uri #api; }
location #api {
include uwsgi_params;
uwsgi_pass unix:/tmp/wsgi.sock;
}
Where nginx is serving my http://example.org/api endpoint only
Check user field on the first line in nginx.conf file. By default it is www-data. Change the name to user root in nginx.conf file if you logged in as root.
I Am Also Getting Same Issue While Deploying Flask Using Nginx And Gunicorn.
I Solved This Issue By putting .Sock file in /temp folder.
There are many things that can cause this particular error, in my case it was the ownership of my PROJECT.socket file that caused it.
Instead of:
srwxr-xr-x 1 yourusername yourusername 0 Nov 4 22:32 PROJECT.sock
it should be: srwxr-xr-x 1 www-data www-data 0 Nov 4 22:32 PROJECT.sock
Just run sudo chown www-data:www-data PROJECT.sock and that's it.