I am running Celery in Kubernetes pod. It can't find the server:
ERROR/MainProcess] consumer: Cannot connect to redis://:**#redis-master:6379/1: Error -3 connecting to redis-master:6379. Lookup timed out..
Trying again in 4.00 seconds... (1/100)
If I connect to the very same pod via "kubectl exec -it" and run the command, I succeed:
redis-cli -u redis://:#redis-master:6379/1 keys '*'
(empty list or set)
How can I troubleshoot this problem?
UPDATE 1:
Problem obviously in DNS:
If I set host to domain name:
export REDIS_HOST=redis-master.dev.svc.cluster.local
celery worker --app src
TIMEOUT
If I set host to domain name:
export REDIS_HOST=10.0.13.13
celery worker --app src
OK
Meanwhile:
# dig redis-master.dev.svc.cluster.local
; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> redis-master.dev.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49071
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;redis-master.dev.svc.cluster.local. IN A
;; ANSWER SECTION:
redis-master.dev.svc.cluster.local. 30 IN A 10.0.13.13
;; SERVER: 10.0.0.10#53(10.0.0.10)
Thus, the problem is narrowed down to:
Why Celery doesn't use DNS?
UPDATE 2:
Problem in Python library dnspython.
Version 2.0.0 - bug in resolving
Version 1.16.0 - works like a charm
SOLUTION
pip install dnspython==1.16.0
Answering myself:
This is a dnspython bug.
Solution:
pip install dnspython==1.16.0
Related
EDIT: weird DNS behavior was some kind of transient issue, and now RHEL/podman works the same way as Ubuntu/podman. I can't reproduce the issue, which makes most part (not 100% though) of this question moot.
I am trying to use Podman and docker-compose to create a compose stack with multiple replicas of backend container, and having a hard time with it.
I use Podman because I have to (comes from Red Hat platform), and picked docker-compose because it is familiar and I use in local dev host, too. I know there are alternatives (podman-compose etc). I learned that Podman 4.1 supported Docker Compose so this sounded like a good candidate.
As an example I have docker-compose.yml with one frontend container and 3 backend containers:
version: "3"
services:
frontend:
image: "nginx:latest"
ports:
- "3000:80"
depends_on:
- backend
backend:
image: "tomcat:latest"
ports:
- "8180-8280:8080"
scale: 3
Note: this stack is just an example. It's purpose is only to highlight the networking aspects of multi-replica docker-compose . I could use something else than nginx:latest, and using e.g. Traefik can solve some of the problems...but sometimes you wish to connect directly from one container to a service with multiple container replicas.
Docker & docker-compose (Ubuntu)
Running this on host which has docker and docker-compose is straightforward.
Backend containers get assigned random ports from range 8180-8280.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
98941117708c nginx:latest "/docker-entrypoint.…" 25 seconds ago Up 22 seconds 0.0.0.0:3000->80/tcp, :::3000->80/tcp example-docker-compose-frontend-1
3d749e25eaba tomcat:latest "catalina.sh run" 26 seconds ago Up 23 seconds 0.0.0.0:8193->8080/tcp, :::8193->8080/tcp example-docker-compose-backend-1
854ba8f60cb3 tomcat:latest "catalina.sh run" 26 seconds ago Up 23 seconds 0.0.0.0:8192->8080/tcp, :::8192->8080/tcp example-docker-compose-backend-2
e57e32181e8e tomcat:latest "catalina.sh run" 26 seconds ago Up 23 seconds 0.0.0.0:8194->8080/tcp, :::8194->8080/tcp example-docker-compose-backend-3
Logging into frontend container, service name backend resolves to all 3 backend containers
dig backend
; <<>> DiG 9.16.27-Debian <<>> backend
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31602
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;backend. IN A
;; ANSWER SECTION:
backend. 600 IN A 172.19.0.3
backend. 600 IN A 172.19.0.2
backend. 600 IN A 172.19.0.4
curl backend:8080 works
Podman & docker-compose (RHEL 8)
Podman came preinstalled, I added docker-compose (standalone) and podman-docker:
# curl -SL https://github.com/docker/compose/releases/download/v2.10.2/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
# chmod a+x /usr/local/bin/docker-compose
# sudo yum install podman-docker
And activated rootless podman socket so that podman and docker-compose can talk to each other:
# systemctl --user enable podman.socket
# systemctl --user start podman.socket
# systemctl --user status podman.socket
# export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
I also switched network backend to netavark, DNS did not work without that change
$ podman info |grep -i networkbackend
networkBackend: netavark
1. Port ranges are not supported
Podman does not like ports: - "8180-8280:8080" due to this bug: https://github.com/containers/podman/issues/15111
[+] Running 1/0
⠿ Network example-docker-compose_default Created 0.0s
⠋ Container example-docker-compose-backend-3 Creating 0.0s
⠋ Container example-docker-compose-backend-1 Creating 0.0s
⠋ Container example-docker-compose-backend-2 Creating 0.0s
Error response from daemon: make cli opts(): strconv.Atoi: parsing "8180-8280": invalid syntax
2. Without port range, address already in use conflict
Changed docker-compose.yml to remove port range
ports:
- "8080:8080"
This results in port conflict, all 3 backend containers try to bind to 8080
Catalina.startup.Catalina.start Server startup in [190] milliseconds
Error response from daemon: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use
3. Scale = 1
Let's try with just one backend container. System starts up
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
217c3e726431 docker.io/library/tomcat:latest catalina.sh run 7 seconds ago Up 6 seconds ago 0.0.0.0:8080->8080/tcp example-docker-compose-backend-1
9c3f86676bde docker.io/library/nginx:latest nginx -g daemon o... 6 seconds ago Up 6 seconds ago 0.0.0.0:3000->80/tcp example-docker-compose-frontend-1
DNS from frontend server looks weird. What are all these different backend IP addresses 10.89.0.3 - 10.89.0.12? Only the last of them works when I curl 10.89.0.x. Still, curl backend:8080 works fine?
4. Scale up from command line
I remove ports and scale from docker-compose.yml and start compose stack with scale=3 option:
version: "3"
services:
frontend:
image: "nginx:latest"
ports:
- "3000:80"
depends_on:
- backend
backend:
image: "tomcat:latest"
$ docker-compose up --scale backend=3
[+] Running 4/4
⠿ Container example-docker-compose-backend-2 Recreated 0.3s
⠿ Container example-docker-compose-backend-3 Recreated 0.2s
⠿ Container example-docker-compose-backend-1 Recreated 0.3s
⠿ Container example-docker-compose-frontend-1 Recreated 0.3s
Attaching to example-docker-compose-backend-1, example-docker-compose-backend-2, example-docker-compose-backend-3, example-docker-compose-frontend-1
Now compose stack starts nicely with 3 backend containers
$ docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b81c44246ae7 docker.io/library/tomcat:latest catalina.sh run 22 minutes ago Up 22 minutes ago example-docker-compose-backend-3
814dc5d307f7 docker.io/library/tomcat:latest catalina.sh run 22 minutes ago Up 22 minutes ago example-docker-compose-backend-2
fb0a5090a456 docker.io/library/tomcat:latest catalina.sh run 22 minutes ago Up 22 minutes ago example-docker-compose-backend-1
c0219d7fded4 docker.io/library/nginx:latest nginx -g daemon o... 22 minutes ago Up 22 minutes ago 0.0.0.0:3000->80/tcp example-docker-compose-frontend-1
DNS from frontend has even more entries
# dig backend
; <<>> DiG 9.16.27-Debian <<>> backend
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29526
;; flags: qr rd ad; QUERY: 1, ANSWER: 11, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 410008265c2b0edb (echoed)
;; QUESTION SECTION:
;backend. IN A
;; ANSWER SECTION:
backend. 86400 IN A 10.89.0.3
backend. 86400 IN A 10.89.0.4
backend. 86400 IN A 10.89.0.6
backend. 86400 IN A 10.89.0.7
backend. 86400 IN A 10.89.0.15
backend. 86400 IN A 10.89.0.16
backend. 86400 IN A 10.89.0.18
backend. 86400 IN A 10.89.0.19
backend. 86400 IN A 10.89.0.20
backend. 86400 IN A 10.89.0.21
backend. 86400 IN A 10.89.0.22
And curl backend:8080 does not work (not sure which port I should use now)
Questions
What's going on here?
Can I achieve a setup of 3 backend containers, so that DNS name backend would resolve to those, with podman & docker-compose?
Podman seems to support docker-compose (or vice versa), but only to a degree. Is there some documentation which tells what docker-compose features are supported on Podman, and which are not?
Podman is my container runtime of choice...unless I'm working with docker-compose, in which case I have found it to be "close but not quite" in terms of its docker API support.
However, for what you're trying to do, you could replace Nginx with Traefik, and let Traefik handle the load balancing. Traefik is a dynamic proxy that uses the Docker API and container labelling to discover containers and configure the proxy rules.
For example:
version: "3"
services:
frontend:
image: "docker.io/traefik:v2.8"
ports:
- "3000:80"
- "127.0.0.1:3080:8080"
command:
- --api.insecure=true
- --providers.docker
volumes:
- /run/user/$UID/podman/podman.sock:/var/run/docker.sock
backend:
labels:
traefik.http.routers.backend.rule: Host(`localhost`)
image: "quay.io/larsks/demoserver:latest"
scale: 3
Here we're mapping all requests for Host: localhost to to our backend containers. This is just for the purposes of a demonstration (since I'll be running curl localhost/...); a more realistic configuration would use specific hostnames, or paths, etc. You can read more in the Routing configuration section of Traefik's Docker documentation, and also in the general Router documentation.
With this configuration, we see the following containers running:
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c29de1d137e2 docker.io/library/traefik:v2.8 --api.insecure=tr... 5 seconds ago Up 6 seconds ago 0.0.0.0:3000->80/tcp, 127.0.0.1:3080->8080/tcp demoserver_frontend_1
f4fac24cb494 quay.io/larsks/demoserver:latest /usr/local/bin/st... 5 seconds ago Up 6 seconds ago demoserver_backend_2
d9d388202be2 quay.io/larsks/demoserver:latest /usr/local/bin/st... 5 seconds ago Up 5 seconds ago demoserver_backend_1
5a3e6330739d quay.io/larsks/demoserver:latest /usr/local/bin/st... 5 seconds ago Up 5 seconds ago demoserver_backend_3
And we can see that requests on port 3000 cycle between the available backends. Running this script:
for i in {1..10}; do
curl http://localhost:3000/hostname
done
Produces as output:
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
VS code often starts listening on a random high port on all interfaces.
$ lsb_release -d
Description: Ubuntu 22.04 LTS
$ code --version
1.67.2
c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5
x64
$ sudo ss -ltpn | grep code
LISTEN 0 511 *:33333 *:* users:(("code",pid=75602,fd=52))
In VS code developer tools console:
CRITI Extension 'ms-vsliveshare.vsliveshare' wants API proposal 'notebookDocumentEvents' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
log.ts:313
ERR UnboundLocalError: local variable 'start_index' referenced before assignment: Error: UnboundLocalError: local variable 'start_index' referenced before assignment
at /home/user/.vscode/extensions/ms-python.python-2022.8.0/out/client/extension.js:2:2107873
at /home/user/.vscode/extensions/ms-python.python-2022.8.0/out/client/extension.js:2:2108167
at Immediate.<anonymous> (/home/user/.vscode/extensions/ms-python.python-2022.8.0/out/client/extension.js:2:2108529)
at processImmediate (node:internal/timers:464:21)
log.ts:313
ERR Error: File not found
at g.handleSetInputError (vscode-file://vscode….main.js:2701:95592)
at async g.setInput (vscode-file://vscode….main.js:2701:94832)
at async doSetInput (vscode-file://vscode…p.main.js:2699:6429)
at async doOpenEditor (vscode-file://vscode…p.main.js:2699:4791)
at async d.openEditor (vscode-file://vscode…p.main.js:2699:3783)
at async vscode-file://vscode….main.js:2906:16950
at async bi.openAnything (vscode-file://vscode….main.js:2712:17542)
I have not been able to reproduce whenever the Microsoft Python extension (Python
v2022.8.0) was disabled, so it might be related to that. There is an error in the console related to ms-python, but I do not immediately see how it would lead to a port being opened, especially not on 0.0.0.0.
The following shows nothing:
$ sudo tcpdump -i any port 33333
The question is what causes this port to be opened (and how to avoid this).
In the process explorer of VS code it shows that the process that opened the port is 'extensionHost'.
I tried 30 day trial of dashDB Local. I followed the steps described in the link:
https://www.ibm.com/support/knowledgecenter/en/SS6NHC/com.ibm.swg.im.dashdb.doc/admin/linux_deploy.html
I did not create a node configuration file because mine is a SMP setup.
Logged into my docker hub account and pulled the image.
docker login -u xxx -p yyyyy
docker pull ibmdashdb/local:latest-linux
The pull took 5 minutes or so. I waited for the image download to complete.
Ran the following command. It completed successfully.
docker run -d -it --privileged=true --net=host --name=dashDB -v /mnt/clusterfs:/mnt/bludata0 -v /mnt/clusterfs:/mnt/blumeta0 ibmdashdb/local:latest-linux
ran logs command
docker logs --follow dashDB
This showed dashDB did not start but exited with error code 130
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0f008f8e413d ibmdashdb/local:latest-linux "/usr/sbin/init" 16 seconds ago Exited (130) 1 seconds ago dashDB
#
logs command shows this:
2017-05-17T17:48:11.285582000Z Detected virtualization docker.
2017-05-17T17:48:11.286078000Z Detected architecture x86-64.
2017-05-17T17:48:11.286481000Z
2017-05-17T17:48:11.294224000Z Welcome to dashDB Local!
2017-05-17T17:48:11.294621000Z
2017-05-17T17:48:11.295022000Z Set hostname to <orion>.
2017-05-17T17:48:11.547189000Z Cannot add dependency job for unit systemd-tmpfiles-clean.timer, ignoring: Unit is masked.
2017-05-17T17:48:11.547619000Z [ OK ] Reached target Timers.
<snip>
2017-05-17T17:48:13.361610000Z [ OK ] Started The entrypoint script for initializing dashDB local.
2017-05-17T17:48:19.729980000Z [100209.207731] start_dashDB_local.sh[161]: /usr/lib/dashDB_local_common_functions.sh: line 1816: /tmp/etc_profile-LOCAL.cfg: No such file or directory
2017-05-17T17:48:20.236127000Z [100209.713223] start_dashDB_local.sh[161]: The dashDB Local container's environment is not set up yet.
2017-05-17T17:48:20.275248000Z [ OK ] Stopped Create Volatile Files and Directories.
<snip>
2017-05-17T17:48:20.737471000Z Sending SIGTERM to remaining processes...
2017-05-17T17:48:20.840909000Z Sending SIGKILL to remaining processes...
2017-05-17T17:48:20.880537000Z Powering off.
So it looks like start_dashDB_local.sh is failing at /usr/lib/dashDB_local_common_functions.sh 1816th line? I exported the image and this is the 1816th line of dashDB_local_common_functions.sh
update_etc_profile()
{
local runtime_env=$1
local cfg_file
# Check if /etc/profile/dashdb_env.sh is already updated
grep -q BLUMETAHOME /etc/profile.d/dashdb_env.sh
if [ $? -eq 0 ]; then
return
fi
case "$runtime_env" in
"AWS" | "V1.5" ) cfg_file="/tmp/etc_profile-V15_AWS.cfg"
;;
"V2.0" ) cfg_file="/tmp/etc_profile-V20.cfg"
;;
"LOCAL" ) # dashDB Local Case and also the default
cfg_file="/tmp/etc_profile-LOCAL.cfg"
;;
*) logger_error "Invalid ${runtime_env} value"
return
;;
esac
I also see /tmp/etc_profile-LOCAL.cfg in the image. Did I miss any step here?
I also created /mnt/clusterfs/nodes file ... but it did not help. The same docker run command failed in the same way.
Please help.
I am using x86_64 Fedora25.
# docker version
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-6.gitae7d637.fc25.x86_64
Go version: go1.7.4
Git commit: ae7d637/1.12.6
Built: Mon Jan 30 16:15:28 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-6.gitae7d637.fc25.x86_64
Go version: go1.7.4
Git commit: ae7d637/1.12.6
Built: Mon Jan 30 16:15:28 2017
OS/Arch: linux/amd64
#
# cat /etc/fedora-release
Fedora release 25 (Twenty Five)
# uname -r
4.10.15-200.fc25.x86_64
#
Thanks for bringing this to our attention. I reached out to our developer team. It seems this is happening because inside the container, tmpfs gets mounted on to /tmp and wipes out all the scripts
We have seen this issue and moving to the latest version of docker seems to fix it. Your docker version commands shows it is an older version.
So please install the latest docker version and retry the deployment of dashdb Local and update here.
Regards
Murali
is there a way to get the mesos master end point using a POST/GET call dynamically? Are there any APIs to call zookeeper which in turn returns the mesos master URL?
You should be able to query for the master in zookeeper by querying your zookeeper DNS endpoint (e.g. zookeeper.example.com) with:
zookeeper.example.com:2181/mesos.
You may also query any of your masters in mesos, and the request will automatically get forwarded to the leading master.
Lastly, there's a project called Mesos-DNS that has a DNS endpoint for any of your mesos masters (with master.mesos).
However, the most important point is that you shouldn't be storing that information. There's no reason to assume that the current leading master (or any master for that matter) won't change.
Use zookeeper command-line.
https://github.com/apache/zookeeper/blob/trunk/bin/zkCli.sh
./bin/zkCli.sh
help
# see how to connect to other zookeeper
# default is: localhost:2181
ls /
# [mesos, zookeeper]
ls /mesos
# [json.info_0000000001]
get /mesos/json.info_0000000001
# It will return the Mesos Master's hostname & IP
The result is like this:
{"address":{"hostname":"ced7f7fb79fc","ip":"172.17.0.3","port":5050},"hostname":"ced7f7fb79fc","id":"c8055c4c-3ae0-40e6-a710-5184bcb957a9","ip":50336172,"pid":"master#172.17.0.3:5050","port":5050,"version":"0.26.0"}
With mesos-dns you can simplye use leader.mesos to get the current master.
dig leader.mesos
; <<>> DiG 9.9.5-3ubuntu0.6-Ubuntu <<>> leader.mesos
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8386
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;leader.mesos. IN A
;; ANSWER SECTION:
leader.mesos. 60 IN A 10.100.12.40
;; Query time: 1 msec
;; SERVER: 10.100.14.206#53(10.100.14.206)
;; WHEN: Wed Sep 14 15:31:55 UTC 2016
;; MSG SIZE rcvd: 46
I'm using python. I did a yum install memcached followed by a easy_install python-memcached
I used the simple test program from the Help(memcache). When I wasn't getting the proper answers I threw in some print statements:
[~/test]$ cat m2.py
import memcache
mc = memcache.Client(['127.0.0.1:11211'], debug=0)
x = mc.set("some_key", "Some value")
print 'Just set a key and value into the cache (suposedly)'
value = mc.get("some_key")
print 'Just retrieved that value from the cache using the key'
print 'X %s' % x
print 'Value %s' % value
[~/test]$ python m2.py
Just set a key and value into the cache (suposedly)
Just retrieved that value from the cache using the key
X 0
Value None
[~/test]$
The question now is, what have I failed to do in my installation? It appears to be working from an API perspective but it fails to put anything into the memcache share area.
I'm using a virtualbox vm running centos
[~]# cat /proc/version
Linux version 2.6.32-358.6.2.el6.i686 (mockbuild#c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Thu May 16 18:12:13 UTC 2013
Is there a daemon that is supposed to be running? I don't see an obvious named one when I do a ps.
I tried to get pylibmc installed on my vm but was unable to find a working installation so for now will see if I can get the above stuff working first.
I discovered if i ran straight from the python console GUI i get a bit more output if I set debug=1
>>> mc = memcache.Client(['127.0.0.1:11211'], debug=1)
>>> mc.stats
{}
>>> mc.set('test','value')
MemCached: MemCache: inet:127.0.0.1:11211: connect: Connection refused. Marking dead.
0
>>> mc.get('test')
MemCached: MemCache: inet:127.0.0.1:11211: connect: Connection refused. Marking dead.
When I try to use per the example telnet to connect to the port i get a connection refused:
[root#~]# telnet 127.0.0.1 11211
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
[root#~]#
I tried the instructions I found on the net for configuring telnet so localhost wouldn't be disabled:
vi /etc/xinetd.d/telnet
service telnet
{
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/sbin/in.telnetd
log_on_failure += USERID
disable = no
}
And then ran the commands to restart the service(s):
service iptables stop
service xinetd stop
service iptables start
service xinetd start
service iptables stop
I ran with both cases (iptables started and stopped) but it has no effect. So I am out of ideas. What do I need to do to make it so the PORT will be allowed? if that is the problem?
Or is there a memcached service that needs to be running that needs to open up the port ?
well this is what it took to get it working: ( a series of manual steps )
1) su -
cd /var/run
mkdir memcached # this was missing
In the memcached file I added "-l 127.0.0.1" to the OPTIONS statement. It's apparently a listen option. Do this for steps 2 & 3. I'm not certain which file is actually used at runtime.
2) cd /etc/sysconfig
cp memcached memcached.old
vi memcached
3) cd /etc/init.d
cp memcached memcached.old
vi memcached
4) Try some commands to see if the server starts now
/etc/init.d/memcached start
/etc/init.d/memcached status
/etc/init.d/memcached stop
/etc/init.d/memcached restart
I tried opening a browser, but it never seemed to actually display anything so I don't really know how valid this approach is. I'm not running apache or anything like this so perhaps its not relevant to my cause. Perhaps I would have to supply a ?key=blah or something.
5) http://127.0.0.1:11211
6) Now it should be ready to go. If one runs the test shown with the following it should work. At least it did for me. doing the help(memcache) will display a simple program. just paste that in and it should work just fine.
[~]$ python
>>> import memcache
>>> help(memcache)