Memcached hangs after "END" when getting stats - memcached

enter code hereI have memcached installed on a Microsoft Azure ubuntu 18.04 server. First we tried installing it in a docker container, but we removed that and installed it using sudo apt-get install memcached libmemcached-tools -y. The configuration is the default one. When I run echo stats | nc 127.0.0.1 11211 it prints out the stats as normal finishing with END, like this
deploy#xyz-api-01:~$ echo stats | nc 127.0.0.1 11211
STAT pid 65896
STAT uptime 6847
STAT time 1584992459
STAT version 1.5.6 Ubuntu
STAT libevent 2.1.8-stable
STAT pointer_size 64
STAT rusage_user 0.597944
STAT rusage_system 0.768785
STAT max_connections 1024
STAT curr_connections 1
STAT total_connections 7
STAT rejected_connections 0
STAT connection_structures 2
STAT reserved_fds 20
STAT cmd_get 1
STAT cmd_set 0
STAT cmd_flush 0
STAT cmd_touch 0
STAT get_hits 0
STAT get_misses 1
STAT get_expired 0
STAT get_flushed 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 64
STAT bytes_written 9509
STAT limit_maxbytes 67108864
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT time_in_listen_disabled_us 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT slab_reassign_rescues 0
STAT slab_reassign_chunk_rescues 0
STAT slab_reassign_evictions_nomem 0
STAT slab_reassign_inline_reclaim 0
STAT slab_reassign_busy_items 0
STAT slab_reassign_busy_deletes 0
STAT slab_reassign_running 0
STAT slabs_moved 0
STAT lru_crawler_running 0
STAT lru_crawler_starts 3825
STAT lru_maintainer_juggles 6891
STAT malloc_fails 0
STAT log_worker_dropped 0
STAT log_worker_written 0
STAT log_watcher_skipped 0
STAT log_watcher_sent 0
STAT bytes 0
STAT curr_items 0
STAT total_items 0
STAT slab_global_page_pool 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0
STAT reclaimed 0
STAT crawler_reclaimed 0
STAT crawler_items_checked 0
STAT lrutail_reflocked 0
STAT moves_to_cold 0
STAT moves_to_warm 0
STAT moves_within_lru 0
STAT direct_reclaims 0
STAT lru_bumps_dropped 0
END
At that point it hangs though, as if it was waiting for something, but typing commands doesn't do anything. It sits there until I type CTRL+C. We had exctly the same issue when it was running in a docker container, which is why we tried installing it directly in the server in case it was an issue with the container. Our application which uses memcached also times out when trying to connect to memcached, so I don't think it's just a terminal problem.
Googling has not found any posts about this specific problem, most people who have issues with memcached hanging find that it hangs for a while and then eventually returns. I've never seen this behaviour before, and I can't find anything in configuration or anywhere that would explain this. I'd appreciate if anyone knows of something that might cause this or some configuration that I need to check.

nc essentially connects and listens, and it won't stop listening until you tell it not to. So CTRL+C, or if you want to force it to timeout after a second:
echo stats | nc -w 1 127.0.0.1 11211
There is nothing wrong happening here. It's about how nc works.
https://www.tutorialspoint.com/unix_commands/nc.htm

You need to send an quit command, for example:
printf "stats\nquit\n" | nc 127.0.0.1 11211

Related

Supervisord sometimes starts celery, sometimes not

I'm deploying my flask api on Kubernetes. The executed command when the container is started is the following:
supervisord -c /etc/supervisor/conf.d/celery.conf
gunicorn wsgi:app --bind=0.0.0.0:5000 --workers 1 --threads 12 --log-level=warning --access-logfile /var/log/gunicorn-access.log --error-logfile /var/log/gunicorn-error.log
You see above that I'm starting celery first with supervisor and after that I'm running the gunicorn server. Content of celery.conf:
[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
identifier = supervisor
directory = /tmp
nocleanup = true
[program:celery]
directory = /mydir/app
command = celery -A celery_worker.celery worker --loglevel=debug
When logged into my pods I can see that sometimes the process of starting celery is working (example in pod 1):
> more /tmp/supervisord.log
2021-06-08 18:19:46,460 CRIT Supervisor running as root (no user in config file)
2021-06-08 18:19:46,462 INFO daemonizing the supervisord process
2021-06-08 18:19:46,462 INFO set current directory: '/tmp'
2021-06-08 18:19:46,463 INFO supervisord started with pid 9
2021-06-08 18:19:47,469 INFO spawned: 'celery' with pid 15
2021-06-08 18:19:48,470 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Sometimes it's not (in pod 2):
> more /tmp/supervisord.log
2021-06-08 18:19:42,979 CRIT Supervisor running as root (no user in config file)
2021-06-08 18:19:42,988 INFO daemonizing the supervisord process
2021-06-08 18:19:42,988 INFO set current directory: '/tmp'
2021-06-08 18:19:42,989 INFO supervisord started with pid 9
2021-06-08 18:19:43,992 INFO spawned: 'celery' with pid 11
2021-06-08 18:19:44,994 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
>>>> 2021-06-08 18:19:58,642 INFO exited: celery (exit status 2; expected) <<<<<HERE
In my pod 1, a ps command shows the following:
> ps aux | grep celery
root 9 0.0 0.0 55308 16376 ? Ss 18:45 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/conf.d/celery.conf
root 23 2.2 0.8 2343684 352940 ? S 18:45 0:05 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 37 0.0 0.5 2341860 208716 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 38 0.0 0.5 2341864 208716 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 39 0.0 0.5 2341868 208716 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 40 0.0 0.5 2341872 208724 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 41 0.0 0.5 2341876 208728 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 42 0.0 0.5 2341880 208728 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 43 0.0 0.5 2341884 208736 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 44 0.0 0.5 2342836 211384 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
In my pod 2, I can see that supervisord/celery process is still there but I don't have all the individual /usr/local/bin/celery processes that I have in pod 1:
> ps aux | grep celery
root 9 0.0 0.0 55308 16296 ? Ss 18:19 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/conf.d/celery.conf
This behavior is not always the same. Sometimes when pods are restarted the two succeed to launch celery, sometimes none of them succeed. In this last scenario if I make a request to my API that is supposed to launch a celery task, I can see on my broker console (RabbitMQ) that a task is created but there is no message "activity" and nothing is written is my database table (the end result of my celery task).
If I start celery manually in my pods:
celery -A celery_worker.celery worker --loglevel=debug
everything works.
What could explain such a behavior?
Following the comments above, the best solution is to have two containers, the first having the entrypoint gunicorn and the other celery celery-worker. If the second is the same image as the first it works very well and I can scale on Kubernetes each container independently. The only thing is that the code sourcing is more difficult, each time I make a code change on the first I must apply the same changes manually on the second, maybe there is a better way to address this specific issue of the code sourcing.

Docker compose install error 'curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104' in Ubuntu

I am trying to install docker compose on the Ubuntu 18.04.2 LTS.
I tried installing using the official link here and followed the Docker Compose documentation given, but when i run the command
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
then after some time it gives me this error
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 617 0 617 0 0 613 0 --:--:-- 0:00:01 --:--:-- 613
24 8280k 24 2056k 0 0 789 0 2:59:06 0:44:27 2:14:39 0
**curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104**
Kindly help me on this i have tried many times but it is not working.
I had the same problem. I assume that you are using Docker Docs, which are usually outdated. You should go to Docker Compose Github instead.
Solution
1 - Open Linux Terminal by pressing Ctrl + Alt + T
2 - Install curl:
sudo apt install curl
3 - Turn on root privileges in terminal for your user (something like admin in Windows OS), with command:
sudo -i
4 - Go to Docker Compose Github. In releases you will find this code. Run it in your linux terminal.
curl -L https://github.com/docker/compose/releases/download/1.25.1-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
5 - Turn off root privileges in terminal for your user, with command:
exit
6 - Check if docker-compose is installed with command:
docker-compose version
Outcome: In your terminal, you should see docker-compose version number and some other informations.

Failed Perlbrew install on shared webspace

I am looking for help to install perlbrew on my shared webspace (Provider: uberspace.de). I used the usually documentation an perlbrew.pl and put:
\curl -L http://install.perlbrew.pl | bash
In my terminal.
Download completed, but than the terminal says:
$ curl -L http://install.perlbrew.pl | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
103 1247 103 1247 0 0 1274 0 --:--:-- --:--:-- --:--:-- 1274
## Download the latest perlbrew
## Installing perlbrew
print() on closed filehandle $fh at perlbrew.YQFev9 line 893.
perlbrew is installed: /package/host/localhost/perlbrew/bin/perlbrew
NOTICE: /package/host/localhost/perlbrew/etc/bashrc already exists and not updated.
NOTICE: /package/host/localhost/perlbrew/etc/cshrc already exists and not updated.
Fail to create /package/host/localhost/perlbrew/etc/csh_reinit. Please check the permission of /package/host/localhost/perlbrew/etc and try `perlbrew init` again. at perlbrew.YQFev9 line 806.
I have no permission to access /package/host/localhost/perlbrew/...
I think that perlbrew has to install on /perl5/perlbrew, right?
How can I change the location of the installation?
Or is there another problem?
Thanks!

Sphinx search doesn't stop nor correctly update

I have problems running sphinx search on my debian Wheezy server.
Currently, there are 2 searchd ports running
root#ns243216:~# netstat -tlpn | grep search
tcp 0 0 0.0.0.0:9306 0.0.0.0:* LISTEN 11266/searchd
tcp 0 0 0.0.0.0:9312 0.0.0.0:* LISTEN 11266/searchd
First Problem
When I want to execute this
sudo /usr/bin/indexer -c /etc/sphinxsearch/sphinx.conf beta_jobs --rotate
It gives me this :
Sphinx 2.2.10-id64-release (2c212e0)
Copyright (c) 2001-2015, Andrew Aksyonoff
Copyright (c) 2008-2015, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/etc/sphinxsearch/sphinx.conf'...
indexing index 'beta_jobs'...
collected 6 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 6 docs, 867 bytes
total 0.046 sec, 18747 bytes/sec, 129.73 docs/sec
total 6 reads, 0.000 sec, 0.4 kb/call avg, 0.0 msec/call avg
total 12 writes, 0.000 sec, 0.9 kb/call avg, 0.0 msec/call avg
WARNING: failed to scanf pid from pid_file '/usr/local/sphinx/var/log/searchd/searchd.pid'.
WARNING: indices NOT rotated.
2 warnings I can't remove...
Second Problem: And when I want to stop my searchd with searchd --stop, it tells me this :
Sphinx 2.2.10-id64-release (2c212e0)
Copyright (c) 2001-2015, Andrew Aksyonoff
Copyright (c) 2008-2015, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/etc/sphinxsearch/sphinx.conf'...
FATAL: stop: failed to read valid pid from '/usr/local/sphinx/var/log/searchd/searchd.pid'
I tried setting chmod 755 to everything inside /usr/local/sphinx/var/log/searchd/, still doesn't work.
My sphinx.conf is here Sphinx.conf on gist
EDIT (answer to #aeryaguzov comment)
root#ns213646:~# sudo cat /usr/local/sphinx/var/log/searchd/searchd.pid
root#ns213646:~# ps aux | grep searchd
root 11265 0.0 0.0 79692 1228 ? S Nov30 0:00 /usr/bin/searchd
root 11266 0.1 0.0 91404 4696 ? Sl Nov30 26:54 /usr/bin/searchd
root 22783 0.0 0.0 8292 632 pts/1 S+ 15:32 0:00 grep searchd
Okay it appears that for some unknown reasons the searchd.pid was badly created by searchd (which is running). So I decided to delete the search.pid and to kill searchd. Then I re-indexed and started searchd with no problems.

How to enable large page memory for the JVM?

In the document(jbossperformancetuning.pdf), it suggest us to enable large page memory for the JVM.
But actually after I added the following to our command-line / script start-up:
"-XX:+UseLargePages"
It didn't work, so I investigated more, enabled the large page memory on OS first, then added "-XX:+UseLargePages -XX:LargePageSizeInBytes=2m" to start-up script.
But unfortunately, it didn't work neither, so could someone give us some suggestions of how to enable the large page memory for the JVM successfully?
Here are some details of our server:
[root#localhost ~]# cat /proc/meminfo
MemTotal: 37033340 kB
MemFree: 318108 kB
Buffers: 179452 kB
Cached: 5934940 kB
SwapCached: 0 kB
...
HugePages_Total: 10251
HugePages_Free: 10251
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
[root#localhost ~]# ps aux | grep java
root 22525 0.2 20.3 28801756 7552420 ? Sl Nov03 31:54 java -Dprogram.name=run.sh -server -Xms1303m -Xmx24g -XX:MaxPermSize=512m -XX:+UseLargePages -XX:LargePageSizeInBytes=2m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -verbose:gc -Xloggc:/tmp/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss-as/lib/endorsed -classpath /opt/jboss-as/bin/run.jar org.jboss.Main -c default -b 0.0.0.0
root 31962 0.0 0.0 61200 768 pts/2 S+ 22:46 0:00 grep java
[root#localhost ~]# cat /etc/sysctl.conf
...
# JBoss is running as root, so the group id is 0
vm.hugetlb_shm_group = 0
# The pages number
vm.nr_hugepages = 12288
Finally I fixed this issue, at first set the large pages memory bigger than JVM heap size, then just reboot the server, because there is no way to make it work unless you upgrade the kernel to the newer one - in RHEL 6.0.