I have a python app behind supervisor and uwsgi.
At a certain point, my app stopped to answer the queries with this message in the logs:
Tue Sep 6 11:06:53 2022 - *** uWSGI listen queue of socket "127.0.0.1:8200" (fd: 3) full !!! (101/100) ***
In my use case,
If a query is not answered within 1s, the answer does not matter anymore; the client app will automatically re-do the request
restarting the whole uwsgi takes around half an hour
Thus I prefer to lose few requests than restarting.
QUESTIONS :
Is it possible to detect a full queue from inside the python app ?
Is it possible to clear the queue from inside ?
Precision: this question is not about fixing the underlying issue. I'm working on that separately. It is only about knowing if this particular workaround is possible and how to implement it.
I'm using uwsgi 2.0.20. Looking at the queue framework does not help since uwsgi has no attribute (e.g.) queue_slot. Doc outdated ?
EDIT
I can reproduce the error with this simple bash script:
#!/bin/bash
for i in {0..200}
do
echo "Number: $i"
sleep 0.2
curl -X POST "http://localhost:1103/my_app" &
done
(my app accepts POST, not GET)
Related
I have a proc which spawns a child process on port 2600. it connects to the process with handle 6, extracts some data, kills the process and then it starts another child process on the same port, I connect to it and when I try to run my commands on the handle I get this following error:
'Cannot write to handle 6. OS reports: Bad File Descriptor
The process is running and I can connect to it manually and get my data. I have a table tracking the connection and I can see the handle for the second time the child process is spawned is also 6. Any suggestions on why this is happening?
UPDATE: Just to make it clear. When I close the handle I send an async message to the proc with "exit 1". Could this be that the process is not killed before I open a connection to it, gets killed after that and then what I think it's the new process is just a garrbage handle to the old, now defunct process?
UPDATE: I do flush the handle after I send "exit 1" and the new process seem to start ok. I can see the new process running on the process (they have different names)
UPDATE: Even thought connection is successful the handle is not added to .z.W. so the handle doesn't appear in .z.W
UPDATE: found the issue. the .IPC.SCON function which I was using which i thought was just a simple wrapper for an error trap ... has logic to check a table of cached handles for the same hst:prt and take that one instead of opening another one. because my process was running on the same host:port, it was using the old cached handle instead of opening it again. thank you all for your help.
What version of kdb are you using? Can you run the below without issues?
KDB+ 4.0 2020.08.28 Copyright (C) 1993-2020 Kx Systems
q)system"q -p 5000";system"sleep 1";h:hopen 5000;0N!h".z.i";neg[h](exit;0)
4475i
q)system"q -p 5000";system"sleep 1";h:hopen 5000;0N!h".z.i";neg[h](exit;0)
4481i
q)system"q -p 5000";system"sleep 1";h:hopen 5000;0N!h".z.i";neg[h](exit;0)
4487i
Update: Without seeing your code or the order of execution, it's hard to pinpoint what the issue is. If the process wasn't terminated before you attempt to start the new process, you would see an Address already in use error. If the process isn't up when you attempt to connect you would see a 'hop error which means hopen failed.
Also, async messages are not sent immediately so depending on the execution order of your code, you may need to flush the async 'exit' message, but like I mentioned, if the original child process is still up when you attempt to start another, you would get an address clash
Update 2: As per my comment above
q)system"q -p 5000"; system"sleep 1"; h:#[hopen;(`::5000;2000);0Ni]; 0N!h".z.i"; neg[h](exit;1)
4365i
q)system"q busy.q -p 5000"; system"sleep 1"; #[hopen;(`::5000;2000);0Ni]; h".z.i" // hopen times out + handle variable not amended
'Cannot write to handle 4. OS reports: Bad file descriptor
[0] system"q busy.q -p 5000"; system"sleep 1"; #[hopen;(`::5000;2000);0Ni]; h".z.i" // hopen times out + handle variable not amended
^
q).z.W
q)
I think what you have written in your UPDATE is correct. The process is trying to connect to that port before your new process is running on it.
I think the best option would be for the newly started process to initiate the connection to the parent rather, that way you know it will be running and don't have to introduce any sleeps.
Other option is to try and reconnect on a timer, once a successful connection occurs remove it from the timer and continue with what you are trying to do.
i am using using activator project which has play 2.4.2 . just for testing i deployed raw project which only listen on port 80 0r 9000 and returning Ok("abc").
but when i check the output of
$ sudo lsof -i | wc -l
the number increasing gradually with time, and after some time let say 24-48 hours. the server crashes with exception too many file open.
i tested with apache benchmark also, after completion of benchmarking, there is still some connections open and never close.
please someone help.
There seems to be some debate around this issue, when sometime back I was working with playframework.
First verify that if your client is asking for connection to be kept alive. In this case playframework would honor the client and keep the connection open. See this disscussion . The takeaway from discussion was play can handle a lot of request, which is questionable if you think about DoS attacks.
The other thing there seems to be options to kill the connection from the action with the header, but I have never tried with those. See this. I am not able to pull any documentation around this option at this moment.
Edit : Seems to be mentioned in 2.2. hightlight.
I wrote a quick PHP page to handle 502 requests. Nginx will re-direct to this page when a 502 is encountered and an email is fired off.
The problem is, most of the time that the 502 is encountered is because PHP has died, so writing to the DB and sending an email using PHP is no longer possible. Tweaks to PHP-FPM settings have done a lot to help (restarting PHP, etc), but I'd still like a fall-back.
There are numerous ways to send an email outside of PHP, but I am curious what others out there are doing with good success? I'd like to keep it simple for configuration (i.e. not have yet another complex dependency to worry about on the servers) and reliability reasons.
Googling and searching SO didn't turn up much, probably because "dies" and "fail" bring back a lot of false positives for my scenario.
What about use a cronjob (bash based) to parse error_log file periodically (x hours) and send an email (mutt/mail) when find something like resuming normal operations in the last period (x hours). I think is simple and effective...
[Thu Dec 27 14:37:52 2012] [notice] caught SIGTERM, shutting down
[Thu Dec 27 14:37:53 2012] [notice] Apache/2.2.22 (Ubuntu) PHP/5.4.6-2~precise+1 configured -- resuming normal operations
UPDATE:
#Brian As #takeshin says cronjobs can run even every second if you want, but some sysadmins could bite you... :|
Here is what I've ended up doing. I've not rolled it out to our prod servers yet, but all testing thus far looks good.
Nginx does not support CGI natively, so you need another means to do it. thttpd fit the bill nicely. There is a good write up the nginx wiki showing how to use it.
I configured thttpd with the following:
dir=/var/www/htdocs
user=thttpd
logfile=/var/log/thttpd.log
pidfile=/var/run/thttpd.pid
port=8000
cgipat=**.cgi
And added this to my nginx config:
error_page 502 #thttpd;
location #thttpd {
include proxy.include;
proxy_pass http://127.0.0.1:8000;
}
Finally, I created a basic CGI script that calls PHP on the command line and passed in my already-written PHP script. This was an ideal solution for me because the script was already set up to log to our alerts table and fire off an email. This is also real-time, as the script will execute as soon as nginx returns a 502 code (subsequent 502s will not hammer me with emails, per the logic of the script).
I was able to run some simulation tests be forcing nginx to return a 502 (see more here).
I'm going to continue tweaking this, but I'm pretty happy with the relative ease of deploying it and that I could re-use existing code.
We have dual solution.
We use shell script to send out email notifications, if PHP dies. We check if php service is running with shell command in the shell script, if it is not running, we'll fire off a shell command to send an email.
This is all in a few lines of Shell Script. Not too hard.
Of course, set it up in cron.
I'm looking at hosting a number of small, static websites and have been looking at a few alternatives including G-WAN. At the moment I'm just trying to get a feel for how well each server suits my needs before picking one.
G-WAN seems to do exactly what I want, though I'm running into problems with updating the configuration (by adding new folders) after the server's started. I can't find anything in the documentation or online about this, so I don't know if I'm doing anything dumb, running an unsupported configuration, or whether it's a feature that doesn't exist in G-WAN.
Here's my setup:
G-WAN 3.3.28 64-bit on Ubuntu 12.04.1 LTS.
I have what I think is the required minimal folder structure:
0.0.0.0_80
#0.0.0.0
www
$site.com
www
$othersite.com
www
I startup gwan via (I'm still messing around, so hopefully ):
sudo .\gwan -d
Everything works brilliantly. I add $thirdsite.com/, $thirdsite.com/www/, and $thirdsite.com/www/index.html; then when I try to visit thirdsite.com it gives me the root host (ie it doesn't seem to pick up the changes).
To reload the modified configuration, I have to either do:
sudo .\gwan -k; sudo .\gwan -d
or kill the non-angel process (kill -s 15) to restart the child process.
Can G-WAN reload the host definitions another way? If so, is it something that works out of the box or is there a command that can cycle the server without dropping requests made to other hosts (/is it safe to kill -s 15 on the non-angel process + if so, is there a reliable way to identify the process)? Thanks in advance!
G-WAN loads the host definitions at startup and does not check them as time goes to reload them dynamically.
To force a reload, you have to stop the child process (when in daemon mode) and v3.9+ keeps the old child alive the time to process any pending request while the new child accepts new connections.
Since stopping the child can also be done from the maintenance script or from a handler or from a servlet by just running exit(0) there is not need for a dedicated command.
Note that when you use kill you can pick the pid file from the gwan directory:
the parent process starts with a capital letter: Gwan_xxxx.pid
the child process starts with a lowercase letter: gwan_xxxx.pid
That will make your life easier.
I am on an embedded platform (mipsel architecture, Linux 2.6 kernel) where I need to monitor IPC between two closed-source processes (router firmware) in order to react to a certain event (dynamic IP change because of DSL reconnect). What I found out so far via strace is that whenever the IP changes, the DSL daemon writes a special message into a UNIX domain socket bound to a specific file name. The message is consumed by another daemon.
Now here is my requirement: I want to monitor the data flow through that specific UNIX domain socket and trigger an event (call a shell script) if a certain message is detected. I tried to monitor the file name with inotify, but it does not work on socket files. I know I could run strace all the time, filtering its output and react to changes in the filtered log file, but that would be too heavy a solution because strace really slows down the system. I also know I could just poll for the IP address change via cron, but I want a watchdog, not a polling solution. And I am interested in finding out whether there is a tool which can specifically monitor UNIX domain sockets and react to specific messages flowing through in a predefined direction. I imagine something similar to inotifywait, i.e. the tool should wait for a certain event, then exit, so I can react to the event and loop back into starting the tool again, waiting for the next event of the same type.
Is there any existing Linux tool capable of doing that? Or is there some simple C code for a stand-alone binary which I could compile on my platform (uClibc, not glibc)? I am not a C expert, but capable of running a makefile. Using a binary from the shell is no problem, I know enough about shell programming.
It has been a while since I was dealing with this topic and did not actually get around to testing what an acquaintance of mine, Denys Vlasenko, maintainer of Busybox, proposed as a solution to me several months ago. Because I just checked my account here on StackOverflow and saw the question again, let me share his insights with you. Maybe it is helpful for somebody:
One relatively easy hack I can propose is to do the following:
I assume that you have a running server app which opened a Unix domain listening socket (say, /tmp/some.socket), and client programs connect to it and talk to the server.
rename /tmp/some.socket -> /tmp/some.socket1
create a new socket /tmp/some.socket
listen on it for new client connections
for every such connection, open another connection to /tmp/some.socket1 to original server process
pump data (client<->server) over resulting pairs of sockets (code to do so is very similar to what telnetd server does) until EOF from either side.
While you are pumping data, it's easy to look at it, to save it, and even to modify it if you need to.
The downside is that this sniffer program needs to be restarted every time the original server program is restarted.
This is similar to what Celada also answered. Thanks to him as well! Denys's answer was a bit more concrete, though.
I asked back:
This sounds hacky, yes, because of the restart necessity, but feasible.
Me not being a C programmer, I keep wondering though if you know a
command line tool which could do the pass-through and protocolling or
event-based triggering work for me. I have one guy from our project in
mind who could hack a little C binary for that, but I am unsure if he
likes to do it. If there is something pre-fab, I would prefer it. Can it
even be done with a (combination of) BusyBox applet(s), maybe?
Denys answered again:
You need to build busybox with CONFIG_FEATURE_UNIX_LOCAL=y.
Run the following as intercepting server:
busybox tcpsvd -vvvE local:/tmp/socket 0 ./script.sh
Where script.sh is a simple passthrough connection
to the "original server":
#!/bin/sh
busybox nc -o /tmp/hexdump.$$ local:/tmp/socket1 0
As an example, I added hex logging to file (-o FILE option).
Test it by running an emulated "original server":
busybox tcpsvd -vvvE local:/tmp/socket1 0 sh -c 'echo PID:$$'
and by connecting to "intercepting server":
echo Hello world | busybox nc local:/tmp/socket 0
You should see "PID:19094" message and have a new /tmp/hexdump.19093 file
with the dumped data. Both tcpsvd processes should print some log too
(they are run with -vvv verbosity).
If you need more complex processing, replace nc invocation in script.sh
with a custom program.
I don't think there is anything that will let you cleanly sniff UNIX socket traffic. Here are some options:
Arrange for the sender process to connect to a different socket where you are listening. Also connect to the original socket as a client. On receipt of data, notice the data you want to notice and also pass everything along to the original socket.
Monitor the system for IP address changes yourself using a netlink socket (RTM_NEWADDR, RTM_NEWLINK, etc...).
Run ip monitor as an external process and take action when it writes messages about added & removed IP addresses on its standard output.