I have a python app behind supervisor and uwsgi.
At a certain point, my app stopped to answer the queries with this message in the logs:
Tue Sep 6 11:06:53 2022 - *** uWSGI listen queue of socket "127.0.0.1:8200" (fd: 3) full !!! (101/100) ***
In my use case,
If a query is not answered within 1s, the answer does not matter anymore; the client app will automatically re-do the request
restarting the whole uwsgi takes around half an hour
Thus I prefer to lose few requests than restarting.
QUESTIONS :
Is it possible to detect a full queue from inside the python app ?
Is it possible to clear the queue from inside ?
Precision: this question is not about fixing the underlying issue. I'm working on that separately. It is only about knowing if this particular workaround is possible and how to implement it.
I'm using uwsgi 2.0.20. Looking at the queue framework does not help since uwsgi has no attribute (e.g.) queue_slot. Doc outdated ?
EDIT
I can reproduce the error with this simple bash script:
#!/bin/bash
for i in {0..200}
do
echo "Number: $i"
sleep 0.2
curl -X POST "http://localhost:1103/my_app" &
done
(my app accepts POST, not GET)
I have a server that runs Postgresql. in the logs I am seeing this message for my resque based 'worker' box, multiple times a minute. Some minutes there isn't a message, others could be 10 times.
2016-01-12 13:40:36 EST:1.1.8.2(33899):[16141]: LOG: could not receive data from client: Connection reset by peer
Now when i go into the 1.1.8.2 box to look at netstat -ntp i don't see a port 33899, and most of them are at least in the 40xxx range by now. That may be conjecture but I'm at a loss to find out why a Redis/Resque/Puma Rails stack would be printing out these messages, let alone what that means even if i get to the bottom of it.
Will I gain memory back if they are closed 'normally'?
Is this a thing to be wary of?
How does one debug OLD ports that are open when the db box and the worker box both don't display the ports any more?
This message is probably due to the resque worker task not closing the database connection before it exits. It's not a huge problem, but presumably Postgres is doing a little extra work to clean it up, and it makes a mess of your log file...
One solution is to add a hook to your resque worker's task file (the same file that contains the self.perform definition):
def self.after_perform(*args)
ActiveRecord::Base.connection.disconnect!
end
i have a problem with replication.
my sql server shows me in log : 3 errors skipped while submit replication.
but how to see records where problems appear ?
:'(
thk in advance.
If you're interested in exploring the differences you should use either a query or the tablediff utility to find the differences between your publisher and subscriber.
https://msdn.microsoft.com/en-us/library/ms162843.aspx
i am using using activator project which has play 2.4.2 . just for testing i deployed raw project which only listen on port 80 0r 9000 and returning Ok("abc").
but when i check the output of
$ sudo lsof -i | wc -l
the number increasing gradually with time, and after some time let say 24-48 hours. the server crashes with exception too many file open.
i tested with apache benchmark also, after completion of benchmarking, there is still some connections open and never close.
please someone help.
There seems to be some debate around this issue, when sometime back I was working with playframework.
First verify that if your client is asking for connection to be kept alive. In this case playframework would honor the client and keep the connection open. See this disscussion . The takeaway from discussion was play can handle a lot of request, which is questionable if you think about DoS attacks.
The other thing there seems to be options to kill the connection from the action with the header, but I have never tried with those. See this. I am not able to pull any documentation around this option at this moment.
Edit : Seems to be mentioned in 2.2. hightlight.
I am on an embedded platform (mipsel architecture, Linux 2.6 kernel) where I need to monitor IPC between two closed-source processes (router firmware) in order to react to a certain event (dynamic IP change because of DSL reconnect). What I found out so far via strace is that whenever the IP changes, the DSL daemon writes a special message into a UNIX domain socket bound to a specific file name. The message is consumed by another daemon.
Now here is my requirement: I want to monitor the data flow through that specific UNIX domain socket and trigger an event (call a shell script) if a certain message is detected. I tried to monitor the file name with inotify, but it does not work on socket files. I know I could run strace all the time, filtering its output and react to changes in the filtered log file, but that would be too heavy a solution because strace really slows down the system. I also know I could just poll for the IP address change via cron, but I want a watchdog, not a polling solution. And I am interested in finding out whether there is a tool which can specifically monitor UNIX domain sockets and react to specific messages flowing through in a predefined direction. I imagine something similar to inotifywait, i.e. the tool should wait for a certain event, then exit, so I can react to the event and loop back into starting the tool again, waiting for the next event of the same type.
Is there any existing Linux tool capable of doing that? Or is there some simple C code for a stand-alone binary which I could compile on my platform (uClibc, not glibc)? I am not a C expert, but capable of running a makefile. Using a binary from the shell is no problem, I know enough about shell programming.
It has been a while since I was dealing with this topic and did not actually get around to testing what an acquaintance of mine, Denys Vlasenko, maintainer of Busybox, proposed as a solution to me several months ago. Because I just checked my account here on StackOverflow and saw the question again, let me share his insights with you. Maybe it is helpful for somebody:
One relatively easy hack I can propose is to do the following:
I assume that you have a running server app which opened a Unix domain listening socket (say, /tmp/some.socket), and client programs connect to it and talk to the server.
rename /tmp/some.socket -> /tmp/some.socket1
create a new socket /tmp/some.socket
listen on it for new client connections
for every such connection, open another connection to /tmp/some.socket1 to original server process
pump data (client<->server) over resulting pairs of sockets (code to do so is very similar to what telnetd server does) until EOF from either side.
While you are pumping data, it's easy to look at it, to save it, and even to modify it if you need to.
The downside is that this sniffer program needs to be restarted every time the original server program is restarted.
This is similar to what Celada also answered. Thanks to him as well! Denys's answer was a bit more concrete, though.
I asked back:
This sounds hacky, yes, because of the restart necessity, but feasible.
Me not being a C programmer, I keep wondering though if you know a
command line tool which could do the pass-through and protocolling or
event-based triggering work for me. I have one guy from our project in
mind who could hack a little C binary for that, but I am unsure if he
likes to do it. If there is something pre-fab, I would prefer it. Can it
even be done with a (combination of) BusyBox applet(s), maybe?
Denys answered again:
You need to build busybox with CONFIG_FEATURE_UNIX_LOCAL=y.
Run the following as intercepting server:
busybox tcpsvd -vvvE local:/tmp/socket 0 ./script.sh
Where script.sh is a simple passthrough connection
to the "original server":
#!/bin/sh
busybox nc -o /tmp/hexdump.$$ local:/tmp/socket1 0
As an example, I added hex logging to file (-o FILE option).
Test it by running an emulated "original server":
busybox tcpsvd -vvvE local:/tmp/socket1 0 sh -c 'echo PID:$$'
and by connecting to "intercepting server":
echo Hello world | busybox nc local:/tmp/socket 0
You should see "PID:19094" message and have a new /tmp/hexdump.19093 file
with the dumped data. Both tcpsvd processes should print some log too
(they are run with -vvv verbosity).
If you need more complex processing, replace nc invocation in script.sh
with a custom program.
I don't think there is anything that will let you cleanly sniff UNIX socket traffic. Here are some options:
Arrange for the sender process to connect to a different socket where you are listening. Also connect to the original socket as a client. On receipt of data, notice the data you want to notice and also pass everything along to the original socket.
Monitor the system for IP address changes yourself using a netlink socket (RTM_NEWADDR, RTM_NEWLINK, etc...).
Run ip monitor as an external process and take action when it writes messages about added & removed IP addresses on its standard output.