i'm building some application, where i have to use memcached.
I found quite nice client:
net.spy.memcached.MemcachedClient
Under this cliend everything works greate except one - i have problem with close connection, and after a while i'm startign to fight with memory leak.
I was looking for possibility for close connection, and i foud "shutdown" method. But if i use this method like this:
MemcachedClient c = new MemcachedClient(new InetSocketAddress(
memcachedIp, memcachedPort));
c.set(something, sessionLifeTime, memcache.toJSONString());
c.shutdown();
I have problem with adding anything do memcached - in logs i see that this method is opening connection, and before it will add anything to memcached, it's closeing the connection.
Do you have any idea, what to do?
Additionally - i found method: c.shutdown(2, TimeUnit.SECONDS); - which should close connection after 2 seconds, but i have connected jmx monitor to my tomcat and i see, that Memcached thread isn't finished after 2 seconds - this thread isn't finished at all...
The reason you are having an issue adding things to memcached like this is that the set(...) function is asynchronous and all it does is put that operation into a queue to be sent to memcached. Since you call shutdown right after this the operation doesn't actually have time to make it out onto the wire. You need to call set(...).get() in order to make your application thread actually wait for the operation to complete before calling shutdown.
Also, I haven't experience IO threads not dying after calling shutdown with a timeout. One way you can confirm that this is an actual bug is by running a standalone program with Spymemached. If the process doesn't terminate when it's completed then you've found an issue.
Related
I have some automated tests that I run in order to test a MongoDB-related library. In order to do that, I start a Mongo server with a temporary data directory and on an ephemeral port, connect to it, and run some tests.
This leads to a race condition, obviously. So in my first version of these tests, I paused for a fixed amount of time and waited to make sure mongod had time to start before the tests began.
This was frustrating (and inefficient), so I decided to monitor the standard output and wait for a line on mongod's standard output stream matching the regular expression:
/\[initandlisten\] waiting for connections/
This got it working. So good, then I prepared to circle back and try to find a more robust way to do it. I recalled that a Java library called "embedmongo" ran MongoDB-based tests, and figured it must solve the problem. And it does this (GitHub):
protected String successMessage() {
return "waiting for connections on port";
}
... and uses that to figure out whether the process has started correctly.
So, are we right? Is examining the mongod process output log (is it ever internationalized? could the wording of the message ever change?) the very best way to do this? Or is there something more robust that we're both missing?
What we do in a similar scenario is:
Try to connect to the configured port (simply new Socket(host, port)) in a loop until it works (10 ms delay) - this ensures, that the mongo client, which starts an internal monitoring thread, does not throw exceptions due to "connection refused"
Connect to the mongodb and query something. This is important, as all mongo client objects are lazy init. (Simple listDatabaseNames() on the client is enough, but make sure to actually read the result.)
All the time, check the process for not being terminated.
I just wrote a small untilMongod command which does just that, which you can use in bash scripting: https://github.com/FGM/untilMongod
Includes a bash + Node.JS example use case.
I have an app in c that listens on a port and creates a pthread upon connection and goes back to the listen. The pthread functions reads from the socket, writes a response and then waits 1/10th of a sec followed by a shutdown() and a close() then pthread_exit(). This can happen very rapidly resulting in possibly hundreds of threads at the same time. My question is can the system reuse a file id before I do the final close()? I'm concerned about the possibility of the socket closing prematurely for some reason. On the listening side the file id cannot be reused until I do the close() call even if the underlying connection is long gone, right? I'm fairly sure that this is how it works but I can't confirm.
On the listening side the file id cannot be reused until I do the
close() call even if the underlying connection is long gone, right?
Yes, this is correct - the file descriptor is not released for re-use until it has been passed to close() (or is an FD_CLOEXEC file descriptor being closed automatically at execve()).
All thread try to enter critical region to be processed if you didn't use semafor,mutex or monitoring probably it uses same id even your files that you get from byte stream may be croupted. I advise to you use semafor, mutex ,or monitoring, and search about dining philosophers problem, because it is very frequent situation. Good luck I hope I can show a clue about your problem.
Ubuntu Linux, 2.6.32-45 kernel, 64b, Perl 5.10.1
I connect many new IO::Socket::UNIX stream sockets to a server, and mostly they work fine. But sometimes in a heavily threaded environment on a faster processor, they return "Resource temporarily unavailable" (EAGAIN/EWOULDBLOCK). I use a timeout on the Connect, so this causes the sockets to be put into non-blocking mode during the connect. But my timeout period isn't occurring - it doesn't wait any noticeable time, it returns quickly.
I see that inside IO::Socket, it tries the connect, and if it fails with EINPROGRESS or EAGAIN/EWOULDBLOCK, it does a select to wait for the write bit to be set. This seems normal so far. In my case the select quickly succeeds, implying that the write bit is set, and the code then tries a re-connect. (I guess this is an attempt to get any error via error slippage?) Anyway, the re-connect fails again with the EAGAIN/EWOULDBLOCK.
In my code this is easy to fix with a re-try loop. But I don't understand why, when the socket becomes writeable, that the socket is not re-connectable. I thought the select guard was always sufficient for a non-blocking connect. Apparently not; so my questions are:
What conditions cause the connect to fail when the select works (the write bit gets set)?
Is there a better way than spinning and retrying, to wait for the connect to succeed? The spinning is wasting cycles. Instead I'd like it to block on something like a select/poll, but I still need a timeout.
Thanx,
-- Steve
But I don't understand why, when the socket becomes writeable, that the socket is not re-connectable.
I imagine it's because whatever needed resource became free got snatched up before you were able to connect again. Replacing the select with a spin loop would not help that.
I am doing remote debugging using eclipse. My requirement is to make 20 requests at the same time, stopping at one point using debug breakpoint and then release all the suspended threads at the same time to test how code is behaving when multiple threads access the code at same time. However, when I tried this I found only one thread is serving all the request
Daemon Thread [http-0.0.0.0-8080-Processor60] (Suspended (breakpoint at line 440 in VcsInfoDAO))
when first request completes, then only second request comes to the breakpoint serving by the same thread mentioned above. Is there any setting in eclipse to make it all the request comes to a single point and then in some way to release the threads at the same time so that all the threads access the code thereafter at the same time.
Any help would be highly appreciated.
Sourabh
Eclipse has nothing to do with what you see. If you set a breakpoint to some place inside a method supposed to be called concurrently, and if your client code really launches 20 concurrent requests, and if you observe that the second request is only handled once the first one has finished, then what you thought was concurrent is not.
I see two possible explanations:
you have a unique thread handling all the requests. If several are sent concurrently, all the requests are queued and handled one by one
you have several threads handling the request concurrently, but the client code sends 20 requests sequentially, rather than sending 20 requests concurrently.
Anyway, using a breakpoint to test such a thing is not a good solution. You'll have to hit the "Continue (F8)" button for each of the 20 threads, thus they won't restart at the same time. You'd bette use a CountDownLatch initialized at 20 to do that:
private CountDownLatch latch = new CountDownLatch(20);
public void run() {
// some code
// here we want to pause all 20 threads and restart them all at the same time
latch.countDown(); // The 20th thread will open the barrier, and they will all restart at the same time
latch.await();
}
Am a c-coder for a while now - neither a newbie nor an expert. Now, I have a certain daemoned application in C on a PPC Linux. I use PHP's socket_connect as a client to connect to this service locally. The server uses epoll for multiplexing connections via a Unix socket. A user submitted string is parsed for certain characters/words using strstr() and if found, spawns 4 joinable threads to different websites simultaneously. I use socket, connect, write and read, to interact with the said webservers via TCP on their port 80 in each thread. All connections and writes seems successful. Reads to the webserver sockets fail however, with either (A) all 3 threads seem to hang, and only one thread returns -1 and errno is set to 104. The responding thread takes like 10 minutes - an eternity long:-(. *I read somewhere that the 104 (is EINTR?), which in the network context suggests that ...'the connection was reset by peer'; or (B) 0 bytes from 3 threads, and only 1 of the 4 threads actually returns some data. Isn't the socket read/write thread-safe? I use thread-safe (and reentrant) libc functions such as strtok_r, gethostbyname_r, etc.
*I doubt that the said webhosts are actually resetting the connection, because when I run a single-threaded standalone (everything else equal) all things works perfectly right, but of course in series not parallel.
There's a second problem too (oops), I can't write back to the client who connect to my epoll-ed Unix socket. My daemon application will hang and hog CPU > 100% for ever. Yet nothing is written to the clients end. Am sure the client (a very typical PHP socket application) hasn't closed the connection whenever this is happening - no error(s) detected either. Any ideas?
I cannot figure-out whatever is wrong even with Valgrind, GDB or much logging. Kindly help where you can.
Yes, read/write are thread-safe. But beware of gethostbyname() and getservbyname() if you're using them - they return pointers to static data, and may not be thread-safe.
errno 104 is ECONNREFUSED (not EINTR). Use strerror or perror to get the textual error message (like 'Connection reset by peer') for a particular errno code.
The best way to figure out what's going wrong is often to do very detailed logging - log the results of every operation, plus details like the IP address/port connecting to, the number of bytes read/written, the thread id, and so forth. And, of course, make sure your logging code is thread-safe :-)
Getting an ECONNRESET after 10 minutes sounds like the result of your connection timing out. Either the web server isn't sending the data or your app isn't receiving it.
To test the former, hookup a program like Wireshark to the local loopback device and look for traffic to and from the port you are using.
For the later, take a look at the epoll() man page. They mention a scenario where using edge triggered events could result in a lockup, because there is still data in the buffer, but no new data comes in so no new event is triggered.