I have a perl-cgi script which is log to cisco devices and run a command. so recently move the script from old solaris server to the newer more powerful vm server. now the script is very slow to log to the device actually got timeout. I am not expert on perl and I don't know how I can troubleshoot.regarding of the network there is no issue detected upon my test. as I said the server and the network at least 10 x faster than the old one. any suggestion? thank in advance.
The script was probably written using blocking sockets. The fact that it moved has probably slowed down the connection between the cisco devices and the server running the CGI. I would check your network path first. If this is still a concern you should write this to fork() a child process, use non-blocking socket techniques or write a CLI app. This doesn't sound like something that is well suited to run as a CGI.
Related
Summary:
I am guessing that the issue here is something to do with how Windows and Linux handle TCP connections, or sockets, but I have no idea what it is. I'm initiating a TCP connection to a piece of custom hardware that someone else has developed and I am trying to understand its behaviour. In doing so, I've created a .Net core 2.2 application; run on a Windows system, I can initiate the connection successfully, but on Linux (latest Raspbian), I cannot.
It appears that it may be because Linux systems do not try to retry/retransmit a SYN after a RST, whereas Windows ones do - and this behaviour seems key to how this peculiar piece of hardware works..
Background:
We have a black box piece of hardware that can be controlled and queried over a network, by using a manufacturer-supplied Windows application. Data is unencrypted and requires no authentication to connect to it and the application has some other issues. Ultimately, we want to be able to relay data from it to another system, so we decided to make our own application.
I've spent quite a long time trying to understand the packet format and have created a library, which targets .net core 2.2, that can be used to successfully communicate with this kit. In doing so, I discovered that the device seems to require a kind of "request to connect" command to be sent, via UDP. Straight afterwards, I am able to initiate a TCP connection on port 16000, although the first TCP attempt always results in a RST,ACK being returned - so a second attempt needs to be made.
What I've developed works absolutely fine on both Windows (x86) and Linux (Raspberry Pi/ARM) systems and I can send and receive data. However, when run on the Raspbian system, there seems to be problems when initiating the TCP connection. I could have sworn that we had it working absolutely fine on a previous build, but none of the previous commits seem to work - so it may well be a system/kernel update that has changed something.
The issue:
When initiating a TCP connection to this device, it will - straight away - reset the connection. It does this even with the manufacturer-supplied software, which itself then immediately re-attempts the connection again and it succeeds; so this kind of reset-once-then-it-works-the-second-time behaviour in itself isn't a "problem" that I have any control over.
What I am trying to understand is why a Windows system immediately re-attempts the connection through a retransmission...
..but the Linux system just gives up after one attempt (this is the end of the packet capture..)
To prove it is not an application-specific issue, I've tried using ncat/netcat on both the Windows system and the Raspbian system, as well as a Kali system on a separate laptop to prove it isn't an ARM/Raspberry issue. Since the UDP "request" hasn't been sent, the connection will never succeed anyway, but this simply demonstrates different behaviour between the OSes.
Linux versions look pretty much the same as above, whereby they send a single packet that gets reset - whereas the Windows attempt demonstrates the multiple retransmissions..
So, does anyone have any answer for this behaviour difference? I am guessing it isn't a .net core specific issue, but is there any way I can set socket options to attempt a retransmission? Or can it be set at the OS level with systemctl commands or something? I did try and see if there are any SocketOptionNames, in .net, that look like they'd control attempts/retries, as this answer had me wonder, but no luck so far.
If anyone has any suggestions as to how to better align this behaviour across platforms, or can explain the reason for this difference is at all, I would very much appreciate it!
Nice find! According to this, Windows´ TCP will retry a connection if it receives a RST/ACK from the remote host after sending a SYN:
... Upon receiving the ACK/RST client from the target host, the client determines that there is indeed no service listening there. In the Microsoft Winsock implementation of TCP, a pending connection will keep attempting to issue SYN packets until a maximum retry value is reached (set in the registry, this value defaults to 3 extra times)...
The value used to limit those retries is set in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpMaxConnectRetransmissions according to the same article. At least in Win10 Pro it doesn´t seem to be present by default.
Although this is a conveniece for Windows machines, an application still should determine its own criteria for handling a failed connect attempt IMO (i. e number of attempts, timeouts etc).
Anyhow, as I said, surprising fact! Living and learning I guess ...
Cristian.
I have a small portable tool which connects to around 150 servers at diverse locations to get a quick status check from them. It is important to get the status for all servers back to the user relatively quickly so the tool connects to the servers in parallel using non-blocking connect, and uses select() to determine when each socket is ready. The use of select() is fairly straightforward, and the tool is failure mature now and works well on Linux. It runs on windows xp, but connections to the vast majority of the servers out there do not complete. The tool staggers the calls to connect to avoid creating what looks like a SYN flood. It connects to one server about 100 msecs. I also have a check in place to ensure FD_SETSIZE is not violated. I have anecdotal evidence from someone else that the behaviour is better on later windows versions, but have not been able to verify.
I have used WinDump to verify that the syn packets are being sent, and I can see ack packets coming back, but select() keeps returning zero, and the code simply can't connect to most of the servers that do exist, and I can connect to just fine with the same code on Linux.
Has anyone seen and or solved any similar issues with many non-blocking connects and select on Windows XP?
After another day or so of digging I seem to have found the answer. On windows XP SP2 there is a limit of 10 concurrent connecting sockets system wide. If 10 or more half open connections exist, a System event is logged noting that the limit has been reached, and new connecting sockets are throttled silently. The System Event number is 4226.
I have fixed my code by adding version checks for Windows XP, and throttled to less than 10 connections on those systems. So far I have no reports of other versions being affected.
Sometimes my media server does not update its database, the only way to fix it is to restart the daemon.
I would like users to be able to simply run an executable or script to do so without revealing any login information to them (which might result in inadvertent headaches).
I feel the easiest way to do this would be to have an executable/script located on a samba share which they can run. The users would be running Windows or Macs which are all on the same local network as the server.
Write up a small CGI script or web app that has a button that when pushed restarts the daemon.
You just need some communication between a privileged daemon (or frequent cron) and the CGI script. Have the CGI script create a file, if the daemon or cron finds it have it restart the service and unlink the file. Primitive, but effective.
As Ignacio said, there are many, many lightweight HTTP servers that support CGI.
I am writing a script that powers on a system via network. And then i need to run a few commands on the other host. How do I know whether the system has powered on?
My programming language is Perl and the target host is RHEL5.
Is there any kernel interrupt or network boot information that indicates the system has powered on and the os has loaded?
[In a different scenario] I was also wondering just in case if i just switch on my Machine manually. when is it exactly said to have powered on. and when is the OS is supposed to have booted completely for a network related operation such as executing a network command there. What if the system is on DHCP how would a remote system then search for this machine [i guess it is possible via mac address. but if i am wrong ].
If I have missed out any info please feel free to ask me. If you have any suggestions to make the task easier please surface them :)
thanx
imkin
Well, I'd say the system is booted when it can perform the request you've made of it. That is, the sshd daemon is running. That's booted sufficiently for your purposes (I assume - substitute for whatever daemon you really need).
So, I'd send the power-on signal, and check back every 15-30 seconds to see if I could connect. If I've failed to connect within whatever is a reasonable time for that machine (2 minutes or 5 minutes or whatever), then I'd send an alert to the IT support team. Well, I'd send it to myself first, and only once I've investigated a few failures or so and found them to all be legitimate would I start sending it directly to IT.
DHCP is kind of a different question. You'd have to start learning about broadcasting, or having a daemon on that machine "call home" during boot to register its current IP address. And it would have to "call home" every time a DHCP renewal changed its IP address. This is decidedly more convoluted. Try to avoid DHCP on such server machines if at all possible.
On the rebooting machine you can install a script in your crontab with the special #reboot assertion (see man 5 crontab). That script could send a notification of some kind to the other machine, notifying it that it's up now.
I think checking for sshd sounds like a good approach.
As for the DHCP problem: if the other computer is on the same subnet you can look it up by MAC address using Net::ARP.
How about adding a script to the remote machine which gets run on startup to have it tell you when it is ready.
What is the best way to access a running mono application via the command line (Linux/Unix)?
Example: a mono server application is running and I want to send commands to it using the command line in the lightest/fastest way possible, causing the server to send back a response (e.g. to stdout).
I would say make a small, simple controller program that takes in your required command line arguments and uses remoting to send the messages to the running daemon.
This would be similar to the tray icon controller program talking to the background service that is prevalent in most Windows service patterns.
Mono's gsharp tool is a graphical REPL that lets you Attach to Process.
#Rich B: This is definately a suitable solution, which I already had implemented - however on the server I have to use, the remoting approach takes around 350ms for a single request.
I've measured the time on the server side of the request handling - the request is handled in less than 10ms, so it has to be the starting of the client program and the tcp connection, that takes up the time.
Hence the hope that I can find another way to post the requests to the server application.
You can use the system.net.sockets abstractions to create a service on a TCP port, and then telnet to that port.
Check the library status page; Mono's coverage here is a bit patchy.