I am working on an application which, through a Java program, links two different robot simulation environments. One simulation environment (let's call it A) sends the current state of the robot to the Java application, which does some calculations and then sends data about this current state, as well as some other information, on to the other simulation environment (let's call it B). Simulation B then updates the state of the robot to match Simulation A's version.
The problem is that as the program continues to run, simulation B begins to lag behind what simulation A is doing. This lag increases continuously, so that after a minute or so simulation B is several seconds behind.
I am using TCP sockets to send data between these environments and the Java program. From background reading on socket programming, I found out it is bad practice to continuously open and close sockets rapidly, so what I am doing currently is just keeping both sockets open. I have a loop running which grabs data from Sim A, does some calculations, and then sends the position data to Sim B and then I have the thread wait for 100ms and then the loop repeats. To be clear, the position data sent to B is unaltered from what is received from A.
Upon researching the lag issue, someone suggested to me that for streams of data it is actually a good idea to open and close sockets, because if you keep the socket open, if one simulation takes a longer time to process things than the other, you end up with the position data stacking up in the buffer and being read sequentially, instead of reading the most recent data. Is this true? Would rewriting my code to open and close sockets every 100ms potentially get rid of the delay? Or is this not how sockets actually work?
Edit for clarification: It is more critical that the simulations stay in sync than that all position data is sent, in other words it is acceptable to not pass along all data points for the sake of staying in sync.
Besides keeping the socket open causing problems, does anyone have any ideas of what might be causing the lag issue?
Thanks in advance for any insight/suggestions/hints!
You are correct about using a single connection. Data can indeed back up, but using multiple connections doesn't change that.
The basic question here is whether the Java program can calculate as fast as the robot can send data. If it can't, it will get behind. You can do various things to the networking to speed it up but if the computations can't keep up they are futile. So you need to investigate your timings.
Related
I'm building a HTML5 / Websockets based multiplayer canvas game for Facebook and I've been working on the server code for a few days now. While the games pretty simple with a 2d top down, WSAD controls and mouseclick fires a projectile to the cursor x/y - I've never had to do real-time multiplayer before. I've read a few great documents but I'm hoping I can overview my general understanding of the topic and someone can validate the approach and/or point out areas for improvement.
Authoritative multiplayer server, client-side prediction and entity interpolation (and questions below)
Client connects to server
Client syncs time to server
Server has two main update loops:
Update the game physics (or game state) on the server at a frequency of 30 per second (tick rate?)
Broadcast the game state to all clients at a frequency of 10 per second
Client stores three updates before being allowed to move, this builds up the cache for entity interpolation between update states (old to new with one redundency in case of packet loss)
Upon input from the user, the client sends input commands to server at a frequency of 10 per second - these input commands are time stamped with the clients time
Client moves player on screen as a prediction of what the server will return as the final (authoritative) position of client
Server applies all updates to its physics / state in the previously mentioned update loop
Server sends out time stamped world updates.
Client (if is behind server time && has updates in the queue) linearly interpolates the old position to the new.
Questions
At 1: possibility to use NTP time and sync between the two?
At 5: time stamped? Is the main purpose here to time-stamp each packet
At 7: The input commands that come in will be out of sync per different latencies of the clients. I'm guessing this needs to be sorted before being applied? Or is this overkill?
At 9: is the lerp always a fixed amount? 0.5f for example? Should I be doing something smarter?
Lots of questions I know but any help would be appreciated!!
At 1 : You're a bit overthinking this, all you have to do in reality is to send the server time to the client and on that side increment that in your update loop to make sure you're tracking time in server-time. Every sync you set your own value to the one that came from the server. Be EXTRA careful about this part, validate every speed/time server-sided or you will get extremely easy-to-do but incredibly nasty hacks.
At 5 : Timestamped is important when you do this communication via UDP, as the order of the packets is not ensured unless you specifically make it so. Via websockets it shouldn't be that big of an issue, but it's still good practice (but make sure to validate those timestamps, or speedhacks ensure).
At 7 : Can be an overkill, depends on the type of the game. If your clients have large lag, they will send less inputs by definition to the server, so make sure you only process those that came before the point of processing and queue the remaining for the next update.
At 9 : This post from gamedev stackexchange might answer this better than I would, especially the text posted by user ggambett at the bottom.
I have problem reading from socket. There is asterisk instance running with plenty of calls (10-60 in a minute) and I'm trying to read and process CDR events related to those calls (connected to AMI).
Here is library which I'm using (not mine, but was pushed to fork because of bugs) https://github.com/warik/gami
Its pretty straightforward, main action goes in gami.go - readDispatcher.
buf := make([]byte, _READ_BUF) // read buffer
for {
rc, err := (*a.conn).Read(buf)
So, there is TCPConn (a.conn) and buffer with size 1024 to which I'm reading messages from socket. So far so good, but eventually, from time to time (this time may vary from 10 minutes to 5 hours independently of data amount which comes through socket) Read operation fails with io.EOF error. I was trying to reconnect and relogin immediately, but its also impossible - connection times out, so i was pushed to wait for about 40-60sec, and this time is very crucial to me, I'm losing a lot of data because of delay. I was googling, reading sources and trying a lot of stuff - nothing. The most strange thing, that simple socket opened in python or php does not fail.
Is it possible that problem because of lack of file descriptors to represent socket on mine machine or on asterisk server?
Is it possible that problem in asterisk configuration (because i have another asterisk on which this problem doesn't reproduce, but also, i have time less calls on last one)?
Is it possible that problem in my way to deal with socket connection or with Go in general?
go version go1.2.1 linux/amd64
asterisk 1.8
Update to latest asterisk. There was bug like that when AMI send alot of data.
For check issue, you have send via ami command like "COMMAND sip show peers"(or any other long output command) and see result.
Ok, problem was in OS socket buffer overflow. As appeared there were to much data to handle.
So, were are three possible ways to fix this:
increase socket buffer volume
increase somehow speed of process which reeds data from socket
lower data volume or frequency
The thing that gami is by default reading all data from asterisk. And i was reading all of them and filter them after actual read operation. According that AMI listening application were running on pretty poor PC it appeared that it simply cannot read all the data before buffer capacity will be exposed.But its possible to receive only particular events, by sending "Events" action to AMI and specifying desired "EventMask".
So, my decision was to do that. And create different connections for different events type.
Let's say I have an internal system of 20+ nodes that pass data back and forth to each other through sockets where low latency is a high, high priority. How do I design it so that if a random server(s) goes down, I can recover/resend the data that was already sent but not processed by the downed server?
For example, if A was streaming data to B, but at some point B goes down without processing some of the data. If we assume A can detect that B went down, and reroute the data to C, how would I design it so that I know what data was sent to B, and now should be rereouted to C?
I'm assuming I'll have to rely on the various message queue software out there, but I'm wondering if there is also another easy way to do this!
The Situation:
I would like to ask what's the best logic for synchronizing objects in a multiplayer 1:1 game using BT or a web server. The game has two players, each of them has multiple guns & bullets, the bullets are created dynamically and disappear after a while, the players my move objects around simultaneously.
The Problem:
I have a real issue with synchronization, since the bullets on one device may be faster than other, also they may have already gone or hit an object on one device while on the other its still in the air.
Possibilities?
What is the best way of handling synchonization in this case? Should all the objects be controlled by one device acting as the server, while th other just gets the values, positions and does very little thinking. Or should control be distributed where each device creates, destroys and moves its own objects and then through synchronization tells the other device.
What is the best to handle transmission delay in this, since BT might be faster than playing over the web?
The best would be a working sample - thanks very much!
You seem to have started on some good ideas about synchronization, but it's possible there are two problems you are running into that are getting overlapped: the synchronization of game clocks and the sychronization of gamestate.
(1) synchronizing game clocks
you need some representation of 'game time' for your game. for a 2 player game it is very reasonable to simply declare one the authority.
so on the authoritative client:
OnUpdate()
gameTime = GetClockTime();
msg.gameTime = gameTime
SendGameTimeMessage(msg);
on the other client might be something like:
OnReceivGameTimeeMessage(msg)
lastGameTimeFromNetwork = msg.gameTime;
lastClockTimeOfGameTimeMessage = GetClockTime();
OnUpdate()
gameTime = lastGameTimeFromNetwork + GetClockTime() - lastClockTimeOfGameTimeMessage;
there are complications like skipping/slipping (ie getting times from over the network that go forward/backward too much) that require further work, but hopefully you get the idea. follow up with another question if you need.
note: this example doesn't differentiate 'ticks' vs 'seconds' nor does is it tied to your network protocol nor the type of device your game is running on (save the requirement 'the device has a local clock').
(2) synchronizing gamestate
after you have a consistent game clock, you still need to work out how to consistently simulate and propagate your gamestate. for synchronizing gamestate you have a few choices:
asynchronous
each unit of gamestate is 'owned' by one process. only that process is allowed to change that gamestate. those changes are propagated to all other processes.
if everything is owned by a single process, this is often called a 'client/server' game.
note, with this model each client has a different view of the game world at any time.
example games: quake, world of warcraft
to optimize bandwidth and hide latency, you can often do some local simulation for fields with a high update frequency. example:
drawPosition = lastSyncPostion + (currentTime - lastSyncTime) * lastSyncVelocity
of course you to having to reconcile new information with your simulated version in this case.
synchronous
each unit of gamestate is identical in all processes.
commands from each process are propagated to each other with their desired initiation time (sometime in the future).
in its simplest form, one process (often called the host) sends special messages indicating when to advance the game time. when everyone recieves that message they are allowed to simulate the game up to that point.
the 'in the future' requirement leads to high latency between input command and gamestate change.
in non-real time games like civilization, this is fine. in a game like starcraft, normally the sound acknowledging the input comes immediately, but the actually gamestate affecting action is delayed. this style is not appropriate for games like shooters that require time-sensitive actions (on the ~100ms scale).
synchronous with resimulation
each unit of gamestate is identical in all processes.
each process sends all other processes its input with its current timestamp. additionally a 'nothing happened' message is periodically sent.
each process has 2 copies of the gamestate.
copy 1 of the gamestate is propagated to the 'last earliest message' it has receive from all other clients. this is equivalent to the synchronous model, but has the weakness that it represents a gamestate from 'a little bit ago'
copy 2 of the gamestate is copy 1 plus all the remaining messages. it is a prediction of what is gamestate at the current time on the client, assuming nothing new happens.
the player interacts with some combination of the two gamestate (ideally 100% copy 2, but some consideration must be taken to avoid pops as new messages come in)
example games: street fighter 4 (internet play)
from your description, options (1) and (3) seem to fit your problem. again if you have further questions or require more detail, ask a follow up.
since the bullets on one device may be faster than other
This should not happen if the game has been architected properly.
Most games these days (particularly multiplayer ones) work on ticks - small timeslices. Each system should get the exact same result when it computes what happened during a tick - no "bullets moving faster on one machine than they do on another".
Then it's a much simpler matter of making sure each system gets the same inputs for each player (you'll need to broadcast each player's input to each other player, along with the tick the input was registered during), and making sure that each system calculates ticks at the same rate.
In order not to flood the remote endpoint my server app will have to implement a "to-send" queue of packets I wish to send.
I use Windows Winsock, I/O Completion Ports.
So, I know that when my code calls "socket->send(.....)" my custom "send()" function will check to see if a data is already "on the wire" (towards that socket).
If a data is indeed on the wire it will simply queue the data to be sent later.
If no data is on the wire it will call WSASend() to really send the data.
So far everything is nice.
Now, the size of the data I'm going to send is unpredictable, so I break it into smaller chunks (say 64 bytes) in order not to waste memory for small packets, and queue/send these small chunks.
When a "write-done" completion status is given by IOCP regarding the packet I've sent, I send the next packet in the queue.
That's the problem; The speed is awfully low.
I'm actually getting, and it's on a local connection (127.0.0.1) speeds like 200kb/s.
So, I know I'll have to call WSASend() with seveal chunks (array of WSABUF objects), and that will give much better performance, but, how much will I send at once?
Is there a recommended size of bytes? I'm sure the answer is specific to my needs, yet I'm also sure there is some "general" point to start with.
Is there any other, better, way to do this?
Of course you only need to resort to providing your own queue if you are trying to send data faster than the peer can process it (either due to link speed or the speed that the peer can read and process the data). Then you only need to resort to your own data queue if you want to control the amount of system resources being used. If you only have a few connections then it is likely that this is all unnecessary, if you have 1000s then it's something that you need to be concerned about. The main thing to realise here is that if you use ANY of the asynchronous network send APIs on Windows, managed or unmanaged, then you are handing control over the lifetime of your send buffers to the receiving application and the network. See here for more details.
And once you have decided that you DO need to bother with this you then don't always need to bother, if the peer can process the data faster than you can produce it then there's no need to slow things down by queuing on the sender. You'll see that you need to queue data because your write completions will begin to take longer as the overlapped writes that you issue cannot complete due to the TCP stack being unable to send any more data due to flow control issues (see http://www.tcpipguide.com/free/t_TCPWindowSizeAdjustmentandFlowControl.htm). At this point you are potentially using an unconstrained amount of limited system resources (both non-paged pool memory and the number of memory pages that can be locked are limited and (as far as I know) both are used by pending socket writes)...
Anyway, enough of that... I assume you already have achieved good throughput before you added your send queue? To achieve maximum performance you probably need to set the TCP window size to something larger than the default (see http://msdn.microsoft.com/en-us/library/ms819736.aspx) and post multiple overlapped writes on the connection.
Assuming you already HAVE good throughput then you need to allow a number of pending overlapped writes before you start queuing, this maximises the amount of data that is ready to be sent. Once you have your magic number of pending writes outstanding you can start to queue the data and then send it based on subsequent completions. Of course, as soon as you have ANY data queued all further data must be queued. Make the number configurable and profile to see what works best as a trade off between speed and resources used (i.e. number of concurrent connections that you can maintain).
I tend to queue the whole data buffer that is due to be sent as a single entry in a queue of data buffers, since you're using IOCP it's likely that these data buffers are already reference counted to make it easy to release then when the completions occur and not before and so the queuing process is made simpler as you simply hold a reference to the send buffer whilst the data is in the queue and release it once you've issued a send.
Personally I wouldn't optimise by using scatter/gather writes with multiple WSABUFs until you have the base working and you know that doing so actually improves performance, I doubt that it will if you have enough data already pending; but as always, measure and you will know.
64 bytes is too small.
You may have already seen this but I wrote about the subject here: http://www.lenholgate.com/blog/2008/03/bug-in-timer-queue-code.html though it's possibly too vague for you.