Lag debugging

[Trompete]

OK. I did some capturing and replay of network data last night on my LAN and discovered a few things:

1. The minimum time from client->server is 16 ms, and it averages that over 107 pings
2. The minimum time from server->client is 16 ms, but it averages 26 ms over 107 pings
3. The client gets behind by 16 frames per ping. This is very perplexing and is likely a big source of syncing problems.

[Trompete]

The good news is I've got the clocks synched up between the client and server, and I've got all incoming messages to the client converted from server timestamp to client timestamp, so I know EXACTLY how old they are. Now I just need to adjust the lag-reading places in the code to use that age to adjust the packet instead of the average lag from server->client. It turns out there is a big difference in lag from packet to packet.

[NiteHawk]

The good news is I've got the clocks synched up between the client and server, and I've got all incoming messages to the client converted from server timestamp to client timestamp, so I know EXACTLY how old they are. Now I just need to adjust the lag-reading places in the code to use that age to adjust the packet instead of the average lag from server->client. It turns out there is a big difference in lag from packet to packet.

Ahh so that might be desyncing it then by sending the average lag since the lag fluctuates?

[Trompete]

The good news is I've got the clocks synched up between the client and server, and I've got all incoming messages to the client converted from server timestamp to client timestamp, so I know EXACTLY how old they are. Now I just need to adjust the lag-reading places in the code to use that age to adjust the packet instead of the average lag from server->client. It turns out there is a big difference in lag from packet to packet.

Ahh so that might be desyncing it then by sending the average lag since the lag fluctuates?

Yes, and it looks like the code doesn't really compensate for lag that much in some cases. I started looking at FIRETUBES before I went to sleep, and I noticed that the lag is not used to figure out how far the torp has gone since the server said it had been fired.

frameUsec = how long a frame lasts in usec (16500)
pktAge = Age of packet in uSec since it was created on server
x,y = where server said the torp was fired
xvel, yvel = server-assigned torp velocity
lagXdiff = xvel * (pktAge/frameUsec)
lagYdiff = yvel * (pktAge/frameUsec)

What I'm saying is going wrong is the initial position of the torp is being assigned as x,y instead of x+lagXDiff, y+lagYdiff, so I think the torps are always out of place.

I'm not sure if that's what other people are seeing as well. We should move this topic to dev discussion to get more descriptions of what the game looks like when it gets out of sync.

[NiteHawk]

Done ;P

[Trompete]

Ok. I made huge strides in terms of lag management in the last couple days. Now all player/AI ship updates are applied with exact precision based on how long ago they were sent on the client/server. This makes the AI players move smoothly and consistently across the screen. You can even hit them with torps and have them explode

The player management is a different matter. I've tried having both the client manage its own position (current) and the server manage the clients' positions. Both have merits and costs.

Having the client manage its position gives the smoothest experience for the client since they are just notifying the server as to where they were 30ms ago or whatever the lag is. The bad part is if they have a slow/laggy connection, they will skip across everybody else's screen. Thanks to the new lag management code, this will be a smooth skip, but still.

Have the server manage the clients' positions has its own problems. The benefit of this scheme is that you are always seeing what the server is seeing, and that is always consistent. The bad part is that the controls are as sluggish and unresponsive as your connection, which makes the game unplayable at 100ms or higher. Also, the small corrections in position due to average lag vs actual lag make small course corrections (a few pixels per update) that make the screen appear to twitch.

As you can see, it's getting better, but there is still a ways to go.


On a side note, I found a funny bug that I'll call "quantum leap". I won't say more about it, but I'm curious if anybody else knows what I'm talking about

[Sandtrooper]

On a side note, I found a funny bug that I'll call "quantum leap". I won't say more about it, but I'm curious if anybody else knows what I'm talking about

Yep I've encountered this bug, first time testing 2.0x's intranet play about 2-3 months ago.

[NiteHawk]

Trompete, did I ever tell that your my heeero.

I'll be releasing in a couple days. Almost done hulls and modifying the modules, aka split up the power vs capacity. But damn we need to redo the modules/ships and rebalance them. Next task after ranks.

[Trompete]

And now the bot positions are adjusted based on super-accurate timestamps generated by the server. Even the 60-bot botmogeddon scenario looks nice and smooth over the network