Server network connection upgrade info

[Borch]

INFO FROM NFOSERVERS

Between 2:30am and 5am CDT on Tuesday, June 30, we will be performing maintenance in Chicago in order to upgrade our connection to the Equinix peering exchange from 20 Gbps to 100 Gbps -- increasing our DDoS mitigation capabilities at this location. This will be a fully scripted maintenance that will involve gracefully shifting traffic off of the old links, taking down one of the routers, making hardware changes, and bringing everything back online.

As a result of the maintenance, we expect for you to see brief periods of routing reconvergence or lost connectivity. However, with a significant operation such as this, there is always the possibility that a misconfiguration or physically bumped cable will lead to longer downtime event. We will be doing our best to be as careful as possible, and to have the on-site technician be as careful as possible.

*****************
Update @ 5:02am CDT on 6/30: Our side of this upgrade, and all hardware changes on both sides, have been completed. After Equinix turns up the interface on their end, peerings will be reestablished and traffic will be back to flowing as normal for customers. We expect no further downtime.

(The maintenance steps were performed smoothly, and as far as we can tell, customers did not see any dropped traffic from the event.)

************************
Update @ midnight CDT on 7/1: Equinix told us previously they they were ready to process the turn-up of the new interface, but they've backpedaled now and say that it won't happen for another week at minimum. In the meantime, this location will continue to hum along with just our 300 Gbps of transit links. Some customers may see different paths than usual due to the peering options being offline (many Canadian customers, for instance, will take a normal backbone to reach us instead of reaching directly).

So considering they have 300 Gbps already and will be moving to 400 Gbps, that is a 25% increase in backbone capacity, which may or may not improve things. Evidently using Equinix will allow for different routing to the server (sounds like hopping onto the Equinix backbone would be like getting in an express lane on a highway, especially for Canadians) which should improve the intermittent hops and reduce ping/lag. But we will see. They say that they are expecting this to happen no sooner than a week and possibly later than this.
It would be premature to do anything about the server before this all settles out, IMHO.

So the options would be, if we find the server inadequate after it has settled out, to relocate back to Atlanta and/or use Knights server.

[Auburnman]

I agree that we should hold off on making any drastic changes ahead of this work. More bandwidth is never a bad thing. Hopefully rerouting (if there is any) won’t hurt anyone.

Borch will you be notified when the upgrade is complete?

[Borch]

Note that I discovered that I had one of the settings on the command line (the port) repeated a large number of times, but the game appeared to be running, but it is possible that this was affecting gameplay.

Message sent to NFO:**************
We are continuing to have lag spikes that hit our server occasionally, clients complain that our server seems to have troubles they do not see on other servers. I've scoured my server to try to find anything that might be contributing to lag and haven't really found anything, which makes me think that the location in Chicago has been much of the source of the problem.

I have run mtr and there are spikes that have occurred between my home and the server, as well as with some other players whom I have had run it. I've also run it on the server to the clients where they have a ping, tend to see some hops with intermittent or more regular ping increases. Nothing seems to be close in to the server, though.

The problems we have had seem to have come about at the same time we moved to Chicago, though that unfortunately corresponded to the increase in internet traffic due to the coronavirus shut down and the increase in network loading due to zoom, etc.

Also, we had a notification from you folks at one time that we were under a denial of service attack, which you were intercepting. Is this something that might still be ongoing?

Any other ideas?

We were also notified about the Equinix upgrade and wondering if that will be of help? When will that be complete?
So we are thinking that the solution might be in returning to Atlanta, but are waiting until the Equinix upgrade is complete.
*******************************
Reply.
The Equinix upgrade should be complete now, as of 7/2. When was the most recent set of spikes that was seen? Chicago overall sees the least issues as it has enough capacity to absorb almost all attacks.
*****************************

[Borch]

I have verified that there is no swap going on and have reduced the memory appropriated to the game.
Also discovered that I no longer have the game logs on the solid state drive, so move them over there today (reduces the possibility of a glitch when b3 reads the logs).

[Shartnado]

Good stuff Borch. Maybe you'll uncover the root cause. Hoping so. But THANK YOU.

[Borch]

Looked at the scripts that start b3, made sure that it isn't possible for b3 to have more than one running at a time.

Chatting with Auburn, he mentions that capturing the flag, sometimes picking up the flag induces lag spikes. I notice that when the lagometer has a yellow blip, the blue triangles that follow become taller and then over time, shrink down in height—I do not know what that specifically indicates. Makes me wonder if there is a problem with data getting from the clients to the server for a time, and then it tries to "catch up."

So with that in mind, I have turned down cl_maxpackets to 50, which might help with that. But will see what it looks like.

Posted a query on patmansan's site about it, if he has any idea. Will need to think about other things could be affecting it. Note that they do have a method by which they intercept denial of service (DOS) attacks, but I did not get a straight answer to my question about if we still had one (though he said that we have enough bandwidth at Chicago that a DOS attack would not overwhelm the network, but do not know how that will affect the server.

[Auburnman]

Did a survey of a few of the least laggy (my experience) US based Cod2 servers. Screenshot is attached. Nothing really jumps out. Most are based in ATL, CHI or NY and many are using InfoServers although we're the only ones on NFO and in Chi.

None of the others are running punk buster and about half are running anti-lag. There's also a fairly even split between Extreme 2.9 and 3.0. Of course there are many more server variables than what's available publicly but I don't see anything here that makes us stand out from the others. FWIW my ping was 45-50 for all of them.

CoD2 Surver Survey.png (99.01 KiB) Viewed 18766 times

[Borch]

Was talking with Auburn about the Lagometer, which does not have any documentation that I can find associated with COD2, but I do know that COD2 is based on the Q3A engine, which is open source, so I went to the folder where I have that and searched for lagometer and found this description:

The Lagometer is the graph in the lower right corner of the status display. It is a test function and not a part of regular game play. The upper graph (blue/yellow) slides one pixel for every rendered frame. Blue lines below the baseline mean that the frame is interpolating between two valid snapshots. Yellow lines above the baseline mean the frame is extrapolating beyond the latest valid time. The length of the line is proportional to the time.
The lower graph (green/yellow/red) slides one pixel for every received snapshot. By default, snapshots come 20 times a second, so if you are running >20 fps, the top graph will move faster, and vice versa. A red bar means the snapshot was dropped by the network. Green and yellow bars are properly received snapshots, with the height of the bar proportional to the ping. A yellow bar indicates that the previous snapshot was intentionally suppressed to stay under the rate limit.
The upper graph indicates the consistency of your connection. Ideally, you should always have blue bars of only a pixel or two in height. If you are commonly getting big triangles of yellow on the graph, your connection is inconsistent.
In a heavy firefight, it is normal for modem players to see yellow bars in the bottom graph, which should return to green when the action quiets down. If you are getting several red bars visible, you may want to look for a server that drops less packets.
There are a few tuning variables for people trying to optimize their connection: The most important one is "rate", which is what the connection speed option in the menu sets. We are fairly conservative with the values we set for the given modem speeds: 2500 for 28.8, 3000 for 33, and 3500 for 56k. You may actually be connecting faster than that, and modem compression may be buying you something, so you might get a better play experience by increasing the values slightly. If you connect at 50000 bps, try a rate of 5000, etc. Quake 3 errs on the conservative side, because too low of a rate will only make the movement of other things in the world choppy, while too high of a rate can cause huge amounts of lag.
Note that the optimal rate will be somewhat lower than a rate for QW or Q2, because Quake 3 now includes the UDP packet header length in the bandwidth estimate. You can ask for a different number of snapshots by changing the "snaps" variable, but there isn't a lot of benefit to that. Dedicated servers run at 40hz, so stick to divisors of that: 40, 20 (default), 10. A "snaps" of 40 will usually just cause you to hit your rate limit a lot faster. It may be useful for tuning rate, if nothing else.
You can adjust the local timing point with "cg_timenudge ", which effectively adds local lag to try to make sure you interpolate instead of extrapolate. If you really want to play on a server that is dropping a ton of packets, a timenudge of 100 or so might make the game smoother.

I am guessing it is the similar for COD2. Snaps in COD2 is set to 20. COD2 does not run at 40 Hz for dedicated servers.

[Auburnman]

That's fascinating.

What variables are dictated by our server besides max frame rate (250) and max packets (?)?

[Borch]

Main server refresh variable is the server fps, which is set to 20. The rate at which updates are sent out to clients is 1/fps seconds. So since we have 20 fps, every 1/20 second (50 msec) the server sents out a sequence of messages to the clients, each message taking an average about 96µsec, so with 20 clients, it takes about 2 msec. I have not seen any delay in these transmissions in the packets I have sampled. I put a 30 minutes sampling of network traffic onto your account directory, Auburn.

The amount of data incoming is determined by the cl_max_packets. It was set to 100, I have set it to 50, which is 2.5 times that of the fps of the server. I notice that since I have done this, the pings are more demonstrative of the actual pings that players have.