Can't get to server select on EQEMU

[WongRomancer]

Anyone else having trouble logging in?

[Trusin]

Yes, and while we can login to the site which should mean we should be able to see the servers up I noticed the forums are running really slow to load at times and I tried to download the update recently and it is hanging. The traceroute looks like this so I think there is a problem somewhere on their end.

traceroute to project1999.org, 30 hops max, 38 byte packets
1 45-16-4-1.lightspeed.rcsntx.sbcglobal.net 2.067 ms 3.990 ms 9.408 ms
2 * * *
3 12.242.112.2 (12.242.112.2) 5.825 ms 3.639 ms 8.150 ms
4 4.68.39.1 (4.68.39.1) 3.346 ms 3.072 ms 3.216 ms
5 * ae-17-3507.ebr3.SanJose1.Level3.net (4.69.209.78) 51.123 ms *
6 FIBERTECH-N.edge2.NewYork6.Level3.net (4.30.181.126) 53.454 ms 55.872 ms 53.393 ms
7 ae3-nycmny83jp1.lightower.net (144.121.35.38) 61.514 ms * 60.947 ms
8 144.121.109.127.lightower.net (144.121.109.127) 54.738 ms 54.179 ms *
9 * * *
10 * ae0-brtnmas6j81.lightower.net (65.254.31.58) 57.967 ms 58.816 ms
11 144.121.7.234.lightower.net (144.121.7.234) 59.500 ms 61.743 ms 61.003 ms
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

[Mcolaluca]

It's not working for me either

[phr0stbyt3]

I haven't been able to connect all day and have also noticed slow forum load times.

[gn0sys]

Here's my tracert results for eqemulator and p99
Microsoft Windows [Version 10.0.18363.836]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\*******>tracert eqemulator.net

Tracing route to eqemulator.net [144.121.19.170]
over a maximum of 30 hops:

1 3 ms 3 ms 3 ms dsldevice.attlocal.net [192.168.1.***]
2 24 ms 27 ms 23 ms 75-27-136-1.lightspeed.*******.sbcglobal.net [********]
3 32 ms 26 ms 29 ms 71.149.76.41
4 * * * Request timed out.
5 39 ms * 46 ms 12.81.225.237
6 49 ms * 43 ms 12.123.18.233
7 31 ms 34 ms 65 ms 4.68.39.1
8 * 115 ms 94 ms ae-17-3507.ebr3.SanJose1.Level3.net [4.69.209.78]
9 91 ms 98 ms * FIBERTECH-N.edge2.NewYork6.Level3.net [4.30.181.126]
10 87 ms 92 ms 86 ms ae3-nycmny83jp1.lightower.net [144.121.35.38]
11 81 ms * * 144.121.109.127.lightower.net [144.121.109.127]
12 * * * Request timed out.
13 * * * Request timed out.
14 * 86 ms * 144.121.7.234.lightower.net [144.121.7.234]
15 * * 97 ms 144.121.19.170.lightower.net [144.121.19.170]

Trace complete.

C:\Users\*******>


Tracing route to project1999.com [144.121.19.170]
over a maximum of 30 hops:

1 4 ms 3 ms 6 ms dsldevice.attlocal.net [192.168.1.***]
2 24 ms 22 ms 22 ms 75-27-136-1.lightspeed.*****.sbcglobal.net [********]
3 24 ms 24 ms 24 ms 71.149.76.41
4 58 ms 24 ms 25 ms 75.8.128.104
5 26 ms 27 ms 26 ms 12.81.225.237
6 35 ms 31 ms 30 ms 12.123.18.233
7 62 ms 46 ms 45 ms 4.68.39.1
8 80 ms 79 ms 81 ms ae-17-3507.ebr3.SanJose1.Level3.net [4.69.209.78]
9 * 81 ms 81 ms FIBERTECH-N.edge2.NewYork6.Level3.net [4.30.181.126]
10 * 89 ms 103 ms ae3-nycmny83jp1.lightower.net [144.121.35.38]
11 * * 80 ms 144.121.109.127.lightower.net [144.121.109.127]
12 * * * Request timed out.
13 87 ms 86 ms 88 ms ae0-brtnmas6j81.lightower.net [65.254.31.58]
14 * 88 ms 85 ms 144.121.7.234.lightower.net [144.121.7.234]
15 * 88 ms 87 ms 144.121.19.170.lightower.net [144.121.19.170]

Trace complete.

[Grimstrike]

Things look fine now, there are 1389 on Green, 949 on Blue and 50 on Red

[gn0sys]

I just tried again and am getting a timeout. There was some weirdness with the internet around the time I tried to connect earlier.

[Donmega]

I cant log in either. When i am able to even get to the server list it is empty.

[jolanar]

I've been having this issue all day. Haven't been able to connect. Made it to character select a few times but got DCed before getting in.

[gn0sys]

I made it to select and also god DC'ed. I'm in TX BTW

[Trusin]

The culprit seems to be lightower.net

probably is only affecting those of us being routed through that IP

[gn0sys]

Yeah. I checked their status and they say they have no issues on the status page

[Leviothan]

Maybe a Texas thing? Is anyone else in TX that this is happening. I noticed someone else say they're in TX and experiencing this.

[Bodybagger]

Same issue, if the culprit is lightower.net what is the solution?

Everything fine on my end but cannot contact servers still.

[Mcolaluca]

I am in Tx as well. I will either see a blank server log in, or it will just time out

[gn0sys]

I may get a wireshark capture, but honestly, It's downstream from my ISP so I don't know who to contact. I would assume other people are seeing similar latency and not just on p99, but their normal internet may be slow.
As well, there is no way for us users to change how the traffic is routed. It'
s all determined by the ISP's lightower.net is part of Crown Castle fiber networks. Essentially, we're at the mercy of their router(s) and I've worked with enough networking guys to know that before they fix the issue, they will deny the issue is their network for s few hours to possibly a day before they actually look at it and find it was indeed their network.
There's a haiku:
It's not DNS

There's no way it's DNS

It was DNS.

[Trusin]

I used openvpn to try to get it to route through a different source and it didn't work. Same problem.

[qbearwa]

For the past 3 days several players have been having issues with the game. Let me run through several scenario's. Starting from login, either it times out, loads into server page and its blank or loads into character screen then fails to load into game. If we manage to load into the game it crashes 98% of time when zoning, porting or deaths. At which point the cycle begins anew. I've tried a number of things and still unable to resolve what's happening. I did a "tracert" for login.eqemulator.net and it keeps failing at lightower.net 144.121.7.234.lightower.net. When the game was not giving me issues and did a "tracrt" for eqemulator.net it completed 19 hops just fine. I spoke with my isp and they claim everything is fine and my tracert confirms as much and the ping usual peaks in the 60. Lastly, I am not sure if this is related or not, but windows did have an update a few days ago. So today I tried doing a compatibility fix but it didn't have any effect. FYI I'm on blue server.

[gn0sys]

Here's the latest traceroute I did to eqemulator.net

1 4 ms 3 ms 6 ms dsldevice.attlocal.net [192.168.1.254]
2 23 ms 27 ms 24 ms 75-27-136-1.lightspeed.austtx.sbcglobal.net [75.27.136.1]
3 24 ms 23 ms 23 ms 71.149.76.41
4 * * * Request timed out.
5 25 ms 27 ms 26 ms 12.81.225.241
6 30 ms 32 ms 34 ms dlstx401igs.ip.att.net [12.123.16.77]
7 30 ms 30 ms 30 ms ae-9.a00.dllstx04.us.bb.gin.ntt.net [129.250.8.237]
8 30 ms 30 ms 30 ms ae-10.r25.dllstx09.us.bb.gin.ntt.net [129.250.4.201]
9 66 ms 66 ms 66 ms ae-8.r23.snjsca04.us.bb.gin.ntt.net [129.250.4.154]
10 66 ms 66 ms 66 ms ae-41.r02.snjsca04.us.bb.gin.ntt.net [129.250.6.119]
11 67 ms 66 ms 67 ms ae-1.a02.snjsca04.us.bb.gin.ntt.net [129.250.3.103]
12 116 ms 182 ms 71 ms ae-7.akamai.snjsca04.us.bb.gin.ntt.net [130.94.195.14]
13 67 ms 66 ms 67 ms a23-217-138-110.deploy.static.akamaitechnologies.com [23.217.138.110]

71.149.76.41 is an IP in Redmond WA
12.81.225.241 is an IP in Morristown NJ. The hop between those two looks to be like the problem.

[gn0sys]

The lightower IP's seem to be in Massachusetts. As a test, I am tethering my laptop to my phone. I'll try to connect. and report back here.

[gn0sys]

Got right on in while tethered.

15 106 ms 79 ms 84 ms ae3-nycmny83jp1.lightower.net [144.121.35.38]
16 92 ms 83 ms 110 ms 144.121.109.127.lightower.net [144.121.109.127]
17 92 ms 81 ms 80 ms 160.72.248.59.lightower.net [160.72.248.59]
18 156 ms 76 ms 93 ms ae0-brtnmas6j81.lightower.net [65.254.31.58]
19 151 ms 75 ms 109 ms 144.121.7.234.lightower.net [144.121.7.234]
20 145 ms 89 ms 78 ms 144.121.19.170.lightower.net [144.121.19.170]

[Leviothan]

Got right on in while tethered.

15 106 ms 79 ms 84 ms ae3-nycmny83jp1.lightower.net [144.121.35.38]
16 92 ms 83 ms 110 ms 144.121.109.127.lightower.net [144.121.109.127]
17 92 ms 81 ms 80 ms 160.72.248.59.lightower.net [160.72.248.59]
18 156 ms 76 ms 93 ms ae0-brtnmas6j81.lightower.net [65.254.31.58]
19 151 ms 75 ms 109 ms 144.121.7.234.lightower.net [144.121.7.234]
20 145 ms 89 ms 78 ms 144.121.19.170.lightower.net [144.121.19.170]

Me too. I was able to login while tethered but my signal is awful where I am. Can't play on that. I imagine this can't be a p99 specific thing.

[Sinphera]

I am also having the same issue. Also located in TX. Is there any reasonable solution to getting onto the game?

[Kawhi]

Also TX and having same issue.

I've seen in the past where something like this happened and everyone happened to be on AT&T internet. Are you all on AT&T as well?

[gn0sys]

Apart from changing your ISP? Not anything that can be done on your end. We'a re at the mercy of lightower. I'm on ATT.

[gn0sys]

Just checked ATT status.

Heads up! We found service issues in your area.
Customers affected: 200+
Looks like there's a TV, internet, or digital home phone problem in your area.
Sign into see if it affects your service.

[Mcolaluca]

I am on att as well, however I dont have any other issues besides trying to log into p99

[qbearwa]

I just tried the tether to phone method and I got right in.. Zoned smoothly as well. logged off and back on with no issues. Here are the trace results.. so know im confused.. same lightower results from tracert as I got on my wired connection through ISP.. not AT&T..

Tracing route to login.eqemulator.net [144.121.19.169]
over a maximum of 30 hops:

1 2 ms 2 ms 5 ms 172.20.10.1
2 41 ms 55 ms 45 ms 10.198.43.41
3 44 ms 40 ms 38 ms 10.0.145.1
4 69 ms 39 ms 39 ms 10.198.43.17
5 53 ms 50 ms 48 ms 10.162.189.206
6 41 ms 141 ms 50 ms 10.162.189.209
7 58 ms 43 ms 39 ms 10.164.176.7
8 89 ms 51 ms 60 ms ae2.mpr2.las1.us.above.net [208.185.88.149]
9 120 ms 115 ms 118 ms ae19.cs2.lax112.us.zip.zayo.com [64.125.26.6]
10 66 ms 165 ms 64 ms ae21.mpr1.slc2.us.zip.zayo.com [64.125.26.19]
11 163 ms 79 ms 201 ms ae4.mpr2.slc2.us.zip.zayo.com [64.125.26.165]
12 221 ms 114 ms 150 ms ae11.cs1.den5.us.zip.zayo.com [64.125.26.42]
13 * * * Request timed out.
14 * * * Request timed out.
15 121 ms 139 ms 129 ms ae8.mpr3.bos2.us.zip.zayo.com [64.125.30.139]
16 114 ms 109 ms 107 ms 209.66.121.142
17 142 ms 125 ms 146 ms ae3-bstpmalljp1.lightower.net [144.121.35.37]
18 123 ms 115 ms 123 ms 160.72.248.57.lightower.net [160.72.248.57]
19 121 ms 112 ms 113 ms ae0-brtnmas6j81.lightower.net [65.254.31.58]
20 120 ms 130 ms 125 ms 144.121.7.234.lightower.net [144.121.7.234]
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

[jolanar]

Weird, also on ATT and also in TX.

No problems this morning though.

[gn0sys]

I played this morning too, nut it looks like it's broken again. ATT is the problem.

[gn0sys]

Pinging login.eqemulator.net [144.121.19.169] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.

[Adron]

Having exactly same problems on google fiber in KCMO.
pings to project1999.org drop rate 11%.
pings to the ip ^ resolves to, 144.121.19.170, drop rate 9%.
if it was just a DNS issue, I don't think that last one would be true.

So it's not just AT&T, unless google fiber's backbone has to hop through AT&T to get to lightower...

my tracert goes through google fiber, gtt.net, then lighttower

[xrend]

Hey guys, I'm currently having this issue... I don't have an issue connecting to anything else, just login.eqemulator.net - I have changed DNS to google, I can't ping... Something is definitely funky with the internet.

[xrend]

So I'm still unable to login. I'm located in Hawaii (Hawaii Telecom). I was playing yesterday just fine, and I zoned out from Sol Ro, crashed, and couldn't get back on to the server. Before I crashed everything was working fine. Now I am unable to ping the server. DNS resolves just fine. Traceroute does not complete.

This looks to be a routing issue, not DNS. I assume that the route was lost for some providers or a duplicate IP address is being leaked from somewhere else. The reason that I was able to play and then when I crashed I suddenly lost the ability to connect is because active routes are cached in most ISP routers to lower the CPU load. Once my active connection was broken (crashed) the route was effectively unlearned from cache and the new route lookup failed as the route was no longer being properly advertised.

Unfortunately, this is going to be a problem that only the devs can correct by talking to their ISP and ensuring their public IP is being advertised correctly and that it's not showing up in routing tables elsewhere. It's not going to affect everyone, just certain ISPs are not getting the route leaked or are getting a duplicate route leaked (which could explain the packet loss in the above users response) so this will affect only users that are on certain backbones. It's very unfortunate.

[xrend]

UPDATE: I have confirmed that there are widespread routing issues happening all over the US right now. I assume this has something to do with COVID and ISP workers WFH/not having direct access to routers/switches. I no longer have routes to Discord voice RTC, and quite a few other routes are flapping, including google DNS 8.8.8.8.

Hopefully this isn't a sign of worse things to come.

[Valewind]

I am also having the same issues. I also have AT&T...

I died twice today because of the unusual lag that I experienced. Anyone know if the GM's would be willing to give me back the exp I lost due to what is clearly a systemic internet connectivity issue?

[mcoy]

The GMs won't rez you but I will. 96% on Blue or 90% on Green. Just send me a PM and tell me where to go.

-Mcoy