A call to (nerdy) arms

[MaximiusM]

So we are all aware of the present situation with 'DDoS' that is impacting the server and, it appears anyway, the larger EQEmu community. Is there anyone else with an IT background willing to volunteer to help?

I'm sure that Rogean and the staff would be open to assistance qualified help. I'm willing to help coordinate volunteers. Even if you're not interested in long term involvement in P99, surely there are a few of us with enough combined expertise to help mitigate this.

Example background (mine): 15 years in tech, last 9 working for fortune 100 tech company that rhymes with crisco, deep understanding of DNS, switched networks, virtualization.

Willing to make offers of service? PM me to keep Rogean's already plump inbox clear.

[Phats]

Sure, take out his IP's.

67.205.76.148

75.144.216.121

http://i41.tinypic.com/11wdfzr.png

Well, 67.205.76.148 is definately the Sleeper it just took me to the website




The second IP was given to me by a source as well.


Have fun

[MaximiusM]

Right, taking down Canadian and US ISPs vs helping Rogean mitigate DDoS at ingress... sounds reasonable.

[Phats]

Toops server is in Canada, not sure about the other one

[Lopretni]

Right, taking down Canadian and US ISPs vs helping Rogean mitigate DDoS at ingress... sounds reasonable.

IPs which we may not be sure are involved, a lot of DDOS uses unsuspecting participants usually don't they?

[MaximiusM]

Without being too much of a prick, you gave me two IPs. Are you familiar with the first D in ddos? If the packets aren't coming from compromised hosts they are being spoofed which means that I doubt that any IP information people have is accurate.

All those packets are going to be UDP so if this person(s) understand the weaknesses of STUN targeting individual IPs/Hosts is like pissing into your own mouth.

edit: Even a fucking retard can use pingback exploits on unpatched wordpress or drupal 6/7 hosts to ghetto flood a target host.

[spoils]

Yea ***** don't u know dem shitz?

[Lopretni]

Maximius, can you gather anything from the image Tiggles posted in another thread?
https://www.project1999.org/forums/showpost.php?p=1034629&postcount=54

[Razdeline]

Without being too much of a prick, you gave me two IPs. Are you familiar with the first D in ddos? If the packets aren't coming from compromised hosts they are being spoofed which means that I doubt that any IP information people have is accurate.

All those packets are going to be UDP so if this person(s) understand the weaknesses of STUN targeting individual IPs/Hosts is like pissing into your own mouth.

edit: Even a fucking retard can use pingback exploits on unpatched wordpress or drupal 6/7 hosts to ghetto flood a target host.

Well, we know it is Toop's doing as far as the terrorism that is happening. I am sure some people can take the information here and do something with it, I am sure a blood-for-blood is a last resort

[rickjames]

Not sure how you can work at cisco and not understand wtf DDoS attacks are, unless you're the receptionist or something.

[LordSterben]

Not sure how you can work at cisco and not understand wtf DDoS attacks are, unless you're the receptionist or something.

Lol^

On a serious note...I am tech savvy as a mofo. I was the editor of the school newspaper in high school so I know the Apple IIe inside and out. And I once programmed a game where you shoot exclamation marks at equal signs in QBasic. If you need my l33t haxxor skills lmk.

[azxten]

There are only two ways to mitigate a DDOS. Expensive hardware (both on the network and server side) and well written firewall rules.

If I had to guess I would say that the firewall rules are already fine and this is purely a matter of money. P99 is already hosted by a company that offers DDOS mitigation. However, as with all things, you get what you pay for. It's likely that if the P99 staff payed out another $5k-$10k a month the attack would be fully mitigated.

What we have right now is the best performance we can get with the resources available.

It would be interesting to see a packet capture from the server side though if any sensitive information could be stripped.

[Nirgon]

Donate button to save your beloved project

Time to have a bake sale or some shit

Need to know if having a bake sale and hitting the donate button is considered RMT first

[MaximiusM]

Not sure how you can work at cisco and not understand wtf DDoS attacks are, unless you're the receptionist or something.

i lol'd :p

but in all seriousness, ty to the people who PM'd. I sent Rogean a note regarding this we'll see where if goes (if anywhere).

[azxten]

Its pretty simple really. There is some volume of traffic directed at the server. Some percentage of this is malicious DDOS attack traffic.

Can that malicious traffic be differentiated from legitimate traffic?

If yes, then it can be filtered out. However, filtering takes CPU cycles and more complex rules typically require more cycles.

Where do we do the filtering? Options are on the server ($) or on an upstream router ($$) or security appliance ($$$). Doing it on your own server is cheap but its also a bit too late because now the server is handling the packet filtering thus overloading itself. Doing it on an upstream router is more expensive because XYZ data center has to maintain the rules for you but moves the load off the server. Doing it on a security appliance is most expensive because they cost to buy but is most effective due to being purpose built.

If no, then all of that traffic needs to hit the server and the server needs to keep up. ($$$$)

This typically means bringing in a load balancer and shifting to a multi-server architecture which typically means the server software will require adjustments to run properly since this is all custom code. ($$$$$)

I'm guessing the server is at the point where the attack is being mitigated on an upstream router/security appliance but its dynamic enough in nature that it can still cause outages until the firewall adjusts to the new sources.

[Nirgon]

Is this ICMP or what?

I know the server is being DDoS'd but we know nothing of the nature of the attack.

Going jihad on Kegz and pissing him off (if its not him) is just going to add more problems.

[azxten]

I'd be interested to see a packet capture as well or hear something from staff.

Considering the amount of trouble this has caused I imagine this is something more custom built which mimics legitimate traffic. Login server login attempts or something like that.

Then I would guess there is a firewall rule which watches for X packets per Y time to the server which is considered abnormal and blacklists the IP for some period of time.

So, every time there is a lag spike its when the attack starts from a new collection of IPs which must be allowed access for a period of time to determine if they are malicious or not.

This is all just a rough guess but hopefully it provides some insight into how these things work for those who are curious.

[trukk]

No clue what is really going on here but if I had to guess, I'd say just plain UDP flooding.

UDP is connectionless, meaning it's one way from the sender to the P99 server. This allows the source IP to be spoofed, thus a single host can pretend to be sending from a few thousand different IP's making blocking very hard.

Most good ISP's will have spoof protection turned on, but it only takes a handful of zombied hosts that don't have the protection on to overwhelm the server.

I would assume that the EQ client uses some UDP connections back to the server, which the port they are spoofing on. You can;t block that port, because it is how the legit clients work.

This is all just a wild guess, but probably pretty close.

-Chris

[rickjames]

There are two potential effects:

A) There is a socket connection and the server is requested to perform some sort of task (i.e. login) This taxes the servers process resources.

B) Its mostly dropped trash traffic (ICMP flood etc.) and overwhelms network equipment.


Scenario B is mitigated by over-provisioning bandwidth usually. Scenario A is much more difficult to respond to unless a solution is developed to trust connections sources and therefore drop packets before they reach the server.

Regardless, to fix the problem by throwing money at it is unlikely to be very efficient or effective.

Now one potential theory I had (im a receptionist at an IT firm too) is that they may be abusing socket connections to the webserver (spam HTTP requests) as it seems the website goes down (colocated/same box/same VM) with the game server. If someone would be able to confirm or refute this, i would be willing to pony up a little cash to help get the website/forums hosted on a separate instance to mitigate that vulnerability.

[Blink]

It's the internet and everyone's an expert!

[MaximiusM]

Now one potential theory I had (im a receptionist at an IT firm too) is that they may be abusing socket connections to the webserver (spam HTTP requests) as it seems the website goes down (colocated/same box/same VM) with the game server. If someone would be able to confirm or refute this, i would be willing to pony up a little cash to help get the website/forums hosted on a separate instance to mitigate that vulnerability.

you've essentially come to the same conclusion as me, fellow internet receptionist. PM'ing with details.

[Pringles]

=
Now one potential theory I had (im a receptionist at an IT firm too) is that they may be abusing socket connections to the webserver (spam HTTP requests) as it seems the website goes down (colocated/same box/same VM) with the game server. If someone would be able to confirm or refute this, i would be willing to pony up a little cash to help get the website/forums hosted on a separate instance to mitigate that vulnerability.


Except that would stand out like a sore thumb as the cause and Rogean would have already resolved it.

[August]

If I can help in any way let me know. I have a background in cryptography and filter analysis and have been in software for 7 years.

More importantly, if you need to find out how to do something, I can find that out almost assuredly. I'm not an expert in cyber security, however.