Threat Action

Threat actor from China and Russia would launch millions of TCP SYNs into IP ranges behind our firewall.

The firewall would hit 10,000,000 million (ten million!) connections and then die. This affected 40,000 staff/students and most likely cost up to $5 million dollars in lost productivity

Solution

1) Black Holing, 2) Internet Router ACL, 3) Firewall Connection Limits (this is what we did, solution at the bottom of this post).

DDOS_1DDOS_2DDOS_3

How?

The threat actor would generate lots of TCP SYNs and send them into our firewall. According to the TCP RFC 793

 Half-Open Connections and Other Anomalies

  An established connection is said to be  "half-open" if one of the
  TCPs has closed or aborted the connection at its end without the
  knowledge of the other, or if the two ends of the connection have
  become desynchronized owing to a crash that resulted in loss of
  memory.  Such connections will automatically become reset if an
  attempt is made to send data in either direction.  However, half-open
  connections are expected to be unusual, and the recovery procedure is
  mildly involved.

So half open are defined in the RFC, so vendors will implement them in the stack, and threat actors will take advantage of them. When I analyzed the data only about 3% of overall traffic was TCP half-open connections.

Our Solution: Firewall Connection Limits

We analyzed all the traffic going through our firewall by looking at the connection tables. We aggregated and averaged this over a two week period. We then checked lots of IP addresses manually and on suspicion known databases. We then enforced two things 1) half-open connection limit, 2) TCP and UDP connection limit. This had the positive affect of keeping the firewall up and running during a DDOS by nullifying the false connections that exceeded our limits.

 

 

Advertisements