Threat actor from China and Russia would launch millions of TCP SYNs into IP ranges behind our firewall.
The firewall would hit 10,000,000 million (ten million!) connections and then die. This affected 40,000 staff/students and most likely cost up to $5 million dollars in lost productivity
1) Black Holing, 2) Internet Router ACL, 3) Firewall Connection Limits (this is what we did, solution at the bottom of this post).
The threat actor would generate lots of TCP SYNs and send them into our firewall. According to the TCP RFC 793
Half-Open Connections and Other Anomalies An established connection is said to be "half-open" if one of the TCPs has closed or aborted the connection at its end without the knowledge of the other, or if the two ends of the connection have become desynchronized owing to a crash that resulted in loss of memory. Such connections will automatically become reset if an attempt is made to send data in either direction. However, half-open connections are expected to be unusual, and the recovery procedure is mildly involved.
So half open are defined in the RFC, so vendors will implement them in the stack, and threat actors will take advantage of them. When I analyzed the data only about 3% of overall traffic was TCP half-open connections.
Our Solution: Firewall Connection Limits
We analyzed all the traffic going through our firewall by looking at the connection tables. We aggregated and averaged this over a two week period. We then checked lots of IP addresses manually and on suspicion known databases. We then enforced two things 1) half-open connection limit, 2) TCP and UDP connection limit. This had the positive affect of keeping the firewall up and running during a DDOS by nullifying the false connections that exceeded our limits.