Mechanisms of SYN Flood Attack and Defense

TCP Packet Exchange Procedure

TCP provides a reliable connection-oriented service. A connection channel must be set up between two parties before one party sends data to the other party. This is called TCP three-way handshake, as shown in Figure 1.

Figure 1 TCP three-way handshake

1. First handshake: The client sends the server a SYN message carrying the server port to connect to and the initial sequence number (ISN).
2. Second handshake: After receiving the SYN message from the client, the server replies with an SYN-ACK message, indicating that the request sent by the client is accepted. In addition, the server sets the acknowledgment number in the SYN-ACK message to the client's ISN plus 1.
3. Third handshake: After receiving the SYN-ACK message from the server, the client sends an ACK message to the server to complete the three-way handshake.

If the server cannot receive an ACK message from the client after the server sending the SYN-ACK message, the server resends an SYN-ACK message to the client. If the server still cannot receive an ACK message from the client within the specified period of time, the incomplete connection is removed.

SYN Flood Attack

An attacker can take advantage of the TCP three-way handshake mechanism to launch SYN flood attacks. As shown in Figure 2, an attacker sends a large number of SYN messages to the server. After receiving these packets, the server replies with SYN-ACK messages but the attacker does not return ACK messages. As a result, the server maintains a large number of half-open connections until they age out. These half-open connections exhaust server resources. Therefore, the server fails to respond to normal requests.

Figure 2 SYN flood attack

The FW defends against SYN flood attacks in TCP proxy or TCP source authentication mode.

TCP Proxy

As a TCP proxy, the FW is deployed between the client and server to establish a three-way handshake with the client on behalf of the server and relay the TCP connection to the server if the three-way handshake is complete. TCP proxy applies only to scenarios in which the forward and return paths are the same.

Figure 3 TCP proxy

• As shown in Figure 3, the FW receives a SYN message, blocks the SYN message, and returns an SYN-ACK message on behalf of the server.
• If the client fails to return an ACK message, the FW considers the SYN message abnormal, and maintains the half-open connection on behalf of the server until the half-open connection expires.
• If the client returns an ACK message, the FW considers the SYN message normal and establishes a three-way handshake with the client. Then, the FW reestablishes a three-way handshake with the server. The subsequent messages on the connection are sent to the server.

The TCP proxy procedure is transparent to both the client and server.

During the TCP proxy procedure, the FW proxies and responds to each SYN message received and maintains half-open connections. Therefore, if a large number of SYN messages are sent to the FW, the FW must have high performance to handle them.

TCP proxy applies only to scenarios in which the forward and return paths are the same.

TCP Source Authentication

TCP source authentication enables the FW can defend against SYN flood attacks when forward and return paths are different. Therefore, compared with TCP proxy, TCP source authentication is more widely used.

Figure 4 TCP source authentication

• As shown in Figure 4, after the FW receives a SYN message from the client, it blocks the SYN message, forges an SYN-ACK message carrying an incorrect sequence number, and sends this message to the client.
• If the source address is fake, no message is sent in response to the incorrect SYN-ACK message.
• If the source address is real, the client replies with an RST message after receiving the incorrect SYN-ACK message so that the FW can send a correct SYN-ACK message. After the FW receives the RST message, it determines that the client is the real source and whitelists the source address. The FW considers the packets sent by this source legitimate and permits the packets without authentication until the whitelist entry expires.

In TCP source authentication, the source client is whitelisted once the client passes the authentication, and authentication is not performed on subsequent SYN messages sent by this source. This implementation greatly improves the defense efficiency and performance and minimizes the resource consumption.

If there are devices that discard source detection packets, the source detection function cannot be used.

First Packet Discarding

Some attacks continuously change source IP addresses or source ports. Such attack packets are different. If the source authentication mode is used for defense, each attack packet is replied. When the attack traffic is heavy, the replying amount is also large, which consumes a lot of performance and causes link congestion.

The combination of the first packet discarding and source authentication can effectively defend against heavy traffic attacks by changing source IP addresses or source port numbers.

Normally, TCP packets are retransmitted upon timeout. If packets are discarded during the interaction, the packets are retransmitted. The first-packet discarding function uses the packet retransmission mechanism. The first packet received is discarded, and subsequent retransmitted packets are directly permitted. When the rate of retransmitted packets reaches the alarm threshold, the packets enter the source authentication process to reduce the number of reply packets.

Figure 5 shows the procedure for processing SYN packets.

Figure 5 Procedure of first packet discarding

Currently, the first-packet discarding function applies only to SYN packets.

First packet discarding applies to IPv4 packets, not IPv6 packets.