Runbook: NodeHighNumberConntrackEntriesUsed

Alert Details

• Alert Name: NodeHighNumberConntrackEntriesUsed
• Expression: (node_nf_conntrack_entries{nanocosmosGroup=~".+", instance=~".+", environment=~".+"} / node_nf_conntrack_entries_limit{nanocosmosGroup=~".+", instance=~".+", environment=~".+"}) >

Description

This alert triggers when the number of conntrack entries used exceeds a certain threshold. Conntrack entries are used by the Linux kernel to track network connections. A high number of entries can indicate potential network issues or misconfigurations.

Possible Causes

• High number of active network connections
• Misconfigured network applications
• Network attacks or unusual traffic patterns
• Insufficient conntrack table size

Troubleshooting Steps

1. Check Current Conntrack Entries

Use the following command to check the current number of conntrack entries:

sudo sysctl net.netfilter.nf_conntrack_count

Expected Output

You should see an output similar to this:

net.netfilter.nf_conntrack_count = 65536

2. Check Conntrack Table Limit

To check the current limit of the conntrack table, use:

sudo sysctl net.netfilter.nf_conntrack_max

Expected Output

You should see an output similar to this:

net.netfilter.nf_conntrack_max = 131072

3. Identify High Connection Sources

To identify sources of high connection counts, use:

sudo conntrack -L | awk '{print $5}' | cut -d= -f2 | sort | uniq -c | sort -nr | head -n 10

Expected Output

This command lists the top 10 sources by connection count:

10000 192.168.1.100
5000  192.168.1.101
...

4. Adjust Conntrack Table Size

If the conntrack table size is insufficient, consider increasing it. For example, to increase the limit to 262144:

sudo sysctl -w net.netfilter.nf_conntrack_max=262144

Expected Output

Verify the new limit:

sudo sysctl net.netfilter.nf_conntrack_max

You should see the updated limit:

net.netfilter.nf_conntrack_max = 262144

5. Review Application Configurations

Review and adjust configurations of applications causing high connection counts. Ensure they are not misconfigured or generating excessive connections.

6. Monitor Network Traffic

Use tools like iftop or nload to monitor real-time network traffic and identify unusual patterns:

sudo apt-get install iftop
sudo iftop -i eth0

Expected Output

You should see real-time network traffic statistics for the specified interface.

Additional Steps

1. Monitor Conntrack Entries

Continuously monitor conntrack entries to ensure they remain within acceptable limits. Use tools like prometheus and grafana to set up dashboards and alerts.

2. Implement Network Security Measures

Implement network security measures such as firewalls and intrusion detection systems to protect against network attacks and unusual traffic patterns.

By following these steps, you should be able to troubleshoot and resolve the “NodeHighNumberConntrackEntriesUsed” alert. If the issue persists, further investigation into the specific network configuration and traffic patterns may be necessary.