Edit

Share via

Capture a TCP dump from a Linux node in an AKS cluster

Summary

Networking problems can occur when you use a Microsoft Azure Kubernetes Service (AKS) cluster. To help you investigate these problems, this article explains how to capture a TCP dump from a Linux node in an AKS cluster, and then download the capture to your local machine.

Prerequisites

Note

You can automate TCP capture through a Helm chart, which can run in the background as a DaemonSet. For more information, see this custom GitHub tool for capturing TCP dumps, or use the steps in the following sections.

Step 1: Find the nodes to troubleshoot

How do you determine which node to pull the TCP dump from? First, get the list of nodes in the AKS cluster by using the Kubernetes command-line client, kubectl. Follow the instructions to connect to the cluster and run the kubectl get nodes --output wide command using the Azure portal or Azure CLI. A node list that's similar to the following output appears:

$ kubectl get nodes --output wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
aks-agentpool-34796016-vmss000000   Ready    agent   45h   v1.20.9   10.240.1.81    <none>        Ubuntu 18.04.6 LTS               5.4.0-1062-azure   containerd://1.4.9+azure
aks-agentpool-34796016-vmss000002   Ready    agent   45h   v1.20.9   10.240.2.47    <none>        Ubuntu 18.04.6 LTS               5.4.0-1062-azure   containerd://1.4.9+azure

Step 2: Connect to a Linux node

The next step is to establish a connection to the AKS cluster node that you want to capture the network trace from. For more information, see Create an interactive shell connection to a Linux node.

Step 3: Make sure tcpdump is installed

After you connect to the AKS Linux node, verify that the tcpdump tool is installed on the node by running tcpdump --version. If tcpdump isn't installed, you see the following error text:

# tcpdump --version
bash: tcpdump: command not found

Then install tcpdump on your pod by running the Advanced Package Tool's package handling utility, apt-get:

apt-get update && apt-get install tcpdump

If tcpdump is installed, you see output similar to the following text:

# tcpdump --version
tcpdump version 4.9.3
libpcap version 1.8.1
OpenSSL 1.1.1  11 Sep 2018

Step 4: Create a packet capture

To capture the dump, run the tcpdump command as follows:

# tcpdump --snapshot-length=0 -vvv -w /capture.cap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
Got 6

Note

Running tcpdump without filtering parameters can significantly increase the size of the Packet Capture (PCAP) file, especially for long runs. Therefore, add filters like source, destination, and port. For example:

  • tcpdump dst 192.168.1.100
  • tcpdump dst host.mydomain.com
  • tcpdump port http or port ftp or port smtp or port imap or port pop3 or port telnet

While the trace is running, replicate your issue many times. This action ensures that the issue is captured within the TCP dump. Note the time stamp while you replicate the issue. To stop the packet capture when you're done, press Ctrl+C:

# tcpdump -s 0 -vvv -w /capture.cap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C526 packets captured
526 packets received by filter
0 packets dropped by kernel

Step 5: Transfer the capture locally

After you complete the packet capture, identify the helper pod so you can copy the dump locally. Open a second console, and then get a list of pods by running kubectl get pods, as shown in the following example.

$ kubectl get pods
NAME                                                    READY   STATUS    RESTARTS   AGE
azure-vote-back-6c4dd64bdf-m4nk7                        1/1     Running   0          3m29s
azure-vote-front-85b4df594d-jhpzw                       1/1     Running   0          3m29s
node-debugger-aks-nodepool1-38878740-vmss000000-jfsq2   1/1     Running   0          60s

The helper pod has a prefix of node-debugger-aks, as shown in the third row. Replace the pod name, and then run the following kubectl command. These commands retrieve the packet capture for your Linux node.

kubectl cp node-debugger-aks-nodepool1-38878740-vmss000000-jfsq2:/capture.cap capture.cap

Note

If you use the chroot /host command when entering the debug pod, add /host before /capture.cap for the source file.

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

  • Last updated on