mozillazg's Blog

ptcpdump v0.33 ~ v0.37 的主要变更内容

2025-11-29T00:00:00+00:00

前言¶

本文将按变更顺序介绍一下 ptcpdump 从上次的 v0.32 之后的 v0.33 版本到最新的 v0.37 版本期间所发布的主要变更内容。

主要变更内容¶

优先使用 BPF ringbuf 代替 BPF perfbuf¶

如果当前系统使用的 Linux 内核支持 BPF ringbuf 的话， ptcpdump 将优先使用 BPF ringbuf 技术优化程序性能。关于 BPF ringbuf 的更多信息，详见 BPF ring buffer 。

修复使用 cgroup-skb 后端时按进程过滤数据包功能失效的问题¶

修复之前的版本中，通过 --backend=cgroup-skb 参数指定使用 cgroup-skb 后端时， --pid 或 --pname 参数对应的功能失效的问题。

支持从标准输入中读取 PcapNG 格式的数据¶

现已支持通过 -r - 从标准输入中读取 PcapNG 格式的数据:

$ cat data.pcapng | ptcpdump -r -
$ ptcpdump -r - < data.pcapng

支持 -C 和 -W 参数实现文件轮转功能¶

现已支持在使用 -w 保存数据到文件中时，使用 -C 或 -W 参数实现文件轮转功能:

sudo ptcpdump -i any -w data.pcapng -C 1mb
sudo ptcpdump -i any -w data.pcapng -C 1mb -W 5

其中:

-C 指定文件的最大大小，当超过该大小时，将在旧的文件名后面追加数字（比如 data.pcapng1 ），来保存历史数据。
-W 自定保存的文件数量，需要与 -C 一起使用，用于限制文件数量。

支持 -F/--expression-file 参数实现从文件中读取 pcap fitler 表达式¶

现已支持使用 -F 或者 --expression-file 参数从文件中读取 pcap filter 包过滤表达式:

sudo ptcpdump -i any -c 10 -F filter.txt
sudo ptcpdump -i any -c 10 --expression-file filter.txt

支持 -tt, -ttt, -tttt, -ttttt 参数实现指定输出的时间格式¶

现已支持使用 -tt, -ttt, -tttt, -ttttt 指定输出包信息时的时间信息格式。

-tt : 显示时间戳格式，比如 1764417816.346098 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...
-ttt : 显示两条记录的时间间隔, 比如 00:00:00.000265 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...
-tttt : 显示日期和时间，比如 2025-11-29 20:03:36.346098 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...
-ttttt : 显示距离第一条记录的时间间隔，比如 00:00:00.002708 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...

支持通过动态链接的方式使用系统安装的 libpcap¶

之前的版本只支持以静态链接的方式编译依赖的 libpcap 库，现已支持通过动态链接的方式使用系统安装的 libpcap 库，满足简化系统库管理的需求。

可以通过下面任一编译方式使用动态链接库:

$ CGO_ENABLED=1 go build -tags dynamic
$ make build-dynamic-link

修复 -s 0 参数会导致程序崩溃的问题¶

在之前的版本中，如果使用 -s 0 参数会导致 ptcpdump 程序崩溃，这个问题现已修复。

修复终端输出中的时间信息不准确的问题¶

在之前的版本中，终端输出的包信息的时间信息随着时间的推移会与实践的事件时间差距越来越大，这个问题现已修复。

实验性支持 armv7 CPU 架构¶

在新版本中已实验性支持 armv7 CPU 架构。因为缺少支持 armv7 架构的 CI 环境，因此只能作为实验性支持，没法保证未来版本的兼容性。如果某个版本不再支持 armv7 了，欢迎反馈。

新增 tp-btf 后端¶

新版本增加了一个新的后端 tp-btf ，当指定使用该后端时，ptcpdump 将使用 BPF_PROG_TYPE_TRACING 类型的 eBPF 程序来捕获数据包。该后端与 tc 后端的主要区别是：tp-btf 默认将能捕获所有网络命名空间下的数据，而 tc 后端默认只捕获当前网络命名空间下的数据（可以通过 --netns 命令指定捕获特定网络命名空间下的数据）。关于该后端与其他后端的区别，详见 Backend 。

可以通过 --backend tp-btf 使用新增的 tp-btf 后端。

新增 socket-filter 后端¶

新版本增加了另一个新的后端 socket-filter ，当指定使用该后端时，ptcpdump 将使用 BPF_PROG_TYPE_SOCKET_FILTER 类型的 eBPF 程序来捕获数据包。使用该后端时，ptcpdump 的输出结果将更接近 tcpdump 的输出结果，因为此时它们两个捕获数据包的时机会更接近。关于该后端与其他后端的区别，详见 Backend 。

可以通过 --backend socket-filter 使用新增的 socket-filter 后端。

在跨网络命名空间的数据结果中显示 inode ID 信息¶

从新版本开始，当输出结果中包含非当前网络命名空间的数据时，程序将会尝试在输出网络接口信息时，显示对应网络命名空间的 inode ID 信息:

12:20:31.336316 eth0@4026533097 curl.405939 Out IP 172.17.0.4.38670 > 1.1.1.1.80: Flags [S], seq 3064539219, win 64240, options [mss 1460,sackOK,TS val 1731159046 ecr 0,nop,wscale 7], length 0, ParentProc [bash.405653]
12:20:31.336382 veth1d387b0 curl.405939 In IP 172.17.0.4.38670 > 1.1.1.1.80: Flags [S], seq 3064539219, win 64240, options [mss 1460,sackOK,TS val 1731159046 ecr 0,nop,wscale 7], length 0, ParentProc [bash.405653]

-v 的输出中增加 IP Options 信息¶

新版本在使用 -v 参数输出详情时，将输出 IP Options 信息:

09:48:31.908813 IP (tos 0x0, ttl 64, id 39183, offset 0, flags [none], proto TCP (6), length 80, options (RR 1.2.3.4, 1.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0,EOL))
    ${IP}.1373 > 1.1.1.1.80: Flags [S], cksum 0xbfb4, seq 122218745, win 512, length 0

新增输出 HTTP 信息¶

如果某一行的数据包是 HTTP 数据，新版本将默认输出基本的 HTTP 信息:

13:27:02.285015 IP ${IP}.57166 > ${IP}.80: Flags [P.], ..., length 73: HTTP: GET / HTTP/1.1
13:27:02.516911 IP ${IP}.80 > ${IP}.57166: Flags [P.], ..., length 349: HTTP: HTTP/1.1 301 Moved Permanently

如果使用 -v 参数输出详情，则会包含详细的 HTTP 信息:

13:27:02.285015 IP (tos 0x0, ttl 64, id 8683, offset 0, flags [DF], proto TCP (6), length 113)
    ..., length 73: HTTP: GET / HTTP/1.1
        GET / HTTP/1.1
        Host: kernel.org
        User-Agent: curl/8.5.0
        Accept: */*

13:27:02.516911 IP (tos 0x0, ttl 128, id 62994, offset 0, flags [none], proto TCP (6), length 389)
    ..., length 349: HTTP: HTTP/1.1 301 Moved Permanently
        HTTP/1.1 301 Moved Permanently
        Server: nginx
        Date: Sat, 05 Jul 2025 13:27:02 GMT
        Content-Type: text/html
        Content-Length: 162
        Connection: keep-alive
        Location: https://kernel.org/

        <html>
        <head><title>301 Moved Permanently</title></head>
        <body>
        <center><h1>301 Moved Permanently</h1></center>
        <hr><center>nginx</center>
        </body>
        </html>

新增输出 TLS 信息¶

如果某一行的数据包是 TLS 数据，新版本将默认输出 TLS Client Hello 或 Server Hello 信息:

13:48:40.371849 IP ${IP}.47812 > ${IP}.443: Flags [P.], ..., length 517: TLSv1.0: Client Hello (SNI=kernel.org)
13:48:40.606677 IP ${IP}.443 > ${IP}.47812: Flags [P.], ..., length 1412: TLSv1.3: Server Hello

新增支持对 L3 网络接口设备抓包¶

之前的版本只支持对 L2 网络接口设备抓包，新版本已增加对 L3 网络接口（比如 TUN 设备）的支持。

改为默认以相对数字显示 TCP 序列号¶

在之前的版本中，默认会显示 TCP 序列号的原始数字，比较难阅读:

20:03:36.346098 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 1457857145:1457857369, ack 1966040526, win 65535, length 224, ParentProc [sshd.16678]
20:03:36.346909 ens33 sshd.183127 In IP ${IP}.64755 > ${IP}.22: Flags [.], seq 1966040526, ack 1457857369, win 64240, length 0, ParentProc [sshd.16678]
20:03:36.348807 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 1457857369:1457857593, ack 1966040526, win 65535, length 224, ParentProc [sshd.16678]

新版本将默认以相对数字的形式显示 TCP 序列号信息:

20:03:36.346098 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 1457857145:1457857369, ack 1966040526, win 65535, length 224, ParentProc [sshd.16678]
20:03:36.346909 ens33 sshd.183127 In IP ${IP}.64755 > ${IP}.22: Flags [.], ack 224, win 64240, length 0, ParentProc [sshd.16678]
20:03:36.348807 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 224:448, ack 0, win 65535, length 224, ParentProc [sshd.16678]

同时新增 -S 和 --absolute-tcp-sequence-numbers 参数用于控制恢复之前的显示原始数字的模式。

如果你对 ptcpdump 有额外的改进或新功能建议，欢迎在评论区或项目 issues 中留言。

Major Changes in ptcpdump versions 0.33 to 0.37

2025-11-29T00:00:00+00:00

Preface¶

This article introduces the main changes in ptcpdump from v0.33 (after v0.32 ) to the latest v0.37, in chronological order.

Main changes¶

Prefer using BPF ringbuf instead of BPF perfbuf¶

If the Linux kernel on the current system supports BPF ringbuf, ptcpdump will prioritize using BPF ringbuf to optimize program performance. For more information about BPF ringbuf, please refer to BPF ring buffer .

Fix the issue where packet filtering by process failed when using the cgroup-skb backend¶

Fixed an issue in previous versions where the features corresponding to the --pid or --pname parameters failed when the cgroup-skb backend was specified via --backend=cgroup-skb.

Support reading PcapNG format data from standard input¶

It now supports reading PcapNG format data from standard input using -r -:

$ cat data.pcapng | ptcpdump -r -
$ ptcpdump -r - < data.pcapng

Support file rotation using -C and -W options¶

It now supports file rotation when saving data to a file using -w, by using the -C or -W options:

sudo ptcpdump -i any -w data.pcapng -C 1mb
sudo ptcpdump -i any -w data.pcapng -C 1mb -W 5

Where:

-C specifies the maximum file size. When this size is exceeded, a number will be appended to the old filename (e.g., data.pcapng1) to save historical data.
-W specifies the number of files to keep. It must be used with -C to limit the number of files.

Support reading pcap filter expressions from a file using -F/--expression-file¶

It now supports reading pcap filter expressions from a file using the -F or --expression-file option:

sudo ptcpdump -i any -c 10 -F filter.txt
sudo ptcpdump -i any -c 10 --expression-file filter.txt

Support specifying output time format using -tt, -ttt, -tttt, -ttttt¶

It now supports using -tt, -ttt, -tttt, and -ttttt to specify the time format when outputting packet information.

-tt : Displays timestamp format, e.g., 1764417816.346098 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...
-ttt : Displays the time interval between two records, e.g., 00:00:00.000265 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...
-tttt : Displays date and time, e.g., 2025-11-29 20:03:36.346098 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...
-ttttt : Displays the time interval since the first record, e.g., 00:00:00.002708 ens33 sshd.183127 Out IP xx.xx.xx.xx.22 > xx.xx.xx.xx.64755:...

Support using system-installed libpcap via dynamic linking¶

Previous versions only supported compiling the dependent libpcap library via static linking. It now supports using the system-installed libpcap library via dynamic linking to meet the needs of simplified system library management.

You can use the dynamic link library through either of the following compilation methods:

$ CGO_ENABLED=1 go build -tags dynamic
$ make build-dynamic-link

Fix the issue where using -s 0 caused the program to crash¶

In previous versions, using the -s 0 option caused the ptcpdump program to crash. This issue is now fixed.

Fix the issue of inaccurate timestamps in terminal output¶

In previous versions, the timestamps of packet information in the terminal output drifted further from the actual event time over time. This issue is now fixed.

Experimental support for armv7 CPU architecture¶

Experimental support for the armv7 CPU architecture has been added in the new version. Since there is no CI environment supporting the armv7 architecture, this is only experimental support, and compatibility in future versions cannot be guaranteed. If a version no longer supports armv7, feedback is welcome.

Add tp-btf backend¶

A new backend tp-btf has been added. When this backend is used, ptcpdump uses BPF_PROG_TYPE_TRACING eBPF programs to capture packets. The main difference from the tc backend is that tp-btf captures data from all network namespaces by default, while the tc backend defaults to capturing data only from the current network namespace (use --netns to specify a specific network namespace). For differences between backends, see Backend .

You can use the new tp-btf backend via --backend tp-btf.

Add socket-filter backend¶

Another new backend socket-filter has been added. When this backend is used, ptcpdump uses BPF_PROG_TYPE_SOCKET_FILTER eBPF programs to capture packets. When using this backend, the output of ptcpdump is closer to the output of tcpdump because the packet capture timing is more similar. For differences between backends, see Backend .

You can use the new socket-filter backend via --backend socket-filter.

Display inode ID in cross-network namespace results¶

Starting from the new version, when the output contains data from a non-current network namespace, the program will attempt to display the inode ID of that network namespace when showing network interface information:

12:20:31.336316 eth0@4026533097 curl.405939 Out IP 172.17.0.4.38670 > 1.1.1.1.80: Flags [S], seq 3064539219, win 64240, options [mss 1460,sackOK,TS val 1731159046 ecr 0,nop,wscale 7], length 0, ParentProc [bash.405653]
12:20:31.336382 veth1d387b0 curl.405939 In IP 172.17.0.4.38670 > 1.1.1.1.80: Flags [S], seq 3064539219, win 64240, options [mss 1460,sackOK,TS val 1731159046 ecr 0,nop,wscale 7], length 0, ParentProc [bash.405653]

Add IP Options information to -v output¶

The new version will output IP Options information when using the -v option for details:

09:48:31.908813 IP (tos 0x0, ttl 64, id 39183, offset 0, flags [none], proto TCP (6), length 80, options (RR 1.2.3.4, 1.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0,EOL))
    ${IP}.1373 > 1.1.1.1.80: Flags [S], cksum 0xbfb4, seq 122218745, win 512, length 0

Add output for HTTP information¶

If a packet contains HTTP data, the new version will output basic HTTP information by default:

13:27:02.285015 IP ${IP}.57166 > ${IP}.80: Flags [P.], ..., length 73: HTTP: GET / HTTP/1.1
13:27:02.516911 IP ${IP}.80 > ${IP}.57166: Flags [P.], ..., length 349: HTTP: HTTP/1.1 301 Moved Permanently

If -v is used for detailed output, detailed HTTP information will be included:

13:27:02.285015 IP (tos 0x0, ttl 64, id 8683, offset 0, flags [DF], proto TCP (6), length 113)
    ..., length 73: HTTP: GET / HTTP/1.1
        GET / HTTP/1.1
        Host: kernel.org
        User-Agent: curl/8.5.0
        Accept: */*

13:27:02.516911 IP (tos 0x0, ttl 128, id 62994, offset 0, flags [none], proto TCP (6), length 389)
    ..., length 349: HTTP: HTTP/1.1 301 Moved Permanently
        HTTP/1.1 301 Moved Permanently
        Server: nginx
        Date: Sat, 05 Jul 2025 13:27:02 GMT
        Content-Type: text/html
        Content-Length: 162
        Connection: keep-alive
        Location: https://kernel.org/

        <html>
        <head><title>301 Moved Permanently</title></head>
        <body>
        <center><h1>301 Moved Permanently</h1></center>
        <hr><center>nginx</center>
        </body>
        </html>

Add output for TLS information¶

If a packet contains TLS data, the new version will output TLS Client Hello or Server Hello information by default:

13:48:40.371849 IP ${IP}.47812 > ${IP}.443: Flags [P.], ..., length 517: TLSv1.0: Client Hello (SNI=kernel.org)
13:48:40.606677 IP ${IP}.443 > ${IP}.47812: Flags [P.], ..., length 1412: TLSv1.3: Server Hello

Add support for capturing on L3 network interface devices¶

Previous versions only supported capturing on L2 network interface devices. The new version adds support for L3 network interfaces (such as TUN devices).

Change to display TCP sequence numbers as relative numbers by default¶

In previous versions, raw TCP sequence numbers were displayed by default, which was hard to read:

20:03:36.346098 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 1457857145:1457857369, ack 1966040526, win 65535, length 224, ParentProc [sshd.16678]
20:03:36.346909 ens33 sshd.183127 In IP ${IP}.64755 > ${IP}.22: Flags [.], seq 1966040526, ack 1457857369, win 64240, length 0, ParentProc [sshd.16678]
20:03:36.348807 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 1457857369:1457857593, ack 1966040526, win 65535, length 224, ParentProc [sshd.16678]

The new version displays TCP sequence numbers as relative numbers by default:

20:03:36.346098 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 1457857145:1457857369, ack 1966040526, win 65535, length 224, ParentProc [sshd.16678]
20:03:36.346909 ens33 sshd.183127 In IP ${IP}.64755 > ${IP}.22: Flags [.], ack 224, win 64240, length 0, ParentProc [sshd.16678]
20:03:36.348807 ens33 sshd.183127 Out IP ${IP}.22 > ${IP}.64755: Flags [P.], seq 224:448, ack 0, win 65535, length 224, ParentProc [sshd.16678]

It also adds the -S and --absolute-tcp-sequence-numbers options to switch back to displaying raw numbers.

If you have any additional improvements or new feature suggestions for ptcpdump, feel free to leave a comment in the comments section or project issues.

使用 elibpcap 为网络相关 eBPF 程序实现 pcap-filter 包过滤语法支持

2025-05-03T00:00:00+00:00

前言¶

我们常用的 tcpdump 抓包工具的一个核心能力是支持使用 pcap-filter 包过滤语法对流量进行过滤，只对符合条件的特定流量进行抓包。

当我们使用 eBPF 技术开发网络相关的工具的时候，如果也能支持 pcap-filter 包过滤语法的话，想必会极大的提升用户体验。因此，我开发的 ptcpdump 工具也内置了对 pcap-filter 包过滤语法的支持。

使用常规方法为 eBPF 程序增加 pcap-filter 支持会需要实现复杂的逻辑。但是，如果你的项目正在使用 cilium/ebpf 作为 eBPF Loader，通过使用本文介绍的 elibpcap 这个 Go package ，你只需要增加几行代码就能为你的项目内置支持 pcap-filter 包过滤语法的能力。

项目介绍¶

如前面所说，通过使用 elibpcap，只需几行代码就可以为 eBPF 项目增加支持 pcap-filter 的能力。

下面是 AI 总结的项目介绍：

elibpcap 是一个 Go 语言包，它提供的功能是：将 pcap 过滤表达式编译为 eBPF 字节码，并将这些过滤器注入到现有的 eBPF 程序中。它将人类可读的 pcap 过滤表达式（例如 tcpdump 等工具中使用的那些表达式）的世界与在 Linux 内核中运行的 eBPF 程序的强大能力和高效率连接了起来。

项目地址¶

https://github.com/jschwinger233/elibpcap

使用示例¶

使用模式¶

为了使用 elibpcap ，我们需要做三方面的修改：

一个是，需要修改 eBPF 程序，在需要实现数据包过滤的位置增加类似如下的 placeholder 逻辑：

// add this
static __noinline bool pcap_filter(void *_skb, void *__skb, void *___skb, void *data, void *data_end) {
     return data != data_end && _skb == __skb && __skb == ___skb;
}

SEC("tc")
int sample_prog(struct __sk_buff *skb){
    // ...

    // add this
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        bpf_printk("pcap_filter not match");
        goto out;
    }
    // ...
}

另一个是，需要修改 Go 程序，调用 elibpcap package 增加集成 pcap-filter 包过滤能力的逻辑：

oldInsts := spec.Programs["sample_prog"].Instructions
newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
        AtBpf2Bpf:  "pcap_filter",
        PacketAccessMode: elibpcap.Direct,
        L2Skb:      true,
})
if err != nil {
        return err
}
spec.Programs["sample_prog"].Instructions = newInsts

其中：

AtBpf2Bpf 用于指定是我们在 eBPF 程序中定义的 static __noinline bool pcap_filter(...) 函数的名称。
PacketAccessMode 用于指定 elibpcap 在生成的字节码时，应当使用哪种方式读取我们在 eBPF 程序中向 pcap_filter 函数所传入的 skb 数据。取值如下：
- elibpcap.Direct ：使用 direct access 的方式读取数据，即，直接通过指针操作读取数据。
- elibpcap.BpfProbeReadKernel ：使用辅助函数 bpf_probe_read_kernel 读取数据。
- elibpcap.BpfSkbLoadBytes ：使用辅助函数 bpf_skb_load_bytes 读取数据。
L2Skb 用于指定我们的 eBPF 程序中 pcap_filter 函数传入的 skb 数据是否包含 L2（链路层）相关的数据。

最后是，因为 elibpcap 依赖了 libpcap ，因此需要在编译时启用 CGO：

静态链接编译示例：

CGO_CFLAGS="-I/path/to/libpcap/include" \
CGO_LDFLAGS="-L/path/to/libpcap/lib -lpcap /path/to/libpcap.a" \
CGO_ENABLED=1 go build -o main \
-tags static -ldflags "-linkmode 'external' -extldflags '-static'"

动态链接编译示例：

CGO_ENABLED=1 go build -o main -tags dynamic

下面将以常见的不同 eBPF 程序类型为例介绍如何集成 elibpcap。

tc/tcx¶

下面是 tc eBPF 程序中需要做的修改：

// add this
static __noinline bool pcap_filter(void *_skb, void *__skb, void *___skb, void *data, void *data_end) {
     return data != data_end && _skb == __skb && __skb == ___skb;
}

SEC("tc")
int sample_prog(struct __sk_buff *skb){
    bpf_skb_pull_data(skb, 0);

    void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;

    // add this
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        bpf_printk("pcap_filter not match");
        goto out;
    }

    bpf_printk("Hello from tc after pcap filter");
out:
    return TC_ACT_UNSPEC;
}

可以看到主要的修改点是：

新增了一个 pcap_filter 函数。
新增了一个 if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) 的判断。

下面是对应的 Go 程序所需要做的修改：

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    if expr == "" {
            return nil
    }
    log.Printf("inject pcap filter: %s", expr)
  // add this
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.Direct,
            L2Skb:      true,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
  // end
    return nil
}

// add this
if err := injectFilter(spec, expr); err != nil {
    log.Fatal(err)
}
if err := spec.LoadAndAssign(&objs, nil); err != nil {
    // ...
}

完整的示例程序详见：tc

xdp¶

xdp 程序集成 pcap-filter 能力所需要做的修改与 tc/tcx 程序基本一致。

c 程序:

SEC("xdp")
int sample_prog(struct xdp_md *ctx){
    // ...
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    if (!pcap_filter((void *)ctx, (void *)ctx, (void *)ctx, data, data_end)) {
        bpf_printk("pcap_filter not match");
        goto out;
    }
  // ...
}

go 程序:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.Direct,
            L2Skb:      true,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}
// ...

完整的示例程序详见：xdp

cgroup-skb¶

cgroup-skb 程序与前面 tc/tcx/xdp 程序所需要做的修改也是基本一致。

c 程序:

SEC("cgroup_skb/egress")
int sample_prog(struct __sk_buff *skb){
      void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        goto out;
    }
    // ...
}

go 程序:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.Direct,
            L2Skb:      false,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}

这里有一个需要注意的点：因为 cgroup-skb 程序中 skb 的数据里未包含 L2 的数据，因此这里需要设置 L2Skb: false 。

完整的示例程序详见：cgroup-skb

kprobe/fentry/tracepoint¶

elibpcap 帮助我们屏蔽了 kprobe/fentry/tracepoint 程序与前面的 tc/tcx/xdp 程序在数据读取方面的差异，让我们可以以基本一致的方式进行集成。

c 程序:

SEC("kprobe/__dev_queue_xmit")
int BPF_KPROBE(sample_prog, struct sk_buff *skb){
    void *skb_head = BPF_CORE_READ(skb, head);
      void *data = skb_head + BPF_CORE_READ(skb, network_header);
      void *data_end = skb_head + BPF_CORE_READ(skb, tail);
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        goto out;
    }
    // ...
}

go 程序:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.BpfProbeReadKernel,
            L2Skb:      false,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}
// ...

这里有两个需要注意的点：

因为在 kprobe 程序中无法通过 “direct packet access“ 的方式读取 skb 中的包数据，因此我们选择设置 PacketAccessMode: elibpcap.BpfProbeReadKernel 使用 bpf_probe_read_kernel 读取数据。
因为我们的 kprobe 程序中检查的是 L3 的数据，因此这里需要设置 L2Skb: false 。

完整的示例程序详见： kprobe , fentry , tracepoint

socket filter¶

socket filter 程序的一个限制是无法使用 direct access 或 bpf_probe_read_kernel 的方式读取数据包，只能使用类似 bpf_skb_load_bytes 的方式读取数据包。

下面是使用 socket filter 程序集成 elibpcap 的示例代码：

c 程序:

SEC("socket")
int sample_prog(struct __sk_buff *skb){
    void *data = (void *)(long)skb;
    char dummy[1];
    bpf_skb_load_bytes(skb, 0, &dummy, sizeof(dummy));
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, (void *)data, (void *)dummy)) {
        goto out;
    }
    // ...
}

go 程序:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:        "pcap_filter",
            PacketAccessMode: elibpcap.BpfSkbLoadBytes,
            L2Skb:            true,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}
// ...

这里有三个需要注意的点：

我们需要使用 bpf_skb_load_bytes 读取数据，因此 PacketAccessMode 被设置为 elibpcap.BpfSkbLoadBytes 。
在 socket filter 程序中，我们是直接从 skb 而不是从 skb->data 中读取的数据，因此 data 被设置为 skb。
在 socket filter 程序中，我们无法直接获取 skb->data_end，并且 pcap_filter 中的 data_end 在使用 BpfSkbLoadBytes 访问方式时只是一个用于申请一个栈变量的 placeholder 变量，因此我们可以直接使用一个 dummy 变量作为 date_end。

完整的示例程序详见： socket-filter 。

结束语¶

如果你的使用 Go 和 eBPF 技术开发的网络项目想要集成 pcap-filter 的能力，但却苦于无从下手，推荐尝试集成 elibpcap 这个巧妙而强大的项目。

参考资料¶

Implement pcap-filter syntax support for network eBPF programs using elibpcap

2025-05-03T00:00:00+00:00

Preface¶

A core feature of the common tcpdump packet capture tool is its support for the pcap-filter syntax. This lets it filter traffic and capture only specific packets matching the filter.

When developing network-related tools with eBPF, supporting the pcap-filter syntax would greatly improve user experience. That's why the ptcpdump tool I developed includes built-in support for the pcap-filter syntax.

Adding pcap-filter support to eBPF programs the usual way involves complex logic. However, if your project uses cilium/ebpf as its eBPF loader, you can use the elibpcap Go package introduced in this article. This lets you add built-in pcap-filter syntax support to your project with just a few lines of code.

Project Introduction¶

As mentioned before, using elibpcap allows you to add pcap-filter support to an eBPF project with just a few lines of code.

Here is a project introduction summarized by AI:

elibpcap is a Go package that provides functionality for compiling pcap filter expressions into eBPF bytecode and injecting these filters into existing eBPF programs. It bridges the world of human-readable pcap filter expressions (like those used in tools such as tcpdump) with the power and efficiency of eBPF programs running in the Linux kernel.

Project Repository¶

https://github.com/jschwinger233/elibpcap

Usage Example¶

Usage Pattern¶

To use elibpcap, we need to make changes in three areas:

First, modify the eBPF program. Add placeholder logic like the following where packet filtering is needed:

// add this
static __noinline bool pcap_filter(void *_skb, void *__skb, void *___skb, void *data, void *data_end) {
     return data != data_end && _skb == __skb && __skb == ___skb;
}

SEC("tc")
int sample_prog(struct __sk_buff *skb){
    // ...

    // add this
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        bpf_printk("pcap_filter not match");
        goto out;
    }
    // ...
}

Second, modify the Go program. Call the elibpcap package to add the logic for integrating pcap-filter capability:

oldInsts := spec.Programs["sample_prog"].Instructions
newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
        AtBpf2Bpf:  "pcap_filter",
        PacketAccessMode: elibpcap.Direct,
        L2Skb:      true,
})
if err != nil {
        return err
}
spec.Programs["sample_prog"].Instructions = newInsts

Where:

AtBpf2Bpf specifies the name of the static __noinline bool pcap_filter(...) function defined in our eBPF program.
PacketAccessMode specifies how elibpcap should read the skb data passed to the pcap_filter function in our eBPF program when generating bytecode. Values are:
- elibpcap.Direct: Uses direct access to read data, i.e., direct pointer operations.
- elibpcap.BpfProbeReadKernel: Uses the helper function bpf_probe_read_kernel to read data.
- elibpcap.BpfSkbLoadBytes: Uses the helper function bpf_skb_load_bytes to read data.
L2Skb specifies if the skb data passed to the pcap_filter function in our eBPF program includes L2 (link layer) related data.

Finally, because elibpcap depends on libpcap, you need to enable CGO during compilation:

Static linking example:

CGO_CFLAGS="-I/path/to/libpcap/include" \
CGO_LDFLAGS="-L/path/to/libpcap/lib -lpcap /path/to/libpcap.a" \
CGO_ENABLED=1 go build -o main \
-tags static -ldflags "-linkmode 'external' -extldflags '-static'"

Dynamic linking example:

CGO_ENABLED=1 go build -o main -tags dynamic

The following sections will show how to integrate elibpcap with examples for different common eBPF program types.

tc/tcx¶

Below are the changes required in the tc eBPF program:

// add this
static __noinline bool pcap_filter(void *_skb, void *__skb, void *___skb, void *data, void *data_end) {
     return data != data_end && _skb == __skb && __skb == ___skb;
}

SEC("tc")
int sample_prog(struct __sk_buff *skb){
    bpf_skb_pull_data(skb, 0);

    void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;

    // add this
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        bpf_printk("pcap_filter not match");
        goto out;
    }

    bpf_printk("Hello from tc after pcap filter");
out:
    return TC_ACT_UNSPEC;
}

The main changes are:

Added a pcap_filter function.
Added an if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) check.

Below are the corresponding changes required in the Go program:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    if expr == "" {
            return nil
    }
    log.Printf("inject pcap filter: %s", expr)
  // add this
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.Direct,
            L2Skb:      true,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
  // end
    return nil
}

// add this
if err := injectFilter(spec, expr); err != nil {
    log.Fatal(err)
}
if err := spec.LoadAndAssign(&objs, nil); err != nil {
    // ...
}

See the complete example program: tc

xdp¶

The modifications needed to integrate pcap-filter capability into XDP programs are basically the same as for tc/tcx programs.

c program:

SEC("xdp")
int sample_prog(struct xdp_md *ctx){
    // ...
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    if (!pcap_filter((void *)ctx, (void *)ctx, (void *)ctx, data, data_end)) {
        bpf_printk("pcap_filter not match");
        goto out;
    }
  // ...
}

go program:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.Direct,
            L2Skb:      true,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}
// ...

See the complete example program: xdp

cgroup-skb¶

The modifications needed for cgroup-skb programs are also basically the same as those for the previous tc/tcx/xdp programs.

c program:

SEC("cgroup_skb/egress")
int sample_prog(struct __sk_buff *skb){
      void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        goto out;
    }
    // ...
}

go program:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.Direct,
            L2Skb:      false,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}

One point to note: Because the skb data in cgroup-skb programs does not contain L2 data, set L2Skb: false here.

See the complete example program: cgroup-skb

kprobe/fentry/tracepoint¶

elibpcap hides the data reading differences between kprobe/fentry/tracepoint programs and the tc/tcx/xdp programs mentioned earlier. This allows us to integrate them in a mostly consistent way.

c program:

SEC("kprobe/__dev_queue_xmit")
int BPF_KPROBE(sample_prog, struct sk_buff *skb){
    void *skb_head = BPF_CORE_READ(skb, head);
      void *data = skb_head + BPF_CORE_READ(skb, network_header);
      void *data_end = skb_head + BPF_CORE_READ(skb, tail);
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, data, data_end)) {
        goto out;
    }
    // ...
}

go program:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:  "pcap_filter",
            PacketAccessMode: elibpcap.BpfProbeReadKernel,
            L2Skb:      false,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}
// ...

Here are two things to note:

Because kprobe programs cannot read packet data from skb using "direct packet access", So we set PacketAccessMode: elibpcap.BpfProbeReadKernel to read data using bpf_probe_read_kernel.
Because our kprobe program checks L3 data, we need to set L2Skb: false here.

See the complete example program: kprobe , fentry , tracepoint

socket filter¶

A limitation of socket filter programs is they cannot read packets using direct access or bpf_probe_read_kernel. They can only read packets using methods similar to bpf_skb_load_bytes.

Below is example code for integrating elibpcap with socket filter programs:

c program:

SEC("socket")
int sample_prog(struct __sk_buff *skb){
    void *data = (void *)(long)skb;
    char dummy[1];
    bpf_skb_load_bytes(skb, 0, &dummy, sizeof(dummy));
    if (!pcap_filter((void *)skb, (void *)skb, (void *)skb, (void *)data, (void *)dummy)) {
        goto out;
    }
    // ...
}

go program:

func injectFilter(spec *ebpf.CollectionSpec, expr string) error {
    // ...
    oldInsts := spec.Programs["sample_prog"].Instructions
    newInsts, err := elibpcap.Inject(expr, oldInsts, elibpcap.Options{
            AtBpf2Bpf:        "pcap_filter",
            PacketAccessMode: elibpcap.BpfSkbLoadBytes,
            L2Skb:            true,
    })
    if err != nil {
            return err
    }
    spec.Programs["sample_prog"].Instructions = newInsts
    return nil
}
// ...

Here are three points to note:

We need to use bpf_skb_load_bytes to read data, so PacketAccessMode is set to elibpcap.BpfSkbLoadBytes.
In socket filter programs, we read data directly from skb, not from skb->data. So, data is set to skb.
In socket filter programs, we cannot directly get skb->data_end. Also, data_end in pcap_filter is just a placeholder variable for allocating a stack variable when using the BpfSkbLoadBytes access mode. So, we can directly use a dummy variable as data_end.

See the complete example program: socket-filter.

Conclusion¶

If you want to add pcap-filter support to your Go and eBPF network project but don't know where to start, try integrating the clever and powerful elibpcap project.

References¶

ptcpdump v0.27 ~ v0.32 的主要变更内容

2025-01-25T00:00:00+00:00

前言¶

本文将按变更顺序介绍一下 ptcpdump 从上次的 v0.26 之后的 v0.27 版本到最新的 v0.32 版本期间所发布的主要变更内容。

主要变更内容¶

新增 -q/--quiet 参数¶

新增 -q/--quiet 参数用于精简输出内容:

13:50:35.524360 lo curl.345650 Out IP 127.0.0.1.58694 > 127.0.0.1.8000: tcp 0, ParentProc [bash.345626]

新增 --context 参数¶

新增 --context 参数用于指定输出中包含的上下文信息，可以通过这个参数控制只显示特定的上下文信息。

通过 --context=process 限制只输出进程信息:

# --context=process
09:32:09.718892 vethee2a302f wget.3553008 In IP 10.244.0.2.33426 > 139.178.84.217.80: Flags [S], seq 4113492822, win 64240, length 0

# -v --context=process
13:44:41.529003 eth0 In IP (tos 0x4, ttl 45, id 45428, offset 0, flags [DF], proto TCP (6), length 52)
    139.178.84.217.443 > 172.19.0.2.42606: Flags [.], cksum 0x5284, seq 3173118145, ack 1385712707, win 118, options [nop,nop,TS val 134560683 ecr 1627716996], length 0
    Process (pid 553587, cmd /usr/bin/wget, args wget kernel.org)

该参数既支持通过英文逗号分割多个值，也支持多次指定参数的方式指定多个值:

# -v --context=process,parentproc
# -v --context=process --context=parentproc

13:44:41.529003 eth0 In IP (tos 0x4, ttl 45, id 45428, offset 0, flags [DF], proto TCP (6), length 52)
    139.178.84.217.443 > 172.19.0.2.42606: Flags [.], cksum 0x5284, seq 3173118145, ack 1385712707, win 118, options [nop,nop,TS val 134560683 ecr 1627716996], length 0
    Process (pid 553587, cmd /usr/bin/wget, args wget kernel.org)
    ParentProc (pid 553296, cmd /bin/sh, args sh)

# -v --context=process,parentproc,container
# -v --context=process --context=parentproc --context=container

13:44:41.529003 eth0 In IP (tos 0x4, ttl 45, id 45428, offset 0, flags [DF], proto TCP (6), length 52)
    139.178.84.217.443 > 172.19.0.2.42606: Flags [.], cksum 0x5284, seq 3173118145, ack 1385712707, win 118, options [nop,nop,TS val 134560683 ecr 1627716996], length 0
    Process (pid 553587, cmd /usr/bin/wget, args wget kernel.org)
    ParentProc (pid 553296, cmd /bin/sh, args sh)
    Container (name test, id d9028334568bf75a5a084963a8f98f78c56bba7f45f823b3780a135b71b91e95, image docker.io/library/alpine:3.18, labels {"io.cri-containerd.kind":"container","io.kubernetes.container.name":"test","io.kubernetes.pod.name":"test","io.kubernetes.pod.namespace":"default","io.kubernetes.pod.uid":"9e4bc54b-de48-4b1c-8b9e-54709f67ed0c"})

在高版本内核中使用 fentry, btf raw tracepoint 以及 tcx ebpf 特性¶

当在高版本内核中执行 ptcpdump 程序时，程序将自动使用 fentry 代替 kprobe，使用 btf raw tracepoint 代替 raw tracepoint, 使用 tcx 代替 tc，通过使用这些新的 ebpf 特性，优化程序在高版本内核下的性能。

新增 --backend 参数¶

新增 --backend 参数用于指定抓包时使用的技术，默认是使用的 tc/tcx 技术。

支持 --backend=cgroup-skb¶

支持通过 --backend=cgroup-skb 参数指定使用 cgroup-skb ebpf 程序进行抓包。使用该方式抓取的流量将不包含链路层信息，ptcpdump 输出中的链路层信息将是一个固定的 fake 数据。

支持 OpenWrt 24.10 x86-64 系统¶

根据用户的需求反馈，提升程序兼容性，支持在 OpenWrt 24.10 x86-64 系统中使用 ptcpdump （前提是系统内核在编译时启用 ebpf 和 BTF 相关参数）。

支持在输出中显示线程信息¶

当通过 --backend=cgroup-skb 进行抓包时，新增在输出中显示线程信息:

$ sudo ptcpdump -i any --backend cgroup-skb -v port 80

10:18:26.846884 ens33 Out IP (tos 0x0, ttl 64, id 57734, offset 0, flags [DF], proto TCP (6), length 478)
    xxx.xxx.xxx.35102 > xxx.xxx.xxx.80: Flags [P.], cksum 0x3381, seq xx:xx, ack xx, win 64240, length 438
    Process (pid 113278, cmd /snap/firefox/5437/usr/lib/firefox/firefox, args /snap/firefox/5437/usr/lib/firefox/firefox)
    Thread (tid 113438, name Socket Thread)
    ParentProc (pid 49607, cmd /usr/bin/xfce4-panel, args xfce4-panel --display :0.0 --sm-client-id xxxx)

支持在输出中显示 uid 以及按 uid 抓包¶

新增在输出中显示 uid 信息:

12:37:40.051539 ens33 Out IP (tos 0x0, ttl 64, id 48697, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.x.x.42906 > 139.x.x.x.443: Flags [S], cksum 0xecc8, seq 940329637, win 64240, options [mss 1460,sackOK,TS val 3421262256 ecr 0,nop,wscale 7], length 0
    Process (pid 99722, cmd /usr/bin/curl, args curl https://kernel.org)
    User (uid 1000)
    ParentProc (pid 18840, cmd /usr/bin/bash, args -bash)

新增支持指定 uid 进行抓包:

$ sudo ptcpdump -i any --uid 100 -v port 80

如果你对 ptcpdump 有额外的改进或新功能建议，欢迎在评论区或项目 issues 中留言。

ptcpdump v0.27 ~ v0.32 的主要变更内容

2025-01-25T00:00:00+00:00

Introduction¶

This article introduces the main changes in ptcpdump from v0.27 (after v0.26) to the latest v0.32, in chronological order.

Main Changes¶

Add -q/--quiet option¶

Added the -q/--quiet option to simplify the output:

13:50:35.524360 lo curl.345650 Out IP 127.0.0.1.58694 > 127.0.0.1.8000: tcp 0, ParentProc [bash.345626]

Add --context option¶

Added the --context option to specify the context information included in the output. This option controls showing only specific context information.

Use --context=process to restrict output to process information only:

# --context=process
09:32:09.718892 vethee2a302f wget.3553008 In IP 10.244.0.2.33426 > 139.178.84.217.80: Flags [S], seq 4113492822, win 64240, length 0

# -v --context=process
13:44:41.529003 eth0 In IP (tos 0x4, ttl 45, id 45428, offset 0, flags [DF], proto TCP (6), length 52)
    139.178.84.217.443 > 172.19.0.2.42606: Flags [.], cksum 0x5284, seq 3173118145, ack 1385712707, win 118, options [nop,nop,TS val 134560683 ecr 1627716996], length 0
    Process (pid 553587, cmd /usr/bin/wget, args wget kernel.org)

This option supports specifying multiple values either by separating them with commas or by specifying the option multiple times:

# -v --context=process,parentproc
# -v --context=process --context=parentproc

13:44:41.529003 eth0 In IP (tos 0x4, ttl 45, id 45428, offset 0, flags [DF], proto TCP (6), length 52)
    139.178.84.217.443 > 172.19.0.2.42606: Flags [.], cksum 0x5284, seq 3173118145, ack 1385712707, win 118, options [nop,nop,TS val 134560683 ecr 1627716996], length 0
    Process (pid 553587, cmd /usr/bin/wget, args wget kernel.org)
    ParentProc (pid 553296, cmd /bin/sh, args sh)

# -v --context=process,parentproc,container
# -v --context=process --context=parentproc --context=container

13:44:41.529003 eth0 In IP (tos 0x4, ttl 45, id 45428, offset 0, flags [DF], proto TCP (6), length 52)
    139.178.84.217.443 > 172.19.0.2.42606: Flags [.], cksum 0x5284, seq 3173118145, ack 1385712707, win 118, options [nop,nop,TS val 134560683 ecr 1627716996], length 0
    Process (pid 553587, cmd /usr/bin/wget, args wget kernel.org)
    ParentProc (pid 553296, cmd /bin/sh, args sh)
    Container (name test, id d9028334568bf75a5a084963a8f98f78c56bba7f45f823b3780a135b71b91e95, image docker.io/library/alpine:3.18, labels {"io.cri-containerd.kind":"container","io.kubernetes.container.name":"test","io.kubernetes.pod.name":"test","io.kubernetes.pod.namespace":"default","io.kubernetes.pod.uid":"9e4bc54b-de48-4b1c-8b9e-54709f67ed0c"})

Use fentry, btf raw tracepoint, and tcx ebpf features on newer kernels¶

When running ptcpdump on newer kernels, the program will automatically use fentry instead of kprobe, btf raw tracepoint instead of raw tracepoint, and tcx instead of tc. Using these new ebpf features optimizes performance on newer kernels.

Add --backend option¶

Added the --backend option to specify the technology used for packet capturing. The default is tc/tcx.

Support --backend=cgroup-skb¶

It now supports using --backend=cgroup-skb to capture packets using cgroup-skb ebpf programs. Traffic captured this way will not contain link-layer information; the link-layer information in ptcpdump output will be fixed fake data.

Support OpenWrt 24.10 x86-64 system¶

Based on user feedback, program compatibility has been improved to support using ptcpdump on OpenWrt 24.10 x86-64 systems (provided the system kernel was compiled with ebpf and BTF parameters enabled).

Support displaying thread information in output¶

When capturing packets using --backend=cgroup-skb, thread information is now displayed in the output:

$ sudo ptcpdump -i any --backend cgroup-skb -v port 80

10:18:26.846884 ens33 Out IP (tos 0x0, ttl 64, id 57734, offset 0, flags [DF], proto TCP (6), length 478)
    xxx.xxx.xxx.35102 > xxx.xxx.xxx.80: Flags [P.], cksum 0x3381, seq xx:xx, ack xx, win 64240, length 438
    Process (pid 113278, cmd /snap/firefox/5437/usr/lib/firefox/firefox, args /snap/firefox/5437/usr/lib/firefox/firefox)
    Thread (tid 113438, name Socket Thread)
    ParentProc (pid 49607, cmd /usr/bin/xfce4-panel, args xfce4-panel --display :0.0 --sm-client-id xxxx)

Support displaying UID in output and capturing by UID¶

It now supports displaying UID information in the output:

12:37:40.051539 ens33 Out IP (tos 0x0, ttl 64, id 48697, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.x.x.42906 > 139.x.x.x.443: Flags [S], cksum 0xecc8, seq 940329637, win 64240, options [mss 1460,sackOK,TS val 3421262256 ecr 0,nop,wscale 7], length 0
    Process (pid 99722, cmd /usr/bin/curl, args curl https://kernel.org)
    User (uid 1000)
    ParentProc (pid 18840, cmd /usr/bin/bash, args -bash)

It now supports specifying a UID for packet capturing:

$ sudo ptcpdump -i any --uid 100 -v port 80

If you have additional improvements or new feature suggestions for ptcpdump, feel free to leave a message in the comments or project issues.

Workload Identity Federation for GKE 特性探索

2025-01-05T00:00:00+00:00

本文将简单探索一下前段时间 GKE 官宣的名为 Workload Identity Federation for GKE 的特性。

功能介绍¶

Workload Identity Federation for GKE 是原有的 GKE Workload Identity 特性的改进版本, 核心的改进是减少了需要配置的信息，提升了用户体验。

使用方法¶

可以通过下面几个步骤体验该特性：

创建一个启用 Workload Identity Federation for GKE 特性的 GKE 集群。具体启用位置是：创建集群 - 安全 - 启用 Workload Identity。

为测试应用使用的 service account 关联 iam 角色，一个 service account 可以关联多个角色。比如:

$ kubectl create ns demo-ns
$ kubectl -n demo-ns create serviceaccount demo-sa

$ gcloud projects add-iam-policy-binding projects/test-gke-XXX \
    --role=roles/container.clusterViewer \
    --member=principal://iam.googleapis.com/projects/23182XXXXXXX/locations/global/workloadIdentityPools/test-gke-XXXXXX.svc.id.goog/subject/ns/<namespace-name>/sa/<service-account-name> \
    --condition=None

$ gcloud storage buckets add-iam-policy-binding gs://<bucket-name> \
--role=roles/storage.objectViewer \
--member=principal://iam.googleapis.com/projects/23182XXXXXXX/locations/global/workloadIdentityPools/test-gke-XXXXXX.svc.id.goog/subject/ns/<namespace-name>/sa/<service-account-name> \
--condition=None

部署测试应用，测试应用的 pod 需要使用第 2 步对应的 namespace 和 service account 以及增加 nodeSelector 确保调度到启用了 Workload Identity Federation for GKE 特性的节点上:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: demo-ns
spec:
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: "true"
  serviceAccountName: demo-sa
  containers:
  - name: test-pod
    image: google/cloud-sdk:slim
    command: ["sleep","infinity"]

待 pod running 后, 进入 test-pod 容器内，访问实例元数据服务获取 sts token，实际的业务应用使用的官方 SDK 也将使用类似的方式访问实例元数据服务获取用于请求云产品 API 的 sts token。

$ gcloud auth print-access-token
 ya29.d.XXX
$ curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
 {"access_token":"ya29.d.XXX","expires_in":3423,"token_type":"Bearer"}

获取的 sts token 将具有前面第 2 步所关联的所有角色的权限.

$ gcloud container node-pools list --zone us-central1 --cluster test
 NAME          MACHINE_TYPE  DISK_SIZE_GB  NODE_VERSION
 default-pool  e2-medium     100           1.30.6-gke.1125000

$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 "https://storage.googleapis.com/storage/v1/b/demo-gke-workload-identity-federation/o"
 {
   "kind": "storage#objects"
 }

工作流程¶

Workload Identity Federation for GKE 特性的工作流程如下：

当应用 Pod 容器内的程序请求实例元数据服务（访问 http://metadata.google.internal ，实际是访问 http://169.254.169.254:80 ）获取 sts token 时, 该请求将被重定向到 169.254.169.252:988。 169.254.169.252:988 是节点上部署的 gke-metadata-server 服务所监听的端口。
gke-metadata-server 服务在收到请求后，将根据请求的 client ip 确定请求来自哪个 Pod，然后再请求 apiserver 生成一个该 Pod 所使用的 service account 对应的 token。
gke-metadata-server 将使用获取的 service account token 访问 GCP 的 STS API 获取一个 sts token，最后 gke-metadata-server 将获取到的 sts token 返回给客户端。

这个流程中有几个关键的组件和信息需要重点关注，下面将逐个说明。

gke-metadata-server¶

当在集群维度或节点池维度启用 Workload Identity Federation for GKE 特性时，集群内将自动部署一个名为 gke-metadata-server 的组件。该组件的工作负载 YAML 如下:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "1"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: gke-metadata-server
  name: gke-metadata-server
  namespace: kube-system
spec:
  minReadySeconds: 90
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: gke-metadata-server
  template:
    metadata:
      annotations:
        components.gke.io/component-name: gke-metadata-server
        components.gke.io/component-version: 0.4.301
        monitoring.gke.io/path: /metricz
      creationTimestamp: null
      labels:
        addonmanager.kubernetes.io/mode: Reconcile
        k8s-app: gke-metadata-server
    spec:
      containers:
      - command:
        - /gke-metadata-server
        - --logtostderr
        - --token-exchange-endpoint=https://securetoken.googleapis.com/v1/identitybindingtoken
        - --workload-pool=test-gke-XXXXXX.svc.id.goog
        - --alts-service-suffixes-using-node-identity=storage.googleapis.com,bigtable.googleapis.com,bigtable2.googleapis.com,bigtablerls.googleapis.com,spanner.googleapis.com,spanner2.googleapis.com,spanner-rls.googleapis.com,grpclb.directpath.google.internal,grpclb-dualstack.directpath.google.internal,staging-wrenchworks.sandbox.googleapis.com,preprod-spanner.sandbox.googleapis.com,wrenchworks-loadtest.googleapis.com,wrenchworks-nonprod.googleapis.com
        - --identity-provider=https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test
        - --passthrough-ksa-list=anthos-identity-service:gke-oidc-envoy-sa,anthos-identity-service:gke-oidc-service-sa,gke-managed-dpv2-observability:hubble-relay,kube-system:antrea-controller,kube-system:container-watcher-pod-reader,kube-system:coredns,kube-system:egress-nat-controller,kube-system:event-exporter-sa,kube-system:fluentd-gcp-scaler,kube-system:gke-metrics-agent,kube-system:gke-spiffe-node-agent,kube-system:heapster,kube-system:konnectivity-agent,kube-system:kube-dns,kube-system:maintenance-handler,kube-system:metadata-agent,kube-system:network-metering-agent,kube-system:node-local-dns,kube-system:pkgextract-service,kube-system:pkgextract-cleanup-service,kube-system:securityprofile-controller,istio-system:istio-ingressgateway-service-account,istio-system:cluster-local-gateway-service-account,csm:csm-sync-agent,knative-serving:controller,kube-system:pdcsi-node-sa,kube-system:gcsfusecsi-node-sa,gmp-system:collector,gke-gmp-system:collector,gke-managed-cim:kube-state-metrics
        - --attributes=cluster-name=test,cluster-uid=392f63049deXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,cluster-location=us-central1
        - --cluster-uid=392f63049ded410XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        - --sts-endpoint=https://sts.googleapis.com
        - --token-exchange-mode=sts
        - --cloud-monitoring-endpoint=monitoring.googleapis.com:443
        - --iam-cred-service-endpoint=https://iamcredentials.googleapis.com
        - --cluster-project-number=2318XXXXXXXX
        - --cluster-location=us-central1
        - --cluster-name=test
        - --component-version=0.4.301
        - --ksa-cache-mode=watchchecker
        - --kcp-allow-watch-checker=true
        - --enable-mds-csi-driver=true
        - --csi-socket=/csi/csi.sock
        - --volumes-db=/var/run/gkemds.gke.io/csi/volumes.boltdb
        env:
        - name: GOMEMLIMIT
          value: "94371840"
        image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metadata-server:gke_metadata_server_20240702.00_p0@sha256:aea9cc887c91b9a05e5bb4bb604180772594a01f0828bbfacf30c77562ac7205
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            host: 127.0.0.1
            path: /healthz
            port: 989
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        name: gke-metadata-server
        ports:
        - containerPort: 987
          name: alts
          protocol: TCP
        - containerPort: 988
          name: metadata-server
          protocol: TCP
        - containerPort: 989
          name: debug-port
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            host: 127.0.0.1
            path: /healthz
            port: 989
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          privileged: true
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kubelet/kubeconfig
          name: kubelet-credentials
          readOnly: true
        - mountPath: /var/lib/kubelet/pki/
          name: kubelet-certs
          readOnly: true
        - mountPath: /var/run/
          name: container-runtime-interface
        - mountPath: /etc/srv/kubernetes/pki
          name: kubelet-pki
          readOnly: true
        - mountPath: /etc/ssl/certs/
          name: ca-certificates
          readOnly: true
        - mountPath: /home/kubernetes/bin/gke-exec-auth-plugin
          name: gke-exec-auth-plugin
          readOnly: true
        - mountPath: /sys/firmware/efi/efivars/
          name: efivars
          readOnly: true
        - mountPath: /dev/tpm0
          name: vtpm
          readOnly: true
        - mountPath: /csi
          name: csi-socket-dir
        - mountPath: /var/run/gkemds.gke.io/csi
          name: state-dir
        - mountPath: /var/lib/kubelet/pods
          mountPropagation: Bidirectional
          name: pods-dir
        - mountPath: /registration
          name: kubelet-registration-dir
      dnsPolicy: Default
      hostNetwork: true
      nodeSelector:
        iam.gke.io/gke-metadata-server-enabled: "true"
        kubernetes.io/os: linux
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: gke-metadata-server
      serviceAccountName: gke-metadata-server
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
        operator: Exists
      - effect: NoSchedule
        operator: Exists
      - key: components.gke.io/gke-managed-components
        operator: Exists
      volumes:
      - hostPath:
          path: /var/lib/kubelet/pki/
          type: Directory
        name: kubelet-certs
      - hostPath:
          path: /var/lib/kubelet/kubeconfig
          type: File
        name: kubelet-credentials
      - hostPath:
          path: /var/run/
          type: Directory
        name: container-runtime-interface
      - hostPath:
          path: /etc/srv/kubernetes/pki/
          type: Directory
        name: kubelet-pki
      - hostPath:
          path: /etc/ssl/certs/
          type: Directory
        name: ca-certificates
      - hostPath:
          path: /home/kubernetes/bin/gke-exec-auth-plugin
          type: File
        name: gke-exec-auth-plugin
      - hostPath:
          path: /sys/firmware/efi/efivars/
          type: Directory
        name: efivars
      - hostPath:
          path: /dev/tpm0
          type: CharDevice
        name: vtpm
      - hostPath:
          path: /var/lib/kubelet/plugins/gkemds.gke.io
          type: DirectoryOrCreate
        name: csi-socket-dir
      - hostPath:
          path: /var/lib/kubelet/pods
          type: Directory
        name: pods-dir
      - hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: Directory
        name: kubelet-registration-dir
      - hostPath:
          path: /var/lib/kubelet/plugins
          type: Directory
        name: kubelet-plugins-dir
      - hostPath:
          path: /var/run/gkemds.gke.io/csi
          type: DirectoryOrCreate
        name: state-dir

gke-metadata-server 组件具有如下特点：

组件的工作负载是一个 DaemonSet。
组件 Pod 使用 hostNetwork: true 以及 privileged: true 。

组件内的服务将监听 987, 988 以及 989 端口, 其中 988 端口将用于接收重定向过来的访问实例元数据服务的请求:

$ ss -atnlp |grep gke
LISTEN 0      1024               *:987              *:*    users:(("gke-metadata-se",pid=183706,fd=10))
LISTEN 0      1024               *:989              *:*    users:(("gke-metadata-se",pid=183706,fd=12))
LISTEN 0      1024               *:988              *:*    users:(("gke-metadata-se",pid=183706,fd=15))

前面所说的 169.254.169.252 这个 IP 是本机 lo 的 IP 地址，所以 gke-metadata-server 监听的 988 端口就包含了前面所说的 169.254.169.252:988 端口:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.169.252/32 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

所有启用了 Workload Identity Federation for GKE 特性的节点在初始化的时候都会配置如下 nftables 规则，确保从业务容器内发起的请求元数据服务 (http://metadata.google.internal , 169.254.169.254:80) 的流量都会被重定向到组件所监听的 169.254.169.252:988 端口:

table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                 counter packets 2143 bytes 169432 jump KUBE-SERVICES
                iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 tcp dport 8080  counter packets 0 bytes 0 dnat to 169.254.169.252:987
                iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 181 bytes 10860 dnat to 169.254.169.252:988
        }
}

前面说到组件将通过 client ip 确定请求的来源 pod 以及会请求 apiserver 获取 service account，这里就涉及到组件是使用的什么凭证来访问 apiserver。组件使用的是节点上 kubelet 的凭证来访问的 apiserver（前面的组件 YAML 中包含了挂载 kubelet kubeconfig 的配置）。同时在部署组件时，将额外为 kubelet 的凭证授予 pods 和 serviceaccounts 的 get/list/watch RBAC 权限，用于获取当前节点的 pod 以及 service account 信息:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    components.gke.io/component-name: gke-metadata-server
    components.gke.io/component-version: 0.4.301
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gce:gke-metadata-server-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gce:gke-metadata-server-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    components.gke.io/component-name: gke-metadata-server
    components.gke.io/component-version: 0.4.301
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gce:gke-metadata-server-reader
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - serviceaccounts
  verbs:
  - get
  - watch
  - list

如果 pod 使用的 service account 没有绑定 IAM 角色，pod 内应用访问元数据服务获取的 sts token 将会是节点默认的 google serivce account 的 sts token。

service account token¶

对于每个 pod 内程序发起的请求，gke-metadata-server 组件不是直接使用 pod 容器所挂载 service account token，而是请求 apiserver 生成了一份新的 service account token。

容器所挂载的 service account token 的 payload 示例如下:

{
  "aud": [
    "https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test"
  ],
  "exp": ...,
  "iat": ...,
  "iss": "https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test",
  "jti": "...",
  "kubernetes.io": {...},
  "nbf": ...,
  "sub": "system:serviceaccount:demo-ns:demo-sa"
}

组件请求 apiserver 所生成的 service account token 的 payload 示例如下:

{
  "aud": [
    "test-gke-XXXXXX.svc.id.goog"
  ],
  "exp": ...,
  "iat": ...,
  "iss": "https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test",
  "jti": "...",
  "kubernetes.io": {...},
  "nbf": ...,
  "sub": "system:serviceaccount:demo-ns:demo-sa"
}

可以看到，主要的区别是 aud 的内容不一样。

sts token¶

gke-metadata-server 组件将使用获取到的 service account token 访问 STS 的 token API 获取一份 sts token。

对应的请求示例如下:

:authority: sts.googleapis.com
:method: POST
:path: /v1/token?alt=json&prettyPrint=false
:scheme: https
x-goog-api-client: gl-go/1.23.0--20240626-RC01 cl/646990413 +5a18e79687 X:fieldtrack,boringcrypto gdcl/0.177.0
user-agent: google-api-go-client/0.5 gke-metadata-server
content-type: application/json
content-length: 1673
accept-encoding: gzip

{"audience":"identitynamespace:test-gke-XXXXXX.svc.id.goog:https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test","grantType":"urn:ietf:params:oauth:grant-type:token-exchange","requestedTokenType":"urn:ietf:params:oauth:token-type:access_token","scope":"https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email","subjectToken":"XXX.XXX.XXX","subjectTokenType":"urn:ietf:params:oauth:token-type:jwt"}

:status: 200
content-type: application/json; charset=UTF-8
vary: Origin
vary: X-Origin
vary: Referer
content-encoding: gzip
date: ...
server: scaffolding on HTTPServer2
content-length: 1061
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff

{"access_token":"ya29.d.XXX","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","token_type":"Bearer","expires_in":3599}

其中请求 body 格式化后的内容如下:

{
  "audience": "identitynamespace:test-gke-XXXXXX.svc.id.goog:https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test",
  "grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
  "requestedTokenType": "urn:ietf:params:oauth:token-type:access_token",
  "scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email",
  "subjectToken": "XXX.XXX.XXX",
  "subjectTokenType": "urn:ietf:params:oauth:token-type:jwt"
}

响应 body 格式化后的内容如下:

{
  "access_token": "ya29.d.XXX",
  "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
  "token_type": "Bearer",
  "expires_in": 3599
}

与 GKE Workload Identity 的区别¶

Workload Identity Federation for GKE 与 GKE Workload Identity 特性（又叫 link Kubernetes ServiceAccounts to IAM）的区别如下：

比较项	GKE Workload Identity	Workload Identity Federation for GKE
需要创建 Google service account (GSA)	需要	不需要
需要配置 GSA 绑定的 IAM 角色	需要	需要配置 k8s service account (KSA) 绑定的角色
需要配置允许使用 KSA 扮演 GSA	需要	不需要
需要在集群内配置 KSA 的 GSA 信息	需要	不需要
绑定角色时支持指定多个 KSA/GSA	不支持	支持指定多个，比如某个命名空间下的所有SA, 某个集群内的所有SA
支持 project + namespace + KSA 维度授权	支持	支持
支持 cluster + namespace + KSA 维度授权	不支持	不支持
支持使用 hostNetwork 的应用	不支持	不支持
依赖部署 gke-metadata-server 组件	依赖	依赖
云产品API对获取的 sts token 的支持情况	几乎所有云产品 API 都支持	大部分云产品API都支持，部分云产品有限支持，少量云产品不支持

BTW, 虽然 Workload Identity Federation for GKE 方案的官方教程和文档中都是说的需要依赖 gke-metadata-server 这个组件，但是从前面的内容中我们也可以看到：我们其实也可以在不安装 gke-metadata-server 组件的情况下，使用该方案。具体来说就是，我们可以通过为应用 Pod 挂载所需的 service account token 然后在应用程序内直接访问 STS 提供的 Token API 的方式来解除对该组件的依赖。

参考资料¶

Exploring Workload Identity Federation for GKE

2025-01-05T00:00:00+00:00

In this article, we will briefly explore a feature called "Workload Identity Federation for GKE" that was recently announced by GKE in their official blog.

Features Overview¶

Workload Identity Federation for GKE is an improved version of the original GKE Workload Identity feature. The main improvement is that it needs less configuration and offers better user experience.

How to Use¶

Follow these steps to try this feature:

Create a GKE cluster with Workload Identity Federation for GKE enabled. You can find this option at: Create Cluster - Security - Enable Workload Identity.

Link IAM roles to the service account used by your test application. One service account can have multiple roles. For example:

$ kubectl create ns demo-ns
$ kubectl -n demo-ns create serviceaccount demo-sa

$ gcloud projects add-iam-policy-binding projects/test-gke-XXX \
    --role=roles/container.clusterViewer \
    --member=principal://iam.googleapis.com/projects/23182XXXXXXX/locations/global/workloadIdentityPools/test-gke-XXXXXX.svc.id.goog/subject/ns/<namespace-name>/sa/<service-account-name> \
    --condition=None

$ gcloud storage buckets add-iam-policy-binding gs://<bucket-name> \
--role=roles/storage.objectViewer \
--member=principal://iam.googleapis.com/projects/23182XXXXXXX/locations/global/workloadIdentityPools/test-gke-XXXXXX.svc.id.goog/subject/ns/<namespace-name>/sa/<service-account-name> \
--condition=None

Deploy the test application. The application pod needs to use the namespace and service account from step 2, and add nodeSelector to ensure scheduling on nodes with the Workload Identity Federation for GKE feature enabled

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: demo-ns
spec:
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: "true"
  serviceAccountName: demo-sa
  containers:
  - name: test-pod
    image: google/cloud-sdk:slim
    command: ["sleep","infinity"]

After the pod is running, enter the test-pod container and access the instance metadata service to obtain the STS token. The official SDK used by actual business applications will use a similar method to access the instance metadata service to obtain STS tokens for requesting cloud product APIs.

$ gcloud auth print-access-token
 ya29.d.XXX
$ curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
 {"access_token":"ya29.d.XXX","expires_in":3423,"token_type":"Bearer"}

The obtained STS token will have all the permissions of the roles associated in step 2.

$ gcloud container node-pools list --zone us-central1 --cluster test
 NAME          MACHINE_TYPE  DISK_SIZE_GB  NODE_VERSION
 default-pool  e2-medium     100           1.30.6-gke.1125000

$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 "https://storage.googleapis.com/storage/v1/b/demo-gke-workload-identity-federation/o"
 {
   "kind": "storage#objects"
 }

Workflow Explained¶

The workflow of Workload Identity Federation for GKE feature is as follows:

When a program in the application Pod container requests the instance metadata service (accessing http://metadata.google.internal, which actually accesses http://169.254.169.254:80) to get an STS token, the request will be redirected to 169.254.169.252:988. 169.254.169.252:988 is the port listened to by the gke-metadata-server service deployed on the node.
When the gke-metadata-server service receives the request, it will determine which Pod the request came from based on the client IP, then request the apiserver to generate a token corresponding to the service account used by that Pod.
gke-metadata-server will use the obtained service account token to access GCP's STS API to get an STS token, finally gke-metadata-server returns the obtained STS token to the client.

There are several key components and information that need special attention in this workflow, which will be explained one by one below.

gke-metadata-server¶

When enabling the Workload Identity Federation for GKE feature at the cluster or node pool level, a component named gke-metadata-server will be automatically deployed in the cluster. The workload YAML for this component is as follows:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "1"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: gke-metadata-server
  name: gke-metadata-server
  namespace: kube-system
spec:
  minReadySeconds: 90
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: gke-metadata-server
  template:
    metadata:
      annotations:
        components.gke.io/component-name: gke-metadata-server
        components.gke.io/component-version: 0.4.301
        monitoring.gke.io/path: /metricz
      creationTimestamp: null
      labels:
        addonmanager.kubernetes.io/mode: Reconcile
        k8s-app: gke-metadata-server
    spec:
      containers:
      - command:
        - /gke-metadata-server
        - --logtostderr
        - --token-exchange-endpoint=https://securetoken.googleapis.com/v1/identitybindingtoken
        - --workload-pool=test-gke-XXXXXX.svc.id.goog
        - --alts-service-suffixes-using-node-identity=storage.googleapis.com,bigtable.googleapis.com,bigtable2.googleapis.com,bigtablerls.googleapis.com,spanner.googleapis.com,spanner2.googleapis.com,spanner-rls.googleapis.com,grpclb.directpath.google.internal,grpclb-dualstack.directpath.google.internal,staging-wrenchworks.sandbox.googleapis.com,preprod-spanner.sandbox.googleapis.com,wrenchworks-loadtest.googleapis.com,wrenchworks-nonprod.googleapis.com
        - --identity-provider=https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test
        - --passthrough-ksa-list=anthos-identity-service:gke-oidc-envoy-sa,anthos-identity-service:gke-oidc-service-sa,gke-managed-dpv2-observability:hubble-relay,kube-system:antrea-controller,kube-system:container-watcher-pod-reader,kube-system:coredns,kube-system:egress-nat-controller,kube-system:event-exporter-sa,kube-system:fluentd-gcp-scaler,kube-system:gke-metrics-agent,kube-system:gke-spiffe-node-agent,kube-system:heapster,kube-system:konnectivity-agent,kube-system:kube-dns,kube-system:maintenance-handler,kube-system:metadata-agent,kube-system:network-metering-agent,kube-system:node-local-dns,kube-system:pkgextract-service,kube-system:pkgextract-cleanup-service,kube-system:securityprofile-controller,istio-system:istio-ingressgateway-service-account,istio-system:cluster-local-gateway-service-account,csm:csm-sync-agent,knative-serving:controller,kube-system:pdcsi-node-sa,kube-system:gcsfusecsi-node-sa,gmp-system:collector,gke-gmp-system:collector,gke-managed-cim:kube-state-metrics
        - --attributes=cluster-name=test,cluster-uid=392f63049deXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,cluster-location=us-central1
        - --cluster-uid=392f63049ded410XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        - --sts-endpoint=https://sts.googleapis.com
        - --token-exchange-mode=sts
        - --cloud-monitoring-endpoint=monitoring.googleapis.com:443
        - --iam-cred-service-endpoint=https://iamcredentials.googleapis.com
        - --cluster-project-number=2318XXXXXXXX
        - --cluster-location=us-central1
        - --cluster-name=test
        - --component-version=0.4.301
        - --ksa-cache-mode=watchchecker
        - --kcp-allow-watch-checker=true
        - --enable-mds-csi-driver=true
        - --csi-socket=/csi/csi.sock
        - --volumes-db=/var/run/gkemds.gke.io/csi/volumes.boltdb
        env:
        - name: GOMEMLIMIT
          value: "94371840"
        image: us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metadata-server:gke_metadata_server_20240702.00_p0@sha256:aea9cc887c91b9a05e5bb4bb604180772594a01f0828bbfacf30c77562ac7205
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            host: 127.0.0.1
            path: /healthz
            port: 989
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        name: gke-metadata-server
        ports:
        - containerPort: 987
          name: alts
          protocol: TCP
        - containerPort: 988
          name: metadata-server
          protocol: TCP
        - containerPort: 989
          name: debug-port
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            host: 127.0.0.1
            path: /healthz
            port: 989
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          privileged: true
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kubelet/kubeconfig
          name: kubelet-credentials
          readOnly: true
        - mountPath: /var/lib/kubelet/pki/
          name: kubelet-certs
          readOnly: true
        - mountPath: /var/run/
          name: container-runtime-interface
        - mountPath: /etc/srv/kubernetes/pki
          name: kubelet-pki
          readOnly: true
        - mountPath: /etc/ssl/certs/
          name: ca-certificates
          readOnly: true
        - mountPath: /home/kubernetes/bin/gke-exec-auth-plugin
          name: gke-exec-auth-plugin
          readOnly: true
        - mountPath: /sys/firmware/efi/efivars/
          name: efivars
          readOnly: true
        - mountPath: /dev/tpm0
          name: vtpm
          readOnly: true
        - mountPath: /csi
          name: csi-socket-dir
        - mountPath: /var/run/gkemds.gke.io/csi
          name: state-dir
        - mountPath: /var/lib/kubelet/pods
          mountPropagation: Bidirectional
          name: pods-dir
        - mountPath: /registration
          name: kubelet-registration-dir
      dnsPolicy: Default
      hostNetwork: true
      nodeSelector:
        iam.gke.io/gke-metadata-server-enabled: "true"
        kubernetes.io/os: linux
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: gke-metadata-server
      serviceAccountName: gke-metadata-server
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
        operator: Exists
      - effect: NoSchedule
        operator: Exists
      - key: components.gke.io/gke-managed-components
        operator: Exists
      volumes:
      - hostPath:
          path: /var/lib/kubelet/pki/
          type: Directory
        name: kubelet-certs
      - hostPath:
          path: /var/lib/kubelet/kubeconfig
          type: File
        name: kubelet-credentials
      - hostPath:
          path: /var/run/
          type: Directory
        name: container-runtime-interface
      - hostPath:
          path: /etc/srv/kubernetes/pki/
          type: Directory
        name: kubelet-pki
      - hostPath:
          path: /etc/ssl/certs/
          type: Directory
        name: ca-certificates
      - hostPath:
          path: /home/kubernetes/bin/gke-exec-auth-plugin
          type: File
        name: gke-exec-auth-plugin
      - hostPath:
          path: /sys/firmware/efi/efivars/
          type: Directory
        name: efivars
      - hostPath:
          path: /dev/tpm0
          type: CharDevice
        name: vtpm
      - hostPath:
          path: /var/lib/kubelet/plugins/gkemds.gke.io
          type: DirectoryOrCreate
        name: csi-socket-dir
      - hostPath:
          path: /var/lib/kubelet/pods
          type: Directory
        name: pods-dir
      - hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: Directory
        name: kubelet-registration-dir
      - hostPath:
          path: /var/lib/kubelet/plugins
          type: Directory
        name: kubelet-plugins-dir
      - hostPath:
          path: /var/run/gkemds.gke.io/csi
          type: DirectoryOrCreate
        name: state-dir

The gke-metadata-server component has the following characteristics:

The component's workload is a DaemonSet.
The component Pod uses hostNetwork: true and privileged: true.

The service inside the component will listen on ports 987, 988, and 989, where port 988 will be used to receive redirected requests for accessing the instance metadata service:

$ ss -atnlp |grep gke
LISTEN 0      1024               *:987              *:*    users:(("gke-metadata-se",pid=183706,fd=10))
LISTEN 0      1024               *:989              *:*    users:(("gke-metadata-se",pid=183706,fd=12))
LISTEN 0      1024               *:988              *:*    users:(("gke-metadata-se",pid=183706,fd=15))

The previously mentioned IP 169.254.169.252 is the local loopback (lo) IP address, so the port 988 listened to by gke-metadata-server includes the previously mentioned 169.254.169.252:988 port:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.169.252/32 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

All nodes with the Workload Identity Federation for GKE feature enabled will be configured with the following nftables rules during initialization, ensuring that traffic from business containers requesting the metadata service (http://metadata.google.internal, 169.254.169.254:80) will be redirected to the 169.254.169.252:988 port listened to by the component:

table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                 counter packets 2143 bytes 169432 jump KUBE-SERVICES
                iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 tcp dport 8080  counter packets 0 bytes 0 dnat to 169.254.169.252:987
                iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 181 bytes 10860 dnat to 169.254.169.252:988
        }
}

As mentioned earlier, the component will determine the source pod through the client IP and request the apiserver to get the service account. This involves what credentials the component uses to access the apiserver. The component uses the kubelet's credentials on the node to access the apiserver (the component YAML shown earlier includes the configuration for mounting kubelet kubeconfig). Additionally, when deploying the component, extra RBAC permissions for get/list/watch on pods and serviceaccounts will be granted to the kubelet's credentials, used for obtaining information about pods and service accounts on the current node:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    components.gke.io/component-name: gke-metadata-server
    components.gke.io/component-version: 0.4.301
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gce:gke-metadata-server-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gce:gke-metadata-server-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    components.gke.io/component-name: gke-metadata-server
    components.gke.io/component-version: 0.4.301
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gce:gke-metadata-server-reader
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - serviceaccounts
  verbs:
  - get
  - watch
  - list

If the service account used by the pod is not bound to any IAM role, the STS token obtained by the application in the pod when accessing the metadata service will be the STS token of the node's default Google service account.

service account token¶

For requests initiated by programs within each pod, the gke-metadata-server component doesn't directly use the service account token mounted in the pod container, but instead requests the apiserver to generate a new service account token.

Here's an example of the payload for a service account token mounted in the container:

{
  "aud": [
    "https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test"
  ],
  "exp": ...,
  "iat": ...,
  "iss": "https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test",
  "jti": "...",
  "kubernetes.io": {...},
  "nbf": ...,
  "sub": "system:serviceaccount:demo-ns:demo-sa"
}

Here's an example of the payload for a service account token generated by the apiserver when requested by the component:

{
  "aud": [
    "test-gke-XXXXXX.svc.id.goog"
  ],
  "exp": ...,
  "iat": ...,
  "iss": "https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test",
  "jti": "...",
  "kubernetes.io": {...},
  "nbf": ...,
  "sub": "system:serviceaccount:demo-ns:demo-sa"
}

As we can see, the main difference is in the content of the aud field.

sts token¶

The gke-metadata-server component will use the obtained service account token to access the STS token API to get an STS token.

Here's an example of the corresponding request:

:authority: sts.googleapis.com
:method: POST
:path: /v1/token?alt=json&prettyPrint=false
:scheme: https
x-goog-api-client: gl-go/1.23.0--20240626-RC01 cl/646990413 +5a18e79687 X:fieldtrack,boringcrypto gdcl/0.177.0
user-agent: google-api-go-client/0.5 gke-metadata-server
content-type: application/json
content-length: 1673
accept-encoding: gzip

{"audience":"identitynamespace:test-gke-XXXXXX.svc.id.goog:https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test","grantType":"urn:ietf:params:oauth:grant-type:token-exchange","requestedTokenType":"urn:ietf:params:oauth:token-type:access_token","scope":"https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email","subjectToken":"XXX.XXX.XXX","subjectTokenType":"urn:ietf:params:oauth:token-type:jwt"}

:status: 200
content-type: application/json; charset=UTF-8
vary: Origin
vary: X-Origin
vary: Referer
content-encoding: gzip
date: ...
server: scaffolding on HTTPServer2
content-length: 1061
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff

{"access_token":"ya29.d.XXX","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","token_type":"Bearer","expires_in":3599}

The formatted content of the request body is as follows:

{
  "audience": "identitynamespace:test-gke-XXXXXX.svc.id.goog:https://container.googleapis.com/v1/projects/test-gke-XXXXXX/locations/us-central1/clusters/test",
  "grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
  "requestedTokenType": "urn:ietf:params:oauth:token-type:access_token",
  "scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email",
  "subjectToken": "XXX.XXX.XXX",
  "subjectTokenType": "urn:ietf:params:oauth:token-type:jwt"
}

The formatted content of the response body is as follows:

{
  "access_token": "ya29.d.XXX",
  "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
  "token_type": "Bearer",
  "expires_in": 3599
}

Differences from GKE Workload Identity¶

Here are the differences between Workload Identity Federation for GKE and GKE Workload Identity feature (also known as linking Kubernetes ServiceAccounts to IAM):

Comparison Item	GKE Workload Identity	Workload Identity Federation for GKE
Requires Google service account (GSA)	Required	Not Required
IAM role binding configuration	Requires GSA role binding	Requires k8s service account (KSA) role binding
KSA impersonating GSA configuration	Required	Not Required
GSA info configuration for KSA in cluster	Required	Not Required
Multiple KSA/GSA binding support	Not supported	Supported (e.g., all SAs in namespace, all SAs in cluster)
Authorization at project + namespace + KSA	Supported	Supported
Authorization at cluster + namespace + KSA	Not supported	Not supported
Support for hostNetwork applications	Not supported	Not supported
Depends on gke-metadata-server component	Required	Required
Cloud API support for obtained STS token	Almost all cloud product APIs	Most cloud APIs supported, some limited support, few unsupported

By the way, although the official tutorials and documentation for Workload Identity Federation for GKE mention that it requires the gke-metadata-server component, from what we discussed earlier, we can see that: we can actually use this solution without installing the gke-metadata-server component. Specifically, we can remove the dependency on this component by mounting the required service account token to the application Pod and then directly accessing the Token API provided by STS within our applications.