Exploring the New Features of Amazon EKS Pod Identity

In this piece, we're taking a quick dive into a fresh security feature for EKS introduced this week by AWS, aptly called Amazon EKS Pod Identity.

Introducing the Features¶

Amazon EKS Pod Identity enhances AWS's existing IAM roles for service accounts (IRSA) feature. This new addition allows users to more easily assign secure AWS API access permissions to Pods. Moreover, all related configuration and management tasks can be handled directly via the AWS API or its console interface.

How to Use¶

To securely provide AWS API access to applications inside Pods using the EKS Pod Identity feature, users simply need to follow these steps:

Start by creating an IAM role, making sure it trusts pods.eks.amazonaws.com. Here's an example of what the role trust policy might look like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

The next step is to install the eks-pod-identity-agent within your EKS cluster, which can be done via the console.

aws eks create-addon \
--cluster-name <CLUSTER_NAME> \
--addon-name eks-pod-identity-agent \
--addon-version v1.0.0-eksbuild.1

Following that, set up the link between the Service Account utilized by your application Pod and the AWS IAM role. This configuration allows apps using this Service Account to assume a designated IAM role, and it can be managed via the console.

aws eks create-pod-identity-association \
  --cluster-name <CLUSTER_NAME> \
  --namespace <NAMESPACE> \
  --service-account <SERVICE_ACCOUNT_NAME> \
  --role-arn <IAM_ROLE_ARN>

Lastly, update your applications to incorporate the most recent AWS SDK version that's compatible with EKS Pod Identity. In your code, either use the SDK's built-in default credential search logic or make explicit calls to the Container credential provider which is a dependency of EKS Pod Identity.

Workflow Explained¶

The workflow for the EKS Pod Identity feature operates in the following manner:

When a user or Controller submits a Pod to the apiserver, this action initiates the mutating webhook process of eks-pod-identity-webhook.
The mutating webhook process of eks-pod-identity-webhook is responsible for mounting the service account OIDC token file to the Pod and setting up the necessary environment variables, namely AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE and AWS_CONTAINER_CREDENTIALS_FULL_URI.

 - env:
  - name: AWS_STS_REGIONAL_ENDPOINTS
    value: regional
  - name: AWS_DEFAULT_REGION
    value: us-west-2
  - name: AWS_REGION
    value: us-west-2
  - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
    value: http://169.254.170.23/v1/credentials
  - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
    value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
  volumeMounts:
  - mountPath: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount
    name: eks-pod-identity-token
    readOnly: true
volumes:
- name: eks-pod-identity-token
  projected:
    defaultMode: 420
    sources:
    - serviceAccountToken:
        audience: pods.eks.amazonaws.com
        expirationSeconds: 86400
        path: eks-pod-identity-token

Applications inside the Pod container, utilizing AWS SDK, will employ the service account OIDC token, sourced from the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment variable. This token is used to access the URL specified by AWS_CONTAINER_CREDENTIALS_FULL_URI (http://169.254.170.23/v1/credentials) to acquire the AWS STS token.

$ curl $AWS_CONTAINER_CREDENTIALS_FULL_URI -H "Authorization: $(cat $AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE)" 2>/dev/null |jq
{
  "AccessKeyId": "ASXXXXXXXXXXXXX",
  "SecretAccessKey": "+j5XXXXXXXX",
  "Token": "IQoJb3JpXXXXXXX",
  "AccountId": "5XXXXXXXXXXX",
  "Expiration": "2023-12-03T13:13:26Z"
}

The URL referred to by AWS_CONTAINER_CREDENTIALS_FULL_URI actually connects to the service exposed by the eks-pod-identity-agent component's Pod, deployed on the same node as the requesting Pod. Upon receiving a request, the eks-pod-identity-agent uses the provided OIDC token to interact with EKS's recently introduced AssumeRoleForPodIdentity API. This process is to acquire the needed AWS STS token, which is then relayed back to the client.

$ aws eks-auth assume-role-for-pod-identity --cluster-name test --token $(cat $AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE) |jq
{
  "subject": {
    "namespace": "default",
    "serviceAccount": "default"
  },
  "audience": "pods.eks.amazonaws.com",
  "podIdentityAssociation": {
    "associationArn": "arn:aws:eks:us-west-2:5XXXXXXXXXXX:podidentityassociation/test/a-6aaXXXXXXXXXXXXXX",
    "associationId": "a-6aaXXXXXXXXXXXXXX"
  },
  "assumedRoleUser": {
    "arn": "arn:aws:sts::5XXXXXXXXXXX:assumed-role/test-eks-pod-identity/eks-test-XXXXXXXXXXXXXXXX",
    "assumeRoleId": "ARXXXXXXXXXXXXXXXXXXX:eks-test-XXXXXXXXXXXXXXXXXX"
  },
  "credentials": {
    "sessionToken": "IQoXXXXX",
    "secretAccessKey": "nR4XXXXXX",
    "accessKeyId": "ASXXXXXXXXXXXXXXXXXX",
    "expiration": "2023-12-03T13:37:22+00:00"
  }
}

The AWS SDK utilized by the application leverages the acquired STS token to access the specific AWS cloud product APIs it needs.

In this process, there are a few crucial components and pieces of information that merit particular emphasis. Let's break them down one by one.

eks-pod-identity-webhook¶

The EKS Pod Identity functionality seamlessly integrates with the eks-pod-identity-webhook component, originally part of the IRSA feature. This component is responsible for automatically adding the necessary OIDC token and environment variable settings to the Pods. Moreover, the eks-pod-identity-webhook deployed in the EKS cluster control plane has been specially tailored for EKS Pod Identity. It only adds EKS Pod Identity configurations to Pods when the corresponding service account lacks IRSA-specific settings and is linked to IAM role information, as outlined in the previously mentioned usage instructions.

The changes implemented in eks-pod-identity-webhook to facilitate EKS Pod Identity can be reviewed in its open-source version, though it's important to note that this version does not include the logic for identifying associated roles. The pertinent Pull Requests (PRs) that detail these modifications are #189 and #196.

eks-pod-identity-agent¶

The EKS Pod Identity implementation suggested by the official documentation involves deploying a component called eks-pod-identity-agent within the cluster. The YAML configuration for this component's workload is outlined below.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "1"
  creationTimestamp: "2023-12-03T04:24:15Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: eks-pod-identity-agent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: eks-pod-identity-agent
    app.kubernetes.io/version: 0.0.25
    helm.sh/chart: eks-pod-identity-agent-1.0.0
  name: eks-pod-identity-agent
  namespace: kube-system
  resourceVersion: "8978"
  uid: 08c03e91-69d4-460b-8acb-b9f9f546dd87
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: eks-pod-identity-agent
      app.kubernetes.io/name: eks-pod-identity-agent
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: eks-pod-identity-agent
        app.kubernetes.io/name: eks-pod-identity-agent
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
                - arm64
              - key: eks.amazonaws.com/compute-type
                operator: NotIn
                values:
                - fargate
      containers:
      - args:
        - --port
        - "80"
        - --cluster-name
        - test
        - --probe-port
        - "2703"
        command:
        - /go-runner
        - /eks-pod-identity-agent
        - server
        env:
        - name: AWS_REGION
          value: us-west-2
        image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/eks-pod-identity-agent:0.0.25
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            host: localhost
            path: /healthz
            port: probes-port
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        name: eks-pod-identity-agent
        ports:
        - containerPort: 80
          name: proxy
          protocol: TCP
        - containerPort: 2703
          name: probes-port
          protocol: TCP
        readinessProbe:
          failureThreshold: 30
          httpGet:
            host: localhost
            path: /readyz
            port: probes-port
            scheme: HTTP
          initialDelaySeconds: 1
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        resources: {}
        securityContext:
          capabilities:
            add:
            - CAP_NET_BIND_SERVICE
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostNetwork: true
      initContainers:
      - command:
        - /go-runner
        - /eks-pod-identity-agent
        - initialize
        image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/eks-pod-identity-agent:0.0.25
        imagePullPolicy: Always
        name: eks-pod-identity-agent-init
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 10%
    type: RollingUpdate
status:
  currentNumberScheduled: 1
  desiredNumberScheduled: 1
  numberAvailable: 1
  numberMisscheduled: 0
  numberReady: 1
  observedGeneration: 1

The eks-pod-identity-agent component is characterized by the following features:

It depends on the node IAM role having been assigned specific IAM permission policies as follows.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "eks-auth:AssumeRoleForPodIdentity",
           ],
           "Resource": "*"
       }
   ]
}

The component's Pod is configured to use hostNetwork.
The service inside the component is set up to listen on ports 80 and 2703.
For port 80, it listens on the IPv4 address 169.254.170.23 and the IPv6 address [fd00:ec2::23].

$ ss -anltp |grep eks-pod
LISTEN 0      4096        127.0.0.1:2703       0.0.0.0:*    users:(("eks-pod-identit",pid=3798,fd=5))
LISTEN 0      4096   169.254.170.23:80         0.0.0.0:*    users:(("eks-pod-identit",pid=3798,fd=4))
LISTEN 0      4096   [fd00:ec2::23]:80            [::]:*    users:(("eks-pod-identit",pid=3798,fd=3))

The component employs an init container to establish a network interface named pod-id-link0 on the node. This interface is configured with the IPv4 address 169.254.170.23 and the IPv6 address [fd00:ec2::23].

pod-id-link0: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 169.254.170.23  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::2078:53ff:fe2f:c723  prefixlen 64  scopeid 0x20<link>
        inet6 fd00:ec2::23  prefixlen 128  scopeid 0x0<global>
        ether 22:78:53:2f:c7:23  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10  bytes 700 (700.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Additionally, the init container configures specific routing rules to guarantee that traffic directed to 169.254.170.23 is handled on the local machine.

169.254.170.23 dev pod-id-link0

AssumeRoleForPodIdentity¶

EKS has introduced a new API, AssumeRoleForPodIdentity. This API is utilized to acquire an STS token that enables the assuming of the IAM role linked to a service account, through the use of the service account's OIDC token.

service account oidc token 中的 payload 的内容示例如下。

{
  "aud": [
    "pods.eks.amazonaws.com"
  ],
  "exp": 1701672885,
  "iat": 1701586485,
  "iss": "https://oidc.eks.us-west-2.amazonaws.com/id/DA16E524AXXXXXX",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "test-596475c6d5-xgvml",
      "uid": "924e66b2-841e-4969-9fa7-a19f3f6f1029"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "eca98cb1-ca2e-469b-8f9d-2be9f5a3354f"
    }
  },
  "nbf": 1701586485,
  "sub": "system:serviceaccount:default:default"
}

This OIDC token, when compared to the one used in the IRSA solution, differs only in its audience (aud) field, which is sts.amazonaws.com in IRSA's case. Otherwise, they are identical.

From this, I infer that the backend process of AssumeRoleForPodIdentity likely follows these steps:

Initially, the client's OIDC token is parsed and its validity and legitimacy are verified based on the OIDC protocol.
Next, the role that needs to be assumed is identified, using the service account's configured association with an IAM role.
Finally, the STS API is invoked to assume the identified role, thereby generating the STS token required by the client.

Association of Service Account with IAM Role¶

EKS has introduced a suite of APIs to streamline the process for users to link service accounts with IAM roles. This enhancement is complemented by new interfaces on the EKS console for managing these associations. Notably, these associations are maintained within EKS and do not alter the trust policy of the roles.

CreatePodIdentityAssociation: To link a role with a service account.
DescribePodIdentityAssociation: To view detailed information about a specific role association.
UpdatePodIdentityAssociation: To alter the role information linked to a service account.
DeletePodIdentityAssociation: To remove a particular role association record.
ListPodIdentityAssociations: To access a list of all current role association records.

Access and Permission Management¶

Under the IRSA approach, it's necessary to include the permissible cluster OIDC provider and service account details in the IAM role's trust policy. However, with EKS Pod Identity, altering the role's trust policy is no longer required. Instead, you can specify which roles are accessible to which service accounts in specific clusters using the EKS console or the EKS API, as explained earlier. These associations are maintained within EKS, reducing the need for regular updates to the trust policy of roles.

The associated IAM role only needs to be configured once with the following trust policy, allowing EKS to assume the role based on the EKS Pod Identity scheme:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

This method allows EKS Pod Identity to address a limitation encountered in IRSA, where a role's trust policy size constraints meant it could only be linked to a few service accounts (details available in #148). With EKS Pod Identity, there's no limit to the number of service accounts that can be associated with a single role.

attribute-based access control (ABAC)¶

EKS Pod Identity also ushers in an advanced attribute-based access control (ABAC) feature. STS tokens acquired via EKS Pod Identity inherently include tags like cluster details, namespace, and service account. Utilizing IAM's tag-based authorization functionality, users can now employ these attributes for more granular access control when setting up permission policies for roles.

STS tokens acquired via EKS Pod Identity automatically include these tags:

eks-cluster-arn: ARN of the current cluster.
eks-cluster-name: Name of the current cluster.
kubernetes-namespace: Name of the namespace containing the service account.
kubernetes-service-account: Name of the service account.
kubernetes-pod-name: Name of the pod utilizing the service account.
kubernetes-pod-uid: UID of the pod that is using the service account.

"tags": [
    {
        "key": "eks-cluster-arn",
        "value": "arn:aws:eks:us-west-2:5XXXXXXXXXXX:cluster/test"
    },
    {
        "key": "eks-cluster-name",
        "value": "test"
    },
    {
        "key": "kubernetes-namespace",
        "value": "default"
    },
    {
        "key": "kubernetes-service-account",
        "value": "default"
    },
    {
        "key": "kubernetes-pod-name",
        "value": "test-596475c6d5-xgvml"
    },
    {
        "key": "kubernetes-pod-uid",
        "value": "924e66b2-841e-4969-9fa7-a19f3f6f1029"
    }
]

In the process of crafting role permission policies, the format ${aws:PrincipalTag/<tag-key>} can be employed. This approach is used within the Condition section of the permission policy to denote the value of a particular tag present in the attributes of the credentials.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectTagging"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "s3:ExistingObjectTag/eks-cluster-name": "${aws:PrincipalTag/eks-cluster-name}"
                },
                "StringEqualsIfExists": {
                    "aws:ResourceTag/kubernetes-namespace": "s3-demo",
                    "aws:Resourcelag/kubernetes-service-account": "${aws:PrincipalTag/kubernetes-service-account}"
                }
            }
        }
    ]
}

With the example permission policy provided earlier, we can tailor the permissions of STS tokens acquired via the EKS Pod Identity feature in the following ways:

Allow access only to S3 objects tagged with eks-cluster-name, where the tag's value aligns with the name of the cluster tied to the current STS token.
Permit access to an S3 object only if it carries a kubernetes-namespace tag, and only if this tag's value is s3-demo.
Authorize access to an S3 object only if it bears a kubernetes-service-account tag, and solely if the tag's value corresponds to the name of the service account associated with the current STS token.

Distinguishing from IRSA¶

Considering the details shared earlier, a comparison between EKS Pod Identity and IRSA highlights several key differences, which are as follows:

Comparison Item	EKS Pod Identity	IRSA
Additional Permissions for Node IAM Role	Requires `eks-auth:AssumeRoleForPodIdentity`	Not Required
Creation of IAM OIDC Provider	Not Required	Necessary for Each Cluster
Associating Roles with Service Account	Via EKS API/Console	Service Account Annotation + Role Trust Policy Modification
No. of Service Accounts Per Role	Unlimited	Limited by 4096-Character Trust Policy, Typically Around 10 Service Accounts or Clusters
No. of Service Accounts Per Role	Unlimited
No. of Roles Per Service Account	One	Unlimited
Role Trust Policy Modification Frequency	Once Only for Trust with `pods.eks.amazonaws.com` and Necessary Actions	At Least Once for Each Cluster
Role Trust Policy Modification Frequency		At Least Once for Each Cluster
Access Control Based on Service Account Attributes	Supported via ABAC	Not Supported
Supported Cluster Types	Only EKS, Not AWS Outposts, EKS Anywhere, AWS Fargate, or Self-Built Clusters	All Except AWS Outposts
Supported Cluster Types		All Except AWS Outposts
Data Plane Dependencies	kubelet + eks-pod-identity-agent	kublet
Stability Risks	Relies on Stability of Cluster Control/Data Plane + EKS API; Dependency on eks-pod-identity-agent Stability; Risks of STS Token Access Issues if Application Pods Launch Before eks-pod-identity-agent Pod (like in Auto-Scaling Scenarios with Same `hostNetwork` Config, Not Recommended for CNI, CSI, etc).	Depends on Stability of Cluster Control/Data Plane + STS API

By the way, although the official tutorials and documentation of the EKS Pod Identity solution mention the need to rely on the eks-pod-identity-agent component, we can also implement this solution without installing the eks-pod-identity-agent component. Specifically, we can remove the dependency on this component by directly accessing the AssumeRoleForPodIdentity API provided by EKS within the application.