# GCP - Vertex AI Enum

{{#include ../../../banners/hacktricks-training.md}}

## Vertex AI

[Vertex AI](https://cloud.google.com/vertex-ai) is Google Cloud's **unified machine learning platform** for building, deploying, and managing AI models at scale. It combines various AI and ML services into a single, integrated platform, enabling data scientists and ML engineers to:

- **Train custom models** using AutoML or custom training
- **Deploy models** to scalable endpoints for predictions
- **Manage the ML lifecycle** from experimentation to production
- **Access pre-trained models** from Model Garden
- **Monitor and optimize** model performance

### Key Components

#### Models

Vertex AI **models** represent trained machine learning models that can be deployed to endpoints for serving predictions. Models can be:

- **Uploaded** from custom containers or model artifacts
- Created through **AutoML** training
- Imported from **Model Garden** (pre-trained models)
- **Versioned** with multiple versions per model

Each model has metadata including its framework, container image URI, artifact location, and serving configuration.

#### Endpoints

**Endpoints** are resources that host deployed models and serve online predictions. Key features:

- Can host **multiple deployed models** (with traffic splitting)
- Provide **HTTPS endpoints** for real-time predictions
- Support **autoscaling** based on traffic
- Can use **private** or **public** access
- Support **A/B testing** through traffic splitting

#### Custom Jobs

**Custom jobs** allow you to run custom training code using your own containers or Python packages. Features include:

- Support for **distributed training** with multiple worker pools
- Configurable **machine types** and **accelerators** (GPUs/TPUs)
- **Service account** attachment for accessing other GCP resources
- Integration with **Vertex AI Tensorboard** for visualization
- **VPC connectivity** options

#### Hyperparameter Tuning Jobs

These jobs automatically **search for optimal hyperparameters** by running multiple training trials with different parameter combinations.

#### Model Garden

**Model Garden** provides access to:

- Pre-trained Google models
- Open-source models (including Hugging Face)
- Third-party models
- One-click deployment capabilities

#### Tensorboards

**Tensorboards** provide visualization and monitoring for ML experiments, tracking metrics, model graphs, and training progress.

### Service Accounts & Permissions

By default, Vertex AI services use the **Compute Engine default service account** (`PROJECT_NUMBER-compute@developer.gserviceaccount.com`), which has **Editor** permissions on the project. However, you can specify custom service accounts when:

- Creating custom jobs
- Uploading models
- Deploying models to endpoints

This service account is used to:
- Access training data in Cloud Storage
- Write logs to Cloud Logging
- Access secrets from Secret Manager
- Interact with other GCP services

### Data Storage

- **Model artifacts** are stored in **Cloud Storage** buckets
- **Training data** typically resides in Cloud Storage or BigQuery
- **Container images** are stored in **Artifact Registry** or Container Registry
- **Logs** are sent to **Cloud Logging**
- **Metrics** are sent to **Cloud Monitoring**

### Encryption

By default, Vertex AI uses **Google-managed encryption keys**. You can also configure:

- **Customer-managed encryption keys (CMEK)** from Cloud KMS
- Encryption applies to model artifacts, training data, and endpoints

### Networking

Vertex AI resources can be configured for:

- **Public internet access** (default)
- **VPC peering** for private access
- **Private Service Connect** for secure connectivity
- **Shared VPC** support

### Enumeration

```bash
# List models
gcloud ai models list --region=<region>
gcloud ai models describe <model-id> --region=<region>
gcloud ai models list-version <model-id> --region=<region>

# List endpoints
gcloud ai endpoints list --region=<region>
gcloud ai endpoints describe <endpoint-id> --region=<region>
gcloud ai endpoints list --list-model-garden-endpoints-only --region=<region>

# List custom jobs
gcloud ai custom-jobs list --region=<region>
gcloud ai custom-jobs describe <job-id> --region=<region>

# Stream logs from a running job
gcloud ai custom-jobs stream-logs <job-id> --region=<region>

# List hyperparameter tuning jobs
gcloud ai hp-tuning-jobs list --region=<region>
gcloud ai hp-tuning-jobs describe <job-id> --region=<region>

# List model monitoring jobs
gcloud ai model-monitoring-jobs list --region=<region>
gcloud ai model-monitoring-jobs describe <job-id> --region=<region>

# List Tensorboards
gcloud ai tensorboards list --region=<region>
gcloud ai tensorboards describe <tensorboard-id> --region=<region>

# List indexes (for vector search)
gcloud ai indexes list --region=<region>
gcloud ai indexes describe <index-id> --region=<region>

# List index endpoints
gcloud ai index-endpoints list --region=<region>
gcloud ai index-endpoints describe <index-endpoint-id> --region=<region>

# Get operations (long-running operations status)
gcloud ai operations describe <operation-id> --region=<region>

# Test endpoint predictions (if you have access)
gcloud ai endpoints predict <endpoint-id> \
  --region=<region> \
  --json-request=request.json

# Make direct predictions (newer API)
gcloud ai endpoints direct-predict <endpoint-id> \
  --region=<region> \
  --json-request=request.json
```

### Model Information Gathering

```bash
# Get detailed model information including versions
gcloud ai models describe <model-id> --region=<region>

# Check specific model version
gcloud ai models describe <model-id>@<version> --region=<region>

# List all versions of a model
gcloud ai models list-version <model-id> --region=<region>

# Get model artifact location (usually a GCS bucket)
gcloud ai models describe <model-id> --region=<region> --format="value(artifactUri)"

# Get container image URI
gcloud ai models describe <model-id> --region=<region> --format="value(containerSpec.imageUri)"
```

### Endpoint Details

```bash
# Get endpoint details including deployed models
gcloud ai endpoints describe <endpoint-id> --region=<region>

# Get endpoint URL
gcloud ai endpoints describe <endpoint-id> --region=<region> --format="value(deployedModels[0].displayName)"

# Get service account used by endpoint
gcloud ai endpoints describe <endpoint-id> --region=<region> --format="value(deployedModels[0].serviceAccount)"

# Check traffic split between models
gcloud ai endpoints describe <endpoint-id> --region=<region> --format="value(trafficSplit)"
```

### Custom Job Information

```bash
# Get job details including command, args, and service account
gcloud ai custom-jobs describe <job-id> --region=<region>

# Get service account used by job
gcloud ai custom-jobs describe <job-id> --region=<region> --format="value(jobSpec.workerPoolSpecs[0].serviceAccount)"

# Get container image used
gcloud ai custom-jobs describe <job-id> --region=<region> --format="value(jobSpec.workerPoolSpecs[0].containerSpec.imageUri)"

# Check environment variables (may contain secrets)
gcloud ai custom-jobs describe <job-id> --region=<region> --format="value(jobSpec.workerPoolSpecs[0].containerSpec.env)"

# Get network configuration
gcloud ai custom-jobs describe <job-id> --region=<region> --format="value(jobSpec.network)"
```

### Access Control

```bash
# Note: IAM policies for individual Vertex AI resources are managed at the project level
# Check project-level permissions
gcloud projects get-iam-policy <project-id>

# Check service account permissions
gcloud iam service-accounts get-iam-policy <service-account-email>

# Check if endpoints allow unauthenticated access
# This is controlled by IAM bindings on the endpoint
gcloud projects get-iam-policy <project-id> \
  --flatten="bindings[].members" \
  --filter="bindings.role:aiplatform.user"
```

### Storage and Artifacts

```bash
# Models and training jobs often store artifacts in GCS
# List buckets that might contain model artifacts
gsutil ls

# Common artifact locations:
# gs://<project>-aiplatform-<region>/
# gs://<project>-vertex-ai/
# gs://<custom-bucket>/vertex-ai/

# Download model artifacts if accessible
gsutil -m cp -r gs://<bucket>/path/to/artifacts ./artifacts/

# Check for notebooks in AI Platform Notebooks
gcloud notebooks instances list --location=<location>
gcloud notebooks instances describe <instance-name> --location=<location>
```

### Model Garden

```bash
# List Model Garden endpoints
gcloud ai endpoints list --list-model-garden-endpoints-only --region=<region>

# Model Garden models are often deployed with default configurations
# Check for publicly accessible endpoints
```

### Privilege Escalation

In the following page, you can check how to **abuse Vertex AI permissions to escalate privileges**:

{{#ref}}
../gcp-privilege-escalation/gcp-vertex-ai-privesc.md
{{#endref}}

## References

- [https://cloud.google.com/vertex-ai/docs](https://cloud.google.com/vertex-ai/docs)
- [https://cloud.google.com/vertex-ai/docs/reference/rest](https://cloud.google.com/vertex-ai/docs/reference/rest)

{{#include ../../../banners/hacktricks-training.md}}
