Set up a storage integration with GCP (Google Cloud Platform)

How to import assets in private buckets on AWS S3 for labeling on Ango Hub

Administrators and project managers can import assets to Ango Hub from GCP private buckets.

Once you have created a storage integration by following the steps in this page, you are ready to import files by providing Ango Hub with URLs to files in your private buckets. You may follow the steps in this page to do so: Importing Cloud (Remote) Assets

Configure CORS

The CORS header below allows Ango Hub to send a request to your cloud storage, and allows your cloud storage to explicitly allow requests from Hub. This is a necessary step to ensure Hub can connect to your private bucket.

The steps to follow to set up CORS with Ango Hub can be found here.

Set Up Google Cloud Platform

Step 1: Create the Service Account

This command creates the new service account within your project.

gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} \
    --project=${PROJECT_ID} \
    --description="Service account to read objects and list project buckets" \
    --display-name="Storage Reader and Bucket Lister"

Step 2: Grant Permissions to Read Objects in a Specific Bucket

This command grants the service account the roles/storage.objectViewer role on a specific bucket. This allows the service account to see and download the objects within that bucket.

gcloud storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
    --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/storage.objectViewer"

Permissions included in roles/storage.objectViewer:

• storage.objects.get: Allows reading/downloading objects.

• storage.objects.list: Allows listing objects within the bucket.

Step 3: Define and Create a Custom Role to List Buckets (Optional for bucket listing on UI)

This is optional for File Explorer bucket listing, you can also manually enter your bucket name if you don't provide this permission.

To securely grant the permission to list all buckets in the project without assigning overly broad roles, we will create a custom IAM role. First, create a file named bucket-lister-role.yaml with the following content. This file defines a new role that contains only the storage.buckets.list permission. bucket-lister-role.yaml:

title: "Bucket Lister"
description: "Allows a user to list GCS buckets in a project."
stage: "GA"
includedPermissions:
- storage.buckets.list

Next, run the following command to create the custom role (named bucketLister) in your project using the definition file.

gcloud iam roles create bucketLister --project=${PROJECT_ID} \
    --file=bucket-lister-role.yaml

Step 4: Grant the Custom Role to the Service Account (Optional for bucket listing on UI) Finally, assign the new bucketLister custom role to your service account at the project level. This grants it the ability to list all buckets in the project.

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="projects/${PROJECT_ID}/roles/bucketLister"

Connect your Cloud Storage

Once you've set up CORS for your bucket, you will need to create a connection between Hub and the bucket itself.

To link your bucket to Hub, you will need a JSON obtained from the GCP dashboard called Service Account Key.

Here are Google's instructions on how to obtain the necessary Service Account Key JSON from your GCP console.

Once you have downloaded the JSON, go to your organization's page, then click on Integrations and Add Integration.

From the dialog that pops up, pick GCP as the provider and drag and drop the Service Account Key JSON you've just downloaded.

After clicking OK, your bucket will be linked to Hub, and it will show up in your list of integrations.

Integrations belong to your organization, not your user.

If you want to be able to access your non-public cloud data in multiple organizations, you will have to integrate Hub with GCP in every one of them.

Preparing and Uploading the JSON for Import

After connecting your bucket, you will need to prepare a JSON file containing each asset’s external ID as well as the asset's full absolute path and, optionally, other information.

Follow the steps outlined in the docs page on Importing Cloud (Remote) Assets.