Copilot

Copilot is an AI-powered assistant that helps you during annotation and review. Copilot can see what you see (if you allow it) and knows how to create annotations and answer classification questions. When enabled in a project, it's always available to all users.

Copilot is powered by your own LLM of choice. You control the API key used to power Copilot.

If you select an LLM hosted in a particular region, and give Copilot the ability to see asset contents, the asset contents will therefore be sent for processing in that region, even if it's different from the region your deployment of Ango Hub is in. (Default: EU)

Modalities in which Copilot is available

• Single image

• Markdown

What Copilot can do

• See image assets and detect what's on them

• See project details, such as the project's category schema

• Create bounding box annotations

• Answer top-level classifications (no attributes)

In addition,

• Granular permissions so it can only see what you tell it to see

• All annotations or classifications created by Copilot will be clearly marked as having created by Copilot, both on the UI and in the export

How to set up Copilot

Step 1: Add an LLM to your organization

This LLM will be used as the conversational model powering your Copilot. If you already have the LLM you wish to use in your organization, you may skip this step.

To add an LLM to your organization, please follow the steps outlined in this docs page: Adding and Managing LLMs. You must be an organization admin or owner.

Step 2: Create an Agent User in your organization

When Copilot creates annotations or answers classification questions, it does so through an API key generated by the Agent User.

To create an Agent User, from the Organization page, navigate to the Agents tab and click on "Add Agent":

From the dialog that appears, give the agent a name and click on OK:

Step 3: Set Copilot settings in your project

Navigate to the project where you'd like to use Copilot. Then, enter the Copilot section of the Settings tab. You'll be able to set all of Copilot's settings from here.

Enable Copilot

This is the master switch to turn Copilot on or off for the project. From here, you can also pick the agent user. Copilot will make changes using this user's name.

Model Selection

You can pick here which LLMs to use for different parts of Copilot.

At the moment, for visual annotation, only the YOLOv11 model is available. It is hosted together with Ango Hub, wherever your current deployment is located.

Conversation history

If enabled, the conversation history between users and Copilot is saved in a remote storage of your choice. You can pick the storage from here.

Conversations will be saved in the bucket you specify at /copilot/${projectId}/${conversationId}.json

Conversation sample

[
  {
    "content": [
      { "type": "text", "text": "Hey! Just wanted to let you know I deployed the latest update." }
    ],
    "role": "assistant",
    "timestamp": "2025-10-28T11:32:45.921Z"
  },
  {
    "content": [
      { "type": "text", "text": "Nice! Did you verify if the API endpoints are returning the correct payloads?" }
    ],
    "role": "bot",
    "timestamp": "2025-10-28T11:34:10.507Z"
  },
  {
    "content": [
      { "type": "text", "text": "Yep, I ran the tests locally and everything passed. The response times are much better too." }
    ],
    "role": "assistant",
    "timestamp": "2025-10-28T11:36:22.153Z"
  },
  {
    "content": [
      { "type": "text", "text": "That’s great to hear. Let’s schedule a quick review call later today." }
    ],
    "role": "bot",
    "timestamp": "2025-10-28T11:38:57.768Z"
  }
]

Project-wide Prompt

The prompt you enter here will be sent to Copilot whenever it's interacted with in this specific project.

Permissions

You are able to set Copilot's permissions from here as per the text next to the checkboxes.

How to use Copilot

In projects where Copilot has been activated and fully set up, anyone opening a supported asset will see a new Copilot button on the right sidebar. After clicking on it, the user can talk to Copilot to ask it to:

• Create bounding boxes on the image

• Answer top-level (not nested) classification questions on the asset

• Answer general questions about the task