# Ango Hub Documentation
Welcome to Ango Hub's documentation pages
Welcome to the documentation for Ango Hub, *a quality-first data annotation platform* designed to support the development of reliable and high-performing AI systems. Ango Hub enables teams to efficiently create, manage, and maintain high-quality labeled datasets across a wide range of domains, including *healthcare*, *banking*, *automotive*, *autonomous systems*, and more.
The platform provides powerful tools for annotation, quality assurance, workflow management, automation, and analytics, allowing organizations to scale their data operations while maintaining strict quality standards.
Whether you are a *data annotator* labeling assets, a *reviewer* ensuring annotation quality, a *project manager* overseeing workflows, or a *solution architect* integrating Ango Hub into your data pipelines, this documentation is designed to help you understand and effectively use the platform.
We hope this documentation helps you get the most out of Ango Hub and build better datasets for your AI projects.
## Quick Start
The Quick Start section helps you quickly navigate the most important parts of the Ango Hub documentation. It provides a curated set of links to essential guides and resources so you can easily find the information you need to begin working with the platform.
## Troubleshooting
Need Help?
If you are experiencing issues with the platform, please visit the [FAQs](/faqs) to see if your question is answered. If not, check out our Troubleshooting section at the bottom of the page list on the left for more specific instructions on issues you may encounter.
Contact Information
For any feedback regarding the documentation, things missing, or wrong, please email one of the maintainers of the docs:
* **lorenzo (at) imerit.net** \[Core Concepts & Popular Topics]
* **onur (at) imerit.net** \[Developer Pages]
* **richa.jain (at) imerit.net** \[3D Multi-Sensor Fusion]
# Video Guides
A series of video guides on how to use Ango Hub
***
## Getting Started with Ango Hub for Labelers
An introduction to the platform basics and core annotation workflow.
{% embed url="" %}
***
## Introduction to the Ango Hub Medical Labeling Editor
Overview of the Medical Labeling Editor interface and essential tools.
{% embed url="" %}
***
## Advanced Features of the Ango Hub Medical Labeling Editor
Explore advanced tools and features for efficient medical annotation.
{% embed url="" %}
***
## Using our First-Party Generative AI Plugins
Learn how to use built-in Generative AI plugins to enhance your workflow.
Video Link
***
## How to use the Asset Builder
{% embed url="" %}
***
## How Benchmarks Work
{% embed url="" %}
***
## Overview of Automation in Ango Hub
{% embed url="" %}
***
## Overview of Deep Reasoning Lab
{% embed url="" %}
***
## Introduction to Multi-DICOM Annotation
{% embed url="" %}
***
## Overview of the Workflow Designer
{% embed url="" %}
***
## How to do Model Evaluation on Ango Hub
{% embed url="" %}
***
## Releases
### What's New in Ango Hub 5.8
{% embed url="" %}
# Changelog
Track recent updates, improvements, and bug fixes
### Getting Started
What is Ango Hub and what problems does it solve?
Ango Hub is a data annotation and workflow management platform designed to help teams efficiently label, review, and manage datasets for machine learning. It supports multiple data types and provides tools for annotation, automation, and collaboration.
How do I create my first project in Ango Hub?
To create a project, navigate to the **Projects** page and click **Create Project**. Enter a project name and description, then click **Create**. Once your project is created, you can proceed to configure its settings, upload assets, define your ontology, and set up workflow stages.
For more details, refer to the [Projects](/core-concepts/projects) documentation page.
What types of data can I upload?
Ango Hub supports a wide range of data types. For a complete list of supported asset file types, please refer to the [Supported Asset File Types](/data/data-in-ango-hub/supported-asset-file-types) and the [Labeling Editors](/labeling/labeling-editor-interface) documentation pages.
What annotation tools are available in Ango Hub?
Ango Hub provides a comprehensive set of annotation tools to support a wide range of data types and use cases. For a complete and up-to-date list of supported annotation tools, please refer to the [Labeling Classes](/labeling/labeling-tools) documentation page.
How do I invite team members to my organization?
You can invite users from the **Organization** page by navigating to the **Members** tab. Enter the user’s email address, assign the appropriate role, and send the invitation. For more details, refer to the [Organizations](/core-concepts/organizations#inviting-new-users-to-your-organization-admin-only) documentation page.
What are the different user roles and permissions?
Ango Hub defines two types of user roles: **organization-level roles** and **project-level roles**.
* **Organization-level roles** include **Owner**, **Admin**, and **Member**.
* **Project-level roles** include **Labeler**, **Reviewer**, **Lead**, and **Manager**.
For more details, refer to the [User Roles](/core-concepts/user-roles) documentation page.
### Assets
How do I import assets into an Ango Hub project?
There are multiple ways to import assets into Ango Hub. For detailed instructions, please refer to the [Import Assets](/data/importing-assets) documentation page.
What is the difference between assets, annotations, tasks, and projects?
* **Projects:** The top-level containers where you organize your work. A project brings together your datasets, team members, ontology, and workflow configuration in a single place.
* **Assets:** The raw data files within a project, such as images, videos, documents, or audio files. Assets represent the content that needs to be annotated.
* **Tasks:** The units of work generated from assets. Tasks move through workflow stages (e.g., labeling, review) and are assigned to users for annotation and quality control.
* **Annotations:** The labels or structured information added to assets during a task. Annotations are created using labeling tools (e.g., bounding boxes, polygons, classifications) and represent the ground truth data used for machine learning.
Can I upload and annotate large datasets? Are there any limits?
Yes, Ango Hub is designed to support large-scale datasets. For optimal performance, we recommend using batch uploads or ingesting data via the API/SDK.
For details on platform limits and best practices, please refer to the [Performance & Compatibility Considerations](/labeling/performance-and-compatibility-considerations) documentation page.
How do I handle duplicate or corrupted files?
It is recommended to validate files before uploading.
Ango Hub does not restrict duplicate uploads, meaning the same file (by name or content) can be uploaded multiple times. If deduplication is required, it should be handled prior to ingestion, or duplicate assets should be removed after upload.
If a corrupted file is opened in the labeling editor, the platform will display an error message indicating that the asset cannot be rendered.
Can I directly annotate data I have stored on cloud storage services?
Yes. Ango Hub allows you to connect to supported cloud storage providers and work directly with your data without manual uploads. For step-by-step instructions, please refer to the [Importing Cloud Assets](/data/importing-assets/asset-cloud-import) documentation page.
Do I have to convert PDFs to images before labeling?
No. Ango Hub supports native PDF annotation, allowing you to label PDF files directly without any conversion. For more details please refer to the [PDF Labeling Editor](/labeling/labeling-editor-interface/pdf-labeling-editor) documentation page.
I'm unable to open files from my bucket in the editor and see the error: “Sorry, the data couldn’t be loaded properly.” What should I do?
This is likely to be an issue with how CORS is set up in the bucket where your assets are located.
In order for Hub to display your assets in AWS S3 or GCP, you need to set up CORS following the steps outlined in [Set up CORS](/data/storages/set-up-cors) page. Try to open the assets again after setting up CORS as mentioned in the page.
If you still see the error, please open a ticket through our built-in help system, clicking on the question mark located at the top right of the screen.
### Labeling
How can I view an object's "objectId" when selecting an annotation?
Either right-click on the annotation, click on the three dots, and click on "Copy Object ID", or when in the labeling editor, click on the Quick Settings toggle () and enable *Show Annotation ID on Hover* to see the ID when hovering over the object.
I am not seeing the "Submit" button. Instead, I see a button saying "Un-assign me".
This happens because, even though you have already annotated this task, as a result of the project's workflow, you have been assigned to annotate it again as part of the consensus calculation, or to review your own task, which is not allowed. Please click on "Un-assign me" and the task will be unassigned from you and put back in the queue for other project members to annotate or review.
How do I review someone's labeling task?
To review a labeling task, you need to be assigned a review role within the project. Tasks are routed to review stages based on your workflow configuration.
Once assigned, you can open the task in the labeling editor, evaluate the annotations, and take appropriate actions such as approving, rejecting, or providing feedback for corrections.
For step-by-step instructions and best practices, please refer to the [Reviewing](/core-concepts/reviewing) documentation page.
How can I perform targeted OCR on images
You can perform targeted OCR on images by adding a Bounding Box tool to your project and enabling the Targeted OCR option. This allows you to extract text from specific regions within an image during annotation.
For more details, please refer to the [Bounding Box](/labeling/labeling-tools/tools/bounding-box) labeling tool documentation page.
I see an "Unknown Classification" warning when opening a task.
This means that this classification was removed from the project's category schema after it was answered in the task.
From your project's *Settings -> Category Schema* section, by clicking on the "Show/Hide Version History" button, you may browse previous versions of your project's category schema to attempt to restore it.
I'm experiencing poor performance of the tool. Where can I know more?
Please refer to the [Performance & Compatibility Considerations](/labeling/performance-and-compatibility-considerations) documentation page.
### Model Integration
How can I import existing annotations (pre-labels) to Ango Hub?
You can import existing annotations into Ango Hub either through the UI or the SDK, depending on your workflow. In the UI, annotations can be uploaded in a supported JSON format during data ingestion or via the import interface, while for programmatic and large-scale operations, you can use the SDK’s [`import_labels`](/sdk/sdk-documentation/project-level-sdk-functions/import_labels) function. For detailed instructions, supported formats, and configuration options, please refer to the [Importing Annotations](/data/importing-and-exporting-annotations/importing-annotations) documentation page.
Can I use my own ML model with Ango Hub?
Yes, you can use your own ML model with Ango Hub in multiple ways depending on your needs. You can run your model externally and import the generated annotations into Ango Hub via the UI or SDK's [`import_labels`](/sdk/sdk-documentation/project-level-sdk-functions/import_labels) function, or you can integrate your model directly into the platform using a [Model Plugin](/plugins/plugin-developer-documentation/model-plugins) to enable live inference within your workflows.
How does pre-labeling work in Ango Hub?
Pre-labeling in Ango Hub enables you to automatically generate initial annotations using model predictions, reducing manual labeling effort and accelerating your workflow.
These auto-generated labels serve as a starting point for annotators, who can review, evaluate, and correct them as needed, ensuring high-quality final outputs while significantly improving efficiency and consistency across your dataset.
How do I perform model evaluation on Ango Hub?
There are currently a few ways to evaluate the outputs of your models. Please refer to this video for a visual explanation on how to accomplish this on Ango Hub:
{% embed url="" %}
If your network does not allow access to the above video, you may download it by clicking on the following link:
{% file src="/files/U17UR2pCEaJQ2eupc9TS" %}
### Plugins
What are plugins in Ango Hub?
The plugin infrastructure allows developers to extend Ango Hub’s core capabilities with custom functionality. Plugins offer you to add new export formats, upload assets, integrate machine learning models, and much more seamlessly and efficiently. For more details, please refer to the [Plugin](/plugins/introduction-to-plugins) documentation.
How do I install or use a plugin?
You can install plugins directly from the Plugin Directory within your organization. Once added, they become available for use in your workflows and projects. For more details, please refer to [How to Add Plugins to Your Organization](/plugins/introduction-to-plugins/how-to-add-plugins-to-your-organization) and [How to Use Plugins in Your Project](/plugins/introduction-to-plugins/how-to-use-plugins-in-your-project) documentation pages.
Can I build custom plugins?
Yes. Developers can create custom plugins using the Ango Hub SDK to extend platform functionality, such as building custom file explorers, model integrations, and export workflows. For more details, please refer to the [Plugin Developer Documentation](/plugins/plugin-developer-documentation) page.
What are some common use cases for plugins?
Plugins extend Ango Hub's functionality and support a wide range of workflows. Common use cases include file import and data ingestion, model integrations for pre-labeling or inference, and exporting data in custom formats.
Ango Hub also provides a set of plugins that are officially supported and maintained by the Ango team. For more details, please refer to the [First-Party Ango Plugins](/plugins/first-party-ango-plugins) documentation page.
If you have specific requirements, you can build your own plugins. For more details, please refer to the [Plugin Developer Documentation](/plugins/plugin-developer-documentation) page.
How can I monitor the status of a plugin run?
You can track the progress and status of your plugin runs directly from the Plugin Sessions window. This interface provides real-time visibility into each run, including its current state, logs, and any errors encountered. For more details, please refer to the [Monitoring Plugin Progress](/plugins/introduction-to-plugins/monitoring-plugin-progress) page.
I'm uploading assets with the File Explorer plugin, but I don't want to upload an asset if it's already been uploaded before. How can I do that?
You can avoid uploading duplicate assets by enabling the prevent\_duplicates parameter in the File Explorer plugin’s configuration JSON. When set to true, the plugin checks for existing assets and skips any duplicates during upload.
How do I know whether a plugin is live or not?
In the plugin list within your project, each plugin has a status indicator next to the “Live” label. A green icon means the plugin is live and active, while a red icon indicates that it is not currently live.
Who should I contact if a plugin goes down?
In the Plugin Directory and in the plugin’s Run Dialog, you can see the email of the person who created and maintains the plugin. It's in the bottom-right corner of the plugin's icon. You may contact them.
### Export
What export formats are supported in Ango Hub?
Ango Hub provides a flexible export system to support a wide range of use cases. By default, annotations can be exported in [Ango Hub's native JSON format](/data/importing-and-exporting-annotations/exporting-annotations/ango-export-format).
In addition, built-in export plugins allow you to convert this format into commonly used standards such as [COCO, YOLO, and KITTI](/plugins/first-party-ango-plugins/ango-export-converter-plugins).
For segmentation tasks, the [Ango to Mask](/plugins/first-party-ango-plugins/ango-to-mask-converter) plugin enables exporting annotations as mask images for both image and video data.
For classification projects, the [CSV Export Classification](/plugins/first-party-ango-plugins/csv-export-for-classification) plugin allows you to export results in a tabular CSV format.
If you have custom export requirements, you can use the SDK's [`export`](/sdk/sdk-documentation/project-level-sdk-functions/export) function, develop your own [Export Plugin](/plugins/plugin-developer-documentation/export-plugins), or reach out to the team for further support.
How do I export annotations for a specific batch?
You can export annotations for a specific batch. In the UI (including built-in export and export plugins), you can simply select the desired batch through the available filters before running the export.
If you are using the SDK, you can specify the target batch by providing the appropriate batch ID within the options parameter of the [`export`](/sdk/sdk-documentation/project-level-sdk-functions/export) function. This allows you to precisely control which batch data is included in the export.
Why is my export empty or missing data?
An empty or incomplete export is typically caused by incorrect batch and stage filters. When exporting via the UI, make sure the selected batch and stage actually contain assets with annotations.
If you are using the SDK, ensure that you are passing the correct identifiers, such as batch names or IDs and stage names or IDs, and that they match your project data.
### Workflow & Project Management
What happens if I change the ontology of a project while people are annotating tasks?
They will receive the new ontology after they have submitted the task they are currently working on. Please refer to the [Managing the Project Ontology](/labeling/managing-the-project-ontology) documentation page.
What happens when a task is sent back to the same stage?
It's assigned to the same user who originally labeled/reviewed it in that stage, and marked for labeling/reviewing again.
How to mute your notifications?
You can mute your notifications so that they don't pop up on your screen anymore. You will still receive notifications, the only difference is that to see them, you'll have to click on the bell icon.
When you mute notifications, you mute them for yourself only, not for the entire organization.
1. Click on the 'bell' icon on the top right corner to open the notification panel.
2. Enable the *Mute* toggle. Notifications will not pop up anymore, and will instead be delivered silently to your notification panel.
How to open an asset provided the Asset ID?
If you have an assetId (which you can find in the *Task Info* drawer on the right-hand side of the labeling editor, or from the SDK) and you wish to open that asset in the labeling editor, navigate to the following URL, replacing {assetId} with the asset ID.
How to transfer project ontologies between projects?
Ango Hub allows you to copy the project ontology from one project and paste it in another, by following the below steps:
1. Navigate to the project containing the ontology you wish to copy.
2. Navigate to the *Settings* tab, then the *Category Schema* section.
3. Click on the "code" button (1) on the right. A panel showing the project's ontology JSON will appear. Copy the JSON to your clipboard using the "copy" button (2).
4. Navigate to the project where you'd like to paste the ontology and follow the same steps, but instead of copying, paste the JSON you have just copied into the box.
5. Click on *Save* at the bottom of the page.
How to transfer project workflows between projects?
Ango Hub allows you to copy the project workflow from one project and paste it in another, by following the below steps:
1. Navigate to the project containing the workflow you wish to copy.
2. Navigate to the *Workflow* tab.
3. Click on the "code" button (1) on the right.
4. A panel showing the project's workflow JSON will appear. Copy the JSON to your clipboard using the "copy" button.
5. Navigate to the project where you'd like to paste the workflow and follow the same steps, but instead of copying, paste the JSON you have just copied into the box.
6. Click on *Save* to save the new workflow.
I get a "0 Tasks Labeled" alert when trying to pre-label tasks.
Most likely, you are attempting to pre-label tasks which are not in the [*Start* stage](/core-concepts/workflow#start). Only tasks in the *Start* stage can be pre-labeled.
By default, tasks skip the *Start* stage and are automatically forwarded to the stage connected to the output of *Start*. This is visible in the *Workflow* tab, when a dotted line runs from the output of Start to the input of another stage:
It is for this reason that by default, pre-labeling is disabled.
**How to Activate Pre-Labeling and Pre-Label Tasks**
1. Go to the *Workflow* tab, then click on the *Start* stage and disable the *Auto Forward* toggle. This will ensure tasks remain in the *Start* stage when created. The dashed line will turn to a fully filled-in line.
2. [Import or upload your assets.](/data/importing-assets)
3. [Import your pre-labels](/data/importing-and-exporting-annotations/importing-annotations), ensuring the tasks you are trying to pre-label are currently in the *Start* stage.
4. When you are finished pre-labeling tasks, go to the Workflow tab again and click on the *Start* stage. Then, click on *Next Stage*. This will push all tasks in Start to the stage following it, sending them into the workflow.
### API & SDK
How do I get my API key?
1. Click on your profile circle on the top-right corner and click on "Account". Alternatively, click [here](https://imerit.ango.ai/account/api).
2. Enter the "API" tab.
3. If an API key is not present, click on the "Create Key" button to the right.
4. Once an API key is present, click on the *Copy to Clipboard* button next to it to copy it to your clipboard. You may now paste it where necessary.
{% hint style="warning" %}
Sharing your API key is the same as sharing your password.
Do not share your API key with anyone, unless you would also share your password with them.
{% endhint %}
How do I get an organization ID?
Each organization is assigned a unique ID. Here's how you can get it from the UI:
1. Navigate to your [Account](https://hub.ango.ai/account) page.
2. You'll see a row called *Organization ID*. Click on the "Clipboard" button next to it to copy the ID to your clipboard.
How do I get a project ID?
To retrieve a Project ID from the UI:
1. Navigate to [Projects](https://imerit.ango.ai/projects) page.
2. Search for your project and click to open it.
3. On the project page, click the copy icon next to the project title to copy the ID to your clipboard.
Alternatively, you can obtain the Project ID from the URL. Project URLs follow this format:
`https://imerit.ango.ai/projects/`
Simply copy the \ portion from the URL.
### Security
How is my data protected?
All data is encrypted in transit and at rest across every deployment of Ango Hub, using current industry standards.
Do we need to move our data into your platform?
No. Ango Hub connects to your storage with scoped, revocable permissions and reads only what you authorize. Access is logged.
Can we run Ango Hub in our own environment?
Yes. Deploy on premises, in your private cloud, or in an air gapped setup for full isolation.
Who can access our projects?
You control access with granular role based permissions and single sign on. We integrate with providers like Okta and follow least privilege by default.
What compliance standards do you meet?
Ango Hub has completed a SOC 2 examination, maintains GDPR compliance, and uses third party auditors to verify security, privacy, and HIPAA controls.
# Assets
Overview of assets as they are used on Ango Hub
An asset is a piece of data to be annotated. Usually, an asset will be in the form of a file, for example, a .jpg image, a .mp3 audio, or a .txt text.
During labeling, Ango Hub will show annotators one asset at a time. The labeler will then annotate the asset using the labeling tools determined in the label set.
Ango Hub supports different categories of assets. [See all the supported file types](/data/data-in-ango-hub/supported-asset-file-types).
[Check out how to import assets into Ango Hub](/data/importing-assets).
## Asset Attributes
### Path
Each asset has a file path associated with it.
To see an asset’s file path, in your project, enter the *Assets* tab, then hover with your cursor over an asset preview. The asset’s absolute path will be shown.
{% hint style="info" %}
Thumbnail previews are only shown for videos, and for images under 10MB in size.
{% endhint %}
Clicking on the “Copy” symbol at the beginning of the path will copy the path to the clipboard.
### External ID
Each asset is given a non-unique External ID.
This ID is used throughout the Ango Hub interface to identify the asset.
When [importing assets from the browser](/data/importing-assets/asset-browser-import), assets are given an External ID equal to their filenames, including the extension. When uploading assets using the [Cloud Import](/data/importing-assets/asset-cloud-import) functionality, External IDs are determined by the uploader in the JSON file.
Hovering the mouse cursor over an externalId will show the full external ID of the asset. Clicking on the “Copy” symbol at the beginning of the ID will copy the ID to the clipboard.
### Created At
The *Created At* attribute shows the date and time when the asset was imported into Ango Hub.
### Tasks
Shown in the *Tasks* column is the task information for each asset.
If a task has been assigned to a user in the stage it is currently in, the letters in the circle are the initials of the user assigned to the task. The small symbol to its top right gives a hint to the task's position in the project's workflow.
A *green checkmark* indicates that the task is in the *Completed* stage.
A *blue empty label* indicates that the task is in a *Labeling* stage waiting to be annotated.
A *yellow review label* indicates that the task is in a *Reviewing* stage waiting to be reviewed.
A *blue filled label* at the top of the circle indicates that the labels in the task have been imported externally into Ango Hub. [More on importing annotations here.](/data/importing-and-exporting-annotations/importing-annotations)
# Attachments
Attachments are media files like text, images, and videos that can be shown to labelers while annotating certain assets.
On Ango Hub, you can upload text, images, videos, or HTML to be shown to labelers next to specific assets during labeling.
An attachment (right) shown next to an asset being labeled.
### Properties of Attachments
* Text, image, video files or arbitrary HTML shown next to specific assets
* Viewable by labelers during labeling
* Unique to each asset, that is, you can show data that belongs to the asset being labeled right next to the asset itself
* Uploadable in bulk with a simple JSON structure
* An asset can have an unlimited number of attachments
### How to View an Attachment
From the labeling editor, navigate to an asset with an attachment.
The *Attachment* tab on the right side of the screen shows the number of attachments available for the current asset. Click it to open the attachment drawer. (1)
The attachment drawer will open. If more than one attachment was uploaded for the asset in question, each attachment is shown in its own tab. (2)
### How to Upload Attachments
From the *Assets* tab in your project, click on *Import.* A dialogue will pop up.
Enter the *Import Attachment* tab.
From here, you'll upload a JSON containing your attachments. Here's what a sample JSON looks like for image, text, or video attachments:
```json
[
{
"externalId": "image-sample.jpg",
"attachments": [
{
"type": "IMAGE",
"label": "image-attachment",
"value": "https://sample-image.jpg"
},
{
"type": "TEXT",
"label": "text-attachment",
"value": "Some sample text."
}
]
},
{
"externalId": "image-sample.jpg",
"attachments": [
{
"type": "VIDEO",
"label": "video-attachment",
"value": "http://sample-video.mp4"
}
]
}
]
```
Here is a sample JSON with an HTML attachment:
```json
[
{
"externalId": "external-id.png",
"attachments": [
{
"type": "TEXT",
"value": " Images in a Row
Header Placeholder Text
"
}
]
}
]
```
{% hint style="info" %}
Only text, .jpg images, and .mp4 videos are supported. Videos must be encoded in such a way that the browser can play them. [See here for a list of codecs supported by browsers](/data/data-in-ango-hub/supported-asset-file-types).
{% endhint %}
{% hint style="info" %}
While PDF files are not directly supported as attachments, you may still display them in attachments by embedding them into an iframe and selecting the type of attachment as "TEXT".
For example, the following will display a PDF attachment:
{% endhint %}
```json
[
{
"data": "https://i.imgur.com/QjQcnU6.jpeg",
"externalId": "demo-asset",
"attachments": [
{
"type": "TEXT",
"value": ""
}
]
}
]
```
{% hint style="warning" %}
URLs may not contain spaces or "+" signs anywhere.
{% endhint %}
If attachment upload is successful, a notification will pop up on the screen notifying you of how many assets were given an attachment.
### Uploading Attachments from Private Buckets
If you have created an integration between Ango Hub and your private storage, either on AWS S3 or on GCP, you can upload attachments that link to files in your private bucket. The file will never be copied or moved anywhere, and will only be shown to labelers when they open the *Attachments* drawer.
First, create an integration from the *Storages* tab of your [Organization page](https://imerit.ango.ai/organization). [You can read how to do so here](/data/storages/importing-private-cloud-assets-aws).
Then, from the same page, copy the storage ID of your newly created storage, by clicking on the "Copy" button next to the ID:
Prepare a JSON in the same format as the one prepared [in the previous section](#how-to-upload-attachments), but with the addition of storage IDs in the URLs. For example:
```json
[
{
"externalId": "image-sample.jpg",
"attachments": [
{
"type": "IMAGE",
"label": "image-attachment",
"value": "https://sample-image.jpg?storageId=1234567890"
},
{
"type": "TEXT",
"label": "text-attachment",
"value": "Some sample text."
}
]
},
{
"externalId": "image-sample.jpg",
"attachments": [
{
"type": "VIDEO",
"label": "video-attachment",
"value": "http://sample-video.mp4?storageId=1234567890"
}
]
}
]
```
## Importing Attachments during Asset Import
Hub allows you to import attachments together with assets, at the same time.
To do so, you will have to use the *Cloud Import JSON* method for importing assets, as explained below:
1. In your project, go to the *Assets* tab and click on the *Add Data* button.
2. From the dialog that appears, enter the *Upload Data URL* tab.
3. Prepare a JSON formatted like the following. It is an array of objects, with each object representing an asset. Each object, then, in its `attachments` property, will have a list of attachments, like so:
```json
[
{
"externalId":"my_external_id",
"data":"https://url-to.asset/video.mp4",
"attachments":[
{"type":"IMAGE","value":"https://angohub-public-assets.s3.eu-central-1.amazonaws.com/uploaded-data-6cbc3c56-58f9-4430-990d-863bd5a1a755.jpg"},
{"type":"TEXT","value":"This is a text attachment."}
]
}
]
```
{% hint style="warning" %}
In AWS S3, if your URL does not contain region information, your attachments may not be visible. When using S3, please ensure the region information is contained in the URL right after the bucket name, like so:
```json
https://bucket-name.s3.eu-central-1.amazonaws.com/filename.JPG?storageId=111bb111389ff80015f2b914
```
{% endhint %}
4. Drag and drop the JSON you've just created on the *Upload Data URL* file box:
5. Your assets with attachments will appear in the *Assets* tab.
# Batches
Batches are a way to organize assets in categories.
Each asset can have multiple batches attached to them. Batches are optional to use, and by default, imported assets have no batch.
You can attach batches to assets to quickly find them later, to assign them all to a certain labeler, to filter performance reports by batch, and much more.
## How To Assign Batches to Assets on Import
### In bulk (the same batch for all assets being imported)
When [importing assets](/data/importing-assets) to Hub, be it [from your local machine](/data/importing-assets/asset-browser-import) or from [public](/data/importing-assets/asset-cloud-import)/[private cloud assets](/data/storages/importing-private-cloud-assets-gcp), you may attach a batch directly during import.
To do so, after clicking on *Add Data* in the *Assets* tab, click on the *Batch Name* dropdown and add the batches you wish to attach.
If you have existing batches, they will show up here. You may also create new batches from this menu.
### Individually (pick which batch each asset will be assigned to)
When importing assets via the Cloud Import method, with either [public](/data/importing-assets/asset-cloud-import) or [private](/data/storages/importing-private-cloud-assets-aws) assets, you may indicate in the JSON you upload the batch(es) you'd like each asset to be assigned to. Please refer to the section *Preparing the JSON* in the docs page on
{% hint style="info" %}
You may also assign batches during upload when using our SDK. See [upload\_files\_cloud](/sdk/sdk-documentation#upload_files_cloud-project_id-assets-integration_id-batches) for more information.
{% endhint %}
### How to Assign Batches After Import
From the *Assets* tab, toggle the checkmark next to the assets you wish to batch together (highlighted in green on the screenshot below). It may help to use the filters at the top of the list to narrow down your search.
You may click on the checkmark highlighted in red on the screenshot above to select all tasks in the current page.
{% hint style="warning" %}
Navigating to a different page of assets *will deselect* the checkmarks you have toggled earlier. Your checkboxes are not kept in memory when navigating between asset pages.
The maximum number of assets you may batch this way at any one time is 1000. This is accomplished by selecting *1000/page* from the bottom right corner of the page, then clicking on the *Select All* checkmark highlighted in red on the screenshot above.
{% endhint %}
From the *Item Actions* dropdown, click on *Set Batches*.
From the dialog that appears, you will be able to select the batches to add to the selected assets from a list of existing project batches.
{% hint style="info" %}
You may also assign batches after upload when using our SDK. See [assign\_batches](/sdk/sdk-documentation#assign_batches-project_id-asset_ids-batches) for more information.
{% endhint %}
### Creating and Editing Batches
To create and edit existing batches, navigate to the *Settings* tab and then to the *Batches* section.
* To add a new batch, click on *Add new batch*.
* To rename an existing batch, click on the icon next to the name of the batch you wish to rename.
* To delete a batch, click on *Delete* on the row of the batch to delete.
{% hint style="info" %}
Don't forget to click on *Save* or your changes will be lost when navigating away from the page.
{% endhint %}
{% hint style="info" %}
You may also create batches after upload by using our SDK. See [create\_batch](/sdk/sdk-documentation#create_batch-project_id-batch_name) for more information.
{% endhint %}
# Benchmarks
Set certain labeling tasks as the gold standard and measure annotator performance.
Ango Hub allows project managers to mark certain labeling tasks as benchmark (also known as 'Test Question' or 'Gold Standard' in other environments). Benchmarks allow the project manager to measure the performance of annotators.
Here's a video overview of Benchmarks:
{% embed url="" %}
{% hint style="warning" %}
Benchmark scoring is not supported for video assets. Video benchmark tasks can be submitted, but their benchmark score will be 0.
{% endhint %}
## Enabling benchmarking for your project
Benchmarking is not enabled by default in new projects.
To enable benchmarking, navigate to your project's *Settings* page, then to the *General* section. Enable the toggle next to *Benchmark.*
Click on the *Save* button at the bottom of the page.
From the same menu, you may also choose the likelihood of annotators being shown a benchmark task whenever they are shown a new task from the queue. By default, it is 10%, meaning that each time an annotator clicks on "Submit" and a new task is shown to them, there is a 10% chance that task is a benchmark (if any benchmarks are left to annotate for that user).
{% hint style="danger" %}
Disabling benchmarking in a project where benchmark tests have taken place **will delete all benchmarking information** from the project completed so far.
It is strongly recommended not to disable benchmarking in projects after it has been enabled.
If you wish to pause benchmarking on your project, you may set the likelihood annotators are shown a benchmark to 0% in the project settings. This will cause benchmark tasks not to appear to annotators.
{% endhint %}
## Setting and removing benchmark tasks
Hub allows you to mark existing tasks as benchmarks. The task must have already been created and existing in the project. You may not mark certain tasks as benchmark during asset upload – assets must first be uploaded – only then they can be marked.
Tasks may be marked as benchmark one at a time (single) or in bulk.
### Single
Navigate to and open the task you would like to set as benchmark. For example, you may click on the task from the *Assets* or the *Tasks* tab.
Once you have opened the task, from the three-dot menu at the top right of the labeling editor, click on *Set as Benchmark.*
To unset a task as benchmark, and turn it back to a normal task, follow the same steps, then click *Remove as Benchmark*. You may need to refresh the page if the benchmark status was changed recently.
The *Set as Benchmark* dialog will appear. Click on *Set as Benchmark* to finalize setting the task as benchmark. The task will be shown to all annotators in all labeling stages in the project.
Ango Hub uses the annotations saved on the source task as its benchmark answers. For classifications, only the answer saved on the source task is accepted as the benchmark answer; you cannot configure additional alternative answers.
### In bulk
From the *Tasks* tab of your project, select one or multiple tasks using the checkboxes to their left. Then, from the *Items* menu, click on *Set as Benchmark*.
To unset tasks as benchmark, and turn them back to normal tasks, follow the same steps, then click *Remove as Benchmark*.
The *Set as Benchmark* dialog will appear. Click on *Set as Benchmark* to finalize setting the task as benchmark. The task will be shown to all annotators in all labeling stages in the project.
## How to see a task's benchmark answers
From the *Tasks* tab, navigate to the task you'd like to examine the benchmark answers of, and open it.
In the labeling editor, a dropdown will appear at the top, allowing you to flip between answers given by different users to this task:
When you are done checking out a user's answer, click on *Cancel* on the top right to return to the task.
## Allowing a user to retake a benchmark
From the Tasks tab, navigate to the benchmark you'd like to reset and click on the "Reset Benchmark" button on the right.
## What happens when you set task(s) as benchmark
In this dialog, you click on *Set as Benchmark* to finalize your benchmark selection.
What this means in practice is that:
1. The tasks you have selected will be marked as benchmarks.
2. The tasks you have selected will be moved to the [*Complete* stage](/core-concepts/workflow/complete). This is because since you have marked the task(s) as the gold standard, they are assumed to be complete.
{% hint style="info" %}
Benchmark tasks may be re-queued to other stages from *Complete*. What this means, however, is that an annotator will annotate it again, or a reviewer may alter it, changing the benchmark for users who have not yet been tested on the benchmark. This is not recommended.
We strongly recommend tasks marked as benchmark not be re-queued from Complete to other stages where they can be edited.
{% endhint %}
3. Hub will make copies of the task(s) you have selected, one for each user in the project, and place them in every user's labeling queues in all label-type stages in the project. The tasks created this way are known as "Benchmark tasks". Benchmark tasks are not included in the final export, are not sent to *Complete*, and are only be shown to users in the stage you have selected in this dialog. They are archived afterwards.\
\
All users who annotate in label-type stages will be shown the benchmark tasks. To limit who can see the benchmark tasks, you must limit who can annotate or review in all label-type stages in your project.
4. Tasks selected as benchmarks will be visually distinguished from other tasks, to the project manager only, by the presence of a small yellow crown in their row, both in the *Assets* and *Tasks* tab:
## How benchmark tasks are shown to users
### Look and Feel
Users will not be able to tell that they are annotating a benchmark task. The task will look and feel exactly like any other task, with no indication whatsoever that the task they are annotating will be utilized in their performance evaluation.
Users will be able to annotate, create issues, skip, save, view instructions, and perform any other action they can normally perform on normal labeling tasks.
The only difference is that completing a benchmark task will not increase the number of "Completed" tasks, as benchmark tasks do not appear in the final export, are not sent to *Complete*, and are only used once in the stage where they have been created to measure the annotator's (or reviewer's) performance.
### Benchmark Frequency
For as long as there are benchmark tasks in the stage, for the user annotating, whenever the user clicks on "Submit", the next task that is shown to them has a, by default, 10% chance of being a benchmark task.
By default, if only benchmark tasks are remaining in the user's queue, the user will be exclusively shown benchmark tasks. However, if you'd like for Ango Hub to stop showing benchmark tasks when the user has completed all of their "regular" tasks, you may enable this toggle:
The frequency at which benchmark tasks are shown can be changed from the project Settings -> General section, under the benchmark toggle:
By default, there is a 10% chance that, whenever an annotator submits a task, the next task they will receive will be a benchmark (if there are any left for the user to annotate).
Setting this to 0% will cause benchmark tasks to stop showing to users, and setting this to 100% will cause only benchmark tasks to be shown to users until benchmark tasks are done. Users would then be shown normal labeling tasks.
## Analyzing annotator performance
As project manager, you can see the performance of each user, as well as the performance of each benchmark question.
To see each user's performance, enter the *Performance* tab. Each user's average benchmark score will be shown on the user's row:
To see the performance of each benchmark question, from the *Tasks* tab, filter by *Benchmark*. You will then only see tasks which have been set as benchmark.
Click on the "+" icon next to a benchmark task to see each user's answers and score as it relates to that benchmark question:
From each benchmark's row, you may see the average benchmark score for that question, as well as the number of annotators who have submitted an answer to that benchmark:
You may also download a JSON containing all information on all tasks used to benchmark users from *Settings* -> *General -> Export Benchmark Tasks.*
## Properties of benchmark tasks
Benchmark tasks shown to users:
* Do not get sent to Complete
* Do not contribute to completion statistics (e.g. the "Tasks Completed" number will not go up as benchmark tasks are completed)
* Do contribute to all other statistics (TPT, etc.)
* Are immediately archived after being submitted (e.g. they are available for project managers to inspect, but they are not present in any stage.
## FAQ
### What is the algorithm used to calculate the benchmark score?
#### Classifications
Let `questionCount` be the total number of classification questions in the project, and `taskCount` the total number of tasks assigned to an asset.
We calculate `x`, the single-question score for a single task as `(sameAnswers / (taskCount - 1))`, where `sameAnswers` is the count of answers that are equal to one another, current one excluded.
We repeat the above calculation for all tasks in the asset, to calculate the final result represented as Σ(x) below.
We calculate `y`, the overall score on a single question (classification) as `(∑(x) / taskCount)`.
We repeat the above calculation for all questions in the asset, to get to the final result represented as Σ(y) below.
The final score, then, is calculated as `∑(y) / questionCount`.
{% hint style="info" %}
**Note on Rank Benchmarking**
In the [Rank](/labeling/labeling-tools/classification-tools/rank) classification tool, if the annotator's answers differ, in any way, with the benchmark, their score for that classification will be 0. If they are the exact same, it wil be 1 (e.g. 100%) for that classification.
{% endhint %}
#### Objects (Bounding Box, Polygon...)
We calculate benchmarks for objects using the Intersection over Union (IoU) method.
We compare objects with one another to generate their IoU scores. If some annotations are completely separate, for example, with not even a pixel in common, their IoU score would be 0. If they overlapped completely, their score would be 100.
We then average the IoU scores of all annotation to calculate the final score.
#### Points
See [Consensus/Points](/core-concepts/workflow/consensus#points).
### Can I edit a benchmark task after it has been set?
Yes. Open the task from the *Tasks* or the *Assets* tab, edit it, and save it.
Existing benchmark scores will not be changed. Users who have not yet been benchmarked on the task will see the new, updated task, and they will be tested on this new version of the task.
### What happens to a user's benchmark score after I have edited a benchmark?
The benchmark score remains unchanged. Once a user is tested on a benchmark task, their score regarding that task is unchangeable. Users who have not yet seen the benchmark task, however, will be tested on the edited version of the task.
### What tools and classification types are included in benchmark calculations?
Tool / Classification Type
Benchmark Support
Notes
Tools
Bounding Box
✅
Calculated with IoU.
Polygon
✅
Calculated with IoU.
Entity
✅
Calculated from overlapping text spans.
Point
✅
Calculated from the distance between points.
Rotated Bounding Box
❌
Polyline
❌
Segmentation
❌
Brush
❌
Voxel Brush
❌
Circle
❌
PDF
❌
Message
❌
Angle
❌
Classifications
Radio
✅
Checkbox
✅
Single-Select Dropdown
✅
Single-Select Tree
✅
Text
✅
Consensus is 0% if the texts differ, even by a single character, and 100% if they are exactly the same.
Slider
✅
Multi-Select Dropdown
❌
Multiple classifications are not available in the Threshold tab.
Multi-Select Tree
❌
Multiple classifications are not available in the Threshold tab.
Frame-Specific Classifications
❌
Frame-specific classifications are not available in the Threshold tab.
Relations
Single Relation
❌
Group Relation
❌
{% hint style="info" %}
The built-in benchmark score calculation supports a limited set of annotation tools and modalities. If your project requires broader tool or modality support, you can use the **Benchmark & Consensus Export** plugin to generate a comprehensive benchmark report. For setup instructions and supported tools, see [the Benchmark & Consensus Export documentation](/plugins/first-party-ango-plugins/benchmark-and-consensus-export).
{% endhint %}
# Category Schema (Ontologies)
Setting up your labeling project's ontology on Ango Hub
A project’s category schema (also known as *ontology*) is the list of labeling tools available in the project.
Ango Hub allows project owners to set the project’s ontology directly from the *Category Schema* section of the *Settings* page.
The ontology created from this section is then reflected in the left sidebar of the labeling editor:
## How to create a project ontology
From the project’s *Settings* tab, enter the *Label Set* section.
Click on the *Add Category* button and choose the labeling tool to add. Each tool has a slightly different setup process. [See here for a list of all labeling tools available in Ango Hub, and how to add them to your project.](/labeling/labeling-tools)
For a guide on how to create nested classifications, [see here](/labeling/labeling-tools/tools/nested-classifications).
## How to copy a project's ontology to a new project
Please [check this guide](/how-to/transfer-project-ontologies-between-projects).
## Browse and Restore Previous Ontology Versions
From the Settings tab, in the *Category Schema* section, click on the "History" icon on the top-right to open a history of all workflow versions in the current project.
{% hint style="info" %}
Category Schema history started being recorded on March 12, 2025. Versions before then had not been saved.
{% endhint %}
To view a previous version of this project's ontology, click on *Preview*. To restore it, click on *Save* while the version you need is previewed.
# Frame Interpolation
When annotating videos, Hub can use keyframes to carry annotation information across multiple frames. This lets annotators label only the frames where something changes, instead of manually editing the same label on every frame.
Frame interpolation is available for all video file types.
## Keyframes
Whenever you create or edit an object or frame-specific classification on a video frame, you are creating a keyframe for that label.
For example, if I create a new bounding box on frame 1, I will have created a keyframe for it on frame 1. Keyframes are marked with white diamonds on the video timeline:
If I navigated to frame 20 and edited the position of the bounding box, I will have created a new keyframe for it:
The position of the bounding box will be interpolated between frames 1 and 20. You can add, remove, and edit keyframes using the video labeling editor interface.
## Object Interpolation
For object tools, interpolation means that Hub calculates the object's shape or position on frames between keyframes.
For example:
1. You draw a bounding box around a car on frame 1.
2. You move to frame 20 and move the same bounding box to the car's new position.
3. Hub calculates the bounding box positions for frames 2 through 19.
{% hint style="info" %}
Interpolation is currently only available for the Bounding Box, Polygon, Segmentation, and Point labeling tools.
{% endhint %}
## Frame-Specific Classifications
Classifications can also be frame-specific. In the project ontology, enable the *Frame-specific* setting on a classification when answers should apply to individual video frames instead of the whole asset.
Frame-specific classifications do not interpolate geometry. Instead, the answer from the latest previous keyframe applies until another keyframe changes the answer, or until the classification's segment ends.
For example, if a frame-specific radio classification is set to *Green* on frame 5, *Red* on frame 15, and *Blue* on frame 40:
* frames 5 through 14 use *Green*
* frames 15 through 39 use *Red*
* frame 40 and later frames in the classification segment use *Blue*
## Import Format Counterparts
When importing video annotations, the UI concepts above map to these Ango Import Format fields:
UI concept
Import field
What it controls
Timeline row for one tracked object or frame-specific classification
objectId
Groups keyframes that belong to the same object or classification instance.
Start and end of the timeline row
segments
The frame range where the object or frame-specific classification exists.
Frames where the object or answer changes
keyFrames
Frame-number keys containing the object geometry or classification answer for that frame.
Hidden / out-of-view spans
outOfViewSegments
Frame ranges where the object or frame-specific classification should not be visible.
A single page in multi-image or DICOM assets
page
The page or frame where a non-interpolated annotation belongs.
Video keyframe numbers are frame indexes. They start from 0.
{% hint style="info" %}
For complete JSON examples, see the video object and video classification sections in the [Ango Import Format](/data/importing-and-exporting-annotations/importing-annotations/ango-import-format#video-and-multi-image-labeling) page.
{% endhint %}
# Geofencing
[Organization](/core-concepts/organizations) administrators can choose to limit access to their organization to certain IP address ranges and zones through a feature we call *Geofencing*.
## Geofencing your organization
From the *Organization* page, navigate to the Geofencing tab, then enable the toggle that appears. You will see the following:
Add the IP ranges and zones which will be allowed to access your organization, then click on *Apply IP and Zone Restrictions* to enable geofencing.
{% hint style="danger" %}
By applying the wrong geofencing settings, you may permanently lock yourself, and everyone else, out of your organization.
Before applying geofencing settings, please ensure that your IP, or your zone, is included in the IP or zones list.
{% endhint %}
{% hint style="info" %}
Zones and IP ranges are cumulative, meaning that if you select an IP range and a country, people from the IP range *and* the country will be able to access your organization.
{% endhint %}
To disable geofencing, enter the *Geofencing* section of the *Organization* page, and turn off the toggle.
## Verifying whether your organization is geofenced
Next to your organization's name in the organization switcher, you will be able to see whether or not your organization is geofenced.
If the "world" icon is grey and without a padlock, your organization is not geofenced:
If the icon is green and showing a padlock, your organization is geofenced. Hovering over the icon will provide details about the geofencing settings:
# Idle Time Detection & Time Tracking
Ango Hub keeps track of the time annotators and reviewers spend on their tasks.
In this page, we explain in detail how time tracking works in Ango Hub, and how you can set up Idle Time Detection in your projects.
## Terminology
### Tabs and Windows
### Time Categories
There are two types of time shown in Ango Hub: *active* and *idle*.
| Type | Description |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Active | The tab is active, and the user is using the mouse or keyboard in the tab area (viewport.) The window can be active or inactive. |
| Idle |
The window is active, the tab is active, the user is not using either the mouse or the keyboard.
OR
The window is inactive, the tab is active, the user is using the mouse but the cursor never hovers over the tab.
OR
The tab is inactive.
|
{% hint style="info" %}
Exports and performance reports show active time. When idle durations are enabled for an export, idle time is included as well.
{% endhint %}
Time during which the browser tab is inactive counts as idle time.
## How Time is Tracked
In Ango Hub, time starts being tracked when a user opens a task. When the user clicks *Save* , the time spent so far is saved on the task, and the timer starts again. When the user clicks *Submit*, the time saved on the task is added to its total duration for the completed stage.
{% hint style="info" %}
This means that if you open a task, then quit it without saving or submitting, this time will not be counted as you were simply 'viewing' the task.
{% endhint %}
When a user first opens an asset, the active timer starts, indicated by the green dot in the top right of the screen:
If, however, the user is idle for a period of time set by the project manager (by default 300 seconds), the *active timer* stops and the *idle timer* starts, and the dot turns gray:
When the dot is gray, the *active timer* stops and the *idle timer* starts. As project manager, you can choose when to show the idleness notice, and thus, when to start considering your users as idle.
## Changing the Idle Time Threshold
To change the duration after which you consider your project members as being idle, navigate to Settings -> General, then change the number, in seconds, under the *Idle Timeout* heading:
{% hint style="info" %}
The Idle Timeout setting is available for standard Ango Hub projects. 3D MSFT projects do not show this setting.
{% endhint %}
Lastly, click on *Save* to save your settings.
## Idle Detection Examples
Assuming a 5-second idle detection threshold.
| Scenario | Durations |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
Open task
Annotate for 15 seconds
Save and quit.
|
Duration: 15s Idle duration: 0s
|
|
Open task
Annotate for 15 seconds
Save and quit.
Open the task and don’t perform any annotation
Quit without saving
Enter the task again and perform annotations for 10 seconds
Save and quit
|
Duration: 25s Idle duration: 0s
|
|
Open task
Annotate for 10 seconds
Change tab for 30 seconds (keeping the browser window active)
Go back to the annotation and annotate for 10 more seconds
Save and quit
|
Duration: 20s Idle duration: 30s
|
|
Open task
Annotate for 10 seconds
Do nothing for 30 seconds with the tab and window active
Move the mouse and annotate for 10 more seconds
Save and quit
|
Duration: 25s (20 seconds of actual annotation + 5 seconds idle detection threshold) Idle duration: 25s (30 seconds of actual idleness minus 5 seconds idle detection threshold)
|
|
Open task
Annotate for 10 seconds
Save and quit
Open task again
Be active, but don’t perform any annotation for 10 seconds
Submit
|
totalDuration (in export): 20 seconds
This occurs because saving keeps the task in its current stage and stores the time spent so far. When the user later submits the task, the saved time from both sessions is added to the task's total duration for that stage.
|
|
Open task
Annotate for 10 seconds
Change the foreground application for 30 seconds (keeping the tab open in the window, but with the window in the background) and never hover over the window.
Go back to annotation and annotate for 10 more seconds
Save and quit
|
Duration: 25s (20 seconds of actual annotation + 5 seconds idle detection threshold) Idle duration: 25s (30 seconds of actual idleness minus 5 seconds idle detection threshold)
|
|
Open task
Annotate for 10 seconds
Change the foreground application for 30 seconds (keeping the tab open in the window, but with the window in the background) and never hover over the window.
Go back to annotation and annotate for 10 more seconds
Submit
Review the asset for 5 seconds
Submit
|
In the export: totalDuration: 25s + 5s = 30s
(review) stageDuration: 5s
|
## Inspecting time spent on tasks
Other than the columns present in the *Performance*, *Assets*, and *Tasks* tabs, you may also inspect a single task's idle and active times, both for the stage it is currently in and as a total.
To do so, open the task you wish to inspect, and open the *Task Info* panel on the right side:
In the highlighted box, you will be able to inspect active and idle times, both for the current stage and for the totality of the task's stage history.
# Instructions
Overview on how to upload and view labeling instructions on Ango Hub.
Project owners and [Managers](/labeling/managing-users-in-projects#managers) can add labeling instructions, in PDF format.
These instructions are then available to see to anyone who can access the project, and are viewable directly from the labeling editor.
## Uploading and Viewing Instructions on Ango Hub
### How to upload instructions
From the project’s *Settings* tab, enter the *Instructions* section. Click on the blue *Upload Instructions* button. You will be prompted to pick a PDF file from your system.
If instructions have already been uploaded, you can replace them by clicking on *Upload Instructions*.
Once uploaded, the instructions will appear under the *Upload Instructions* button.
### How to view instructions
#### Managers, Reviewers, and Labelers
From the labeling editor, click on the *Instructions* button in the right sidebar. A panel showing the uploaded instructions will appear.
The buttons above the PDF allow you to zoom in, out, fit the PDF to screen, and to navigate between pages.
#### Labelers Only
From the *Samples* tab, click on *Instructions*.
# Issues
Open and talk about labeling issues for all reviewers and managers to see.
Labelers, reviewers, and managers can open issues on labeling tasks to have a conversation about a particular asset or label.
Labelers can open issues on the tasks they have been assigned to, for example, to clarify how a certain asset needs to be labeled. Mentioned reviewers and project managers will get notified of the new issue so that they may take action and respond.
Reviewers can open issues on tasks to, for example, communicate to the labeler how they should label similar tasks in the future. Labelers get notified of the opened issues opened this way.
Users can have real-time conversations on issue threads, mention people to be notified, and delete/resolve issues.
## Opening and Viewing Issues on Ango Hub
### How to Open an Issue
#### General Task Issue
Users can open issues that do not point to a specific region of the asset, and are about the task in general.
To do so, open the task you'd like to open an issue in. From the right-hand sidebar, click on *Issues* and write your issue in the box. Mention other users if necessary by @ing them, and press Enter. You can also press "I" on your keyboard if no object is selected to quickly open a task-level issue.
{% hint style="info" %}
You can only mention users which are also members of the project.
{% endhint %}
When you get a reply, you will get a notification.
You can edit or delete the issue using the *Edit* and *Delete* buttons on the issue.
#### Specific Area Issue
In visual and audio labeling tasks, you can place an issue marker on the image/audio to direct attention to a particular region of the asset related to your issue. This is available for images, videos, PDF, audio, and medical data.
To do so, open the task in which you'd like to open an issue, and click on the speech bubble icon in the top-right corner of the screen. Then, click on the asset region of interest. Type your issue and press Enter.
In audio files, after clicking on the issue bubble icon, click and drag on the waveform where you would like to place your issue:
{% hint style="info" %}
Issues created this way will also be visible in the right-hand sidebar.
If an issue is obstructing your view, you can hide it by clicking on the  icon, and you can unhide it by clicking it again from the sidebar.
{% endhint %}
For medical data, you can additionally draw a line when the issue cursor is active, to further point at a specific area when opening a Specific Area issue:
#### Object-Level Issues
To open an issue tied to a particular object, right-click on the object, expand the context menu, click on the three dots and choose *Open Issue*:
Alternatively, from the *Objects* list on the bottom left of the screen, click on the three dots on the row of the object you'd like to open the issue on, then click on *Create Issue.*
Alternatively, click on an object, then press "I" on your keyboard.
When creating an object-level issue, the dialog displays the object's *Object ID*. You can use the copy button next to the ID to copy it in full, for example when referencing the same object in another conversation or system.
#### Video Issues
When you create a specific-area or object-level issue in the Video Editor, the issue dialog displays *Start frame* and *End frame* fields. The start frame is the frame where you opened the dialog. Set the end frame to create an issue that applies across a range; the end cannot be greater than the video's final frame.
The Issues sidebar displays *View on Frame N* for a single-frame issue or *View on Frame N \[N-M]* for a range. Clicking the action moves the video to the issue's start frame. An issue marker remains visible throughout its frame range.
#### Classification-level Issue
Click on the three dots next to the classification answer you'd like to create an issue about, and click on *Create Issue*:
#### Multi-Image Issues
In [multi-image assets](/data/importing-assets/bundled-assets/importing-multiple-images-in-one-asset-grid-or-carousel), you may open an issue spanning multiple images.
To do so, navigate to the image where this issue starts occurring. Then, select the issue "speech bubble" icon and click anywhere on the asset to open the *Open Issue* dialog:
At the *End* selector, pick the image number where the issue ends. Click on the arrow to create the issue. You'll be able to see the issue page range when browsing issues in the issue sidebar, as well as in the [issue export](#downloading-all-issues-in-a-project).
Page range in the Issue Sidebar
In the issue export:
Issue object with page range in issue export
```json
{
"_id": "65364c3bc1407000158d96d3",
"points": [],
"status": "Open",
"errorType": "Comment",
"labelTask": "651d596274373600156401ad",
"asset": {
"_id": "651d596274373600156401a9",
"dataset": [
"URLs"
],
"batches": [],
"externalId": "my-asset-external-id-1",
"data": "https://angohub-public-assets.s3.eu-central-1.amazonaws.com/3e6e15b9-c32c-4b73-97f9-dbda784926ab.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIATAGM6WLISC5CRH7S%2F20231004%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20231004T122403Z&X-Amz-Expires=120000&X-Amz-Signature=d8f4e7ba5794faf2e5b27a83375b93bc234935c930bf371fdcb936e66fd8f012&X-Amz-SignedHeaders=host&x-id=GetObject",
"head": "https://angohub-public-assets.s3.eu-central-1.amazonaws.com/3e6e15b9-c32c-4b73-97f9-dbda784926ab.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIATAGM6WLISC5CRH7S%2F20231004%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20231004T122403Z&X-Amz-Expires=120000&X-Amz-Signature=d8f4e7ba5794faf2e5b27a83375b93bc234935c930bf371fdcb936e66fd8f012&X-Amz-SignedHeaders=host&x-id=GetObject"
},
"project": "64e5e49785eb730015b4eb7b",
"position": "[266.8706411698538,280.4731774415406]",
"content": "I'm not sure how to annotate this car between these frames.",
"stage": "Complete",
"contentMentions": [],
"page": 0, // Page start, zero-indexed
"pageEnd": 3, // Page end, zero-indexed
"createdBy": "lorenzo@example.net",
"organization": "64db3c87c14b35001503e10b",
"createdAt": "2023-10-23T10:34:35.018Z",
"updatedAt": "2023-10-23T10:34:35.018Z",
"comments": [],
"stageId": "Complete"
}
```
### Viewing and Responding to Issues
To view and respond to issues in the project, go to the *Issues* tab and click on the issue of interest:
You will be directed to the task where the issue was opened, and the issue will be highlighted for you. To respond, simply type your response in the box and press Enter:
For an object-level issue, click its *View on Image*, *View on Page*, *View on Frame*, *View on Text*, or *View on Timeline* action to navigate to the relevant location and select the corresponding object in the editor.
On a single-image task, *View on Image* also centers supported shape objects in the viewport. Segmentation and brush objects are selected without changing the viewport.
If you believe your issue was resolved, click on *Resolve* on the top-right corner of the issue.
### Filtering Issues by Task ID
From the project *Issues* tab, click *Filter* and select *Task ID* to show only issues associated with specific tasks. Enter a Task ID and press Enter to add it to the filter. You can add multiple Task IDs; each ID appears as a separate filter value.
The Task ID filter can be combined with the other available issue filters. It is also available to labelers from the *My Issues* table.
### How to hide all issue bubbles from the asset
Click on the "Eye" icon in the issue panel:
## Downloading All Issues in a Project
Navigate to the *Issues* tab in your project, and click on *Export Issues*:
You will receive a JSON file containing all details pertaining to all issues in the project.
# Issue Error Codes
Reviewers may assign error codes to tasks they review. This way, instead of writing a issue by hand, they can affix a pre-built error code detailing the nature of the problem with the task. (e.g. *Slightly incorrect, Very Incorrect, Missing*, etc.)
## Setting Up Error Codes
To set up error codes in your project, navigate to *Settings -> General* and enable the toggle next to *Error Codes:*
From this interface, you can add, remove, and edit error code categories, and add individual error codes in them.
To add a new error code category, click on the "+ Add Error Code" button. To edit the name of an existing one, click on the *pen* icon. To delete, click on the *trash can* icon.
To create an error code, type the name of the error code in one of the categories, then press Enter. There is no limit to how many error codes a category may contain. Press *Save* at the bottom of the page to save your error codes.
## Assigning Error Codes during Review
There are currently two ways to assign error codes to assets: through issues or when rejecting a task during review.
### Error Codes on Issues
During review, or any other stage and role in which issues can be placed, click on the "issue" icon in the top right and click on the asset where you'd like to place the issue, then pick the error code category and an error code:
Write your issue and press the Enter key or the *send* button to finalize your issue.
Alternatively, you may create issues directly on objects by selecting the object and pressing on the letter I on your keyboard, and assign error codes that way.
### Error Codes on Rejection
When in a review stage, click on the *Reject* button at the top right to mark the task as rejected. If the task contains no issues, a popup will appear.
Pick a category and an error code as shown in the previous section and click on *Open Issue & Reject*.
{% hint style="info" %}
Is it currently not possible to add error codes to issues created from the *Issues* panel on the right of the screen.
{% endhint %}
# Label Validation
⚠️ Warning for validations on projects created before Ango Hub 3.10 (3 Jan 2024)
Prior to Ango Hub 3.10, a toggle in the project settings allowed the project manager to decide whether the validations in the project prevented submission or not. This toggle was global for the entire project, meaning that all validations in the project either prevented submission or not.
Since Ango Hub 3.10, the toggle has been deprecated. Instead, the project manager can now decide whether each validation prevents submission or not by including a boolean named `preventSubmission` to the error. If `true`, the annotator will not be able to submit when the error is shown. If `false`, the annotator will be shown the option to submit the task anyway.
For validation code created before this change, or wherever the `preventSubmission` boolean is not present, we default to preventing submission.
You can show errors to labelers when they try to submit annotations that don't fit requirements of your choice, and if you so wish, you can prevent them from submitting their label entirely.
Ango Hub provides a fully customizable and programmatic way to validate annotations, allowing you to create JavaScript functions that hook directly into Ango Hub's code and run when annotators attempt to save or submit their annotations.
{% hint style="info" %}
Label validation is run only when the labeler clicks on the *Validate* or the *Submit* button.
{% endhint %}
{% hint style="info" %}
Label validation will not run unless you enable the *Enable Custom Validation* toggle in the *Custom Validation* section of the project settings:
{% endhint %}
## Enable Label Validation
Navigate to the project where you'd like to enable label validation, then go to the *Settings* tab and enter the *Label Validation* section:
**Enable Custom Validation**: when enabled, validation will be activated for the current project.
In the code area below, you will enter your JavaScript function containing the logic of your label validation. You may click on *Validate Code* to check for syntax errors and try your code on sample labels.
## Set Up Label Validation
In the code area of the *Label Validation* section, enter a JavaScript function that takes one parameter (we'll call it *answers,)* and returns a list of dicts with the errors you'd like to show. The function will be run every time an annotator tries to save or submit annotations.
### Function Parameters
Ango Hub will pass one parameter to your function, a JSON object which contains all objects, relations, and classifications the annotator is trying to submit.
This is a sample 'answers' parameter you'll receive from Ango Hub. In this case, the user created two PDF areas, created a single relation between them, and replied to a classification with two answers:
```json
{
"tools": [
{
"pdf": {
"position": {
"boundingRect": {
"x1": 157,
"y1": 166.1374969482422,
"x2": 200,
"y2": 226.1374969482422,
"width": 671.001473820641,
"height": 670.9999999999999
},
"rects": [],
"pageNumber": 1
},
"content": {
"image": "data:image/png;base64="
}
},
"objectId": "7e0bd6f41a01c78c15ae608",
"classifications": [],
"metadata": {
"createdAt": 1674132659608,
"createdBy": "614348d554e17400149964b1"
},
"schemaId": "801a2b8b8079b50d4541789"
},
{
"pdf": {
"position": {
"boundingRect": {
"x1": 106,
"y1": 271.1374969482422,
"x2": 135,
"y2": 332.1374969482422,
"width": 671.001473820641,
"height": 670.9999999999999
},
"rects": [],
"pageNumber": 1
},
"content": {
"image": "data:image/png;base64="
}
},
"objectId": "8083daa5d68a090aca6c738",
"classifications": [],
"metadata": {
"createdAt": 1674132663738,
"createdBy": "614348d554e17400149964b1"
},
"schemaId": "801a2b8b8079b50d4541789"
}
],
"classifications": [
{
"schemaId": "fe5af3c7471de99e3868448",
"answer": [
"3b935728b7f43b18b1f7553",
"cb48de7b8c1a10908b2d747"
],
"page": 0,
"metadata": {
"createdAt": 1674132625127,
"createdBy": "614348d554e17400149964b1",
"updatedAt": 1674132625364,
"updatedBy": "614348d554e17400149964b1"
}
}
],
"relations": [
{
"from": "7e0bd6f41a01c78c15ae608",
"to": "8083daa5d68a090aca6c738",
"objectId": "5ccee3e5af7eeb84b59a358",
"schemaId": "c3ec9fbe6c52567a0dfb479",
"direction": "straight",
"metadata": {
"createdAt": 1674132666358,
"createdBy": "614348d554e17400149964b1"
}
}
]
}
```
For more on what can be passed in 'answers', we recommend following the steps in the next section. Following that, consulting the page [Ango Annotation Format](/data/importing-and-exporting-annotations/exporting-annotations/ango-export-format) will also help, as it details the format in which annotations are exported, which is the same as what you'll receive as parameter.
#### Quickly previewing the 'answers' parameter
We developed a fast and efficient way to preview what will be passed to your function based on your needs:
1. Open the labeling editor and perform a test annotation on Hub with the tools and answers you need.
2. Without having to save or submit the annotation, from the three-dot menu in the top-right corner, click on *Copy Answers:*
You will have copied exactly what Ango Hub would have passed as parameter in your validation function based on the labels on the screen.
### Function Returns
Your function will need to return a list of dictionaries, containing the ID of the object which is causing the error, an error message string, as well as whether the error should prevent the annotator from submitting the task or not.
This is what a sample return will look like:
```javascript
[
{
objectId: "12345678",
message: "A point has an X coordinate below 100.",
preventSubmission: false
},
{
objectId: "12345690",
message: "Maximum 3 answers allowed.",
preventSubmission: true
}
]
```
### Testing and Debugging your Validation Code
From Ango Hub, you can test and debug your function without having to perform labeling and submitting test annotations every time.
1. Obtain a sample of what will be passed into your function by following the steps [in this section](#quickly-previewing-the-answers-parameter).
2. From your project dashboard, enter the *Settings* tab, then the *Label Validation* section. In the code area, enter your validation function.
3. To test and debug your code, click on the *Validate Code* button below the code area:
A dialog will open:
5\. Paste the input parameter you have copied in Step 1, and click on *Run*.
Ango Hub will scan your function for syntax errors, and show them to you if present.
If your function successfully returned errors based on the annotations JSON you've just passed it as parameter, Ango Hub will show them to you below the code area:
By moving back and forth between writing your function and the validation dialog, you can create a quick build-debug workflow allowing you to create and test your functions quickly.
## Code Examples
Give an error if a point exists with its X coordinate under 100:
```javascript
function (answers) {
let errors = [];
const points = answers.objects.filter((obj) => obj.tool === "point" && obj["point"][0] < 100);
points.map((b) => {
errors.push(
{
objectId: b.objectId,
message: "Point X under 100.",
preventSubmission: false
}
);
})
return errors;
}
```
Give an error if an empty group relation exists:
```javascript
function (answers) {
let errors = [];
const groupRelation = answers.relations.find((r) => r.tool === "group");
if (groupRelation?.group.length > 0) {
errors.push({
objectId: groupRelation.objectId,
message: "Group relation cannot be empty.",
preventSubmission: false
});
}
return errors;
}
```
Give an error if a bounding box is narrower than 100px:
```javascript
function (answers) {
let errors = [];
const bbs = answers.objects.filter((obj) => obj.tool === "bounding-box" && obj["bounding-box"]["width"] < 100);
bbs.map((b) => {
errors.push(
{
objectId: b.objectId,
message: "BB is narrower than 100px.",
preventSubmission: false
}
);
})
return errors;
}
```
Give an error if a group relation you specify contains objects with names different from the ones you specify:
```javascript
function (answers) {
let errors = [];
groupRelationToCheck = "TITLE OF GROUP RELATION TO CHECK"
const acceptableObjectTypes = [
"TITLE OF ACCEPTABLE OBJECT 1",
"TITLE OF ACCEPTABLE OBJECT 2"
]
const groupRelations = answers.relations.filter((r) => r.tool === "group" && r.title === groupRelationToCheck);
groupRelations.forEach(groupRelation => {
groupRelation.group.forEach(object => {
if (!acceptableObjectTypes.includes(object.title)) {
errors.push({
objectId: groupRelation.objectId,
message: `A ${groupRelationToCheck} group relation contains a ${object.title} object.`,
preventSubmission: false
})
}
})
})
return errors;
}
```
Give an error if a group relation you specify contains more than one object from the ones you specify:
```javascript
function (answers) {
let errors = [];
groupRelationToCheck = "TITLE OF GROUP RELATION TO CHECK"
const acceptableObjectTypes = [
"TITLE OF ACCEPTABLE OBJECT 1",
"TITLE OF ACCEPTABLE OBJECT 2"
]
const groupRelations = answers.relations.filter((r) => r.tool === "group" && r.title === groupRelationToCheck);
groupRelations.forEach(groupRelation => {
acceptableObjectTypes.forEach(objectType => {
matchingObjects = groupRelation.group.filter(object => object.title === objectType)
if (matchingObjects.length > 1) {
errors.push({
objectId: groupRelation.objectId,
message: `A group relation contains more than one ${objectType} object.`,
preventSubmission: false
})
}
})
})
return errors;
}
```
In tasks with brush tools, return an error if any pixel in the image has not been covered with a brush trace:
```javascript
function (answers) {
const data = currentBrushData.data;
const size = data.length;
for(let i = 0; i < size; i+=4) {
if(!data[i] && !data[i+1] && !data[i+2]) {
return [{message: 'Empty pixel!'}];
}
}
return [];
}
```
# Labeler Performance
Viewing and analyzing labeler performance on Ango Hub
From your project's dashboard, enter the *Performance* tab. You'll be able to see detailed information about your labelers' performance.
## Performance Page Columns
### Completed tasks
**Labels** shows how many labeling tasks an annotator has completed.
**Labeling Time** is how long each annotator has spent on the project.
**Avg. Labeling Time**: How long the annotator has spent, on average, annotating each asset.
**Reviews** is how many reviews a labeler has completed in the project.
**Review Time**: How long the annotator has spent, cumulatively, reviewing.
**Avg. Review Time**: How long the user has spent, on average, on reviewing each task.
### In progress tasks
## Exporting Performance CSV
You may get a CSV of all data in the Performance tab by clicking on the *Download* button in the top-right corner of the table:
# Labeling
Overview of performing data labeling on Ango Hub.
Labeling is the main action performed on Ango Hub.
Annotators open the [labeling editor](/labeling/labeling-editor-interface), in which [assets](/core-concepts/assets) are shown. They then select [labeling tools](/labeling/labeling-tools) and place labels on assets. [See all supported file types on Ango Hub](/data/data-in-ango-hub/supported-asset-file-types).
## How to start labeling
From the project’s dashboard, click on the *Start Labeling* button on the top right.
You will be shown the first available unlabeled asset in the labeling queue.
If the project [workflow](/core-concepts/workflow) contains multiple ["Label"-type stages](/core-concepts/workflow/label), you will see a dropdown next to the "Start Labeling" button. From this dropdown, you will be able to select the labeling stage you'd like to work on. Labeling stages in this dropdown are ordered by the time they were added to the workflow. If you click on "Start Labeling", Ango Hub will try to find a task for you from the first labeling stage in the dropdown, then if it cannot find one, it will attempt the same for the second stage, and so on.
To annotate, select a labeling tool from the *Tools* section in the left sidebar, then place the label on the asset. If there are no tools in the *Tools* section, answer the questions in the *Questions* section. ([More details on labeling tools](/labeling/labeling-tools).)
Once you have finished labeling ([see here an overview of the labeling editor interface](/labeling/labeling-editor-interface)) click on *Submit*. You will be shown the next unlabeled asset in the queue. Otherwise, click on the arrow next to Submit and click on "Submit & Exit" to submit the task and exit the queue (i.e. not get a new task). [See more on the labeling queue here](/core-concepts/labeling-queue).
# Labeling Queue
Order in which assets are shown to labelers on Ango Hub.
The labeling queue is the list of [tasks](/core-concepts/tasks) that are distributed to labelers, in order.
If your project's workflow has multiple labeling stages, each stage will have its own queue.
## Using the Labeling Queue
From the dashboard, click on the *Start Labeling* button on the top-right:
You will be shown the first labeling task that is available to you from one of the labeling queues. If your project has more than one labeling queue, the queue will be picked at random. Once you label it and click on *Submit*, you will be shown the next, and so on.
{% hint style="info" %}
You may choose to label a certain task without following queue order by going to the *Task* tab and clicking on any task, or by clicking on a circle under the *Labels* column in the *Assets* tab.
Doing so does not invoke the labeling queue, and the *Submit* button will not be available to you. Instead, you will see the *Save* button, indicated by a floppy disk icon. This is because labeling and saving the asset will not move you forward one task. Essentially, you will only edit that one particular task.
{% endhint %}
### Properties of the Labeling Queue
* The task at the top of the queue is the task that will be shown next to labelers in the stage.
* Each project has one labeling queue for each *Label* stage in your [workflow](/core-concepts/workflow). When labelers open a new task, they pick from the top of the queue the first task that is unassigned or assigned to them.
* All tasks in a *Label* stage are automatically part of a labeling queue.
* Tasks get added to the bottom of the queue as they are created. Thus, the queue is ordered by task creation time, first in first out.
* Tasks that have been assigned to user X (for example, X is currently labeling them) will be unavailable to Y. Y will be given the next task in the queue that is unassigned or assigned to them.
Whenever users press on the *Start Labeling* button on the top-right corner of the dashboard, they are shown the first task in the labeling queue. When they complete labeling and press on *Submit*, they pick another task from the top of the queue, and so on.
{% hint style="info" %}
If the project has multiple labeling queues as a result of having more than one *Label*-type stage, clicking on *Start Labeling* will choose one of the labeling queues at random.
If you wish to label in a particular queue, click on the downward-facing chevron next to the *Start Labeling* button a pick a particular queue you'd like to work on.
{% endhint %}
## Showing Queue Task Counts
Project managers can show labelers and reviewers how many tasks are currently waiting in their accessible queues.
To enable queue counts, open the project's *Settings* tab, go to *General*, and enable *Show Queue Task Counts* in the *Task Queue* section. Click *Save* to apply the change.
When this setting is enabled, the *Start Labeling* and *Start Reviewing* buttons display the total number of tasks available to the current user. If the project has multiple Label or Review stages, open the arrow next to the button to see the count for each accessible stage. Opening the stage menu refreshes the displayed counts.
The number in parentheses is the number of tasks available in the queue.
The setting is disabled by default. When it is disabled, the buttons and stage menus do not display queue counts.
## Reordering the Labeling Queue
For projects created on or before Ango Hub 4.6, expand this. Otherwise, keep reading below.
You may reorder the labeling (and review) queues to prioritize certain batches.
For example, say you have `Batch_1`, `Batch_2`, and `Batch_3` in your project. While normally tasks are presented randomly from any batch, you may reorder tasks in such a way that tasks from Batch 1 are presented first, then when they are done, tasks from Batch 2, and so on.
{% hint style="info" %}
Reordering reorders tasks simultaneously in all stages. Currently there is no way to reorder tasks in a single stage without using the `priority` SDK function.
{% endhint %}
To reorder all queues, navigate to the *Workflow* tab, then click on the arrow next to *Sync Queues.* Lastly, click on *Re-order & Sync:*
A popup will appear. Select the batches you'd like to prioritize, in order, then click on *OK.*
In this example, labelers and reviewers will be shown tasks from the batch named 1_4 first, then 4_1, and so on.
Now, when picking labeling or reviewing tasks from any queue, tasks from the batch you selected first will be shown, followed by the next, and so on.
{% hint style="info" %}
If there are tasks which were already assigned to a user but not completed, belonging to a non-priority batch, those tasks will be shown first.
In short, this is the prioritization algorithm:
1. Assigned tasks from prioritized batches, in order of priority
2. Assigned tasks from non-prioritized batches
3. Non-assigned tasks from prioritized batches, in order of priority
4. Non-assigned tasks from non-prioritized batches
{% endhint %}
Please refer to the [Priority](/core-concepts/priority) docs page.
# Multiple Classification
You can mark certain classification classes as being "Multiple". When you do so, those classifications can be answered more than once by annotators.
## Set Up Multiple Classifications
From your project's dashboard, go to Settings -> Category Schema. From the *Add Category* dropdown, add a classification to your project. Click on it to expand it, and enable the *Multiple* toggle:
This classification will be marked as multiple, and annotators will be able to answer it more than once.
## Answer Multiple Classifications
When labeling, if a classification has been marked as multiple, it will have a "+" sign next to it. Click on it to answer the classification more than once. You can click on the trash can next to a classification to remove it.
## Importing Multiple Classifications
Here is a sample [import](/data/importing-and-exporting-annotations/importing-annotations) for importing multiple classifications:
```json
[
{
"data": "https://angohub-public-assets.s3.eu-central-1.amazonaws.com/6514e5c0-a3f1-4fd7-818f-8b1a2826107a.jpg",
"classifications": [
{
"schemaId": "0e7e61deb89204036819943",
"title": "Color",
"answer": "Black"
},
{
"schemaId": "0e7e61deb89204036819943",
"title": "Color",
"answer": "White"
},
{
"schemaId": "0e7e61deb89204036819943",
"title": "Color",
"answer": "Red"
}
]
}
]
```
This import will provide this result:
### Importing Frame-Specific Multiple Classifications
When you have a video, or an asset with multiple pages/frames, you may need to have frame-specific classifications, that is, a unique answer for each frame. You can combine frame-specific classifications and multiple classifications to have multiple, frame-specific classifications.
Here is a sample import for a video, with multiple frame-specific annotations:
```json
[
{
"data": "https://angohub-public-assets.s3.eu-central-1.amazonaws.com/10c65b5f-702d-419c-83b9-bdc526008fb6.mp4",
"classifications": [
{
"schemaId": "1331ff47f975d7d06e8e053",
"objectId": "object_1",
"tool": "radio",
"title": "Multiple Radio",
"answer": "1",
"page": 1
},
{
"schemaId": "1331ff47f975d7d06e8e053",
"objectId": "object_2",
"tool": "radio",
"title": "Multiple Radio",
"answer": "2",
"page": 4
},
{
"schemaId": "1331ff47f975d7d06e8e053",
"objectId": "object_1",
"tool": "radio",
"title": "Multiple Radio",
"answer": "2",
"page": 20
},
{
"schemaId": "1331ff47f975d7d06e8e053",
"objectId": "object_2",
"tool": "radio",
"title": "Multiple Radio",
"answer": "1",
"page": 15
}
]
}
]
```
In the above example, we use object IDs to be able to edit different answers to the same classification.
We have only one classification class, called `Multiple Radio`.
On Page 1 the classification is answered once with the answer `1`. On page 4, we add a new answer, and we answer with `2`.
Then, on page 20, we change our first answer from `1` to `2`. We are able to do this because we used the same object ID. Hub, then, knows not to add a new answer but to change the one that already exists.
On page 15, we do the same and this time change our second answer from `2` to `1`.
{% hint style="info" %}
From the import, it is not currently possible to remove/reset answers, as well as stopping interpolation of an answer.
What this means is that once you set an answer, if you do not change it until the end of the video, will remain the same for all frames until the end.
{% endhint %}
## Preventing Users from Deleting Multiple Classification Answers
If you have imported pre-labels containing multiple classification answers, and you wish to prevent users from deleting them, you may lock the classification answer from the labeling editor.
When the multiple classification belongs to a locked annotation, annotators cannot add new answers, edit existing answers, or delete existing answers for that classification until the annotation is unlocked.
# Notifications
Ango Hub features a built-in notification system. Notifications allow you to get informed about events related to your account.
## Types of Notifications
Type
Action on Click
An export you requested is ready to download.
Download the export.
An issue has been opened in a project where you have been assigned as reviewer or manager.
Navigate to task where the issue was opened.
You have been invited to an organization.
Switch to the organization where you have been invited.
You have been assigned to a project.
Navigate to the project where you were assigned.
A batch model plugin has finished processing.
An export plugin has finished processing.
Download the export from the plugin.
## Receiving Notifications
Whenever you receive a notification, if the *Mute* toggle is disabled, you will see a notification popup appear from the top-right hand side of the screen:
Until you click on the notification, the notification will be marked as 'Unread'. The badge indicator on the icon indicates how many unread notifications you have.
Clicking on the notification will mark it as read and will allow you to navigate to the task indicated by the notification, or to download a file, depending on the type of notification. The notification will automatically disappear after a few seconds, or you can dismiss it by clicking on the "X".
## Accessing Past Notifications
To access your past notifications, click on the icon in the top-right corner of the screen. The notification dialog will open:
You may mark all notifications as 'read' by clicking on 'Mark All as Read' in the top-right corner of the notification dialog.
You may refresh the dialog with the latest notifications by clicking on the icon.
## Muting Notifications
You may have notifications stop appearing as a popup by muting them. To mute notifications for your account, open the notification dialog, then enable the toggle.
Muted notifications will still appear in the notification dialog.
## Filtering Notifications
You may filter notifications by type by opening the notification dialog and clicking on one of the filters:
# Organizations
Overview of organizations and organization management on Ango Hub
Organizations are the main way users are organized within Ango Hub. In short, organizations are groups of users with access to the same projects and plugins.
## Quick Facts About Organizations
* All users are always part of at least one organization. It is not possible for a user to not be part of an organization.
* You can belong to more than one organization at the same time.
* Plugins are managed organization-wide. That is, if you have activated a plugin while you were part of Org A, the plugin is active for all users of Org A.
* It is not possible to have a project shared between two organizations.
* The only way to join an existing organization is through an invite sent by an administrator of the organization you wish to join.
## Creating an Organization
### When Signing Up
If you signed up to Ango Hub independently, without having received an invite from an existing user, you have created an organization with a name equal to what you wrote in the *Company Name* field during signup.
An organization with the same name as your company has been created for you automatically, and you have been added as its first and only user.
### After Having Signed Up (From the Organization Switcher)
To create a new organization, click on the name of your organization at the top-right corner of the screen, then click on "New Organization".
You will be prompted to input the name of your new organization. Once you have entered it and clicked on OK, you will be switched to your new organization.
{% hint style="info" %}
If you receive a warning that you are not able to create further organizations, it likely means you are on the free plan and have run out of organizations to create. [Get in touch with our team](https://ango.ai/get-demo/) to upgrade.
{% endhint %}
### Via Invitation
If you signed up to Ango Hub via clicking on an email invite, you have automatically been added to the inviter's organization. No further action is required on your part.
If you are already signed up and logged into Ango Hub, and you receive an invitation to join an organization, you may click on the Hub notification or on the email link to join the new organization.
You will not be removed from your old organization.
## Switching Organizations
If you are part of more than one organization, you may switch between your new and old organization through the *Organization Switcher* from the top-right corner of the screen, by clicking on your current organization, then on the name of the organization you'd like to switch to.
## Users and Roles in Organizations
You can invite as many users to your organization as your plan allows.
Every user will have either the *admin* or the *member* role.
### Organization-Level Roles
* **Admins** can see all projects and all data within projects. Admins can also invite and remove users, and can change the organization- and project-level roles of all other users. Admins can also see organization-level statistics and plan usage.
* **Members** can only see the projects they have been assigned to. In projects, they only see data according to the project-level role they have been assigned to. They cannot manage or invite other users, nor can they see plan usage and org-level statistics.
### Inviting new users to your organization (Admin only)
Click on the *Organization* link in the top bar. You will be brought to your organization's dashboard. From this dashboard you can manage members of your organization.
Click on the red *Invite Member* button on the top right of the screen. In the dialog that pops up, enter the email of the member(s) you'd like to invite, as well as the org role you'd like them to take on. [See more on roles here](/core-concepts/user-roles). You may add multiple members at once quickly by copy-pasting a list of comma-separated email addresses.
You may also expand the *Pre-add Users to Projects* dropdown to also pre-add them to projects, in different roles. When the users you invited accept their invitations, they will automatically be added to the projects you picked here.
An email will be sent with an invitation link to the email you entered. Once the user you invited clicks on the link and signs up, they will appear as members in your organization.
To resend all pending organization invitations, click *Resend Invitations* from the invitations list. Ango Hub will ask you to confirm before sending the new invitation emails.
{% hint style="info" %}
If the user you are inviting is already signed up to Ango Hub, they will receive an email with a link to sign in to their profile and a notification allowing them to switch to your organization. Invitations sent this way have no expiration date.
If the user you are inviting is not already signed up to Ango Hub, they will receive an email with a link to sign up to Ango Hub. Once they signed up, they will receive a notification they will need to click to switch to your organization.
{% endhint %}
### Managing Organization Users
{% hint style="warning" %}
If an admin has added plugins to the organization, changing that user's role to member, or removing them from the org, will also remove their plugins from the org.
{% endhint %}
From the *Organization* page, you can change a member's status, as well as remove them from the organization entirely from the three-dot menu on each members' row:
Before removing a member, Ango Hub shows their email address and explains that they will lose access to the organization and its projects. If the member has added plugins, a second confirmation lists the affected plugin connections before they are removed from the organization.
## Removing Yourself from an Organization (Admin only)
At the present time, you can only remove yourself from an organization if you have admin privileges in the organization and you are not the owner (e.g. original creator) of the organization.
From the *Organization* page, click on the three dots next to your name and click on *Remove From Organization*.
{% hint style="info" %}
If you remove yourself from your only organization, you will be automatically added to a new organization called *Untitled Organization*.
This is because it is not possible for a user not to be part of an organization.
{% endhint %}
## Deleting an Organization
If you are the organization owner, you can delete your current organization.
To do so, from the *Organization* page, enter the *Danger Zone* tab and click on *Delete Organization* to start the process.
If you are an organization admin but not the owner, you can check who the owner is from the *Members* tab in the *Organization* page.
## Organization Statistics
You may access statistics related to your entire organization from the Overview tab of the Organization page:
By default, Ango Hub displays organization statistics on a project by project basis, but you can switch to the *Users* tab to access user-by-user statistics and download a CSV of them by using the *Export* button:
## Sending a link to a specific organization
Normally, if you send someone a link to a project, if they are part of the organization the project is in, they will be asked to switch to the organization and then they'll be directed to the project.
However, if you wish to send a link to a specific organization without sending a project link, you may do so by sending the following link:
```
https://imerit.ango.ai/switch-org/YOUR_ORG_ID
```
Replace `YOUR_ORG_ID` with the ID of the organization you'd like the recipient to switch to. You can find your organization's ID in your "Account" page accessible by clicking on your avatar on the top-right corner:
# Priority
Each task can be assigned a priority level, according to which it will be processed in the queues.
{% hint style="info" %}
Priority on tasks can only be set in projects using the **new** labeling queue logic (Dynamic Queue).
All projects created in Ango Hub versions prior to and including 4.6 use the **old** logic. Projects created in Ango Hub 4.7+ use the **new** logic.
You may upgrade your old projects to use the new logic. To do so, navigate to the project's *Workflow* tab, click on the chevron next to the *Sync Queues* button, and click on *Switch to Dynamic Queue.* The project will be upgraded instantly.
Please note that this process is **irreversible**. Projects upgraded this way lose the "Re-order and Sync" functionality, and gain the "Priority" functionality.
{% endhint %}
Project managers may assign granular priority levels to tasks in their projects. Tasks with higher priority levels will be shown to users (labelers/reviewers) before tasks with lower priority levels.
Tasks can be assigned a priority level between -1000 and +1000.
If you do not assign a priority level to a task, its default priority level is 0.
## How to set a task's priority
### During Import
#### In bulk from the *Add Data* import dialog
For the following import methods in the *Add Data* dialog:
* Cloud Upload
* Local Upload
* File Explorer
You may set the priority of the tasks you are importing from the *Priority Value* field.
{% hint style="info" %}
This field will not appear in projects using the old labeling queue logic. Refer to the hint box at the top of this documentation page for information on how to upgrade your project.
{% endhint %}
{% hint style="warning" %}
In the *Cloud Upload* import method, if you assigned individual priority levels in the JSON, the priority value you set in the *Add Data* dialog will overwrite the priority levels in the JSON.
{% endhint %}
#### Individually from JSON
When importing assets using the Cloud Upload JSON method, you may add a `priority` key-value pair to each task to assign a priority level to that task.
For example:
```json
[
{
"data": "https://url.com",
"externalId": "cute-cat.jpg",
"priority": 300
}
]
```
{% hint style="info" %}
If, while uploading the JSON to Ango Hub using the *Cloud Upload* dialog, you set a priority level in the dialog itself, that will override all priority levels set in the JSON.
Leave the priority field blank in the *Add Data* dialog to use the priority levels you set in the JSON.
{% endhint %}
### From the *Tasks* tab
Click on the checkbox next to the task(s) you wish to set the priority of.
Click on the *\[Number] Items* dropdown, and then on the *Set Priority* button.
The following dialog will appear. Enter a priority value between -1000 and 1000, and click on *Set Priority*.
The selected tasks will be given the priority level you have just entered.
### From the SDK
Use the [update\_tasks\_priority()](/sdk/sdk-documentation/project-level-sdk-functions/update_tasks_priority) function. Please refer to its documentation page for more information on its usage.
## How to see a task's priority
### From the *Tasks* tab
From the *Tasks* tab, click on the *Columns* dropdown, then enable the *Priority* column:
This will make the *Priority* column appear in your table:
### Filtering tasks by priority
From the *Tasks* tab, click on *Filter*, then select *Priority*.
The *Priority* filter is a numeric filter. You may filter tasks whose priority:
* is equal to a specific value
* is greater than a specific value
* is less than a specific value
* is between two values
Priority filters support the full priority range from -1000 to 1000, including 0 and negative priority values.
### From the labeling editor
With a task open in any labeling editor, click on the *Task Info* button on the right to open the panel containing that task's priority level:
# Projects
Overview of projects and project management in Ango Hub
Projects are the main way to organize and distribute labeling tasks in Ango Hub.
In projects, project owners, [managers](/labeling/managing-users-in-projects#managers), and leads can upload up to 200k [assets](/core-concepts/assets), add annotators, add [reviewers](/core-concepts/reviewing), create label sets (annotation ontologies), export their results, and more. Everything done in Ango Hub is done within projects.
## Project Dashboard
Every project opens on the *Dashboard* tab. The Dashboard summarizes task progress, duration, annotations, and other project statistics in a set of widgets. Use the date and batch filters at the top of the page to limit the data shown in the widgets.
Labelers and reviewers see statistics for their own work. Leads, managers, and project owners can switch between project-wide and personal statistics.
In the *Annotation Count* widget, *Overall* counts annotations currently present in the project's tasks. *Me* counts annotations you contributed across task history, including annotations that were later changed or removed in a subsequent workflow stage. As a result, your personal annotation count can be higher than the overall count.
### Burndown Chart
The *Burndown Chart* tracks the project's total scope, completed tasks, and remaining tasks over time. It is hidden by default; leads, managers, and project owners can add it from the *Widget Library* as described below.
The Burndown Chart can display task counts or percentages.
Use the controls above the chart to:
* **Group by** day, week, or month. This changes both the chart intervals and the recent-progress window used for the projection.
* **Show** task counts or percentages. Percentages are calculated against the project's scope at each point in time, so adding tasks can raise the remaining percentage.
When the project has enough recent progress, the widget shows an estimated completion date and the current completion velocity. The estimate uses the latest 14 chart intervals, so changing *Group by* can produce a different date. If the project is complete, has too little history, or has no recent progress, the widget explains why a date cannot be projected.
The Burndown Chart follows the Dashboard date and batch filters. The date range limits the period shown without resetting the task totals accumulated before its start; the completion projection still uses the full history of the matching tasks. Archived tasks count as completed automatically. Completion dates are based on each task's last update, so a task edited after completion appears on the date of that edit. Tasks without a usable timestamp are excluded.
### Customizing the Dashboard
Leads, managers, and project owners can customize the project-wide Dashboard:
1. Hover over a widget to display its controls.
2. Drag the handle to move the widget, or click the `x` to remove it from the Dashboard.
3. To restore a removed widget, click *Widget Library* in the top-right corner and drag the widget from the panel onto the Dashboard.
Hover over a widget to display its drag and remove controls.
Drag a widget from the Widget Library to restore it to the Dashboard.
The Widget Library only lists widgets available for the current project. For example, *Consensus Distribution* is available only when the project's workflow includes a Consensus stage.
To reuse a layout in another project, click *Copy Layout*, open the other project's Dashboard, and click *Paste Layout*. *Copy Layout* places the layout on your system clipboard, so you can also send the copied text to another user. After they copy the shared text to their clipboard, they can click *Paste Layout* on their project's Dashboard. This copies the widget order and which widgets are visible; it does not copy project data or filters.
{% hint style="info" %}
Applied Dashboard layouts are stored in your current browser, separately for each project. They do not change the Dashboard for other users or sync to other browsers or devices unless the layout is copied and shared as described above.
{% endhint %}
## Managing Projects in Ango Hub
### Creating Projects
From Ango Hub’s *Projects* page, click on “Create Project” on the top right or the center.
A dialog will pop up asking to input the project type, name, and description. Click on *Create Project* at the bottom to create your project. In most cases, you will want to use *Standard* projects. If you need to annotate 3D Multi-Sensor Fusion data such as point clouds, then select *3D Multi-Sensor Fusion*.
#### Importing Sample Projects
Organization admins can click *Import Sample Projects* on the *Projects* page to add one or more ready-to-use examples to their organization. Select the samples you want, then click *Load Samples*.
Imported samples retain their existing batches, label validation settings and rules, and project issues. This lets you explore their configured workflows without recreating the sample setup.
### Pinning Projects
You may pin projects to the top of your project list for easy access.
{% hint style="info" %}
Pinning a project will only pin it for your specific user account.
Other members of your organization will not see your pin.
{% endhint %}
To pin a project, hover over the left side of the row of the project you'd like to pin. A pin icon will appear. Click on it to pin the project.
### Project Tags
Organization admins can create tags and use them to organize projects on the *Projects* page. Project tags are shared across the organization.
To manage the tags available in your organization, open the *Projects* page and click *Manage Tags*.
From the *Manage Tags* dialog, you can create, rename, and delete project tags.
{% hint style="info" %}
Deleting a project tag removes it from the organization and from every project where it was used.
{% endhint %}
To add tags to a project, click the three dots in the same row as the project and click *Add Tags*. Select one or more tags, then click *Add Tags*.
Projects with tags show them in the *Tags* column of the project list. Admins can remove a tag from a project by clicking the `x` on the tag.
When your organization has project tags, the *Projects* table also includes a *Tags* filter, letting you filter the project list by one or more tags.
### Cloning Projects
You may clone a project to create a duplicate of it.
To clone a project, from the *Projects* page, click on the three dots in the same row as the project you wish to clone and click on *Clone Project*.
A dialog will appear, allowing you to select what parts of the project you would like to clone:
Click on *Clone*. A duplicate of the selected project will be added to your project list.
### Deleting Projects
To delete a project, from the *Projects* page, click on the three dots in the same row as the project you wish to delete and click on *Delete*.
{% hint style="danger" %}
Deleting projects is an irreversible and destructive action. Once a project has been deleted, it cannot be recovered.
There is **no** "Trash" to which recently deleted projects go.
{% endhint %}
# Requeuing
Reset labeling tasks and put them back in the labeling or reviewing queue.
Once a [labeling task](/core-concepts/tasks) is created, it flows through the [workflow](/core-concepts/workflow), moving from stage to stage as users, annotators, and plugins process tasks.
Ango Hub provides organization admins, or project users with the Lead or Manager role, with a way to manually override workflow and place a task in a specific stage of your choice with requeuing.
{% hint style="info" %}
From a task's stage history, you are able to see if and when a task has been requeued.
However, requeuing moves a task from one stage to another **without** creating a task history entry.
This means that, for example, if you:
* Have a task in the Start stage
* Import pre-labels to it, or make any modifications
* Then requeue it to another stage, for example a Label stage
* An annotator makes further modifications to it in the Label stage
You would not be able to tell the difference between the modifications made in the Start stage and the Label stage. This is because the task in Start, as-is, was shifted to the Label stage.
To create stage history entries, tasks must be submitted or forwarded using the Start stage settings in the Workflow tab. Saving *does not* create new task history entries.
{% endhint %}
## How to requeue a task
This section will show you how to open the requeue dialog. The sections below it will explain each option in the re-queue dialog.
### Re-queue multiple tasks from the *Tasks* tab
Enter the *Tasks* tab (1), then toggle the checkbox (2) next to each task you wish to re-queue. Then, from the *Item* dropdown click on *Re-queue* (3). The re-queue dialog will open.
### Requeue all tasks in a specific stage
From the *Workflow* tab of your project, left-click on the stage containing the tasks you'd like to re-queue to a different stage. A dialog will appear in the top-right corner of the workflow panel. Click on the three dots, then on *Bulk Re-queue*.
### Requeue a Consensus sub-task
When a task is sent to a Consensus-type stage, Ango Hub creates copies of it, called sub-tasks, and places them in each consensus sub-stage.
For example, if you have a consensus stage with three Label sub-stages, for each task entering the Consensus stage, Ango Hub will create three copies of it and place each copy in its own Label sub-stage queue.
If you wish to requeue one such sub-task, navigate to the *Tasks* tab, expand the main Consensus task, and click on the "Requeue" button on the sub-task:
## Re-queuing options
First, open the re-queue dialog in one of the ways shown in the previous sections. The re-queue dialog will appear:
**Select a Stage**: select here the stage where you'd like to send the tasks.
**Remove Annotations**: Clears all annotations, including objects, brush traces, classifications, and relations.
**Remove Assignees**: If the task was assigned for labeling/reviewing to a specific user, toggling this on will remove the assignment.
**Remove Stage History**: Deletes the stage history on the selected tasks.
# Reviewing
Overview of reviewing annotations on Ango Hub.
Reviewing is a key component in ensuring your labels are kept at the highest possible level of quality.
Reviewing is the act of looking at a labeling [task](/core-concepts/tasks) again and confirming that it was done correctly, or optionally, to fix it if it wasn't done correctly. Only users with the roles *Reviewer*, *Lead*, or *Manager* can perform reviews.
## How to review tasks (Reviewer, Lead, or Manager roles only)
From your project's dashboard, click on *Start Reviewing* on the top-right corner of the screen.
You will be shown the first unreviewed labeling task in the review queue.
{% hint style="info" %}
If your project's [workflow](/core-concepts/workflow) has more than one *Review* stage, this means the project has more than one review queue. The review queue will be picked at random when clicking on *Start Reviewing*.
If you wish to pick a specific review queue to work from, click on the downward-pointing chevron next to the *Start Reviewing* button and select the stage you'd like to review the tasks of.
{% endhint %}
From here, you can accept the labels as they are by clicking on *Accept,* or you can reject the labeling task by clicking on *Reject* in the top-right corner of the screen.
{% hint style="info" %}
*Accept* and *Reject* are flags attached to the particular task. What they mean for your project depends on how the project's workflow was set up.
For example, you may set a workflow up in such a way that rejected tasks are sent to labeling again, and accepted ones are marked as done. See more on [the documentation page for workflows](/core-concepts/workflow).
{% endhint %}
If the current review stage has the *Read-Only* toggle turned on, then you will not be able to make modifications to the annotations. (More on setting up read-only reviewing in the [Workflow documentation page](/core-concepts/workflow#review))
If, however, the current stage is not read only, you may choose to fix the labeling task, make your modifications, then click on either the *Accept or Reject* button.
After submitting your choice, you will be shown the next unreviewed task, until none are left in the current queue. If the project has more than one review queue in which you may perform reviews, clicking on *Start Reviewing* again will open the review queue of a different stage where there are still tasks left to review
## Review Error Codes
Reviewers can assign specific error codes to tasks they review (e.g. *Slightly Incorrect, Very Incorrect, Empty*, etc.) See more on error codes in its docs page here:
{% content-ref url="/pages/7NiolGPeQPyP2qP2nr1f" %}
[Issue Error Codes](/core-concepts/issues/issue-error-codes)
{% endcontent-ref %}
## Reviewing Consensus Tasks
In many projects, the "Disagreement" output of a [Consensus stage](/core-concepts/workflow/consensus) will be linked to the input of a "Review" stage, such that if no agreement was found among annotators, the task will be sent to a reviewer.
### Which annotations get reviewed?
For each task that is output from a Consensus stage and input into a Review stage, the annotations shown to the reviewer are determined by the Consensus stage's adjudication setting.
If *Best Answer* is selected, the reviewer sees a composite task made from the highest-scoring answers for each supported class. If *Select All* is selected, the reviewer sees the annotations from all consensus sub-stages merged together.
### How can I compare different judgments?
Reviewers have two ways, currently, to look at other annotators' judgments and make comparisons.
For classifications, you will see a percentage next to the classification's name. Clicking on it will reveal a popover:
From this popover, you are able to see who answered what question and how. The percentage at the top-left is the consensus score for that classification. If threshold has been met for that classification, the percentage will be shown in green, otherwise, it will be shown in red.
For other annotation tools (e.g. not classifications), currently, the only way to compare judgments is by opening the "Stage History" drawer from the right side of the screen and clicking between judgments. This will allow reviewers to inspect those judgments.
# Review Queue
The review queue is the list of [tasks](/core-concepts/tasks) that are distributed to reviewers, in order.
If your project's workflow has multiple review stages, each stage will have its own queue.
## Using the Review Queue
From the dashboard, click on the *Start Reviewing* button on the top-right.
You will be shown the first review task that is available to you from one of the review queues. If your project has more than one review queue, the queue will be picked at random. Once you review it and click on *Accept* or *Reject*, you will be shown the next, and so on.
{% hint style="info" %}
You may choose to review a certain task without following queue order by going to the *Task* tab and clicking on any task, or by clicking on a circle under the *Labels* column in the *Assets* tab.
Doing so does not invoke the review queue, and reviewing the asset will not move you forward one task. Essentially, you will only review that one particular task.
{% endhint %}
### Properties of the Review Queue
* The task at the top of the queue is the task that will be shown next to reviewers in the stage.
* Each project has one review queue for each *Review* stage in your [workflow](/core-concepts/workflow). When reviewers open a task, they pick from the top of the queue the first task that is unassigned or assigned to them.
* All tasks in a *Review* stage are automatically part of a labeling queue.
* Tasks get added to the bottom of the queue as they are created. Thus, the queue is ordered by task creation time, first in first out.
* Tasks that have been assigned to user X (for example, X is currently reviewing them) will be unavailable to Y. Y will be given the next task in the queue that is unassigned or assigned to them.
Whenever users press on the *Start Reviewing* button on the top-right corner of the dashboard, they are shown the first task in the review queue. When they complete reviewing and press on *Accept/Reject*, they pick another task from the top of the queue, and so on.
{% hint style="info" %}
If the project has multiple review queues as a result of having more than one *Review*-type stage, clicking on *Start Reviewing* will choose one of the labeling queues at random.
If you wish to review in a particular queue, click on the downward-facing chevron next to the *Start Reviewing* button a pick a particular queue you'd like to work on.
{% endhint %}
Project managers can optionally display the number of tasks waiting in Label and Review queues. See [Showing Queue Task Counts](/core-concepts/labeling-queue#showing-queue-task-counts) for setup and behavior.
# Skipping
Project managers may allow users to skip working on certain tasks. Skipping does not generate a new entry in the task's stage history.
## How to configure skipping
In your project settings, go to *General*, and scroll down to the *Skipping* section. You will see two toggles:
**Enable Skipping** (on by default): When enabled, users will see a "Skip" button next to the "Submit" button. Clicking on the "Skip" button will allow users to move on to the next task without submitting the current task.
When skipping is disabled, users will see the following:
**Unassign on Skip** (off by default): When enabled, when a user skips a task, the task will be unassigned from them. This means that that task will be placed back into the common task pool, for anyone to eventually pick it up.
# Stage History
The Stage History of a task is a record keeping track of which [Workflow](/core-concepts/workflow) stages the task has travelled through.
For example, if your task:
* started in the Start stage
* was manually requeued to a Label stage
* was annotated in a Label stage
* went through a Review stage
* went to Complete
This is what the task's Stage History would look like:
## How to access a task's Stage History
### From the Labeling Editor
Open the task in the labeling editor (e.g. click on it). Then, from the right sidebar, click on "Stage History".
### From the *Tasks* Tab
In your project, enter the "Tasks" tab. Find the task you'd like to see the stage history of, then click on the "History" icon to see that task's stage history.
## Stage History in Consensus tasks
For tasks that have passed through a [Consensus](/core-concepts/workflow/consensus) stage, each consensus sub-stage appears as its own entry in Stage History.
When a consensus score is available for a sub-stage, Ango Hub shows the score in that entry. You can use this to see how closely each submitted consensus answer matched the other answers for that task.
# Tasks
Overview of labeling tasks in Ango Hub
Every labeling project is broken down into [assets](/core-concepts/assets), then individual tasks.
### Creating Tasks
Tasks are created automatically when assets are added.
Ango Hub creates one labeling task for each asset.
{% hint style="warning" %}
When an asset is deleted, the tasks assigned to the asset are deleted as well. Importing the asset again will not restore the tasks that have been deleted this way.
{% endhint %}
Tasks are shown in the *Tasks* tab.
**Task ID**. Unique identifier associated with each task.
**External ID**. Non-unique identifier for each asset. If importing assets with [drag and drop](/data/importing-assets#browser-import), this ID will be equal to the filename of each asset. If uploading using [a JSON](/data/importing-assets#cloud-import) or [the SDK](/sdk/sdk-documentation), you will determine this ID on import.
**Stage**. Stage where the task is currently located and waiting.
**Assignee**. User currently assigned to annotate the task.
**Updated At**. Time and date when the task was last updated (e.g. submitted.)
**Duration**. Total duration of the task. See the [page on idle detection for more details](/core-concepts/idle-time-detection-and-time-tracking).
**Consensus**. The task's overall consensus score. When a workflow contains multiple Consensus stages, the column also shows a completed/total count for each stage. Hover over the value to see the stage names and the users who completed each one.
Consensus progress across multiple stages
**Last review**. Result of the last review the task went through.
**Open Issues**. Number of open issues on the task.
Other column types are also available. To show/hide them, click on the *Columns* dropdown:
### Bulk accepting or rejecting review tasks
Project Leads and Managers can bulk accept or reject tasks from the *Tasks* tab when the task list is filtered to a single *Review* stage.
To do so:
1. Open the *Tasks* tab of your project.
2. Filter the table to one *Review* stage from the stage filter.
3. Add any other filters you need, such as batch or assignee.
4. Select the tasks you want to accept or reject.
5. Open the selected-items dropdown and choose *Bulk Accept* or *Bulk Reject*.
If you use *Select all items in the list*, the bulk action applies to every task matching the current filters, not only the tasks currently visible on the page.
*Bulk Accept* and *Bulk Reject* only appear when the table is filtered to exactly one *Review* stage. They are hidden when no stage is selected, when multiple stages are selected, or when the selected stage is not a *Review* stage.
### Deleting Tasks
To delete a labeling task, select the task(s) you wish to delete with the checkboxes on the left, then click on *Actions* and click on *Delete*.
{% hint style="warning" %}
Deleting tasks is a destructive and irreversible action.
{% endhint %}
# Usage
Depending on your Ango Hub plan, your organization may have different usage limits.
## Checking your Current Usage
To see your current usage, as the organization manager, click on the *Organization* link in the top navigation bar, then click on *Usage*.
## What Counts as Usage?
Every time you invite a new member to your organization, and that member accepts the invitation, it is counted as 1 seat.
Every time you upload an asset, of any kind, it is counted as 1 asset. For multi-image assets, each image in the asset is counted as 1 asset.
## Increasing Usage
To increase your usage limits, please contact your iMerit sales representative, or contact us [here](https://imerit.net/contact-us/).
# User Roles
Overview of user roles and permission levels in Ango Hub
All users in Ango Hub belong to both organizations and projects. As such, all users have different roles both at the organization and the project level.
This page is an overview of all roles available to users, both at the organization and at the project level.
{% hint style="info" %}
For more on inviting new users to your organization, and on organization-level roles in detail, check the [Organizations](/core-concepts/organizations#inviting-new-users-to-your-organization-admin-only) page.
For more on inviting users to projects, and on project-level roles in detail, check the [Managing Users in Projects](/labeling/managing-users-in-projects) page.
{% endhint %}
## Organization-level user roles
Level
Role
Properties
Organization
Owner
Can see and edit all data in all projects in organization (equivalent to having the Manager role in all projects)
Can invite and remove org members
Can edit other members' org- and project-level role
Can see org-level statistics
Can see Ango Hub usage/plan details
Can install and remove plugins
Can create API key
Can delete the organization
Currently, there can only be one owner in an organization. Ownership transfer is not yet available from the user interface.
Organization
Admin
Everything the owner can do, except deleting the organization.
Organization
Member
Cannot see/edit organization-level data (statistics, plugins, usage, etc.)
Can only see assigned projects
In projects, can only see data according to assigned project-level role
See assigned projects and act in their according to their project role
✅
✅
See and edit all data in all projects
✅
❌
Invite organization members
✅
❌
Remove organization members
✅
❌
Edit other members’ organization-level roles
✅
❌
Edit other members’ project-level roles
✅
❌
See organization-level statistics
✅
❌
See Ango Hub usage/plan details
✅
❌
Install plugins
✅
❌
Remove plugins
✅
❌
Create API key
✅
❌
Access unassigned projects
✅
❌
### **Project**
Task
Labeler
Reviewer
Lead
Manager
Labeling assets
✅
✅
✅
✅
Viewing assets they labeled
✅
✅
✅
✅
Viewing tasks they completed
✅
✅
✅
✅
Viewing project instructions
✅
✅
✅
✅
Viewing personal statistics
✅
✅
✅
✅
Viewing issues they are mentioned in or opened
✅
✅
✅
✅
Viewing tasks assigned to them
✅
✅
✅
✅
Opening, responding to, solving, deleting own issues
✅
✅
✅
✅
Reviewing labels
❌
✅
✅
✅
Opening all issues
❌
✅
✅
✅
Responding to, solving, deleting all issues
❌
✅
✅
✅
Setting benchmark tasks
❌
❌
✅
✅
Requeuing tasks
❌
❌
✅
✅
Adding/editing/removing project members
❌
❌
✅
✅
Exporting labels
❌
❌
✅
✅
Viewing project-wide statistics
❌
❌
✅
✅
Run export plugins
❌
❌
✅
✅
Set tasks as benchmarks
❌
❌
✅
✅
Assign tasks to users
❌
❌
✅
✅
Assign tasks to batches
❌
❌
✅
✅
Adding and deleting assets
❌
❌
❌
✅
Viewing settings of the project
❌
❌
❌
✅
Editing settings of project
❌
❌
❌
✅
# Workflow
Overview of workflow stages in Ango Hub
Workflows allow you to create custom data labeling pipelines tailored to the specific needs of your project. This can range from simple single-stage labeling tasks to more intricate multi-stage processes involving [reviews](/core-concepts/reviewing), [webhooks](/core-concepts/workflow/webhook), [plugins](/plugins/introduction-to-plugins), logical gates, random sampling, and more. All with a simple and intuitive UI requiring no coding or scripting.
This page will guide you through the various aspects of Workflow and its configuration options.
## Setting up a workflow
From your project, navigate to the *Workflow* tab. You'll be presented with the default workflow, which has a single labeling stage:
In this default workflow, when you upload new [assets](/core-concepts/assets), their tasks are sent to the *Label* stage, where they can be labeled by any annotator in your project. After the annotator submits their annotations, the labeling task is sent to *Complete* and is considered done and ready for export. This is the simplest possible workflow on Hub.
### Adding stages
To add a new stage, for example to add a reviewing stage after the labeling stage, click on one of the stages from the row at the top and drag it on the main workflow panel. Once you release the left mouse button, the stage will be added to the workflow panel, and you can drag it around with the left mouse button:
Adding the stage to the panel is not enough, however: you'll need to plug it into your current workflow.
For example, if we want the review to happen between the *Label* and *Complete* stages, we will first plug the output of *Label* into the input of *Review*. Then, if we want to make it so that when a task is rejected in review it it sent to the *Label* stage again, and if it's accepted, it goes to *Complete,* here's how we would do it:
{% hint style="info" %}
The position of the stages on the Workflow panel is merely cosmetic — functionality-wise, only the connections between stages matter.
{% endhint %}
{% hint style="info" %}
If you add a stage to the Workflow panel but then do not connect it to your workflow, the stage will not be active. Ensure you connect all stages you intend to add to your workflow.
{% endhint %}
The small number at the top right of each stage indicates how many tasks are currently in the stage. *(e.g., if there are 4 tasks in a label stage, it means that 4 tasks are waiting to be shown to annotators for labeling.)*
### Editing and deleting stages
To edit a stage, left-click on it. A dialog will appear in the top-left of the Workflow panel:
To rename a stage, click on the *Pen* icon next to the stage's name in the dialog.
By clicking on the three dots, a number of options will appear:
**Delete** deletes the selected stage.
**List Tasks in Stage** opens the *Tasks* tab with a filter, only displaying the tasks in the current stage.
**Remove Edges** removes all connections to and from the selected stage. To delete connections, you can also click on any one connection to highlight it and then press *Del* (⌫ on macOS) on your keyboard to achieve the same result.
**Bulk Re-queue** gives you the option to [re-queue](/core-concepts/requeuing) all tasks currently in the selected stage. By re-queuing, you can send the tasks to any other stage of your choice. You can remove their annotations, their assigned annotator, and/or their issues. You may read more on re-queuing [here](/core-concepts/requeuing).
Additionally, each stage type has its own set of configurable options. You can read more about each stage's options in [this section](#workflow-stages).
### Saving your Workflow
Click on the red *Save* button in the top left of the screen to save your workflow.
{% hint style="warning" %}
You will not be able to save your workflow if:
* a stage has unconnected inputs/outputs
* a logic stage does not have a logical condition selected
* a consensus stage has less than two labeling or plugin stages
{% endhint %}
## Browse and Restore Previous Workflow Versions
From the Workflow tab, click on the "History" icon on the top-right to open a history of all workflow versions in the current project.
{% hint style="info" %}
Workflow history started being recorded on March 12, 2025. Workflow versions before then had not been saved.
{% endhint %}
To view a previous version of this project's workflow, click on *Preview*. To restore it, click on *Save* while the version you need is previewed.
## Syncing Queues
If, after changing your workflow, you click on *Start Labeling* or *Start Reviewing* and, for any reason, you get an error regarding a lack of tasks to annotate/review, you may use the "Syncing Queues" button to synchronize the state between workflow and tasks.
In short, syncing queues is something to be done as a first step only when troubleshooting errors. If there are no errors in your project, you will not need to use the button.
## Workflow Stages
Here's more about each stage and their options.
# Complete
This is a fixed stage and it cannot be added, edited, or removed. *Complete* is the point at which all tasks end.
Tasks in the *Complete* stage are read-only as they are considered to have been completed.
## Settings
**Prevent Requeuing Tasks to Complete** – when checked, users with requeuing permissions will not be able to requeue tasks to the *Complete* stage.
## Output
*Complete* has no output.
# Consensus
A Consensus stage is a way for you to present tasks to multiple annotators, and have the task be output in either *Agreement* or *Disagreement* conditional upon how much the annotators agree with one another.
{% hint style="warning" %}
Because of the way the Consensus mechanism works under the hood, logic stages of the type "Annotator" and "Duration" may not work as expected when processing tasks output from a Consensus stage.
Requeuing tasks with issues which have been output from a Consensus stage might lead to unexpected behavior regarding the issues. We recommend closing all issues on such tasks before requeuing them.
{% endhint %}
Essentially, the Consensus stage is a container for other Label or Plugin sub-stages.
The Consensus stage accepts plugin sub-stages, such that, for example, you can have a task be labeled by an annotator and a plugin, and you may return the task based on how similarly the annotator labeled the task compared to the plugin.
There is a limit of ten maximum sub-stages you may add to the Consensus stage.
{% hint style="warning" %}
The Consensus stage, by default, does *not* prevent the same task from being labeled by the same person. To prevent that from happening, you will have to assign different annotators to different label stages, as mentioned in [the section for the Label stage](#label). This can be done automatically by clicking on [*Auto Assign*](#auto-assign) in the settings for the consensus stage.
More details in the section for [Auto Assign](#auto-assign).
{% endhint %}
{% hint style="info" %}
Consensus is intended for single-page assets. For multi-page or sequence-like assets, including video, PDF, TIFF, DICOM, NRRD, NIfTI, and assets made from multiple files, tool-based consensus may not be calculated as expected. We do not recommend using consensus in those projects yet.
{% endhint %}
Tool / Classification Type
Consensus Support
Notes
Tools
Bounding Box
✅
Calculated with IoU.
Polygon
✅
Calculated with IoU.
Entity
✅
Calculated from overlapping text spans.
Point
✅
Calculated from the distance between points.
Rotated Bounding Box
❌
Polyline
❌
Segmentation
❌
Brush
❌
Voxel Brush
❌
Circle
❌
PDF
❌
Message
❌
Angle
❌
Classifications
Radio
✅
Checkbox
✅
Single-Select Dropdown
✅
Single-Select Tree
✅
Text
✅
Consensus is 0% if the texts differ, even by a single character, and 100% if they are exactly the same.
Slider
✅
Multi-Select Dropdown
❌
Multiple classifications are not available in the Threshold tab.
Multi-Select Tree
❌
Multiple classifications are not available in the Threshold tab.
Frame-Specific Classifications
❌
Frame-specific classifications are not available in the Threshold tab.
Relations
Single Relation
❌
Group Relation
❌
{% hint style="info" %}
The built-in consensus score calculation supports a limited set of annotation tools and modalities. If your project requires broader tool or modality support, you can use a **Consensus** stage together with the **Calculate Consensus Score** plugin. For setup instructions and supported tools, see [the Calculate Consensus Score documentation](/plugins/first-party-ango-plugins/calculate-consensus-score).
{% endhint %}
## Diagram of how Consensus works
As mentioned in the diagram above, whenever a task enters the Consensus stage, it is 'duplicated' into sub-tasks, and each sub-task is sent to its own sub-stage.
You may examine individual sub-tasks and check their current status from the "Tasks" tab, by clicking on the "Plus" next to the Consensus task to expand it and see details pertaining to the sub-tasks:
If a sub-task is in the "Archive" stage it means it has been completed and submitted.
If a sub-task has been assigned to a specific annotator, but you wish to unassign it from that person and place it back in the queue for that specific consensus sub-stage, you can click on the "Un-assign and send back to queue" button for that sub-task:
Once all sub-stages have been annotated, they will be archived and they will no longer be accessible through the "Tasks" tab. They will, however, be accessible from the "Stage History" panel in the labeling editor when opening the main task.
## Settings
### Auto Assign
By default, Ango Hub does not prevent the same annotator from annotating the same asset more than once as part of a consensus stage.
For example, if you add two Label tasks which can be annotated by *Anyone*, like so:
Labeler A will open their labeling queue and go through the tasks in Consensus\_1.
If no other annotator has opened the tasks annotated by Labeler A, and Labeler A clicks on *Start Labeling* and enters the labeling queue, they may enter the Consensus\_2 queue and label the same tasks again. This way, consensus will not be calculated between two different annotators, as usually expected, since the same annotator will have annotated both tasks themselves.
To prevent this, you'd have to assign each labeling stage in consensus to different annotators. *Auto Assign* automates this process for you.
From the Consensus stage settings, click on *Auto Assign*. The following dialog will pop up:
Toggle on the users you'd like to assign to the stages within the selected consensus container, and they'll be distributed to every consensus stage in the container. If, after doing so, there are no consensus stages in your container assigned to *Anyone*, then you have guaranteed that no labeler will see the same task twice.
### Stage Settings
#### Setup
Clicking on *Add Label* will add a label stage. Clicking on *Add Plugin* will add a plugin stage. Click on each individual stage to change their options. Click on the trash can to delete the stage.
#### Threshold
From this view, you will be able to pick what will be determined as *Agreement* and *Disagreement*. You will see a list of labeling tools present in your project.
To have a tool or classification be used to decide whether the task goes to *Agreement* or *Disagreement*, enable the toggle next to it. Disabled items are not available for consensus thresholds.
In the example above, we have three tools: a bounding box named Vehicle, a radio classification named Color, and a single dropdown named Model. In this example, the task will be considered in agreement when at least 30% of the annotators give the same answer to Color, *and* at least 30% of annotators give the same answer to Model. When both of these conditions are satisfied, the task is marked as being in *Agreement.*
Since the "Vehicle" bounding box had its toggle turned off, annotations from that class will not be used to decide whether the task goes to *Agreement* or *Disagreement*.
{% hint style="info" %}
If no thresholds are selected, or if all selected thresholds are set to 0%, all tasks sent through the Consensus stage will go to *Agreement*.
{% endhint %}
#### Adjudication
The task sent as output is not the judgment from a single annotator – it is instead a composite task, the contents of which will be determined by the adjudication method you pick here.
**Best Answer**
The output task contains the annotations with the highest consensus score, for each class, for classes where consensus can be calculated.
* For example, if the consensus stage has three judgment sub-stages, and the task has three radio classifications A, B, and C, and one bounding box class D, the task output at the end will have, for each classification, the answer annotators coalesced on the most, and for class D, the bounding boxes created by the annotator with the highest class D consensus score.
* For classes where consensus cannot be calculated (e.g. assume in our project there is a rotated bounding box class E), the final task will have the non-calculable classes from the first user who has submitted them in the consensus stage.
* So in this case, we would have the best answers from classes A, B, and C, then the bounding boxes drawn by the user with the highest class D consensus score, and for class E, we would have the answers given by the first user to submit them in the consensus stage.
If, in the Consensus stage, some annotators did not create annotations using a certain class, or did not answer some classification answers, but others did, the output task will contain them, even if not all consensus annotators responded.
* For example, if we have a project with a bounding box class A, a polygon class B, a radio classification C, and a text classification D, assuming:
* User 1 only created 1 bounding box with class A, and answered the radio classification C (no other answers/annotations)
* User 2 only created 1 bounding box with class A, and a polygon with class B (no other answers/annotations)
* User 3 only created 1 bounding box with class A, answered the text classification D (no other answers/annotations)
* The output composite task will have:
* The class A bounding boxes drawn by the user with the highest class A consensus score
* The class B polygons created by User 2
* The class C radio answer from User 1
* The class D text answer from User 3
Here is a visual representation of the algorithm, given three annotators working on the same image:
**Select All**
The output task contains all annotations from all consensus stages/judgments, merged together.
Here is a visual representation of the algorithm, given three annotators working on the same image:
**None**
The output task contains no annotations. Use this option when the labeler or reviewer in the next stage should start with an empty editor instead of receiving a merged consensus answer.
The annotations created in the Consensus sub-stages remain available in the task's *Stage History*, where they can still be inspected. The adjudication method is applied when the task completes the Consensus stage; changing it later does not change tasks which have already completed the stage.
## How Consensus is Calculated
### Classifications
Let `questionCount` be the total number of supported classification questions in the project, and `taskCount` the total number of tasks assigned to an asset.
We calculate the single-question consensus for a single task as `sameAnswers / taskCount`, where `sameAnswers` is the count of answers that are equal to one another, current one included.
We repeat the above calculation for all tasks in the asset, the overall consensus on a single question (classification) is the highest value achieved during the repetitions, (y).
We repeat the above calculation for all questions in the asset, to get to the final result represented as Σ(y) below.
The final consensus score, then, is calculated as `∑(y) / questionCount`.
### Entities
For entity annotations, Ango Hub compares overlapping text spans. Fully matching spans receive the highest score. Partially overlapping spans receive a partial score based on the amount of overlap, and non-overlapping spans receive 0.
### Points
The algorithm checks the distance between two points against the longest distance on the image, which is the image diagonal. For example, in an image with a height of 500 pixels and a width of 1200 pixels, the longest distance is 1300 pixels.
The closer two points are to one another, the higher their consensus score will be. Points in the same position receive 100%. Points separated by the image diagonal, or by a longer distance, receive 0%.
See the following visual examples:
### Other objects (Bounding Box, Polygon)
We calculate consensus for objects using the Intersection over Union (IoU) method.
We compare objects with one another to generate their IoU scores. If some annotations are completely separate, for example, with not even a pixel in common, their IoU score would be 0. If they overlapped completely, their score would be 100.
Noting that objects are compared to themselves too, hence for the above not-intersecting objects example the score of each object would be 50. The highest score achieved for all tasks will be taken into account as the consensus score of that tool (a tool means a unique schema id here, not the tool type).
We then average the IoU scores of all tools to calculate the final consensus score.
## Output
The Consensus stage has two outputs: *Agreement* and *Disagreement*.
If the consensus threshold has been achieved for all labeling tools and classifications specified in the [stage setup](#threshold), the consensus task will be output from the *Agreement* output. Otherwise, it will be sent from the *Disagreement* output.
### The Output Task
The task you will get as the output will be determined by the method you pick in the stage's *Adjudication* tab. The task sent as output is not the judgment from a single annotator; it is a composite task.
# Hold
Tasks in a *Hold* stage will be held there until moved by a project manager. Tasks in a *Hold* stage, when opened, can be edited and saved, but not submitted to be advanced to a further stage.
#### Output
Tasks will be sent through the output plug of a *Hold* stage when manually sent through by a project manager from the *Hold* stage settings.
#### Options
**Manual Forward.** Clicking on *Next Stage* sends all tasks currently in the Hold stage to the stage connected to the output of the Hold stage. If you wish to only send forward tasks belonging to one or more batches, by clicking on the dropdown/text area to the left, you may search for and pick a batch (or more) to forward.
# Label
Tasks is in the *Label* stage will be shown to any user in your project after they click on *Start Labeling* in the project dashboard for annotation.
{% hint style="info" %}
Once a labeler is assigned to annotate a task in a labeling stage, they will remain the assignee for that task for this stage even if, later, the task moves to other stages and then comes back to it.
So if someone annotates a task, and then it gets sent back to the same labeling stage as a result of review later, it will be annotated by the same labeler.
{% endhint %}
#### Output
The task is sent from the output to the next stage when the annotator clicks on *Submit*.
#### Options
**Labeling Assignment.** By default, tasks in a *Label* stage will be presented for labeling to any annotator in your project. By clicking on *Select Labelers* and picking one or more labelers in your project from the dropdown, you can ensure that tasks in this stage will only be shown for labeling to the labeler(s) you picked.
# Logic
The Logic stage directs tasks to different outputs according to logical rules you set.
## Settings
From the Logic stage's settings panel, you can pick one or more logical functions (rules) with which to direct tasks.
Rules will be run top to bottom. When a task is sent to the logic stage, the first rule will be run on the task. If the task matches the rule, then it will be sent out from that rule's output. If not, the second rule will be run on the task, and so on. If, after running all rules, none match, the task is sent out from the 'Else' output.
The Settings panel will start you off with a single rule. To add more rules, click on the button. To remove a rule, click on the button to the rule's number:
## Rule Options
### Annotation Type
Returns as output tasks which contain a specific annotation type, such as an object or a classification answer. All conditions must be true for the logic stage to send the task from the `true` output.
In the example above, for a task to be sent from `true`, it must have a Vehicle-class object, and it must not have a Person-class object. If any of those is false (e.g. it does not have a vehicle, or it has a person) then the task will be sent out from the `False` output. (in short, assume there is an "AND" operator between all conditions.
### Annotator
Returns as output tasks which were annotated by one or more annotators you specify.
When a task reaches a Logic stage, the Annotator rule checks the task's current assignee. This is the user from the stage that sent the task into the Logic stage.
For example, if a task goes from a Label stage to a Review or QA stage, and then from that Review or QA stage to a Logic stage, an Annotator rule in that Logic stage checks the Review or QA user, not the earlier Label stage annotator.
### Random Sample
Returns as output a random percentage of tasks. For example, if you enter 55%, a randomly selected \~55% of tasks will be sent from the True output and a \~45% of tasks from the False output.
{% hint style="warning" %}
The percentage you set in the *Random Sample* logic gate is the likelihood a task passed in is passed out from the True output.
For example, if you set the percentage to 50%, each asset passed in has a 50% chance of being passed out from the True output. This calculation is repeated for each asset passed in and is independent from the number of tasks passed to the outputs before.
As a result of this, asset numbers passed out might be slightly different from what you expect. For example, setting up a 20% / 80% ratio and then sending through 100 items might result in a 83/17 split. Similarly, setting the percentage to 50% on a batch of 10'000 assets may lead to a 4998/5002 split instead of a precise 5000/5000 one.
Because of the way the random sample logic stage works, the more assets are passed to it, the smaller the divergence from your intended percentage. Passing only 4 assets with a 50/50 split may cause a 0/4 or a 1/3 split, while larger datasets will tend towards the average, causing a lower divergence from the intended ratio.
{% endhint %}
### Task Duration
Returns as output tasks based on how long it has taken to annotate them.
### Batch
Returns as output tasks belonging to one or more batches you specify.
### Issue Error Code
Returns as output tasks having at least one issue with an error code you specify.
# Plugin
If you have any Single Model plugins active in your project, tasks passed as input will be sent to be processed by the plugin. The processed task, if the plugin returned no errors, will be returned as output.
The green or red dot next to the chosen plugin's name indicates its liveness. Red means the plugin is down, and green means the plugin is up.
{% hint style="warning" %}
If you re-queue tasks to a *Plugin* stage, Hub will not automatically send them to be processed by the plugin. They will instead be shown as being in the *Plugin* stage unprocessed.
To have the tasks be processed by the plugin, from the *Plugin* stage's settings, click on the three dots and then on *Re-Run*. The plugin will be run on all tasks currently in the *Plugin* stage.
{% endhint %}
#### Setup
In the first dropdown, select the *Model* plugin to use in this stage. Tasks passed as input to this stage will be sent to the *Model* plugin you choose here.
In the *Load Preset* dropdown, you can select a plugin setting preset which has been saved earlier for the current project and selected plugin. [See here for more information on setting and loading presets](/plugins/introduction-to-plugins/plugin-configuration-and-preset-management).
If you disable **overwrite existing annotations**, the plugin will not overwrite annotations existing on the asset – it will instead append to them.
In the dropdowns below, pair the plugin's classes with classes in your project. For example, if your plugin is a detector, detecting various types of objects, it may have classes such as `car`, `traffic_light`, etc. Here, you would match them with your project's own "Car" and "Traffic Light" classes.
You may additionally pass configuration as JSON in the text field.
# Review
Tasks in a *Review* stage will be shown to a user with at least review permissions (e.g. Reviewers, Leads, and Managers) when they click on *Start Reviewing* for reviewing.
{% hint style="info" %}
Once a reviewer is assigned to review a task in a Review stage, they will remain the assignee for that task for this stage even if, later, the task moves to other stages and then comes back to the original Review stage.
Therefore, if someone reviews a task, and then it gets sent back to the same review stage as a result of other workflow actions later, it will be reviewed by the same reviewer.
{% endhint %}
#### Output
The *Review* stage has two outputs, one for each of the review options. If a reviewer, while reviewing, clicks on *Accept*, it is sent from the *Accepted* output. Alternatively, if the reviewer clicks on *Reject*, it is sent from the *Rejected* output.
This allows you to create workflows where, for example, you send rejected tasks to a second review, or back to labeling [as shown in this previous section](#adding-stages).
#### Output Percentages
The percentages displayed on the stage indicate the percentage of times tasks have been sent out of the Review stage from the Accepted or Rejected outputs.
For example, if tasks left the Review stage 100 times, and 76 of these times it was through the Accepted output, then the acceptance percentage displayed will be 76%.
{% hint style="info" %}
Please note that this includes tasks which go through the Review stage multiple times.
For example, in a new Review stage, if a task is initially rejected, this it will count as a rejection, and the rejection rate will go to 100%. Later, if that same task is sent back to this Review stage and accepted, this will count as an acceptance, but for the purposes of the percentage displayed on screen, the earlier rejection is **not** removed. The percentages displayed at this point would then be 50%/50% as there has been one rejection and one acceptance.
{% endhint %}
#### Options
**Read Only.** By activating this toggle, reviewers will not be able to make changes to the annotations. They will only be able to click on *Accept* or *Reject*.
**Bulk Accept/Reject**. From the left dropdown, you may select all tasks in the stage or only a particular batch. Then, by clicking on Accept or Reject, you can bulk accept/reject them. This is akin to clicking on "Accept" or "Reject" on all tasks.
{% hint style="info" %}
In 3D MSFT projects, bulk review actions only apply to unassigned tasks. Ango Hub shows how many tasks in the selected stage or batch are eligible and disables the actions when there are no eligible tasks.
{% endhint %}
You can also bulk accept or reject tasks from the [Tasks tab](/core-concepts/tasks). This is useful when you need to filter the task list first, for example to accept only tasks with a specific label or assignee while leaving the rest of the review queue untouched.
**Assignees.** By default, tasks in a *Review* stage are shown to any user with review permissions who clicks on *Start Reviewing*. To restrict tasks to specific reviewers, click on "Select Reviewer" and choose one or more users from the dropdown. This ensures that only the selected user(s) can review tasks in this stage.
# Start
*Start* is the point from which all tasks start.
When new assets are uploaded, depending on your settings for the *Start* stage, tasks are either held in *Start* (e.g., if you wish to pre-label them before sending them to a labeling task), or are immediately sent to the stage plugged into the output of *Start*.
The *Start* stage will show you a number of statistics related to your project, such as how many tasks are in your project and how many have and have not reached the *Complete* stage.
#### Settings
**Auto Forward on Upload** (default: on): When Auto Forward is toggled on, whenever new tasks are created, e.g. by uploading new assets, they are immediately sent to the stage connected to the output of *Start.* They are not held in start. By toggling it off, tasks will stay in the *Start* stage until you re-queue them to another stage.
{% hint style="warning" %}
If you have tasks (e.g. Task 1, Task 2, and Task 3) in the Start stage, then turn on Auto Forward, then upload more assets (Task 4, Task 5), **all** tasks in the Start stage will be forwarded to the next stage, including Task 1, 2, and 3, even though they were uploaded when Auto Forward was off.
{% endhint %}
**Manual Forward**: Clicking on *Next Stage* will send all tasks currently in the *Start* stage to the stage connected to the output of *Start*. If you wish to only forward tasks from a certain stage, you may do so by clicking on the search bar next to the button and by picking the stage(s) to forward from the dropdown. You may type to quickly search for batches.
# Webhook
Fires a webhook every time a task is passed as input. Returns the same task as output.
## Settings
**URL.** The url where to send the webhook.\
**Auth Type.** The authentication method you'd like to use. Please see [this section below](#authentication-methods) for more information.\
**Logs.** Whenever a task enters the *Webhook* stage and a webhook is fired or an error occurs, this is logged. Clicking on *Logs* allows you to scrutinize logs for all webhooks, with the format: `[Date][Status] external_id`
## Authentication Methods
### No Auth
A standard POST request to your URL with the JSON event payload.
On your server, you may accept the request without verification.
#### Server Code Examples
Flask
```python
from flask import Flask, request, Response
app = Flask(__name__)
@app.route("/hook", methods=["POST"])
def hook():
# No auth — just process the payload
event = request.get_json(force=True, silent=True)
if event is None:
return Response("Invalid JSON", status=400)
# ... handle event ...
return "ok", 200
```
### Secret (X-Hub-Signature)
#### What we send
* A POST request containing the raw body.
* Header: X-Hub-Signature: ``
{% hint style="info" %}
Note: The header value is the plain hex digest (no `sha1=` prefix).
{% endhint %}
#### How verification works
1. Read the raw request body bytes.
2. Compute `HMAC-SHA1(secret, body)` and hex-encode it.
3. Compare the computed digest to the `X-Hub-Signature` header using a constant-time comparison.
4. Reject if they don’t match.
#### Server Code Examples
Flask
```python
import json, hmac, hashlib
from flask import Flask, request, Response
secret = b"your_secret_key"
app = Flask(__name__)
@app.route("/hook", methods=["POST"])
def hook():
computed = hmac.new(secret, msg=request.data, digestmod=hashlib.sha1).hexdigest()
provided = request.headers.get("X-Hub-Signature", "")
# Constant-time compare is recommended:
if not hmac.compare_digest(provided, computed):
return Response("Invalid signature", status=400)
print(json.dumps(request.get_json(), indent=2))
return "ok", 200
```
Node/Express
```javascript
import express from "express";
import crypto from "crypto";
const app = express();
// Use raw body to avoid re-serialization differences
app.post("/hook", express.raw({ type: "*/*" }), (req, res) => {
const secret = Buffer.from(process.env.WEBHOOK_SECRET || "your_secret_key");
const expected = crypto.createHmac("sha1", secret).update(req.body).digest("hex");
const provided = (req.get("X-Hub-Signature") || "");
// timingSafeEqual requires equal-length buffers
const a = Buffer.from(provided, "utf8");
const b = Buffer.from(expected, "utf8");
if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
return res.status(400).send("Invalid signature");
}
// ... handle event ...
res.send("ok");
});
app.listen(3000);
```
{% hint style="info" %}
Things to note:
* Compute the HMAC over the exact raw bytes received, not parsed/re-serialized JSON.
* Use constant-time comparison (`hmac.compare_digest, timingSafeEqual`) to avoid timing attacks.
* If you proxy/transform requests, ensure the body is unchanged before verification.
{% endhint %}
### Bearer Token
#### What we send
* A POST request containing the event payload.
* Header: `Authorization: Bearer `
#### How verification works
1. Read the Authorization header.
2. Check it equals `Bearer `.
3. Reject if missing or mismatched.
#### Server Code Examples
Flask
```python
import os
from flask import Flask, request, Response
TOKEN = os.getenv("WEBHOOK_TOKEN", "your_token")
app = Flask(__name__)
@app.route("/hook", methods=["POST"])
def hook():
auth = request.headers.get("Authorization", "")
expected = f"Bearer {TOKEN}"
if auth != expected:
return Response("Unauthorized", status=401)
# ... handle event ...
return "ok", 200
```
Node/Express
```javascript
import express from "express";
const app = express();
app.use(express.json());
app.post("/hook", (req, res) => {
const token = process.env.WEBHOOK_TOKEN || "your_token";
const provided = req.get("Authorization") || "";
const expected = `Bearer ${token}`;
// Constant-time compare where possible
if (provided !== expected) {
return res.status(401).send("Unauthorized");
}
// ... handle event ...
res.send("ok");
});
app.listen(3000);
```
## General Recommendations
* Respond with a 2xx status once you’ve accepted the event; do heavy work asynchronously. This helps webhooks flow.
* Log the event ID and timestamp (if present) for idempotency/retries.
* Keep your secret/token out of source control and rotate regularly.
## Webhook Content
### Sample Webhook Output
Sample Webhook Output
```json
{
"projectId": "67a3be4088cb6f0f7976327b",
"categorySchema": {
"tools": [
{
"title": "",
"tool": "bounding-box",
"required": false,
"schemaId": "d3fb304c505050a0631e201",
"ocrEnabled": false,
"classifications": [],
"multiple": false,
"color": "#f44336",
"shortcutKey": "1"
}
],
"classifications": [],
"relations": []
},
"asset": "https://angohub-test-assets.s3.eu-central-1.amazonaws.com/67a3be4088cb6f0f7976327b/assets/9a02ef5e-1281-48a6-902e-c959c1e1b201.png",
"assetId": "67a3beaf88cb6f0f797634ff",
"dataset": [],
"overlay": [],
"externalId": "task5.png",
"metadata": {
"width": 1001,
"height": 1001
},
"batches": [
"67a3beb088cb6f0f79763515",
"67a3beb088cb6f0f79763516"
],
"batchNames": [
"batch-1",
"batch-2"
],
"task": {
"type": "default",
"taskId": "67a3beb088cb6f0f79763514",
"stage": "08960214-cb68-4109-ac75-33ed521e9cac",
"stageId": "08960214-cb68-4109-ac75-33ed521e9cac",
"stageName": "Webhook",
"updatedAt": "2025-02-06T10:13:28.709Z",
"updatedBy": "lorenzo@imerit.net",
"duration": 0,
"idleDuration": 0,
"totalDuration": 3549,
"totalIdleDuration": 0,
"tools": [
{
"bounding-box": {
"x": 401.2628984836038,
"y": 398.68154506437764,
"height": 200.2,
"width": 201.91845493562232
},
"objectId": "49c2c2d71cccba94e471625",
"classifications": [],
"metadata": {
"createdAt": "2025-02-06T10:13:27.625Z",
"createdBy": "lorenzo@imerit.net",
"createdIn": "Label"
},
"schemaId": "d3fb304c505050a0631e201",
"title": ""
}
],
"classifications": [],
"relations": [],
"stageHistory": [
{
"stageId": "Start",
"duration": 0,
"idleDuration": 0,
"completedAt": "2025-02-05T19:40:32.305Z",
"isSkipped": false,
"_id": "67a3beb088cb6f0f79763577"
},
{
"stageId": "Label",
"duration": 3549,
"idleDuration": 0,
"completedAt": "2025-02-06T10:13:28.745Z",
"taskHistory": "67a48b48a31195645b7a257b",
"completedBy": "lorenzo@imerit.net",
"submittedBy": "lorenzo@imerit.net",
"isSkipped": false,
"_id": "67a48b48a31195645b7a257d"
},
{
"stageId": "fd4b3696-96ef-49aa-8e2d-c545db740f61",
"duration": 0,
"idleDuration": 0,
"completedAt": "2025-02-06T10:14:08.803Z",
"taskHistory": "67a48b70a31195645b7a2acf",
"consensusId": null,
"isSkipped": false,
"_id": "67a48b70a31195645b7a2ae1"
}
],
"brushDataUrl": null,
"medicalBrushDataUrl": null
}
}
```
### Differences between Webhook Output and Export
Export
Webhook
The batches property provides batches as names.
"batches": [
"Le Croissànt"
]
The batches property provides batches as IDs.
"batches": [
"651521e299f4bf0015872c91"
]
In the webhook output, batch names are provided in the batchNames property.
stageHistory field contains label contents.
stageHistory field only contains metadata, no label contents.
## Webhook Errors
In case the webhook cannot be sent (e.g. Ango Hub does not receive a 200 response), Ango Hub will keep the task in the Webhook stage and display a visual warning in the Workflow editor:
To view the tasks the webhooks of which were not sent, navigate to the *Tasks* tab and filter by stage from the left-hand side.
To attempt to send the webhook again, click on the Webhook stage, then click on the three dots on the top right of its settings panel, and click on *Re-run.*
## Set Up a Sample Webhook Server (X-Hub-Signature Auth Method)
This sample is a minimum server setup you can use to test whether your webhook configuration is working or not.
1. Run this Python script, changing `your_secret_key` with a secret key of your choice.
Sample Webhook Python Script
```python
import json
from flask import request, Flask, Response
import hmac
import hashlib
secret = b'your_secret_key' # Secret Key you entered while adding the integration
app = Flask(__name__)
@app.route('/hook', methods=['POST']) # Custom Endpoint
def hook():
computed_signature = hmac.new(secret, msg=request.data, digestmod=hashlib.sha1).hexdigest()
if request.headers["X-Hub-Signature"] != computed_signature:
return Response("Invalid signature", status=400)
else:
print(json.dumps(request.get_json(), indent=2))
return "ok"
if __name__ == '__main__':
app.run(debug=True)
```
2. Install ngrok on your system. Instructions on installing ngrok can be found [here](https://ngrok.com/download).
3. Once ngrok is installed, from the command line/terminal, run `ngrok http 127.0.0.1:5000`
4. You will see a screen like the following. Copy the URL highlighted in red.
5. Go to your Ango Hub project and set up your workflow to have a Webhook stage plugged in. In this case, for example, the Webhook stage will fire every time a labeler submits a task in the *Label* stage:
6. Click on the Webhook plugin to open its settings.
7. In the *URL* field, paste the URL we copied before, adding `/hook` at the end. For example, if the URL provided by ngrok was `https://47f2-88-243-68-208.ngrok.io`, you will paste it and add `/hook` at the end, forming `https://47f2-88-243-68-208.ngrok.io/hook`.
8. In the *Secret* field, type the secret key you entered in the Python script during step 1.
9. Save your workflow.
10. In your project, perform an action which would trigger a webhook. In our example above, it would be submitting a tasl from the *Label* stage.
If the webhook worked correctly, you will see a `200 OK` code in the ngrok window:
And the webhook content will be sent to your server where you ran the Python script. If you ran it in PyCharm, for example, you will see the webhook contents in the *Run* tab:
# Managing Users in Projects
Overview of inviting, managing, and removing users from projects, as well as their roles
All users, in Ango Hub, belong to organizations, and have certain roles assigned to them.
Every user has two different role levels: an organization-level one and a project-level one.
This page is about project-level user roles. For organization-level user roles, and for more on organizations, check out the [Organizations](/core-concepts/organizations) page.
## Adding organization members to a project
Here's how you can add an existing organization member to your project:
1. Enter the project’s *Settings* tab
2. Enter the *Members* section
3. Click on the *Add Member from Org* button.
From the dialog that pops up, choose the member you wish to add, their role, and click on *OK*
Alternatively, to add users in bulk, enter the "From CSV" tab. Create a CSV file conforming to the sample shown in the "Sample CSV" section, drag and drop it into the upload box, and click on "Review". Review the users which will be added and then click on "Add".
## Adding non-organization members to a project
You may add users directly to your project, letting Ango Hub taking care of inviting them to the organization.
When you invite users to your project this way, they will automatically be added to your organization in the "member" role.
1. Enter the project's *Settings* tab.
2. Enter the *Members* section.
3. Click on the *Invite to Project* button.
From the dialog that appears, enter the emails of the users you wish to add, separated by commas. Then, pick the project role you wish to apply to the new users and click on *Invite*.
{% hint style="info" %}
If you have a comma-separated list of emails, you may paste it into the *User Email* field to quickly add all emails.
{% endhint %}
Alternatively, to add users in bulk, enter the "From CSV" tab. Create a CSV file conforming to the sample shown in the "Sample CSV" section, drag and drop it into the upload box, and click on "Review". Review the users which will be added and then click on "Add".
If your project has pending invitations, you can resend them from the *Pending Invitations* section. Click *Resend Invitations*, then confirm to send the invitation emails again.
## Project-level Roles
Project administrators can assign org members to one of three roles: labeler, reviewer, and manager.
### Labeler
A labeler can, in essence, only annotate data and open issues. Labelers can also view data related to their own performance within the project.
Labelers (or annotators) can access personalized versions of the following tabs:
* Overview
* Tasks
* Samples
* My Issues
The *Overview* shown to labelers is a simplified version of that shown to project administrators and reviewers, only containing data about the labeler’s own performance.
The *Issues* and *Tasks* tabs shown to labelers are also limited, only containing the issues and tasks the labeler logged in has created.
The following are the actions that labelers can perform in Ango Hub:
* Create labels
* Create, reply to, resolve, and delete issues on their own tasks
####
The following are the data that labelers can view in Ango Hub:
* Their own performance from the Overview tab
* Their own issues from the *My Issues* tab
* Their own tasks from the *Tasks* tab
* All samples created in the project from the *Samples* tab
* Project instructions, task attachment, and task info
### Reviewers
Reviewers are equal to *Labelers*, with the only difference being that they have access to the *Start Reviewing* button.
### Leads
Leads are given significantly more permissions than labelers in Ango Hub.
The following are the tabs that leads can access in the projects where they are assigned:
* Overview
* Performance
* Assets
* Tasks
* Samples
* Issues
The level of information leads can see in the above tabs is the same as project administrators.
####
The following are the actions that leads can take within the platform:
* Label
* Review
* Delete and add label tasks
* Set benchmarks
* Set samples
* Rate tasks
* Create, reply to, resolve, and delete issues
The following is the data viewable by leads:
* All labeling data of all project members
* All performance data of all project members
* All assets
* All tasks
* All samples
* All issues
The following is a sample of what a lead can see from the Overview tab:
### Managers
Project managers are equal to project owners: they can access everything project owners can, with no differences.
# Profile Page
Overview of the Profile Page in Ango Hub
From any page in Ango Hub, click on your initials on the top-right corner of the screen, and then on *Account*.
You will be brought to your personal profile page.
### Profile
From this page, you can change your display name and see which organization you belong to, as well as your role and unique ID.
{% hint style="info" %}
Changing your name will also change your initials' circle's background color.
{% endhint %}
### API
By clicking on *Create Key*, you may create a new key to use with our [API](/api/api-documentation-deprecated). If you already had a key, the old key will be revoked. the "Copy" button to the right of your API Key copies it to clipboard.
# Managing the Project Ontology
You can set and manage the ontology of your project by navigating to your project's settings, then to the *Category Schema* section.
## Managing Classes through the UI
### Adding new classes
You can add a new labeling tool (e.g., a new class) to your project by clicking on the *Add Category* button, then selecting the type of tool you'd like to add.
There is no limit to how many labeling tools you may have in your project.
### Editing existing classes
Once you have added classes to your project, each of your classes will be represented by a row in the *Category Schema* section of your project.
To edit a class, click on its row and the row will expand to show the class's details:
All labeling tools share four common properties:
* **Schema ID**: A unique identifier for your class.
* **Title**: What your class should be known as (e.g. `vehicle`, `person`, `lesion`)
* **Description**: Optional text shown to annotators in the labeling editor. For tools and relations, the description appears as a tooltip in the tool list. For classifications, the description appears next to the classification title. Descriptions also work for nested classifications.
* **Required**: Whether or not labelers are required to use this class before submitting their annotations.
* For classifications, annotators will not be able to submit their labeling task without having answered it.
* For visual tools (e.g. bounding box, polygon), annotators will not be able to submit their annotations without adding at least one annotation using the class specified.
For all properties of all labeling tools in Ango Hub, please consult the [Labeling Tools](/labeling/labeling-tools) page and navigate to the docs page describing the tool of your choice.
### Deleting and changing the order of classes
You may change the order in which classes are shown on the labeling editor by using the "up" and "down" arrows on each class's row.
Nested classifications can also be reordered. Use the arrows on a nested classification row to move it within its own group of sibling classifications.
You may delete a class by clicking on the "rubbish bin" icon in each class's row.
## Managing Classes through JSON
You can view and edit the underlying JSON Ango Hub uses to store project ontologies by clicking on the *Show/Hide JSON* button on the top right of the *Category Schema* section.
This allows you to do a number of things. You can:
* Copy the JSON of a project ontology to your clipboard
* Paste an ontology into another project
* Edit the JSON directly
As you update your classes from the UI on the left, you will see the updates change the JSON in real time. This works the other way too – changing the JSON will instantly change what's shown on the UI.
See [here](/how-to/transfer-project-ontologies-between-projects) for a quick how-to guide on how to copy an ontology from a project and pasting it into another.
## What happens if I change the ontology of a project while people are annotating tasks?
They will receive the new ontology after they have submitted the task they are currently working on.
# Labeling Editor Interface
Overview of the labeling editor interfaces and their key features
Ango Hub has labeling editors for each of the supported formats: audio, image, DICOM video, PDF, and text.
Even though the data types supported by Ango Hub are all different, we designed our labeling editors to be as similar to one another as possible. This way, once an annotator has learned how to use one, they will be able to quickly switch to the other, without having to spend time retraining.
In this page, we will go through the features in common between all editors. You can find further information on each editor (e.g. image, audio, etc.) in the pages found within this page in the left sidebar of the docs.
The following is an overview of the editor’s interface.
## Overview of the Labeling Editor on Ango Hub
### Left Sidebar
#### Tools (when available)
At the top of the left sidebar is the list of all labeling tools available in the project. Bounding boxes, polygons, points, and entities, for example, are all labeling tools.
Tools of different types can be present in the same project. To annotate, the labeler clicks on the tool from this section then clicks on the asset, where the label needs to be placed.
#### Classifications (when available)
In the middle of the sidebar are the top-level classification tools for this project.
If created from the *Label Set* section of the *Settings* tab, classifications will be shown here. From this section, annotators can categorize the asset shown, with radio buttons, dropdowns, free text, and more.
In the Markdown Labeling Editor, the left sidebar is hidden when every classification is embedded directly in the asset and there are no tool or relation objects to display. If an existing annotation does not match the current project schema, the sidebar remains visible and lists it as an unknown object.
#### Objects
Below the classification section is a list of all annotations (objects) placed on the current asset.
As the annotator adds labels to the asset, they will be shown in this list.
The “eye” icon next to *Objects* allows you to hide all labels on the screen.
The button allows you to expand/collapse all object rows.
If the label has a nested question, by clicking on it in the list, you can expand it.
The eye icon allows you to hide the label from the asset. You can also use the shortcut H while hovering over an annotation to hide it. To unhide all annotations, press Shift + H.
The trash can deletes it. If present, nested labels will be shown below. In the example above, we ask the labeler what kind of vehicle they have just labeled with the bounding box tool.
Clicking on the three dots will open a submenu:
#### Create Issue
Create an [issue](/core-concepts/issues) bound to the selected object.
#### Change category
If in your project you have more than one of the same tool, you can quickly change the label’s category from here. If we had another bounding box tool called “tree”, for example, we would be able to change this label’s category from “vehicle” to “tree.”
{% hint style="warning" %}
It is not possible to change an annotation to a label of a different type. (for example, a bounding box into a polygon.)
{% endhint %}
{% hint style="info" %}
Another way to quickly change a label's category is to hover over it with the mouse cursor and pressing Alt + the new category's keyboard shortcut.
{% endhint %}
#### Update Description
Add text which will be shown in the object list. This text will be bound to the object and will appear in the final export.
{% hint style="info" %}
A property in common to all classification tools is the *Description*. By default, classifications have no description.
Once the description is changed, it will take place of the title.
To change a classification's description, click on the three-dot menu in the classification's row and click on *Update Description*. The description can also be set during pre-label import, as described in the section on [Ango Hub's import format](/data/importing-and-exporting-annotations/importing-annotations/ango-import-format).
{% endhint %}
#### Copy Object ID
Each object (annotation) has a unique ID. Clicking this button will copy the annotation's ID to the clipboard.
#### Lock
A locked label cannot be edited. The lock function is particularly useful when [importing labels](/data/importing-and-exporting-annotations/importing-annotations), if there are labels you'd like the annotators not to change.
When a locked annotation has nested classification answers, those answers are also locked. Annotators cannot edit the nested answers, add new answers to multiple-answer classifications under the locked annotation, or update the nested classification description until the annotation is unlocked.
#### Grouping
If your project ontology contains group relation classes, by picking the group relation from the "Grouping" menu, you may create a new group with the selected class and add the selected object to such new group.
#### Bring to Front/Back
You may bring the selected annotation to the front or the back. When you do, the annotation will visually appear in front or behind other annotations of its class. (e.g. if an annotation of class A is behind an annotation of class B, clicking on "Bring to Front" will not bring this annotation in front of that of class B.)
In terms of the final export, annotations that have been brought to the back will appear last (after all annotations, even those not in the same class), and annotations that have been brought to the front will appear first (before all annotations, even those not in the same class).
#### Quick Object Scrolling
You can quickly move between objects in the editor by hovering over an object such that it is selected, and then by pressing on Shift + Up or Shift + Down to navigate between objects. The selected object will automatically be shown in the middle of the screen and the zoom level will be accommodated to better view the object.
### Top bar
#### Project Navigation
On the left-hand side are links to navigate between projects and within the current project.
The *Ango Logo* opens your project list.\
The *Home* opens the current project’s dashboard.\
The *Back* button brings you back one page.
#### Navigation Arrows
The arrows allow users to move back to the previous asset, and forward to the next asset.
The navigation arrows are only shown if the current task was opened by following an internal link from within the platform. Directly opening a task from an URL will prevent the arrows from showing.
#### Current Asset External ID
#### Task Stage History
Whenever a task moves from one stage to another, the circle highlighted in this section changes.
The circle at the foreground represents the current stage. The ones on its left, the previous stages, and the ones to the right the expected future stages for the task.
By hovering over each circle, with the mouse, you are able to access more information about the task in each stage.
#### New Targeted Issue Button
By clicking on the speech bubble, you activate the *Targeted Issue* cursor. When this cursor is active, you may, on images, videos, and medical data, click on the asset to leave issues which are specifically tied to a particular X,Y coordinate on the asset.
#### Overflow Dropdown
By clicking on the three dots, you may access the overflow menu, housing options used less frequently.
**Reset**: Return the annotations to their original state.
**Copy**: Copy the current asset's annotations to the system clipboard.
**Paste**: Paste annotations from the clipboard to the current asset. Only works for annotations copied using the *Copy* button in the overflow menu.
**Clear**: Deletes all annotations and resets all classifications in the current asset.
**Set as Sample**: Sets the current task as [sample](broken://pages/VKnp024GiL8Cr4FDQvpP).
**Re-queue**: Opens the [re-queue](/core-concepts/requeuing) dialog for the current task.
**Copy Answers**: Copies to clipboard a JSON of the annotations and classifications currently on the asset, previewing what the asset annotations will look like in the final export. Especially useful in [label validation](/core-concepts/label-validation) workflows.
#### Submit/Save
At the top right of the editor, we find the *Save* and *Submit* buttons.
The Save button is indicated by a floppy disc icon. Clicking on it will save your current annotations without submitting them.
When you click on *Submit*, you will be shown a confirmation dialog asking you if you are sure you wish to submit. When you click on OK on the dialog, changes will be saved and you will be brought to the next unlabeled asset.
If you wish to submit but not be brought to the next task in the queue, please click on the arrow next to the "Submit" button and click on "Submit & Exit":
{% hint style="info" %}
You can also submit by pressing the Enter button on your keyboard, then pressing Enter when the confirmation dialog appears.
{% endhint %}
{% hint style="warning" %}
There is no autosave.
If you try to leave the page without saving, you will be asked to save before exiting.
If your session expires while you are away, Ango Hub may ask you to log in again before you continue working. To avoid losing unsaved annotations, save periodically during long labeling sessions.
{% endhint %}
### Right Sidebar
#### Issues
From the right sidebar, open the *Issues* panel. This will show all issues present in the current task.
Users can type text in the *+ Create Issue* field and press Enter to create issues, which will be sent as in-app notifications to task assignees, project owners, and project reviewers.
The user who created the issue, as well as reviewers and project owners, will be able to reply to the issue, resolve it, or delete it from the buttons on the right side of the panel.
#### Instructions
The *Instructions* button opens a drawer showing labeling instructions for the current project, if present. Project owners can upload instructions as a PDF file from the project’s *Settings* tab.
The buttons above the PDF allow the user to reset the zoom, zoom in and out, and to navigate between pages.
[More on instructions here](/core-concepts/instructions).
#### Samples
The *Samples* button opens a drawer showing all samples set in the project. [More on samples here.](broken://pages/VKnp024GiL8Cr4FDQvpP)
Clicking on a sample in the list will open its related asset and description. Clicking on the *Arrow in box* button in the sample will open an enlarged view of the sample:
#### Attachment
Clicking on *Attachment* will open a drawer showing the attachment linked to the current asset, if available. [More on attachments here](#attachment).
#### Task info
Clicking on *Task info* will show information related to the current task.
#### Stage History
See [Stage History](/core-concepts/stage-history).
### Assets drawer (Project Owner, Manager, and Reviewer only)
Clicking on the right-facing arrow on the left side of the screen will open the *Assets* drawer.
The *assets drawer* contains a list of all assets in the current project, together with their related information such as their status, consensus score, assignees, and more. Clicking on an item in the list will open the related asset.
From the top, clicking on will refresh the asset list. Clicking on *X* will close the drawer. At the bottom, arrows and page numbers allow you to navigate the asset list.
### Labeling Settings
Clicking on the red *Settings* button on the bottom right of the editor will open a panel with a number of quick labeling settings.
Each data type has its own set of settings. This page will only go through those that are common to all data types. For more details on each data type's settings, visit its page, for example, [Image Labeling Editor](/labeling/labeling-editor-interface/image-labeling-editor).
#### Show Object Indices
When enabled, the first three characters of an object's ID will be shown both on the object itself and on the Object row.
#### Show Annotation Classes
If active, when hovering over an annotation, you'll see a summary of the information related to the annotation.
#### Show NER Entity Classes
In the text labeling editor, toggles displaying class names next to NER entities.
#### Hide Segmentation Points
Activate to hide all segmentation points in the asset. Will improve performance in larger projects.
#### Keep tool selected after annotation
By default, after placing an annotation, the current labeling tool is deselected, and the annotator needs to click on a labeling tool again to place another annotation.
When this toggle is selected, placing annotations will not deselect the currently selected labeling tool, and the annotator can immediately place another one. The annotator can deselect the currently selected labeling tool by pressing *Esc*.
#### Show context menu after object creation
If enabled, the context menu will appear instantly upon creating an annotation.
#### Expand context menu automatically
When this setting is activated, right-clicking on an asset will automatically open the expanded version of the context menu, instead of opening the collapsed one.
The object's row in the Objects list will also open and be focused upon selecting the object.
#### Hide Unknown Objects
On occasion, for example as a result of running a plugin, annotations may not be associated with a category. Such annotations are known as "unknown". Activate to hide all unknown annotations.
#### Image Smoothing Enabled
Disable to disable image smoothing. Useful for pixel-wise annotation tasks.
#### Enable Crosshair
When this toggle is enabled, dashed lines will be visible around the cursor's location.
**Enable Halo Box**
When this toggle is enabled, a halo box will become visible around bounding boxes, with the margin set through the "Halo Box Margin" setting, in pixels. This is a purely cosmetic/visual change and has no effect on the annotations.
See also [Bounding Box -> Halo Box](/labeling/labeling-tools/tools/bounding-box#how-to-add-a-halo-around-bounding-boxes).
**Opacity**
Alter the opacity of annotations on screen.
#### Brightness, Contrast, Saturation
The *brightness, contrast,* and *saturation* sliders apply to the asset on screen.
#### Box Border Thickness
Changes the thickness of bounding boxes.
#### Point Radius
Change the size of point annotations.
#### Invert
Invert the colors of the asset.
# Audio Labeling Editor
Overview of the Audio Labeling Editor in Ango Hub
Ango Hub provides a labeling editor with which audio files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub’s audio labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The audio labeling editor supports audio assets with the following file extensions:
* .mp3
* .wav
* .ogg
### Supported Labeling Tools
The audio labeling editor supports following labeling tools:
**Tools**
* Entity
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### Audio Interface Elements
#### Playback bar
* allow you to move the playhead to the beginning of the audio file, five seconds back, play/pause, five second forward, and to the end of the file.
* The button becomes active when a segment is selected, and allows you to start playback starting from the beginning of the selected segment. (Pressing R on your keyboard will also enable this.)
* indicates the position of the playhead / the total duration of the audio file.
* When a segment is selected, allow you to move to the next/previous segment.
* If *Auto Scroll* is enabled, the list of segments below the playback bar will move with the playhead, always displaying as selected the currently playing segmentation.
* *Loop* repeats playback continuously. With no segment selected, it loops the entire audio asset. With a segment selected, pressing Play loops that segment; if the playhead is outside the segment, playback begins at the segment's start. Selecting a different segment while playback is active changes the loop target, and clearing the selection switches back to looping the full asset. Press G to turn Loop on or off.
* The icon enables you to pick a playback speed.
* The zoom slider enables you to pick a zoom level. You may also zoom in/out with the scroll wheel. Both types of zoom always zoom towards the current location of the playhead.
* The volume slider allows you to change the playback volume.
* Your playback speed and volume remain selected when you move to another audio asset using the navigation arrows, *Submit*, or *Skip*.
* The *Show Spectrogram* button displays a spectrogram below the waveform. The spectrogram uses the same timeline as the waveform, so playback, seeking, zooming, scrolling, and segment editing stay aligned across both views.
* For audio assets with more than one channel, the playback bar will also show a *Show Separate Channels* button. Clicking it displays each audio channel as a separate waveform, making it easier to inspect left and right channels independently.
* For stereo audio assets, the playback bar also includes a *Mixer* button. From the Mixer, you may move the stereo balance between *Left*, *Center*, and *Right*, or set an intermediate value with the slider. This allows you to lower one side of the audio and focus on the other channel while labeling.
#### Spectrogram View
The spectrogram view lets you inspect the frequency content of an audio asset while labeling. It appears under the waveform and can be shown or hidden from the playback bar.
When the spectrogram is visible:
* The waveform remains visible above it, unless you resize the panels to give the spectrogram the full available height.
* It uses a grayscale color scale to make differences in audio intensity easier to inspect.
* The frequency scale stays pinned on the left while you zoom and scroll through the audio. Select the settings icon in this gutter to choose the frequency range you want to inspect and adjust the analysis window. These settings are saved separately for each project.
* Hovering over the spectrogram shows the timestamp and approximate frequency under the cursor.
* Audio entity segments stay visible on both the waveform and spectrogram views.
To resize the waveform and spectrogram areas, drag the divider between them. The divider snaps between preset heights, letting you keep the waveform full size, shrink it, or hide it while keeping the spectrogram visible.
#### Top Bar
Click on the "Model Plugins" button to open a list of [model-type plugins](/plugins/plugin-developer-documentation) available in your organization.
If a [default preset](/plugins/introduction-to-plugins/plugin-configuration-and-preset-management) has been set for the plugin, the button will be clickable, and clicking on it once will run the plugin on the current asset with the default settings.
Clicking on the three dots next to the model name will open the Model Run Dialog, allowing you to customize the plugin's run settings and to run the plugin on the current asset with settings of your choice.
#### Segment List and Layered Text View
Use the view selector to switch between *Segment List* and *Layered Text View*. Ango Hub remembers the selected view after you reload the task.
The *Layered Text View* button is also available in the editor's right-side toolbar.
The *Segment List* allows you to see, at a glance, each segment currently present in the audio file. If segments have [nested classifications](/labeling/labeling-tools/tools/nested-classifications), those classifications will be displayed in the segment list. This can, for example, be a text classification, a dropdown, radio, or any other available [classification type](/data/importing-and-exporting-annotations/exporting-annotations/ango-export-format/asset/task/classifications).
*Layered Text View* organizes segments into numbered tiers. Select a tier to focus the waveform on its segments, then use the edit icon in a tier box to update its selection text. To move one or more selected segments to another tier, right-click them and choose *Change Tier*. New segments are added to the selected tier.
Clicking on the three dots next to each class's name will open a menu where you can open [object-specific issues](/core-concepts/issues#object-level-issues), or copy the unique Object ID of that object.
The `id` column displays a unique ID for the segment. These are the first three characters of the Object ID.
The *Classifications* column displays nested classifications of the class belonging to the segment, if any.
The *Start* and *End* columns display the timestamp when the segment starts and ends in the audio file. The *Duration* displays the duration of the segment.
#### Editing segment times precisely
Select one segment to edit its timing with millisecond precision. When exactly one segment is selected, its *Start* and *End* values in the *Segment List* become editable. Enter a value as seconds, `m:ss.mmm`, or `mm:ss:mmm`, then press Enter or click outside the field to save it. Press Tab to move from *Start* to *End*, or Escape to discard the edit.
While editing a *Start* or *End* value, use the Up and Down arrow keys to adjust it by 10 milliseconds. Hold Shift to adjust it by 100 milliseconds, or Ctrl (Command on macOS) to adjust it by 1 millisecond.
You can also move the entire selected segment without changing its duration:
* Alt (Option on macOS) + Left/Right Arrow moves it by 10 milliseconds.
* Alt/Option + Shift + Left/Right Arrow moves it by 100 milliseconds.
* Ctrl + Shift + Left/Right Arrow on Windows and Linux, or Command + Shift + Left/Right Arrow on macOS, moves it by 1 millisecond.
Hold a shortcut to move the segment continuously. If an overlap-prevention setting is enabled, the segment stops at the nearest applicable segment boundary.
If you hide an audio entity from the *Objects* panel using the eye icon, the same entity is also hidden from the waveform and from the *Segment List*. Hidden entities are skipped when using the next and previous segment controls. Showing the entity again from the *Objects* panel makes it appear in the waveform and *Segment List* again.
You may additionally delete the segment by clicking on the trash can.
## How to Annotate Audio
From the playback buttons on the bottom bar, start and stop playback of the audio file as necessary.
{% hint style="info" %}
If you get no sound, make sure that:
* Your system volume is on
* The volume slider on the bottom bar is not all the way to the left
* You have selected the right output device on your computer
{% endhint %}
From the *Tools* section on the left sidebar, select an *Entity* labeling tool, marked with an underlined *A* icon.
Click on the waveform where you’d like the annotation to start. Keep the left mouse button pressed and drag until where you’d like the annotation to end. Release the left mouse button.
You can change the start and end points of the annotation by selecting it with left-click, then dragging on one of the ends. You can drag the entire annotation by, after selecting it, clicking and dragging from the middle of the label.
To prevent audio entity annotations from overlapping, open the editor settings from the bottom-right corner of the editor. Choose *Prevent All Region Overlap* to prevent any overlap, or *Prevent Same-Class Region Overlap* to prevent overlap only between segments of the same class. Ango Hub remembers the selected overlap setting after reload.
When an overlap-prevention setting is enabled, newly drawn audio entity annotations cannot overlap according to the selected rule. If part of a new annotation would overlap an applicable existing annotation, Ango Hub shortens or moves its boundary to the nearest available range. If the new annotation is fully inside an applicable existing annotation, it is not created. Existing annotations follow the same rule when you edit their start or end points.
### Changing Audio Segment Opacity
To make the waveform easier to inspect behind audio entity segments, open the editor settings from the bottom-right corner of the editor and adjust the *Opacity* slider. The setting ranges from 0 (transparent) to 1 (opaque) and updates segment fills immediately in the waveform, separate-channel, and spectrogram views.
Selected segments remain slightly more visible than other segments. Issue regions and pending issue highlights keep their own visibility and are not changed by this setting.
Press `Y` to switch the opacity between 0 and the last non-zero value.
If the labeling tool has nested questions, right-click on the label and click on the menu that appears to see and answer the nested questions.
If classification questions are present, you may answer them from the *Questions* panel on the left sidebar.
## Opening Spot Issues
Besides issues about the asset as a whole and about individual objects, as outlined in the docs page on [issues](/core-concepts/issues), in audio assets you may also open 'spot' issues about specific timestamps or ranges of timestamps.
To do so, click on the *issue bubble* icon at the top-right of the screen. Then, either click once on the audio where you would like to open the issue, or click and drag on the audio waveform over the section of audio related to the issue:
## Merging two entities
Select multiple entities by holding Shift + clicking. Then, click on the "Merge" button that appears:
If the entities had attributes (nested questions):
* For all attribute types other than Text, only the attributes of the first entity are retained. (The first entity is the entity that starts first in the audio)
* For the Text attribute type, the text of the second entity is appended to the text of the first entity. (First and second refer to the timestamp at which the entities start in the audio.
## Keyboard Shortcuts
A list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the [top bar](/labeling/labeling-editor-interface#top-bar):
### Further reading
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# Image Labeling Editor
Overview of the Image Labeling Editor in Ango Hub
Ango Hub provides a labeling editor with which image files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub’s image labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The image labeling editor supports image assets with the following file extensions:
* .jpg
* .jpeg
* .png
* .tif
* .tiff
* .bmp
### Supported Labeling Tools
The image labeling editor supports following labeling tools:
**Tools**
* Bounding Box
* Rotated Bounding Box
* Polygon
* Polyline
* Segmentation
* Point
* Circle
* Brush
* Skeleton
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### Toolbar
#### Zoom Buttons
The three buttons allow you to zoom in, zoom out, and fit the asset to screen, respectively.
Scrolling with the mouse wheel will zoom in and out of the asset. Double-clicking will zoom in on the zone that’s been double-clicked.
The zoom percentage displayed is relative to the actual size of the image. This means that an image displayed at a 100% zoom rate is displayed in its original size.
#### Mode Switch
If the asset is multi-image, you can switch between Grid and Carousel modes here. [Read here for more on importing and annotating multi-image assets on Ango Hub](/data/importing-assets/bundled-assets/importing-multiple-images-in-one-asset-grid-or-carousel).
#### Rotate Images
The above buttons will rotate the image by 90 degrees counterclockwise or clockwise. This has no effect on the final export.
## How to Annotate Images
From the *Tools* panel on the left sidebar, select a supported labeling tool. Then, follow the instructions found on each tool's docs page.
If no tools are present in the project, only answer the questions in the *Classifications* panel.
### Nested Questions and Classifications
If the labels have nested questions, click on a label to select it, then right-click on it and open the the menu item that appears to see and answer the nested questions.
If classification questions are present, you may answer them from the *Classifications* panel on the left sidebar.
## Quick Settings Menu
The following is a list of quick settings unique to Ango Hub's image labeling editor. For an explanation of the settings common to all data types, visit the [*Labeling Editor Interface*](/labeling/labeling-editor-interface) docs page.
**Hide Segmentation Points**: when enabled, segmentation points are disabled, improving performance.
**Hide Unknown Objects**: when enabled, unknown objects (annotations without categorization, for example, annotations created with OCR or Lung Detector) are not shown. They will still appear in the export.
**Image Smoothing Enabled**: when enabled, images will be smoothed out so that individual pixels are not discernible.
**Opacity**: sets the opacity of the annotations.
**Point Radius**: sets the radius of point annotations.
**Border Thickness**: sets the thickness of the borders of bounding boxes.
**Invert**: inverts the colors of the image.
**Brightness**: sets the image's brightness.
**Contrast**: sets the image's contrast.
## Cloning Annotations
In [multi-image assets](/data/importing-assets/bundled-assets/importing-multiple-images-in-one-asset-grid-or-carousel), you can quickly clone the annotations present in one page to other pages.
To do so, navigate to the page containing the annotations you wish to clone. Then, click on the three dots at the top-right of the screen, and click on "Clone":
The 'Clone' dialog will appear:
From here, you may choose whether to clone all of the annotations in the current frame or only the annotations being selected.
If you pick *Clone All*, you will see the following checkboxes:
Select the type of annotations that you wish to clone among tools, classifications, and relations. Then, select the frames to which you wish to clone the annotations. In the example above, the objects and relations in the frame 1 will be copied to frames 2-10 included.
Existing annotations in frames 2-10 will not be deleted or overwritten.
Click on 'Clone' to clone the annotations.
## Keyboard Shortcuts
A full list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the top bar:
## Further reading
{% content-ref url="/pages/-MjiniaaX8Bc-1CsxFDT" %}
[Audio Labeling Editor](/labeling/labeling-editor-interface/audio-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# Video Labeling Editor
Overview of the Video Labeling Editor in Ango Hub
Ango Hub provides a labeling editor with which video files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub’s video labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The video labeling editor supports video assets with the following file extensions:
* .mp4
* .webm
* .mov
* .mkv
{% hint style="warning" %}
Because `.mov` playback support can vary by operating system and browser, Ango Hub cannot guarantee that every `.mov` file will play correctly on every device. When possible, prefer `.mp4` files encoded with H.264.
{% endhint %}
### Supported Labeling Tools
The video labeling editor supports following labeling tools:
**Tools**
* Bounding Box
* Rotated Bounding Box
* Polygon
* Polyline
* Segmentation
* Point
* Circle
* Skeleton
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### Playback Bar
The back and forward arrows allow you to move backwards and forwards one frame at a time, or to the beginning/end of the video. The Play  button starts and stops playback. The slider allows you to move between frames by clicking on the playhead and dragging it to find the frame you need.
Turn on *Loop* from the playback controls to restart the video automatically whenever it reaches the end. You can also press G to turn Loop on or off. The Loop button is highlighted while looping is enabled.
Video Loop control
{% hint style="info" %}
When dragging with the slider, the frame selected will only be loaded when you release the left mouse button.
{% endhint %}
You may also navigate to a specific frame by typing its number and pressing Enter or clicking outside of the text field:
The volume button allows you to change the volume of the audio when playing the file back. The "1x" button allows you to change the playback speed. Playback speed can be set from 0.1x to 10x, in 0.1x increments.
Your playback speed and volume remain selected when you move to another video asset using the navigation arrows, *Submit*, or *Skip*.
#### Video annotation counts
Hover over the *Video annotation counts* button in the top-right toolbar to see the following totals for the current video:
* **Total annotations** counts every frame where an object is visible, including keyframes and interpolated frames.
* **Total keyframe annotations** counts the visible keyframes across all objects.
* **Total number of annotated objects** counts the objects annotated in the video.
The totals update automatically when you add, edit, or remove annotations, change an object's frame range, or mark frames as out of view. Frames marked as out of view are excluded from the annotation and keyframe totals.
To the right, there is a three-dot menu. Clicking it will make the following options appear:
**Annotation length** determines the default segment length when creating an annotation on the video.
**Jump size** lets you customize how many frames to jump backward or forward when using the Shift + Z/C keyboard shortcuts.
#### Videos with Variable Frame Rate
{% hint style="danger" %}
Ango Hub can import and open videos with variable frame rate, but **frame-level annotation on Variable Frame Rate (VFR) videos is not supported**.
{% endhint %}
See [Variable Frame Rate (VFR) Video Compatibility](/data/data-in-ango-hub/variable-frame-rate-videos) for details and recommendations.
In videos with variable frame rate (VFR), the following warning is displayed:
#### Buffering
Ango Hub does not load the entire video in memory right from the start, as that would be computationally expensive, delay loading, and would use unnecessary memory. Instead, Hub loads it chunk by chunk, buffering it.
You can see how much of the video has been downloaded (buffered) by looking at the playback bar as below:
If your download speed is not sufficiently high to smoothly play the video, a warning will appear:
### Timeline
The timeline view allows you to see the annotations throughout the video in a visual way.
When you create a new annotation, a new row will be added to the timeline view. You can click on the row to select its annotation, or click on the annotation to select its row:
Objects belonging to the same class are automatically grouped and collapsed. To see each object, click on the arrow:
Next to each object's class name, the first few characters of its unique Object ID will be visible. Hover over the characters to display and be able to copy to your clipboard the object's ID:
You can change each object row's order in the timeline by dragging and dropping it using its handle:
#### Managing timeline tracks
Each object row in the timeline has a lock button and a three-dot menu with additional track actions:
The lock button locks or unlocks the object track directly from the timeline row.
In the three-dot menu, **Move to top** moves the track to the top of its current class group, and **Move to bottom** moves it to the bottom of its current class group.
**Completed** marks the track green and moves it to the bottom of the object list, helping annotators keep finished tracks separate from tracks that still need work.
The **Track color** swatches let you color-code a track. To return to the track's class color, click **Reset to default color** above the swatches, or click the currently selected swatch again. Locked tracks cannot have their color changed.
Frame-specific classification rows also support move and color actions, but do not show the lock button.
#### Adding a keyframe
When you move the object, or change the answer in a frame-specific classification, a keyframe is added to the row, indicated by a white rhombus on the line:
You may also add a keyframe without editing the object, by clicking on the segment, to the frame where you would like to add the frame, and then on the "Add Keyframe" button .
#### Removing a keyframe
To remove the keyframe, navigate to the frame where the keyframe is located, and click on the row or the annotation of which you'd like to delete the keyframe. A *Remove Keyframe* button will appear. Click on it to remove the keyframe:
#### Stopping an object or classification from appearing
**Manually**
You can mark a certain annotation as being "out of view". To do so, navigate to the frame where the annotation has gone out of view, click on the "Set as start of out of view" button.
To bring the annotation back, navigate to the last frame where the annotation is not in view, click on the segment you wish to bring back, then click on "Set as end of out of view":
Please note that when you mark an annotation out of view, while you cannot see it on the asset anymore, and it does not appear in the "Objects" list, you can still interact with it (for example to delete it, or mark it back in view) by right-clicking on the annotation segment in the timeline.
And even though nested classifications for out-of-view objects do not appear in the "Objects" view, they will appear in the final export, and in the context menu that opens when its segment is right-clicked.
**By marking the start and end of the out of view segment**
1. Navigate to the frame where you'd like the out of view segment to start.
2. Click on the annotation you'd like to mark as out of view.
3. Click on the icon to start the out of view segment.
4. Navigate to frame where you'd like for your out of view segment to end.
5. Click again on the same button. All frames in that annotation in the interval will be marked as out of view.
#### Making annotations appear for shorter/longer
There are two ways.
The first way is to click and drag on the handles on the left/right side of the segment to shrink/extend the annotation duration:
The second way is to click on the segment, then to navigate to the frame where you'd like the segment to start or end, and click on the "Set Annotation Start" (or "Set Annotation End") button:
#### Giving Annotations Names
You may give each annotation its own name by clicking on it and pressing "D" on your keyboard. Alternatively, you may right-click on it, click on the three-dot menu, and click on "Update Description".
Once you update an annotation's description, it will appear on the timeline:
#### Split and Merge Segments
To split a segment into two separate segments, navigate to the frame where you'd like the segments to split and click on the "Split" button that appears:
To merge two segments, click on the first segment, then Shift + click on the second segment. Then, click on the "Merge" button that appears.
#### Removing Segment Keyframes over an Interval
When a segment is selected, hovering over the icon will cause this dialog to appear:
Pick the frame interval where you would like to remove the keyframes in the selected segment and click on "Remove".
## How to Annotate Videos
The following are the labeling tools supported on videos:
* [Bounding Box](/labeling/labeling-tools/tools/bounding-box)
* [Rotated Bounding Box](/labeling/labeling-tools/tools/rotated-bounding-box)
* [Polygon](/labeling/labeling-tools/tools/polygon)
* [Polyline](/labeling/labeling-tools/tools/polyline)
* [Segmentation](/labeling/labeling-tools/tools/segmentation)
* [Point](/labeling/labeling-tools/tools/point)
* [Circle](/labeling/labeling-tools/tools/circle)
* [Skeleton](/labeling/labeling-tools/tools/skeleton)
* [Entity](/labeling/labeling-tools/tools/entity) (only in Waveform View)
From the *Tools* panel on the left sidebar, select a supported labeling tool. Then, follow the instructions found on each tool's docs page, linked to above.
If no tools are present in the project, only answer the questions in the *Classifications* panel.
### Labeling Properties Specific to Videos
#### Frame Interpolation
Between keyframes, objects are automatically linearly interpolated. So for example, if you create a keyframe on frame 1 for an object on the top-left corner of the video, and in frame 100 of the same object being in the bottom-right corner of the video, Ango Hub will automatically fill in the contents of frames between 1-100 with interpolation, having the object smoothly move from one corner to the next.
{% hint style="info" %}
Interpolation is currently only available for the Bounding Box, Polygon, Segmentation, and Point labeling tools.
{% endhint %}
#### Frame-Specific Classifications
When creating classifications such as [radio](/labeling/labeling-tools/classification-tools/radio), [dropdown](/labeling/labeling-tools/classification-tools/single-dropdown), and others, project managers may choose to make those classifications general (e.g., one response per video) or frame-specific (e.g. one response per frame.)
If the classification has been marked as "frame-specific", then it will appear in the timeline and it will behave like a tool-based object with keyframes.
## Annotating Audio Entities in Videos (Waveform View)
{% hint style="info" %}
Certain video formats might not have Waveform View available.
{% endhint %}
You can switch to Waveform View by pressing this button:
In Waveform View, the timeline will be replaced with the video's sound waveform. All tools other than Entity will be disabled. Annotating audio in this view is equal to doing so in the [audio labeling editor](/labeling/labeling-editor-interface/audio-labeling-editor). Please consult the documentation page on the audio labeling editor for more information.
For video assets, Waveform View shows the current frame and total frame count under the time display. The frame readout uses the same 1-based frame numbering convention as the video editor's frame box.
You can switch back to the video labeling editor at any time by pressing on the same button again.
If the waveform appears as a flat line, but you were expecting a rich sound wave, please click on the *Show Soundwave* button.
### Keyboard Shortcuts
A full list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the top bar:
### Further reading
{% content-ref url="/pages/-MjiniaaX8Bc-1CsxFDT" %}
[Audio Labeling Editor](/labeling/labeling-editor-interface/audio-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# DICOM Labeling Editor
Overview of the DICOM Labeling Editor interface on Ango Hub
Ango Hub provides a labeling editor with which single-frame and multi-frame DICOM files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub's DICOM labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The DICOM labeling editor supports DICOM assets with the following file extensions:
* .dcm
Both grayscale and color DICOM images are supported.
### Supported Labeling Tools
The DICOM labeling editor supports following labeling tools:
**Tools**
* Bounding Box
* Polygon
* Polyline
* Point
* Voxel Brush
* Angle
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### DICOM Interface Elements
When you open a DICOM asset, Ango Hub displays the DICOM file in a DICOM-specific image editor. Single-frame DICOM files behave like image assets. Multi-frame DICOM files also include frame navigation controls.
If the asset contains multiple DICOM files, Ango Hub displays them in a multi-view grid.
#### Multi-frame Navigation
For multi-frame DICOM files, use the playback bar at the bottom of the editor to move through frames. You can move one frame at a time, drag the playhead, play through the frames, or enter a frame number.
When multiple DICOM files are displayed at once, frame navigation applies to the active DICOM view. Click a DICOM view to make it active before navigating its frames.
If the active DICOM has more than one frame, a vertical slice navigator is also shown on the left side of that view. Use the slider or the up and down arrows to move through the DICOM's frames.
You may also use the left and right arrow shortcuts to move backward and forward through frames in the active DICOM.
#### Multi-view Grid
In multi-DICOM assets, each DICOM file appears in its own view. The active view is highlighted. Click a view to make it active, or hover over a view while a tool is selected to annotate in that view.
You can rearrange views by dragging them with the handle in the top-left corner of each view. You can also enlarge a single view with the expand button in the top-right corner of that view, then restore the full grid.
While holding Alt (Option) on your keyboard, click DICOM views to select a smaller set of views. When the selected count matches an available layout, a layout button appears. Click it to show only the selected DICOMs in a compact layout. Click *Default* to return to the full grid.
The alignment button at the top of the grid lets you align the displayed DICOM views.
#### DICOM Filenames
By default, the filename of each DICOM is displayed on the bottom-left of each view. You can turn this off from the editor's settings menu by toggling off *Show DICOM Filename*.
### How to Annotate DICOM Files
Annotating DICOMs is similar to annotating images. For multi-frame DICOMs, annotations are created on the current frame of the active DICOM.
The following are the labeling tools supported on DICOMs:
* [Bounding Box](/labeling/labeling-tools/tools/bounding-box)
* [Polygon](/labeling/labeling-tools/tools/polygon)
* [Polyline](/labeling/labeling-tools/tools/polyline)
* [Point](/labeling/labeling-tools/tools/point)
* [Voxel Brush](/labeling/labeling-tools/tools/voxel-brush)
* [Angle](/labeling/labeling-tools/tools/angle)
From the *Tools* panel on the left sidebar, select a supported labeling tool. Then, follow the instructions found on each tool's docs page, linked to above.
If no tools are present in the project, only answer the questions in the *Classifications* panel.
When done, move forward to the next frame or DICOM and repeat the process until all required frames and DICOMs are labeled.
#### Nested Questions and Classifications
If the labels have nested questions, right-click on each label and click on the menu item that appears to see and answer the nested questions.
If classification questions are present, you may answer them from the *Questions* panel on the left sidebar.
## Annotating Multiple DICOM Files at Once
{% hint style="info" %}
For information on how to import multi-DICOM assets, please consult [this docs page](/data/importing-assets/bundled-assets/importing-multiple-dicom-files-to-be-annotated-and-displayed-at-once).
{% endhint %}
### Focusing the Grid
When multiple DICOM files are displayed on screen, you may choose to focus the layout on a smaller number of them.
By default, the filename of each DICOM is displayed on the bottom-left of each view. You can turn this off from the editor's settings menu by toggling off the *Show DICOM Filename* option.
While holding Alt (Option) on your keyboard, click on the DICOMs you would like to focus on. If the selected number of DICOMs matches an available compact layout, a layout button will appear.
Clicking the layout button will display the selected DICOMs in their own, larger layout.
To return to the full grid, click *Default*.
### Copying and pasting annotations from one DICOM to another
You may select a single annotation, or multiple by holding Shift, then use Control + C (⌘ + C on macOS) to copy the objects to the clipboard, even if the objects are on different slices in the same DICOM. You may then paste the new objects to the other DICOM by clicking on the target DICOM and pressing Control + V (⌘ + V on macOS).
Due to a current limitation, all objects in the clipboard will be pasted on the slice (frame) you are currently viewing, regardless of whether they came from different frames.
## Changing Window Levels
Click the *Window* tool in the toolbar to open windowing options for the active DICOM.
You can adjust the window width and level using the sliders or by entering values. You can also choose from presets such as *Abdomen*, *Bone*, *Air*, *Brain*, *Lung*, and *Liver*, or reset width and level to the DICOM's default values.
You can apply a preset to the active DICOM with its keyboard shortcut, even when the Window options are closed:
| Preset | Shortcut | Width | Level |
| ------- | ----------- | ----: | ----: |
| Abdomen | `Shift + 1` | 350 | 40 |
| Bone | `Shift + 2` | 1000 | 400 |
| Air | `Shift + 3` | 1000 | -426 |
| Brain | `Shift + 4` | 100 | 50 |
| Lung | `Shift + 5` | 1400 | -500 |
| Liver | `Shift + 6` | 150 | 30 |
To save your current width and level as a custom preset, click the save icon next to *Preset* and enter a name. You can save up to two custom presets, which are assigned `Shift + 7` and `Shift + 8`. Custom presets are stored for the current project in your browser and are available in both 2D DICOM and NRRD tasks in that project. Delete a custom preset from the dropdown to replace it. Ango Hub does not let you save values that already match another preset.
Preset shortcuts do not activate while you are typing in an input field.
Alternatively, right-click and drag on the DICOM view to change the window width and level directly.
### Smoothing DICOM Images
The *Window* tool also provides two ways to smooth the active DICOM image:
* **Despeckle** reduces isolated pixels that differ sharply from the pixels around them.
* **Block Averaging** smooths groups of neighboring pixels.
Turn on either option, then adjust its *Strength* slider or enter a value. The image updates when you release the slider. Higher values apply stronger smoothing, and you can use both options at the same time.
These smoothing options are available for DICOM images, including color DICOM images. They are not available for NRRD assets.
### Keyboard Shortcuts
A full list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the top bar:
### Further reading
{% content-ref url="/pages/-MjiniaaX8Bc-1CsxFDT" %}
[Audio Labeling Editor](/labeling/labeling-editor-interface/audio-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# Medical Labeling Editor
Ango Hub provides a medical labeling editor, in which annotators can label medical files in the NRRD and NIFTI formats.
{% hint style="info" %}
DICOM files are annotated in the [DICOM labeling editor](/labeling/labeling-editor-interface/dicom-labeling-editor).
{% endhint %}
## Overview
### Supported File Types
The 3D Medical labeling editor supports NRRD and NIFTI assets with the following file extensions:
* .nrrd
* .nii
* .nii.gz
NRRD files must be compatible 3D medical volumes with supported spatial metadata. See [NRRD File Compatibility](/data/data-in-ango-hub/nrrd-file-compatibility) for the exact NRRD files Ango Hub can and cannot open.
### Supported Labeling Tools
The 3D Medical labeling editor supports following labeling tools:
**Tools**
* Bounding Box
* Rotated Bounding Box
* Polygon
* Polyline
* Point
* Voxel Brush
* Angle
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
By default, the *Axial* view will be shown in the top left, the *Coronal* view in the bottom left, and the *Sagittal* view in the bottom right. At the moment, this layout cannot be changed.
A 3D reconstruction of the annotations created so far is shown in the top-right view, the 3D Viewer, when the toggle in its top left is active.
### **Tool/Class List**
You select your class (tool) from the ones created in the project settings in the tool list. Only one class can be selected at a time. The selected class is highlighted in blue.
If the icon is filled in, this means that there are voxels in the current asset which have been painted with the related class.
By clicking on the eye icon next to a class name, you can hide brush traces belonging to the class. Clicking on the three dots and then on the trash can will delete all segmentations from the class.
Once a class is selected, you can annotate on the asset using its functions – brush, pen, threshold, and more, by clicking on the function from the *Function Bar*.
#### Jumping to a class's largest segmentation in all views
Click on the three dots next to the class name and then on Jump. All visible views will jump to the slice containing the highest amount of voxels painted with the class you specified.
#### Voxel Brush Tool Attributes
{% hint style="info" %}
Voxel brush attributes will not appear until at least a voxel has been painted with the voxel brush tool in question. This is to prevent attributes from being set when no object is present.
{% endhint %}
If you have added nested classifications (aka "attributes") to a Voxel Brush class in the project settings, the class will have a downward arrow, and can be expanded:
From this section, you are able to set attributes for the class in this asset.
### **Function Bar**
By default:
* **Crosshair Toggle**: toggles the crosshair on or off.\
Read more about the crosshair [here](#multi-planar-translation-and-crosshair).
* **Reformat**: toggles oblique reformat mode for NRRD assets.\
Read more about Reformat [here](#reformat).
* **Window Options**: Change the window width and level.\
Read more about window options [here](#windowing-options).
* **Smoothing**: Opens the *Smoothing* tool, allowing you to smooth out your annotations.\
Read more about smoothing [here](#smoothing).
* **Island Tools**: Tools related to deleting and changing categories of islands.\
Read more about island tools [here](#island-tools).
* **Threshold Tool**: Paint all pixels with values falling between two values you specify.\
Read more about threshold [here](#threshold).
* **Logical Operators**: Perform logical operation on labeling classes, like invert, subtract, add, copy, and more.\
Read more about logical operators [here](#logical-operators).
* **Fill Between Slices**: read more about this function [here](/labeling/labeling-editor-interface/medical-labeling-editor/fill-between-slices).
* **Level Tracing**: When toggled on, hovering on a voxel will display a trace of the voxels at the same level of the hovered voxel. Clicking confirms the selection.
* **Grow**: When toggled on, click and drag on the asset to perform a "grow" motion. You may alter the sensitivity of this tool from the settings on the left-hand side of the screen. If you wish to limit the effect of the grow tool to a specific Hounsfield unit range, also activate the Threshold function and select your range.
* **Borders**: Toggle whether the brush traces shown should be filled in or not. The change is merely cosmetic and has no bearing on the export.
* **Keypoints**: When enabled, clicking anyhere on the view creates a new keypoint.\
Read more about keypoints [here](#keypoints).
* **3D Bounding Box**: When enabled, clicking and dragging on the asset will create a 3D bounding box. Read more about 3D bounding boxes [here](/labeling/labeling-editor-interface/medical-labeling-editor/3d-bounding-box).
* **Slice Navigator**: Slider allowing you to move between slices in the view. Drag the slider up and down to move between slices, or click on the up and down arrows on the top left of the view. Additionally containing the name of the current view, current slice/total number of slices, and in [assets with reference volumes](/data/importing-assets/importing-reference-medical-data-during-asset-import), whether the currently-viewed volume is a main or a reference one.
Additionally, when a tool is selected:
* **Brush Tool**: Create a segmentation by painting.
* **Pen Tool**: Create a segmentation by drawing a closed loop.
* **Adaptive Brush**: When active, the shape of the brush will act as a "bucket" tool. You may change the sensitivity of this tool from the left-hand side of the screen.
### 3D Viewer
By enabling the 3D toggle, you'll be able to see a 3D reconstruction of the annotations completed so far in the top-right view, including supported medical objects such as points, polylines, polygons, bounding boxes, rotated bounding boxes, and voxel brush traces.
* Rotate the view by dragging with the mouse cursor
* Zoom in and out with the scroll wheel
* Pan with Ctrl + Left Mouse Button
Note that the view does not get updated in real time. You will need to click on the refresh button in the top left of the 3D view to update the view with the latest changes.
By default, the 3D viewer will display axis indicators and a 3D cube around your segmentations. You may turn each off from the 3D viewer's three-line menu.
### Data Probe
When hovering over a voxel in a view with the mouse cursor, the Data Probe, at the bottom left of the screen, will display information about the voxel being hovered on.
**Voxel Location** refers to the location of the voxel, in voxels, assuming voxel 0, 0, 0 is the left-most, front-most, bottom-most voxel in the Coronal view.
**Voxel Category** (if any) refers to the class the voxel belongs to.
**Patient Coordinates** refers to the real world coordinates of the voxel from the indicated plane. For example, R 6.35 signifies that the voxel is 6.35 millimeters from the R (Right) plane.\
The different planes are L (Left), R (Right), A (Anterior), P (Posterior), I (Inferior), S (Superior).
**Hounsfield Value** refers to the Hounsfield density value of the pixel being hovered.
### View Options
Each of the three main views has a three-line menu at the top from which you can access options related to that view.
#### **Display Options**
* **Fit to View** will make the entire currently visible volume fit in the current view.
* **Pin** enlarges the view to cover the entire screen. This can also be accomplished by double-clicking on the view. Double-click again to return to the previous stage.
* **Set as Main** (on reference volumes only). Sets the current reference volume as the main volume to be annotated. This will erase all annotations in the current task.
* **Select view** allows you to change the view for the volume you are currently viewing in the view.
* **Select volume** allows you to switch between your main volume and a reference volume, or to another reference volume. The project manager must have imported references volumes for this feature to work. See the following page for more: [Importing Reference Medical Volumes on Asset Import](/data/importing-assets/importing-reference-medical-data-during-asset-import).
{% hint style="info" %}
Only the main volume can be annotated. Reference volumes are for reference purposes only.
You may set a reference volume to be the main volume using the *Set as Main* button. This will switch the position of the current reference volume with that of the main volume. This, however, will also permanently delete all annotations in the current task.
{% endhint %}
#### Brush Trace Options
* **Clone \[class name] traces...** appears when a class has been selected. This option allows you to clone traces in the current view and in the current slice, belonging to the selected class, to other slices in the same view.\
\
For example, if you have some *Lung* traces in slice number 50 in the axial view you'd like to copy to all slices between 25 and 75, navigate to that slice and from the view options menu, select *Clone Lung traces*... – the following dialog will appear:
Indicate the frames to where you'd like the traces to be cloned and click on *Clone*.
### Quick Settings
You may access a number of quick settings by clicking on the following button on the bottom-right corner of the screen:
## Functions
Whenever you enable a function by clicking on one from the Function Bar, its options will appear in the Active Functions List, located in the bottom left of the screen. This section covers all functions and their options.
For any function, you may collapse its dialog using the chevron on the left:
### Windowing Options
Click the "Window" icon in the function bar to open the windowing options and enable the window adjustment shortcuts detailed in the [Keyboard Shortcuts](#keyboard-shortcuts) section below.
**Window Width** and **Level**: Move the slider, or type in a value to change the window's width and level.
Ango Hub saves the most recently used width and level for each project in your current browser. The values are restored when you reload the editor or open another NRRD task in the same project. In a multi-volume NRRD task, the same values apply to every volume; resetting them uses the main volume's defaults. NRRD and 2D DICOM window values are saved separately.
**Preset**: Ango Hub has a variety of width/level presets, ideal for quickly setting a W/L pair allowing you to highlight certain elements of the view. For example, selecting the *Lung* preset will set W/L in such a way that lungs are highlighted, and so on for all other presets.
You can select a preset from the dropdown or use its keyboard shortcut. Preset shortcuts apply to the active view in any Medical editor layout, even when the Window options are closed.
| Preset | Shortcut | Width | Level |
| ------- | ----------- | ----: | ----: |
| Abdomen | `Shift + 1` | 350 | 40 |
| Bone | `Shift + 2` | 1000 | 400 |
| Air | `Shift + 3` | 1000 | -426 |
| Brain | `Shift + 4` | 100 | 50 |
| Lung | `Shift + 5` | 1400 | -500 |
| Liver | `Shift + 6` | 150 | 30 |
To save your current width and level as a custom preset, click the save icon next to *Preset* and enter a name. You can save up to two custom presets, which are assigned `Shift + 7` and `Shift + 8`. Custom presets are stored for the current project in your browser and are available in both NRRD and 2D DICOM tasks in that project. Delete a custom preset from the dropdown to replace it. Ango Hub does not let you save values that already match another preset.
Preset shortcuts do not activate while you are typing in an input field.
### Multi-planar Translation & Crosshair
The crosshair allows you to gather your bearings in 3D space and to understand where your cursor is positioned relative to all views.
It also allows you to synchronize all views in such a way that they all show the same pixel you are selecting with the crosshair. This is also known as multiplanar translation.
To toggle the crosshair, click on the *Crosshair Toggle*:
Move your cursor over a view while holding *Shift*. You will see that all views will navigate between slices in such a way as to always show the pixel you are hovering over with the crosshair:
When no annotation tool is selected, you can also drag either crosshair line to move only that plane. Dragging a horizontal line moves it vertically, while dragging a vertical line moves it horizontally. The linked view and slice number update as you drag, and the pointer changes to a move icon over a draggable line.
To hide the crosshair, click on the *Crosshair Toggle* again.
You can still perform multi-planar translation even with the crosshair turned off, just by doing Shift + Move Cursor.
The plane indicators and crosshair lines use consistent colors throughout the editor: Axial is red, Coronal is green, and Sagittal is yellow. Each view shows crosshair lines in the colors of the other two planes, making it easier to identify which plane each line represents.
### Reformat
The Reformat tool allows you to work on oblique planes in NRRD assets. When Reformat is enabled, you can rotate the viewing plane from one view and have the other views update to match the reformatted plane.
To use Reformat, click the *Reformat* button in the function bar and make sure no annotation tool is selected. Move the pointer over a crosshair line in the Axial, Coronal, or Sagittal view:
* Drag the inner or middle section of the line to move that plane.
* Drag the outer section, near either end of the line, to rotate the plane. The other views update to show the corresponding oblique plane.
The pointer changes to show whether dragging will move or rotate the plane. Clicking or dragging away from the crosshair lines does not move or rotate them.
Reformatted views are marked with an *R* next to the plane name. The crosshair lines remain perpendicular and the three views remain aligned even when the dataset uses different voxel spacing along each axis.
When Reformat is active, you can use Brush and Pen annotations on the reformatted plane. Point, Polyline, and Polygon annotations created on a reformatted plane store their reformat position, so selecting those annotations from the object list later restores the same oblique view.
Rotated Bounding Box annotations are not displayed or editable while Reformat is active.
In the Ango export, Point, Polyline, and Polygon annotations created on a reformatted plane include an `nrrdReformat` object with the active plane, angle, and center voxel used for that oblique view. The annotation coordinates themselves are still exported as 3D voxel coordinates, with patient-space coordinates where available.
Voxel Brush annotations do not include per-annotation reformat metadata in the JSON export. They are exported as the final 3D segmentation volume through `medicalBrushDataUrl`, the same as other medical brush annotations. 3D Bounding Box annotations are also exported as their normal 3D coordinates and do not include `nrrdReformat`.
To leave Reformat mode and return the views to their regular orthogonal planes, click the *Reformat* button again.
### Brush Tool
After selecting a class, activate the Brush tool by clicking on its icon: .
**Brush Size**: Slider to adjust the size of the brush tool.
**Brush Mode**: Selector to pick whether the brush/pen should add (Paint) or whether it should erase (Eraser.)
**Brush Type**: Pick between 2D (flat) and 3D (spherical) brush.
**Editable Area**: When using the Threshold tool, choose where the threshold operation can write voxels. Select *Everywhere* to apply the threshold to the whole asset, or select a Voxel Brush class to restrict the operation to voxels already assigned to that class. When the threshold is restricted to a class, *Overwrite* or *Overlap* must be enabled for the threshold operation to change existing voxels.
**Overwrite Toggle**: When active, when you paint with the brush/pen over pre-existing traces, the traces that have been covered will be deleted and overwritten with the new traces.
**Overlap Toggle**: When active, your brush traces will overlap pre-existing traces, without overwriting (deleting) them. This way, you can segment the same voxel with multiple classes.
To use the *Medical Brush* tool, simply click with the left mouse button where you'd like to place the annotation. You can click and drag if necessary, and you may use either the slider mentioned above or the keyboard shortcuts mentioned below to change the size of the brush and other settings.
#### Brush in Threshold Range
You may choose to limit where you can paint with the brush to a threshold range you specify.
To do so, while the Brush tool is active, click on the "Threshold" icon and pick a range. As long as the threshold tool is active, the brush will only paint within the specified threshold range. To allow the brush to paint anywhere, close the Threshold tool by clicking on its icon again.
#### Sphere Brush
In the brush settings, switching the brush type to 3D will make the brush a sphere. Clicking anywhere on the asset will create a sphere, with the center on the cursor and with a diameter equal to the brush size.
{% hint style="info" %}
Visually, the sphere might not always look spherical. This is because often, voxels have different sizes in the different dimensions.
For example, if the voxels in the asset are taller than they are larger, the sphere will look oblong (e.g. taller) than a normal sphere.
This is normal, expected, and intended.
{% endhint %}
### Threshold
The Threshold tool allows you to paint pixels with a value falling between a range you specify.
To activate the threshold tool, first select a Medical Brush tool from the toolbar on the left, then click on the *Threshold* button. From the dialog that pops up, specify two values using the slider. The view will update and show in a pulsing light pink the pixels that would be painted with the values currently selected:
{% embed url="" %}
From the *Editable Area* dropdown, select where the threshold operation should apply. To apply it to the whole asset, select *Everywhere*. When you wish to finalize the threshold annotation, click on *Apply*. The highlighted pixels will be painted over with the tool previously selected.
#### Threshold Area
You may have threshold only apply to an area where you have already created brush traces.
For example, if we were to have painted an area like so with the 'Green' brush:
We may have Hub only run threshold, with another tool, inside of the green area. This can be useful if, for example, we wished to annotate the bones in the selected area.
To do so, after you have painted the area where you'd like to run threshold, select a different brush, then click on the 'Threshold' icon (). From the dialog that appears on the left side of the screen, select the threshold range. Then, from the *Editable Area* dropdown, select the brush you have used to draw the initial area.
Ensure the 'Overwrite' or 'Overlap' toggle is on for the brush to run the threshold with, then click on Apply. Threshold will be applied in the area.
### Pen
The Pen tool allows you to draw a region of pixels to paint, similar to the [Segmentation](/labeling/labeling-tools/tools/segmentation) tool for images.
The pen tool is a subset of the Medical Brush tool. To activate the pen tool, first select a Medical Brush tool from the toolbar on the left, then click on the *Pen* button.
With the pen tool selected, draw over the image surrounding the pixels you'd like to paint. When you are done, press N to close the loop. All pixels found inside the loop will be painted.
### Smoothing
-> See the [Smoothing](/labeling/labeling-editor-interface/medical-labeling-editor/smoothing) docs page.
### Logical Operators
Logical operators allow you to perform logical operations (e.g. add, subtract, invert) on brush traces.
Some operations (e.g. *invert*) operate on a single brush class: the one you selected. Others (e.g. *copy, subtract*) operate between two classes of brush traces: the *selected* class (the one you clicked on in the Tools list which is highlighted in blue) and the *target* class (the one you select in the Logical Operators panel).
To activate logical operators, click on a class in the Tools list – this will be your *selected* class. Then, click on the Logical Operators icon in the Function Bar () and select an operation. If required by the operation, select the target tool. Then click on *Apply*.
#### Logical Operations
**Copy.** Removes all existing segmentations of the selected class, and creates new ones copying, and overlapping, the target class.
**Add.** Creates new brush traces belonging to selected class, copying and overlapping the brush traces of target class.
**Subtract.** Wherever traces from selected and target class overlap, segmentations from the target class are subtracted from segmentations of the selected class.
**Intersect.** Deletes all traces of selected class, except where they overlap with target class.
**Invert.** Inverts traces from selected class.
**Clear.** Deletes all traces belonging to the selected class.
**Fill.** Paints all voxels of the asset with the selected brush, overlapping existing segmentations.
### Keypoints
Click on the button to enable placing keypoints on the asset. Clicking anywhere on the asset will place a keypoint with three-dimensional coordinates:
{% @arcade/embed url="" flowId="45V5yqbQlVYd6EXs69bE" %}
As shown in the video, you may double-click on the keypoint's listing under the "Objects" panel to change its name. You may change its coordinates manually by single-clicking on the listing.
### Island Tools
Islands are groups of contiguous voxels.
Island tools are a group of utilities which allow you to remove or change the categories of specific islands in your asset. For example, you may have Hub automatically delete all islands smaller than 1000 voxels.
Read how to use Hub's island tools in the [Island Tools docs page](/labeling/labeling-editor-interface/medical-labeling-editor/island-tools).
## Importing NRRD Segmentations
Ango Hub allows you to import existing NRRD segmentations as pre-labels.
To do so, follow the steps outlined in the page [Importing NRRD Annotations](/data/importing-and-exporting-annotations/importing-annotations/importing-nrrd-annotations).
## Keyboard Shortcuts
Condition
Shortcut
Action
Description
Always
Space + Drag Cursor
Pan
Pans the view.
Always
Ctrl + Mouse Wheel
Zoom
Zooms the view being currently hovered by the mouse cursor.
Always
Shift + Move Cursor
Multiplanar Translation
Navigates between slices in all other views to show the pixel you are hovering over in all views.
Always
Ctrl + Z
Undo
Undo the last action.
Always
Ctrl + Shift + Z
Redo
Redo the last undone action.
Always
Mouse Wheel
Navigate Slices
Move up and down between slices.
Always
Middle Mouse Button
Pan
Pans the view.
Always, except while typing in an input field
Shift + 1–8
Apply Window Preset
Applies the corresponding predefined or saved custom window/level preset to the active view.
When no labeling tools are selected
Click and Drag
Pan
Pans the view.
When no labeling tools are selected
Double Click
Full Screen
Switch between four-view and single-view modes.
When a labeling tool is selected
Shift + Mouse Wheel
Change Tool Size
Changes the size of the tool.
When a labeling tool is selected
W and Q
Change Tool
Move up and down in the tool list, changing the currently active tool.
When the "Window" icon is selected
Drag cursor up/down
Change Brightness
Dragging the cursor downwards will increase the brightness, dragging it upwards will decrease it.
When the "Window" icon is selected
Drag cursor left/right
Change Contrast
Moving the cursor to the left while holding Ctrl will increase contrast, moving it to the right will decrease it.
When the "Pen" tool is selected
N
Close Pen Trace
Closes the loop drawn with the pen and paints the pixels found inside it.
## NRRD Exports
Exports obtained from the NRRD editor work differently from exports obtained with our other editors.
While you can obtain the export [the same way](/data/importing-and-exporting-annotations/exporting-annotations), the brush data is not contained directly within the text export itself, instead, the text export contains a link to a NRRD file containing your annotations.
A sample NRRD export looks like this:
```json
[{
"asset": "https://asset.url/asset.nrrd",
"externalId": "RegLib_C01_1.nrrd",
"metadata": {},
"labeledAt": "2023-02-01T10:14:35.584Z",
"status": "Labeled",
"labelDuration": 8152266,
"consensus": "",
"tasks": [
{
"completedBy": "NAME SURNAME",
"completedAt": "2023-02-01T10:14:35.584Z",
"duration": 8152266,
"isSkipped": false,
"review": {
"status": "Todo",
"completedBy": [],
"completedAt": null,
"duration": 0,
"isSkipped": false
},
"status": "Completed",
"updatedBy": "NAME SURNAME",
"updatedAt": "2023-02-01T10:14:40.947Z",
"isBenchmark": false,
"benchmark": "",
"issues": [],
"taskId": "63d0e32ad8157d000eace053",
"medicalBrushDataUrl": "https://angohub-public-assets.s3.eu-central-1.amazonaws.com/63d0e32ad8157d000eace053.zip",
"objects": [],
"classifications": [],
"relations": []
}
],
"batches": []
}]
```
The brush data is contained in the URL linked to in the `medicalBrushDataUrl` property. Once you enter the URL, download the .zip and unzip it, you will receive a .nrrd file containing your annotations. For example, this is what the open NRRD would look like, with simple circular annotations:
# 3D Bounding Box
The 3D Bounding Box feature allows you to create 3D bounding boxes in medical assets.
## Creating a 3D Bounding Box
In the medical labeling editor, click on a "Bounding Box"-type class in the "Tools" section on the left.
Click and drag with the left mouse button on the asset wherever you would like to create the bounding box.
Release the left mouse button to finalize two dimensions of the bounding box:
A 3D bounding box will be created with the two dimensions you specified, and the other dimension being, initially, 1px:
To edit the 3D bounding box's dimensions, click and drag on the editing points on the bounding box, in any view.
## Extending a 3D Bounding Box to Another Slice
You can extend an existing 3D bounding box to another slice while keeping its size and position in the current view.
Select the 3D bounding box, navigate to the slice where the bounding box should end, and press `V`. You may also click the *Snap BB Edges* button in the top toolbar while the bounding box is selected.
## Changing the class to which a bounding box belongs
Right-click anywhere on the 3D bounding box and click on the three dots.
Then, from the "Change Category" sub-menu, pick the class you'd like to change the 3DBB to. The 3D bounding box will be assigned to that class.
## Adding a Description / Name to a Bounding Box
You may add a manual text description or name to a bounding box by clicking on it and pressing "D" on your keyboard.
# Fill Between Slices
Fill Between Slices allows you to fill in the content between brush traces with a single click.
## How to use Fill Between Slices
1. Ensure no classes are selected.
2. Click on the *Fill Between Slices* icon in the top bar. () Ango Hub will calculate the areas that will be filled. The areas to be filled will be displayed in a pink color.
3. Click on "Apply" if you wish to apply the changes, or exit the Fill Between Slices function to cancel.
While Ango Hub is calculating or applying Fill Between Slices, the Fill Between Slices and Apply controls show a loading state and cannot be clicked again. Wait for the operation to finish before changing slices or continuing to edit.
# Island Tools
Islands are groups of contiguous voxels.
Island tools are a group of utilities which allow you to remove or change the categories of specific islands in your asset. For example, you may have Hub automatically delete all islands smaller than 1000 voxels.
## Keep Largest Island
Deletes all islands in the selected category, keeping only the largest one. But if the largest island is smaller than the given voxel count value, it too will be removed.
### Usage
1. Select from the 'Tools' list the category from which you'd like to remove (delete) the small islands and keep the largest one.
2. Click on the 'Islands' icon (). A dialog will appear on the left side of the screen:
3. From the selector that appears on the left side of the screen, select the "Keep largest island" tool.
4. In the "Minimum Size" number input, enter the minimum size islands must be to be kept. If even the largest island in the given category is smaller than the given voxel count value, it will be deleted.
5. Click on "Apply".
## Remove Small Islands
Removes all islands in the selected category, below a given voxel count value.
### Usage
1. Select from the 'Tools' list the category from which you'd like to remove (delete) the small islands.
2. Click on the 'Islands' icon (). A dialog will appear on the left side of the screen:
3. From the selector that appears on the left side of the screen, select the "Remove small islands" tool.
4. In the "Minimum Size" number input, enter the minimum size islands must be to be kept. All islands smaller than the area you input here will be deleted.
5. Click on "Apply".
## Keep Selected Island
In a given category, deletes all islands except the one clicked on.
### Usage
1. Select from the 'Tools' list the category from which you'd like to remove (delete) the islands.
2. Click on the 'Islands' icon (). A dialog will appear on the left side of the screen:
3. From the selector that appears on the left side of the screen, select the "Keep selected island" tool.
4. From any one of the 2D views (Axial, Coronal, or Sagittal) click on the island you'd like to keep.
## Remove Selected Island
In a given category, removes the island clicked on.
### Usage
1. Select from the 'Tools' list the category from which you'd like to remove (delete) the island.
2. Click on the 'Islands' icon (). A dialog will appear on the left side of the screen:
3. From the selector that appears on the left side of the screen, select the "Remove selected island" tool.
4. From any one of the 2D views (Axial, Coronal, or Sagittal) click on the island you'd like to remove.
## Add Selected Island
Changes the category of the selected island
### Usage
1. Select from the 'Tools' list the category to which you'd like to change the selected island. For example, if you'd like to change the category of an island from "Bone" to "Brain", you'd select the "Brain" category.
2. Click on the 'Islands' icon (). A dialog will appear on the left side of the screen:
3. From the selector that appears on the left side of the screen, select the "Add selected island" tool.
4. From any one of the 2D views (Axial, Coronal, or Sagittal) click on the island you'd like to change the category of.
# Line (Tape Measure)
The Line feature allows you to create two-point lines in medical assets.
## Creating a Line
In the medical labeling editor, click on a "Polyline"-type class from the "Tools" section on the left.
Click with the left mouse button on the volume wherever you would like to start the line. Then click again where you would like to end it.
To edit the line, click and drag on the editing points on the line.
As lines are 2D, they will only be viewable in the view where you have created them.
### Viewing the Line Measurement
As you hover over the line, or click on it, its real-world measurement in millimeters will become visible:
The Objects panel also shows the measurement next to each line. When multiple lines use the same class, the class header shows their total measurement. If the lines use different units, Ango Hub displays a separate total for each unit.
Measurements and totals update when you move, edit, or delete a line.
## Changing the Class of a Line
Left click on the line to select it, then right-click anywhere on the line and click on the three dots.
Then, from the "Change Category" sub-menu, pick the class you'd like to change the line to. The line will be assigned to that class.
## Adding a Description / Name to a Line
You may add a manual text description or name to a line by clicking on it and pressing "D" on your keyboard.
## Quickly Navigating to a Line
You may quickly navigate to the slice where a line was drawn by holding Ctrl and hovering with the mouse cursor over the line's row in the "Objects" panel on the left-hand side of the screen.
# Smoothing
The *smoothing* tool allows you to smooth out annotations created with a brush tool you specify.
The *Smoothing* tool is a subset of the *Medical Brush* tool. To activate smoothing, first select a Medical Brush tool from the toolbar on the left, then click on the *Smoothing* button:
## Options
Options for the smoothing tool will appear in the bottom left corner of the screen:
**Preset**: Pick here the smoothing type you'd like to apply. Read more about each smoothing preset in the [Smoothing Presets](#smoothing-presets) section.
**Kernel Size:** Pick here the degree to which you'd like to smooth your segmentations, from 1 to 7.\
If you pick a 3x3x3 kernel size for example, the algorithm, for each pixel in 3D space, will consider its 26 immediate neighbors and take the pixel as center of a 3x3x3 cube when applying smoothing.\
In short, the higher the kernel size, the smoother the result (this, however depends on the smoothing type).
While the smoothing tool is active, you may smooth segmentations in the following ways:
* By directly painting over the parts you wish to smooth, effectively functioning as a smoothing brush.
* By clicking on *Apply* in the smoothing settings, applying smoothing to all of the segmentations created with the selected tool in the task.
## Smoothing Presets
Presets are different types of smoothing available on Ango Hub.
Here are details about each:
### Median Smoothing
The median algorithm fills very small holes and smoothes out the general surface of the asset, while keeping the general shape of the asset the same.
Example of median smoothing:
->
The larger the kernel size, the more aggressive the smoothing.
### Opening Smoothing
The opening algorithm never *adds* anything to the asset, only removing extrusions smaller than the selected kernel size. It is useful in removing noise.
The opening algorithm performs erosion (removing noise, shrinking the volume) followed by dilation (expanding the volume while keeping its general shape intact).
Example of opening (click to expand):
->
The larger the kernel, the larger the areas that will be removed. (e.g. more will be eroded.)
### **Closing Smoothing**
The closing algorithm never *removes* anything from the asset, filling holes smaller than the selected kernel size. It is useful in closing small holes or connecting broken parts of an object.
The opening algorithm performs dilation (expanding the volume while keeping its general shape intact) followed by erosion (removing noise, shrinking the volume).
Example of closing (click to expand):
->
The larger the kernel, the larger the areas that will be covered (closed). For example, with a 7x7x7 kernel, an area sized 6x6x6 that is completely empty, yet surrounded by full voxels, will be filled in.
# PDF Labeling Editor
Overview of the PDF Labeling Editor in Ango Hub
Ango Hub provides a labeling editor with which PDF files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub’s PDF labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The PDF labeling editor supports document assets with the following file extensions:
* .pdf
### Supported Labeling Tools
The PDF labeling editor supports following labeling tools:
**Tools**
* PDF
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### PDF Interface Elements
#### Navigation buttons
The arrows allow you to move forwards and backward between pages. You can also directly type in the page number and press Enter to navigate to that page.
Scrolling with the mouse wheel will also navigate between pages in the asset.
With the + and - zoom buttons you can adjust the PDF's zoom level. resets the zoom level and jumps you to the bottom-most annotation in the PDF:
### How to Annotate PDFs
From the *Tools* panel on the left sidebar, select a *PDF* labeling tool, marked with an A enclosed in a square. (If none are present, only answer the questions in the *Questions* panel.)
#### Area
With the *PDF* tool selected, click and drag where you’d like the area to be placed.
After creating the area and selecting by clicking on it, you can change its size by dragging it from its sides and corners. You can drag the entire area by dragging it with the mouse cursor.
You can perform OCR on the contents of the box by selecting it by clicking it, then right-clicking on the box and pressing the *OCR* button.
After having performed OCR, you'll be shown how confident Hub is that the OCR it output is correct:
#### Nested Questions and Classifications
If the labels have nested questions, after selecting a label by clicking on it, right-click on a label and click on the menu item that appears to see and answer the nested questions.
If classification questions are present, you may answer them from the *Questions* panel on the left sidebar.
### Quick Annotation Navigation
Since PDFs can be long, manually navigating to where each label was placed can take time.
It is possible to directly "jump" to the location of each label by keeping Ctrl pressed and hovering over the label from the *Objects* section in the left sidebar. The editor will automatically navigate to the page where the label is, and the label selected will be highlighted.
### Keyboard Shortcuts
A full list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the top bar:
### Further reading
{% content-ref url="/pages/-MjiniaaX8Bc-1CsxFDT" %}
[Audio Labeling Editor](/labeling/labeling-editor-interface/audio-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mk6DvRXUK6-NIvfsXeq" %}
[Image Labeling Editor](/labeling/labeling-editor-interface/image-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# Text Labeling Editor
Overview of the text labeling editor in Ango Hub
Ango Hub provides a labeling editor with which text files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub’s text labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The text labeling editor supports text assets with the following file extensions:
* .txt
### Supported Labeling Tools
The text labeling editor supports following labeling tools:
**Tools**
* Entity
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### Top Bar
Click on the "Model Plugins" button to open a list of [model-type plugins](/plugins/plugin-developer-documentation) available in your organization.
If a [preset](/plugins/introduction-to-plugins/plugin-configuration-and-preset-management) has been set for a plugin, the button will be clickable, and clicking on it once will run the plugin on the current asset with the default settings.
Clicking on the gear icon next to the model name will open the Model Run Dialog, allowing you to customize the plugin's run settings and to run the plugin on the current asset with settings of your choice.
## How to Annotate Text
Please refer to the [Text](/labeling/labeling-tools/tools/entity#text) section in the docs page about the Entity tool for more information on how to use the tool.
Once you have created entities on the text, if classification questions are present, you may answer them from the *Questions* panel on the left sidebar.
### Quickly Jump to Annotation
To quickly jump to a specific annotation, hold the *Ctrl* button (on both Mac and PC) and hover over the annotation you'd like to jump to in the *Objects* list at the bottom left of the screen.
### Display entity class names next to classes
Press Shift + T on your keyboard to toggle displaying class names next to each entity:
### Inspect overlapping entities
Hover over an entity to see its class name. If two or more entities overlap at the cursor position, the tooltip shows all of their class names side by side and highlights the overlapping entities. As you move between overlapping and non-overlapping parts of the spans, the tooltip updates to show the classes under the cursor.
### Display entity content in the *Objects* panel
From the quick settings, enable the *Show Text Object Selection on List* toggle.
The text content of spans will appear in the *Objects* panel instead of the class name for each span:
### Keyboard Shortcuts
A full list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the top bar:
### Further reading
{% content-ref url="/pages/-MjiniaaX8Bc-1CsxFDT" %}
[Audio Labeling Editor](/labeling/labeling-editor-interface/audio-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mk6DvRXUK6-NIvfsXeq" %}
[Image Labeling Editor](/labeling/labeling-editor-interface/image-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mk6GBzBlgn96o0VYkRi" %}
[PDF Labeling Editor](/labeling/labeling-editor-interface/pdf-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# LLM Chat Labeling Editor
Overview of the text labeling editor in Ango Hub
Ango Hub provides a labeling editor with which users can have live conversations with LLMs, and with which existing LLM conversations can be annotated.
The LLM Chat Labeling Editor is opened when a user opens a LLM Chat-type asset. For more information on how to create and/or import such assets, please [read this docs page](/data/importing-assets/creating-and-importing-llm-chat-assets).
{% hint style="info" %}
This article will exclusively go over Ango Hub’s LLM Chat labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported Labeling Tools
The text labeling editor supports following labeling tools:
**Tools**
* Message
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
## Pre-Existing Conversations and Live Conversations
As outlined in [the docs page on creating LLM Chat-type assets](/data/importing-assets/creating-and-importing-llm-chat-assets), there are two possible types of LLM Chat assets: "pre-existing" and "live" conversations.
Pre-existing conversations are conversations which already happened outside of the Ango Hub platform and you import them to be annotated.
Live conversations start empty, and the annotator has a live conversation with your LLM directly from Ango Hub.
### How to chat with your LLM live from Ango Hub
{% hint style="info" %}
This functionality is not available on private cloud and on-premise deployments of Ango Hub.
{% endhint %}
Once the LLM Chat-type assets have been created, if they are empty (of the "live" type), when the user enters the asset, they will be greeted with an empty message UI:
To prompt the LLM, enter your text in the text area labeled *Enter prompt here...* and hit Enter or the Send button. The LLM will receive your prompt and answer.
#### Entering Markdown and LaTeX in prompts
You may use Markdown and, between dollar signs, LaTeX in your prompts. To preview the final, formatted version of your prompt, click on the "Markdown" icon to the left of the prompt text area:
If the model's outputs also contain Markdown or LaTeX, they will automatically be formatted. To view a message's output in plain text (unformatted), click on the *Switch to Raw Text* button below the message:
## How to Annotate LLM Chats on Ango Hub
Regardless of whether the chat is a pre-existing one or a live one, from the *Tools* section on the left-hand side of the screen, select a [*Message*](/labeling/labeling-tools/tools/message)-type tool. Then, click on the chat message you would like to classify. The classifications that have been nested under the Message-type tool will appear, and you will be able to answer them.
Click on *Save* or *Submit* when you are done.
# Markdown Labeling Editor
Overview of the Markdown labeling editor in Ango Hub
Ango Hub provides a labeling editor with which Markdown files can be annotated.
{% hint style="info" %}
This article will exclusively go over Ango Hub’s Markdown labeling interface. Features common to all labeling editors are instead [explained here](/labeling/labeling-editor-interface).
{% endhint %}
## Overview
### Supported File Types
The markdown labeling editor supports text assets with the following file extensions:
* .md
### Supported Labeling Tools
The markdown labeling editor supports following labeling tools:
**Classifications**
* Radio
* Checkbox
* Single-Select Dropdown
* Multi-Select Dropdown
* Single-Select Tree
* Multi-Select Tree
* Text
**Relations**
* Single Relation
* Group Relation
### Top Bar
### Model Plugins
Click on the "Model Plugins" button to open a list of [model-type plugins](/plugins/plugin-developer-documentation) available in your organization.
If a [default preset](/plugins/introduction-to-plugins/plugin-configuration-and-preset-management) has been set for the plugin, the button will be clickable, and clicking on it once will run the plugin on the current asset with the default settings.
Clicking on the three dots next to the model name will open the Model Run Dialog, allowing you to customize the plugin's run settings and to run the plugin on the current asset with settings of your choice.
### KaTeX and LaTeX Rendering
The *Enable KaTeX Rendering* button will render any mathematical LaTeX markup visible in the asset into formatted mathematical notation.
The *Compile as LaTeX* button will take the text content of the current asset and treat it as LaTeX markup, then attempt to compile the document. If compilation was successful, it will display the resulting PDF. Through the option toggles, you may ask Ango Hub to wrap the text with a document environment (i.e. `\begin{document}` and `\end{document}`) and to add a document-type class.
## How to Annotate Markdown
In Markdown assets, only classifications are available.
Answer classifications embedded in the Markdown directly in the asset. Any classification that is not embedded appears in the *Classifications* panel on the left sidebar. When every classification is embedded and there are no objects to display, Ango Hub hides the left sidebar to give the asset more space.
When done, click on *Submit* on the top-right corner of the screen.
### Keyboard Shortcuts
A full list of keyboard shortcuts is available by clicking on the *Keyboard* button on the right side of the top bar:
### Further reading
{% content-ref url="/pages/-MjiniaaX8Bc-1CsxFDT" %}
[Audio Labeling Editor](/labeling/labeling-editor-interface/audio-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mk6DvRXUK6-NIvfsXeq" %}
[Image Labeling Editor](/labeling/labeling-editor-interface/image-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mk6GBzBlgn96o0VYkRi" %}
[PDF Labeling Editor](/labeling/labeling-editor-interface/pdf-labeling-editor)
{% endcontent-ref %}
{% content-ref url="/pages/-Mjifco2debe91gXbTS1" %}
[Labeling Editor Interface](/labeling/labeling-editor-interface)
{% endcontent-ref %}
# Labeling Classes
Reference guide for all annotation tools available in Ango Hub
Labeling classes define the categories and annotation structures used during the labeling process in Ango Hub. Each class is associated with a specific labeling tool and determines how annotators interact with the data while creating annotations.
Ango Hub supports a wide range of labeling tools designed for different data types and annotation needs, such as spatial annotations (e.g., bounding boxes or polygons), classification tools, text-based inputs, and relationship annotations.
## List of Labeling Tools on Ango Hub
### Tools
# Tools
Available tool types in Ango Hub
# Angle
Overview of the Angle labeling tool in Ango Hub
The Angle labeling tool allows you to draw and measure angles in 3D medical assets.
{% hint style="info" %}
The Angle tool is supported across the DICOM and Medical labeling editors.
{% endhint %}
## How to add an Angle tool to your project
From the project’s *Settings* tab, enter the *Category Schema* section.
Click on *Add Category*. From the list that appears, click on *Angle*.
A new row will appear named *Angle*. Click on it to expand it.
**Schema ID**: A unique ID which identifies this class in your project.
**Title:** The name of your class.
**Required:** Whether or not annotators have to add at least one instance of this class in order to submit their annotation task.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the annotation, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### How to Draw an Angle
Click on the image where you’d like the angle vertex to be. Then, click on it or on its edge nodes to change its angle or where it is located.
## Example Export
This is what the Angle tool output looks like in the final export. See [its section on our documentation page on our export](/data/importing-and-exporting-annotations/exporting-annotations/ango-export-format/asset/task/tools#angle) for more.
```json
[
{
"asset": "https://angohub-test-assets.s3.eu-central-1.amazonaws.com/67ed4234d51cdf151a019b3e/assets/72579265-072d-4a8e-893f-1fd3225fd80c.nii.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA3FV7TSYIZFWAQDUV%2F20250902%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20250902T112709Z&X-Amz-Expires=86400&X-Amz-Signature=3d6ee78dce799adc3f2eae6e1d77f0251bf27d63df24f7a54363f308d890d2df&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject",
"assetId": "67ed4269cce65e74f1a24a0d",
"externalId": "medical-compressed-nifti.nii.gz",
"batches": [],
"task": {
"taskId": "67ed4269cce65e74f1a24a1f",
"type": "default",
"stage": "Complete",
"stageId": "Complete",
"updatedAt": "2025-09-02T11:25:46.054Z",
"updatedBy": "example@example.net",
"totalDuration": 0,
"tools": [
{
"angle": {
"vertex": [
121,
112,
60
],
"p1": [
17,
166,
60
],
"p2": [
170,
176,
60
],
"angle": 100
},
"objectId": "5b1e9542a735f72f5edc815",
"classifications": [],
"schemaId": "e83ed8591f14d6c804f9919",
"title": "Angle Tool"
}
],
"classifications": [],
"relations": [],
"brushDataUrl": null,
"medicalBrushDataUrl": null
}
}
]
```
# Bounding Box
Overview of the Bounding Box labeling tool in Ango Hub
The bounding box labeling tool allows you to draw a box around an object or point of interest in an image. It also allows you to perform OCR on the area marked by the box.
{% hint style="info" %}
The Bounding Box tool is supported across the Image, Video, DICOM, and Medical labeling editors.
{% endhint %}
## How to add a Bounding Box tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Bounding Box*.
A new row will appear named *Bounding Box*. Click on it to expand it.
**Schema ID**: A unique ID which identifies this class in your project.
**Title:** The name of your class.
**Required:** Whether or not annotators have to add at least one instance of this class in order to submit their annotation task.
**Targeted OCR**: By enabling this, you will be able to perform OCR and get the OCR results in the exports for the areas highlighted by the bounding boxes belonging to this class.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the bounding box, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### How to Draw a Bounding Box
Click on the image where you’d like the bounding box to start. Click again where you’d like the bounding box to end.
After creating the box, you can change its size by selecting it by clicking it, then dragging on its points. You can drag the entire bounding box by selecting it then dragging it with the mouse cursor.
## How to Add a Halo around Bounding Boxes
To add a visual "halo" around bounding boxes, in the labeling editor, click on the quick settings icon, then on "Enable Halo Box". Enter the Halo Box Margin in pixels.
This halo box is purely visual and has no impact on the final export.
## Perform OCR on the contents of the bounding box
Hub allows you to perform OCR on the contents of bounding boxes, both single and in bulk.
To do so, when adding your bounding box class to your project, enable the *Targeted OCR* toggle as mentioned [in this section](#how-to-add-a-bounding-box-tool-to-your-project).
### On a Single Bounding Box
Once you are in the labeling editor, draw a bounding box over the text you'd like to perform OCR on. Then, right-click on the box and expand the context menu. This is what you should see:
Click on thebutton to perform OCR on the area highlighted by the box. The OCR results will appear in the context menu, together with how confident our OCR module is about the OCR results:
### On Multiple Bounding Boxes
In the labeling editor, press *Shift* and click on the bounding boxes you'd like to perform OCR on. Once you have selected all boxes, right click on any one of them and click on *Run OCR*.
To see the OCR results, right-click on a box and expand the context menu that appears. Alternatively, expand the rows in the *Objects* section in the lower-left corner of the UI.
## 3D Bounding Box Tool in Medical Labeling Editor
{% embed url="" %}
# Brush
Overview of the Brush labeling tool in Ango Hub
The brush labeling tool is a pixel-wise tool to assign pixels on images and videos to certain classes.
{% hint style="info" %}
The Brush tool is supported across the Image labeling editor.
{% endhint %}
## Adding a Brush tool to your project
From the project’s *Settings* tab, enter the *Category Schema* section.
Click on *Add Category*. From the list that appears, click on *Brush*.
A new row will appear named *Brush*. Click on it to expand it.
Give your brush tool a title.
Enable the *Required* toggle if you want to force labelers to create brush traces for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating brush traces using the class you've just created.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing brush traces, on each individual brush trace instance, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### Brush Options
#### After selecting a Brush tool, in Pen mode
When you select a *Brush* tool, a number of options will appear on screen. Here is what will appear once you select a brush:
**Enable Brush Mode**: By default, the Brush will start in the "Pen" tool, allowing you to create traces by drawing their borders. Clicking on this icon will switch you to the Brush mode.
**Enable Pen Mode**: Switches the tool back to Pen mode.
**Overwrite**: Normally, the brush will not paint over existing traces. When this toggle is enabled, it will.
#### After selecting a Brush tool, in Brush mode
**Brush Size Slider**: Change the diameter of the brush. Also available using Shift + Scroll Wheel.
#### After clicking on a brush trace instance
**Eraser**: Enter Eraser mode. In Eraser mode, traces you create will remove pixels from the currently selected instance.
## Using the Brush Tool
Select a *Brush* tool from the *Tools* section in the left sidebar of the labeling editor.
By default, the Brush tool will open in Pen mode. Draw the border of the brush trace you would like to create. Click back on the start dot or press N on your keyboard to close the trace.
Ango Hub will automatically select and highlight the trace you have just created. To continue adding pixels to this instance, keep drawing. To remove pixels from this instance, click on the "Scissors" icon at the top. To close this instance for now, deselect the Brush tool by clicking on it in the Tools list, or by pressing Esc on your keyboard.
If in Brush mode, click and drag on the image where you'd like to draw traces.
### Bucket Fill Mode
While a Brush-type tool is selected, you may switch to "Bucket fill" mode by clicking on the bucket button at the top, or the "V" shortcut.
When Bucket Fill mode is enabled, a single click on the image will select contiguous pixels sharing similar colors.
Increasing the "Threshold" will make it so that more pixels are selected. Decreasing it makes it so that only pixels with very close color values will be selected.
At first, your bucket selection will appear in white (1). You may then click on different parts of the image and change your threshold to edit your selection. Once you are done and you wish to confirm your selection, click on the "Tick" button (2). This will turn your selection into a brush instance.
## Editing Brush Instances
### Adding/Removing pixels to/from the instance
Click on the instance you'd like to edit, then from the top toolbar, select the tool you would like to use to edit the instance, either Pen or Brush. This will enter "Edit Mode" for that instance.
To continue adding pixels to this instance, draw on screen. To remove pixels from this instance, click on the "Scissors" icon at the top and draw where you would like to remove pixels.
Deselect the Brush tool by clicking on it in the Tools list, or by pressing Esc on your keyboard when you are done.
### Changing the class of an existing instance
#### With the mouse
Left-click on an existing instance to select it. Right-click on it to open the context menu. From the three dots, navigate to "Change Category" and select the new class for the instance. See video:
#### With the keyboard
Left-click on an existing instance to select it. On your keyboard, press the Alt (Options) button plus the keyboard shortcut number for the class you would like to change the instance to.
For example, in the GIF above, the class Brush\_3 has a keyboard shortcut number of 3. If I wanted to change the existing trace to that class, I would left-click on it then press Alt + 3.
# Circle
Overview of the Circle labeling tool in Ango Hub
The circle labeling tool allows you to annotate using circles created out of three points.
{% hint style="info" %}
The Circle tool is supported across the Image and Video labeling editors.
{% endhint %}
## Adding a Circle tool to your project
From the project’s *Settings* tab, enter the *Category Schema* section.
Click on *Add Category*. From the list that appears, click on *Circle*.
A new row will appear named *Circle*. Click on it to expand it.
Give your circle tool a title.
Enable the *Required* toggle if you want to force labelers to create at least one circle for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating circles using the class you've just created.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing a circle, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
## Using the Circle Tool
Select a *Circle* tool from the *Tools* section in the left sidebar of the labeling editor.
Click three times on the image to create three points. They will form the circle.
To edit the circle, click on it to select it. Then, drag it by using your mouse cursor, or drag each point to edit the circle.
## Data format
{% hint style="warning" %}
In the final export, circles are represented only by the coordinates of the three points used to create them.
{% endhint %}
Please see the page on the Ango Export Format to learn how the circle appears in the final export:
{% content-ref url="/pages/X5TLHCxEmd25AH9AlKGG" %}
[Ango Export Format](/data/importing-and-exporting-annotations/exporting-annotations/ango-export-format)
{% endcontent-ref %}
# Entity
Overview of the Entity labeling tool in Ango Hub
The entity labeling tool is a multi-functional tool, used in both audio and text labeling to identify entities.
{% hint style="info" %}
The Entity tool is supported across the Audio, and Text labeling editors.
{% endhint %}
## How to add an Entity tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Entity*.
A new row will appear named *Entity*. Click on it to expand it.
Give your entity tool a title.
Enable the *Required* toggle if you want to force labelers to place one. When the toggle is disabled, labelers will be able to save and move to the next asset without creating an entity.
{% hint style="info" %}
By default, when using the Entity tool on text assets, the content of the spans is included in the export for your convenience. If you wish to turn this feature off, please enable the *No Span Content* toggle in the entity class settings.
{% endhint %}
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after placing the entity, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
## How to Label an Entity
### Audio
From the *Tools* section on the left sidebar, select an *Entity* labeling tool, marked with an underlined *A* icon.
Click on the waveform where you’d like the annotation to start. Keep the left mouse button pressed and drag until where you’d like the annotation to end. Release the left mouse button.
You can change the start and end points of the annotation by selecting it by clicking it, then by dragging on one of the ends. You can drag the entire annotation by selecting it, then clicking and dragging from the middle of the label.
### Text
From the *Tools* panel on the left sidebar, select an *Entity* labeling tool, marked with an underlined *A*. (If none are present, only answer the questions in the *Questions* panel.)
#### Character Mode
With the *Entity* tool selected, click and drag on text to highlight the span you’d like to label:
#### Token Mode
Double click on a token to quickly annotate it. Double click and drag to add multiple tokens to a span:
If the labels have nested questions, select a label by clicking on it, then right-click on each label and click on the menu item that appears to see and answer the nested questions.
#### Editing Text Spans
You may click on a text annotation, then drag and drop its ends to modify its length:
# Message
Overview of the Message labeling tool in Ango Hub
The Message labeling tool is used to classify individual messages in [LLM Chat assets](/data/importing-assets/creating-and-importing-llm-chat-assets) in the [LLM Chat labeling editor](/labeling/labeling-editor-interface/llm-chat-labeling-editor).
{% hint style="info" %}
The Message tool is supported across the LLM Chat labeling editor.
{% endhint %}
## How to add a Message tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Message*.
A new row will appear named *Message*. Click on it to expand it.
Give your message tool a title.
Enable the *Required* toggle if you want to force labelers to classify at least one message. When the toggle is disabled, labelers will be able to save and move to the next asset without classifying a message using this class.
In order to ask labelers questions about the individual message, for example, if you want to show a further *radio* after placing the message-type class, you must click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
## How to Label a Message
From the *Tools* section on the left sidebar, select a *Message* labeling tool, then click on the message you would like to classify.
The nested classifications of the class will appear on the right-hand side of the screen. Answer the questions to classify the message.
# Nested Classifications
Overview of the Nested Classifications labeling tool in Ango Hub
Ango Hub allows annotators to answer multiple, nested questions regarding individual labels, relations, or top-level classifications.
In the example above, we ask annotators to draw a polygon. Then, they answer questions related to the polygon they’ve just drawn. Each question’s answer determines which question will be asked next.
Here, if annotators select the “clothing” answer, they will be shown a question asking what type of clothing it is. If then they select “shirt” they will be asked the shirt’s gender, and so on.
You may nest classifications under existing classifications conditionally (i.e. to appear once a user selected a specific answer) or unconditionally (the nested classification appears regardless of what answer the user picks.)
## Setting Up Nested Classifications on Ango Hub
From the project’s *Settings* tab, enter the *Category Schema* section.
Click on the label or relation on which you’d like to ask nested questions. In our example, we expand a “Vehicle” bounding box.
Click on the *Add Classification* button towards the bottom.
If you are adding a nested classification to a tool or relation class, you will be prompted to select the type of classification you need.
If you are adding a nested classification to another classification, you will be prompted to select the classification answer that should be selected by the user for the nested classification to trigger. ([More on classification in Ango Hub](/labeling/labeling-tools/classification-tools).)
If you are nesting a classification under an existing classification, and if you wish for the nested classification to appear regardless of the answer picked by the user, pick the *Any* option from the dropdown:
{% hint style="info" %}
If you are adding a classification to a *Text*-type classification tool, the nested classification will be displayed regardless of the text being input as soon as the user starts entering text.
{% endhint %}
{% hint style="info" %}
The *Add Classification* button will not appear for classifications if no options have been added.
{% endhint %}
A new labeling tool will appear. Fill up its title and description as before, and add options if available.
From here, if you click on *Add Classification* again, you can nest a further question. For example, we can show annotators a free text tool if they answer “Other.”
Enable the *Required* toggle on a nested classification if labelers must answer it before submitting the task. Required conditional classifications are enforced only when their parent answer makes them visible.
### How to Answer Nested Questions
In the [labeling editor](/labeling/labeling-editor-interface), right-click on the annotation where you’d like to answer the nested questions. You can also expand an annotation or relation in the Objects panel and answer its questions there.
A contextual menu will appear. Clicking on it will expand the questions if available.
Relation questions are available wherever relations are supported, except in 3D MSFT projects and the DICOM series editor. When a required relation question is unanswered or an answer does not match its configured format, Ango Hub blocks submission and identifies the relation that needs attention.
# PDF Tool
Overview of the PDF labeling tool in Ango Hub
The PDF labeling tool allows you to annotate text and arbitrary rectangular areas on PDF documents.
{% hint style="info" %}
The PDF tool is supported across the PDF labeling editor.
{% endhint %}
## How to add a PDF tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *PDF*.
A new row will appear named *PDF*. Click on it to expand it.
Give your PDF tool a title and description.
Enable the *Required* toggle if you want to force labelers to create a PDF label for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating the PDF label.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after placing a PDF tool, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
## How to Label with the PDF Tool
From the *Tools* panel on the left sidebar, select a *PDF* labeling tool, marked with an A enclosed in a square.
With the *PDF* tool selected, click and drag where you’d like the bounding box to be placed.
### OCR
{% hint style="info" %}
By default, targeted OCR only detects English. To change, add, or remove which languages should be detected in your project, head over to *Settings -> General* to do so:\
{% endhint %}
#### Single OCR
To perform OCR on the area you've just drawn, click on the area to select it, then right-click it and click on the *OCR* button to perform OCR.
After performing OCR, Hub will display how confident it is that the OCR results are correct.
#### Group OCR
To perform OCR on a group of annotations at once, hold *Shift* then click on each annotation for which you'd like to perform OCR. Right-click on any one of them and click on *OCR.*
# Point
Overview of the Point labeling tool in Ango Hub
The point labeling tool allows annotators to place a point on an object or point of interest in an image.
{% hint style="info" %}
The Point tool is supported across the Image, Video, DICOM, and Medical labeling editors.
{% endhint %}
## The Point Labeling tool in Ango Hub
### How to add a Point tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Point*.
A new row will appear named *Point*. Click on it to expand it.
Give your point tool a title.
Enable the *Required* toggle if you want to force labelers to create a point. When the toggle is disabled, labelers will be able to save and move to the next asset without creating a point label on the asset.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the point, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### How to Place a Point
Select the Point tool from the Tools section on the left sidebar.
Click on the image where you’d like the point to be.
You may change the position of the point by clicking to select it, then dragging it.
## Point Snapping
While drawing a point, if you press E, Hub will highlight the points of other polygons, polylines, and points. By clicking on the other object's points, you can snap your current point to one of them:
By snapping at least two points to the existing polygon and holding Alt (⌥ on macOS) + clicking, you can quickly anchor a series of points to the existing object, as shown in the GIF above.
# Polygon
Overview of the Polygon labeling tool in Ango Hub
The polygon labeling tool allows you to draw a polygon around an object or point of interest in an image.
{% hint style="info" %}
The Polygon tool is supported across the Image, Video, DICOM, and Medical labeling editors.
{% endhint %}
{% hint style="info" %}
If you are deciding between Polygon and Segmentation, see [What's the difference between the Polygon tool and the Segmentation tool?](/other/whats-the-difference-between-the-polygon-tool-and-the-segmentation-tool)
{% endhint %}
## How to add a Polygon tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Polygon*.
A new row will appear named *Polygon*. Click on it to expand it.
Give your polygon tool a title and description.
Enable the **Required** toggle if you want to force labelers to create a polygon on every asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating a polygon.
If you would like for your polygon to appear curved, enable the **Curved** toggle. Please note that this is a purely visual change and does not affect the final export in any way.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the polygon, click on *Add Classification* and add a further question. More[ on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
## How to Draw a Polygon
Click on the image where you’d like the first point of the polygon to be. Click again where you’d like the second point to be, and so on. When you are done, click on the first point again or press on *N* on your keyboard to close the polygon.
While drawing, you may click anywhere with the right mouse button to delete the last point you have placed.
To add a new point after closing the polygon, hold CTRL and click anywhere on an edge to add a point. To delete a point, hold CTRL and right-click on the point you’d like to delete:
To quickly trace a polygon without having to click on each point manually, draw the first point, then hold *Shift* and move your cursor. In the Medical labeling editor for NRRD assets, hold *Alt* instead; *Shift* remains available for crosshair and multi-planar navigation.
## Editing Polygons
After selecting a polygon in the labeling editor, you can use the polygon editing controls to modify it:
* **Auto-Overwrite**: Draw a new polygon that removes overlapping areas from existing polygons.
* **Auto-Subtract**: Draw a new polygon that excludes areas already covered by existing polygons.
* **Auto-Merge**: Select a polygon, then draw an additional area to merge into that polygon.
* **Eraser**: Select a polygon, then draw an area to remove from it.
* **Merge**: Select multiple polygons and click *Merge* to combine them.
* **Subtract**: Select two polygons and click *Subtract* to remove the first selected polygon's area from the second selected polygon.
While drawing with **Auto-Merge** or **Eraser**, the selected polygon stays fixed so the new outline can be completed without accidentally dragging the existing annotation. Press *Esc* to cancel an unfinished helper-tool outline before applying it.
Polygon annotations can contain only one region and no holes. If an editing operation would create multiple separate regions or create a hole, Ango Hub will not apply the operation and will show a warning.
## Edge Sharing
While drawing a polygon, if you press E, Hub will highlight the points of other polygons, polylines, and points. By clicking on the other object's points, you can snap your current polygon's points to them:
You can also share edges with the polygon itself the same way:
By snapping at least two points to the existing polygon and holding Alt (⌥ on macOS) + clicking, you can quickly anchor a series of points to the existing object:
## Nudge Tool
You can apply small changes to your polygons by 'nudging' them.
To nudge a polygon, click on the polygon on the asset, then pick the type of nudge tool you'd like to use:
Nudge Erase will allow you to reduce the size of your polygon while applying small changes. Nudge Draw will add area to your polygon.
Once you have picked your nudging tool of choice by clicking on either the "plus" or the "minus", click and drag around the polygon's edge to refine the edge of your polygon:
# Polyline
Overview of the Polyline labeling tool in Ango Hub
The polyline labeling tool allows you to draw a line on an image.
{% hint style="info" %}
The Polyline tool is supported across the Image, Video, DICOM, and Medical labeling editors.
{% endhint %}
## The Polyline Labeling Tool in Ango Hub
### How to add a Polyline tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Polyline*.
A new row will appear named *Polyline*. Click on it to expand it.
Give your polyline tool a title and description.
Enable the *Required* toggle if you want to force labelers to create a polyline on every asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating a polyline label using this class.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the polyline, click on *Add Classification* and add a further question. More[ on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### How to Draw a Polyline
Click on the image where you’d like the first point of the polyline to be. Click again where you’d like the second point to be, and so on. When you are done, press *N* on your keyboard to finish the polyline.
While drawing, you may click anywhere with the right mouse button to delete the last point you have placed.
To edit a polyline's points, select it by clicking on it, then drag the polyline itself or one of its points.
## Edge Sharing
While drawing a polyline, if you press E, Hub will highlight the points of other polygons, polylines, and points. By clicking on the other object's points, you can snap your current polyline's points to them:
By snapping at least two points to the existing polygon and holding Alt (⌥ on macOS) + clicking, you can quickly anchor a series of points to the existing object, as shown in the GIF above.
# Rotated Bounding Box
Overview of the Rotated Bounding Box labeling tool in Ango Hub
The rotated bounding box labeling tool allows you to draw a box around an object or point of interest and rotate it.
{% hint style="info" %}
The Rotated Bounding Box tool is supported across the Image, Video, and Medical labeling editors. In the Medical editor, it is available for NRRD assets in the standard Axial, Coronal, and Sagittal views.
{% endhint %}
## The Rotated Bounding Box Tool in Ango Hub
### How to add a Rotated Bounding Box tool to your project
From the project’s *Settings* tab, enter the *Category Schema* section.
Click on *Add Category*. From the list that appears, click on *Rotated Bounding Box*.
A new row will appear named *Rotated Bounding Box*. Click on it to expand it.
Give your rotated bounding box tool a title.
**Required:** Enable the *Required* toggle if you want to force labelers to create a rotated bounding box for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating the rotated bounding box.
**Targeted OCR**: By enabling this, you will be able to perform OCR and get the OCR results in the exports for the areas highlighted by the bounding boxes belonging to this class.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the rotated bounding box, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### How to Draw a Rotated Bounding Box
Click on the image where you'd like the top-left corner of the rotated bounding box to be. Click again where you'd like the box to end.
The line and point outside the box indicate the direction of the rotated bounding box. When the *Show Rotated Bounding Box Rotation Handles* editor setting is enabled, they remain visible after you deselect the object, making it easier to see the box's orientation at a glance.
You can change the angle of the box by selecting it, then clicking and dragging on the point located just outside the box. When the object is not selected, visible direction handles are read-only and cannot be used to rotate the box. Turn off *Show Rotated Bounding Box Rotation Handles* if you do not want unselected rotated bounding boxes to show the direction handle and line.
## Rotated Bounding Boxes in the Medical Labeling Editor
In an NRRD task, select a Rotated Bounding Box class and draw the box in the Axial, Coronal, or Sagittal view. Before rotating the box, resize it from one of the other standard views to set its depth. You can then return to the view where you created it to move, resize, or rotate it.
The labels next to the box show its physical dimensions, based on the medical volume's real-world spacing. Its rotation is also shown next to the box. Changes to the box are reflected in the other standard views and persist when you save and reopen the task.
Selecting a rotated bounding box from the Objects list navigates the Axial, Coronal, and Sagittal views to the box. Selecting it directly in a view leaves the current slices unchanged.
The 3D Viewer displays rotated bounding boxes with their edges and center lines. Refresh the 3D Viewer after editing a box to see the latest geometry.
To add pre-existing rotated bounding boxes while importing NRRD assets, see [How to Import Rotated Bounding Boxes into NRRD Assets](/data/importing-and-exporting-annotations/importing-annotations/importing-nrrd-annotations#how-to-import-rotated-bounding-boxes-into-nrrd-assets).
{% hint style="warning" %}
Rotated bounding boxes are not displayed or editable in reformatted views.
{% endhint %}
## Perform OCR on the contents of the bounding box
Hub allows you to perform OCR on the contents of rotated bounding boxes, both single and in bulk.
To do so, when adding your rotated bounding box class to your project, enable the *Targeted OCR* toggle as mentioned in the previous section.
{% hint style="warning" %}
Targeted OCR will only work if the rotation handle is at the **top**.
{% endhint %}
### On a Single Rotated Bounding Box
Once you are in the labeling editor, draw a bounding box over the text you'd like to perform OCR on. Then, right-click on the box and expand the context menu.
Click on thebutton to perform OCR on the area highlighted by the box. The OCR results will appear in the context menu, together with how confident our OCR module is about the OCR results:
### On Multiple Rotated Bounding Boxes
In the labeling editor, press *Shift* and click on the rotated bounding boxes you'd like to perform OCR on. Once you have selected all boxes, right click on any one of them and click on *Run OCR*.
To see the OCR results, right-click on a box and expand the context menu that appears. Alternatively, expand the rows in the *Objects* section in the lower-left corner of the UI.
# Segmentation
Overview of the Segmentation labeling tool in Ango Hub
The Segmentation labeling tool allows you to segment images and create complex polygons with holes, with support for merging and subtracting polygons from one another.
{% hint style="info" %}
The Segmentation tool is supported across the Image and Video labeling editors.
{% endhint %}
{% hint style="info" %}
If you are deciding between Polygon and Segmentation, see [What's the difference between the Polygon tool and the Segmentation tool?](/other/whats-the-difference-between-the-polygon-tool-and-the-segmentation-tool)
{% endhint %}
## How to add a Segmentation tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Segmentation*.
A new row will appear named *Segmentation*. Click on it to expand it.
Give your segmentation tool a title and description.
Enable the *Required* toggle if you want to force labelers to create a segmentation for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating the segmentation.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the segmentation, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
## Using the Segmentation Tool
Click on a segmentation tool from the *tools* section on the left of the screen.
### Manually
Click on the image where you’d like the segmentation to start. Click again somewhere else to create a new point, and so on:
To create a fine-grained segmentation, keep the left mouse button (LMB) pressed to draw your segmentation:
When you are done with your segmentation, press N to close and finalize it, or click on the initial point as shown in the video above.
### With AI Assistance
Follow the instructions on the page for [Auto Suggestion](/labeling/labeling-ai-assistance/auto-suggestion).
### Cutting Out From / Creating a Hole in a Segmentation
Click on a segmentation on the image, then press on the "scissors" icon that appears. This will switch to "delete" mode. In delete mode, you can create segmentations the contents of which will be removed from any segmentation present on the image at the moment:
### Merging Segmentations
Select two segmentations belonging to the same class by shift-clicking them. The "merge" button will become active. Clicking it will merge the two segmentations. If they overlap, then they will then behave as one single segmentation, which you are able to adjust:
You can also merge segmentations not overlapping. In that case, while they will not behave as one segmentation, they will be semantically grouped together in one instance, and in the export they'll be shown as a single 'object':
### Subtracting Segmentations
By default, you can draw a segmentation over another, and they will overlap without subtracting one another.
If you have overlapping segmentations, Hub allows you make it so that one "cuts" into the other, such that the segmentation found below is cut (subtracted) by the one above, having no overlap in the end.
To do so, select two segmentations, by shift-clicking on them directly, or by shift-clicking on their list item in the *Objects* panel. The first segmentation you choose will be the one that will have no data removed. The segmentation chosen second will have part of it subtracted. Then, click on *Subtract*. You can see it in action here:
{% hint style="info" %}
If you can't click on a segmentation because it is obstructed by another one on top, you can:
* Click on its row in the *Objects* list instead, or
* Hide the object overlapping it by clicking on the  icon in its row, then select the object, then unhide the overlapping object by clicking on its eye once more.
{% endhint %}
### Nudging Segmentations
You can apply small changes to your segmentations by 'nudging' them.
To nudge a segmentation, click on the segmentation on the asset, then pick the type of nudge tool you'd like to use:
Nudge Erase will allow you to reduce the size of your segmentation while applying small changes. Nudge Draw will add area to your segmentation.
Once you have picked your nudging tool of choice by clicking on either the "plus" or the "minus", click and drag around the segmentation's edge to refine the edge of your segmentation:
### Adding to an existing segmentation
The *Auto-Merge* feature allows you to extend an existing segmentation instance.
To activate it, click on an existing segmentation, then click on the "Auto-merge" icon. Draw where you would like to extend the segmentation, then press N.
{% embed url="" %}
# Skeleton
Overview of the Skeleton labeling tool in Ango Hub
The Skeleton tool allows you to create skeleton annotations on visual assets, such as images and videos. Individual skeleton points can be linked to other points in the skeleton, and can have unique attributes.
## How to add a Skeleton class to a project
From the project's *Settings* tab, enter the *Category Schema* section. From the *Add Category* dropdown, pick *Skeleton*.
A Skeleton-type class will be added to your project.
## How to define the skeleton's structure
Expand the Skeleton class in your category schema. After having given the class a name, click on *Define Skeleton*.
The *Define Skeleton* dialog will open. From here, you can click on the canvas to define your skeleton's structure. Start by clicking to create the skeleton's points:
You can rename each individual point by clicking on the pen icon next to the point's number in the list on the right-hand side of the screen.
By clicking on the "Upload Image" button, you can upload an image that will be used as background for the define skeleton canvas, for you to use as reference.
Then, click on *Connect Points* to create connections between the points you have just created:
{% hint style="info" %}
The button may now look different on the platform than on the video below, but the function remains the same.
{% endhint %}
You can delete connections by clicking on the trach can icon next to the connection on the right hand side of the screen. (you may have to scroll down)
Once done, click on *Apply*.
If necessary, you can then add attributes to each individual skeleton point. From the *Category Schema* section of your project's settings, click on an individual point of the skeleton, and then on *Add Classification*.
You can, for example, have a radio button, text, or dropdown as attribute to a point.
## How to annotate using the Skeleton tool
From within the labeling editor, click on the skeleton tool of your choice from the *Tools* panel on the left-hand side of the screen.
Then, click on the asset where you'd like to place the skeleton. The skeleton will appear.
Click on the skeleton to select it. You can then click and drag to move or resize it as a whole.
You can also move each individual point by double clicking on the point and moving it. Please see the video below:
To add attributes to a point, right-click the point.
# Voxel Brush
Overview of the Voxel Brush labeling tool in Ango Hub
The voxel brush labeling tool is a 3D pixel-wise tool to assign voxels on medical imaging assets to certain classes.
{% hint style="info" %}
The Voxel Brush tool is supported across the DICOM and Medical labeling editors.
{% endhint %}
## Adding a Voxel Brush tool to your project
From your project's *Settings* tab, enter the *Category Schema* section. Click on *Add Category*, then on *Voxel Brush*. The tool will be added to your project:
## Voxel Brush Settings
Once the tool has been added to your project, clicking on it wil reveal its settings.
**Schema ID**: This is a unique ID Ango Hub assigns to your tool. It's most important when attempting to import pre-annotations. [Read more about medical pre-annotations here](/data/importing-and-exporting-annotations/importing-annotations/importing-nrrd-annotations).
**Title**: A name of your choosing for this tool/class.
**Required**: If enabled, the annotator must have created at least one segmentation with this tool before being able to submit the task.
**Add Classification**: From this section, you may add nested classifications (attributes) to your class. Please refer to [the docs page on the medical labeling editor](/labeling/labeling-editor-interface/medical-labeling-editor#tool-class-list) for more information on how to annotate attributes.
## Threshold Editable Area
The in-editor *Editable Area* setting is documented with the other Medical editor brush controls, including *Brush Size*, *Brush Mode*, and *Brush Type*. See the [Medical Labeling Editor](/labeling/labeling-editor-interface/medical-labeling-editor#brush-tool) page.
# Spline
Overview of the Spline labeling tool in Ango Hub
The Spline labeling tool allows you to draw splines on images and videos. Splines are similar to polylines, with the exception that they smoothly curve along keypoints rather than being jagged.
## How to add a Spline tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Spline*.
A new row will appear named *Spline*. Click on it to expand it.
Give your Spline tool a title and description.
Enable the *Required* toggle if you want to force labelers to create a spline on every asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating a spline label using this class.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* after drawing the spline, click on *Add Classification* and add a further classification. More[ on nested classifications here](/labeling/labeling-tools/tools/nested-classifications).
### How to Draw a Spline
Click on the image where you’d like the first point of the spline to be. Click again where you’d like the second point to be, and so on. When you are done, click on the first point again or press on *N* on your keyboard to close the spline.
You may resize the width of each point on the spline by using the grab points on the side.
To edit a spline's points, select it by clicking on it, then drag the spline itself or one of its points.
To add new points, select the spline, then press and hold Ctrl on your keyboard, then click on the spline where you'd like to add the point.
To remove a point, select the spline, then right-click on the point you wish to delete.
You may use the keyboard shortcuts Y and U to lenghten/shorten the current point's width by 1 pixel, or Alt+Y and Alt+U to do so by 0.5 pixels.
### Edge Sharing
As you draw a spline, you may activate the *Edge Sharing* functionality. If you do so, Ango Hub will allow you to snap the point you are about to create to the existing point of another spline.
As you draw, use the keyboard shortcut "E" or click on the "Edge Sharing" button on the top of the editor. Then, as you draw, move your cursor close to a point of an already existing spline. Ango Hub will highlight that point and allow you to, if you choose to click, quickly snap your new point to it.
### Marking spline points
After having drawn a spline, you may mark certain points. This mark may have any meaning you choose to assign to it. It can, for example, be used to mark that a certain road marking is occluded in the current view.
To mark a certain point on a spline, after you have drawn a spline, click it to select it. Then, hover over the point you'd like to mark and press "B" on your keyboard. The point will be marked. You may unmark a point the same way.
# Classification
Available classification types in Ango Hub
# Radio
Overview of the Radio classification tool in Ango Hub
The radio labeling tool allows you to ask labelers close-ended questions with one right answer.
{% hint style="info" %}
The Radio tool is supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
In this case, users are asked the type of image shown. Clicking on another answer would deselect the current answer.
While there are no limits to the number of answers a radio tool can have, the tool is ideal for questions with a limited number of answers. For questions with more answers than that, you may consider using the single dropdown tool.
## The Radio Labeling Tool in Ango Hub
### How to add a Radio tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Radio*.
A new row will appear named *Radio*. Click on it to expand it.
Give your radio tool a title and description. From the *Options* text box, you can enter the answers. Press *Enter* after entering each one.
Enable the *Required* toggle if you want to force labelers to answer it. When the toggle is disabled, labelers will be able to save and move to the next asset without answering this question.
If you would like to ask labelers further questions, for example, if you want to show a further *radio* when labelers answer *Top-down*, click on *Add Classification* and add a further question. More on nested questions here.
### Differences between Radio and Single-Select Dropdown
The tool is functionally identical to the single dropdown, with the main difference being visual.
The major benefit of the radio tool over the dropdown tool is that it can be answered by a labeler with a single click, instead of the two required for dropdowns. The downside is that it cannot easily display large numbers of items. Here’s an example of how to ideally use radios and dropdowns:
When the number of possible answers is limited, we recommend choosing the Radio tool. For questions with a significant amount of more answers, the dropdown tool is preferable.
# Checkbox
Overview of the Checkbox classification tool in Ango Hub
The checkbox labeling tool allows you to ask labelers close-ended questions with multiple right answers.
{% hint style="info" %}
The Checkbox tool is supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
## The Checkbox Labeling Tool in Ango Hub
### How to add a Checkbox tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Checkbox*.
A new row will appear named *Checkbox*. Click on it to expand it.
Give your checkbox tool a title and description. From the *Options* text box, you can enter the answers. Press *Enter* after entering each one.
Enable the *Required* toggle if you want to force labelers to answer it. When the toggle is disabled, labelers will be able to save and move to the next asset without answering this question.
If you would like to ask labelers further questions, for example, if you want to show a Text tool when labelers select a particular answer, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### Differences between Checkbox and Multi-Select Dropdown
The tool is functionally identical to the Multi-Select Dropdown tool, with the main difference being visual.
The major benefit of the checkbox tool over the dropdown tool is that it can be answered by a labeler with a fewer amount of clicks, instead of the having to first open the dropdown to answer. The downside of checkbox is that it cannot easily display large numbers of items.
When the number of possible answers is limited, we recommend choosing the checkbox tool. For questions with a significant amount of more answers, the dropdown tool is preferable.
### Differences between Radio and Checkbox tools
In [radio](/labeling/labeling-tools/classification-tools/radio), labelers can only select one answer. In checkboxes, labelers can select more than one.
# Single-Select Dropdown
Overview of the Single-Select Dropdown classification tool in Ango Hub.
The single-select dropdown labeling tool allows you to ask labelers close-ended questions with one right answer.
{% hint style="info" %}
The Single-Select Dropdown tool is supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
## The Single-Select Dropdown Labeling Tool in Ango Hub
### How to add a Single-Select Dropdown tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Single-Select Dropdown*.
A new row will appear named *Single-Select Dropdown*. Click on it to expand it.
Give your single-select dropdown tool a title and description. From the *Options* text box, you can enter the answers. Press *Enter* after entering each one.
Enable the *Required* toggle if you want to force labelers to answer it. When the toggle is disabled, labelers will be able to save and move to the next asset without answering this question.
If you would like to ask labelers further questions, for example, if you want to show a Text tool when labelers answer *Other*, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### Differences between Radio and Single-Select Dropdown
The tool is functionally identical to the [Radio tool](/labeling/labeling-tools/classification-tools/radio), with the main difference being visual.
The major benefit of the radio tool over the dropdown tool is that it can be answered by a labeler with a single click, instead of the two required for dropdowns. The downside is that it cannot easily display large numbers of items. Here’s an example of how to ideally use radios and dropdowns:
When the number of possible answers is limited, we recommend choosing the Radio tool. For questions with a significant amount of more answers, the dropdown tool is preferable.
### Differences between the Single and Multi-Select Dropdown tools
In single-select dropdowns, labelers can only select one answer. In multi-select dropdowns, labelers can select more than one.
# Multi-Select Dropdown
Overview of the Multiple Dropdown classification tool in Ango Hub
The multi-select dropdown labeling tool allows you to ask labelers close-ended questions with multiple right answers.
{% hint style="info" %}
The Multi-Select Dropdown tool is supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
## The Multi-Select Dropdown Labeling Tool in Ango Hub
### How to add a Multi-Select Dropdown tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Multi-Select Dropdown*.
A new row will appear named *Multi-Select Dropdown*. Click on it to expand it.
Give your multi-select dropdown tool a title and description. From the *Options* text box, you can enter the answers. Press *Enter* after entering each one.
Enable the *Required* toggle if you want to force labelers to answer it. When the toggle is disabled, labelers will be able to save and move to the next asset without answering this question.
If you would like to ask labelers further questions, for example, if you want to show a Text tool when labelers select a particular answer, click on *Add Classification* and add a further question. [More on nested questions here](/labeling/labeling-tools/tools/nested-classifications).
### Differences between Checkbox and Multi-Select Dropdown
The tool is functionally identical to the [Checkbox tool](/labeling/labeling-tools/classification-tools/checkbox), with the main difference being visual.
The major benefit of the checkbox tool over the dropdown tool is that it can be answered by a labeler with a fewer amount of clicks, instead of the having to first open the dropdown to answer. The downside of checkbox is that it cannot easily display large numbers of items.
When the number of possible answers is limited, we recommend choosing the checkbox tool. For questions with a significant amount of more answers, the dropdown tool is preferable.
### Differences between the Single and Multi-Select Dropdown tools
In [single-select dropdowns](/labeling/labeling-tools/classification-tools/single-dropdown), labelers can only select one answer. In multi-select dropdowns, labelers can select more than one.
# Tree Dropdown Tools
Overview of the Tree Dropdown classification tool in Ango Hub
The Tree Dropdown labeling tool allows you to ask labelers to pick leaves and trees from complex, nested tree structures.
Two types of this tool exist: multiple and single selection. Single selection tree dropdowns let users only pick one option, while the multiple selection allows users to pick multiple options at once.
{% hint style="info" %}
The Tree Dropdown tools are supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
## The Tree Dropdown Labeling Tool in Ango Hub
### How to add a Tree Dropdown tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Tree Dropdown*.
A new row will appear named *Tree Dropdown*. Click on it to expand it.
### Properties of the Tree Dropdown Tool
* **Schema ID**: A unique ID assigned to all instances of labeling tools on Ango Hub. You can use it when [importing labels for pre-labeling](/data/importing-and-exporting-annotations/importing-annotations).
* **Title**: The title of the labeling tool you are creating. This will show up both in the editor for labelers to see, as well as in the final export.
* **Required**: Toggle this to make labelers answer this classification before saving.
* **Frame-specific**: For video assets, determine whether the answer to this classification applies to the whole video or to each individual frame only. Toggling will make it frame-specific.
* **Show Dropdown**: Normally, trees are shown embedded within the classification tool and are immediately, fully visible to annotators. Toggling this option will make it so that to see the tree, annotators will have to click on an area, and as a result of the click the tree will show as a "dropdown". This can especially be useful for larger trees.
Normal "embedded" view:
With "Show Dropdown" enabled:
#### Adding Branches and Leaves
Click on *Configure* from the label tool's options:
From the dialog that pops up, click on "Add Node" to add a new node stemming from the root:
To add a new children node stemming from a parent node, click on the "+" button next to the parent node.
You may rename nodes by clicking on the "pencil" button next to the node's name, entering the name, then pressing "Enter".
If you wish to allow annotators to select parent nodes (branches) in addition to leaves, turn on the "Allow Parent Selection" toggle:
# Text
Overview of the Text tool in Ango Hub
The text labeling tool allows you to ask labelers open-ended questions.
{% hint style="info" %}
The Text tools are supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
### How to add a Text tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Text*.
A new row will appear named *Text*. Click on it to expand it.
### Text Tool Options
**Schema ID**: The unique ID assigned to the tool.
**Title**: The name of your labeling tool.
**Required**: If you enable this toggle, labelers will be required to answer this field before saving.
**Regex Validation**: If you enter a regular expression in this field (without the initial and final `\` slashes), labelers' answers to this field will have to match the regex being input before saving.
Labelers answer in a plain text box. This box grows automatically as the labeler types longer answers, up to about 10 visible rows. After that, the field remains scrollable, and labelers can still resize it vertically.
**KaTeX Preview**: If enabled, you may have annotators see a live preview of KaTeX, a mathematical subset of LaTeX:
**Frame-specific**: If you enable this toggle, the answer to this classification will only be valid for one frame (in videos and multi-frame images/DICOMs) instead of the whole asset.
**Multiple**: If enabled, annotators can answer this classification multiple times.
Once you have selected your options, click on *Save* at the bottom to save your changes.
# Slider
Overview of the Slider classification tool in Ango Hub
The Slider classification tool allows you to ask labelers to answer numeric questions by choosing a value on a configurable range.
{% hint style="info" %}
The Slider tool is supported across the Audio, Image, Video, DICOM, Medical, PDF, Text, LLM, and Markdown labeling editors.
{% endhint %}
### How to add a Slider tool to your project
From the project's *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Slider*.
A new row will appear named *Slider*. Click on it to expand it.
### Slider Tool Options
**Schema ID**: The unique ID assigned to the tool.
**Title**: The name of your Slider classification.
**Description**: Optional text shown to labelers in the labeling editor.
**Required**: If you enable this toggle, labelers will be required to answer this field before saving.
**Min Value**: The lowest numeric value labelers can select.
**Max Value**: The highest numeric value labelers can select.
**Step Size**: The interval between selectable slider values. For example, a step size of `1` allows whole numbers, while `0.5` allows half-step values.
**Min Label**: Optional text shown at the low end of the slider.
**Max Label**: Optional text shown at the high end of the slider.
**Frame-specific**: If you enable this toggle, the answer to this classification will only be valid for one frame in videos and multi-frame images/DICOMs instead of the whole asset.
**Data-specific**: If you enable this toggle, the answer to this classification will be stored separately for each data item in grouped assets.
**Multiple**: If enabled, annotators can answer this classification multiple times.
Slider answers are saved and exported as numeric values.
Once you have selected your options, click on *Save* at the bottom to save your changes.
# Data
# File Upload Box
Overview of the file upload box tool in Ango Hub
The *Upload Box* tool allows you to add a classification where users can upload a file while labeling a task.
Each upload box can contain one file per task. If you need to collect more than one file, add multiple upload boxes to the project, or ask users to upload one archive file containing all required files.
## How to add an Upload Box tool to a project
From your project's *Settings* tab, enter the *Category Schema* section. Click on *Add category*, then pick *Upload box*. An *Upload box* tool will appear in your ontology:
## Upload Box Settings
Setting name
Description
Title
The upload box's title. This text will appear immediately above the upload box itself.
Required
Whether or not the user is required to upload a file to this box before submitting the task.
Data storage
By clicking on Pick folder, you're able to select the storage integration (e.g. AWS S3, GCP, Azure), bucket or Azure container, and subfolder where uploaded files will be written.
When selecting a bucket or container, you can pick one from the list or switch to search mode and type the name manually. Manual search is useful when the connected storage does not allow Ango Hub to list buckets or containers, but does allow access to the one you enter.
Please note: Ango Hub must have write permissions to the bucket or container you are pointing to. See our page on storages for more.
Accepted Formats
You may optionally list the file extensions you'd like to accept for this upload box. For example, .jpg and .png. If you leave this blank, any file extension will be accepted.
Enter extensions with the dot, such as .pdf, and use lowercase extensions.
Filename
If you fill in this field, the uploaded file will be saved with this filename. The filename is also shown to users in the upload box.
If you leave this blank, Ango Hub generates a filename from the upload box title, upload time, and original filename.
## Where uploaded files are stored
If you configure *Data storage*, uploaded files are written to the selected storage integration, bucket or Azure container, and folder. Ango Hub also adds project and task-specific path segments so that files from different projects and tasks do not overwrite each other.
If you do not configure *Data storage*, uploaded files are stored in Ango Hub's default storage for upload-box files.
The final object path is built as follows:
```
//upload-box//
```
If you do not select a storage folder, the path in Ango Hub's default storage is:
```
/upload-box//
```
For example, if your selected folder is `uploads/evidence`, the project ID is `PROJECT_ID`, the task ID is `TASK_ID`, and the uploaded filename is `evidence.pdf`, the uploaded object will be written under:
```
uploads/evidence/PROJECT_ID/upload-box/TASK_ID/evidence.pdf
```
{% hint style="warning" %}
If you are using your own storage integration, Ango Hub needs write access to the selected bucket or Azure container and folder. If Ango Hub cannot create the upload URL or write the file, the user will see a file upload failure.
{% endhint %}
## How a user can upload a file on the upload box
When the user is in the task, they will see the upload box as a classification.
They can drag and drop a file onto the upload box, or click on the box to open their OS's file picker.
If the upload box already has a file, uploading another file replaces the answer for that upload box.
Once a file has been uploaded, the user will see a link to the uploaded file and a delete button. Deleting the answer removes the file from the task's answer, but does not necessarily delete the object from the underlying storage bucket or container.
## Export format
Upload box answers are exported in the task's `classifications` list, like other classification answers.
In the export, the classification's `tool` is `upload-box`, and the `answer` is a signed URL to the uploaded file.
```json
{
"objectId": "b0f7f4c7f7e14d2a8b3fcb42",
"schemaId": "58af1b9c7f334a939afbbf89",
"tool": "upload-box",
"title": "Upload supporting file",
"answer": "https://example-bucket.s3.amazonaws.com/project-id/upload-box/task-id/evidence.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256...",
"classifications": []
}
```
Because the exported `answer` is a signed URL generated at export time, it may expire. If you need long-term access to uploaded files, use your storage provider's object path or bucket tooling as the durable source of truth.
See [Classifications in the Ango Export Format](/data/importing-and-exporting-annotations/exporting-annotations/ango-export-format/asset/task/classifications) for the full classification export structure.
# Relation
Available relation types in Ango Hub
Relations connect annotations to describe how they are associated. Ango Hub provides two relation types:
* [Single Relation](/labeling/labeling-tools/relation/single-relation), which connects two annotations and can include a direction.
* [Group Relation](/labeling/labeling-tools/relation/group-relation), which groups two or more annotations.
Both relation types can contain [nested classifications](/labeling/labeling-tools/tools/nested-classifications). This lets project managers ask questions about the relation itself, such as the type, confidence, or context of the connection. Relation classifications can be required, can contain conditional follow-up questions, and are included when annotations are saved, imported, exported, or sent through a webhook.
Relation classifications are available wherever relations are supported, except in 3D MSFT projects and the DICOM series editor.
# Single Relation
Overview of the Single Relation tool in Ango Hub
The Single Relation labeling tool allows you to create a relation between two annotations.
{% hint style="info" %}
The Single Relation tool is supported across the Audio, Image, Video, DICOM, PDF, and Text labeling editors.
{% endhint %}
{% hint style="info" %}
Single relations are not visible unless the user is hovering over them with the mouse cursor.
{% endhint %}
## How to add a Single Relation tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Single Relation*.
A new row will appear named *Single Relation*. Click on it to expand it.
Give your single relation tool a title and description.
Enable the *Required* toggle if you want to force labelers to create a single relation for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating the single relation.
### How to Draw a Single Relation
Click on the label where you’d like the single relation to start. Click on the second label where you’d like the relation to end. Press N on your keyboard to finalize the relation.
After having created the relation, you can change its direction by clicking on the small arrow inside the relation's box.
#### Single Relation Directions
* **Forward:** the first label points at the second label.
* **Backward:** the second label points at the second label.
* **Bidirectional:** The labels point at each other.
# Group Relation
Overview of the Group Relation tool in Ango Hub
The Group Relation labeling tool allows you to create a relation between two or more annotations.
{% hint style="info" %}
The Group Relation tool is supported across the Audio, Image, Video, DICOM, PDF, and Text labeling editors.
{% endhint %}
### How to add a Group Relation tool to your project
From the project’s *Settings* tab, enter the *Label Set* section.
Click on *Add Category*. From the list that appears, click on *Group*.
A new row will appear named *Group*. Click on it to expand it.
Give your group relation tool a title and description.
Enable the *Required* toggle if you want to force labelers to create a group relation for each asset. When the toggle is disabled, labelers will be able to save and move to the next asset without creating the single relation.
## How to Draw a Single Relation
There are two ways to add a group relation in Hub.
1. Click on the first label to include in the group relation. Then, shift-click on all other annotations you wish to include. Right-click on any one of them, and select your group relation under the *Grouping* menu item:
2. Click on any one of the *Group Relation* tools you have created from the *Tools* section on the left of the screen. Then, left-click on all annotations you wish to include. Press "N" on your keyboard when you are done to close the group relation.
{% hint style="info" %}
If you wish to add an annotation to more than one group relation:
1. Hide the relation it is currently part of by pressing "H" on your keyboard.
2. Add the annotation to the second group relation.
3. Press "Shift + H" to unhide all relations.
{% endhint %}
# Labeling AI Assistance
