Ango Hub Docs
Open Ango HubContact iMerit
  • Ango Hub Documentation
  • Video Guides
  • Changelog
  • FAQs & Troubleshooting
  • All Keyboard and Mouse Shortcuts
  • Core Concepts
    • Assets
    • Attachments
    • Batches
    • Benchmarks
    • Category Schema (Ontologies)
    • Frame Interpolation
    • Geofencing
    • Idle Time Detection & Time Tracking
    • Instructions
    • Issues
      • Issue Error Codes
    • Label Validation
    • Labeler Performance
    • Labeling
    • Labeling Queue
    • Multiple Classification
    • Notifications
    • Organizations
    • Projects
    • Requeuing
    • Reviewing
    • Review Queue
    • Skipping
    • Stage History
    • Tasks
    • Usage
    • User Roles
    • Workflow
      • Complete
      • Consensus
      • Hold
      • Label
      • Logic
      • Plugin
      • Review
      • Start
      • Webhook
  • Labeling
    • Managing Users in Projects
      • Profile Page
    • Managing the Project Ontology
    • Labeling Editor Interface
      • Audio Labeling Editor
      • Image Labeling Editor
      • Video Labeling Editor
      • DICOM Labeling Editor
      • Medical Labeling Editor
        • 3D Bounding Box
        • Fill Between Slices
        • Island Tools
        • Line (Tape Measure)
        • Smoothing
      • PDF Labeling Editor
      • Text (NER) Labeling Editor
      • LLM Chat Labeling Editor
      • Markdown Labeling Editor
      • 3D Multi-Sensor Fusion Labeling Editor
    • Labeling Classes
      • Tools
        • Bounding Box
        • Brush
        • Entity
        • Message
        • Nested Classifications
        • PCT
        • PDF Tool
        • Point
        • Polygon
        • Polyline
        • Rotated Bounding Box
        • Segmentation
        • Spline
        • Voxel Brush
      • Classification
        • Checkbox
        • Multiple Dropdown
        • Radio
        • Rank
        • Single Dropdown
        • Text
        • Tree Dropdown Tools (Single and Multiple Selection)
      • Relation
        • Single Relation
        • Group Relation
    • Magnetic Lasso
    • Performance & Compatibility Considerations
  • Data
    • Data in Ango Hub
      • Embedding Private Bucket Files in MD Assets
    • Importing Assets
      • Asset Builder
      • Bundled Assets
        • Importing Multiple Images as One Multi-Page Asset
        • Importing Multiple Single-Frame DICOM Files as One Multi-Page Asset
        • Importing multiple DICOM files to be annotated and displayed at once
        • Importing Multiple Single-Frame DICOM Files as a DICOM Series
        • Importing Multiple Markdown Files as One Multi-Page Asset
      • File Explorer
      • Supported Asset File Types & Codecs
      • Importing Cloud (Remote) Assets
      • Importing From Local Machine
      • Creating and Importing LLM Chat Assets
      • Importing Data in the 3D Multi-Sensor Fusion Labeling Tool
      • Bulk Importing Markdown/HTML Assets
      • Importing Attachments during Asset Import
      • Importing Annotations during Asset Import
      • contextData: Adding Extra Data to Assets
      • Importing Reference Images as Overlay
      • Importing Reference Medical Data During Asset Import
    • Importing and Exporting Annotations
      • Importing Annotations
        • Ango Import Format
        • Importing Brush Traces
        • Importing NRRD Annotations
      • Exporting Annotations
        • Ango Export Format
          • Asset
            • Task
              • Tools
              • Classifications
              • Relations
          • Stage History
    • Adding and Managing LLMs
    • Storages
      • Set up a storage integration with Azure
      • Set up a storage integration with AWS S3
      • Set up a storage integration with MinIO and S3-compatible custom storage services
      • Set up a storage integration with GCP (Google Cloud Platform)
      • Set up CORS
      • Validating Storage Integrations
    • Purging Data from Ango Hub
  • Plugins
    • Overview of Plugins in Ango Hub
      • Installing Plugins
      • Plugin Setting Presets
      • Monitoring Plugin Progress
    • First-Party Plugins
      • Ango Export Converter Plugins
      • Asset Converter Plugins
      • Ango to Mask Converter
      • Batch Assignment
      • ChatGPT
      • Column-Agnostic Markdown Generator
      • CSV Export for Classification
      • DALL-E
      • DALL-E (Model Plugin)
      • File Explorer Plugin
      • General Object Detector
      • General Object Segmenter
      • Markdown Generator
      • One-Click Segmentation
      • Open World Object Detection
      • Optical Character Recognition
      • TPT Export
      • YOLO | Instance Segmentation
      • YOLO | Pose Estimation
      • YOLO | Object Detection
      • YOLO | Image Classification
    • Plugin Developer Documentation
      • Export Plugins
      • Batch Model Plugins
      • Model Plugins
      • File Explorer Plugins
      • Markdown Generator Plugins
      • Plugin Logger
      • [WIP] Deploying your Plugin
      • Plugin 'Host' Information
  • SDK
    • SDK Documentation
      • Project Level SDK Functions
        • add_members_to_project
        • assign_batches
        • assign_task
        • create_attachment
        • create_batch
        • create_issue
        • create_label_set
        • create_project
        • delete_issue
        • export
        • exportV3
        • get_assets
        • get_batches
        • get_issues
        • get_metrics
        • get_project
        • get_project_performance
        • get_task
        • get_tasks
        • get_task_history
        • import_labels
        • list_projects
        • requeue_tasks
        • rerun_webhook
        • update_workflow_stages
        • upload_files
        • upload_files_cloud
        • upload_files_with_asset_builder
        • upload_chat_assets
      • Organization Level SDK Functions
        • create_storage
        • delete_organization_invites
        • delete_organization_members
        • delete_storage
        • get_organization_invites
        • get_organization_members
        • get_storages
        • invite_members_to_org
        • update_organization_members_role
    • SDK - Useful Snippets
    • SDK Changelog
  • API
    • API Documentation
  • How-To
    • Add Members
      • Add multiple users to a project
    • Annotate
      • Annotate 3D Point Cloud Files on Ango Hub
      • Perform targeted OCR on images
    • Export Data
      • Automatically send Ango Hub Webhook contents to Google Sheets, Email, Slack, and more with Zapier
      • Download a JSON of your project ontology
      • Download DICOM Segmentation Masks
      • Download your annotations in the COCO, KITTI, or YOLO format
      • Download your Segmentation Masks
      • Get your export as separate JSON files for each asset
    • Manage a Project
      • Get your API Key
      • Get your Organization ID
      • Mute your notifications
      • Open an asset provided the Asset ID
      • Pre-label assets
      • Share a filtered view of the Tasks table with others
      • Transfer project ontologies between projects
      • Transfer project workflows between projects
    • Perform Model Evaluation on Ango Hub
  • Troubleshooting
    • I get a "0 Tasks Labeled" alert when trying to pre-label tasks
    • I get a 'The data couldn't be loaded properly' error when opening certain assets
    • I get a "Unknown Classification" warning when opening a task
  • Feature Availability Status for projects of the 3D Multi-Sensor Fusion type
  • Comparison between QuickServe and Ango Hub
  • Changes from Ango Hub Legacy
  • Video V2 Breaking Changes and Transition
  • Data Access, Storage, and Security
  • Two-Factor Authentication
  • Single Sign-On (SSO) Support
  • Customer Support
  • Ango Hub Status Page
Powered by GitBook
On this page
  • Diagram of how Consensus works
  • Settings
  • Auto Assign
  • Stage Settings
  • How Consensus is Calculated
  • Classifications
  • Points
  • Other objects (Bounding Box, Polygon, PDF Area)
  • Output
  • The Output Task
  1. Core Concepts
  2. Workflow

Consensus

PreviousCompleteNextHold

Last updated 3 months ago

Because of the way the Consensus mechanism works under the hood, logic stages of the type "Annotator" and "Duration" may not work as expected when processing tasks output from a Consensus stage.

Requeuing tasks with issues which have been output from a Consensus stage might lead to unexpected behavior regarding the issues. We recommend closing all issues on such tasks before requeuing them.

A Consensus stage is a way for you to present tasks to multiple annotators, and have the task be output in either Agreement or Disagreement conditional upon how much the annotators agree with one another.

Essentially, the Consensus stage is a container for other Label or Plugin sub-stages.

The Consensus stage accepts plugin sub-stages, such that, for example, you can have a task be labeled by an annotator and a plugin, and you may return the task based on how similarly the annotator labeled the task compared to the plugin.

There is a limit of ten maximum sub-stages you may add to the Consensus stage.

Consensus agreement cannot be calculated for the following class types:

  • Brush

  • Voxel Brush

  • Segmentation

  • Polyline

  • Rotated Bounding Box

In video assets, consensus calculation ignores pages. Because of this, we do not recommend using consensus in video-based tasks yet.

Diagram of how Consensus works

As mentioned in the diagram above, whenever a task enters the Consensus stage, it is 'duplicated' into sub-tasks, and each sub-task is sent to its own sub-stage.

You may examine individual sub-tasks and check their current status from the "Tasks" tab, by clicking on the "Plus" next to the Consensus task to expand it and see details pertaining to the sub-tasks:

If a sub-task is in the "Archive" stage it means it has been completed and submitted.

If a sub-task has been assigned to a specific annotator, but you wish to unassign it from that person and place it back in the queue for that specific consensus sub-stage, you can click on the "Un-assign and send back to queue" button for that sub-task:

Once all sub-stages have been annotated, they will be archived and they will no longer be accessible through the "Tasks" tab. They will, however, be accessible from the "Stage History" panel in the labeling editor when opening the main task.

Settings

Auto Assign

By default, Ango Hub does not prevent the same annotator from annotating the same asset more than once as part of a consensus stage.

For example, if you add two Label tasks which can be annotated by Anyone, like so:

Labeler A will open their labeling queue and go through the tasks in Consensus_1.

If no other annotator has opened the tasks annotated by Labeler A, and Labeler A clicks on Start Labeling and enters the labeling queue, they may enter the Consensus_2 queue and label the same tasks again. This way, consensus will not be calculated between two different annotators, as usually expected, since the same annotator will have annotated both tasks themselves.

To prevent this, you'd have to assign each labeling stage in consensus to different annotators. Auto Assign automates this process for you.

From the Consensus stage settings, click on Auto Assign. The following dialog will pop up:

Toggle on the users you'd like to assign to the stages within the selected consensus container, and they'll be distributed to every consensus stage in the container. If, after doing so, there are no consensus stages in your container assigned to Anyone, then you have guaranteed that no labeler will see the same task twice.

Stage Settings

Setup

Threshold

From this view, you will be able to pick what will be determined as Agreement and Disagreement. You will see a list of labeling tools present in your project.

To have a tool be included in the Consensus calculation, enable the toggle next to it.

In the example above, we have three tools: a bounding box named Vehicle, a radio classification named Color, and a single dropdown named Model. In this example, the task will be considered in agreement when at least 30% of the annotators give the same answer to Color, and at least 30% of annotators give the same answer to Model. When both of these conditions are satisfied, the task is marked as being in Agreement.

Since the "Vehicle" bounding box had its toggle turned off, annotations from that class will not be counted in the consensus score calculation.

Adjudication

The task sent as output is not the judgment from a single annotator – it is instead a composite task, the contents of which will be determined by the adjudication method you pick here.

Best Answer

The output task contains the annotations with the highest consensus score, for each class, for classes where consensus can be calculated.

  • For example, if the consensus stage has three judgment sub-stages, and the task has three radio classifications A, B, and C, and one bounding box class D, the task output at the end will have, for each classification, the answer annotators coalesced on the most, and for class D, the bounding boxes created by the annotator with the highest class D consensus score.

  • For classes where consensus cannot be calculated (e.g. assume in our project there is a points class E and a rotated bounding box class F), the final task will have the non-calculable classes from the first user who has submitted them in the consensus stage.

  • So in this case, we would have the best answers from classes A, B, and C, then the bounding boxes drawn by the user with the highest class D consensus score, and for classes E and F, we would have the answers given by the first user to submit them in the consensus stage.

If, in the Consensus stage, some annotators did not create annotations using a certain class, or did not answer some classification answers, but others did, the output task will contain them, even if not all consensus annotators responded.

  • For example, if we have a project with a bounding box class A, a polygon class B, a radio classification C, and a text classification D, assuming:

    • User 1 only created 1 bounding box with class A, and answered the radio classification C (no other answers/annotations)

    • User 2 only created 1 bounding box with class A, and a polygon with class B (no other answers/annotations)

    • User 3 only created 1 bounding box with class A, answered the text classification D (no other answers/annotations)

  • The output composite task will have:

    • The class A bounding boxes drawn by the user with the highest class A consensus score

    • The class B polygons created by User 2

    • The class C radio answer from User 1

    • The class D text answer from User 3

Here is a visual representation of the algorithm, given three annotators working on the same image:

All Answers

The output task contains all annotations from all consensus stages/judgments, merged together.

Here is a visual representation of the algorithm, given three annotators working on the same image:

How Consensus is Calculated

Classifications

Let questionCount be the total number of classification questions in the project, and taskCount the total number of tasks assigned to an asset.

We calculate the single-question consensus for a single task as sameAnswers / taskCount, where sameAnswers is the count of answers that are equal to one another, current one included.

We repeat the above calculation for all tasks in the asset, the overall consensus on a single question (classification) is the highest value achieved during the repetitions, (y).

We repeat the above calculation for all questions in the asset, to get to the final result represented as Σ(y) below.

The final consensus score, then, is calculated as ∑(y) / questionCount.

Note on Rank Benchmarking

Points

The algorithm checks the proportion between the distance of two points and the longest distance on the image. For example, let’s say we have an image with 500 height and 1200 width. The longest distance for the image here will be 1300.

And I have a point from Consensus one [200,200] and from Consensus two [500,600]. The distance between these points is 500. The consensus result of this two points will be 1300 / 500 = 26(%) And since we also compare the point with itself to adjust the consensus, which will be 100%, the overall consensus will be 63%.

See the following visual examples:

Other objects (Bounding Box, Polygon, PDF Area)

We calculate consensus for objects using the Intersection over Union (IoU) method.

We compare objects with one another to generate their IoU scores. If some annotations are completely separate, for example, with not even a pixel in common, their IoU score would be 0. If they overlapped completely, their score would be 100.

Noting that objects are compared to themselves too, hence for the above not-intersecting objects example the score of each object would be 50. The highest score achieved for all tasks will be taken into account as the consensus score of that tool (a tool means a unique schema id here, not the tool type).

We then average the IoU scores of all tools to calculate the final consensus score.

Output

The Consensus stage has two output: Agreement and Disagreement.

The Output Task

The task you will get as the output will be determined by the method you pick in the stage's Adjudication tab. Please refer to the section on adjudication for more on the task being output.

The task sent as output is not the judgment from a single annotator – it is instead a composite task. Here are this task's properties:

  • The output task contains the annotations with the highest consensus score, for each class, for classes where consensus can be calculated.

    • For example, if the consensus stage has three judgment sub-stages, and the task has three radio classifications A, B, and C, and one bounding box class D, the task output at the end will have, for each classification, the answer annotators coalesced on the most, and for class D, the bounding boxes created by the annotator with the highest class D consensus score.

    • For classes where consensus cannot be calculated (e.g. assume in our project there is a points class E and a rotated bounding box class F), the final task will have the non-calculable classes from the first user who has submitted them in the consensus stage.

    • So in this case, we would have the best answers from classes A, B, and C, then the bounding boxes drawn by the user with the highest class D consensus score, and for classes E and F, we would have the answers given by the first user to submit them in the consensus stage.

  • If, in the Consensus stage, some annotators did not create annotations using a certain class, or did not answer some classification answers, but others did, the output task will contain them, even if not all consensus annotators responded.

    • For example, if we have a project with a bounding box class A, a polygon class B, a radio classification C, and a text classification D, assuming:

      • User 1 only created 1 bounding box with class A, and answered the radio classification C (no other answers/annotations)

      • User 2 only created 1 bounding box with class A, and a polygon with class B (no other answers/annotations)

      • User 3 only created 1 bounding box with class A, answered the text classification D (no other answers/annotations)

    • The output composite task will have:

      • The class A bounding boxes drawn by the user with the highest class A consensus score

      • The class B polygons created by User 2

      • The class C radio answer from User 1

      • The class D text answer from User 3

The Consensus stage, by default, does not prevent the same task from being labeled by the same person. To prevent that from happening, you will have to assign different annotators to different label stages, as mentioned in . This can be done automatically by clicking on in the settings for the consensus stage.

More details in the section for .

Clicking on Add Label will add a label stage. Clicking on Add Plugin will add a plugin stage. Click on each individual stage to change their options. Enable the grey toggle to mark the stage as dynamic. (see section on ). Click on the trash can to delete the stage.

In the classification tool, if the annotator's answers differ, in any way, with the benchmark, their score for that classification will be 0. If they are the exact same, it wil be 1 (e.g. 100%) for that classification.

If the consensus threshold has been achieved for all labeling tools and classifications specified in the , the consensus task will be output from the Agreement output. Otherwise, it will be sent from the Disagreement output.

Rank
the section for the Label stage
Auto Assign
Auto Assign
Dynamic Consensus
stage setup