Ctrlk
Contact iMeritAngo Hub WebsiteLog In to Ango Hub

Stage Comparison

Overview

The Stage Comparison Plugin compares annotations created in two different workflow stages and identifies differences between them. This is especially useful for QA workflows, gold standard validation, reviewer vs labeler analysis, and auditing annotation consistency across stages.

Cover

Stage Comparison

Plugin Functionality

The Stage Comparison Plugin compares annotations created in two different workflow stages and generates a detailed reports to highlight changes between them. It helps teams evaluate labeling quality, measure review impact, and audit annotation consistency across stages.

  • Compares annotations between a selected evaluation and stages to identify differences.

  • Detects added, removed, and modified annotations across supported tools, classifications and relations.

  • Highlights geometry-level and attribute-level changes.

  • Generates annotation and annotator level metrics in a structured report for analysis and monitoring.

  • Supports object-level matching using a configurable attribute to track changes to the same object across stages.

Running the Stage Comparison plugin on your project generates multiple CSV files:

Comparison File

This file contains a table where each row represents an object and shows the comparison results between two workflow stages.

Aggregate File

Each row corresponds to an asset. This table aggregates data from the Comparison File and summarizes the comparison result statistics.

Metric File

This file provides overall metrics for the comparison.

The CSV files contains following columns:

  • Batch

  • External ID

  • Assignee

    • Evaluation Stage Assignee

    • Gold Stage Assignee

  • Page

  • Schema ID

  • Object ID

  • Tool Type

  • Answer

    • Evaluation Stage Answer

    • Gold Stage Answer

  • Description

    • Evaluation Stage Description

    • Gold Stage Description

  • Comparison

Comparison Categories

Classifications

Comparison
Description

Missing

The answer in the Gold stage is available, but the answer in the Evaluation stage is not

Unqualified

The answer in the Evaluation stage is available, but the answer in the Gold stage is not

Match

The answer in the Gold stage is exactly equal to the answer in the Evaluation stage

Wrong*

The answer in the Gold stage is different from the answer in the Evaluation stage

Wrong-Missing Answer*

The answer exists in the Gold stage, but one or more expected answers are missing in the Evaluation stage.

Wrong-Unqualified Answer*

The Evaluation stage contains one or more additional answers that are not present in the Gold stage.

The "Wrong" category is applicable only to Radio, Single-Select Dropdown, Single-Select Tree, and Text classifications.

The "Wrong-Missing Answer" and "Wrong-Unqualified Answer" categories are applicable only to Checkbox, Multi-Select Dropdown, and Multi-Select Tree classifications.

Tools

Comparison
Description

Missing

The tool exists in the Gold stage, but no corresponding tool exists in the Evaluation stage.

Unqualified

The tool exists in the Evaluation stage, but no corresponding tool exists in the Gold stage.

Match

The tool class, coordinates, and description are exactly equal between the Gold and Evaluation stages.

Wrong-Class

The tool exists in both stages, but the assigned classes are different.

Wrong-Coordinates

The tool exists in both stages, but the coordinates are different.

Wrong-Description

The tool exists in both stages, but the descriptions are different.

Wrong-[Class+Coordinates+Description]

Multiple mismatches exist simultaneously for the same tool.

Relations

Comparison
Description

Missing

The relation exists in the Gold stage, but no corresponding relation exists in the Evaluation stage.

Unqualified

The relation exists in the Evaluation stage, but no corresponding relation exists in the Gold stage.

Match

The relation class and connected entities exactly match between the Gold and Evaluation stages.

Wrong-Class

The relation exists in both stages, but the relation classes are different.

Wrong-Description

The relation exists in both stages, but the descriptions or attributes are different.

Wrong-Class+Description

Multiple mismatches exist simultaneously for the same relation.

Supported Data Types

  • Compatible with all data types available in AngoHub.

Supported Annotation Tools

  • Classifications - All

  • Tools

    • Bounding Box

    • Rotated Bounding Box

    • Polygon

    • Segmentation

    • Entity

    • Polyline

    • Point

    • PDF

    • Nested Classifications

  • Relations

    • Single

    • Group

Plugin Configuration

From the Stage Filter field, pick the stages containing the tasks you'd like to get the export of. Similarly, from the Batch Filter field, you may select one or more batches the tasks of which will be exported. By default, all tasks are exported from all batches.

If you wish to receive an email when the export is complete, toggle Send Email on.

You may vary a number of settings related to your export from the Config JSON field. Each option is detailed below:

  • "gold_stage": The stage used as the gold standard reference for comparison.

    • Example:

      • "gold_stage": "Review"

  • "evaluation_stage": The stage whose annotations are evaluated against the gold standard.

    • Example:

      • "evaluation_stage": "Label"

  • "start_date": Start date for filtering assets included in the comparison (YYYY-MM-DD).

    • Example:

      • "start_date": "2020-12-31"

  • "end_date": End date for filtering assets included in the comparison (YYYY-MM-DD).

    • Example:

      • "end_date": "2021-12-31"

  • "ignored_schema_ids": List of schema IDs to exclude from the comparison.

    • Example:

      • "ignored_schema_ids": ["12345", "12346"]

  • "merge_output": Whether to merge comparison results into a single consolidated output.

    • Example:

      • "merge_output": true

      • "merge_output": false

  • "additional_attribute_name": Attribute name used for object-level matching across stages.

    • Example:

      • "additional_attribute_name": "Object ID"

  • "include_key_frames_only": Specifies whether only key frames should be included in the metrics calculation. (For video assets only)

    • Example:

      • "include_key_frames_only": true

      • "include_key_frames_only": false

  • "logging_frequency": Defines how frequently progress logs are displayed; setting this value to 0 disables logging entirely, while any positive integer enables logging at the specified interval. For more information on how to view plugin logs, see here.

    • Example:

      • "logging_frequency": 0

      • "logging_frequency": 100

Last updated