Stage Comparison

Overview

The Stage Comparison Plugin compares annotations created in two different workflow stages and identifies differences between them. This is especially useful for QA workflows, gold standard validation, reviewer vs labeler analysis, and auditing annotation consistency across stages.

Stage Comparison

Plugin Functionality

The Stage Comparison Plugin compares annotations created in two different workflow stages and generates a detailed reports to highlight changes between them. It helps teams evaluate labeling quality, measure review impact, and audit annotation consistency across stages.

• Compares annotations between a selected evaluation and stages to identify differences.

• Detects added, removed, and modified annotations across supported tools, classifications and relations.

• Highlights geometry-level and attribute-level changes.

• Generates annotation and annotator level metrics in a structured report for analysis and monitoring.

• Supports object-level matching using a configurable attribute to track changes to the same object across stages.

  

Running the Stage Comparison plugin on your project generates multiple CSV files:

Comparison File

This file contains a table where each row represents an object and shows the comparison results between two workflow stages.

Aggregate File

Each row corresponds to an asset. This table aggregates data from the Comparison File and summarizes the comparison result statistics.

Metric File

This file provides overall metrics for the comparison.

The CSV files contains following columns:

Comparison File

• Batch

• External ID

• Assignee

  • Evaluation Stage Assignee

  • Gold Stage Assignee

• Page

• Schema ID

• Object ID

• Tool Type

• Answer

  • Evaluation Stage Answer

  • Gold Stage Answer

• Description

  • Evaluation Stage Description

  • Gold Stage Description

• Comparison

Aggregate File

Metric File

Supported Data Types

• Compatible with all data types available in AngoHub.

Supported Annotation Tools

• Classifications - All

• Tools

  • Bounding Box

  • Rotated Bounding Box

  • Polygon

  • Segmentation

  • Entity

  • Polyline

  • Point

  • PDF

  • Nested Classifications

• Relations

  • Single

  • Group

Plugin Configuration

From the Stage Filter field, pick the stages containing the tasks you'd like to get the export of. Similarly, from the Batch Filter field, you may select one or more batches the tasks of which will be exported. By default, all tasks are exported from all batches.

If you wish to receive an email when the export is complete, toggle Send Email on.

You may vary a number of settings related to your export from the Config JSON field. Each option is detailed below:

{
  "gold_stage": null,
  "evaluation_stage": null,
  "start_date": "2020-12-31",
  "end_date": null,
  "ignored_schema_ids": [],
  "merge_output": false,
  "additional_attribute_name": null,
  "include_key_frames_only": true,
  "logging_frequency": 0
}

• "gold_stage": The stage used as the gold standard reference for comparison.

  • Example:

    • "gold_stage": "Review"

• "evaluation_stage": The stage whose annotations are evaluated against the gold standard.

  • Example:

    • "evaluation_stage": "Label"

• "start_date": Start date for filtering assets included in the comparison (YYYY-MM-DD).

  • Example:

    • "start_date": "2020-12-31"

• "end_date": End date for filtering assets included in the comparison (YYYY-MM-DD).

  • Example:

    • "end_date": "2021-12-31"

• "ignored_schema_ids": List of schema IDs to exclude from the comparison.

  • Example:

    • "ignored_schema_ids": ["12345", "12346"]

• "merge_output": Whether to merge comparison results into a single consolidated output.

  • Example:

    • "merge_output": true

    • "merge_output": false

• "additional_attribute_name": Attribute name used for object-level matching across stages.

  • Example:

    • "additional_attribute_name": "Object ID"

• "include_key_frames_only": Specifies whether only key frames should be included in the metrics calculation. (For video assets only)

  • Example:

    • "include_key_frames_only": true

    • "include_key_frames_only": false

• "logging_frequency": Defines how frequently progress logs are displayed; setting this value to 0 disables logging entirely, while any positive integer enables logging at the specified interval. For more information on how to view plugin logs, see here.

  • Example:

    • "logging_frequency": 0

    • "logging_frequency": 100