# Benchmark & Consensus Export
## Overview
The Benchmark & Consensus Export plugin is an export plugin designed to measure the agreement between multiple annotators and generate a comprehensive CSV report summarizing the results
It analyzes annotations across annotators, compares them using selected metrics, and produces CSV reports to support quality assessment, benchmarking, and adjudication workflows.
## Plugin Functionality
The Benchmark & Consensus Export plugin analyzes annotations produced by multiple annotators and generates structured CSV outputs for quality evaluation and benchmarking. The plugin is fully configurable using calculation modes and text comparison metrics.
* The plugin processes tasks for consensus generation or benchmark evaluation. In consensus mode, annotations from multiple annotators are compared to identify agreement and derive a consensus result. In benchmark mode, annotator outputs are evaluated against ground truth annotations to measure performance.
* Compares text-based annotations using a configurable evaluation metric. Supported metrics include: exact match, case-insensitive match and BLEU score.
* Allows specific annotation classes (schemas) to be excluded from consensus and benchmark calculations. Any schema IDs listed in ignored\_schema\_ids are ignored during metric computation and report generation.
* Generates detailed CSV reports summarizing annotator agreement, consensus results, and benchmark metrics.
### Supported Data Types
* Compatible with all data types available in AngoHub.
### Supported Annotation Tools
* Classifications
* Radio
* Checkbox
* Single-Dropdown
* Multi-Dropdown
* Tree Dropdown
* Text
* Nested Classifications
* Multiple Classifications
* Tools
* Bounding-Box
* Rotated Bounding-Box
* Polygon
* Polyline
* Segmentation
* Entity
* Point
* Brush
* Voxel Brush
{% hint style="danger" %}
Following tools are not supported by the plugin:
* PDF, Message, Angle
* Nested classifications within tools
* Relations
{% endhint %}
## Plugin Configuration
From the **Stage Filter** field, pick the stages containing the tasks you'd like to get the export of. Similarly, from the **Batch Filter** field, you may select one or more batches the tasks of which will be exported. By default, all tasks are exported from all batches.
If you wish to receive an email when the export is complete, toggle **Send Email** on.
You may vary a number of settings related to your export from the **Config JSON** field. Each option is detailed below:
```json
{
"mode": "consensus",
"text_metric": "exact_match",
"ignored_schema_ids": [],
"include_key_frames_only": false,
"logging_frequency": 0
}
```
* **"mode":** Determines what types of tasks the plugin will process.
* Options:
* "consensus"
* "benchmark"
* **"text\_metric":** Defines the metric used to compare text-based annotations.
* Options:
* "exact\_match"
* "case\_insensitive\_match"
* "bleu\_score"
* "**ignored\_schema\_ids":** Specifies which classes should be excluded from the consensus calculation.
* Example:
* "ignored\_schema\_ids": \["12345", "12346"]
* **"include\_key\_frames\_only":** Specifies whether only key frames should be included in the metrics calculation. (For video assets only)
* Example:
* "include\_key\_frames\_only": true
* "include\_key\_frames\_only": false
* **"logging\_frequency":** Defines how frequently progress logs are displayed; setting this value to 0 disables logging entirely, while any positive integer enables logging at the specified interval. For more information on how to view plugin logs, [see here](/plugins/introduction-to-plugins/monitoring-plugin-progress.md).
* Example:
* "logging\_frequency": 0
* "logging\_frequency": 100
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.imerit.net/plugins/first-party-ango-plugins/benchmark-and-consensus-export.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
