# Benchmark & Consensus Export
## Overview
The Benchmark & Consensus Export plugin is an export plugin designed to measure the agreement between multiple annotators and generate a comprehensive CSV report summarizing the results
It analyzes annotations across annotators, compares them using selected metrics, and produces CSV reports to support quality assessment, benchmarking, and adjudication workflows.
## Plugin Functionality
The Benchmark & Consensus Export plugin analyzes annotations produced by multiple annotators and generates structured CSV outputs for quality evaluation and benchmarking. The plugin is fully configurable using calculation modes and text comparison metrics.
* The plugin processes tasks for consensus generation or benchmark evaluation. In consensus mode, annotations from multiple annotators are compared to identify agreement and derive a consensus result. In benchmark mode, annotator outputs are evaluated against ground truth annotations to measure performance.
* Compares text-based annotations using a configurable evaluation metric. Supported metrics include: exact match, case-insensitive match and BLEU score.
* Allows specific annotation classes (schemas) to be excluded from consensus and benchmark calculations. Any schema IDs listed in ignored\_schema\_ids are ignored during metric computation and report generation.
* Generates detailed CSV reports summarizing annotator agreement, consensus results, and benchmark metrics.
### Supported Data Types
* Compatible with all data types available in AngoHub.
### Supported Annotation Tools
* Classifications
* Radio
* Checkbox
* Single-Dropdown
* Multi-Dropdown
* Tree Dropdown
* Text
* Nested Classifications
* Multiple Classifications
* Tools
* Bounding-Box
* Rotated Bounding-Box
* Polygon
* Polyline
* Segmentation
* Entity
* Point
* Brush
* Voxel Brush
{% hint style="danger" %}
Following tools are not supported by the plugin:
* PDF, Message, Angle
* Nested classifications within tools
* Relations
{% endhint %}
## Plugin Configuration
From the **Stage Filter** field, pick the stages containing the tasks you'd like to get the export of. Similarly, from the **Batch Filter** field, you may select one or more batches the tasks of which will be exported. By default, all tasks are exported from all batches.
If you wish to receive an email when the export is complete, toggle **Send Email** on.
You may vary a number of settings related to your export from the **Config JSON** field. Each option is detailed below:
```json
{
"mode": "consensus",
"text_metric": "exact_match",
"ignored_schema_ids": [],
"include_key_frames_only": false,
"logging_frequency": 0
}
```
* **"mode":** Determines what types of tasks the plugin will process.
* Options:
* "consensus"
* "benchmark"
* **"text\_metric":** Defines the metric used to compare text-based annotations.
* Options:
* "exact\_match"
* "case\_insensitive\_match"
* "bleu\_score"
* "**ignored\_schema\_ids":** Specifies which classes should be excluded from the consensus calculation.
* Example:
* "ignored\_schema\_ids": \["12345", "12346"]
* **"include\_key\_frames\_only":** Specifies whether only key frames should be included in the metrics calculation. (For video assets only)
* Example:
* "include\_key\_frames\_only": true
* "include\_key\_frames\_only": false
* **"logging\_frequency":** Defines how frequently progress logs are displayed; setting this value to 0 disables logging entirely, while any positive integer enables logging at the specified interval. For more information on how to view plugin logs, [see here](https://docs.imerit.net/plugins/introduction-to-plugins/monitoring-plugin-progress).
* Example:
* "logging\_frequency": 0
* "logging\_frequency": 100
