Labeling Queue

Order in which assets are shown to labelers on Ango Hub.

The labeling queue is the list of tasks that are distributed to labelers, in order.

If your project's workflow has multiple labeling stages, each stage will have its own queue.

Using the Labeling Queue

From the dashboard, click on the Start Labeling button on the top-right:

You will be shown the first labeling task that is available to you from one of the labeling queues. If your project has more than one labeling queue, the queue will be picked at random. Once you label it and click on Submit, you will be shown the next, and so on.

You may choose to label a certain task without following queue order by going to the Task tab and clicking on any task, or by clicking on a circle under the Labels column in the Assets tab.

Doing so does not invoke the labeling queue, and the Submit button will not be available to you. Instead, you will see the Save button, indicated by a floppy disk icon. This is because labeling and saving the asset will not move you forward one task. Essentially, you will only edit that one particular task.

Properties of the Labeling Queue

• The task at the top of the queue is the task that will be shown next to labelers in the stage.

• Each project has one labeling queue for each Label stage in your workflow. When labelers open a new task, they pick from the top of the queue the first task that is unassigned or assigned to them.

• All tasks in a Label stage are automatically part of a labeling queue.

• Tasks get added to the bottom of the queue as they are created. Thus, the queue is ordered by task creation time, first in first out.

• Tasks that have been assigned to user X (for example, X is currently labeling them) will be unavailable to Y. Y will be given the next task in the queue that is unassigned or assigned to them.

Whenever users press on the Start Labeling button on the top-right corner of the dashboard, they are shown the first task in the labeling queue. When they complete labeling and press on Submit, they pick another task from the top of the queue, and so on.

If the project has multiple labeling queues as a result of having more than one Label-type stage, clicking on Start Labeling will choose one of the labeling queues at random.

If you wish to label in a particular queue, click on the downward-facing chevron next to the Start Labeling button a pick a particular queue you'd like to work on.

Reordering the Labeling Queue

You may reorder the labeling (and review) queues to prioritize certain batches.

For example, say you have Batch_1, Batch_2, and Batch_3 in your project. While normally tasks are presented randomly from any batch, you may reorder tasks in such a way that tasks from Batch 1 are presented first, then when they are done, tasks from Batch 2, and so on.

Reordering reorders tasks simultaneously in all stages. Currently there is no way to reorder tasks in a single stage without using the priority SDK function.

To reorder all queues, navigate to the Workflow tab, then click on the arrow next to Sync Queues. Lastly, click on Re-order & Sync:

A popup will appear. Select the batches you'd like to prioritize, in order, then click on OK.

In this example, labelers and reviewers will be shown tasks from the batch named 1_4 first, then 4_1, and so on.

Now, when picking labeling or reviewing tasks from any queue, tasks from the batch you selected first will be shown, followed by the next, and so on.

If there are tasks which were already assigned to a user but not completed, belonging to a non-priority batch, those tasks will be shown first.

In short, this is the prioritization algorithm:

1. Assigned tasks from prioritized batches, in order of priority

2. Assigned tasks from non-prioritized batches

3. Non-assigned tasks from prioritized batches, in order of priority

4. Non-assigned tasks from non-prioritized batches