Workflow
Last updated
Last updated
Workflows allow you to create custom data labeling pipelines tailored to the specific needs of your project. This can range from simple single-stage labeling tasks to more intricate multi-stage processes involving reviews, webhooks, plugins, logical gates, random sampling, and more. All with a simple and intuitive UI requiring no coding or scripting.
This page will guide you through the various aspects of Workflow and its configuration options.
From your project, navigate to the Workflow tab. You'll be presented with the default workflow, which has a single labeling stage:
In this default workflow, when you upload new assets, their tasks are sent to the Label stage, where they can be labeled by any annotator in your project. After the annotator submits their annotations, the labeling task is sent to Complete and is considered done and ready for export. This is the simplest possible workflow on Hub.
To add a new stage, for example to add a reviewing stage after the labeling stage, click on one of the stages from the row at the top and drag it on the main workflow panel. Once you release the left mouse button, the stage will be added to the workflow panel, and you can drag it around with the left mouse button:
Adding the stage to the panel is not enough, however: you'll need to plug it into your current workflow.
For example, if we want the review to happen between the Label and Complete stages, we will first plug the output of Label into the input of Review. Then, if we want to make it so that when a task is rejected in review it it sent to the Label stage again, and if it's accepted, it goes to Complete, here's how we would do it:
The position of the stages on the Workflow panel is merely cosmetic — functionality-wise, only the connections between stages matter.
If you add a stage to the Workflow panel but then do not connect it to your workflow, the stage will not be active. Ensure you connect all stages you intend to add to your workflow.
The small number at the top right of each stage indicates how many tasks are currently in the stage. (e.g., if there are 4 tasks in a label stage, it means that 4 tasks are waiting to be shown to annotators for labeling.)
To edit a stage, left-click on it. A dialog will appear in the top-left of the Workflow panel:
To rename a stage, click on the Pen icon next to the stage's name in the dialog.
By clicking on the three dots, a number of options will appear:
Delete deletes the selected stage.
Remove Edges removes all connections to and from the selected stage. To delete connections, you can also click on any one connection to highlight it and then press Del (⌫ on macOS) on your keyboard to achieve the same result.
Bulk Re-queue gives you the option to re-queue all tasks currently in the selected stage. By re-queuing, you can send the tasks to any other stage of your choice. You can remove their annotations, their assigned annotator, and/or their issues. You may read more on re-queuing here.
Additionally, each stage type has its own set of configurable options. You can read more about each stage's options in this section.
Click on the red Save button in the top left of the screen to save your workflow.
You will not be able to save your workflow if:
a stage has unconnected inputs/outputs
a logic stage does not have a logical condition selected
a consensus stage has less than two labeling or plugin stages
a consensus stage has no threshold set up
If, after changing your workflow, you click on Start Labeling or Start Reviewing and, for any reason, you get an error regarding a lack of tasks to annotate/review, you may use the "Syncing Queues" button to synchronize the state between workflow and tasks.
In short, syncing queues is something to be done as a first step only when troubleshooting errors. If there are no errors in your project, you will not need to use the button.
Here's more about each stage and their options.
StartLabelReviewConsensusLogicPluginWebhookComplete
