Workflow
Last updated
Last updated
Workflows allow you to create custom data labeling pipelines tailored to the specific needs of your project. This can range from simple single-stage labeling tasks to more intricate multi-stage processes involving , , , logical gates, random sampling, and more. All with a simple and intuitive UI requiring no coding or scripting.
This page will guide you through the various aspects of Workflow and its configuration options.
From your project, navigate to the Workflow tab. You'll be presented with the default workflow, which has a single labeling stage:
To add a new stage, for example to add a reviewing stage after the labeling stage, click on one of the stages from the row at the top and drag it on the main workflow panel. Once you release the left mouse button, the stage will be added to the workflow panel, and you can drag it around with the left mouse button:
Adding the stage to the panel is not enough, however: you'll need to plug it into your current workflow.
For example, if we want the review to happen between the Label and Complete stages, we will first plug the output of Label into the input of Review. Then, if we want to make it so that when a task is rejected in review it it sent to the Label stage again, and if it's accepted, it goes to Complete, here's how we would do it:
The small number at the top right of each stage indicates how many tasks are currently in the stage. (e.g., if there are 4 tasks in a label stage, it means that 4 tasks are waiting to be shown to annotators for labeling.)
To edit a stage, left-click on it. A dialog will appear in the top-left of the Workflow panel:
To rename a stage, click on the Pen icon next to the stage's name in the dialog.
By clicking on the three dots, a number of options will appear:
Delete deletes the selected stage.
List Tasks in Stage opens the Tasks tab with a filter, only displaying the tasks in the current stage.
Remove Edges removes all connections to and from the selected stage. To delete connections, you can also click on any one connection to highlight it and then press Del (⌫ on macOS) on your keyboard to achieve the same result.
Click on the red Save button in the top left of the screen to save your workflow.
You will not be able to save your workflow if:
a stage has unconnected inputs/outputs
a logic stage does not have a logical condition selected
a consensus stage has less than two labeling or plugin stages
a consensus stage has no threshold set up
From the Workflow tab, click on the "History" icon on the top-right to open a history of all workflow versions in the current project.
To view a previous version of this project's workflow, click on Preview. To restore it, click on Save while the version you need is previewed.
If, after changing your workflow, you click on Start Labeling or Start Reviewing and, for any reason, you get an error regarding a lack of tasks to annotate/review, you may use the "Syncing Queues" button to synchronize the state between workflow and tasks.
In short, syncing queues is something to be done as a first step only when troubleshooting errors. If there are no errors in your project, you will not need to use the button.
Here's more about each stage and their options.
In this default workflow, when you upload new , their tasks are sent to the Label stage, where they can be labeled by any annotator in your project. After the annotator submits their annotations, the labeling task is sent to Complete and is considered done and ready for export. This is the simplest possible workflow on Hub.
Bulk Re-queue gives you the option to all tasks currently in the selected stage. By re-queuing, you can send the tasks to any other stage of your choice. You can remove their annotations, their assigned annotator, and/or their issues. You may read more on re-queuing .
Additionally, each stage type has its own set of configurable options. You can read more about each stage's options in .
