Rank

Tool to rank different options in Ango Hub. Ideal for RLHF projects.

The Rank tool allows you to have annotators rank different options. These options may include text, images, videos, or any custom HTML.

This tool is often used in RLHF (Reinforced Learning from Human Feedback) projects, where multiple LLM answers need to be ranked and/or rated.

How to add a Rank tool to your project

From your project's Settings tab, enter the Category Schema section. Click on Add Category, then click on Rank. This will add a Rank tool to your project.

By itself, this will not populate ranking options – it will only display to users the option titles. For example, if, in the project set up in the GIF above, we were to upload an asset normally, without including context data, the rank tool would appear this way:

To include text, or any HTML in the ranking options itself, please follow the instructions in the Populating Ranking Options section below.

Populating Ranking Options

Automatic Ordering

To enrich ranking options to include content of any HTML-supported type, you must upload your asset using the JSON Import option and by including contextData in the format below:

[
  {
    "externalId": "your-asset-external-id",
    "data": "What are the main features of the Python programming language?",
    "contextData": {
      "responses": [
        "Python is renowned for its simplicity and readability, which make it an excellent choice for beginners. It supports multiple programming paradigms, including procedural, object-oriented, and functional programming.",
        "Python has a vast ecosystem of libraries and frameworks, such as NumPy for numerical calculations, Pandas for data manipulation, and Django for web development, making it versatile for various applications.",
        "Python is an interpreted language, which means that it is executed line-by-line which can simplify debugging and testing. It also seamlessly integrates with other languages, allowing for more flexibility in software development."
      ]
    }
  },
  { ... }
]

The value of data may be plain text, any HTML, or a URL to a file.

The values in the responses array may be plain text or any HTML.

When a JSON is imported this way, Ango Hub will make the assumption that the ranking options are presented in order. So for example, if, in your category schema, you have created three ranking options "Response 1", "Response 2", and "Response 3", Ango Hub will assume that in the responses list, the first item is "Response 1", the second "Response 2", and so on.

Manual Ordering

If the responses in the JSON are not ordered, or you wish to maintain one order in the JSON and another order on Ango Hub, you may alternatively import the JSON with the following format:

{
    "externalId": "Test Ranking Asset 2",
    "data": "What are the main features of the Python programming language?",
    "contextData": {
      "responses": [
        {
          "id": "Response 1",
          "label": "Python is renowned for its simplicity and readability, which make it an excellent choice for beginners. It supports multiple programming paradigms, including procedural, object-oriented, and functional programming."
        },
        {
          "id": "Response 2",
          "label": "Python has a vast ecosystem of libraries and frameworks, such as NumPy for numerical calculations, Pandas for data manipulation, and Django for web development, making it versatile for various applications."
        }
      ]
    }
  }

Where id is the name of the option as created in the Category Schema section of the project settings, and label is the content of the ranking option.

Once ranking options are populated, they will display the following way: