Label Validation

Breaking Change Notice

Prior to Ango Hub 3.10, a toggle in the project settings allowed the project manager to decide whether the validations in the project prevented submission or not. This toggle was global for the entire project, meaning that all validations in the project either prevented submission or not.

Since Ango Hub 3.10, the toggle has been deprecated. Instead, the project manager can now decide whether each validation prevents submission or not by including a boolean named preventSubmission to the error. If true, the annotator will not be able to submit when the error is shown. If false, the annotator will be shown the option to submit the task anyway.

For validation code created before this change, or wherever the preventSubmission boolean is not present, we default to preventing submission.

You can show errors to labelers when they try to submit annotations that don't fit requirements of your choice, and if you so wish, you can prevent them from submitting their label entirely.

Ango Hub provides a fully customizable and programmatic way to validate annotations, allowing you to create JavaScript functions that hook directly into Ango Hub's code and run when annotators attempt to save or submit their annotations.

Label validation is run only when the labeler clicks on Save or Submit.

Label validation will not run unless you enable the Enable Custom Validation toggle in the Custom Validation section of the project settings:

Enable Label Validation

Navigate to the project where you'd like to enable label validation, then go to the Settings tab and enter the Label Validation section:

Enable Custom Validation: when enabled, validation will be activated for the current project.

In the code area below, you will enter your JavaScript function containing the logic of your label validation. You may click on Validate Code to check for syntax errors and try your code on sample labels.

Set Up Label Validation

In the code area of the Label Validation section, enter a JavaScript function that takes one parameter (we'll call it answers,) and returns a list of dicts with the errors you'd like to show. The function will be run every time an annotator tries to save or submit annotations.

Function Parameters

Ango Hub will pass one parameter to your function, a JSON object which contains all objects, relations, and classifications the annotator is trying to submit.

This is a sample 'answers' parameter you'll receive from Ango Hub. In this case, the user created two PDF areas, created a single relation between them, and replied to a classification with two answers:

{
  "tools": [
    {
      "pdf": {
        "position": {
          "boundingRect": {
            "x1": 157,
            "y1": 166.1374969482422,
            "x2": 200,
            "y2": 226.1374969482422,
            "width": 671.001473820641,
            "height": 670.9999999999999
          },
          "rects": [],
          "pageNumber": 1
        },
        "content": {
          "image": "data:image/png;base64="
        }
      },
      "objectId": "7e0bd6f41a01c78c15ae608",
      "classifications": [],
      "metadata": {
        "createdAt": 1674132659608,
        "createdBy": "614348d554e17400149964b1"
      },
      "schemaId": "801a2b8b8079b50d4541789"
    },
    {
      "pdf": {
        "position": {
          "boundingRect": {
            "x1": 106,
            "y1": 271.1374969482422,
            "x2": 135,
            "y2": 332.1374969482422,
            "width": 671.001473820641,
            "height": 670.9999999999999
          },
          "rects": [],
          "pageNumber": 1
        },
        "content": {
          "image": "data:image/png;base64="
        }
      },
      "objectId": "8083daa5d68a090aca6c738",
      "classifications": [],
      "metadata": {
        "createdAt": 1674132663738,
        "createdBy": "614348d554e17400149964b1"
      },
      "schemaId": "801a2b8b8079b50d4541789"
    }
  ],
  "classifications": [
    {
      "schemaId": "fe5af3c7471de99e3868448",
      "answer": [
        "3b935728b7f43b18b1f7553",
        "cb48de7b8c1a10908b2d747"
      ],
      "page": 0,
      "metadata": {
        "createdAt": 1674132625127,
        "createdBy": "614348d554e17400149964b1",
        "updatedAt": 1674132625364,
        "updatedBy": "614348d554e17400149964b1"
      }
    }
  ],
  "relations": [
    {
      "from": "7e0bd6f41a01c78c15ae608",
      "to": "8083daa5d68a090aca6c738",
      "objectId": "5ccee3e5af7eeb84b59a358",
      "schemaId": "c3ec9fbe6c52567a0dfb479",
      "direction": "straight",
      "metadata": {
        "createdAt": 1674132666358,
        "createdBy": "614348d554e17400149964b1"
      }
    }
  ]
}

For more on what can be passed in 'answers', we recommend following the steps in the next section. Following that, consulting the page Ango Annotation Format will also help, as it details the format in which annotations are exported, which is the same as what you'll receive as parameter.

Quickly previewing the 'answers' parameter

We developed a fast and efficient way to preview what will be passed to your function based on your needs:

1. Open the labeling editor and perform a test annotation on Hub with the tools and answers you need.

2. Without having to save or submit the annotation, from the three-dot menu in the top-right corner, click on Copy Answers:

   

You will have copied exactly what Ango Hub would have passed as parameter in your validation function based on the labels on the screen.

Function Returns

Your function will need to return a list of dictionaries, containing the ID of the object which is causing the error, an error message string, as well as whether the error should prevent the annotator from submitting the task or not.

This is what a sample return will look like:

[
  {
    objectId: "12345678",
    message: "A point has an X coordinate below 100.",
    preventSubmission: false
  },
  {
    objectId: "12345690",
    message: "Maximum 3 answers allowed.",
    preventSubmission: true
  }
]

Testing and Debugging your Validation Code

From Ango Hub, you can test and debug your function without having to perform labeling and submitting test annotations every time.

1. Obtain a sample of what will be passed into your function by following the steps in this section.

2. From your project dashboard, enter the Settings tab, then the Label Validation section. In the code area, enter your validation function.

3. To test and debug your code, click on the Validate Code button below the code area:

A dialog will open:

5. Paste the input parameter you have copied in Step 1, and click on Run.

Ango Hub will scan your function for syntax errors, and show them to you if present.

If your function successfully returned errors based on the annotations JSON you've just passed it as parameter, Ango Hub will show them to you below the code area:

By moving back and forth between writing your function and the validation dialog, you can create a quick build-debug workflow allowing you to create and test your functions quickly.

Code Examples

Give an error if a point exists with its X coordinate under 100:

function (answers) {
  let errors = [];

  const points = answers.objects.filter((obj) => obj.tool === "point" && obj["point"][0] < 100);

  points.map((b) => {
    errors.push(
      {
        objectId: b.objectId,
        message: "Point X under 100.",
        preventSubmission: false
      }
    );
  })

  return errors;
}

Give an error if an empty group relation exists:

function (answers) {
  let errors = [];

  const groupRelation = answers.relations.find((r) => r.tool === "group");

  if (groupRelation?.group.length > 0) {
    errors.push({
      objectId: groupRelation.objectId,
      message: "Group relation cannot be empty.",
      preventSubmission: false
    });
  }

  return errors;
}

Give an error if a bounding box is narrower than 100px:

function (answers) {
  let errors = [];

  const bbs = answers.objects.filter((obj) => obj.tool === "bounding-box" && obj["bounding-box"]["width"] < 100);

  bbs.map((b) => {
    errors.push(
      {
        objectId: b.objectId,
        message: "BB is narrower than 100px.",
        preventSubmission: false
      }
    );
  })

  return errors;
}

Give an error if a group relation you specify contains objects with names different from the ones you specify:

function (answers) {
  let errors = [];

  groupRelationToCheck = "TITLE OF GROUP RELATION TO CHECK"

  const acceptableObjectTypes = [
    "TITLE OF ACCEPTABLE OBJECT 1",
    "TITLE OF ACCEPTABLE OBJECT 2"
  ]

  const groupRelations = answers.relations.filter((r) => r.tool === "group" && r.title === groupRelationToCheck);

  groupRelations.forEach(groupRelation => {
    groupRelation.group.forEach(object => {
      if (!acceptableObjectTypes.includes(object.title)) {
        errors.push({
          objectId: groupRelation.objectId,
          message: `A ${groupRelationToCheck} group relation contains a ${object.title} object.`,
          preventSubmission: false
        })
      }
    })
  })

  return errors;
}

Give an error if a group relation you specify contains more than one object from the ones you specify:

function (answers) {
  let errors = [];

  groupRelationToCheck = "TITLE OF GROUP RELATION TO CHECK"

  const acceptableObjectTypes = [
    "TITLE OF ACCEPTABLE OBJECT 1",
    "TITLE OF ACCEPTABLE OBJECT 2"
  ]

  const groupRelations = answers.relations.filter((r) => r.tool === "group" && r.title === groupRelationToCheck);

  groupRelations.forEach(groupRelation => {
    acceptableObjectTypes.forEach(objectType => {
      matchingObjects = groupRelation.group.filter(object => object.title === objectType)
      if (matchingObjects.length > 1) {
        errors.push({
          objectId: groupRelation.objectId,
          message: `A group relation contains more than one ${objectType} object.`,
          preventSubmission: false
        })
      }
    })
  })

  return errors;
}