Dataset Cleaning Redefined: Harnessing Human Intelligence to increase quality of Objects365
Discover a new approach to dataset cleaning. Learn how human intelligence enhances Objects365, elevating data quality for your projects.
Introduction
The Intuition Machines labeling framework integrates our labeling workflow generator (which allows you to construct complex labeling workflows from smaller atomic components) with third-party labeling workforces/solutions. You can choose to add labels in-house, bring your own workforce, or work with BPO partners already on our platform.
Dataset construction is mediated by best-in-class machine learning which builds ML models that can quickly and accurately label datasets at reduced cost. Our framework uses an active learning loop to iteratively decide which samples need human labels and which samples can be labeled by the model.
This approach scales well and reduces cost and time to label, since human labels are only used sparingly. In this series of blog posts we’ll introduce the different features of our framework through examples of cleaning/improving an existing public dataset - Objects365. Each blog post will cover a different scenario, starting with easy and moving to more complex use cases.
1. Importance of Dataset Cleaning
The first step to building a good machine learning model is to have high quality data. Poor data quality can result in models that give wrong insights, misleading results, or poor model performance. Dataset cleaning is the process of ensuring the highest possible reliability and overall integrity of the dataset and involves various curation tasks such as identifying and rectifying errors, inconsistencies or inaccuracies in the dataset, or standardization and normalization of the data.
2. Introduction to Objects365 Dataset
The Objects365 dataset, a widely used dataset in computer vision, is a large-scale set of images for object detection. Currently, it contains over 2 million pictures in 365 different object categories. Each annotation in the dataset includes a class label and the bounding box coordinates of the object. Introduced in 2019, the Objects365 dataset serves as the next step in object detection benchmarks following PASCAL VOC and COCO datasets (Shao et al., 2019, 8430). It is significantly larger than both PASCAL VOC and COCO and encompasses the same classes within its 365 different object categories.
All large datasets are prone to error and, after some investigation, we have identified multiple mistakes in the Objects365 dataset. Our goal is to rectify these mistakes using our labeling tool to improve the overall quality of the dataset and enhance both model training and performance. Additionally, we would like to show how machine learning models can be combined with human answers to produce a human-in-the-loop solution that can be used to label or clean dataset annotations faster and cheaper than all-human solutions.
3. Approaches to Data Cleaning
In the realm of dataset cleaning for our case, the primary task is correcting inaccurate bounding boxes in the dataset.
To tackle these tasks, we employ our Model-in-the-Loop (MITL) tool. Through MITL, we harness the power of active learning, which synergistically combines machine learning models with human insights. When the models confront ambiguous images, human feedback will be used to support the models’ training process.
For sourcing human work we will use the HUMAN Protocol, a decentralized blockchain platform for organizing and compensating labor. It is especially useful for tasks such as data labeling for machine learning. A job is posted on the blockchain with funds held in escrow, workers do the job, their work is validated by other workers and upon approval, they are paid proportional to the accuracy of their work, as determined by human consensus and other methods.
With MITL, we can use this work in a smarter way. Computer Vision models are used to label all datasets, then we pick up the hardest cases and only a smaller subset of the whole dataset is presented to humans to label those images. Next, MITL's models leverage this information and label all images with better accuracy than before. This approach will be crucial for cleaning the Objects365 dataset, considering its enormity. If we were to label millions of images solely through human labor, it would be a long and costly process. However, by combining humans with models via MITL, we can effectively scale the cleaning and labeling process across all images in Objects365.
4. Implementation of Data Cleaning
For this blog post, we have created a small subset of images from the Objects365 dataset. The focus of this labeling exercise is to determine whether an image contains a dog or not, following the classic "Dog or not" labeling approach. To generate this subset, we extracted 1000 bounding boxes of dogs and 1000 bounding boxes of non-dogs from the Objects365 annotations. As a result, we now have a total of 2000 images that require labeling.
The images in this subset were randomly selected, and we have already identified instances where incorrect labels are present.
4.1. Factored Cognition (Workflow Orchestrator)
To orchestrate the cleaning process, we use a tool called Factored Cognition. It is part of our MITL solution and its role is to orchestrate the launching of multiple labeling jobs and aggregating the results. Utilizing Factored Cognition is straightforward, requiring only minimal configuration through sending POST requests to the relevant endpoints.
4.2. The Process of Cleaning the Objects365 Dataset
Let's dive into the specifics of applying our chosen data cleaning method to the Objects365 dataset.
In order to launch a workflow, we first need to create a manifest. Essentially, a manifest is a detailed description of the job we're aiming to launch.
We'll be launching the workflow consisting of two distinct jobs. The first one will be completed via HUMAN Protocol, where all images will be labeled by people. The second job is an MITL (Model In The Loop) job, where a portion of the images will be labeled by people, while the rest will be processed through MITL’s model.
For these jobs, the type we'll be using is "image_label_binary", which means binary classification. The widget for this job looks like this:
The manifest includes the following sections:
- "question": This is a clear and concise question that appears at the top of the widget.
- "labels": This section includes example images as references, which guide the user in providing the expected answer.
- "groundtruth_uri": This is a URI that references the 'ground truth' images. The file contains URI’s for dog bounding box crops classified as "true" and for other classes classified as "false". An example file looks like this:
Verify the labeling jobs status:
And that's it. Usually jobs take some time to be finished, but after the job is completed, we can analyze the results.
5. Visualization and Comparison of the Original vs Cleaned Dataset
After implementing our label cleaning method, we can visualize the results and compare the original and cleaned Objects365 datasets. Data visualization plays a vital role in understanding the improvements achieved through the cleaning process.
We have one workflow where all the images were labeled by humans. One of them with only a portion of the images being labeled by humans and rest being taken care of MITL’s model.
We can quickly analyze the results by sending a post request to the relevant endpoint.
Found 114 annotations to fix in HUMAN Protocol routed labels.
Found 76 annotations to fix in MITL labels.
Out of the 2000 images this workflow had, 114 (5.7%) was labeled as “non-dog” by labelers via HUMAN Protocol, while 76 (3.8%) of the images were labeled as “non-dog” by MITL’s model. All of the images below were labeled as “Dog” in original Objects365 annotations, while humans and MITL decided that they were not dogs.
5.1. Results of the human job.
5.2. Results of the MITL job.
Conclusion
In this first blog post, we demonstrated a method to detect incorrectly labeled images within a dataset. Our approach utilizes two distinct workflows. One relies solely on the discerning prowess of human labelers, while the other combines the power of human labelers with the advanced capabilities of the Model-In-The-Loop (MITL) model.
This workflow structure can be easily scaled to clean up the entirety of a large dataset like Objects365, which currently contains 2 million images.
In the next blogpost in this series, we dive deeper into the capabilities of Workflow Orchestrator with a feature called “Job Chaining”, where you can compose workflows with jobs interacting with each other as well as a new job type called “Image Label Multiple Choice”.
The code and dataset used for this blog post can be found here. If you wish to learn more about labeling in detail, check out our website! Also if you wish to use our services, feel free to reach out to our support team at https://www.imachines.com.
References
Shao, S., Zhao, Z., Li, B., Xiao, G., Yu, G., Zhang, X., & Sun, J. (2019). Objects365: A Large-scale, High-quality Dataset for Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
