maml: Multi-annotator Machine Learning 👥

Author: Marek Herde

This project implements an ecosystem for multi-annotator learning approaches, aiming to learn from data with noisy annotations provided by multiple error-prone (mostly human) annotators. This research area is also often referred to as learning from crowds or learning from crowd-sourced labels. A graphical model describing this problem setting is given below.

Papers Employing maml 📃

• Marek Herde, Lukas Lührs, Denis Huseljic, and Bernhard Sick. Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension. In ECAI, 2024.
• Marek Herde, Denis Huseljic, Lukas Rauch, and Bernhard Sick. dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans. In NeurIPS, 2024.

Multi-annotator Learning Approaches 🤖

Multi-annotator learning approaches estimate annotators' performances for improving neural networks' generalization performances during training:

$$P(y_n, z_{nm} | x_n, a_m) = P(y_n | x_n) \times P(z_{nm} | x_n, a_m, y_n),$$

where

• $P(y_n | x_n)$ represents the class probabilities.
• $Pr(z_{nm} | x_n, a_m, y_n)$ represents the confusion matrix.

The approaches differ in their training and architectures to estimate these two quantities or proxies of them.

Approach	Authors	Venue (Year)	Annotator Performance Model	Training
dawid-skene	Dawid and Skene	J. R. Stat. (1979)	confusion matrix per annotator	em-algorithm + standard cross-entropy
cl	Rodrigues et al.	AAAI (2018)	noise adaption layer per annotator	cross-entropy
trace-reg	Tanno et al.	CVPR (2019)	confusion matrix per annotator	cross-entropy + regularization
conal	Chu et al.	AAAI (2021)	confusion matrix per and across annotators	cross-entropy + regularization
union-net	Wei et al.	TNNLS (2022)	noise adaption layer across annotators	cross-entropy
geo-reg-w	Ibrahim et al.	ICLR (2023)	confusion matrix per annotator	cross-entropy + regularization
geo-reg-f	Ibrahim et al.	ICLR (2023)	confusion matrix per annotator	cross-entropy + regularization
madl	Herde et al.	TMLR (2023)	confusion matrix per instance-annotator pair	cross-entropy + regularization
crowd-ar	Cao et al.	SIGIR (2023)	reliability scalar per instance-annotator pair	two-model cross-entropy
annot-mix	Herde et al.	ECAI (2024)	confusion matrix per instance-annotator pair	cross-entropy + mixup extension

Datasets 💾

Beyond implementing multi-annotator machine learning approaches, maml provides code to download and train with publicly available datasets annotated by multiple error-prone humans (e.g., crowdworkers). The table below lists the currently supported datasets, including their main characteristics. Some of these datasets also come with certain variants to emulate lower or higher levels of annotation noise, e.g., cifar10n, cifar100n, and dopanim.

Dataset	spc	mgc	labelme	cifar10h	cifar10n	cifar100n	dopanim
Authors	Rodrigues et al.	Rodrigues et al.	Rodrigues et al.	Peterson et al.	Wei et al.	Wei et al.	Herde et al.
Venue	PRL (2013)	PRL (2013)	AAAI (2018)	CVPR (2019)	ICLR (2022)	ICLR (2022)	NeurIPS (2024)
Data Modality	text	sound	image	image	image	image	image
Training Instances [#]	4,999	700	1,000	10,000	50,000	50,000	10,484
Validation Instances [#]	❌	❌	500	❌	❌	❌	750
Test Instances [#]	5,428	300	1,188	50,000	10,000	10,000	4,500
Classes [#]	2	10	8	10	10	100	15
Annotators [#]	203	42	59	2,571	747	519	20
Annotation Platform	AMT	AMT	AMT	AMT	AMT	AMT	LabelStudio
Annotator Metadata	❌	❌	❌	❌	❌	❌	✅
Annotation Times	❌	❌	❌	✅	✅	✅	✅
Soft Class Labels	❌	❌	❌	❌	❌	❌	✅
Annotations per Instance [#]	5.6	4.2	2.5	51.4	3.0	1.0	5.0
Annotations per Annotator [#]	137	67	43	200	201	96	2,602
Overall Accuracy [%]	78.9	56.0	74.0	94.9	82.3	59.8	67.3
Accuracy per Annotator [%]	77.1	73.3	69.2	94.9	82.1	55.6	65.6

Code Snippet 💻

The following code snippet exemplarily demonstrates how to train and test the multi-an:x:tator classifier crowd layer (cl) on the dataset music genres classification (mgc) via maml.

from lightning.pytorch import Trainer
from maml.data import MusicGenres
from torch import nn
from torch.utils.data import DataLoader

# Download the training, validation, and test dataset.
ds_train = MusicGenres(root=".", version="train", download=True)
ds_train = MusicGenres(root=".", version="test", download=True)

# Build data loaders.
dl_train = DataLoader(dataset=ds_train, batch_size=64, shuffle=True, drop_last=True)
dl_test = DataLoader(dataset=ds_train, batch_size=128)

# Define neural network architecture.
gt_embed_x = nn.Sequential(
      nn.Linear(in_features=124, out_features=128),
      nn.BatchNorm1d(num_features=neuron_list[i + 1]),
      nn.ReLU(),
)
gt_output = nn.Linear(in_features=128, out_features=ds_train.get_n_classes())

# Define optimizer with parameters for the ground truth (GT) model estimating class
# probabilities and the annotator performance (AP) model estimating annotators'
# performances.
optimizer = RAdam
optimizer_gt_dict = {"lr": 0.01}
optimizer_ap_dict = {"lr": 0.01}

# Build multi-annotator learning classifier `cl`.
clf = CrowdLayerClassifier(
        n_classes=ds_train.get_n_classes(),
        n_annotators= ds_train.get_n_annotators(),
        gt_embed_x=gt_embed_x,
        gt_output=gt_output,
        optimizer=optimizer,
        optimizer_gt_dict=optimizer_gt_dict,
        optimizer_ap_dict=optimizer_ap_dict,
)

# Train multi-annotator classifier.
trainer = Trainer(max_epochs=50, accelerator="gpu")
trainer.fit(model=clf, train_dataloaders=dl_train)

# Evaluate multi-annotator classifier after the last epoch.
trainer.test(dataloaders=dl_test)

Setup of Conda Environment 🐍

As a prerequisite, we assume to have a Linux distribution as operating system.

1. Download a conda version to be installed on your machine.
2. Setup the environment via

projectpath$ conda env create -f environment.yml

3. Activate the new environment

projectpath$ conda activate maml

4. Verify that the maml (multi-annotator machine learning) environment was installed correctly:

projectpath$ conda env list

Empirical Evaluation 📊

We provide scripts and Jupyter notebooks to benchmark and visualize multi-annotator machine learning approaches on datasets annotated by multiple error-prone annotators.

Experiments

The Python script for executing a single experiment is perform_experiment.py and the corresponding main config file is evaluation. In this config file, you also need to specify the mlruns_path defining the path, where the results will be saved via mlflow. Further, you can select the 'gpu' or 'cpu' as accelerator.

1. Before starting a single experiment or Jupyter notebook, check whether the dataset is downloaded. For example, if you want to ensure that the dataset dopanim is downloaded, update the download flag in its config file dopanim.yaml.
2. An experiment can then be started by executing the following commands

projectpath$ conda activate maml
projectpath$ cd empirical_evaluation/python_scripts
projectpath/empirical_evaluation/python_scripts$ python perform_experiment.py data=dopanim data.class_definition.variant="full" classifier=majority_vote seed=0

3. Since there are many possible experimental configurations, including repetitions with different seeds, you can create Bash scripts. As an example, you can reproduce the experiments of the dopanim paper by following the instructions in write_bash_scripts.py and then execute the following commands

projectpath$ conda activate maml
projectpath$ cd empirical_evaluation/python_scripts
projectpath/empirical_evaluation/python_scripts$ python write_bash_scripts.py

4. There is a bash script for the hyperparameter search, each dataset variant of the benchmark, and use cases. For example, executing the benchmark experiments for the variant full via SLURM can be done according to

projectpath$ conda activate maml
projectpath$ sbatch path_to_bash_scripts/dopanim_benchmark_full.sh

Results

Once an experiment is completed, its associated results can be loaded via mlflow. To get a tabular presentation of these results, you need to start the Jupyter notebook tabular_results.ipynb and follow its instructions.

projectpath$ conda activate maml
projectpath$ cd empirical_evaluation/jupyter_notebooks
projectpath/empirical_evaluation/jupyter_notebooks$ jupyter-notebook tabular_results.ipynb

Structure 🏛️

• empirical_evaluation: scripts to reproduce or adjust our empirical evaluation, including the benchmark and case studies
  • hydra_configs: collection of hydra config files for defining hyperparameters
    • architecture: config group of config files for network architectures
    • classifier: config group of config files for multi-annotator classification approaches
    • data: config group of config files for datasets
    • ssl_model: config group of config files for self-supervised learning models as backbones
    • experiment.yaml: config file to define the architecture(s), dataset, and multi-annotator classification approach for an experiment
  • jupyter_notebooks: Jupyter notebooks to analyze results
    • tabular_results.ipynb: Jupyter notebook to create the tables of results obtained after executing the experiments for the dataset dopanim
  • python_scripts: collection of scripts to perform experimental evaluation
    • perform_experiments.py: script to execute a single experiment for a given configuration
    • write_bash_scripts.py: script to write Bash or Slurm scripts for evaluation
• maml: Python package for multi-annotator machine learning consisting of several sub-packages
  • architectures: implementations of network architectures for the ground truth and annotator performance models
  • classifiers: implementations of multi-annotator machine learning approaches using pytorch_lightning modules
  • data: implementations of pytorch data sets with class labels provided by multiple, error-prone annotators
  • utils: helper functions, e.g., for visualization
• environment.yml: file containing all package details to create a conda environment

Trouble Shooting 🚨

If you encounter any problems, watch out for any TODO comments, which give hints or instructions to ensure the code's functionality. If the problems are still not resolved, feel free to create a corresponding GitHub issue or contact us directly via the e-mail marek.herde@uni-kassel.de