[NeurIPS 2025] NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception

Repo Feature

• Modality Support

  • LiDAR
  • Camera
  • LiDAR + Camera

• Heterogeneity Support

  • Sensor Data Heterogeneity: We have multiple LiDAR data (16/32/64-line) and camera data (w./w.o. depth sensor) in the same scene.
  • Modality Heterogeneity: You can assign different sensor modality to agents in the way you like!
  • Model Heterogeneity: You can assign different model encoders (together with modality) to agents in the way you like!

• Dataset Support

  • OPV2V
  • V2XSet
  • V2X-Sim 2.0
  • DAIR-V2X-C

• Detector Support

  • PointPillars (LiDAR)
  • SECOND (LiDAR)
  • Pixor (LiDAR)
  • VoxelNet (LiDAR)
  • Lift-Splat-Shoot (Camera)

• multiple heterogeneous collaborative perception methods

  • STAMP [ICLR2025]
  • MPDA [ICDCS2022]
  • PnPDA [ECCV2024]

Data Preparation

• OPV2V: Please refer to this repo.
• OPV2V-H: We store our data in Huggingface Hub. Please refer to Downloading datasets tutorial for the usage.
• V2XSet: Please refer to this repo.
• V2X-Sim 2.0: Download the data from this page. Also download pickle files from google drive.
• DAIR-V2X-C: Download the data from this page. We use complemented annotation, so please also follow the instruction of this page.

Note that you can select your interested dataset to download. OPV2V and DAIR-V2X-C are heavily used in this repo, so it is recommended that you download and try them first.

Create a dataset folder under NegoCollab and put your data there. Make the naming and structure consistent with the following:

NegoCollab/dataset

. 
├── DAIR_V2X_C_Complemented
│   ├── cooperative
│   ├── infrastructure-side
│   ├── vehicle-side
│   ├── train.json
│   └── val.json
├── OPV2V
│   ├── additional
│   ├── test
│   ├── train
│   └── validate
├── OPV2V_Hetero
│   ├── test
│   ├── train
│   └── validate
├── V2XSET
│   ├── test
│   ├── train
│   └── validate
├── V2V4Real
│   ├── test
│   ├── train
│   └── validate

Installation

Step 1: Basic Installation

conda create -n ngcb python=3.8
conda activate ngcb
# install pytorch. Cudatoolkit 11.3 are tested in our experiment.
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
# install dependency
pip install -r requirements.txt
# install this project. It's OK if EasyInstallDeprecationWarning shows up.
python setup.py develop

Step 2: Install Spconv (1.2.1 or 2.x)

We use spconv 1.2.1 to generate voxel features.

To install spconv 1.2.1, please follow the guide in https://github.com/traveller59/spconv/tree/v1.2.1.

Tips for installing spconv 1.2.1:

1. make sure your cmake version >= 3.13.2
2. CUDNN and CUDA runtime library (use nvcc --version to check) needs to be installed on your machine.

Step 3: Bbx IoU cuda version compile

Install bbx nms calculation cuda version

python opencood/utils/setup.py build_ext --inplace

Step 4: Install pypcd by hand for DAIR-V2X LiDAR loader.

pip install git+https://github.com/klintan/pypcd.git

Step 5: Dependencies for FPV-RCNN (optional)

Install the dependencies for fpv-rcnn.

cd NegoCollab
python opencood/pcdet_utils/setup.py build_ext --inplace

---

To align with our agent-type assignment in our experiments, please make a copy of the assignment file under the logs folder

# In NegoCollab directory
mkdir opencood/logs
cp -r opencood/modality_assign opencood/logs/heter_modality_assign

New Style Yaml and Old Style Yaml

We introduced identifiers such as m1, m2, ... to indicate the modalities and models that an agent will use.

However, yaml files without identifiers like m1 (if you are familiar with the CoAlign repository) still work in this repository. For example, PointPillar Early Fusion.

Note that there will be some differences in the weight key names of their two models' checkpoint. For example, training with the m1 identifier will assign some parameters's name with prefix like encoder_m1., backbone_m1, etc. But since the model structures are the same, you can convert them using the rename_model_dict_keys function in opencood/utils/model_utils.py.

Agent type identifier

• The identifiers like m1, m2 in opv2v_4modality.json are used to assign agent type to each agent in the scene. With this assignment, we ensure the validation scenarios for all methods are consistent and fixed. To generate these json files, you can refer to heter_utils.py.

• The identifiers like m1, m2 in ${METHOD}.yaml are used to specify the sensor configuration and detection model used by this agent type (like m2 in the case of camera_pyramid.yaml).

In ${METHOD}.yaml, there is also a concept of mapping_dict. It maps the given agent type of opv2v_4modality.json to the agent type in the current experiment. As you can see, camera_pyramid.yaml is a homogeneous collaborative perception setting, so the type of all agents should be the same, which can be referred to by m2.

Just note that mapping_dict will not take effect during the training process to introduce more data augmentation. Each agent will be randomly assigned an agent type that exists in the yaml.

Quick Start

The trained model files for NegoCollab, STAMP, MPDA-P, and PnPDA-P has uploaded to the Hugging Face repository. Create a checkpoints directory under the NegoCollab/ folder, extract the downloaded model files checkpoints.zip into this checkpoints directory, then you can refer to the bash code in the corresponding script files for testing.

NegoCollab inference_nego.sh checkpoints/zz-opv2v_nego_from_m1m2
STAMP inference_stamp.sh checkpoints/zz-opv2v_stamp_from_m0
MPDA-P inference_protocol.sh checkpoints/zz-opv2v_mpda_protocol_from_m0
PnPDA-P inference_protocol.sh checkpoints/zz-opv2v_pnpda_protocol_from_m0

Training Process

Step 1: Homogeneous Collaborative Training.

Suppose you are now in the NegoCollab/ folder. Then train the homogeneous collaboration models for m1, m2, m3, and m4 separately. The following bash script demonstrates the training process for the m1 homogeneous model, while the training methods for m2, m3, and m4 are similar.

# Create Directory
model_dir="checkpoints/opv2v_single_pyramid_m1"
yaml_file="opencood/hypes_yaml/opv2v/Homo/m1_pointpillar_pyramid.yaml"

mkdir "${model_dir}"
cp "${yaml_file}" "${model_dir}/config.yaml"

# Start Training
python opencood/tools/train.py \
-y $yaml_file \
--model_dir $model_dir

Step 2: Initial Alliance Negotiation.

stage 1: Common Representation Negotiate

After completing the training of homogeneous collaboration model, we negotiate the common representation from the initial collaboration alliance. Taking the initial alliance formed by m1 and m2 as an example, first create the negotiation training folder checkpoints/opv2v_nego_pyramid_m1m2, then copy the trained homogeneous collaboration model files of m1 and m2 from the first step into this folder, rename them as stage0_mx_collab, and run train_nego_onlytrain.py to start training.

# Create Directory
model_dir="checkpoints/opv2v_nego_pyramid_m1m2"
yaml_file='opencood/hypes_yaml/opv2v/NGCB/stage1/m1m2_pyramid.yaml'

mkdir "${model_dir}"
cp "${yaml_file}" "${model_dir}/config.yaml"

# Copy homogeneous collaborative models
cp checkpoints/opv2v_single_pyramid_m1 checkpoints/opv2v_nego_pyramid_m1m2/stage0_m1_collab
cp checkpoints/opv2v_single_pyramid_m2 checkpoints/opv2v_nego_pyramid_m1m2/stage0_m2_collab

# Start Training
python opencood/tools/train_nego_onlytrain.py \
-y None \
-s nego \
--model_dir $model_dir \
--val True \
--gpu_for_test 1

stage 2: Task Adaption

After completing the common representation negotiation, we adjust the receivers' parameters using the collaboration task loss. First, create an adaptation training folder checkpoints/opv2v_nego_pyramid_m1m2-ft, then copy the stage0_mx_collab and the best-performing AP model file from the stage1 training folder into this adaptation folder for continued training, enabling the model to adapt to downstream tasks.

# Create Directory
model_dir="checkpoints/opv2v_nego_m1m2_pyramid-ft"
yaml_file='opencood/hypes_yaml/opv2v/NGCB/stage1/m1m2_pyramid.yaml'

mkdir "${model_dir}"
cp "${yaml_file}" "${model_dir}/config.yaml"

# Copy the trained models of stage 1
cp checkpoints/opv2v_nego_pyramid_m1m2/stage0_m1_collab checkpoints/opv2v_nego_pyramid_m1m2-ft/stage0_m1_collab
cp checkpoints/opv2v_nego_pyramid_m1m2/stage0_m2_collab checkpoints/opv2v_nego_pyramid_m1m2-ft/stage0_m2_collab
cp checkpoints/opv2v_nego_pyramid_m1m2/net_epoch15.pth checkpoints/opv2v_nego_pyramid_m1m2-ft

# Start Training
python opencood/tools/train_nego_onlytrain.py \
-y None \
-s ft \
--model_dir $model_dir \
--val True \
--gpu_for_test 1

Step 3: New Agent Alignment.

stage 1: Align to the Common Representaion

After completing the agent training within the initial alliance, we proceed to train new agents. The training utilizes the negotiator obtained from stage1 in step2, along with the sender and receiver from the initial alliance agents, to facilitate the alignment of the new agent's local representations to the common representations through the sender.

Taking the training for new agent m3 as an example, first use find_best_ap50_and_save_bak.py to extract and save the negotiator network from the best-performing epoch during negotiation into the negotiator directory, while saving the trained sender and receiver parameters of m1 and m2 into stage0_mx_collab. Then create a new agent alignment training folder, copy the negotiator, stage0_mx_collab, and rename the homogeneous collaboration model file of m3 as stage0_m3_collab into the alignment training directory.

# Save Negotiator, senders and receivers
python opencood/tools/find_best_ap50_and_save_bak.py \
--eval_epoch 5(bestap) \
--model_dir checkpoints/opv2v_nego_m1m2_pyramid

# Create Directory
model_dir="checkpoints/opv2v_nego_m1m2_guide_m3_pyramid"
yaml_file='opencood/hypes_yaml/opv2v/NGCB/stage2/m1m2_guide_m3_pyramid.yaml'

mkdir "${model_dir}"
cp "${yaml_file}" "${model_dir}/config.yaml"

# Copy the negotiator, sender and receiver from models trained in stage2.
cp checkpoints/opv2v_nego_m1m2/stage0_m1_collab checkpoints/opv2v_nego_m1m2_guide_m3_pyramid
cp checkpoints/opv2v_nego_m1m2/stage0_m2_collab checkpoints/opv2v_nego_m1m2_guide_m3_pyramid
cp checkpoints/opv2v_nego_m1m2/negotiator checkpoints/opv2v_nego_m1m2_guide_m3_pyramid

# Copy homogeneous collaborative models of m3
cp checkpoints/opv2v_single_pyramid_m3 checkpoints/opv2v_nego_m1m2_guide_m3_pyramid

python opencood/tools/train_nego_onlytrain.py \
-y None \
-s align \
--model_dir $model_dir \
--val True \
--gpu_for_test 0

stage 2: Task Adaption

After the new agent aligns with the common representations, the receiver parameters are fine-tuned using the collaboration task loss.

First, create an adaptation training folder checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft. Then copy the best-performing model (by AP value) obtained from stage1 training into this adaptation folder to commence training.

# Create Directory
model_dir="checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft"
yaml_file='opencood/hypes_yaml/opv2v/NGCB/stage2/m1m2_guide_m3_pyramid.yaml'

mkdir "${model_dir}"
cp "${yaml_file}" "${model_dir}/config.yaml"

# Copy models trained in stage1
cp checkpoints/opv2v_nego_m1m2_guide_m3_pyramid/stage0_m1_collab checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft/stage0_m1_collab
cp checkpoints/opv2v_nego_m1m2_guide_m3_pyramid/stage0_m2_collab checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft/stage0_m2_collab
cp checkpoints/opv2v_nego_m1m2_guide_m3_pyramid/stage0_m3_collab checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft/stage0_m3_collab

cp checkpoints/opv2v_nego_m1m2_guide_m3_pyramid/net_epoch15.pth checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft

# Start Training
python opencood/tools/train_nego_onlytrain.py \
-y None \
-s ft \
--model_dir $model_dir \
--val True \
--gpu_for_test 0

Inference

During inference, the agents are equipped with the sender and receiver after the training phase of task adaptation, and the heterogeneous collaborative performance in each scenario is tested.

After completing downstream task adaptation training for both the initial alliance and new agents, first execute find_best_ap50_and_save_bak.py to save the parameters of senders and receivers to the stage0_mx_collab directory. Then create an inference directory, copy both the scenario-specific yaml configuration files and all participating agents' stage0_mx_collab files into it, and run inference_nego.py to conduct the inference test.

Taking m1m2m3m4 collaborative inference as an example, the testing procedure is as follows:

# Save the senders and receivers after the task adaptation training phase.
python opencood/tools/find_best_ap50_and_save_bak.py \
--eval_epoch 5(bestap) \
--model_dir checkpoints/opv2v_nego_m1m2_pyramid-ft

python opencood/tools/find_best_ap50_and_save_bak.py \
--eval_epoch 5(bestap) \
--model_dir checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft

python opencood/tools/find_best_ap50_and_save_bak.py \
--eval_epoch 5(bestap) \
--model_dir checkpoints/opv2v_nego_m1m2_guide_m4_pyramid-ft

# Create Directory
model_dir="checkpoints/opv2v_nego_m1m2m3m4_pyramid-ft"
yaml_file='opencood/hypes_yaml/opv2v/NGCB/inference/m1m2m3m4_pyramid.yaml'

mkdir "${model_dir}"
cp "${yaml_file}" "${model_dir}/config.yaml"

# Copy the agent models with senders and receivers to the inference directory
cp checkpoints/opv2v_nego_m1m2_pyramid-ft/stage0_m1_collab checkpoints/opv2v_nego_m1m2m3m4_pyramid-ft/stage0_m1_collab
cp checkpoints/opv2v_nego_m1m2_pyramid-ft/stage0_m2_collab checkpoints/opv2v_nego_m1m2m3m4_pyramid-ft/stage0_m2_collab
cp checkpoints/opv2v_nego_m1m2_guide_m3_pyramid-ft/stage0_m3_collab checkpoints/opv2v_nego_m1m2m3m4_pyramid-ft/stage0_m2_collab
cp checkpoints/opv2v_nego_m1m2_guide_m4_pyramid-ft/stage0_m4_collab checkpoints/opv2v_nego_m1m2m3m4_pyramid-ft/stage0_m4_collab

# Start Inference
python opencood/tools/inference_nego.py \
--model_dir $model_dir