Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs

This repository contains the code for the experiments of our VLM circuit overlap paper and the project website.

Repository structure

data: Contains generated prompt csv files for each model and task. These files can also be generated automatically by the scripts and code. Each task directory also contains an images.tar.gz file that hosts all images on git-lfs.
data_generation: Contains scripts to generate the images (that are found in the data folder).
third_party: Contains a fork of the TransformerLens library with edits we made to make it support VLMs (specifically the VLMs we analyzed as well as Llama3.2).
All script files (script_.*.py) contain the code for running the experiments described in the paper as GPU jobs.
Other files contain processes used in experiments (attribution patching, faithfulness evaluations, alignment between modalities, circuit overlap measurements, utility functions for each task, etc).
docs contains code for the project website.

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
data		data
data_generation		data_generation
docs		docs
third_party/TransformerLens/transformer_lens		third_party/TransformerLens/transformer_lens
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
activation_patching.py		activation_patching.py
analysis_utils.py		analysis_utils.py
arithmetic_utils.py		arithmetic_utils.py
attr_patching.py		attr_patching.py
clevr_color_ordering_utils.py		clevr_color_ordering_utils.py
component.py		component.py
evaluation_utils.py		evaluation_utils.py
factual_recall_utils.py		factual_recall_utils.py
figures_and_results_processing.ipynb		figures_and_results_processing.ipynb
general_utils.py		general_utils.py
metrics.py		metrics.py
modality_alignment_utils.py		modality_alignment_utils.py
object_counting_utils.py		object_counting_utils.py
script_backpatch_vqav2.py		script_backpatch_vqav2.py
script_backpatching_experiment.py		script_backpatching_experiment.py
script_node_circuit_discovery_and_eval.py		script_node_circuit_discovery_and_eval.py
script_node_cross_modality_analysis.py		script_node_cross_modality_analysis.py
script_node_intersection.py		script_node_intersection.py
sentiment_analysis_utils.py		sentiment_analysis_utils.py
vision_language_prompts.py		vision_language_prompts.py
visualization_utils.py		visualization_utils.py