Leveraging Generative Modelling for Rich Representations

• We model representation space as continuous dynamical system (NODEL, CARL)
• We model representation space as distribution (DARe)
• We leverage EBMs for rich representations (LEMa)
• We leverage Score for rich representations (ScAlRe)

Workflows

ScAlRe (Score Alignment Regularization for Representation Learning)

Results

Algorithm	CIFAR10 (R50)		CIFAR100 (R50)		CIFAR10 (R18)		CIFAR100 (R18)		Timg (R18)
	LR	kNN	LR	kNN	LR	kNN	LR	kNN	LR	kNN
SimCLR	92.5	90.9	69.1	62.0	91.2	89.4	62.6	58.0	46.2	37.9
SimCLR-ScAlRe-E	92.6	90.2	69.4	61.7	91.0	89.3	63.9	57.8	45.8	37.1
SimCLR-ScAlRe-S					91.5	89.9	63.9	57.9	46.2	37.5
Barlow Twins	90.7	86.3	71.7	60.3	90.1	87.2	67.7	59.0	53.0	39.8
Barlow Twins-ScAlRe-E	91.4	87.4	71.3	60.9	90.1	87.6	66.8	58.6	53.8	41.8
Barlow Twins-ScAlRe-S	92.1	88.7	70.4	57.0	90.5	87.4	65.9	56.3	54.0	41.7
BYOL					84.8	82.7	63.2	56.3
BYOL-ScAlRe-E					86.4	81.4	62.7	56.7
BYOL-ScAlRe-S					87.0	82.0	61.3	53.9
SimSiam	91.8	88.9	63.3	56.2	90.4	88.5	62.6	57.1	47.2	39.3
SimSiam-ScAlRe-E	91.5	88.8	62.9	55.2	90.5	89.1	62.7	58.0	47.6	38.9
SimSiam-ScAlRe-S	90.9	87.7	63.3	55.7	90.6	88.8	62.8	57.9	47.0	39.1
VicReg	90.5	87.7	68.6	57.8	87.7	84.2	62.7	52.2	48.5	33.9
VicReg-ScAlRe-E	90.8	88.0	68.9	57.3	87.8	84.1	62.4	52.0	48.0	34.2
VicReg-ScAlRe-S	90.9	87.5	68.1	57.1	87.5	84.3	62.8	52.3	48.0	34.1

Clustering Metrics Results

Algorithm	CIFAR10 (R18)				CIFAR100 (R18)
	ARI	NMI	Silhoutte	DBS	ARI	NMI	Silhoutte	DBS
SimCLR	0.589	0.707	0.082	3.246	0.244	0.535	0.115	2.493
SimCLR-ScAlRe-E	0.605	0.703	0.082	3.345	0.231	0.535	0.114	2.451
SimCLR-ScAlRe-S	0.557	0.677	0.075	3.477	0.235	0.531	0.113	2.515
Barlow Twins	0.407	0.541	0.034	4.428	0.179	0.472	0.052	3.206
Barlow Twins-ScAlRe-E	0.471	0.582	0.038	4.289	0.171	0.464	0.053	3.207
Barlow Twins-ScAlRe-S	0.376	0.514	0.032	4.475	0.157	0.437	0.045	3.264
SimSiam	0.576	0.674	0.059	3.879	0.228	0.514	0.076	2.935
SimSiam-ScAlRe-E	0.553	0.678	0.060	3.720	0.227	0.516	0.079	2.877
SimSiam-ScAlRe-S	0.581	0.669	0.059	3.905	0.223	0.513	0.077	2.921
VicReg	0.435	0.520	0.051	3.595	0.150	0.414	0.048	3.078
VicReg-ScAlRe-E	0.399	0.496	0.048	3.792	0.159	0.423	0.050	3.053
VicReg-ScAlRe-S	0.400	0.492	0.047	3.644	0.155	0.420	0.049	3.062
BYOL	0.382	0.497	0.071	3.145	0.204	0.511	0.084	2.724
BYOL-ScAlRe-E	0.411	0.503	0.065	3.376	0.212	0.516	0.084	2.760
BYOL-ScAlRe-S	0.418	0.522	0.069	3.365	0.206	0.506	0.086	2.715

Reproducing the results

• lookout for more commands in run.sh

python train.py --config configs/simclr.yaml --dataset cifar10 --gpu 1 --model resnet18 --epochs 800 --epochs_lin 100 --save_path simclr.c10.r18.pth > logs/simclr.c10.r18.log

**Test the pretrained model

python test.py --dataset cifar10 --model resnet18 --saved_path saved_models/simclr.c10.r18.pth --cmet --knn --lreg --linprobe --tsne --gpu 0 --verbose