verified waterbirds results

Author: Lisa Dunlap

README.md

@@ -3,7 +3,7 @@ Official Implementation of [LADS (Latent Augmentation using Domain descriptionS)
 
 ![LADS method overview.](figs/lads-method-2-1.png "LADS method overview")
 
-*WARNING: this is still WIP, please raise an issue or email me if you run into any bugs.*
+*WARNING: this is still WIP, please raise an issue if you run into any bugs.*
 
 ```
 @article{dunlap2023lads,
@@ -14,14 +14,6 @@ Official Implementation of [LADS (Latent Augmentation using Domain descriptionS)
 }
 ```
 
-## TODOs
-[X] clean up emb saving/loading
-[] fix the Directional vs LADS acc diff
-[X] add e2e method for DA
-[] get E2E to work well
-[X] add in selective augmentation (run lp, check per class acc, augment poor performing finetuned classes more towards the text emb)
-[] run 2 layer mlp baselines
-
 ## Getting started
 
 1. Install the dependencies for our code using Conda. You may need to adjust the environment YAML file depending on your setup.
@@ -38,21 +30,30 @@ Official Implementation of [LADS (Latent Augmentation using Domain descriptionS)
 ## Code Structure 
 The configurations for each method are in the `configs` folder. To try say the baseline of doing normal LR on the CLIP embeddings:
 ```
-python main.py --config configs/Waterbirds/base.yaml
+python main.py --config configs/Waterbirds/mlp.yaml 
+```
+
+you can also override parameters like so
+```
+python main.py --config configs/Waterbirds/mlp.yaml METHOD.MODEL.LR=0.1 EXP.PROJ=new_project
 ```
 
+### Datasets
+
 Datasets supported are in the [helpers folder](./helpers/data_helpers.py). Currently they are:
-* Waterbirds (100% and 95%)
-* ColoredMNIST (LNTL version and simplified version)
-* DomainNet
-* CUB Paintings
-* OfficeHome
+* Waterbirds (100% and 95%) [our specific split](https://drive.google.com/file/d/1zJpQYGEt1SuwitlNfE06TFyLaWX-st1k/view) [code to generate data](https://github.com/kohpangwei/group_DRO)
+* ColoredMNIST (LNTL version and simplified version) NOTEBOOK COMING SOON
+* DomainNet (the version used in the paper is `DATA.DATASET=DomainNetMini`) [full dataset](http://ai.bu.edu/DomainNet/)
+* CUB Paintings [photos dataset](https://www.vision.caltech.edu/datasets/cub_200_2011/) [paintings dataset](https://github.com/thuml/PAN)
+* OfficeHome COMING SOON
+
+You can download the CLIP embeddings of these datasets [here](https://drive.google.com/drive/folders/1ItjhX7RPfQ6fQQk6_bEYJPewnkVdcfOC?usp=sharing). We also have the embeddings for CUB, Waterbirds, and DomainNetMini in the [embeddings](./embeddings/) folder.
 
-You can download the CLIP embeddings of these datasets [here](https://drive.google.com/drive/folders/1ItjhX7RPfQ6fQQk6_bEYJPewnkVdcfOC?usp=sharing)
+Since computing the CLIP embeddings for each train/val/test set is time consuming, you can store the embeddings by setting `DATA.LOAD_CACHED=False`, then it should store the embeddings into a file `embeddings/{dataset}/clip_{openai,LAION}_{model_name}`
 
-Since computing the CLIP embeddings for each train/val/test set is time consuming, you can store the embeddings by setting `DATA.LOAD_CACHED=False` and `DATA.SAVE_PATH=[path you want to save to]`
+### Methods
 
-Then, add the path to the saved embeddings to DATASET_PATHS in [data_helpers](./helpers/data_helpers.py) and set `DATA.LOAD_CACHED=Tue` in your yaml file
+All the augmenation methods (i.e. LADS and BiasLADS) are in `methods/augmentations`, while the classifiers and baselines are in `methods/clip_transformations.py`
 
 More description of each method and the config files in the config folder. 
 

configs/CUB/lads.yaml

@@ -6,7 +6,7 @@ EXP:
   TEXT_PROMPTS: ['a painting of a {} bird.']
   NEUTRAL_TEXT_PROMPTS: ['a photo of a {} bird.']
   AUGMENTATION: 'LADS'
-  EPOCHS: 200
+  EPOCHS: 400
   ENSAMBLE: True
 
 
@@ -18,12 +18,9 @@ DATA:
 METHOD:
   MODEL:
     NUM_LAYERS: 1
-    DOM_WEIGHT: 1.0
     LR: 0.001
     WEIGHT_DECAY: 0.05
-    CHECKPOINT_NAME: 'lads'
-    RESUME: False
-  USE_DOM_GT: True
+    CHECKPOINT_NAME: 'cub_lp'
 
 AUGMENTATION:
   MODEL:

configs/DomainNet/test.yaml

@@ -30,16 +30,16 @@ METHOD:
 AUGMENTATION:
   MODEL:
     LR: 0.005
-    WEIGHT_DECAY: 0.005
-    NUM_LAYERS: 2
-    HIDDEN_DIM: 384
+    WEIGHT_DECAY: 0.05
+    NUM_LAYERS: 1
+    HIDDEN_DIM: 512
   EPOCHS: 50
   GENERIC: False
   DOM_LABELS: ["forest", "water"]
-  DOM_SPECIFIC_XE: False
+  DOM_SPECIFIC_XE: true
   ALPHA: 0.75
   # CLIP_NN_LOSS: True
   # COMPARE_BEFORE_AUG: True
   # NN_INCLUDE_SAMPLE: True
-  DOM_WEIGHT: 1.0
-  NN_WEIGHT: 1.0
+  # DOM_WEIGHT: 1.0
+  # NN_WEIGHT: 1.0

configs/DomainNet/test_aug.yaml

@@ -61,4 +61,4 @@ AUGMENTATION:
   DOM_SPECIFIC_XE: False
   DOM_LABELS: []
   NN_WEIGHT: 0.0
-  REG_WEIGHT: 0.0
+  REG_WEIGHT: 0.1

configs/Waterbirds/lads.yaml

@@ -85,36 +85,13 @@ def flatten_config(dic, running_key=None, flattened_dict={}):
     model = model.to(device)
 
 # # load data
-# if args.DATA.LOAD_CACHED:
-#     cache_file = f"{args.DATA.SAVE_PATH}/{args.DATA.DATASET}/{args.EXP.IMAGE_FEATURES}_{args.EXP.CLIP_PRETRAINED_DATASET}_{args.EXP.CLIP_MODEL.replace('/','_')}.pt"
-#     dataset_classes, dataset_domains = dh.DATASET_CLASSES[args.DATA.DATASET], dh.DATASET_DOMAINS[args.DATA.DATASET]
-#     assert os.path.exists(cache_file), f"{cache_file} does not exist. To compute embeddings, set DATA.LOAD_CACHED=False"
-#     print(f"Loading cached embeddings from {cache_file}")
-#     train_features, train_labels, train_groups, train_domains, train_filenames, val_features, val_labels, val_groups, val_domains, val_filenames, test_features, test_labels, test_groups, test_domains, test_filenames = load_embeddings(cache_file, args.DATA.DATASET)
-# load data
-# if args.DATA.LOAD_CACHED:
-#     print(args.DATA.LOAD_CACHED)
-#     if args.EXP.IMAGE_FEATURES == 'clip' or args.EXP.IMAGE_FEATURES == 'openclip':
-#         model_name = args.EXP.CLIP_MODEL
-#     else:
-#         model_name = args.EXP.IMAGE_FEATURES
-#     cache_file, dataset_classes, dataset_domains = dh.get_cache_file(DATASET_NAME, model_name, args.EXP.IMAGE_FEATURES)
-#     assert os.path.exists(cache_file), f"{cache_file} does not exist. To compute embeddings, set DATA.LOAD_CACHED=False"
-#     data = torch.load(cache_file)
-#     train_features, train_labels, train_groups, train_domains, train_filenames = data['train_features'], data['train_labels'], data['train_groups'], data['train_domains'], data['train_filenames']
-#     val_features, val_labels, val_groups, val_domains, val_filenames = data['val_features'], data['val_labels'], data['val_groups'], data['val_domains'], data['val_filenames']
-#     test_features, test_labels, test_groups, test_domains, test_filenames = data['test_features'], data['test_labels'], data['test_groups'], data['test_domains'], data['test_filenames']
-#     # move some val data to test 
-#     if args.DATA.DATASET != 'ColoredMNISTBinary':
-#         val_features, val_labels, val_groups, val_domains, val_filenames = data['val_features'][::2], data['val_labels'][::2], data['val_groups'][::2], data['val_domains'][::2], data['val_filenames'][::2]
-#         test_features, test_labels, test_groups, test_domains, test_filenames = np.concatenate((data['test_features'], data['val_features'][1::2])), np.concatenate((data['test_labels'], data['val_labels'][1::2])), np.concatenate((data['test_groups'], data['val_groups'][1::2])), np.concatenate((data['test_domains'], data['val_domains'][1::2])), np.concatenate((data['test_filenames'], data['val_filenames'][1::2]))
 cache_file = f"{args.DATA.SAVE_PATH}/{args.DATA.DATASET}/{args.EXP.IMAGE_FEATURES}_{args.EXP.CLIP_PRETRAINED_DATASET}_{args.EXP.CLIP_MODEL.replace('/','_')}.pt"
 dataset_classes, dataset_domains = dh.DATASET_CLASSES[args.DATA.DATASET], dh.DATASET_DOMAINS[args.DATA.DATASET]
 if os.path.exists(cache_file):
     print(f"Loading cached embeddings from {cache_file}")
     train_features, train_labels, train_groups, train_domains, train_filenames, val_features, val_labels, val_groups, val_domains, val_filenames, test_features, test_labels, test_groups, test_domains, test_filenames = load_embeddings(cache_file, args.DATA.DATASET)
 else:
-    # print(f"Computing embeddings and saving to {cache_file}")
+    print(f"Computing embeddings and saving to {cache_file}")
     trainset, valset, testset = dh.get_dataset(DATASET_NAME, preprocess)
     dataset_classes, dataset_domains = dh.get_class(DATASET_NAME), dh.get_domain(DATASET_NAME)
     train_loader = torch.utils.data.DataLoader(trainset, batch_size=cfg.DATA.BATCH_SIZE, shuffle=True)
@@ -154,7 +131,7 @@ def flatten_config(dic, running_key=None, flattened_dict={}):
 print("Advice Method", args.EXP.ADVICE_METHOD)
 bias_correction = getattr(CLIPTransformations, args.EXP.ADVICE_METHOD)(prompts, clip_model, args, neutral_prompts)
 
-# old_train_features, old_train_labels, old_train_groups, old_train_domains, old_train_filenames = train_features, train_labels, train_groups, train_domains, train_filenames
+
 old_val_features, old_val_labels, old_val_groups, old_val_domains, old_val_filenames = val_features, val_labels, val_groups, val_domains, val_filenames
 old_test_features, old_test_labels, old_test_groups, old_test_domains, old_test_filenames = test_features, test_labels, test_groups, test_domains, test_filenames
 
@@ -251,26 +228,21 @@ def flatten_config(dic, running_key=None, flattened_dict={}):
 print(f"Test accuracy: {group_accuracy} \n Test domain accuracy: {domain_accuracy}")
 
 if 'E2E' in args.EXP.ADVICE_METHOD:
-    # features, labels, groups, domains, filenames = np.concatenate([old_val_features, old_test_features]), np.concatenate([old_val_labels, old_test_labels]), np.concatenate([old_val_groups, old_test_groups]), np.concatenate([old_val_domains, old_test_domains]), np.concatenate([old_val_filenames, old_test_filenames])
     aug_features, aug_labels, aug_domains, aug_filenames = bias_correction.augment_dataset(train_features, train_labels, train_domains, train_filenames)
     sample_idxs = random.sample(list(range(len(aug_filenames))), 1000)
-    # print("SAMPLE SHAPE: ", sample_filenames.shape, sample_domains.shape)
     sample_features, sample_domains, sample_labels, sample_filenames = aug_features[sample_idxs], aug_domains[sample_idxs], aug_labels[sample_idxs], aug_filenames[sample_idxs]
     neighbor_domains, neighbor_labels, domain_acc, class_acc, neighbor_samples, prop_unique, mean_cs = get_nn_metrics(sample_features, sample_domains, sample_labels, old_test_features, old_test_domains, old_test_labels)
     wandb.log({"mean CS for NN": mean_cs})
     print(neighbor_samples)
     plt.rcParams["figure.figsize"] = (20,5)
     f, (axs_orig, axs_new) = plt.subplots(2, 10, sharey=True)
     for i, (original_idx, sample_idx) in enumerate(neighbor_samples):
-        # try:
         print(sample_filenames[original_idx])
         axs_orig[i].imshow(Image.open(sample_filenames[original_idx]).resize((224, 224)))
         axs_orig[i].set_title(f"{dataset_domains[int(sample_domains[int(original_idx)])]} - {sample_labels[int(original_idx)]}")
         axs_orig[i].axis('off')
         axs_new[i].imshow(Image.open(old_test_filenames[sample_idx]).resize((224, 224)))
         axs_new[i].set_title(f"{dataset_domains[int(old_test_domains[int(sample_idx)])]} - {old_test_labels[int(sample_idx)]}")
         axs_new[i].axis('off')
-        # except:
-        #     print(f"sample idx {sample_idx} is not a valid index")
     wandb.log({"train features NN": wandb.Image(f), "domain consistency acc": domain_acc, "class consistency acc": class_acc, "unique nn": prop_unique})
     wandb.sklearn.plot_confusion_matrix(sample_domains, neighbor_domains, dataset_domains)