Code for Exhaustive parallel search

[doomysday]

Hi, thank you for your great work!

I sincerely hope to know the details of the exhaustive parallel search mentioned in the Section. J.1.

I cannot find any clues in this repo, and I would appreciate it if you could upload the related code scripts. This would be very helpful for me to better understand and follow your nice work.

[gabrielilharco]

Hi @doomysday, thanks for the kind words! Exhaustive parallel search is simply looking at all possible combinations α1 = {0, 0.05, 0.10, 0.15, ..., 1.00} and α2 = {0, 0.05, 0.10, 0.15, ..., 1.00}, computing the weights θ = (1 − α1 − α2) · θzs + α1 · θ1 + α2 · θ2 for two finetuned models θ1 and θ2, and measuring the accuracy of the resulting model. So each combination of α1 and α2 corresponds to a model, which is a single pixel in the plots from Figs. 27-29. Hope this helps!

[doomysday]

Okay, you got it! I would appreciate it if you could upload the corresponding code of the exhaustive parallel search and the results in from Figs. 27-29.

BTW, another question is: if the model becomes significantly smaller—for example, with only around 10M parameters—do 'patching' or 'model arithmetic' remain effective? Have you explored this further, or do you have any intuitive advice?