Zero order search

[sayakpaul]

Config:

{
    "pretrained_model_name_or_path": "black-forest-labs/FLUX.1-dev",
    "torch_dtype": "bf16",
    "pipeline_call_args": {
        "height": 1024,
        "width": 1024,
        "max_sequence_length": 512,
        "guidance_scale": 3.5,
        "num_inference_steps": 50
    },
    "verifier_args": {
        "name": "gemini", 
        "max_new_tokens": 800,
        "choice_of_metric": "overall_score"
    },
    "search_args": {
        "search_method": "zero-order",
        "search_rounds": 4,
        "threshold": 0.95,
        "num_neighbors": 4
    }
}

[willisma]

LGTM!

[willisma]

Let me know your opinion on this:
we also observed similar effect as in the sample you shown above, that the search converge to some fix point without further improving, so I wonder if we should either:

1. for one search round, if all neighbor noises are worse, restart the round without increment the round number
2. for one search round, if all neighbor noises are worse, reject this round, save the center noise n, and sample a new noise to perform the rest of the search, and in the end compare the saved noise and the new noise to see which is better

[sayakpaul]

for one search round, if all neighbor noises are worse, reject this round, save the center noise n, and sample a new noise to perform the rest of the search, and in the end compare the saved noise and the new noise to see which is better

We could do this. We will have datapoint from each search that will help us do it easily I believe.

for one search round, if all neighbor noises are worse, reject this round

Regarding this we compare the performance of the neighbor noises w.r.t the base noise right?

[willisma]

for one search round, if all neighbor noises are worse, reject this round, save the center noise n, and sample a new noise to perform the rest of the search, and in the end compare the saved noise and the new noise to see which is better

We could do this. We will have datapoint from each search that will help us do it easily I believe.

for one search round, if all neighbor noises are worse, reject this round

Regarding this we compare the performance of the neighbor noises w.r.t the base noise right?

yes that's correct

[sayakpaul]

@willisma I went with 2.

{
    "pretrained_model_name_or_path": "black-forest-labs/FLUX.1-dev",
    "torch_dtype": "bf16",
    "pipeline_call_args": {
        "height": 1024,
        "width": 1024,
        "max_sequence_length": 512,
        "guidance_scale": 3.5,
        "num_inference_steps": 50
    },
    "verifier_args": {
        "name": "gemini", 
        "max_new_tokens": 800,
        "choice_of_metric": "overall_score"
    },
    "search_args": {
        "search_method": "zero-order",
        "search_rounds": 4,
        "threshold": 0.95,
        "num_neighbors": 4
    }
}

python main.py --prompt="a tiny astronaut hatching from an egg on the moon" --num_prompts=None --pipeline_config_path=configs/flux.1_dev_zero_order.json

If you could take a look and let me know your comments, that would be helpful.

[willisma]

I'm a little confused about here & the should_generate_noise flag; what's the difference between the two?

[sayakpaul]

@willisma sorry to be bothering again, hopefully, this is the last one. Could you check the changes in 8b47348?

When rejecting the current round, I am:

• Not serializing the artifacts (datapoint JSON and images)
• Sampling a new base noise and continuing with the rest of the process.

LMK.