Zebra Simple Test

Implementation of Zebra Puzzle for LLM, simple variation of ZebraLogic bench.

What it does exactly:

• loads ZebraLogic dataset for local work if none yet
• queries and checks N zebra puzzles of specified SIZE for each MODEL in list
• stores json log for each model and summary, extracts model answer and think

Key differences from ZeroEval:

• requires only openai and duckdb pip packages and access to APIs
• works via OpenAI API also with Groq, Google, Anthropic and DeepSeek
• switches thinking mode with model suffix none/default/low/medium/high for all APIs
• extracts <think> or <thought> tag reasoning to separate thought data from results
• DOES NOT implement any other benches except zebra puzzle of specified size

Installation

pip install openai
pip install duckdb

Usage

# Configure keys for your models
export GROQ_API_KEY="set your key for @groq models"
export GOOGLE_API_KEY="set your key for @google models"
export NVIDIA_API_KEY="set your key for @nvidia models"

# Run 10 random zebra puzzles 2*2 for models from models.txt
python -u zebra.py -f models.txt -s 2*2 -r 10

# Run 2 exact zebra puzzles 3*3 #0 and #1 for one Groq model
python -u zebra.py -m llama-3.3-70b-versatile@groq -s 3*3 -i 0,1 -t 4096

Results example

Finished processing 4 models * 10 tests in 0:04:09.991346

{
    "duration": "0:04:09.991346",
    "size": "2*2",
    "count": 10,
    "max_tokens": 1024,
    "models_summary": {
        "nvidia/llama-3.1-nemotron-70b-instruct@nvidia": 90,
        "nvidia/llama-3.1-nemotron-ultra-253b-v1@nvidia": 100,
        "gemini-2.5-flash-preview-05-20-none@google": 100,
        "gemini-2.5-flash-preview-05-20-low@google": 100
    }
}

Supported models

Common rules

• model should contain OpenAI API compatible model name, optional reasoning_effort suffix and @source
• each source addressed via OpenAI API with key from environment variable as GOOGLE_API_KEY for @google
• reasoning_effort controlled by suffix after the model name: none, default, low, medium, high
• @google and @nvidia have specific way of enabling thinking to allow catching <think> or <thought>
• use response- model prefix for @openai to request via Response API and extract reasoning.summary
• full model name example with reasoning suffix and source: gemini-2.5-flash-preview-05-20-low@google

Models by source

• @groq models https://console.groq.com/docs/models
• @google models https://ai.google.dev/gemini-api/docs/models
• @nvidia models https://docs.api.nvidia.com/nim/reference/llm-apis
• @xai models https://docs.x.ai/docs/models?cluster=us-east-1
• @deepseek models https://api-docs.deepseek.com/
• @anthropic models https://docs.anthropic.com/en/docs/about-claude/models/overview#model-aliases
• @openai models https://platform.openai.com/docs/pricing
• @nebius models https://studio.nebius.com/?modality=text2text

Keys by source

• GROQ_API_KEY https://console.groq.com/keys
• GOOGLE_API_KEY https://makersuite.google.com/app/apikey
• NVIDIA_API_KEY https://build.nvidia.com/settings/api-keys
• XAI_API_KEY https://console.x.ai/
• DEEPSEEK_API_KEY keys https://platform.deepseek.com/api-keys
• ANTHROPIC_API_KEY https://console.anthropic.com/settings/keys
• OPENAI_API_KEY https://platform.openai.com/api-keys
• NEBIUS_API_KEY https://studio.nebius.com/settings/api-keys
• CUSTOM_API_KEY and CUSTOM_API_URL could point to any, @custom model suffix

Models examples

gpt-4.1-nano-high@openai                            # no free
response-o3-medium@openai                           # reasoning.summary returned in think field
claude-sonnet-4-0-high@anthropic                    # no free
deepseek-reasoner-high@deepseek                     # no free
grok-3-mini-high@xai                                # no free
nvidia/llama-3.1-nemotron-70b-instruct@nvidia       # free for devs, no reasoning
nvidia/llama-3.1-nemotron-ultra-253b-v1-high@nvidia # free for devs, have <think>
gemini-2.5-pro-exp-03-25-none@google                # pro is not free
gemini-2.5-flash-preview-05-20-default@google       # have <thought>
gemini-2.0-flash@google                             # no reasoning
qwen/qwen3-32b-default@groq                         # reasoning none and default, have <think>
qwen-qwq-32b@groq                                   # no reasoning control, but have <think>
deepseek-r1-distill-llama-70b@groq                  # no reasoning control, but have <think>
llama-3.3-70b-versatile@groq                        # no reasoning
llama3-8b-8192@groq                                 # no reasoning
gemma2-9b-it@groq                                   # no reasoning  
allam-2-7b@groq                                     # no reasoning, weak and good to test size 2*2
microsoft/phi-4@nebius                              # 
deepseek-ai/DeepSeek-R1@nebius                      # 

Acknowledgements

1. First thanks to Creative Workshop team inspired me to do this simple test
2. Authors of ZebraLogic and paper created good and reusable results, nice work!
3. Big Thanks for WildEval for publishing dataset with solutions
4. And warmest thanks to author of Sherlock game implementing the puzzle. Many happy hours in the last millennium I spent with this game, now it is available as Watson desktop and Sherlok mobile

History

• 2025-06-12 Initial version with dataset load (ow, original ZebraLogic by allenai/ZeroEval has no solution)
• 2025-06-13 Extracted and modified prompt from ZeroEval, completion request and results comparison logic
• 2025-06-14 Created results saving logic, tested models from different sources, improved completion request
• 2025-06-15 Research for reasoning_effort and similars to control thinking, extract <think> or <thought>
• 2025-06-16 Polished results for easy usage by Creative Workshop team, prepared for publishing on github
• 2025-06-16 Added support for command-line args and usage help, added direct models in args and items list
• 2025-06-17 Used new Response API to extract reasoning summary for OpenAI o3- and o1- models
• 2025-07-18 Added obfuscated dataset (with entities and names replaced) and command line option -ood to use it
• 2025-07-18 Added NEBIUS_API_KEY and @nebius models support

Tasks

• Add and test zebra.py
• Add saving parquet if not exists, do not store in our repos, read license
• Add config.py and results to .gitignore
• Add delay option between tests (bypassing RPM/TPM free limits)
• Add and fill LICENSE, check used parts license and reference here
• Collect links to models and keys retrieval, models examples with comments
• Implement command-line args retrieval and usage help output