Test issues (local, m1, 32GB)

[planetf1]

Environment

• Machine: Apple M1, 32 GB, macOS 25.4.0
• Python: 3.12.8
• Ollama: 0.17.5

Issues

1. test_generate_from_raw — intermittent hang

I was unable to get this test reliable initially - it may be a bug in ollama or machine limitations. With no timeout (we didn't have one in the test originally) it could hang indefinately. The test looks ok, but is dependent on parallel execution. Using the granite4:micro model should be ok at 4 in parallel (and default context of 32k).

Experimenting with enforcing timeouts.

This was the main issue leading to long execution runs or failures

2. test_generate_from_raw_with_format — stale @pytest.mark.xfail

This is marked as xfail, yet was consistently passing. I think we could remove the marker now. The current test seems reliable

3. test_format (ollama + openai_ollama) — MAX_NEW_TOKENS too small

This also failed intermittently....

MAX_NEW_TOKENS = 2**8 (256 tokens) insufficient for structured JSON output; causes
truncated responses and assertion failures. Raising to 2**10 (1024 tokens) seems to inject reliability ...

4. researcher.py — 84-second test not gated by slow
pyproject.toml described slow as ">5 minutes". researcher.py (~84s) ran in every
default suite pass, dominating duration.

I think we need to add slow to its # pytest: markers; ( and having a policy of ">1 minute" seems more reasonable )

5. asyncio_default_fixture_loop_scope=None — pytest-asyncio warning

A warning - suspect will ifgnore this for now

6. Unclosed aiohttp ClientSession / TCPConnector at suite teardown
Unclosed client session: <aiohttp.client.ClientSession object at 0x156d68950>

Similar - and only occurs at the end of the test suite

[planetf1]

Will propose a PR with fixes to address the important issues here that are preventing clean execution of a default test run on m1 32GB mac (ie uv run pytest)

[planetf1]

A little more on the ollama test:

• The ollama test uses async to execute up to 4 llm calls in parallel
• sometimes one or more of these threads just seems to hang (yet to determine why - could be an ollama bug, macos, async libraries....). This can lead to tests running for a long time 30 mins+
• adding a httpx timeout can ensure we at least abort early - for example after 60s - but this doesn't make the test reliable. We could possibly mark it as XFAIL.
• Running standalone it's generally reliable as a 1-off, but will fail if run repeatedly - and often fails as part of a large test run

Possible fixes include adding retry into the backend, but it would be good to understand what is going on first. Reducing parallelism may also avoid the issue, but that may not be helpful

Other thoughts include - is it load related? Do we need to reduce context window since for 4 threads we need 4x ... (but monitoring, memory load seemed ok with granite4:micro)

The best 'quick fix' for now would be

• mark as xfail
• include a timeout of ~1 minute

Seeing if I can determine underlying cause..

[planetf1]

Unreliability rate is around 20%. For example:

Run	Time	Duration	Result
1	13:17:09	22.54s	PASS
2	13:17:35	19.39s	PASS
3	13:17:58	65.34s	PASS
4	13:19:07	26.88s	PASS
5	13:19:37	76.72s	PASS
6	13:20:56	23.32s	PASS
7	13:21:23	28.44s	PASS
8	13:21:54	25.00s	PASS
9	13:22:22	108.50s	PASS
10	13:24:13	135s	FAIL
11	13:26:32	133.80s	FAIL
12	13:28:49	134.84s	FAIL
13	13:31:07	69.53s	PASS
14	13:32:20	64.00s	PASS
15	13:33:28	15.00s	PASS
16	13:33:46	26.97s	PASS
17	13:34:16	70.40s	PASS
18	13:35:29	20.07s	PASS
19	13:35:52	134.75s	FAIL
20	13:38:10	16.76s	PASS

Note that runs 10-12 failed consecutively -- it does seem related to other load. This is with a httpx timeout of 120s (each test hit 134s). Note that before timeouts I saw runs like 35s + ... One other run (9) got close to failing. Good runs can be as low as 20s, but timing is very variable

Next check - this test uses a 32K context (default for ollama backend) but is doing trivial work. Trying with 2k context. (so overall context goes from 128K -> 8K ie massive K-V cache reduction)

[planetf1]

With a smaller context only marginally better (and probably down to small # samples)

Run	Duration	Result
1	137.55s	FAIL
2	21.38s	pass
3	24.53s	pass
4	23.80s	pass
5	134.97s	FAIL
6	76.98s	pass
7	32.58s	pass
8	61.25s	pass
9	136.35s	FAIL
10–20	17–72s	all pass

This looks like a concurrency bug, not resource limitation

[planetf1]

The assertions used in the test are not valid. In the event of failure we get "" (which is not None) so assertion passes... (which isn't helpful)

Additionally ..the cause of "" ? - this is what we end up returning in the ollama backend if anything fails... the original exception is swallowed and not reported anywhere :-(

For now will add a log entry (to help to debug the issue). But we need to consider how to surface exceptions. Will open an issue

[planetf1]

Related: opened #574 as the initial problem I got was a missing model. No code change but required an action

[jakelorocco]

I believe

6. Unclosed aiohttp ClientSession / TCPConnector at suite teardown
   Unclosed client session: <aiohttp.client.ClientSession object at 0x156d68950>

is related to: BerriAI/litellm#11657.

[psschwei]

I have also seen Ollama hang, but for me I don't think it was related to these tests... I had granite4:micro stopping, but it seems to have gotten stuck stopping. Manually killing it / restarting the ollama service fixed the problem (and next run went much quicker)

[planetf1]

@jakelorocco yes agreed - actually I hit a similar issue in another project

@paul different machine spec may be a factor - I'm sure it's some kind of concurrency issue. The more serious aspect is that any errors are masked. So probably where initial focus should be.

[planetf1]

Created a PR with the misc fixes I needed when running tests, and created a new issue specifically to track the Ollama concurrency issue as #599 . There have also been many misc fixes to tests recently, including in error handling - so it would make sense to work on this issue on top of those fixes.