Explain potential reasons for performance difference

[DavidFarago]

To understand the tool calling performance difference and to be able to better choose the vendor, please explain potential reasons for the difference.

Is there some internal nondeterminism leading to vendor-specific differences in logits output when the same model weights are used? Or are there differences in sampling or schema validation? Or are different model weights / quantization being used by different vendors?

[TonyGeez]

To understand the tool calling performance difference and to be able to better choose the vendor, please explain potential reasons for the difference.

Is there some internal nondeterminism leading to vendor-specific differences in logits output when the same model weights are used? Or are there differences in sampling or schema validation? Or are different model weights / quantization being used by different vendors?

Most regular users don't realize this, but quantization plays a huge role in model performance.

Peoples (myself included) assumed for long time, that when you select a model from a provider, they simply download it and handle the inference.

The reality is much more complex.
I strongly recommend trying Fireworks and comparing it to other providers. This one is kinda easy to notice.

Even the evaluations here confirm just how poorly their models perform, they're not just slightly worse than the second worst provider.

That kind of performance gap cannot be explained by normal variations in quantization or sampling parameters.

Something fundamentally wrong is happening with what they're serving.

Fireworks markets itself heavily on speed and cost, which reveals everything about their priorities.
They have clearly optimized their infrastructure to serve models as cheaply as possible.

This likely means aggressive quantization, almost certainly INT4 or lower.

No surprise Fireworks is one of the few providers nowadays that refuses to disclose the precision they use for their inferenced models.

But my guess is these kind of providers, which have way worst results than other is probably also involves model pruning, where they remove parameters deemed less important.

It could include knowledge distillation, where they train a smaller model to mimic a larger one.
It might mean running on older, cheaper hardware with limited memory bandwidth.

Any combination of these techniques could explain why some provider models perform so poorly on structured tasks like tool calling.

The worst part is the complete lack of transparency. You select a model from provider API, you receive an endpoint, but you have no idea what precision it runs (we are starting to see more provider starting to disclose that) at, what optimizations have been applied, or even what hardware it operates on.

For basic text generation this might be acceptable, but for tool calling where reliability and precision are critical, you're essentially gambling.

What truly concerning is the potential for even more extreme cost cutting measures.
When you call an API, you're trusting that the provider is running what they claim to be running.

However, there's no verification mechanism.
They could be serving smaller models than advertised, heavily distilled versions, or other substitutes.

A well prompted 70B model can sometimes produce outputs that appear plausible to average users compared to a 400B model, especially when users aren't testing rigorously.

The cost savings from serving a smaller model would be enormous, and most average user would never discover the difference.