Rails Apples to Apples Benchmarks

[adsteel]

I'm not entirely sure where to put this as I was a little behind the ball and did all this work with fast_jsonapi, but I'd love to get community feedback and impressions on it if anyone cares at this point. If there's a way to make Rails work with these comparison specs, maybe we could pull my repo into this repo?

Full repo with tests and instructions: https://github.com/adsteel/jas_vs_ams_comparison

Data visualization of benchmark speeds, in seconds, over 10 consecutive benchmark trials per serializer solution.

[stas]

Heya and thank you for starting the discussion @adsteel

While I'm not a fan of looking at the performance through the rails prism, I understand why that would be interesting to others.

Speaking specifically about the approach you took, there are a couple of suggestions we should consider if we actually want to properly test the rails context.

Particularly, set up an isolated rails app and use a server benchmarking tool to generate and collect the stats. We had a performance suite previously using the rspec-benchmark, and the current setup is way better.

In regards to the implementation, AMS applies optimizations for ActiveRecord queries, which potentially should look faster, and is not something JSONAPI::Serializer aims to solve. So to actually have a comparable setup, the queries sent to both libraries should be equally optimized (consider adding .includes(:rel1, :rel2) and such.

Final point, we should test small and large datasets (see all.yml for context). Having a couple of records is generally not enough to challenge the performance aspects of an implementation.

I'll leave this conversation open for other folks to share their opinions, but at this point I'd wait until all of the above is considered before adopting a rails-specific benchmark.