Write a script to aggregate OSS data on Ruff configuration

[charliermarsh]

Right now, it's hard for us to make data-informed decisions. It'd be nice to leverage our open-source usage to help understand questions like:

• Which rules are commonly turned on?
• Which rules are commonly turned off?
• Which rules are commonly # noqa ignored (i.e., false positives)?

This is good prior art (\ht @konstin): rust-lang/rust-clippy#7666

[charliermarsh]

Labeling this as "good first issue", not because it's well-scoped, but because it's a relatively independent project (and could be done without writing any Rust, if anyone is eager to help out).

[akx]

I could take a look – https://github.com/search?q=path%3A**%2Fpyproject.toml+ruff&type=code seems like a good starting point (along with **/ruff.toml).

[konstin]

fwiw it seems that github doesn't expose an api to their dependency graph, so i'd also go with that code search (or something api friendlier if github has too tight query limits)

[akx]

@konstin Dependency graph probably wouldn't directly pick up e.g. ruff-pre-commit in a pre-commit config file (which is how I tend to use ruff). Anyway, I have a skeleton of an aggregator up already 🔧

[akx]

Alright, here goes with some initial data 😁

• Results: https://gist.github.com/akx/de1fddb5950d763c9d9d934e5522e6f0#file-results-md
  • JSON: https://gist.github.com/akx/de1fddb5950d763c9d9d934e5522e6f0#file-results-json
• Code: https://github.com/akx/ruff-usage-aggregate

The GitHub Search API doesn't seem to be using the new Code Search stuff and is heavily rate-limited, so the dataset may not be great just yet (PRs welcome, of course). Let me know if you want the files I have so far!

[JonathanPlasse]

What about the ALL data?

[akx]

What about the ALL data?

If you mean the ALL selector, it's enabled in 23/306 files seen. (You can see it in the "Other values" list in the Select section.) It would probably be interesting to see if those repos also have lots of ignores.

Similarly, I guess a more holistic view (that knew about all of the codes Ruff knows about) would be an interesting next step!

[charliermarsh]

Oooh this is so useful! Thank you @akx! The other piece of data that would be really useful to see, though not sure whether it can fit into this paradigm, is how often various codes are used in # noqa, since a # noqa is often indicative of a false positive.

Another question I'd have (though not your responsibility to answer, only if you're curious): I know the data is based on 306 TOML files. I'd be interested to know how often various fields are set vs. unset. (E.g., the .md version says 200 files enable E and F. Are the remaining 306 omitting them? Or do they use the default config, and so they're enabled anyway?)

[akx]

@charliermarsh Sure! I updated the gist with "Fields set in configuration" (since I already had that data anyway, it just wasn't aggregated). It's also based on a slightly larger dataset (I realized I didn't process ruff.tomls correctly before).

Finding noqas will require a bit more magic, but now that I do have a good starter dataset of which repos may have Ruff enabled, we can go from there (but not right now since I have 9% laptop battery left 😂)

[akx]

I updated the gist with a bigger dataset (1352 files!) – and since the JSON data was getting unwieldy, had to remove it from there. If someone needs it for analysis, let me know... though that said, everything should now be recreatable with just running make in ruff-usage-aggregate :)

The numbers remain roughly the same:

• empty select/extend-select: 61%
• empty ignore/extend-ignore: 69%
• empty fixable: 95%
• empty unfixable: 88%
• most-configured field: line-length
• median configured line-length: 100
• most popular configured line-length: 120
• most popular configured Python version: py310
• top unfixable by far: F401 (unused import)
• top ignore: E501 (line length), D203 (blank line before class docstring), D213 (multi-line docstring summary should start at 2nd line)

[akx]

Welp, turns out there was a bug or two that conflated "unset" with "empty set" 🙄 – I think this data is now more or less correct:

Name	Value
Total TOML files	1613
Unique TOML files	1352
Deduplicated TOML files	261
No select	28.7%
No ignore	35.3%
No fixable	94.8%
No unfixable	84.2%
Most popular configured Python version	py310
Median configured line length	100
Most common unfixable	F401
Most common ignore	E501
Most common select	F

Also updated into the gist, with all of the details...

[akx]

Updated with some 90+ new repos in a fortnight: https://gist.github.com/akx/211308a4d2b31aaf4412558af6fe62a1

Name	Value
Total TOML files	1846
Unique TOML files	1577
Deduplicated TOML files	269
No select	29.4%
No ignore	36.0%
No fixable	94.2%
No unfixable	84.6%
Most popular configured Python version	py310
Median configured line length	100.0
Most common unfixable	F401
Most common ignore	E501
Most common select	F

[charliermarsh]

Awesome, thank you for this! (Gonna close issue :))