The Push for 90% Test Coverage

[kshyatt]

The testing situation right now

The latest Codecov run shows a test coverage percentage of 83.24%. This is amazing work by all, but now let's push for that beautiful green badge, following the great work at #9493. @ViralBShah asked us to open new issues rather than keep accumulating comments over there. There are many files in Julia which are poorly covered, but some are more target-rich than others. In particular, the files / folders with the most lines missed are:

• base/multi.jl - 348 lines missed
• base/pkg/entry.jl - 284 lines missed
• base/linalg/lapack.jl - 293 52 lines missed
• base/inference.jl - 229 lines missed
• base/printf.jl - 234 lines missed
• base/show.jl - 170 lines missed
• base/LineEdit.jl - 199 lines missed
• base/abstractarray.jl - 196 19 lines missed - claimed by @davidagold
• base/REPL.jl - 184 lines missed
• base/stream.jl - 169 lines missed
• base/interactiveutil.jl - 163 lines missed - claimed by @Rory-Finnegan
• base/string.jl - 139 lines missed - claimed by @ScottPJones
• base/int.jl - 127 81 lines missed
• libgit2 folder - 400 lines missed

To get to 90% coverage, we need to cover 3580 lines. If we focus on the above files, we can drastically improve their coverage and Julia's at once. I also like this set because there's a good spread across Julia functionality - linear algebra, strings, parallel processing, the REPL, etc.

Isn't That a Lot of Lines?

Yeah, it is. Luckily, we've already made great progress (we passed 80%!) thanks to a great group of new contributors. Spread out across a bunch of people, it's not so much work.

File specifics

lapack.jl is the file I know best. There are many functions wrapped in there that "can't" be tested right now because they require matrix storage formats that Julia doesn't support, like banded or RFP. But there are also functions that do complicated factorizations that can be difficult to test without a lot of domain expertise. Linear algebra experts welcome!

multi.jl, the file in the worst state, has many methods for starting/stopping/talking to workers that have not been tested at all. Maybe this is an issue with the buildbots that @tkelman or @staticfloat can clarify more?

In general the state of the REPL testing is quite poor. If someone would like to familiarize themselves with the REPL internals writing these tests would be a wonderful way to do so.

How you can help

If there's an area in the above list you have some expertise in or want to learn more about, feel free to "claim" the file and start testing the heck out of it! That's how I got involved in the linear algebra testing and it worked out pretty well 😉. The first set of tests was the hardest to write, but following the instructions in the OP of #9493 will get you most of the way there. You absolutely do not need to read the following 150+ comments! Following the Coveralls repo or the Codecov repo is a good way to track our progress, and you will be able to see your numbers go up.

If someone claims a file as their testing baby, I will check it off in the list above. If there's no checkbox, that file is up for grabs and in need of some awesome Julians, newbies or experienced, to write some tests for it!

We now have a Gitter chat-room: https://gitter.im/kshyatt/julia-coverage

"Help! I'm a first time contributor and I have no idea how to use Coveralls!"

Here's a step-by-step image guide to how I find things to test using Coveralls.

1. Click on the latest build on the JuliaLang julia repo on Coveralls
   
2. Coveralls by default doesn't show us a lot of files. Let's change that.
   
3. Let's view 100 files per page. Since Julia base has about 220 files, we can capture all the ones in the worst state this way
   
4. Now let's look at all the files in Julia, rather than just the ones changed in the latest merge
   
5. lapack.jl here is in a bad state. Look at all those missed lines! Let's click on it and find something to test.
   
6. Here's a line in red that's not tested. It's an error throw, which are often missed. Writing a test for this functionality should be pretty simple.
   

[tkelman]

There was a question elsewhere that I don't think has been answered yet about what happens when parallel workers try to write coverage data to the same cov file - I'm not sure whether or not this is happening on the buildbots. Setting the environment variable JULIA_CPU_CORES=1 may help (or hurt the coverage?).

Also #11802 may make these numbers more honest (worse) when it gets merged.

[ScottPJones]

I'd be happy to try to handle the lines in string.jl, if nobody else wants it.
Would people mind me combining that with splitting up string.jl into a number of independent files,
and then splitting up test/strings.jl (my nemesis, because I keep having to rebase when somebody changes it) as well as test/unicode.jl into files matching the split up string.jl (and the ones that are already separate, i.e. utf8.jl, utf16.jl, and utf32.jl)?

[ScottPJones]

@tkelman That was me asking, and I have a very strong suspicion that it isn't being handled, and that maybe the coverage is actually higher, because esp. heavily used things like abstractarray.jl.cov may be getting overwritten by the different processes.

[Ismael-VC]

I'll keep helping the best I can! 👍 for 100%

[tkelman]

Maybe dumb possibility: write files to ".jl.$(getpid()).cov" then merge them in a post-processing step?

[ViralBShah]

That may be the simplest thing to do.

[ViralBShah]

@ScottPJones It would be best IMO to not mix up the testing with other PRs. That way we can fast track merging things that are focussed.

[ScottPJones]

Sounds like a good first step to me, as long as the same worker processes are used during the runtests (just in case the os could reuse the pid if the workers are forked during the testing)

[timholy]

I haven't dug into how the buildbots are running things, but AFAIK we've always been measuring coverage of base with JULIA_CPU_CORES=1. Indeed, the current lack of info back from workers may be the major factor behind multi.jl's apparently-poor coverage.

[ScottPJones]

@timholy That's good, I'd been trying to run coverage tests on my laptop, and was getting things overwritten. It would be good to solve this though, so that we can speed up coverage testing.
@ViralBShah Whatever people feel best, I was thinking of splitting things up by 3-4 commits in one PR,
but I can also do it as separate PRs for splitting up string.jl, splitting up test/strings.jl test/unicode.jl, and adding any tests that are not covered

[davidagold]

Happy to jump on base/abstractarray.jl, will work around pending changes from #11895.

[kshyatt]

Wonderful! Thanks so much, @davidagold. I have a weak commitment to lapack.jl since there are a few functions in there that need better error messages anyway.