More comprehensive test coverage

[timholy]

Updated Feb 23, 2015.

This would make a whole bunch of good "first contribution" projects, and does not require deep insider knowledge of the language.

The basic idea is to expand the test suite to make sure that julia's base code works as promised. Here is one recommended way to contribute toward this goal:

The easy way

1. Go visit https://coveralls.io/r/JuliaLang/julia. (At the time of this writing, coverage numbers are in the 70s%. If you see numbers that are much lower than this, don't trust the reports; there is likely a problem in measuring coverage.)
2. Browse through the source files and find some untested functionality (highlighted in red) that you think you might be able to write a test for.
3. Write a test that exercises this functionality---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

The manual method

1. Make sure you have an up-to-date git checkout and are on the master branch. Build julia with make
2. Copy/paste the following script into an editor window and save it. Here I'll use the filename /tmp/coverage_tests.jl:

using Base.Test
testnames = ["core", "keywordargs", "numbers", "strings", "dates",
    "hashing", "remote", "iobuffer", "staged", "arrayops",
    "subarray", "reduce", "reducedim", "random", "intfuncs",
    "simdloop", "blas", "fft", "dsp", "sparse", "bitarray", "copy", "math",
    "functional", "bigint", "sorting", "statistics", "spawn",
    "backtrace", "priorityqueue", "arpack", "file", "suitesparse", "version",
    "pollfd", "mpfr", "broadcast", "complex", "socket",
    "floatapprox", "readdlm", "reflection", "regex", "float16", "combinatorics",
    "sysinfo", "rounding", "ranges", "mod2pi", "euler", "show",
    "lineedit", "replcompletions", "repl", "test", "goto",
    "llvmcall", "grisu", "nullable", "meta", "profile",
    "libgit2", "docs", "base64", "pkg", "linalg1", "linalg2",
    "linalg3", "linalg4", "linalg/lapack", "linalg/triangular", "linalg/tridiag", 
    "linalg/pinv", "linalg/cholmod", "linalg/umfpack", "linalg/givens"
]
for tst in testnames
    println(tst)
    include(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia","test","$tst.jl"))
end

(This is the list of tests currently in test/runtests.jl, with 3 omissions (resolve, reflection, and meta) and one addition (pkg). The omitted tests explicitly test inlining, which we're going to disable, or are problematic when inlining is disabled.)

1. From the top-level directory, type rm usr/lib/julia/sys.so. Deleting sys.so will prevent julia from using any pre-compiled functions, increasing the accuracy of the results. (This also makes startup a bit slower, but that's fine for this test---and once you're done, simply typing make will cause this file to be rebuilt).
2. Via the command prompt, navigate to julia's test/ directory.
3. Start julia with julia --code-coverage=all --inline=no. This turns on code-coverage and prevents inlining (inlining makes it difficult to accurately assess whether a function has been tested).
4. At the julia prompt, type include("/tmp/coverage_tests.jl").
5. Once the tests complete successfully, quit julia
6. Navigate to the base/ directory.
7. Browse through the *.jl.cov files, and look for functions (or branches within functions) that either have 0 in front of them (indicating that they were run 0 times), or have - in front of them (indicating that they were never compiled).
8. Write a test that exercises some function that is not adequately covered---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

[JeffBezanson]

💯

[sbromberger]

This sounds like a fantastic idea, but I think I'm missing something: I fire up julia* from test/ with
--code-coverage=all and follow the instructions (the tests all appear to run correctly; that is, I get SUCCESS at the end), but then when I go to base/, I don't see any .cov files.

*I've tried using /usr/local/bin/julia which was installed via make install, as well as the one in the source tree under usr/bin.

julia> VERSION
v"0.4.0-dev+2251"

[IainNZ]

I just did it, and it worked for me. Are you definitely using the julia you are building from source?

[sbromberger]

Pretty sure. See https://gist.github.com/sbromberger/282ed8d69695b34d4279 for the log.

ETA: I do see a runtests.jl.cov in test/:

-rw-r--r--  1 seth  staff  2675 Dec 30 01:51 runtests.jl.cov

[IainNZ]

Why are you doing ../usr/bin/julia, or is that symlinked to /Users/seth/dev/julia/julia/julia?

Anyway, I thought I'd try to give this a go, but I'm not getting sensible results from the coverage. For example, I got that intersect and setdiff weren't run - sure enough they're in the tests. I added some printlns just to be sure, and they do get run, but I see

        - function intersect(s::Set, sets::Set...)
        1     println(:intersect, s, sets)
        0     i = similar(s)
        0     for x in s
        0         inall = true
        0         for t in sets
        0             if !in(x,t)
        0                 inall = false
        0                 break
        -             end
        -         end
        0         inall && push!(i, x)
        -     end
        0     return i
        - end

which is even harder to explain ¯\_(ツ)_/¯.

[sbromberger]

Why are you doing ../usr/bin/julia, or is that symlinked to /Users/seth/dev/julia/julia/julia?

Because that's where the compiled julia (pre-make install) lives. There is no /Users/seth/dev/julia/julia/julia. It's

-rwxr-xr-x  1 seth  staff  20404 Dec 24 01:19 /Users/seth/dev/julia/julia/usr/bin/julia*

(On my system, $SRCROOT/julia was a broken symlink to $SRCROOT/usr/bin/julia-debug, which broke when I moved the directory. Even after fixing the broken symlink, I did not get the expected results.)

[sbromberger]

OK, mystery solved. Because I moved the source root after the build, I have broken symlinks all over the place. Specifically, $SRCROOT/usr/share/julia/base is a symlink that is not relative (that is, it points to a full path starting with /), which ties it to the filesystem at the time of the build. Since I moved the source root, that absolute symlink now points to an invalid directory, which (I guess) forces $SRCROOT/usr/bin/julia to try to find some other base/ (though I don't know where).

Why the symlinks within the build directory are absolute-pathed (at least on OSX) is a mystery to me. Far better, instead of having

lrwxr-xr-x  1 seth  staff  26 Dec 24 00:43 base@ -> /Users/seth/dev/julia/base

in $SRCROOT/usr/share/julia, we should have

lrwxr-xr-x  1 seth  staff  26 Dec 24 00:43 base@ -> ../../../base

so that the build directory is independent of the overall filesystem and the source root (/Users/seth/dev/julia) can be moved without problem.

[waldyrious]

@timholy, step 7 appears to be less detailed than the other ones. Perhaps it should at least link to http://julia.readthedocs.org/en/latest/stdlib/test/ for those new to unit testing. (The rest of the list is crystal clear, thanks for doing that!)

Additionally, CONTRIBUTING.md mentions "Make sure you test your code as described here", but the "here" is kinda ambiguous. Does it means the list that follows shortly afterwards, or was it meant to be a link, perhaps to the docs I pointed to above?

[timholy]

@IainNZ: wow. OK, I'll take a peek and see if I can figure out what's happening.

@waldyrious: very good suggestions. I've edited the post up top to include your link, and will look into editing CONTRIBUTING.md.

[timholy]

@IainNZ, the following test script works for me:

using Base.Test
s1 = Set(1,2,3)
s2 = Set(2,5,7)
s = intersect(s1, s2)
@test s == Set(2)

I get useful coverage results with or without #9354 and --inline=no:

        - function intersect(s::Set, sets::Set...)
        1     i = similar(s)
        1     for x in s
        3         inall = true
        3         for t in sets
        3             if !in(x,t)
        2                 inall = false
        2                 break
        -             end
        -         end
        3         inall && push!(i, x)
        -     end
        1     return i
        - end

Does that work for you? If not, what platform and commit are you on?

[svaksha]

Hi,
@timholy, I ran the tests for commit: 5219315, which threw this error:

julia> include("runtests.jl")
From worker 3: * linalg2
From worker 2: * linalg1
From worker 2: * linalg3
From worker 3: * linalg4
From worker 2: * linalg/lapack
From worker 2: * linalg/triangular
exception on 3: ERROR: test failed: transpose(qrfact(randn(3,3))) did not throw ErrorException
in expression: transpose(qrfact(randn(3,3)))
in error at error.jl:21
in default_handler at test.jl:27
in do_test_throws at test.jl:69
in runtests at /home/mom/julia/test/testdefs.jl:5
in anonymous at multi.jl:852
in run_work_thunk at multi.jl:603
in anonymous at task.jl:852
while loading linalg4.jl, in expression starting on line 205
ERROR: test failed: transpose(qrfact(randn(3,3))) did not throw ErrorException
in expression: transpose(qrfact(randn(3,3)))
in anonymous at task.jl:1616
while loading linalg4.jl, in expression starting on line 205
while loading /home/mom/julia/test/runtests.jl, in expression starting on line 42

I had another question regarding running these tests: Currently, for personal devel stuff I prefer to use the nightly julia build from Elliot/staticfloat for Ubuntu-14.04-LTS. But, for the test coverage, is it kosher to use and build julia in the /home/user/julia directory? I figured that since the nightlies run off a different space, this arrangement will give me a sandbox (not exactly, but then julia does not have an Anaconda / Canopy/ virtualenv env like python does) space. What are the drawbacks of doing this, if any? TIA.