Can't run tests in CI: TestItemRunner.compute_line_column(content, 1) == (line = 1, column = 1) error

[ianfiske]

I can use TestItemRunner locally fine. It works both in VSCode's UI and also doing ] test in my package using a basic runtests.jl

using TestItemRunner

@run_package_tests

However, when I run in our CI system, the tests fail because it's trying to run a test from within TestItemRunner itself.

  Test threw exception
  Expression: TestItemRunner.compute_line_column(content, 1) == (line = 1, column = 1)
  UndefVarError: `TestItemRunner` not defined
  Stacktrace:
   [1] macro expansion
     @ /usr/local/julia/share/julia/stdlib/v1.10/Test/src/Test.jl:669 [inlined]
   [2] top-level scope
     @ /builds/internal/MyPackage.jl/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl:508

I tried adding a filter to ensure it wouldn't pick up tests in TestItemRunner:

using TestItemRunner

# Filter necessary to avoid some Gitlab errors where it
# was detecting tests in TestItemRunner itself.
@run_package_tests filter = ti -> contains(dirname(ti.filename), "MyPackage")

but this didn't work. Any ideas?

[ianfiske]

In the (abridged) report, I see

Test Summary:                                                | Pass  Error  Total   Time
Package                                                      |   57      9     66  36.6s
  test/basic_tests.jl                                        |   16            16  22.4s
  test/components.jl                                         |   29            29   9.1s
  .julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl |           9      9   2.1s
    compute_line_column                                      |           9      9   2.1s

so TestItemRunner is clearly picking up tests within its own source code.

[ianfiske]

I was able to work around this and get tests to pass in CI with this

@run_package_tests filter = ti -> !contains(dirname(ti.filename), "TestItemRunner")

but I don't understand why it's picking these tests up in the first place and the workaround is needed.

[davidanthoff]

It looks to me as if the Julia depot being used is a child folder of the package your are testing? This path /builds/internal/MyPackage.jl/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl suggests that.

So what happens then is that all the packages that you are using are cloned into MyPackage.jl/.julia/packages etc and in general the test item framework picks up any test items that are in a Julia folder that is a sub folder of MyPackage.

Is there a reason why the depot ends up being there? That is a bit of a weird choice ;)

[davidanthoff]

But apart from that, if we fix #41 then this should also address the situation here.

[ianfiske]

Ah, good catch! Thanks for the explanation. Indeed we did intentionally set JULIA_DEPOT_PATH: "./.julia". This was done to allow Gitlab to cache the precompiled dependencies to speed up our CI jobs. Maybe there is another way. Or we can just wait for the #41 fix. Or we're good with my workaround for right now.

[kenahoo]

Hi folks - the workaround of doing

@run_package_tests filter = ti -> !contains(dirname(ti.filename), "TestItemRunner")

turns out not to be sufficient, because I've been getting the following error recently:

┌ Warning: Error in your test item or test setup definition
│   file = "/blah-blah/MyProject.jl/.julia/packages/ReTestItems/VrjGK/test/testfiles/_duplicate_id_test.jl"
│   errors =
│    2-element Vector{Any}:
│     (name = "name1", message = "Unknown keyword argument.", range = 1:52)
│     (name = "name2", message = "Unknown keyword argument.", range = 54:105)
└ @ TestItemRunner /blah-blah/MyProject.jl/.julia/packages/TestItemRunner/Q6zqe/src/TestItemRunner.jl:174

So in addition to TestItemRunner, it's necessary to filter out ReTestItems now too. I could add another heuristic that looks for something like /test/ in the path, but it seems like a better option might be to restrict the search to just the test/ directory in the first place. Is that feasible?

[kenahoo]

By way of pre-emptive explanation on "why is .julia/ inside the package directory" - like @ianfiske, we're using GitLab, and its cache mechanism requires that any cached items (in this case, downloaded packages) must be inside the working directory of the build:

Both artifacts and caches define their paths relative to the project directory, and can’t link to files outside it.

( https://docs.gitlab.com/ee/ci/caching/ )

[davidanthoff]

We can't in general limit things to the test folder as tests are allowed to be written inline in the package code.

The easiest change would just be to adjust the filter? Something like @run_package_tests filter = ti -> startswith(dirname(ti.filename), "PackageName/test") would presumably do the trick?

We could think about some kind of flag that limits things to the test folder, but I'm not super sure how to best do that as we don't really have any concept of a global test configuration...

[kenahoo]

Thanks for the suggestion, but unfortunately I can't fix this with a filter, because the error happens before any filtering is applied, in the if length(errors_for_file) > 0 partway through this code:

https://github.com/julia-vscode/TestItemRunner.jl/blob/main/src/TestItemRunner.jl#L174

    # Find all Julia files in this folder and sub folders
    julia_files = String[]
    for (root, _, files) in walkdir(path)
        for file in files
            if endswith(lowercase(file), ".jl")
                push!(julia_files, normpath(joinpath(root, file)))
            end
        end

    end

    # Find all @testitems and @testsetup
    testitems = Dict{String,Vector}()
    # testsetups maps @testsetup NAME => (filename, code, name, line, column)
    testsetups = Dict{Symbol,Any}()
    for file in julia_files
        content = read(file, String)

        stream = JuliaSyntax.ParseStream(content; version=VERSION)
        JuliaSyntax.parse!(stream; rule=:all)
        tree = JuliaSyntax.build_tree(JuliaSyntax.SyntaxNode, stream)

        testitems_for_file = []
        testsetups_for_file = []
        errors_for_file = []
        TestItemDetection.find_test_detail!(tree, testitems_for_file, testsetups_for_file, errors_for_file)

        if length(errors_for_file) > 0
            @warn "Error in your test item or test setup definition" file errors=errors_for_file
            error("There is an error in your test item or test setup definition, we are aborting.")
        end

        if length(testitems_for_file) > 0
            testitems[file] = [(filename=file, code=content[i.code_range], name=i.name, option_tags=i.option_tags, option_default_imports=i.option_default_imports, option_setup=i.option_setup, compute_line_column(content, i.code_range.start)...) for i in testitems_for_file]
        end
        for i in testsetups_for_file
            testsetups[i.name] = (filename=file, code=content[i.code_range], name=Symbol(i.name), kind=i.kind, compute_line_column(content, i.code_range.start)...)
        end
    end

    # Filter @testitems
    if filter !== nothing
        for file in keys(testitems)
            testitems[file] = Base.filter(i -> filter((filename=file, name=i.name, tags=i.option_tags)), testitems[file])
            isempty(testitems[file]) && pop!(testitems, file)
        end
    end

It seems like we would need a way to prevent those files from being scanned in the first place. A parameter list of starting directories would help a lot, I bet 99% of projects would have all tests in ["src", "test"]?