ENH: Update MLIR backend to LLVM 20.dev

[mtsokol]

This PR replaces #787 and fixes #797

---

This PR updates MLIR backend to current LLVM 20.dev (so main branch):

• I ran it locally against latest LLVM version.
• Moved COO format to SoA convention link.
• Updated tensor.empty call link.

As one can see it fixes a bunch of skips in the test suite: sparse/mlir_backend/tests/test_simple.py

• Dense+Dense and COO+COO now works.
• Reshaping Dense and COO formats also works.

It's thanks to changes already present in main branch, added after 19.x branched, and:

• [mlir][sparse] fix bug with all-dense assembler llvm/llvm-project#108615
• [MLIR][sparse] Add soa property to sparse_tensor Python bindings llvm/llvm-project#109135
• [MLIR][Python] Add encoding argument to tensor.empty Python function llvm/llvm-project#110656
• [MLIR][sparse] Fix SparseTensor test_output.py test llvm/llvm-project#110882

[codspeed-hq]

CodSpeed Performance Report

Merging #799 will degrade performances by 24.56%

_{Comparing updated-llvm-nightly-test (d511c5c) with main (bbe2b58)}

Summary

❌ 1 regressions
✅ 339 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	updated-llvm-nightly-test	Change
❌	test_index_slice[side=100-rank=2-format='gcxs']	2.5 ms	3.4 ms	-24.56%

[mtsokol]

We still need more wheels for finch-milr package on PyPI but the PR itself can be already reviewed.

[hameerabbasi]

Thanks for the test changes, overall the essence of this change LGTM, it is mlir -> Finch_mlir and figuring out the COO format better. Thanks!

[hameerabbasi]

This should probably handle SOA and without SOA separately.

[mtsokol]

In my opinion we should only support SoA singleton format:

1. Non-SoA singleton looks to be buggy for basic operations link
2. Mixed singleton levels aren't allowed: https://github.com/llvm/llvm-project/blob/8d38fbf2f027c72332c8ba03ff0ff0f83b4dcf02/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp#L811

What would be the benefit of supporting non-SoA singleton levels separately?

[hameerabbasi]

We'd be able to support the current COO format only for non-SoA.

[mtsokol]

What do you mean by that? Can you give some example?
With this PR we can support COO format (also input objects of type scipy.sparse.coo_array) where MLIR-backend implementation uses SoA representation.

[hameerabbasi]

The existing Numba COO format uses the non-SoA format, and we pretty much have to support this to be backwards compatible. Doesn't have to be in this PR though.

[mtsokol]

I think I'm still missing the point here. Numba is a separate backend that supports only 1D and 2D COO arrays. MLIR backend supports >=2D COO arrays.
What do you mean by backward compatibility here? Can you give an example where backward compatibility breaks here? If a user passes scipy.sparse.coo_array object to sparse.asarray function it it will work for any backend regardless of an internal representation (SoA or non-SoA).

[hameerabbasi]

So, sparse.COO has a constructor that takes coords and a .coords attribute. The attribute is a 2D NumPy array.

Ideally, I'd like to keep it a 2D np.ndarray as otherwise I'm not 100% sure how much will break.

We could do this with np.stack, but that would incur a performance penalty.

Also the current Numba backend supports ND, not 2D. SciPy supports only 2D, however.

I'm thinking of a future where all of sparse is powered by Finch-MLIR, ideally.

[mtsokol]

I think for accessing coords, np.stack could be an option. My caveat for non-SoA in MLIR backend is that it already has some issues with basic operations: https://discourse.llvm.org/t/passmanager-fails-on-simple-coo-addition-example/81247

[hameerabbasi]

Just one comment plus CI/Windows and should be good to go.