The inverse matrix algorithm used by this library is the cofactor method which is extremely inefficient when n becomes large enough. It scales in O(n^5).
With Gaussian elimination it scales in O(n^3) which with a matrix in the order of hundreds means 10^4 times faster. I tried a naive gaussian elimination method and tests are passing.
But a question arises now in my mind: was there any specific reason why gaussian method wasn't taken in consideration?
Otherwise I might have a PR ready