Release state and E2E/integration testing

[brentforder]

I really like this library, but I think we may have jumped the gun on leaving pre-release / beta release status with 10.1.x.
This downstream library of the native libraries is inherently unstable without an E2E approach (featuring good integration testing coverage).

I propose that we:

1. Switch to "pre-release" status and beta versioning
2. Plan some kind of untimed sprint to resolve the properly-reported bugs directly related to MLRN component methods and events
3. Require adding integration (MLRN<->native) tests related to each method/event fixed (see Create more E2E Tests #541)
4. Document integration tests for all supported native methods/events as a contributor priority
5. Return to normal release status and versioning

[tyrauber]

@brentforder, I don't understand what you mean by "leaving pre-release / beta release status with v10".

What exactly is the problem with our automated release workflow?

@KiwiKilian, Are betas not tagged as pre-release? Wouldn't that resolve the issue?

Regarding E2E tests, I think E2E tests is the wrong terminology here. E2E testing in my experience is verifying an application's functionality from the user's perspective, typically for react-native done with detox, or the more modern, maestro.

Our automated CI/CD E2E system is frankly amazing. Every alpha, beta or main change, builds the app and runs all the examples for both react-native app through maestro for both android and ios.

I think the issue that you are raising is that we might need more unit testing on the integration layer between this library and the upstream native libraries. That seems where the majority of the bugs are now coming - either upstream in the library, or our integration. Writing additional unit tests to determine where the "fault" lies would be helpful.

I think the bigger issue is the question of new arch support. That adds qanother entire layer of complexity. We probably need a new arch branch, in addition to alpha and beta, that configures the react-native and expo apps to use the new arch. Then we can start building tests around the new arch specifically, as there are definitely new arch only bugs related to implementation and integration.

With that being said - full new arch support would require a substantial rewrite, and most likely not be backwards compatible. Unfortunately, doing that right would probably require contracting someone like Software Mansion or Margelo to assist, which would require signifiant funding.

Sorry for the brain dump, but I feel like this is the crux of the issue:

• Keeping up with changes to native SDKs is tough. They move quickly.
• Debugging native SDK bugs and triaging between cross-library team is tough
• Re-writing for new arch would require significant investment and most likely make the library new-arch only

@KiwiKilian Does that about sum the state of things. Please correct me if I misspoke or missed any pertinent points.

[brentforder]

@tyrauber

I don't understand what you mean by "leaving pre-release / beta release status with v10".

Oops, I mean specifically with 10.1.x We were in pre-release / beta in 10.0.x, and that is no longer the case in 10.1.x.

What exactly is the problem with our automated release workflow?

The problem with the automated release workflow is that it does not employ integration tests for the native libraries.

Integration tests can use the same testing framework as unit tests, but because their concern is the integration of two systems (MLRN + native library), they should be at a higher-level than MLRN code units. The MLRN examples do not contain such tests (i.e. per #541), and the examples' successful execution amounts to low coverage % of the integrated functionality. This means that very simple bugs are easy to sneak in each release.

Also yes, it's true that E2E is a broader concept than integration testing. I coupled the terms E2E and integration testing here because of #541, but it makes more sense to say that an E2E testing approach prescribes integration testing for a downstream library like MLRN.

[KiwiKilian]

... and runs all the examples

It only runs the most basic example. That's why there is #541.

No clue, if they should rather be called E2E or integration tests.

[KiwiKilian]

I can repeat, what I answered to the similar question on Slack a month ago. Most issues getting reported are:

• Limitations of MLN (e.g. marker rendering behavior)
• Upstream issues (e.g. problems with specific tiles etc)
• Implementation issues (misunderstanding of the lib or MapLibre in general)
• Bugs with the new architecture

Have a look at the currently open bugs. It's quite imminent, that there are few to none critical problems, when on the old arch. Why should v10 still be in a pre-release state? It offers lots of improvements for users of the old arch and the JS/TS codebase. New arch support is basically "it compiles".

Yes, more tests are highly appreciated. Anyone can feel free to open PRs adding some! But looking into the close future, I personally would prioritize moving to the new arch first and add tests along that. Because stuff will change along that migration.