CTS tests failing - xtdb

[planetf1]

The CTS tests for xtdb have been been failing for over a week.

For example https://github.com/odpi/egeria-test-cts/actions/runs/5239075143 reports:

The xtdb test times out - details showing:

It's not obvious that there is a specific xtdb or base change that has caused this behaviour

[planetf1]

Actions to consider

• Update test script to ensure the timeout for running the test is enough under 6 hours, so that we can at least continue the script if the test times out. For example using 5:45 may be prudent, in addition to adding some steps to extract k8s console logs
• Use the built in 'ssh' capability so that the test is initiated, and then interactive k8s activities can be used to identify what is going on
• try running CTS for xtdb locally, or in another more accessible environment (openshift, rancher etc)

[planetf1]

I've added a PR to adjust timeouts & try to capture console log.

Meanwhile I ran a session with ssh access, and can see that there is a failure to start the tests:

Mon Jun 12 11:35:48 GMT 2023 cts Information CONFORMANCE-SUITE-0008 The Open Metadata Repository Conformance Workbench repository-workbench is waiting for server tut to join the cohort
Mon Jun 12 11:35:58 GMT 2023 cts Information CONFORMANCE-SUITE-0008 The Open Metadata Repository Conformance Workbench repository-workbench is waiting for server tut to join the cohort
Mon Jun 12 11:36:08 GMT 2023 cts Information CONFORMANCE-SUITE-0008 The Open Metadata Repository Conformance Workbench repository-workbench is waiting for server tut to join the cohort
Mon Jun 12 11:37:04 GMT 2023 cts Information CONFORMANCE-SUITE-0008 The Open Metadata Repository Conformance Workbench repository-workbench is waiting for server tut to join the cohort
Mon Jun 12 11:37:29 GMT 2023 cts Information CONFORMANCE-SUITE-0008 The Open Metadata Repository Conformance Workbench repository-workbench is waiting for server tut to join the cohort

This likely means the tut never started -- and the most common cause of that is library loading issues

In fact in the logs, there are zero log entries from the tut server itself -- and even the cts server is reporting restarts

Possible dependency clash? will check locally.

[planetf1]

The logs show that an exception occurs initializing the server:

"OCF-CONNECTION-400-005 Unknown Connector Provider class org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSRepositoryConnectorProvider passed in connection <Unknown>","The system is unable to create the requested connector instance because the Connector Provider's class is not known to the JVM.  This may be because the Co

(truncated)

Note that this is logged when configuring the cts -- so not in the platform output log....

The cause is that the download of the connector (also not clearly reported) fails, due to

wget http://repository.sonatype.org/service/local/artifact/maven/redirect\?r\=central-proxy\&g\=org.odpi.egeria\&a\=egeria-connector-xtdb\&v\=LATEST\&c\=jar-with-dependencies
--2023-06-12 16:08:01--  http://repository.sonatype.org/service/local/artifact/maven/redirect?r=central-proxy&g=org.odpi.egeria&a=egeria-connector-xtdb&v=LATEST&c=jar-with-dependencies
Resolving repository.sonatype.org (repository.sonatype.org)... 54.89.192.189, 34.198.184.168
Connecting to repository.sonatype.org (repository.sonatype.org)|54.89.192.189|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://repository.sonatype.org/service/local/artifact/maven/redirect?r=central-proxy&g=org.odpi.egeria&a=egeria-connector-xtdb&v=LATEST&c=jar-with-dependencies [following]
--2023-06-12 16:08:01--  https://repository.sonatype.org/service/local/artifact/maven/redirect?r=central-proxy&g=org.odpi.egeria&a=egeria-connector-xtdb&v=LATEST&c=jar-with-dependencies
Connecting to repository.sonatype.org (repository.sonatype.org)|54.89.192.189|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

This previously failed and was reported at https://community.sonatype.com/t/issues-or-something-changed-cannot-download-artifact-without-user-pass-prompt/10593

It was then fixed, but in the last week+ has again failed.

A more resiliant fix would be to

• enable charts to download via maven API instead
• when building xtdb etc, post in github as an artifact, and use within egeria test cts

Both require dev work.

Some improvements may be useful in the way failed downloads are detected/reported.

cc: @mandy-chessell @lpalashevski

[planetf1]

One of the sonatype community leaders has posted to the thread advising that the team has been informed, so hopefully someone will look at the download issues & then our CTS will be running again.