starting build "60af624b-8d8b-4bea-b2a8-38c0943adaab"

FETCHSOURCE
From https://github.com/datacommonsorg/website
 * branch            895221a467226970695e8733322c48e96baad118 -> FETCH_HEAD
Updating files:  89% (2180/2446)
Updating files:  90% (2202/2446)
Updating files:  91% (2226/2446)
Updating files:  92% (2251/2446)
Updating files:  93% (2275/2446)
Updating files:  94% (2300/2446)
Updating files:  95% (2324/2446)
Updating files:  96% (2349/2446)
Updating files:  97% (2373/2446)
Updating files:  98% (2398/2446)
Updating files:  99% (2422/2446)
Updating files: 100% (2446/2446)
Updating files: 100% (2446/2446), done.
HEAD is now at 895221a Bringing it back to working with UV
GitCommit:
895221a467226970695e8733322c48e96baad118
BUILD
Starting Step #0 - "explore_tests"
Step #0 - "explore_tests": Pulling image: python:3.11.3
Step #0 - "explore_tests": 3.11.3: Pulling from library/python
Step #0 - "explore_tests": bd73737482dd: Pulling fs layer
Step #0 - "explore_tests": 6710592d62aa: Pulling fs layer
Step #0 - "explore_tests": 75256935197e: Pulling fs layer
Step #0 - "explore_tests": c1e5026c6457: Pulling fs layer
Step #0 - "explore_tests": f0016544b8b9: Pulling fs layer
Step #0 - "explore_tests": 1d58eee51ff2: Pulling fs layer
Step #0 - "explore_tests": 93dc7b704cd1: Pulling fs layer
Step #0 - "explore_tests": caefdefa531e: Pulling fs layer
Step #0 - "explore_tests": 93dc7b704cd1: Waiting
Step #0 - "explore_tests": caefdefa531e: Waiting
Step #0 - "explore_tests": f0016544b8b9: Verifying Checksum
Step #0 - "explore_tests": f0016544b8b9: Download complete
Step #0 - "explore_tests": 6710592d62aa: Verifying Checksum
Step #0 - "explore_tests": 6710592d62aa: Download complete
Step #0 - "explore_tests": 1d58eee51ff2: Verifying Checksum
Step #0 - "explore_tests": 1d58eee51ff2: Download complete
Step #0 - "explore_tests": 75256935197e: Verifying Checksum
Step #0 - "explore_tests": 75256935197e: Download complete
Step #0 - "explore_tests": bd73737482dd: Verifying Checksum
Step #0 - "explore_tests": bd73737482dd: Download complete
Step #0 - "explore_tests": 93dc7b704cd1: Verifying Checksum
Step #0 - "explore_tests": 93dc7b704cd1: Download complete
Step #0 - "explore_tests": caefdefa531e: Verifying Checksum
Step #0 - "explore_tests": caefdefa531e: Download complete
Step #0 - "explore_tests": bd73737482dd: Pull complete
Step #0 - "explore_tests": c1e5026c6457: Verifying Checksum
Step #0 - "explore_tests": c1e5026c6457: Download complete
Step #0 - "explore_tests": 6710592d62aa: Pull complete
Step #0 - "explore_tests": 75256935197e: Pull complete
Step #0 - "explore_tests": c1e5026c6457: Pull complete
Step #0 - "explore_tests": f0016544b8b9: Pull complete
Step #0 - "explore_tests": 1d58eee51ff2: Pull complete
Step #0 - "explore_tests": 93dc7b704cd1: Pull complete
Step #0 - "explore_tests": caefdefa531e: Pull complete
Step #0 - "explore_tests": Digest: sha256:3a619e3c96fd4c5fc5e1998fd4dcb1f1403eb90c4c6409c70d7e80b9468df7df
Step #0 - "explore_tests": Status: Downloaded newer image for python:3.11.3
Step #0 - "explore_tests": docker.io/library/python:3.11.3
Step #0 - "explore_tests": --explore ### Running explore page integration tests
Step #0 - "explore_tests": [NOTICE]  server/.venv does not exist. Setting up website python virtual environment...
Step #0 - "explore_tests": installing server/requirements.txt
Step #0 - "explore_tests":   DEPRECATION: langdetect is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Step #0 - "explore_tests":   DEPRECATION: data-gemma is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Step #0 - "explore_tests":   DEPRECATION: flask_testing is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Step #0 - "explore_tests": 
Step #0 - "explore_tests": [notice] A new release of pip available: 22.3.1 -> 25.3
Step #0 - "explore_tests": [notice] To update, run: pip install --upgrade pip
Step #0 - "explore_tests": 
Step #0 - "explore_tests": [notice] A new release of pip available: 22.3.1 -> 25.3
Step #0 - "explore_tests": [notice] To update, run: pip install --upgrade pip
Step #0 - "explore_tests": Using ENV_PREFIX=Staging
Step #0 - "explore_tests": Starting servers using run_servers.sh...
Step #0 - "explore_tests": [NOTICE]  nl_server/.venv directory not found. Running './run_test.sh --setup_nl'.
Step #0 - "explore_tests": --setup_nl ### Set up NL python requirements
Step #0 - "explore_tests": installing nl_server/requirements.txt
Step #0 - "explore_tests": ============================= test session starts ==============================
Step #0 - "explore_tests": platform linux -- Python 3.11.3, pytest-9.0.2, pluggy-1.6.0 -- /workspace/server/.venv/bin/python3
Step #0 - "explore_tests": cachedir: .pytest_cache
Step #0 - "explore_tests": rootdir: /workspace
Step #0 - "explore_tests": configfile: pytest.ini
Step #0 - "explore_tests": plugins: rerunfailures-10.2, flakefinder-1.1.0, xdist-3.2.1, anyio-4.12.0
Step #0 - "explore_tests": gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
Step #0 - "explore_tests": 
[gw0] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw1] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw2] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw3] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw4] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw5] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw6] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw7] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw8] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw9] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw10] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw11] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw12] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw13] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw14] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw15] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw16] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw17] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw18] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw19] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw20] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw21] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw22] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw23] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw24] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw25] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw26] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw27] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw28] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw29] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw30] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw31] linux Python 3.11.3 cwd: /workspace
Step #0 - "explore_tests": 
[gw0] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw1] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw2] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw3] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw4] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw5] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw6] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw7] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw8] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw10] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw9] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw11] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw13] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw12] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw14] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw15] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw16] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw17] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw18] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw19] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw20] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw21] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw22] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw23] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw24] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw26] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw25] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw27] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw28] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw29] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw30] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": 
[gw31] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #0 - "explore_tests": gw0 [50] / gw1 [50] / gw2 [50] / gw3 [50] / gw4 [50] / gw5 [50] / gw6 [50] / gw7 [50] / gw8 [50] / gw9 [50] / gw10 [50] / gw11 [50] / gw12 [50] / gw13 [50] / gw14 [50] / gw15 [50] / gw16 [50] / gw17 [50] / gw18 [50] / gw19 [50] / gw20 [50] / gw21 [50] / gw22 [50] / gw23 [50] / gw24 [50] / gw25 [50] / gw26 [50] / gw27 [50] / gw28 [50] / gw29 [50] / gw30 [50] / gw31 [50]
Step #0 - "explore_tests": 
Step #0 - "explore_tests": scheduling tests via LoadScheduling
Step #0 - "explore_tests": 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_undata_ilo 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestEE1::test_e2e_answer_places 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_bugs 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_nl_size 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_sdg 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_context 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_bio 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestEE1::test_e2e_edge_cases2 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_uae 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_bio 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_undata_dev 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_undata 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_statvars 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_sdg_global 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_undata 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_translate 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_comparison 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_basic 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_sdg_global_specialvars 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_sfr 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_basic_sdg 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_sdg_specialvars 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_sdg 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestEE1::test_e2e_india_demo 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestEE1::test_e2e_edge_cases 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_multivar 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestDetection::test_detection_statvars 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestEE1::test_e2e_correlation_bugs 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_correlation 
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestEE1::test_e2e_electrification_demo 
Step #0 - "explore_tests": 
Step #0 - "explore_tests": [notice] A new release of pip available: 22.3.1 -> 25.3
Step #0 - "explore_tests": [notice] To update, run: pip install --upgrade pip
Step #0 - "explore_tests": server/integration_tests/explore_test.py::ExploreTestFulfillment::test_fulfillment_explore_more FLASK_ENV=integration_test
Step #0 - "explore_tests": GOOGLE_CLOUD_PROJECT=datcom-website-staging
Step #0 - "explore_tests": ENABLE_MODEL=true
Step #0 - "explore_tests": Starting NL Server...
Step #0 - "explore_tests": Starting Website server...
Step #0 - "explore_tests": [22:28:08][INFO    ][config_reader.py:120] Loading index and model catalog from: /workspace/nl_server/../deploy/helm_charts/dc_website/nl/catalog.yaml 
Step #0 - "explore_tests": [22:28:08][INFO    ][config_reader.py:86] server config:
Step #0 - "explore_tests": {
Step #0 - "explore_tests":   "version": "1",
Step #0 - "explore_tests":   "default_indexes": [
Step #0 - "explore_tests":     "base_uae_mem"
Step #0 - "explore_tests":   ],
Step #0 - "explore_tests":   "indexes": {
Step #0 - "explore_tests":     "base_uae_mem": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #0 - "explore_tests":       "model": "uae-large-v1-model",
Step #0 - "explore_tests":       "healthcheck_query": "Life expectancy",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "bio_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/bio",
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Gene",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "medium_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": null,
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Life expectancy",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "base_mistral_mem": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #0 - "explore_tests":       "model": "sfr-embedding-mistral-model",
Step #0 - "explore_tests":       "healthcheck_query": "Life expectancy",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "sdg_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/sdg",
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Hunger",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "undata_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/undata",
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Hunger",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "undata_ilo_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/undata_ilo",
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Employment",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv"
Step #0 - "explore_tests":     }
Step #0 - "explore_tests":   },
Step #0 - "explore_tests":   "models": {
Step #0 - "explore_tests":     "cross-encoder-ms-marco-miniilm-l6-v2": {
Step #0 - "explore_tests":       "type": "VERTEXAI",
Step #0 - "explore_tests":       "usage": "RERANKING",
Step #0 - "explore_tests":       "score_threshold": null,
Step #0 - "explore_tests":       "project_id": "datcom-website-dev",
Step #0 - "explore_tests":       "location": "us-central1",
Step #0 - "explore_tests":       "prediction_endpoint_id": "3977846152316846080"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "cross-encoder-mxbai-rerank-base-v1": {
Step #0 - "explore_tests":       "type": "VERTEXAI",
Step #0 - "explore_tests":       "usage": "RERANKING",
Step #0 - "explore_tests":       "score_threshold": null,
Step #0 - "explore_tests":       "project_id": "datcom-website-dev",
Step #0 - "explore_tests":       "location": "us-central1",
Step #0 - "explore_tests":       "prediction_endpoint_id": "284894457873039360"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "uae-large-v1-model": {
Step #0 - "explore_tests":       "type": "VERTEXAI",
Step #0 - "explore_tests":       "usage": "EMBEDDINGS",
Step #0 - "explore_tests":       "score_threshold": 0.7,
Step #0 - "explore_tests":       "project_id": "datcom-nl",
Step #0 - "explore_tests":       "location": "us-central1",
Step #0 - "explore_tests":       "prediction_endpoint_id": "8110162693219942400"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "ft-final-v20230717230459-all-MiniLM-L6-v2": {
Step #0 - "explore_tests":       "type": "LOCAL",
Step #0 - "explore_tests":       "usage": "EMBEDDINGS",
Step #0 - "explore_tests":       "score_threshold": 0.5,
Step #0 - "explore_tests":       "gcs_folder": "gs://datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "sfr-embedding-mistral-model": {
Step #0 - "explore_tests":       "type": "VERTEXAI",
Step #0 - "explore_tests":       "usage": "EMBEDDINGS",
Step #0 - "explore_tests":       "score_threshold": 0.5,
Step #0 - "explore_tests":       "project_id": "datcom-website-dev",
Step #0 - "explore_tests":       "location": "us-central1",
Step #0 - "explore_tests":       "prediction_endpoint_id": "224012300019826688"
Step #0 - "explore_tests":     }
Step #0 - "explore_tests":   },
Step #0 - "explore_tests":   "enable_reranking": true
Step #0 - "explore_tests": } 
Step #0 - "explore_tests": [22:28:08][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #0 - "explore_tests": [22:28:08][INFO    ][gcs.py:50] Download datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 to /tmp/datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 
Step #0 - "explore_tests": [22:28:09][INFO    ][SentenceTransformer.py:113] Load pretrained SentenceTransformer: /tmp/datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 
Step #0 - "explore_tests": [22:28:10][INFO    ][SentenceTransformer.py:219] Use pytorch device_name: cpu 
Step #0 - "explore_tests": [22:28:10][INFO    ][gcs.py:50] Download datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv to /tmp/datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv 
Step #0 - "explore_tests": [22:28:11][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s][22:28:13][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #0 - "explore_tests": 
Generating train split: 7305 examples [00:02, 3354.92 examples/s]
Generating train split: 7305 examples [00:02, 3348.61 examples/s]
Step #0 - "explore_tests": [22:28:14][INFO    ][gcs.py:50] Download datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv to /tmp/datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv 
Step #0 - "explore_tests": [22:28:14][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 92 examples [00:00, 1358.44 examples/s]
Step #0 - "explore_tests": [22:28:15][INFO    ][gcs.py:50] Download datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv to /tmp/datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv 
Step #0 - "explore_tests": [22:28:16][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 10000 examples [00:00, 11257.85 examples/s]
Generating train split: 20000 examples [00:01, 12529.86 examples/s][22:28:18][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #0 - "explore_tests": 
Generating train split: 30000 examples [00:02, 12285.84 examples/s]
Generating train split: 40000 examples [00:03, 12926.71 examples/s]
Generating train split: 42418 examples [00:03, 12712.18 examples/s]
Generating train split: 42418 examples [00:03, 12539.69 examples/s]
Step #0 - "explore_tests": [22:28:20][INFO    ][gcs.py:50] Download datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv to /tmp/datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv 
Step #0 - "explore_tests": [22:28:23][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s][22:28:23][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #0 - "explore_tests": [22:28:28][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #0 - "explore_tests": 
Generating train split: 7259 examples [00:09, 742.08 examples/s]
Generating train split: 7259 examples [00:09, 741.50 examples/s]
Step #0 - "explore_tests": [22:28:33][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #0 - "explore_tests": [22:28:33][INFO    ][gcs.py:50] Download datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv to /tmp/datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv 
Step #0 - "explore_tests": [22:28:34][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4565 examples [00:00, 10767.41 examples/s]
Generating train split: 4565 examples [00:00, 10729.89 examples/s]
Step #0 - "explore_tests": [22:28:34][INFO    ][gcs.py:50] Download datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv to /tmp/datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv 
Step #0 - "explore_tests": [22:28:35][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5474 examples [00:00, 11059.87 examples/s]
Generating train split: 5474 examples [00:00, 11024.96 examples/s]
Step #0 - "explore_tests": [22:28:35][INFO    ][gcs.py:50] Download datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv to /tmp/datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv 
Step #0 - "explore_tests": [22:28:36][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv 
Step #0 - "explore_tests": 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 86 examples [00:00, 1294.84 examples/s]
Step #0 - "explore_tests": [22:28:36][INFO    ][flask.py:79] NL Server Flask app initialized 
Step #0 - "explore_tests": [22:28:36][INFO    ][nl_app.py:27] Run nl server in local mode (host=localhost), port=6070 
Step #0 - "explore_tests":  * Serving Flask app 'nl_server.flask'
Step #0 - "explore_tests":  * Debug mode: on
Step #0 - "explore_tests": [22:28:46][INFO    ][config_reader.py:120] Loading index and model catalog from: /workspace/nl_server/../deploy/helm_charts/dc_website/nl/catalog.yaml 
Step #0 - "explore_tests": [22:28:46][INFO    ][config_reader.py:86] server config:
Step #0 - "explore_tests": {
Step #0 - "explore_tests":   "version": "1",
Step #0 - "explore_tests":   "default_indexes": [
Step #0 - "explore_tests":     "base_uae_mem"
Step #0 - "explore_tests":   ],
Step #0 - "explore_tests":   "indexes": {
Step #0 - "explore_tests":     "base_uae_mem": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #0 - "explore_tests":       "model": "uae-large-v1-model",
Step #0 - "explore_tests":       "healthcheck_query": "Life expectancy",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "bio_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/bio",
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Gene",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "medium_ft": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": null,
Step #0 - "explore_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #0 - "explore_tests":       "healthcheck_query": "Life expectancy",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv"
Step #0 - "explore_tests":     },
Step #0 - "explore_tests":     "base_mistral_mem": {
Step #0 - "explore_tests":       "store_type": "MEMORY",
Step #0 - "explore_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #0 - "explore_tests":       "model": "sfr-embedding-mistral-model",
Step #0 - "explore_tests":       "healthcheck_query": "Life expectancy",
Step #0 - "explore_tests":       "embeddings_path": "gs://datcom-nl-models/bas
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds/60af624b-8d8b-4bea-b2a8-38c0943adaab?project=879489846695.]
...
xplore_tests": [gw11] [100%] PASSED server/integration_tests/explore_test.py::ExploreTestEE2::test_e2e_single_date 
Step #0 - "explore_tests": 
Step #0 - "explore_tests": =============================== warnings summary ===============================
Step #0 - "explore_tests": server/.venv/lib/python3.11/site-packages/flask_babel/__init__.py:183: 32 warnings
Step #0 - "explore_tests":   /workspace/server/.venv/lib/python3.11/site-packages/flask_babel/__init__.py:183: DeprecationWarning: 'locked_cached_property' is deprecated and will be removed in Flask 2.4. Use a lock inside the decorated function if locking is needed.
Step #0 - "explore_tests":     @locked_cached_property
Step #0 - "explore_tests": 
Step #0 - "explore_tests": shared/lib/constants.py:33
Step #0 - "explore_tests": shared/lib/constants.py:33
Step #0 - "explore_tests":   /workspace/shared/lib/constants.py:33: DeprecationWarning: invalid escape sequence '\s'
Step #0 - "explore_tests":     f'(?<!{rate_word}\srate)(?<!{rate_word}\srates)'
Step #0 - "explore_tests": 
Step #0 - "explore_tests": shared/lib/constants.py:251
Step #0 - "explore_tests":   /workspace/shared/lib/constants.py:251: DeprecationWarning: invalid escape sequence '\W'
Step #0 - "explore_tests":     "WetBulb": ["wet(\W?)bulb",],
Step #0 - "explore_tests": 
Step #0 - "explore_tests": -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
Step #0 - "explore_tests": ======= 49 passed, 1 skipped, 35 warnings, 34 rerun in 375.86s (0:06:15) =======
Finished Step #0 - "explore_tests"
Starting Step #1 - "nl_tests"
Step #1 - "nl_tests": Already have image (with digest): python:3.11.3
Step #1 - "nl_tests": --nl ### Running nl page integration tests
Step #1 - "nl_tests": Using ENV_PREFIX=Staging
Step #1 - "nl_tests": Starting servers using run_servers.sh...
Step #1 - "nl_tests": FLASK_ENV=integration_test
Step #1 - "nl_tests": GOOGLE_CLOUD_PROJECT=datcom-website-staging
Step #1 - "nl_tests": ENABLE_MODEL=true
Step #1 - "nl_tests": Starting NL Server...
Step #1 - "nl_tests": Starting Website server...
Step #1 - "nl_tests": [22:30:11][INFO    ][config_reader.py:120] Loading index and model catalog from: /workspace/nl_server/../deploy/helm_charts/dc_website/nl/catalog.yaml 
Step #1 - "nl_tests": [22:30:11][INFO    ][config_reader.py:86] server config:
Step #1 - "nl_tests": {
Step #1 - "nl_tests":   "version": "1",
Step #1 - "nl_tests":   "default_indexes": [
Step #1 - "nl_tests":     "base_uae_mem"
Step #1 - "nl_tests":   ],
Step #1 - "nl_tests":   "indexes": {
Step #1 - "nl_tests":     "base_uae_mem": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #1 - "nl_tests":       "model": "uae-large-v1-model",
Step #1 - "nl_tests":       "healthcheck_query": "Life expectancy",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "bio_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/bio",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Gene",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "medium_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": null,
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Life expectancy",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "base_mistral_mem": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #1 - "nl_tests":       "model": "sfr-embedding-mistral-model",
Step #1 - "nl_tests":       "healthcheck_query": "Life expectancy",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "sdg_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/sdg",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Hunger",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "undata_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/undata",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Hunger",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "undata_ilo_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/undata_ilo",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Employment",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv"
Step #1 - "nl_tests":     }
Step #1 - "nl_tests":   },
Step #1 - "nl_tests":   "models": {
Step #1 - "nl_tests":     "cross-encoder-ms-marco-miniilm-l6-v2": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "RERANKING",
Step #1 - "nl_tests":       "score_threshold": null,
Step #1 - "nl_tests":       "project_id": "datcom-website-dev",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "3977846152316846080"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "cross-encoder-mxbai-rerank-base-v1": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "RERANKING",
Step #1 - "nl_tests":       "score_threshold": null,
Step #1 - "nl_tests":       "project_id": "datcom-website-dev",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "284894457873039360"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "uae-large-v1-model": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "EMBEDDINGS",
Step #1 - "nl_tests":       "score_threshold": 0.7,
Step #1 - "nl_tests":       "project_id": "datcom-nl",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "8110162693219942400"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "ft-final-v20230717230459-all-MiniLM-L6-v2": {
Step #1 - "nl_tests":       "type": "LOCAL",
Step #1 - "nl_tests":       "usage": "EMBEDDINGS",
Step #1 - "nl_tests":       "score_threshold": 0.5,
Step #1 - "nl_tests":       "gcs_folder": "gs://datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "sfr-embedding-mistral-model": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "EMBEDDINGS",
Step #1 - "nl_tests":       "score_threshold": 0.5,
Step #1 - "nl_tests":       "project_id": "datcom-website-dev",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "224012300019826688"
Step #1 - "nl_tests":     }
Step #1 - "nl_tests":   },
Step #1 - "nl_tests":   "enable_reranking": true
Step #1 - "nl_tests": } 
Step #1 - "nl_tests": [22:30:12][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #1 - "nl_tests": [22:30:12][INFO    ][gcs.py:50] Download datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 to /tmp/datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 
Step #1 - "nl_tests": [22:30:13][INFO    ][SentenceTransformer.py:113] Load pretrained SentenceTransformer: /tmp/datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 
Step #1 - "nl_tests": ============================= test session starts ==============================
Step #1 - "nl_tests": platform linux -- Python 3.11.3, pytest-9.0.2, pluggy-1.6.0 -- /workspace/server/.venv/bin/python3
Step #1 - "nl_tests": cachedir: .pytest_cache
Step #1 - "nl_tests": rootdir: /workspace
Step #1 - "nl_tests": configfile: pytest.ini
Step #1 - "nl_tests": plugins: rerunfailures-10.2, flakefinder-1.1.0, xdist-3.2.1, anyio-4.12.0
Step #1 - "nl_tests": gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
Step #1 - "nl_tests": 
[gw0] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw1] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw2] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw3] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw4] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw5] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw6] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw7] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw8] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw9] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw10] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw11] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw12] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw13] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw14] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw15] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw16] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw17] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw18] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw19] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw20] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw21] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw22] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw23] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw24] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw25] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw26] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw27] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw28] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw29] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw30] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw31] linux Python 3.11.3 cwd: /workspace
Step #1 - "nl_tests": 
[gw0] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw1] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw2] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw3] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw4] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw5] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw6] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw7] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw8] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw10] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw9] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw11] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw12] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw13] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw16] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw14] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw15] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw18] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw17] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw19] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw20] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw22] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw21] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw24] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw23] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw25] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw26] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw27] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw28] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw29] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw30] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": 
[gw31] Python 3.11.3 (main, May 23 2023, 13:25:46) [GCC 10.2.1 20210110]
Step #1 - "nl_tests": gw0 [14] / gw1 [14] / gw2 [14] / gw3 [14] / gw4 [14] / gw5 [14] / gw6 [14] / gw7 [14] / gw8 [14] / gw9 [14] / gw10 [14] / gw11 [14] / gw12 [14] / gw13 [14] / gw14 [14] / gw15 [14] / gw16 [14] / gw17 [14] / gw18 [14] / gw19 [14] / gw20 [14] / gw21 [14] / gw22 [14] / gw23 [14] / gw24 [14] / gw25 [14] / gw26 [14] / gw27 [14] / gw28 [14] / gw29 [14] / gw30 [14] / gw31 [14]
Step #1 - "nl_tests": 
Step #1 - "nl_tests": scheduling tests via LoadScheduling
Step #1 - "nl_tests": 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_international 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_sdg 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_strict_default_place 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestDemo::test_demo_climatetrace 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestDemo::test_demo_feb2023 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_strict_low_confidence 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestDemo::test_textbox_sample 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_strict_multi_verb 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestDemo::test_demo_fallback 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_demo_usa_map_types 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestDemo::test_demo_multisv 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_translate 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestMisc::test_place_detection_e2e_dc 
Step #1 - "nl_tests": [22:30:14][INFO    ][SentenceTransformer.py:219] Use pytorch device_name: cpu 
Step #1 - "nl_tests": [22:30:14][INFO    ][gcs.py:50] Download datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv to /tmp/datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv 
Step #1 - "nl_tests": [22:30:15][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv 
Step #1 - "nl_tests": [22:30:16][INFO    ][gcs.py:50] Download datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv to /tmp/datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv 
Step #1 - "nl_tests": [22:30:16][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv 
Step #1 - "nl_tests": [22:30:16][INFO    ][gcs.py:50] Download datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv to /tmp/datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv 
Step #1 - "nl_tests": [22:30:17][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #1 - "nl_tests": [22:30:17][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv 
Step #1 - "nl_tests": [22:30:18][INFO    ][gcs.py:50] Download datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv to /tmp/datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv 
Step #1 - "nl_tests": [22:30:22][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #1 - "nl_tests": [22:30:25][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv 
Step #1 - "nl_tests": [22:30:27][INFO    ][gcs.py:50] Download datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv to /tmp/datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv 
Step #1 - "nl_tests": [22:30:27][INFO    ][util.py:607] ['http://localhost:6070/healthz'] not ready, waiting for 5 seconds 
Step #1 - "nl_tests": [22:30:27][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv 
Step #1 - "nl_tests": [22:30:28][INFO    ][gcs.py:50] Download datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv to /tmp/datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv 
Step #1 - "nl_tests": [22:30:28][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv 
Step #1 - "nl_tests": [22:30:29][INFO    ][gcs.py:50] Download datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv to /tmp/datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv 
Step #1 - "nl_tests": [22:30:29][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv 
Step #1 - "nl_tests": [22:30:29][INFO    ][flask.py:79] NL Server Flask app initialized 
Step #1 - "nl_tests": [22:30:29][INFO    ][nl_app.py:27] Run nl server in local mode (host=localhost), port=6070 
Step #1 - "nl_tests": server/integration_tests/nl_test.py::NLTestDemo::test_demo_cities_feb2023  * Serving Flask app 'nl_server.flask'
Step #1 - "nl_tests":  * Debug mode: on
Step #1 - "nl_tests": [22:30:50][INFO    ][config_reader.py:120] Loading index and model catalog from: /workspace/nl_server/../deploy/helm_charts/dc_website/nl/catalog.yaml 
Step #1 - "nl_tests": [22:30:50][INFO    ][config_reader.py:86] server config:
Step #1 - "nl_tests": {
Step #1 - "nl_tests":   "version": "1",
Step #1 - "nl_tests":   "default_indexes": [
Step #1 - "nl_tests":     "base_uae_mem"
Step #1 - "nl_tests":   ],
Step #1 - "nl_tests":   "indexes": {
Step #1 - "nl_tests":     "base_uae_mem": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #1 - "nl_tests":       "model": "uae-large-v1-model",
Step #1 - "nl_tests":       "healthcheck_query": "Life expectancy",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "bio_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/bio",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Gene",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "medium_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": null,
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Life expectancy",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "base_mistral_mem": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/base",
Step #1 - "nl_tests":       "model": "sfr-embedding-mistral-model",
Step #1 - "nl_tests":       "healthcheck_query": "Life expectancy",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "sdg_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/sdg",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Hunger",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "undata_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/undata",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Hunger",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "undata_ilo_ft": {
Step #1 - "nl_tests":       "store_type": "MEMORY",
Step #1 - "nl_tests":       "source_path": "/workspace/tools/nl/embeddings/input/undata_ilo",
Step #1 - "nl_tests":       "model": "ft-final-v20230717230459-all-MiniLM-L6-v2",
Step #1 - "nl_tests":       "healthcheck_query": "Employment",
Step #1 - "nl_tests":       "embeddings_path": "gs://datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv"
Step #1 - "nl_tests":     }
Step #1 - "nl_tests":   },
Step #1 - "nl_tests":   "models": {
Step #1 - "nl_tests":     "cross-encoder-ms-marco-miniilm-l6-v2": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "RERANKING",
Step #1 - "nl_tests":       "score_threshold": null,
Step #1 - "nl_tests":       "project_id": "datcom-website-dev",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "3977846152316846080"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "cross-encoder-mxbai-rerank-base-v1": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "RERANKING",
Step #1 - "nl_tests":       "score_threshold": null,
Step #1 - "nl_tests":       "project_id": "datcom-website-dev",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "284894457873039360"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "uae-large-v1-model": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "EMBEDDINGS",
Step #1 - "nl_tests":       "score_threshold": 0.7,
Step #1 - "nl_tests":       "project_id": "datcom-nl",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "8110162693219942400"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "ft-final-v20230717230459-all-MiniLM-L6-v2": {
Step #1 - "nl_tests":       "type": "LOCAL",
Step #1 - "nl_tests":       "usage": "EMBEDDINGS",
Step #1 - "nl_tests":       "score_threshold": 0.5,
Step #1 - "nl_tests":       "gcs_folder": "gs://datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2"
Step #1 - "nl_tests":     },
Step #1 - "nl_tests":     "sfr-embedding-mistral-model": {
Step #1 - "nl_tests":       "type": "VERTEXAI",
Step #1 - "nl_tests":       "usage": "EMBEDDINGS",
Step #1 - "nl_tests":       "score_threshold": 0.5,
Step #1 - "nl_tests":       "project_id": "datcom-website-dev",
Step #1 - "nl_tests":       "location": "us-central1",
Step #1 - "nl_tests":       "prediction_endpoint_id": "224012300019826688"
Step #1 - "nl_tests":     }
Step #1 - "nl_tests":   },
Step #1 - "nl_tests":   "enable_reranking": true
Step #1 - "nl_tests": } 
Step #1 - "nl_tests": [22:30:50][INFO    ][SentenceTransformer.py:113] Load pretrained SentenceTransformer: /tmp/datcom-nl-models/ft_final_v20230717230459.all-MiniLM-L6-v2 
Step #1 - "nl_tests": [22:30:52][INFO    ][SentenceTransformer.py:219] Use pytorch device_name: cpu 
Step #1 - "nl_tests": [22:30:52][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/base_uae_mem_2025_11_03_07_10_42/embeddings.csv 
Step #1 - "nl_tests": [22:30:52][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/bio_ft_2024_11_08_19_00_38/embeddings.csv 
Step #1 - "nl_tests": [22:30:53][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/embeddings_medium_2024_05_09_18_01_32.ft_final_v20230717230459.all-MiniLM-L6-v2.csv 
Step #1 - "nl_tests": [22:30:53][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/base_mistral_mem_2024_07_01_10_23_43/embeddings.csv 
Step #1 - "nl_tests": [22:30:54][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/sdg_ft_2024_06_24_23_45_46/embeddings.csv 
Step #1 - "nl_tests": [22:30:55][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/undata_ft_2024_06_24_23_47_04/embeddings.csv 
Step #1 - "nl_tests": [22:30:55][INFO    ][memory.py:67] Loading embeddings file: /tmp/datcom-nl-models/undata_ilo_ft_2024_10_14_13_45_50/embeddings.csv 
Step #1 - "nl_tests": [22:30:56][INFO    ][flask.py:79] NL Server Flask app initialized 
Step #1 - "nl_tests": [22:30:56][INFO    ][nl_app.py:27] Run nl server in local mode (host=localhost), port=6070 
Step #1 - "nl_tests": [22:30:56][WARNING ][_internal.py:97]  * Debugger is active! 
Step #1 - "nl_tests": [22:30:56][INFO    ][util.py:593] http://localhost:6070/healthz is up running 
Step #1 - "nl_tests": [22:30:56][INFO    ][gcs.py:50] Download datcom-website-config/nl_bad_words.txt to /tmp/datcom-website-config/nl_bad_words.txt 
Step #1 - "nl_tests": [22:30:57][INFO    ][util.py:593] https://staging.api.datacommons.org/version is up running 
Step #1 - "nl_tests": [22:30:57][INFO    ][web_app.py:28] Run web server in local mode 
Step #1 - "nl_tests":  * Serving Flask app 'server.__init__'
Step #1 - "nl_tests":  * Debug mode: on
Step #1 - "nl_tests": [22:31:08][INFO    ][util.py:593] http://localhost:6070/healthz is up running 
Step #1 - "nl_tests": [22:31:09][INFO    ][util.py:593] https://staging.api.datacommons.org/version is up running 
Step #1 - "nl_tests": [22:31:09][INFO    ][web_app.py:28] Run web server in local mode 
Step #1 - "nl_tests": [22:31:10][WARNING ][_internal.py:97]  * Debugger is active! 
Step #1 - "nl_tests": 
Step #1 - "nl_tests": [gw11] [  7%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_strict_low_confidence 
Step #1 - "nl_tests": [gw9] [ 14%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_strict_default_place 
Step #1 - "nl_tests": [22:31:16][INFO    ][llm_api.py:87] Gemini model used for LLM API: gemini-2.5-flash 
Step #1 - "nl_tests": [22:31:16][INFO    ][models.py:4993] AFC is enabled with max remote calls: 10. 
Step #1 - "nl_tests": [gw12] [ 21%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_strict_multi_verb 
Step #1 - "nl_tests": [gw13] [ 28%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_translate 
Step #1 - "nl_tests": [gw5] [ 35%] PASSED server/integration_tests/nl_test.py::NLTestDemo::test_textbox_sample 
Step #1 - "nl_tests": [22:31:20][INFO    ][_client.py:1025] HTTP Request: POST https://generativelanguage.googleapis.com/v1/models/gemini-2.5-flash:generateContent "HTTP/1.1 200 OK" 
Step #1 - "nl_tests": [gw1] [ 42%] PASSED server/integration_tests/nl_test.py::NLTestDemo::test_demo_climatetrace 
Step #1 - "nl_tests": [gw10] [ 50%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_sdg 
Step #1 - "nl_tests": [gw6] [ 57%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_demo_usa_map_types 
Step #1 - "nl_tests": [gw4] [ 64%] PASSED server/integration_tests/nl_test.py::NLTestDemo::test_demo_multisv 
Step #1 - "nl_tests": [gw8] [ 71%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_place_detection_e2e_dc 
Step #1 - "nl_tests": [gw2] [ 78%] PASSED server/integration_tests/nl_test.py::NLTestDemo::test_demo_fallback 
Step #1 - "nl_tests": [gw7] [ 85%] PASSED server/integration_tests/nl_test.py::NLTestMisc::test_international 
Step #1 - "nl_tests": [gw0] [ 92%] PASSED server/integration_tests/nl_test.py::NLTestDemo::test_demo_cities_feb2023 
Step #1 - "nl_tests": [gw3] [100%] PASSED server/integration_tests/nl_test.py::NLTestDemo::test_demo_feb2023 
Step #1 - "nl_tests": 
Step #1 - "nl_tests": =============================== warnings summary ===============================
Step #1 - "nl_tests": server/.venv/lib/python3.11/site-packages/flask_babel/__init__.py:183: 32 warnings
Step #1 - "nl_tests":   /workspace/server/.venv/lib/python3.11/site-packages/flask_babel/__init__.py:183: DeprecationWarning: 'locked_cached_property' is deprecated and will be removed in Flask 2.4. Use a lock inside the decorated function if locking is needed.
Step #1 - "nl_tests":     @locked_cached_property
Step #1 - "nl_tests": 
Step #1 - "nl_tests": -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
Step #1 - "nl_tests": ================= 14 passed, 32 warnings in 102.11s (0:01:42) ==================
Finished Step #1 - "nl_tests"
PUSH
DONE