!here causes ValueError/IndexError

[LongHairedHacker]

I've just set up pears on local machine for testing, according to the README.md.

Steps I used for set up

$ git clone https://github.com/PeARSearch/PeARS-federated.git 
# Commit the got checked out is the lastest main f4b1c972def1d5b4b9e1a0998209f8676480b31b
$ python -m venv .venv
$ . .venv/bin/activate
$ pip install -r requirements.txt
$ cp .env-template .env
# Replaced SECRET_KEY, SECURITY_PASSWORD_SALT and CSRF_SESSION_KEY with random strings
$ flask pears install-language de
# Set PEARS_LANGS to en,de
$ flask pears create-user sebastian XXXXXXX  sebastian@example.com
$ flask pears setadmin sebastian

If I run it using python3 run.py, open http://localhost:8080/ and search for !here test I get:

$ python3 run.py
PATH /home/sebastian/build/PeARS-federated/app
Installed languages: ['en', 'de']
	>> ERROR: filter_instances_by_language: got non-200 status code when trying to access https://pears.cc/api/languages...
 * Serving Flask app 'app'
 * Debug mode: on
PATH /home/sebastian/build/PeARS-federated/app
Installed languages: ['en', 'de']
	>> ERROR: filter_instances_by_language: got non-200 status code when trying to access https://pears.cc/api/languages...

>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in en
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG en
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁experiments', '▁detection', '▁screening', '▁test', '▁evaluation', '▁preliminary', '▁tested', '▁testing', '▁tests', '▁trials']]
Traceback (most recent call last):
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 46, in index
    clean_query, results = get_local_search_results(query)
                           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 132, in get_local_search_results
    r, s = score_pages.run_search(clean_query, lang, extended=app.config['EXTEND_QUERY'])
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 194, in run_search
    document_scores = compute_scores(query, q_vectors, lang)
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 99, in compute_scores
    m, bins, podnames, urls = load_vec_matrix(lang)
                              ~~~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 90, in load_vec_matrix
    m, bins, podnames, urls = mk_vec_matrix(lang)
                              ~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 78, in mk_vec_matrix
    m = vstack(m)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/scipy/sparse/_construct.py", line 849, in vstack
    return _block([[b] for b in blocks], format, dtype, return_spmatrix=True)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/scipy/sparse/_construct.py", line 966, in _block
    raise ValueError('blocks must be 2-D')
ValueError: blocks must be 2-D

I assumed this was because there were no categories/themes/pods (the naming in the code is bit confusing to me) on this instance.
Still this should not cause an exception.

So I added a bunch of URLs from my blog test things out. As soons as I had two pods the error change to:

>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in en
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG en
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁screening', '▁preliminary', '▁testing', '▁test', '▁tests', '▁tested', '▁evaluation', '▁trials', '▁experiments', '▁detection']]
>> TIMER: Function 'mk_vec_matrix' executed in 0.0018s
>> TIMER: Function 'compute_scores' executed in 0.0025s

>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in de
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG de
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁test', '▁start', '▁pilot', 'flüge', '▁gestartet', '▁durchgeführt', '▁experiment', 'hubschrauber', '▁versuch', 'training']]
Traceback (most recent call last):
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 46, in index
    clean_query, results = get_local_search_results(query)
                           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 132, in get_local_search_results
    r, s = score_pages.run_search(clean_query, lang, extended=app.config['EXTEND_QUERY'])
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 194, in run_search
    document_scores = compute_scores(query, q_vectors, lang)
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 99, in compute_scores
    m, bins, podnames, urls = load_vec_matrix(lang)
                              ~~~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 90, in load_vec_matrix
    m, bins, podnames, urls = mk_vec_matrix(lang)
                              ~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 74, in mk_vec_matrix
    npz = npz[idvs,:]
          ~~~^^^^^^^^
IndexError: index 2 is out of bounds for axis 0 with size 2

Worth noting: My blog is in german and english, so I naturally have pods in both languages now.
As far as I can tell from the log output the english search worked just fine. However the german one failed.

I'm assuming both errors are somewhat related, since both point towards mk_vec_matrix.

Please let me know if you need any further input or if there is something I should try to narrow this down further.

[LongHairedHacker]

I just did some crude print()-debugging and I think the second error might be another issue entirely.

My instance has two pods now:
app/pods/sebastian/en/sebastians-site.de.u.sebastian.npz and app/pods/sebastian/de/sebastians-site.de.u.sebastian.npz.

The shape of npz = load_npz(npzs[i]).toarray() for the en one is (3, 16000) whereas for the de one I get (2, 16000).
us = db.session.query(Urls).filter_by(pod=podnames[i]).all() returns three URLs: [https://www.sebastians-site.de/software/nix-i2c/, https://www.sebastians-site.de/software/we-have-ai-at-home/, https://www.sebastians-site.de/news/].
idvs = [u.vector for u in us if u.vector is not None] results in [1, 2, 1]

So that's where the IndexError comes from.

Either something went wrong during indexing and I missed the error message (entirely possible, I'll check that later).
In that case there is simply one row missing in the de npz file.
Or it's okay for that row to be missing, however in that case it is unclear to me how URL.vector and the entries in the npz files are kept in sync.

Looking through the other open issues, this might be related to #89 (maybe).

[minimalparts]

Hi! Thanks for reporting this!

So indeed, I was able to recreate your initial error and it was due to !here creating havoc when there is nothing in the local pods. Silly us, we will fix it! (And thanks for pointing it out.)

The second one is indeed about the matrices having got into an inconsistent state. We have been struggling on and off with this. I wonder what caused it in your case. The easiest way to fix this, since you only have a few indexed pages, would be to delete your pods/ folder and app.db in app/ and start from scratch (you'll need to recreate your user). But that doesn't tell us where how the issue occurred in the first place.

Do you remember in which order you ran the language install and indexing? Did you add the German models after indexing some English pages or right at the beginning? Did the indexer complain while indexing any of your pages?

Thanks very much for helping us investigate!!

[minimalparts]

Ah, sorry, I have just seen your lines about set up. So nothing to do with the order of language installs...

[LongHairedHacker]

Thanks for getting back to me this quickly.
I'll reset to a clean state and try to reproduce it later today.
This time I'll make sure, to note down the order, pay attention to the output and make a test query after each url was added.

Just for my understanding (and to help me look at the right places in the code):
If I create a new category that will create one npz and one pos file for each language that I have installed.
The npz file contains the word embedding vectors for that combination of language and category.
Since that language distinction is not present in Url.pod the index stored in Url.vector is the same across all embedding files for the different languages.
So every Url is to added to all embedding files regardless of the language detected for it.
This mean the npz should always contain an matrix of the same shape across all languages right?

[LongHairedHacker]

Okay I did clean setup, following the same steps I've outline above.

First Url

https://www.sebastians-site.de/software/nix-i2c/

>> INDEXER: run_indexer_url: Running indexer over suggested URL.
Computing vector for https://www.sebastians-site.de/software/nix-i2c/ ( sebastians-site.de )
CONTENT TYPE text/html
REMOVING BOILERPLATES FOR LANG en ( English ).

$ ls -alh app/pods/sebastian/*/*.npz
-rw-r--r-- 1 sebastian users 9.3K Oct  3 17:36 app/pods/sebastian/en/sebastians-site.de.u.sebastian.npz

The de folder is missing, considering how I think the query code works is odd.

Searching !here test runs in the ValueError when searching with QUERY LANG: de:

>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in en
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG en
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁testing', '▁preliminary', '▁experiments', '▁evaluation', '▁screening', '▁tests', '▁tested', '▁test', '▁trials', '▁detection']]
>> TIMER: Function 'mk_vec_matrix' executed in 0.0025s
>> TIMER: Function 'compute_scores' executed in 0.0033s


>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in de
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG de
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁experiment', '▁versuch', 'flüge', '▁durchgeführt', 'training', 'hubschrauber', '▁start', '▁gestartet', '▁pilot', '▁test']]
Traceback (most recent call last):
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 46, in index
    clean_query, results = get_local_search_results(query)
                           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 132, in get_local_search_results
    r, s = score_pages.run_search(clean_query, lang, extended=app.config['EXTEND_QUERY'])
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 204, in run_search
    document_scores = compute_scores(query, q_vectors, lang)
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 106, in compute_scores
    m, bins, podnames, urls = load_vec_matrix(lang)
                              ~~~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 98, in load_vec_matrix
    m, bins, podnames, urls = mk_vec_matrix(lang)
                              ~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 86, in mk_vec_matrix
    m = vstack(m)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/scipy/sparse/_construct.py", line 849, in vstack
    return _block([[b] for b in blocks], format, dtype, return_spmatrix=True)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/scipy/sparse/_construct.py", line 966, in _block
    raise ValueError('blocks must be 2-D')
ValueError: blocks must be 2-D

Makes sense. As no matrix has been saved for de yet.

Second Url

https://www.sebastians-site.de/software/we-have-ai-at-home/

>> Indexer : from_url
>> INDEXER: run_indexer_url: Running indexer over suggested URL.
Computing vector for https://www.sebastians-site.de/software/we-have-ai-at-home/ ( sebastians-site.de )
CONTENT TYPE text/html
REMOVING BOILERPLATES FOR LANG en ( English ).

Still no de file:

-rw-r--r-- 1 sebastian users 23K Oct  3 17:47 app/pods/sebastian/en/sebastians-site.de.u.sebastian.npz

So same issue again:

>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in en
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG en
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁testing', '▁preliminary', '▁experiments', '▁evaluation', '▁screening', '▁tests', '▁tested', '▁test', '▁trials', '▁detection']]
>> TIMER: Function 'mk_vec_matrix' executed in 0.0019s
>> TIMER: Function 'compute_scores' executed in 0.0024s


>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in de
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG de
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁experiment', '▁versuch', 'flüge', '▁durchgeführt', 'training', 'hubschrauber', '▁start', '▁gestartet', '▁pilot', '▁test']]
Traceback (most recent call last):
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 46, in index
    clean_query, results = get_local_search_results(query)
                           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 132, in get_local_search_results
    r, s = score_pages.run_search(clean_query, lang, extended=app.config['EXTEND_QUERY'])
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 204, in run_search
    document_scores = compute_scores(query, q_vectors, lang)
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 106, in compute_scores
    m, bins, podnames, urls = load_vec_matrix(lang)
                              ~~~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 98, in load_vec_matrix
    m, bins, podnames, urls = mk_vec_matrix(lang)
                              ~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 86, in mk_vec_matrix
    m = vstack(m)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/scipy/sparse/_construct.py", line 849, in vstack
    return _block([[b] for b in blocks], format, dtype, return_spmatrix=True)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/scipy/sparse/_construct.py", line 966, in _block
    raise ValueError('blocks must be 2-D')
ValueError: blocks must be 2-D

Third Url

https://www.sebastians-site.de/news/

>> Indexer : from_url
>> INDEXER: run_indexer_url: Running indexer over suggested URL.
Computing vector for https://www.sebastians-site.de/news/ ( sebastians-site.de )
CONTENT TYPE text/html
REMOVING BOILERPLATES FOR LANG de ( German ).

This time the file exists:

$ ls -alh app/pods/sebastian/*/*.npz
-rw-r--r-- 1 sebastian users 17K Oct  3 17:51 app/pods/sebastian/de/sebastians-site.de.u.sebastian.npz
-rw-r--r-- 1 sebastian users 23K Oct  3 17:47 app/pods/sebastian/en/sebastians-site.de.u.sebastian.npz

If we search !here test now:

>> SEARCH:CONTROLLERS:get_local_search_results: searching in en
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG en
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁testing', '▁preliminary', '▁experiments', '▁evaluation', '▁screening', '▁tests', '▁tested', '▁test', '▁trials', '▁detection']]
>> TIMER: Function 'mk_vec_matrix' executed in 0.0020s
>> TIMER: Function 'compute_scores' executed in 0.0026s


>>>>>>>>>>>>>>>>>>>>>>
>> SEARCH:CONTROLLERS:get_local_search_results: searching in de
>>>>>>>>>>>>>>>>>>>>>>

 Getting results on this instance
QUERY LANG de
QUERY SPLIT: ['test']
WORDS TOKENIZED: [['▁test']]
WORDS TOKENIZED EXPANDED: [['▁experiment', '▁versuch', 'flüge', '▁durchgeführt', 'training', 'hubschrauber', '▁start', '▁gestartet', '▁pilot', '▁test']]
Traceback (most recent call last):
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sebastian/build/PeARS-federated/.venv/lib/python3.13/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 46, in index
    clean_query, results = get_local_search_results(query)
                           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/controllers.py", line 132, in get_local_search_results
    r, s = score_pages.run_search(clean_query, lang, extended=app.config['EXTEND_QUERY'])
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 204, in run_search
    document_scores = compute_scores(query, q_vectors, lang)
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 106, in compute_scores
    m, bins, podnames, urls = load_vec_matrix(lang)
                              ~~~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 98, in load_vec_matrix
    m, bins, podnames, urls = mk_vec_matrix(lang)
                              ~~~~~~~~~~~~~^^^^^^
  File "/home/sebastian/build/PeARS-federated/app/utils.py", line 254, in wrap_func
    result = func(*args, **kwargs)
  File "/home/sebastian/build/PeARS-federated/app/search/score_pages.py", line 82, in mk_vec_matrix
    npz = npz[idvs, :]
          ~~~^^^^^^^^^
IndexError: index 2 is out of bounds for axis 0 with size 2

So app/pods/sebastian/de/sebastians-site.de.u.sebastian.npz gets created with one row less than the en version I guess?
If so this might even be reproducible with only two Urls: One en and one de.

[minimalparts]

Hi again,

ok, bug found, it is pretty weird. The error occurs when the same category /pod name is used for different languages. You used 'sebastians-site.de' for both your German and English pages and this has weird consequences. I tried to index your three pages on a fresh install using different category names and everything worked fine.

I now remember we had caught this at some point but the fix doesn't seem to have made it into main. Sorry about that and thank you for bringing it back to light! Will fix.

[LongHairedHacker]

Okay ... now I see the issue:

mk_page_vector.compute_vector uses app.utils_db.create_pod_npz_pos to create a new npz (and pos) file if none of those exist already.
The npz file is initialized with pod = np.zeros((1,VEC_SIZE)).
So for every npz file the row 0 will always be a bunch of zeros.
That's why my de file has two row in it. The first is just zeros and the second is the actual vector for my German url.

In the en file I have one row of zeros and than two vectors for my two english urls.

So there are now two Urls in my database with pod == "sebastians-site.de" with url.vector == 1
This means a search with language en there will be one wrong result if the vector at index 1 matches (the German url that also happens to have url.vector == 1.
For a German search it will stumble upon the Url with pod == "sebastians-site.de" with url.vector == 2 and fail while trying to access the non-existing third row in the de npz file.

I guess this relates to #116.

To fix this I'd suggest making the pod property of the Url model a foreign key into the Pods table.
As far as the Pods table is concerned the en and the de pod are two different entities already, and they even have a language property.
This would resolve the issue at hand and would also prevent the Url objects from breaking if a pod is renamed.

[minimalparts]

Just for my understanding (and to help me look at the right places in the code): If I create a new category that will create one npz and one pos file for each language that I have installed. The npz file contains the word embedding vectors for that combination of language and category. Since that language distinction is not present in Url.pod the index stored in Url.vector is the same across all embedding files for the different languages. So every Url is to added to all embedding files regardless of the language detected for it. This mean the npz should always contain an matrix of the same shape across all languages right?

The way it is set up is that there is a unique database with metadata for all urls. The database has a 'vector' field with an ID number that points at the row in the npz file that the document vector is contained in. But documents are only inserted in the npz file that correspond to their language. So row 3 in the English npz will be different from from row 3 in the German one. Basically, the sum of the non-zero rows in all npz files should be equal to the number of entries in the database.

[minimalparts]

To fix this I'd suggest making the pod property of the Url model a foreign key into the Pods table. As far as the Pods table is concerned the en and the de pod are two different entities already, and they even have a language property. This would resolve the issue at hand and would also prevent the Url objects from breaking if a pod is renamed.

This sounds very sensible! I am copying in @gossminn because I think they had already started on the issue and I don't want to duplicate work :) Let's see what they say.

Thanks again!

[gossminn]

Hi @LongHairedHacker! Thanks again for your interest and detailed bug reports 😊 And sorry for the late reply! As @minimalparts mentioned above, we are aware of the issue with pods in different languages sharing the same name. The solution that you describe could definitely work, but I recently already implemented another fix (in a fork that we made for a specialized version of PeARS for a client) which encodes the language code into the pod name, e.g. if you index something in German under "news", your pod name will be news.l.de.u.sebastian. We haven't integrated this into main because so far we've used the fix only on new instances, and the migration code for renaming pods on existing instances isn't ready yet. I'll make the fix available on a branch in this repo tomorrow, and once the migration code is done (hopefully later this week) we'll merge it to main.

[gossminn]

The solution with language codes integrated in the pod names is now available from the branch linked to #116 - as noted above we'll merge it into main as soon as possible. If you have time to check it out, let us know if you run into any new issues! I just tried it on my local setup:

• starting with a clean slate (rm -r app/pods/* and rm app.db)
• indexed two sample pages, in different languages (in my local setup I have Dutch nl and German de installed), but with the same pod names
• then in the Categories page, you should see something like:

[LongHairedHacker]

Thanks for pushing that branch. As far as I can tell it solves my problem with multiple languages.