Dictionaries not working

[Mohyoo]

Idk if it's a bug or I still don't know how to use your software.

I put stardict & mdict dictionaries inside the source folder; but they don't appear in the web UI at all.
So I had to manually add them in the Library tab in the web UI; it only accepted the (.mdx) file; but when trying to search for a word; it says: "word not found"; even though it appears in the search results; and when clicking in a suggested word; boom it crashes.

the traceback error (sorry i deleted SilverDict before copying the error) mentioned sth about the database file in the (.cache) folder; i deleted the cache folder and tried to import dictionaries again; but same thing.

Sorry for long explanation; I'm not used to report errors; but I did now cause I liked your app.

[Crissium]

Well in the first place if it is so hard to figure out how to use it, it probably indicates bad user interface design.

it only accepted the (.mdx) file;

Perhaps there was an error when adding a StarDict file?

the traceback error (sorry i deleted SilverDict before copying the error) mentioned sth about the database file in the (.cache) folder; i deleted the cache folder and tried to import dictionaries again;

The database file is not inside .cache, so I think there might be a bug related to how I handle static files inside dictionaries.

Sorry for long explanation; I'm not used to report errors; but I did now cause I liked your app.

I'm sorry that it's not useful in your case, and I'll leave this issue open in case someone runs into something similar.

[Mohyoo]

No need to apologize, you created a great educational software, it just needs some fine tunes.

EDIT: I was going step by step to reproduce the same errors that happened before; but suddenly SilverDict is working properly; I'll check my system and see if I can find why it didn't work before; so yeah your software works properly!

I launched SilverDict again, and here are more details to help you.

-First, I'm running Windows 8.1 x64 Pro.
-I put one MDict & one StarDict dictionaries (The dictionaries 100% work in their native softwares) inside the SilverDict\source folder; and I launch the app without any errors in the console, the output is:

INFO:app.langs:Using speller Sibel
INFO:app.dictionaries:Dictionaries loaded.
INFO:waitress:Serving on http://0.0.0.0:2628
WARNING:waitress.queue:Task queue depth is 1
WARNING:waitress.queue:Task queue depth is 2
WARNING:waitress.queue:Task queue depth is 1
WARNING:waitress.queue:Task queue depth is 1
WARNING:waitress.queue:Task queue depth is 1
WARNING:waitress.queue:Task queue depth is 2

-As you see in the picutres above, it didn't find any dictionary; so instead of organizing the dictionaries into sub folders in the SilverDict\source directory; i tried to put the (.mdx) & (.ifo) files directly inside the source folder (ofc alongside their other files like mdd, dict.dz, etc...); but it still doesnt find them.
-Now, I tried to open the web UI -> Library tab -> Dictionaries tab -> Click the add (+) button at top right. they get imported normally as u see here (although earlier, the StarDict dictionary gave an error):

-So, idk how they are working now.

By saying the .cache folder before; I meant: "C:\Users\Username\.cache\SilverDict"; and the database file (.db) was located here earlier; now the folder structure changed; I have no idea what happened.

[Crissium]

As you see in the picutres above, it didn't find any dictionary;

The source folder is not automatically scanned on a file being added. It happens only when you click on the 'refresh' button in the top right corner of the 'Sources' tab of Library.

[Mohyoo]

Well; you are right; sorry for the inconvenience.

-But I still get this error when typing in the search field or when clicking in a specific word in the results:

-It often gets followed by this console output:

Traceback (most recent call last):
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask_cors\extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\api\lookup.py", line 45, in query
suggestions = dicts.suggestions(group_name, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\dictionaries.py", line 385, in suggestions

keys_expanded.extend(db_manager.expand_key(key_simplified,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\Users\Halima\Desktop\SilverDict\program\server\app\db_manager.py", line 280, in expand_key
rows = cursor.execute(statement, ngrams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such table: ngrams

-It doesnt crash, i can still lookup for other words; it just happens for specific words.
-And one last thing, the Oxford dictionary in StarDict format gives results when searching; while the Merriam Webster in MDict format doesnt appear in the results at all, although it gets load normally.

-Thanks for your patience!

[Crissium]

OK I think I know what's happening now… I accidentally set the default value of suggestions_mode to both-sides accidentally, and such suggestion expansion requires the n-gram index to be built, which I don't do until requested. A temporary solution is either to build the n-gram index manually (in Settings) or to set suggestions_mode to right-side in ~/preferences.yaml.

I'll release a new version as soon as possible.

[Crissium]

Could you help me verify if the either of the two fixes work for you?

A temporary solution is either to build the n-gram index manually (in Settings) or to set suggestions_mode to right-side in ~/preferences.yaml.

[Mohyoo]

So, the error below disappeared, it was happening when clicking in some specific results, I've tested that with as many results as I can, thus it should be working fine now.

However, when putting a DSL dictionary in the source folder, or when trying to manually import it in Library -> Dictionaries, this happens (in Thorium browser (Chromium-based)):

And in Mozilla, the error appears like this:

Followed by this console output:

INFO:app.settings:Found dictionary oxford (C:\Users\Halima\Desktop\SilverDict\so
urce\dsl01_oxford\oxford.dsl) during re-scanning.
ERROR:app:Exception on /api/management/scan [GET]
Traceback (most recent call last):
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py",
line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py",
line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask_cors\exte
nsion.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py",
line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\env\Lib\site-packages\flask\app.py",
line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) #
type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\api\management.py"
, line 100, in scan_sources
dicts.add_dictionary(dictionary_info)
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\dictionaries.py",
line 165, in add_dictionary
self._load_dictionary(dictionary_info)
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\dictionaries.py",
line 91, in _load_dictionary
DSLReader(dictionary_info['dictionary_name'],
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\dicts\dsl_reader.p
y", line 123, in init
self._clean_up(filename)
File "C:\Users\Halima\Desktop\SilverDict\program\server\app\dicts\dsl_reader.p
y", line 65, in _clean_up
text = data.decode(detect_encoding(data)) # TODO: json's detect_encoding() i
s not always reliable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 34595: inva
lid start byte

EDIT: (.dsl.dz) files work, so the problem is with (.dsl) files only

And one last thing, some dictionaries (StarDict, MDict & .dsl.dz) results don't work, I mean, the words inside the dictionaries appear in the results, but clicking them gives this:

Clicking on one of the suggestion gives this page:

Clicking again does nothing, and there is no console output.

I hope this can help you; if you want me to do other things (such as testing some other dictionaries), don't hesitate.

[Crissium]

Could you send me the problematic dictionaries so that I can debug it, including the one that cannot be added and those which can be added but cannot be searched? I think you can simply drop them here.

By the way, other users have complained about the encoding problem (the DSL dictionary that failed to be indexed due to an unknown encoding), too, and it seems it has to be addressed now.

[Mohyoo]

So, I tested

• (1) dsl.dz file ---- (Worked)
• (2) dsl files ------ (Didn't work)
• (5) MDict files --- (Only 3 worked)
• (4) StarDict files - (Only 1 worked)

The ones that didn't get imported at all (I have included an error screenshot for each one):
Not imported.zip

The ones that got imported, but their search results didn't work when clicking them:
Imported but not working.zip

The ones that worked just fine:
Working.zip

Just a request, when launching the application or rescanning the source folder, can you make it automatically remove invalid dictionaries paths (The ones that got moved or removed via file manager)?

EDIT, by saying the decoding error, I think you meant this?

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 34595: invalid start byte