fix issues

Author: gingerwizard

scripts/search/README.md

@@ -36,22 +36,28 @@ options:
 | 20/01/2024 | 0.4700           | [View Results](https://pastila.nl/?008231f5/bc107912f8a5074d70201e27b1a66c6c#cB/yJOsZPOWi9h8xAkuTUQ==) | Baseline                                         |
 | 21/01/2024 | 0.5021           | [View Results](https://pastila.nl/?00bb2c2f/936a9a3af62a9bdda186af5f37f55782#m7Hg0i9F1YCesMW6ot25yA==) | Index `_` character and move language to English |
 | 24/01/2024 | 0.7072           | [View Results](https://pastila.nl/?065e3e67/e4ad889d0c166226118e6160b4ee53ff#x1NPd2R7hU90CZvvrE4nhg==) | Process markdown, and tune settings.             |
+| 24/01/2024 | 0.7412           | [View Results](https://pastila.nl/?0020013d/e69b33aaae82e49bc71c5ee2cea9ad46#pqq3VtRd4eP4JM5/izcBcA==) | Include manual promotions for ambigious terms.   |
+
+
 
 ## Issues
 
 1. Some pages are not optimized for retrieval e.g. 
    a. https://clickhouse.com/docs/en/sql-reference/aggregate-functions/combinators#-if will never return for `countIf`, `sumif`, `multiif`
-2. Some pages are hidden e.g. https://clickhouse.com/docs/en/install#from-docker-image - this needs to be separate page.
-3. Some pages e.g. https://clickhouse.com/docs/en/sql-reference/statements/alter need headings e.g. `Alter table`
-4. https://clickhouse.com/docs/en/optimize/sparse-primary-indexes needs to be optimized for primary key
-5. `between` we need to likely manually promote.
-6. `case when` - https://clickhouse.com/docs/en/sql-reference/functions/conditional-functions needs to be improved. Maybe keywords or a header
-7. `has` - https://clickhouse.com/docs/en/sql-reference/functions/array-functions#hasarr-elem tricky
-8. `clickhouse` - manual promotion
-9. `codec` - we need better content
-10. `shard` - need a better page
-11. `populate` - we need to have a subheading on the mv page
-12. `contains` - https://clickhouse.com/docs/en/sql-reference/functions/string-search-functions needs words
-13. `client` - maybe promote manually
-14. `config.xml` - manually promote
-15. `replica` - need more terms on https://clickhouse.com/docs/en/architecture/horizontal-scaling but we need a better page
+1. Some pages are hidden e.g. https://clickhouse.com/docs/en/install#from-docker-image - this needs to be separate page.
+1. Some pages e.g. https://clickhouse.com/docs/en/sql-reference/statements/alter need headings e.g. `Alter table`
+1. https://clickhouse.com/docs/en/optimize/sparse-primary-indexes needs to be optimized for primary key
+1. case `when` - https://clickhouse.com/docs/en/sql-reference/functions/conditional-functions needs to be improved. Maybe keywords or a header
+1. `has` - https://clickhouse.com/docs/en/sql-reference/functions/array-functions#hasarr-elem tricky
+1. `codec` - we need better content
+1. `shard` - need a better page
+1. `populate` - we need to have a subheading on the mv page
+1. `contains` - https://clickhouse.com/docs/en/sql-reference/functions/string-search-functions needs words
+1. `replica` - need more terms on https://clickhouse.com/docs/en/architecture/horizontal-scaling but we need a better page
+
+
+Algolia configs to try:
+
+- minProximity - 1
+- minWordSizefor2Typos - 7
+- minWordSizefor1Typo- 3

scripts/search/index_pages.py

@@ -7,14 +7,24 @@
 from slugify import slugify
 from algoliasearch.search.client import SearchClientSync
 import networkx as nx
+from urllib.parse import urlparse, urlunparse
 
 DOCS_SITE = 'https://clickhouse.com/docs'
+with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'settings.json'), 'r') as f:
+    settings = json.load(f)
 HEADER_PATTERN = re.compile(r"^(.*?)(?:\s*\{#(.*?)\})$")
 object_ids = set()
 files_processed = set()
 link_data = []
 
 
+def split_url_and_anchor(url):
+    parsed_url = urlparse(url)
+    url_without_anchor = urlunparse(parsed_url._replace(fragment=""))
+    anchor = parsed_url.fragment
+    return url_without_anchor, anchor
+
+
 def read_metadata(text):
     parts = text.split("\n")
     metadata = {}
@@ -124,6 +134,7 @@ def clean_content(content):
     content = re.sub(r'```.*?```', '', content, flags=re.DOTALL)  # replace code blocks
     return content
 
+
 def inject_snippets(directory, content):
     snippet_pattern = re.compile(
         r"import\s+(\w+)\s+from\s+['\"]@site/((.*?))['\"];",
@@ -207,20 +218,20 @@ def parse_markdown_content(metadata, content):
     current_h1 = metadata.get('title', '')
     current_h2 = None
     current_h3 = None
-    current_h4 = None
     current_subdoc = {
         'file_path': metadata.get('file_path', ''),
         'slug': heading_slug,
         'url': f'{DOCS_SITE}{heading_slug}',
         'h1': current_h1,
+        'h1_camel': current_h1,
         'title': metadata.get('title', ''),
         'content': metadata.get('description', ''),
         'keywords': metadata.get('keywords', ''),
         'objectID': get_object_id(heading_slug),
         'type': 'lvl1',
         'hierarchy': {
-            'lvl0': metadata.get('title', ''),
-            'lvl1': metadata.get('title', '')
+            'lvl0': current_h1,
+            'lvl1': current_h1
         }
     }
     for line in lines:
@@ -234,6 +245,7 @@ def parse_markdown_content(metadata, content):
             current_subdoc['slug'] = heading_slug
             current_subdoc['url'] = f'{DOCS_SITE}{heading_slug}'
             current_subdoc['h1'] = current_h1
+            current_subdoc['h1_camel'] = current_h1
             current_subdoc['title'] = current_h1
             current_subdoc['type'] = 'lvl1'
             current_subdoc['object_id'] = custom_slugify(heading_slug)
@@ -254,6 +266,7 @@ def parse_markdown_content(metadata, content):
                 'url': f'{DOCS_SITE}{heading_slug}',
                 'title': current_h2,
                 'h2': current_h2,
+                'h2_camel': current_h2,
                 'content': '',
                 'keywords': metadata.get('keywords', ''),
                 'objectID': get_object_id(f'{heading_slug}-{current_h2}'),
@@ -281,6 +294,7 @@ def parse_markdown_content(metadata, content):
                 'url': f'{DOCS_SITE}{heading_slug}',
                 'title': current_h3,
                 'h3': current_h3,
+                'h3_camel': current_h3,
                 'content': '',
                 'keywords': metadata.get('keywords', ''),
                 'objectID': get_object_id(f'{heading_slug}-{current_h3}'),
@@ -305,6 +319,7 @@ def parse_markdown_content(metadata, content):
                 'url': f'{DOCS_SITE}{heading_slug}',
                 'title': current_h4,
                 'h4': current_h4,
+                'h4_camel': current_h4,
                 'content': '',
                 'keywords': metadata.get('keywords', ''),
                 'objectID': get_object_id(f'{heading_slug}-{current_h4}'),
@@ -336,6 +351,9 @@ def process_markdown_directory(directory, base_directory):
                     files_processed.add(md_file_path)
                     metadata, content = parse_metadata_and_content(directory, base_directory, md_file_path)
                     for sub_doc in parse_markdown_content(metadata, content):
+                        url_without_anchor, anchor = split_url_and_anchor(sub_doc['url'])
+                        sub_doc['url_without_anchor'] = url_without_anchor
+                        sub_doc['anchor'] = anchor
                         update_page_links(directory, base_directory, metadata.get('file_path', ''), sub_doc['url'],
                                           sub_doc['content'])
                         yield sub_doc
@@ -371,9 +389,22 @@ def compute_page_rank(link_data, damping_factor=0.85, max_iter=100, tol=1e-6):
     return page_rank
 
 
+def create_new_index(client, index_name):
+    try:
+        client.delete_index(index_name)
+        print(f'Temporary index \'{index_name}\' deleted successfully.')
+    except:
+        print(f'Temporary index \'{index_name}\' does not exist or could not be deleted')
+    client.set_settings(index_name, settings['settings'])
+    client.save_rules(index_name, settings['rules'])
+    print(f"Settings applied to temporary index '{index_name}'.")
+
+
 def main(base_directory, sub_directories, algolia_app_id, algolia_api_key, algolia_index_name,
          batch_size=1000, dry_run=False):
+    temp_index_name = f"{algolia_index_name}_temp"
     client = SearchClientSync(algolia_app_id, algolia_api_key)
+    create_new_index(client, temp_index_name)
     docs = []
     for sub_directory in sub_directories:
         directory = os.path.join(base_directory, sub_directory)
@@ -388,13 +419,22 @@ def main(base_directory, sub_directories, algolia_app_id, algolia_api_key, algol
     for i in range(0, len(docs), batch_size):
         batch = docs[i:i + batch_size]  # Get the current batch
         if not dry_run:
-            send_to_algolia(client, algolia_index_name, batch)
+            send_to_algolia(client, temp_index_name, batch)
         else:
             for d in batch:
                 print(f"{d['url']} - {d['page_rank']}")
         print(f'{'processed' if dry_run else 'indexed'} {len(batch)} records')
         t += len(batch)
-    print(f'total for {directory}: {'processed' if dry_run else 'indexed'} {t} records')
+    print(f'total {'processed' if dry_run else 'indexed'} {t} records')
+    print('switching temporary index...', end='')
+    client.operation_index(
+        index_name=temp_index_name,
+        operation_index_params={
+            "operation": "move",
+            "destination": algolia_index_name
+        },
+    )
+    print('done')
 
 
 if __name__ == '__main__':

scripts/search/results.csv

@@ -60,7 +60,7 @@ mater,https://clickhouse.com/docs/en/materialized-view,,
 bitmap,https://clickhouse.com/docs/en/sql-reference/functions/bitmap-functions,,
 docker,https://clickhouse.com/docs/en/install#from-docker-image,,
 match,https://clickhouse.com/docs/en/sql-reference/functions/string-search-functions#match,https://clickhouse.com/docs/en/sql-reference/functions/string-search-functions,
-alter table,https://clickhouse.com/docs/en/sql-reference/statements/alter,https://clickhouse.com/docs/en/sql-reference/statements/alter/delete",
+alter table,https://clickhouse.com/docs/en/sql-reference/statements/alter,https://clickhouse.com/docs/en/sql-reference/statements/alter/delete,
 partition by,https://clickhouse.com/docs/en/partitions,https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/custom-partitioning-key,https://clickhouse.com/docs/en/engines/table-engines/special/url#partition-by
 group,https://clickhouse.com/docs/en/sql-reference/statements/select/group-by,,
 null,https://clickhouse.com/docs/en/sql-reference/data-types/nullable,,
@@ -133,7 +133,7 @@ timezone,https://clickhouse.com/docs/en/sql-reference/functions/date-time-functi
 union,https://clickhouse.com/docs/en/sql-reference/statements/select/union,,
 dict,https://clickhouse.com/docs/en/dictionary,https://clickhouse.com/docs/en/sql-reference/dictionaries,
 array join,https://clickhouse.com/docs/en/sql-reference/statements/select/array-join,https://clickhouse.com/docs/en/sql-reference/functions/array-join,
-clickhouse,https://clickhouse.com/,https://clickhouse.com/docs/en/intro,
+clickhouse,https://clickhouse.com/docs/en/intro,
 nested,https://clickhouse.com/docs/en/sql-reference/data-types/nested-data-structures/nested,https://clickhouse.com/docs/en/sql-reference/data-types/nested-data-structures/nested#nestedname1-type1-name2-type2-,
 sample,https://clickhouse.com/docs/en/sql-reference/statements/select/sample,,
 distinct,https://clickhouse.com/docs/en/sql-reference/statements/select/distinct,,
@@ -151,7 +151,7 @@ materi,https://clickhouse.com/docs/en/materialized-view,,
 max_threads,https://clickhouse.com/docs/en/operations/settings/settings#max_threads,,
 limit,https://clickhouse.com/docs/en/sql-reference/statements/select/limit,,
 toint,https://clickhouse.com/docs/en/sql-reference/functions/type-conversion-functions,,
-shard,https://clickhouse.com/docs/concepts/concepts/glossary#shard,,
+shard,https://clickhouse.com/docs/concepts/concepts/glossary#shard,https://clickhouse.com/docs/en/concepts/glossary,
 timeout,https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings,https://clickhouse.com/docs/en/operations/settings/settings#timeout_overflow_mode,
 date_diff,https://clickhouse.com/docs/en/sql-reference/functions/date-time-functions#date_diff,,
 default,https://clickhouse.com/docs/en/operations/settings/settings-users,https://clickhouse.com/docs/knowledgebase/remove-default-user,