Enhance TUI indexing dashboard with OIDC support and metrics

[Szowesgad]

This pull request restructures the project into a Cargo workspace, introduces a new memex-contracts crate for shared data types, and expands configuration and authentication support. It also updates copyright and contact information throughout the project.

Project structure and workspace modernization:

• Converts the project to a Cargo workspace, splitting the codebase into crates/rust-memex (main logic) and crates/memex-contracts (shared contracts/types). Updates Cargo.toml files accordingly and moves dependencies to workspace-level management. [1] [2] [3]

New shared contracts crate:

• Adds the memex-contracts crate with shared types for audit, progress, stats, and timeline data, making it easier to share data structures between the backend and future frontends. [1] [2] [3] [4] [5] [6]

Configuration and authentication enhancements:

• Expands the configuration file format to support dashboard-only OIDC authentication, additional auth modes, and query token support. Updates config loading logic to handle these new options. (crates/rust-memex/src/bin/cli/config.rsR88-R118, Fd9f55daL653R653)

Documentation and copyright updates:

• Updates copyright and contact information to reference vetcoders.io instead of loct.io across documentation, issue templates, and security policy. [1] [2] [3] [4]

Cleanup and refactoring:

• Removes obsolete local plan/report files and unnecessary #[allow(dead_code)] attributes from configuration and definition modules. [1] [2] [3] [4] [5] [6]

Let me know if you want to discuss the new workspace structure, how to use the shared contracts, or the new authentication options!

[gemini-code-assist]

Code Review

This pull request transitions the project to a workspace structure, introduces a new memex-contracts crate for shared data structures, and implements a robust multi-token authentication system with Argon2id hashing and OIDC support. It significantly enhances the indexing pipeline with resume capabilities, adaptive throughput control, and LLM-based outer layer synthesis via Ollama. Additionally, it adds comprehensive diagnostic tools for database maintenance, including quality audits, deduplication based on source hashes, and write repair. Feedback focuses on improving efficiency by batching database operations during backfills and rollbacks, reusing HTTP clients, and optimizing hot paths in keyword extraction. There is also a critical note regarding an inverted hierarchy in the simplified onion slicing logic and a suggestion to replace fragile manual path parsing for ACL checks with a more robust routing-based approach.

[gemini-code-assist]

In create_core_only_slice, the hierarchy is inverted compared to the standard onion model. The Core slice is set as the parent of the Outer slice, whereas in a typical onion structure, the Outer slice (summary) should be the root. This inversion may cause unexpected behavior when navigating the hierarchy via expand or parent operations in the TUI or API.

    vec![
        OnionSlice {
            id: outer_id.clone(),
            layer: SliceLayer::Outer,
            content: content.to_string(),
            parent_id: None,
            children_ids: vec![core_id.clone()],
            keywords: outer_keywords,
        },
        OnionSlice {
            id: core_id,
            layer: SliceLayer::Core,
            content: content.to_string(),
            parent_id: Some(outer_id),
            children_ids: vec![],
            keywords: core_keywords,
        },
    ]

[gemini-code-assist]

Performing a delete_document and add_to_store for every single document during backfill is extremely inefficient. In LanceDB, this will create a new table version for every row updated, leading to significant storage overhead and slow performance. These operations should be batched per page (e.g., collect all new_docs and their IDs, then perform one batch delete and one batch add per page).

[gemini-code-assist]

Manual path parsing to extract namespaces for ACL checks is fragile and highly coupled to the specific URL structure. If routes are added or modified (e.g., adding a version prefix or changing a verb), this function may fail to correctly identify the namespace, potentially leading to security bypasses or unexpected denials. Consider using a more robust approach, such as extracting parameters from the matched route provided by the web framework's router.

[gemini-code-assist]

Creating a new reqwest::Client for every document processed in the pipeline is inefficient. It prevents the reuse of connection pools and increases overhead for every HTTP call to Ollama. The client should be instantiated once (e.g., as a field in RAGPipeline) and reused across all synthesis calls.

[gemini-code-assist]

Creating a HashSet of vowels on every call to looks_like_path_fragment is inefficient, as this function is called for every token during keyword extraction. This check should be optimized using a simple matches! macro or a constant bitmask to avoid repeated allocations.

    let mut max_vowel_run = 0;
    let mut current_run = 0;
    for ch in token.chars() {
        if matches!(ch.to_ascii_lowercase(), 'a' | 'e' | 'i' | 'o' | 'u' | 'y') {

[gemini-code-assist]

The rollback logic currently deletes chunks one by one, which is inefficient. Since a batch delete_documents method was added to StorageManager in this PR, it should be used here to perform the rollback in a single operation per namespace.

Suggested change

	for (namespace, id) in stored_doc_refs.iter().rev() {
	match storage.delete_document(namespace, id).await {
	Ok(count) => deleted += count,
	Err(err) => {
	failures += 1;
	warn!(
	"Failed to roll back partially stored chunk {}/{}: {}",
	namespace, id, err
	);
	}
	}
	}
	let mut by_ns: std::collections::HashMap<String, Vec<&str>> = std::collections::HashMap::new();
	for (ns, id) in stored_doc_refs {
	by_ns.entry(ns.clone()).or_default().push(id.as_str());
	}
	for (ns, ids) in by_ns {
	match storage.delete_documents(&ns, &ids).await {
	Ok(count) => deleted += count,
	Err(err) => {
	failures += ids.len();
	warn!("Failed to roll back partially stored chunks for namespace {}: {}", ns, err);
	}
	}
	}