fix(ci): resolve HuggingFace tokenizer failures in CI environments

Enhanced the AIRedisConfiguration to properly handle HuggingFace tokenizer
initialization in CI/offline environments by:

1. Bundling the tokenizer.json file for msmarco-distilbert-dot-v5 in resources
2. Implementing fallback logic to try local resources first, then remote download
3. Improving error handling and logging for tokenizer initialization
4. Removing test-specific CI workarounds in favor of comprehensive solution

This addresses the root cause of CI test failures where network restrictions
prevent HuggingFace tokenizer downloads, ensuring all tests can run reliably
in both local and CI environments.

Author: bsbodden

redis-om-spring-ai/src/main/java/com/redis/om/spring/AIRedisConfiguration.java

@@ -386,10 +386,11 @@ public Pipeline defaultImagePipeline(AIRedisOMProperties properties) {
   /**
    * Creates a HuggingFace sentence tokenizer for text processing.
    * This tokenizer is used to prepare text for sentence embedding models.
-   * Checks connectivity to HuggingFace before attempting to download the tokenizer.
+   * First attempts to load from bundled resources, then falls back to downloading
+   * from HuggingFace if network is available.
    *
    * @param properties AI Redis OM configuration properties containing tokenizer settings
-   * @return a configured HuggingFaceTokenizer, or null if unable to connect or load
+   * @return a configured HuggingFaceTokenizer, or null if unable to load
    */
   @Bean(
       name = "djlSentenceTokenizer"
@@ -400,12 +401,26 @@ public HuggingFaceTokenizer sentenceTokenizer(AIRedisOMProperties properties) {
         "modelMaxLength", properties.getDjl().getSentenceTokenizerModelMaxLength() //
     );
 
+    // First try to load from bundled resources (for CI/offline environments)
+    try {
+      String resourcePath = "/tokenizers/" + properties.getDjl().getSentenceTokenizerModel() + "/tokenizer.json";
+      var resourceStream = getClass().getResourceAsStream(resourcePath);
+      if (resourceStream != null) {
+        logger.info("Loading HuggingFace tokenizer from bundled resources: " + resourcePath);
+        return HuggingFaceTokenizer.newInstance(resourceStream, options);
+      }
+    } catch (Exception e) {
+      logger.debug("Failed to load tokenizer from bundled resources, will try downloading", e);
+    }
+
+    // Fall back to downloading from HuggingFace (for normal environments)
     try {
       //noinspection ResultOfMethodCallIgnored
       InetAddress.getByName("www.huggingface.co").isReachable(5000);
+      logger.info("Loading HuggingFace tokenizer from remote: " + properties.getDjl().getSentenceTokenizerModel());
       return HuggingFaceTokenizer.newInstance(properties.getDjl().getSentenceTokenizerModel(), options);
     } catch (IOException ioe) {
-      logger.warn("Error retrieving default DJL sentence tokenizer");
+      logger.warn("Unable to download HuggingFace tokenizer (network unavailable or restricted environment)");
       return null;
     }
   }