diff --git a/README.md b/README.md index 3916ed0e..58b819a8 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,44 @@
-
-

AI Resources

-
- -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -![Language](https://img.shields.io/github/languages/top/redis-developer/redis-ai-resources) -![GitHub last commit](https://img.shields.io/github/last-commit/redis-developer/redis-ai-resources) +AI Resources +

AI Resources

+

+ License: MIT + Language + GitHub last commit + Discord + Twitter +

-
-
+

โœจ A curated repository of code recipes, demos, tutorials and resources for basic and advanced Redis use cases in the AI ecosystem. โœจ +

+ +
+

+ Getting Started | + Demos | + Recipes | + Tutorials | + Integrations | + Resources +

-
+

-[**Demos**](#demos) | [**Recipes**](#recipes) | [**Tutorials**](#tutorials) | [**Integrations**](#integrations) | [**Content**](#content) | [**Benchmarks**](#benchmarks) | [**Docs**](#docs) +## Getting Started -
-
+New to Redis for AI applications? Here's how to get started: + +1. **First time with Redis?** Start with our [Redis Intro notebook](python-recipes/redis-intro/00_redis_intro.ipynb) +2. **Want to try vector search?** Check our [Vector Search with RedisVL](python-recipes/vector-search/01_redisvl.ipynb) recipe +3. **Building a RAG application?** Begin with [RAG from Scratch](python-recipes/RAG/01_redisvl.ipynb) +4. **Ready to see it in action?** Play with the [Redis RAG Workbench](https://github.com/redis-developer/redis-rag-workbench) demo + +
## Demos No faster way to get started than by diving in and playing around with a demo. @@ -36,111 +54,127 @@ No faster way to get started than by diving in and playing around with a demo. ## Recipes -Need quickstarts to begin your Redis AI journey? **Start here.** - -### Non-Python Redis AI Recipes - -#### โ˜•๏ธ Java - -A set of Java recipes can be found under [/java-recipes](/java-recipes/README.md). - +Need quickstarts to begin your Redis AI journey? ### Getting started with Redis & Vector Search -| Recipe | Description | -| --- | --- | -| [/redis-intro/00_redis_intro.ipynb](/python-recipes/redis-intro/00_redis_intro.ipynb) | The place to start if brand new to Redis | -| [/vector-search/00_redispy.ipynb](/python-recipes/vector-search/00_redispy.ipynb) | Vector search with Redis python client | -| [/vector-search/01_redisvl.ipynb](/python-recipes/vector-search/01_redisvl.ipynb) | Vector search with Redis Vector Library | -| [/vector-search/02_hybrid_search.ipynb](/python-recipes/vector-search/02_hybrid_search.ipynb) | Hybrid search techniques with Redis (BM25 + Vector) | -| [/vector-search/03_dtype_support.ipynb](/python-recipes/vector-search/03_dtype_support.ipynb) | Shows how to convert a float32 index to float16 or integer dataypes| +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿ **Redis Intro** - The place to start if brand new to Redis | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/redis-intro/00_redis_intro.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/redis-intro/00_redis_intro.ipynb) | +| ๐Ÿ” **Vector Search with RedisPy** - Vector search with Redis python client | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/00_redispy.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/00_redispy.ipynb) | +| ๐Ÿ“š **Vector Search with RedisVL** - Vector search with Redis Vector Library | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/01_redisvl.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/01_redisvl.ipynb) | +| ๐Ÿ”„ **Hybrid Search** - Hybrid search techniques with Redis (BM25 + Vector) | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/02_hybrid_search.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/02_hybrid_search.ipynb) | +| ๐Ÿ”ข **Data Type Support** - Shows how to convert a float32 index to float16 or integer dataypes | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/vector-search/03_dtype_support.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/03_dtype_support.ipynb) | ### Retrieval Augmented Generation (RAG) -**Retrieval Augmented Generation** (aka RAG) is a technique to enhance the ability of an LLM to respond to user queries. The **retrieval** part of RAG is supported by a vector database, which can return semantically relevant results to a userโ€™s query, serving as contextual information to **augment** the **generative** capabilities of an LLM. +**Retrieval Augmented Generation** (aka RAG) is a technique to enhance the ability of an LLM to respond to user queries. The **retrieval** part of RAG is supported by a vector database, which can return semantically relevant results to a user's query, serving as contextual information to **augment** the **generative** capabilities of an LLM. To get started with RAG, either from scratch or using a popular framework like Llamaindex or LangChain, go with these recipes: -| Recipe | Description | -| --- | --- | -| [/RAG/01_redisvl.ipynb](python-recipes/RAG/01_redisvl.ipynb) | RAG from scratch with the Redis Vector Library | -| [/RAG/02_langchain.ipynb](python-recipes/RAG/02_langchain.ipynb) | RAG using Redis and LangChain | -| [/RAG/03_llamaindex.ipynb](python-recipes/RAG/03_llamaindex.ipynb) | RAG using Redis and LlamaIndex | -| [/RAG/04_advanced_redisvl.ipynb](python-recipes/RAG/04_advanced_redisvl.ipynb) | Advanced RAG techniques | -| [/RAG/05_nvidia_ai_rag_redis.ipynb](python-recipes/RAG/05_nvidia_ai_rag_redis.ipynb) | RAG using Redis and Nvidia NIMs | -| [/RAG/06_ragas_evaluation.ipynb](python-recipes/RAG/06_ragas_evaluation.ipynb) | Utilize the RAGAS framework to evaluate RAG performance | -| [/RAG/07_user_role_based_rag.ipynb](python-recipes/RAG/07_user_role_based_rag.ipynb) | Implement a simple RBAC policy with vector search using Redis | +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿงฉ **RAG from Scratch** - RAG from scratch with the Redis Vector Library | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/01_redisvl.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/01_redisvl.ipynb) | +| โ›“๏ธ **LangChain RAG** - RAG using Redis and LangChain | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/02_langchain.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/02_langchain.ipynb) | +| ๐Ÿฆ™ **LlamaIndex RAG** - RAG using Redis and LlamaIndex | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/03_llamaindex.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/03_llamaindex.ipynb) | +| ๐Ÿš€ **Advanced RAG** - Advanced RAG techniques | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/04_advanced_redisvl.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/04_advanced_redisvl.ipynb) | +| ๐Ÿ–ฅ๏ธ **NVIDIA RAG** - RAG using Redis and Nvidia NIMs | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/05_nvidia_ai_rag_redis.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/05_nvidia_ai_rag_redis.ipynb) | +| ๐Ÿ“Š **RAGAS Evaluation** - Utilize the RAGAS framework to evaluate RAG performance | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/06_ragas_evaluation.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/06_ragas_evaluation.ipynb) | +| ๐Ÿ”’ **Role-Based RAG** - Implement a simple RBAC policy with vector search using Redis | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/RAG/07_user_role_based_rag.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/07_user_role_based_rag.ipynb) | ### LLM Memory -LLMs are stateless. To maintain context within a conversation chat sessions must be stored and resent to the LLM. Redis manages the storage and retrieval of message histories to maintain context and conversational relevance. -| Recipe | Description | -| --- | --- | -| [/llm-message-history/00_message_history.ipynb](python-recipes/llm-message-history/00_llm_message_history.ipynb) | LLM message history with semantic similarity | -| [/llm-message-history/01_multiple_sessions.ipynb](python-recipes/llm-message-history/01_multiple_sessions.ipynb) | Handle multiple simultaneous chats with one instance | +LLMs are stateless. To maintain context within a conversation chat sessions must be stored and re-sent to the LLM. Redis manages the storage and retrieval of message histories to maintain context and conversational relevance. + +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿ’ฌ **Message History** - LLM message history with semantic similarity | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/llm-message-history/00_llm_message_history.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/llm-message-history/00_llm_message_history.ipynb) | +| ๐Ÿ‘ฅ **Multiple Sessions** - Handle multiple simultaneous chats with one instance | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/llm-message-history/01_multiple_sessions.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/llm-message-history/01_multiple_sessions.ipynb) | ### Semantic Caching An estimated 31% of LLM queries are potentially redundant ([source](https://arxiv.org/pdf/2403.02694)). Redis enables semantic caching to help cut down on LLM costs quickly. -| Recipe | Description | -| --- | --- | -| [/semantic-cache/00_semantic_caching_gemini.ipynb](python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb) | Build a semantic cache with Redis and Google Gemini | -| [/semantic-cache/01_doc2cache_llama3_1.ipynb](python-recipes/semantic-cache/01_doc2cache_llama3_1.ipynb) | Build a semantic cache using the Doc2Cache framework and Llama3.1 | -| [/semantic-cache/02_semantic_cache_optimization.ipynb](python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb) | Use CacheThresholdOptimizer from redisvl to setup best cache config | +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿง  **Gemini Semantic Cache** - Build a semantic cache with Redis and Google Gemini | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb) | +| ๐Ÿฆ™ **Llama3.1 Doc2Cache** - Build a semantic cache using the Doc2Cache framework and Llama3.1 | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/semantic-cache/01_doc2cache_llama3_1.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/semantic-cache/01_doc2cache_llama3_1.ipynb) | +| โš™๏ธ **Cache Optimization** - Use CacheThresholdOptimizer from redisvl to setup best cache config | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb) | ### Semantic Routing -Routing is a simple and effective way of preventing misuses with your AI application or for creating branching logic between data sources etc. +Routing is a simple and effective way of preventing misuse with your AI application or for creating branching logic between data sources etc. -| Recipe | Description | -| --- | --- | -| [/semantic-router/00_semantic_routing.ipynb](python-recipes/semantic-router/00_semantic_routing.ipynb) | Simple examples of how to build an allow/block list router in addition to a multi-topic router | -| [/semantic-router/01_routing_optimization.ipynb](python-recipes/semantic-router/01_routing_optimization.ipynb) | Use RouterThresholdOptimizer from redisvl to setup best router config | +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿ”€ **Basic Routing** - Simple examples of how to build an allow/block list router in addition to a multi-topic router | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/semantic-router/00_semantic_routing.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/semantic-router/00_semantic_routing.ipynb) | +| โš™๏ธ **Router Optimization** - Use RouterThresholdOptimizer from redisvl to setup best router config | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/semantic-router/01_routing_optimization.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/semantic-router/01_routing_optimization.ipynb) | ### AI Gateways AI gateways manage LLM traffic through a centralized, managed layer that can implement routing, rate limiting, caching, and more. -| Recipe | Description | -| --- | --- | -| [/gateway/00_litellm_proxy_redis.ipynb](python-recipes/gateway/00_litellm_proxy_redis.ipynb) | Getting started with LiteLLM proxy and Redis. | +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿšช **LiteLLM Proxy** - Getting started with LiteLLM proxy and Redis | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/gateway/00_litellm_proxy_redis.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/gateway/00_litellm_proxy_redis.ipynb) | ### Agents -| Recipe | Description | -| --- | --- | -[/agents/00_langgraph_redis_agentic_rag.ipynb](python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb) | Notebook to get started with lang-graph and agents | -[/agents/01_crewai_langgraph_redis.ipynb](python-recipes/agents/01_crewai_langgraph_redis.ipynb) | Notebook to get started with lang-graph and agents | -[/agents/02_full_featured_agent.ipynb](python-recipes/agents/02_full_featured_agent.ipynb) | Notebook builds full tool calling agent with semantic cache and router | +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿ•ธ๏ธ **LangGraph Agents** - Notebook to get started with lang-graph and agents | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb) | +| ๐Ÿ‘ฅ **CrewAI Agents** - Notebook to get started with CrewAI and lang-graph | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/agents/01_crewai_langgraph_redis.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/agents/01_crewai_langgraph_redis.ipynb) | +| ๐Ÿง  **Memory Agent** - Building an agent with short term and long term memory using Redis | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/agents/03_memory_agent.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/agents/03_memory_agent.ipynb) | +| ๐Ÿ› ๏ธ **Full-Featured Agent** - Notebook builds full tool calling agent with semantic cache and router | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/agents/02_full_featured_agent.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/agents/02_full_featured_agent.ipynb) | ### Computer Vision -| Recipe | Description | -| ------ | ----------- | -| [/computer-vision/00_facial_recognition_facenet.ipynb](python-recipes/computer-vision/00_facial_recognition_facenet.ipynb) | Build a facial recognition system using the Facenet embedding model and RedisVL. +| Recipe | GitHub | Google Colab | +| ------ | ------ | ------------ | +| ๐Ÿ‘ค **Facial Recognition** - Build a facial recognition system using the Facenet embedding model and RedisVL | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/computer-vision/00_facial_recognition_facenet.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/computer-vision/00_facial_recognition_facenet.ipynb) | ### Recommendation Systems -| Recipe | Description | -| --- | --- | -| [/recommendation-systems/00_content_filtering.ipynb](python-recipes/recommendation-systems/00_content_filtering.ipynb) | Intro content filtering example with redisvl | -| [/recommendation-systems/01_collaborative_filtering.ipynb](python-recipes/recommendation-systems/01_collaborative_filtering.ipynb) | Intro collaborative filtering example with redisvl | -| [/recommendation-systems/02_two_towers.ipynb](python-recipes/recommendation-systems/02_two_towers.ipynb) | Intro deep learning two tower example with redisvl | +| Recipe | GitHub | Google Colab | +| --- | --- | --- | +| ๐Ÿ“‹ **Content Filtering** - Intro content filtering example with redisvl | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/recommendation-systems/00_content_filtering.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/recommendation-systems/00_content_filtering.ipynb) | +| ๐Ÿ‘ฅ **Collaborative Filtering** - Intro collaborative filtering example with redisvl | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/recommendation-systems/01_collaborative_filtering.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/recommendation-systems/01_collaborative_filtering.ipynb) | +| ๐Ÿ—๏ธ **Two Towers** - Intro deep learning two tower example with redisvl | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/recommendation-systems/02_two_towers.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/recommendation-systems/02_two_towers.ipynb) | ### Feature Store -| Recipe | Description | -| ------ | ----------- | -| [/feature-store/00_feast_credit_score.ipynb](python-recipes/feature-store/00_feast_credit_score.ipynb) | Credit scoring system using Feast with Redis as the online store. +| Recipe | GitHub | Google Colab | +| ------ | ------ | ------------ | +| ๐Ÿ’ณ **Credit Scoring** - Credit scoring system using Feast with Redis as the online store | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/feature-store/00_feast_credit_score.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/feature-store/00_feast_credit_score.ipynb) | +| ๐Ÿ” **Transaction Search** - Real-time transaction feature search with Redis | [![Open In GitHub](https://img.shields.io/badge/View-GitHub-green)](python-recipes/feature-store/01_card_transaction_search.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/feature-store/01_card_transaction_search.ipynb) | + +### โ˜•๏ธ Java AI Recipes + +A set of Java recipes can be found under [/java-recipes](/java-recipes/README.md). + ## Tutorials Need a *deeper-dive* through different use cases and topics? -| Tutorial | Description | -| -------- | ------------ | -| [Agentic RAG](https://github.com/redis-developer/agentic-rag) | A tutorial focused on agentic RAG with LlamaIndex and Cohere | -| [RAG on VertexAI](https://github.com/redis-developer/gcp-redis-llm-stack/tree/main) | A RAG tutorial featuring Redis with Vertex AI | -| [Recommendation Systems w/ NVIDIA Merlin & Redis]((https://github.com/redis-developer/redis-nvidia-recsys)) | Three examples, each escalating in complexity, showcasing the process of building a realtime recsys with NVIDIA and Redis | - + + + + + + +
+ ๐Ÿค– Agentic RAG +
+ A tutorial focused on agentic RAG with LlamaIndex and Cohere + 		+ โ˜๏ธ RAG on VertexAI +
+ A RAG tutorial featuring Redis with Vertex AI + 		+ ๐Ÿ” Recommendation Systems +
+ Building realtime recsys with NVIDIA Merlin & Redis +
+ +
## Integrations Redis integrates with many different players in the AI ecosystem. Here's a curated list below: @@ -157,6 +191,7 @@ Redis integrates with many different players in the AI ecosystem. Here's a curat | [RelevanceAI](https://relevance.ai/) | Platform to tag, search and analyze unstructured data faster, built on Redis | | [DocArray](https://docs.docarray.org/user_guide/storing/index_redis/) | DocArray Integration of Redis as a VectorDB by Jina AI | +
# Other Helpful Resources @@ -167,4 +202,23 @@ Redis integrates with many different players in the AI ecosystem. Here's a curat - [NVIDIA RecSys with Redis](https://developer.nvidia.com/blog/offline-to-online-feature-storage-for-real-time-recommendation-systems-with-nvidia-merlin/) - [Benchmarking results for vector databases](https://redis.io/blog/benchmarking-results-for-vector-databases/) - Benchmarking results for vector databases, including Redis and 7 other Vector Database players. - [Redis Vector Library Docs](https://docs.redisvl.com) -- [Redis Vector Search API Docs](https://redis.io/docs/interact/search-and-query/advanced-concepts/vectors/) - Official Redis literature for Vector Similarity Search. \ No newline at end of file +- [Redis Vector Search API Docs](https://redis.io/docs/interact/search-and-query/advanced-concepts/vectors/) - Official Redis literature for Vector Similarity Search. + +
+ +## Contributing + +We welcome contributions to Redis AI Resources! Here's how you can help: + +1. **Add a new recipe**: Create a Jupyter notebook demonstrating a Redis AI use case +2. **Improve documentation**: Enhance existing notebooks or README with clearer explanations +3. **Fix bugs**: Address issues in code samples or documentation +4. **Suggest improvements**: Open an issue with ideas for new content or enhancements + +To contribute: +1. Fork the repository +2. Create a feature branch +3. Make your changes +4. Submit a pull request + +Please follow the existing style and format of the repository when adding content. diff --git a/python-recipes/feature-store/00_feast_credit_score.ipynb b/python-recipes/feature-store/00_feast_credit_score.ipynb index f65f894b..022fb1ae 100644 --- a/python-recipes/feature-store/00_feast_credit_score.ipynb +++ b/python-recipes/feature-store/00_feast_credit_score.ipynb @@ -31,7 +31,7 @@ }, "source": [ "## Let's Begin!\n", - "\"Open\n" + "\"Open\n" ] }, { @@ -3744,4 +3744,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +} diff --git a/python-recipes/feature-store/01_card_transaction_search.ipynb b/python-recipes/feature-store/01_card_transaction_search.ipynb new file mode 100644 index 00000000..2b9bc7e3 --- /dev/null +++ b/python-recipes/feature-store/01_card_transaction_search.ipynb @@ -0,0 +1,3394 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", + "# Card Transaction Search and Analytics with RedisVL\n", + "\n", + "In this recipe, we will explore a dataset of card transactions generated by multiple users, across multiple vendors over a period of time. We will showcase the power, speed, and flexibility of the Redis Query Engine for search, filtering, vector similarity, and complex aggregations using RedisVL (Redis Vector Library).\n", + "\n", + "Transaction search and analytics have many use cases - but primarily this data is useful for building realtime feature stores that can power fraud or anomaly detection machine learning models.\n", + "\n", + "## What we'll cover\n", + "1. Loading transaction data into Redis\n", + "2. Vectorizing transaction data for semantic similarity search\n", + "3. Search techniques\n", + " - Exact match filtering\n", + " - Vector search\n", + " - Full text search and fuzzy matching\n", + " - Hybrid search\n", + "4. Complex aggregation queries\n", + " - Calculate average transaction volume per week\n", + " - Identify spending patterns\n", + " - Generate user spending profiles" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Let's Begin!\n", + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prepare data\n", + "\n", + "Our dataset is a list of 200 credit card transactions (fake)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n", + "!mv temp_repo/python-recipes/feature-store/resources .\n", + "!rm -rf temp_repo" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Install Required Packages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q \"redisvl==0.6.0\" sentence-transformers pandas nltk" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Install Redis Stack\n", + "\n", + "In this tutorial, Redis will be used to store, index, and query vector\n", + "embeddings created from transaction data. **We need to make sure we have a Redis\n", + "instance available**.\n", + "\n", + "#### For Colab\n", + "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "%%sh\n", + "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", + "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", + "sudo apt-get update > /dev/null 2>&1\n", + "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", + "redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### For Alternative Environments\n", + "There are many ways to get the necessary redis-stack instance running\n", + "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", + "own version of Redis Enterprise running, that works too!\n", + "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", + "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Define the Redis Connection URL\n", + "\n", + "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import warnings\n", + "\n", + "warnings.filterwarnings('ignore')\n", + "\n", + "# Replace values below with your own if using Redis Cloud instance\n", + "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", + "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", + "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", + "\n", + "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", + "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create Redis client and test connection" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from redis import Redis\n", + "\n", + "client = Redis.from_url(REDIS_URL)\n", + "client.ping()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Optional: clear all data in Redis if needed\n", + "client.flushall()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load Transaction Dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 200 transaction entries\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
transaction_iduser_idmerchant_iditem_nameamountcurrencytimestamplatloncard_providerlocation
0txn_0001u_002m_009Headphones1154.59USD174618255127.584806-71.730465VISA-71.730465, 27.584806
1txn_0002u_013m_018Dinner501.64USD174697095128.831898-104.441434AMEX-104.441434, 28.831898
2txn_0003u_008m_006Laptop1359.33USD174684135146.087128-102.099503VISA-102.099503, 46.087128
3txn_0004u_011m_024Gaming Console157.54USD174700335127.226349-115.753846VISA-115.753846, 27.226349
4txn_0005u_010m_014Concert Ticket718.00USD174543375145.108103-79.409905AMEX-79.409905, 45.108103
\n", + "
" + ], + "text/plain": [ + " transaction_id user_id merchant_id item_name amount currency \\\n", + "0 txn_0001 u_002 m_009 Headphones 1154.59 USD \n", + "1 txn_0002 u_013 m_018 Dinner 501.64 USD \n", + "2 txn_0003 u_008 m_006 Laptop 1359.33 USD \n", + "3 txn_0004 u_011 m_024 Gaming Console 157.54 USD \n", + "4 txn_0005 u_010 m_014 Concert Ticket 718.00 USD \n", + "\n", + " timestamp lat lon card_provider location \n", + "0 1746182551 27.584806 -71.730465 VISA -71.730465, 27.584806 \n", + "1 1746970951 28.831898 -104.441434 AMEX -104.441434, 28.831898 \n", + "2 1746841351 46.087128 -102.099503 VISA -102.099503, 46.087128 \n", + "3 1747003351 27.226349 -115.753846 VISA -115.753846, 27.226349 \n", + "4 1745433751 45.108103 -79.409905 AMEX -79.409905, 45.108103 " + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "# Load transactions from JSON file\n", + "df = pd.read_json(\"resources/transactions_200.json\")\n", + "print(f\"Loaded {len(df)} transaction entries\")\n", + "\n", + "# # Convert timestamp to datetime for easier manipulation\n", + "df[\"timestamp\"] = df[\"timestamp\"].apply(lambda s: int(pd.to_datetime(s).timestamp()))\n", + "df['location'] = df.apply(lambda r: f\"{r.lon}, {r.lat}\", axis=1)\n", + "\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Let's examine the transaction data" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "count 200.00000\n", + "mean 747.57135\n", + "std 426.08199\n", + "min 26.63000\n", + "25% 373.06250\n", + "50% 696.15500\n", + "75% 1130.19750\n", + "max 1499.87000\n", + "Name: amount, dtype: float64" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Basic statistics on transaction amounts\n", + "df['amount'].describe()" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "card_provider\n", + "DISCOVER 54\n", + "AMEX 52\n", + "VISA 51\n", + "MASTERCARD 43\n", + "Name: count, dtype: int64" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Count of transactions by card provider\n", + "df['card_provider'].value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "item_name\n", + "Plane Ticket 15\n", + "Hotel Stay 15\n", + "Groceries 14\n", + "Dinner 14\n", + "Headphones 13\n", + "Gym Membership 13\n", + "Bicycle 12\n", + "Gaming Console 11\n", + "Streaming Subscription 11\n", + "Smartphone 9\n", + "Name: count, dtype: int64" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Look at the most common items purchased\n", + "df['item_name'].value_counts().head(10)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "user_id\n", + "u_013 16\n", + "u_012 14\n", + "u_008 13\n", + "u_006 13\n", + "u_014 12\n", + "u_018 12\n", + "u_007 11\n", + "u_011 10\n", + "u_010 10\n", + "u_020 10\n", + "u_009 9\n", + "u_002 9\n", + "u_016 9\n", + "u_001 9\n", + "u_017 8\n", + "u_015 8\n", + "u_005 7\n", + "u_019 7\n", + "u_003 7\n", + "u_004 6\n", + "Name: count, dtype: int64" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Look at how many users there are\n", + "df['user_id'].value_counts()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Vectorize Transaction Data for Semantic Search\n", + "\n", + "We'll use a Hugging Face sentence transformer to create vector embeddings for transaction data. The text we'll vectorize will be a combination of:\n", + "- Merchant name\n", + "- Item purchased\n", + "- Transaction amount" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "13:15:47 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n", + "13:15:47 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 7.46it/s]\n" + ] + } + ], + "source": [ + "from redisvl.utils.vectorize import HFTextVectorizer\n", + "\n", + "# Set environment variable to avoid parallelism warnings\n", + "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n", + "\n", + "# Initialize the vectorizer with a small but powerful model\n", + "hf = HFTextVectorizer(\"sentence-transformers/all-MiniLM-L6-v2\")" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
transaction_idvector_text
0txn_0001Merchant m_009 selling Headphones for $1154.59
1txn_0002Merchant m_018 selling Dinner for $501.64
2txn_0003Merchant m_006 selling Laptop for $1359.33
\n", + "
" + ], + "text/plain": [ + " transaction_id vector_text\n", + "0 txn_0001 Merchant m_009 selling Headphones for $1154.59\n", + "1 txn_0002 Merchant m_018 selling Dinner for $501.64\n", + "2 txn_0003 Merchant m_006 selling Laptop for $1359.33" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Create a combined text field for vectorization\n", + "def create_text_for_vectorization(row):\n", + " return f\"Merchant {row['merchant_id']} selling {row['item_name']} for ${row['amount']:.2f}\"\n", + "\n", + "df['vector_text'] = df.apply(create_text_for_vectorization, axis=1)\n", + "\n", + "# Display some examples\n", + "df[['transaction_id', 'vector_text']].head(3)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Generating vectors for transactions...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 8.56it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 13.23it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 77.89it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 70.84it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 68.89it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 68.21it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 76.25it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 82.86it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 87.50it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 72.29it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 74.83it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 70.85it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 72.97it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 83.08it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 87.37it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 85.30it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 73.55it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 77.35it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 78.44it/s]\n", + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 79.18it/s]\n" + ] + } + ], + "source": [ + "# Generate vectors for each transaction\n", + "print(\"Generating vectors for transactions...\")\n", + "df[\"vector\"] = hf.embed_many(df[\"vector_text\"].tolist(), as_buffer=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Define Redis Index Schema\n", + "\n", + "We'll create a schema that includes both standard fields and vector field for our transaction data." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "from redisvl.schema import IndexSchema\n", + "from redisvl.index import SearchIndex\n", + "\n", + "\n", + "# Define the index schema with fields we want to search and filter on\n", + "schema = IndexSchema.from_dict({\n", + " \"index\": {\n", + " \"name\": \"transactions\",\n", + " \"prefix\": \"transactions:entry\",\n", + " \"storage_type\": \"hash\"\n", + " },\n", + " \"fields\": [\n", + " {\n", + " \"name\": \"transaction_id\",\n", + " \"type\": \"tag\",\n", + " \"attrs\": {\n", + " \"sortable\": True\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"user_id\",\n", + " \"type\": \"tag\",\n", + " \"attrs\": {\n", + " \"sortable\": True\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"merchant_id\",\n", + " \"type\": \"tag\",\n", + " \"attrs\": {\n", + " \"sortable\": True\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"item_name\",\n", + " \"type\": \"text\",\n", + " \"attrs\": {\n", + " \"sortable\": True\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"amount\",\n", + " \"type\": \"numeric\",\n", + " \"attrs\": {\n", + " \"sortable\": True\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"currency\",\n", + " \"type\": \"tag\",\n", + " },\n", + " {\n", + " \"name\": \"timestamp\",\n", + " \"type\": \"numeric\",\n", + " \"attrs\": {\n", + " \"sortable\": True\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"card_provider\",\n", + " \"type\": \"tag\",\n", + " },\n", + " {\n", + " \"name\": \"location\",\n", + " \"type\": \"geo\",\n", + " },\n", + " {\n", + " \"name\": \"vector\",\n", + " \"type\": \"vector\",\n", + " \"attrs\": {\n", + " \"dims\": 384, # Based on the all-MiniLM-L6-v2 model\n", + " \"distance_metric\": \"cosine\",\n", + " \"algorithm\": \"flat\",\n", + " \"datatype\": \"float32\"\n", + " }\n", + " }\n", + " ]\n", + "})\n", + "\n", + "# Create the index\n", + "index = SearchIndex(schema, client)\n", + "index.create(overwrite=True, drop=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "Index Information:\n", + "โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n", + "โ”‚ Index Name โ”‚ Storage Type โ”‚ Prefixes โ”‚ Index Options โ”‚ Indexing โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "| transactions | HASH | ['transactions:entry'] | [] | 0 |\n", + "โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n", + "Index Fields:\n", + "โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n", + "โ”‚ Name โ”‚ Attribute โ”‚ Type โ”‚ Field Option โ”‚ Option Value โ”‚ Field Option โ”‚ Option Value โ”‚ Field Option โ”‚ Option Value โ”‚ Field Option โ”‚ Option Value โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ transaction_id โ”‚ transaction_id โ”‚ TAG โ”‚ SEPARATOR โ”‚ , โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ user_id โ”‚ user_id โ”‚ TAG โ”‚ SEPARATOR โ”‚ , โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ merchant_id โ”‚ merchant_id โ”‚ TAG โ”‚ SEPARATOR โ”‚ , โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ item_name โ”‚ item_name โ”‚ TEXT โ”‚ WEIGHT โ”‚ 1 โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ amount โ”‚ amount โ”‚ NUMERIC โ”‚ SORTABLE โ”‚ UNF โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ currency โ”‚ currency โ”‚ TAG โ”‚ SEPARATOR โ”‚ , โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ timestamp โ”‚ timestamp โ”‚ NUMERIC โ”‚ SORTABLE โ”‚ UNF โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ card_provider โ”‚ card_provider โ”‚ TAG โ”‚ SEPARATOR โ”‚ , โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ location โ”‚ location โ”‚ GEO โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ”‚ vector โ”‚ vector โ”‚ VECTOR โ”‚ algorithm โ”‚ FLAT โ”‚ data_type โ”‚ FLOAT32 โ”‚ dim โ”‚ 384 โ”‚ distance_metric โ”‚ COSINE โ”‚\n", + "โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n" + ] + } + ], + "source": [ + "# Check the index information\n", + "!rvl index info -i transactions -u {REDIS_URL}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Populate Redis with Transaction Data\n", + "\n", + "Now that our index is created, let's load the transaction data into Redis." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 200 transactions into Redis\n" + ] + }, + { + "data": { + "text/plain": [ + "['transactions:entry:txn_0001',\n", + " 'transactions:entry:txn_0002',\n", + " 'transactions:entry:txn_0003',\n", + " 'transactions:entry:txn_0004',\n", + " 'transactions:entry:txn_0005']" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Load data into Redis\n", + "transaction_ids = index.load(\n", + " data=df.to_dict(orient=\"records\"),\n", + " id_field=\"transaction_id\"\n", + ")\n", + "print(f\"Loaded {len(transaction_ids)} transactions into Redis\")\n", + "\n", + "# Display the first few transaction IDs loaded\n", + "transaction_ids[:5]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Part I: Transaction Search Techniques\n", + "\n", + "Now that we have our data loaded into Redis, let's explore different search techniques." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Exact Match Queryies & Sorting\n", + "\n", + "Let's start with some basic exact match filtering to find transactions with specific properties." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idtransaction_iduser_idmerchant_iditem_nameamountcard_provider
0transactions:entry:txn_0002txn_0002u_013m_018Dinner501.64AMEX
1transactions:entry:txn_0005txn_0005u_010m_014Concert Ticket718AMEX
2transactions:entry:txn_0006txn_0006u_017m_016Hotel Stay1232.8AMEX
3transactions:entry:txn_0015txn_0015u_001m_018Clothing114.86AMEX
4transactions:entry:txn_0032txn_0032u_013m_002Concert Ticket585.69AMEX
\n", + "
" + ], + "text/plain": [ + " id transaction_id user_id merchant_id \\\n", + "0 transactions:entry:txn_0002 txn_0002 u_013 m_018 \n", + "1 transactions:entry:txn_0005 txn_0005 u_010 m_014 \n", + "2 transactions:entry:txn_0006 txn_0006 u_017 m_016 \n", + "3 transactions:entry:txn_0015 txn_0015 u_001 m_018 \n", + "4 transactions:entry:txn_0032 txn_0032 u_013 m_002 \n", + "\n", + " item_name amount card_provider \n", + "0 Dinner 501.64 AMEX \n", + "1 Concert Ticket 718 AMEX \n", + "2 Hotel Stay 1232.8 AMEX \n", + "3 Clothing 114.86 AMEX \n", + "4 Concert Ticket 585.69 AMEX " + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from redisvl.query import FilterQuery\n", + "from redisvl.query.filter import Tag, Num\n", + "\n", + "# Find all AMEX transactions\n", + "card_filter = Tag(\"card_provider\") == \"AMEX\"\n", + "\n", + "query = FilterQuery(\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", + " filter_expression=card_filter,\n", + " num_results=5\n", + ")\n", + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idamounttransaction_iduser_idmerchant_iditem_namecard_provider
0transactions:entry:txn_00611499.87txn_0061u_006m_004GroceriesVISA
1transactions:entry:txn_01111471.73txn_0111u_014m_006CoffeeVISA
2transactions:entry:txn_01471462.78txn_0147u_018m_003DinnerMASTERCARD
3transactions:entry:txn_00191462.52txn_0019u_012m_005DinnerDISCOVER
4transactions:entry:txn_01681450.52txn_0168u_016m_014GroceriesAMEX
\n", + "
" + ], + "text/plain": [ + " id amount transaction_id user_id merchant_id \\\n", + "0 transactions:entry:txn_0061 1499.87 txn_0061 u_006 m_004 \n", + "1 transactions:entry:txn_0111 1471.73 txn_0111 u_014 m_006 \n", + "2 transactions:entry:txn_0147 1462.78 txn_0147 u_018 m_003 \n", + "3 transactions:entry:txn_0019 1462.52 txn_0019 u_012 m_005 \n", + "4 transactions:entry:txn_0168 1450.52 txn_0168 u_016 m_014 \n", + "\n", + " item_name card_provider \n", + "0 Groceries VISA \n", + "1 Coffee VISA \n", + "2 Dinner MASTERCARD \n", + "3 Dinner DISCOVER \n", + "4 Groceries AMEX " + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Find high-value transactions (over $1000)\n", + "amount_filter = Num(\"amount\") > 1000\n", + "\n", + "query = FilterQuery(\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", + " filter_expression=amount_filter,\n", + " num_results=5,\n", + ").sort_by(\"amount\", asc=False)\n", + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idamounttransaction_iduser_idmerchant_iditem_nametimestamp
0transactions:entry:txn_01901413.99txn_0190u_013m_005Plane Ticket1745570551
1transactions:entry:txn_01451382.3txn_0145u_013m_024Hotel Stay1746927751
2transactions:entry:txn_01031360.17txn_0103u_013m_015Coffee1744911751
3transactions:entry:txn_00671311.19txn_0067u_013m_010Headphones1746715351
4transactions:entry:txn_00651231.32txn_0065u_013m_012Plane Ticket1746675751
5transactions:entry:txn_00601094.44txn_0060u_013m_001Ride Share1744857751
6transactions:entry:txn_01501075.13txn_0150u_013m_003Plane Ticket1746812551
7transactions:entry:txn_01251032.48txn_0125u_013m_018Shoes1747143751
8transactions:entry:txn_0058916.96txn_0058u_013m_024Plane Ticket1745523751
9transactions:entry:txn_0113733.8txn_0113u_013m_001Software License1745566951
\n", + "
" + ], + "text/plain": [ + " id amount transaction_id user_id merchant_id \\\n", + "0 transactions:entry:txn_0190 1413.99 txn_0190 u_013 m_005 \n", + "1 transactions:entry:txn_0145 1382.3 txn_0145 u_013 m_024 \n", + "2 transactions:entry:txn_0103 1360.17 txn_0103 u_013 m_015 \n", + "3 transactions:entry:txn_0067 1311.19 txn_0067 u_013 m_010 \n", + "4 transactions:entry:txn_0065 1231.32 txn_0065 u_013 m_012 \n", + "5 transactions:entry:txn_0060 1094.44 txn_0060 u_013 m_001 \n", + "6 transactions:entry:txn_0150 1075.13 txn_0150 u_013 m_003 \n", + "7 transactions:entry:txn_0125 1032.48 txn_0125 u_013 m_018 \n", + "8 transactions:entry:txn_0058 916.96 txn_0058 u_013 m_024 \n", + "9 transactions:entry:txn_0113 733.8 txn_0113 u_013 m_001 \n", + "\n", + " item_name timestamp \n", + "0 Plane Ticket 1745570551 \n", + "1 Hotel Stay 1746927751 \n", + "2 Coffee 1744911751 \n", + "3 Headphones 1746715351 \n", + "4 Plane Ticket 1746675751 \n", + "5 Ride Share 1744857751 \n", + "6 Plane Ticket 1746812551 \n", + "7 Shoes 1747143751 \n", + "8 Plane Ticket 1745523751 \n", + "9 Software License 1745566951 " + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Combine filters: Find transactions for a specific user with high amounts\n", + "user_filter = Tag(\"user_id\") == \"u_013\" # Specific user\n", + "amount_filter = Num(\"amount\") > 500 # High amount threshold\n", + "\n", + "# Combine filters with logical AND\n", + "combined_filter = user_filter & amount_filter\n", + "\n", + "query = FilterQuery(\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"timestamp\"],\n", + " filter_expression=combined_filter,\n", + ").sort_by(\"amount\", asc=False)\n", + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Vector Search\n", + "\n", + "Now let's use vector search to find transactions semantically similar to a query." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 10.19it/s]\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idvector_distanceamounttransaction_idmerchant_iditem_name
0transactions:entry:txn_00570.5002093911171201.71txn_0057m_011Laptop
1transactions:entry:txn_00980.497323393822503.01txn_0098m_012Headphones
2transactions:entry:txn_01700.500393152237374.23txn_0170m_010Headphones
3transactions:entry:txn_00400.495004236698159.33txn_0040m_017Headphones
4transactions:entry:txn_01690.494512319565153.22txn_0169m_008Headphones
\n", + "
" + ], + "text/plain": [ + " id vector_distance amount transaction_id \\\n", + "0 transactions:entry:txn_0057 0.500209391117 1201.71 txn_0057 \n", + "1 transactions:entry:txn_0098 0.497323393822 503.01 txn_0098 \n", + "2 transactions:entry:txn_0170 0.500393152237 374.23 txn_0170 \n", + "3 transactions:entry:txn_0040 0.495004236698 159.33 txn_0040 \n", + "4 transactions:entry:txn_0169 0.494512319565 153.22 txn_0169 \n", + "\n", + " merchant_id item_name \n", + "0 m_011 Laptop \n", + "1 m_012 Headphones \n", + "2 m_010 Headphones \n", + "3 m_017 Headphones \n", + "4 m_008 Headphones " + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from redisvl.query import VectorQuery\n", + "\n", + "# Search for expensive electronics\n", + "user_query = \"Expensive electronics purchase\"\n", + "\n", + "# Vectorize the user's query\n", + "embedded_user_query = hf.embed(user_query, as_buffer=True)\n", + "\n", + "# Create vector query\n", + "vec_query = VectorQuery(\n", + " vector=embedded_user_query,\n", + " vector_field_name=\"vector\",\n", + " num_results=5,\n", + " return_fields=[\"transaction_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", + " return_score=True,\n", + ").sort_by(\"amount\", asc=False)\n", + "\n", + "# Execute the query\n", + "results = index.query(vec_query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Vector Search with Filters\n", + "\n", + "We can combine vector search with exact match filters to get more precise results." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 70.97it/s]\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idvector_distanceamounttransaction_iduser_idmerchant_iditem_name
0transactions:entry:txn_00420.533764362335742.36txn_0042u_017m_012Groceries
1transactions:entry:txn_00410.511252999306612.59txn_0041u_017m_003Clothing
2transactions:entry:txn_00080.563050031662564.91txn_0008u_017m_022Shoes
3transactions:entry:txn_00180.553927659988462.71txn_0018u_017m_013Dinner
4transactions:entry:txn_01950.5286039114429.52txn_0195u_017m_002Groceries
\n", + "
" + ], + "text/plain": [ + " id vector_distance amount transaction_id user_id \\\n", + "0 transactions:entry:txn_0042 0.533764362335 742.36 txn_0042 u_017 \n", + "1 transactions:entry:txn_0041 0.511252999306 612.59 txn_0041 u_017 \n", + "2 transactions:entry:txn_0008 0.563050031662 564.91 txn_0008 u_017 \n", + "3 transactions:entry:txn_0018 0.553927659988 462.71 txn_0018 u_017 \n", + "4 transactions:entry:txn_0195 0.5286039114 429.52 txn_0195 u_017 \n", + "\n", + " merchant_id item_name \n", + "0 m_012 Groceries \n", + "1 m_003 Clothing \n", + "2 m_022 Shoes \n", + "3 m_013 Dinner \n", + "4 m_002 Groceries " + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Search for expensive purchases by a specific user\n", + "user_query = \"Large purchases\"\n", + "embedded_user_query = hf.embed(user_query)\n", + "\n", + "# Filter for a specific user\n", + "user_filter = Tag(\"user_id\") == \"u_017\"\n", + "\n", + "# Create vector query with filter\n", + "vec_query = VectorQuery(\n", + " vector=embedded_user_query,\n", + " vector_field_name=\"vector\",\n", + " num_results=5,\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", + " return_score=True,\n", + " filter_expression=user_filter\n", + ").sort_by(\"amount\", asc=False)\n", + "\n", + "# Execute the query\n", + "results = index.query(vec_query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 14.38it/s]\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idvector_distancetransaction_iduser_idmerchant_iditem_nameamount
0transactions:entry:txn_00460.681029856205txn_0046u_013m_018Hotel Stay588.18
1transactions:entry:txn_01580.68172955513txn_0158u_015m_020Hotel Stay835.78
2transactions:entry:txn_00480.690501689911txn_0048u_010m_012Hotel Stay588.93
3transactions:entry:txn_01910.707059979439txn_0191u_001m_014Plane Ticket912.33
4transactions:entry:txn_00580.725707709789txn_0058u_013m_024Plane Ticket916.96
\n", + "
" + ], + "text/plain": [ + " id vector_distance transaction_id user_id \\\n", + "0 transactions:entry:txn_0046 0.681029856205 txn_0046 u_013 \n", + "1 transactions:entry:txn_0158 0.68172955513 txn_0158 u_015 \n", + "2 transactions:entry:txn_0048 0.690501689911 txn_0048 u_010 \n", + "3 transactions:entry:txn_0191 0.707059979439 txn_0191 u_001 \n", + "4 transactions:entry:txn_0058 0.725707709789 txn_0058 u_013 \n", + "\n", + " merchant_id item_name amount \n", + "0 m_018 Hotel Stay 588.18 \n", + "1 m_020 Hotel Stay 835.78 \n", + "2 m_012 Hotel Stay 588.93 \n", + "3 m_014 Plane Ticket 912.33 \n", + "4 m_024 Plane Ticket 916.96 " + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Search for travel expenses with price range filter\n", + "user_query = \"Travel-related expenses\"\n", + "embedded_user_query = hf.embed(user_query)\n", + "\n", + "# Price range filter\n", + "min_amount = Num(\"amount\") >= 500\n", + "max_amount = Num(\"amount\") <= 1000\n", + "price_range = min_amount & max_amount\n", + "\n", + "# Create vector query with filter\n", + "vec_query = VectorQuery(\n", + " vector=embedded_user_query,\n", + " vector_field_name=\"vector\",\n", + " num_results=5,\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", + " return_score=True,\n", + " filter_expression=price_range\n", + ")\n", + "\n", + "# Execute the query\n", + "results = index.query(vec_query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Full Text Search\n", + "\n", + "Redis also provides powerful full-text search capabilities." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idscoretransaction_iduser_idmerchant_iditem_nameamount
0transactions:entry:txn_00043.246753txn_0004u_011m_024Gaming Console157.54
1transactions:entry:txn_00133.246753txn_0013u_001m_005Gaming Console293.8
2transactions:entry:txn_00333.246753txn_0033u_008m_021Gaming Console402.54
3transactions:entry:txn_00363.246753txn_0036u_020m_002Gaming Console758.65
4transactions:entry:txn_00723.246753txn_0072u_007m_003Gaming Console68.88
5transactions:entry:txn_00883.246753txn_0088u_011m_015Gaming Console26.63
6transactions:entry:txn_00963.246753txn_0096u_014m_021Gaming Console1393.99
7transactions:entry:txn_01023.246753txn_0102u_016m_021Gaming Console697.55
8transactions:entry:txn_01093.246753txn_0109u_007m_020Gaming Console43.49
9transactions:entry:txn_01273.246753txn_0127u_001m_021Gaming Console508.48
10transactions:entry:txn_01403.246753txn_0140u_014m_008Gaming Console884.5
11transactions:entry:txn_00122.435065txn_0012u_018m_016Plane Ticket234.87
12transactions:entry:txn_00582.435065txn_0058u_013m_024Plane Ticket916.96
13transactions:entry:txn_00652.435065txn_0065u_013m_012Plane Ticket1231.32
14transactions:entry:txn_00702.435065txn_0070u_003m_005Plane Ticket1000.78
\n", + "
" + ], + "text/plain": [ + " id score transaction_id user_id merchant_id \\\n", + "0 transactions:entry:txn_0004 3.246753 txn_0004 u_011 m_024 \n", + "1 transactions:entry:txn_0013 3.246753 txn_0013 u_001 m_005 \n", + "2 transactions:entry:txn_0033 3.246753 txn_0033 u_008 m_021 \n", + "3 transactions:entry:txn_0036 3.246753 txn_0036 u_020 m_002 \n", + "4 transactions:entry:txn_0072 3.246753 txn_0072 u_007 m_003 \n", + "5 transactions:entry:txn_0088 3.246753 txn_0088 u_011 m_015 \n", + "6 transactions:entry:txn_0096 3.246753 txn_0096 u_014 m_021 \n", + "7 transactions:entry:txn_0102 3.246753 txn_0102 u_016 m_021 \n", + "8 transactions:entry:txn_0109 3.246753 txn_0109 u_007 m_020 \n", + "9 transactions:entry:txn_0127 3.246753 txn_0127 u_001 m_021 \n", + "10 transactions:entry:txn_0140 3.246753 txn_0140 u_014 m_008 \n", + "11 transactions:entry:txn_0012 2.435065 txn_0012 u_018 m_016 \n", + "12 transactions:entry:txn_0058 2.435065 txn_0058 u_013 m_024 \n", + "13 transactions:entry:txn_0065 2.435065 txn_0065 u_013 m_012 \n", + "14 transactions:entry:txn_0070 2.435065 txn_0070 u_003 m_005 \n", + "\n", + " item_name amount \n", + "0 Gaming Console 157.54 \n", + "1 Gaming Console 293.8 \n", + "2 Gaming Console 402.54 \n", + "3 Gaming Console 758.65 \n", + "4 Gaming Console 68.88 \n", + "5 Gaming Console 26.63 \n", + "6 Gaming Console 1393.99 \n", + "7 Gaming Console 697.55 \n", + "8 Gaming Console 43.49 \n", + "9 Gaming Console 508.48 \n", + "10 Gaming Console 884.5 \n", + "11 Plane Ticket 234.87 \n", + "12 Plane Ticket 916.96 \n", + "13 Plane Ticket 1231.32 \n", + "14 Plane Ticket 1000.78 " + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from redisvl.query import TextQuery\n", + "from redisvl.query.filter import Text\n", + "\n", + "# Search for specific items\n", + "text_query = TextQuery(\n", + " text=\"Gaming system, plane tickets, and hotel rooms\",\n", + " text_field_name=\"item_name\",\n", + " text_scorer=\"BM25\",\n", + " num_results=15,\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", + ")\n", + "\n", + "results = index.query(text_query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Fuzzy search is another popular technique to help with record linkage tasks." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idtransaction_iduser_idmerchant_idamountitem_name
0transactions:entry:txn_0032txn_0032u_013m_002585.69Concert Ticket
1transactions:entry:txn_0058txn_0058u_013m_024916.96Plane Ticket
2transactions:entry:txn_0065txn_0065u_013m_0121231.32Plane Ticket
3transactions:entry:txn_0150txn_0150u_013m_0031075.13Plane Ticket
4transactions:entry:txn_0190txn_0190u_013m_0051413.99Plane Ticket
\n", + "
" + ], + "text/plain": [ + " id transaction_id user_id merchant_id amount \\\n", + "0 transactions:entry:txn_0032 txn_0032 u_013 m_002 585.69 \n", + "1 transactions:entry:txn_0058 txn_0058 u_013 m_024 916.96 \n", + "2 transactions:entry:txn_0065 txn_0065 u_013 m_012 1231.32 \n", + "3 transactions:entry:txn_0150 txn_0150 u_013 m_003 1075.13 \n", + "4 transactions:entry:txn_0190 txn_0190 u_013 m_005 1413.99 \n", + "\n", + " item_name \n", + "0 Concert Ticket \n", + "1 Plane Ticket \n", + "2 Plane Ticket \n", + "3 Plane Ticket \n", + "4 Plane Ticket " + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from redisvl.query.filter import Text\n", + "\n", + "user_filter = Tag(\"user_id\") == \"u_013\" # Specific user\n", + "fuzzy = Text(\"item_name\") % \"%%tickt%%\"\n", + "\n", + "fuzzy_match = FilterQuery(\n", + " filter_expression=user_filter & fuzzy,\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"amount\", \"item_name\"]\n", + ")\n", + "\n", + "results = index.query(fuzzy_match)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idtransaction_iduser_idmerchant_iditem_nameamountcard_provider
0transactions:entry:txn_0004txn_0004u_011m_024Gaming Console157.54VISA
1transactions:entry:txn_0036txn_0036u_020m_002Gaming Console758.65VISA
2transactions:entry:txn_0102txn_0102u_016m_021Gaming Console697.55VISA
\n", + "
" + ], + "text/plain": [ + " id transaction_id user_id merchant_id \\\n", + "0 transactions:entry:txn_0004 txn_0004 u_011 m_024 \n", + "1 transactions:entry:txn_0036 txn_0036 u_020 m_002 \n", + "2 transactions:entry:txn_0102 txn_0102 u_016 m_021 \n", + "\n", + " item_name amount card_provider \n", + "0 Gaming Console 157.54 VISA \n", + "1 Gaming Console 758.65 VISA \n", + "2 Gaming Console 697.55 VISA " + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Exact text match combined with other filters\n", + "text_filter = Text(\"item_name\") % \"Gaming\" # Full text search for Laptop\n", + "card_filter = Tag(\"card_provider\") == \"VISA\" # Only VISA card transactions\n", + "combined_filter = text_filter & card_filter\n", + "\n", + "query = FilterQuery(\n", + " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", + " filter_expression=combined_filter,\n", + ")\n", + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Part II: Record Linkage Examples\n", + "\n", + "Let's use our various search techniques to tackle a simple record linkage task. Below we have a \"fake\" transaction without a unique transaction ID. It may or may not be a duplicate of the data in our index already.\n", + "\n", + "Because Redis is fast we can perform fast record linkage techniques and serve transaction search clients as well.\n", + "\n", + "**Record linkage techniques in Redis:**\n", + "- Exact match & fuzzy text search & timestamp range\n", + "- Semantic search with vectors\n", + "- Bloom filters (probabalistic data structures -- not shown here)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [], + "source": [ + "# Create a duplicate transaction that's similar to an existing one\n", + "fake_transaction = {\n", + " \"user_id\": \"u_013\", # Same user as txn_0032\n", + " \"merchant_id\": \"m_002\", # Same merchant as txn_0032\n", + " \"item_name\": \"Concert Tickt\", # Same item slightly mispelled\n", + " \"amount\": 585.69, # Same amount\n", + " \"currency\": \"USD\",\n", + " \"timestamp\": 1746765800, # Very close timestamp\n", + " \"card_provider\": \"AMEX\", # Same card provider\n", + " \"lat\": 36.173155, \n", + " \"lon\": -79.595479, \n", + " \"location\": \"36.173155,-79.595479\" \n", + "}\n", + "\n", + "# In this example, the transaction is a mistaken duplicate charge by the vendor" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Batches: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:00<00:00, 14.15it/s]\n" + ] + } + ], + "source": [ + "# User ID and Merchant ID must be the same\n", + "exact_matches = (Tag(\"user_id\")==\"u_013\") & (Tag(\"merchant_id\")==\"m_002\")\n", + "\n", + "# Fuzzy match on Item Name\n", + "terms = fake_transaction['item_name'].split()\n", + "fuzzy_item_name = \" | \".join([f\"%%{term}%%\" for term in terms])\n", + "fuzzy_match = Text(\"item_name\") % fuzzy_item_name\n", + "\n", + "# Timestamp range - create 60 second window on either side of transaction timestamp\n", + "from redisvl.query.filter import Timestamp\n", + "\n", + "start_ts = fake_transaction['timestamp'] - 60\n", + "end_ts = fake_transaction['timestamp'] + 60\n", + "timestamp_range = Timestamp(\"timestamp\").between(start_ts, end_ts)\n", + "\n", + "# Make transaction vector\n", + "transaction_vector = hf.embed(create_text_for_vectorization(fake_transaction), as_buffer=True)\n", + "\n", + "# Build query\n", + "query = VectorQuery(\n", + " vector=transaction_vector,\n", + " vector_field_name=\"vector\",\n", + " filter_expression=exact_matches & fuzzy_match & timestamp_range,\n", + " return_fields=[\"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"timestamp\", \"location\"],\n", + " num_results=3\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'(((@user_id:{u_013} @merchant_id:{m_002}) @item_name:(%%Concert%% | %%Tickt%%)) @timestamp:[1746765740.0 1746765860.0])=>[KNN 3 @vector $vector AS vector_distance] RETURN 7 user_id merchant_id item_name amount timestamp location vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 3'" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "str(query)" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'id': 'transactions:entry:txn_0032',\n", + " 'vector_distance': '0.0979611873627',\n", + " 'user_id': 'u_013',\n", + " 'merchant_id': 'm_002',\n", + " 'item_name': 'Concert Ticket',\n", + " 'amount': '585.69',\n", + " 'timestamp': '1746765751',\n", + " 'location': '-79.595479, 36.173155'}]" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "index.query(query)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This kind of search op for entity resolution can be very fast!" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "479 ฮผs ยฑ 19.7 ฮผs per loop (mean ยฑ std. dev. of 7 runs, 1,000 loops each)\n" + ] + } + ], + "source": [ + "%%timeit\n", + "\n", + "index.query(query)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Part III: Complex Aggregations\n", + "\n", + "Now let's explore Redis's powerful aggregation capabilities to analyze transaction data. This can be useful for feature store workloads, anomaly detection models, and even basic realtime analytics." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Basic Aggregations\n", + "\n", + "First, let's look at some simple aggregations to understand spending patterns." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [], + "source": [ + "import redis.commands.search.reducers as reducers\n", + "\n", + "from redisvl.redis.utils import convert_bytes, make_dict\n", + "from redisvl.query.aggregate import AggregationQuery" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
card_provideravg_amounttotal_amountcount
0DISCOVER661.07148148135697.8654
1VISA717.11470588236572.8551
2MASTERCARD800.72976744234431.3843
3AMEX823.31115384642812.1852
\n", + "
" + ], + "text/plain": [ + " card_provider avg_amount total_amount count\n", + "0 DISCOVER 661.071481481 35697.86 54\n", + "1 VISA 717.114705882 36572.85 51\n", + "2 MASTERCARD 800.729767442 34431.38 43\n", + "3 AMEX 823.311153846 42812.18 52" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Calculate average transaction amount by card provider\n", + "agg_query = AggregationQuery(\"*\") \\\n", + " .group_by(\n", + " \"@card_provider\",\n", + " reducers.avg(\"amount\").alias(\"avg_amount\"),\n", + " reducers.sum(\"amount\").alias(\"total_amount\"),\n", + " reducers.count().alias(\"count\")\n", + " ) \\\n", + " .sort_by(\"@avg_amount\")\n", + "\n", + "results = index.aggregate(agg_query)\n", + "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", + "\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
item_nametotal_spentcount
0furniture28793
1camera2700.875
2laptop4597.796
3concert ticket3771.986
4software license6070.978
5shoes7181.089
6coffee7959.159
7clothing5670.879
8smartphone6312.799
9ride share6462.659
10book7423.419
11gaming console5236.0511
12streaming subscription10386.9311
13bicycle8212.1212
14gym membership11243.3213
15headphones6981.0513
16dinner10573.2714
17groceries10242.8414
18plane ticket11890.2815
19hotel stay13717.8515
\n", + "
" + ], + "text/plain": [ + " item_name total_spent count\n", + "0 furniture 2879 3\n", + "1 camera 2700.87 5\n", + "2 laptop 4597.79 6\n", + "3 concert ticket 3771.98 6\n", + "4 software license 6070.97 8\n", + "5 shoes 7181.08 9\n", + "6 coffee 7959.15 9\n", + "7 clothing 5670.87 9\n", + "8 smartphone 6312.79 9\n", + "9 ride share 6462.65 9\n", + "10 book 7423.41 9\n", + "11 gaming console 5236.05 11\n", + "12 streaming subscription 10386.93 11\n", + "13 bicycle 8212.12 12\n", + "14 gym membership 11243.32 13\n", + "15 headphones 6981.05 13\n", + "16 dinner 10573.27 14\n", + "17 groceries 10242.84 14\n", + "18 plane ticket 11890.28 15\n", + "19 hotel stay 13717.85 15" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Count transactions by item category\n", + "agg_query = AggregationQuery(\"*\") \\\n", + " .group_by(\n", + " \"@item_name\",\n", + " reducers.sum(\"amount\").alias(\"total_spent\"),\n", + " reducers.count().alias(\"count\")\n", + " ) \\\n", + " .sort_by(\"@count\", max=20)\n", + "\n", + "\n", + "results = index.aggregate(agg_query)\n", + "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", + "\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. User Transaction Features\n", + "\n", + "Let's analyze spending profiles by user. Probably most useful for feature store workloads" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
user_idavg_transaction_amounttransaction_countstdev_transaction_amount
0u_013882.147516423.525319582
\n", + "
" + ], + "text/plain": [ + " user_id avg_transaction_amount transaction_count stdev_transaction_amount\n", + "0 u_013 882.1475 16 423.525319582" + ] + }, + "execution_count": 35, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Let's look at some user spending features\n", + "\n", + "user_filter = Tag(\"user_id\") == \"u_013\"\n", + "\n", + "agg_query = AggregationQuery(str(user_filter)) \\\n", + " .group_by(\n", + " \"@user_id\",\n", + " reducers.avg(\"amount\").alias(\"avg_transaction_amount\"),\n", + " reducers.count().alias(\"transaction_count\"),\n", + " reducers.stddev(\"amount\").alias(\"stdev_transaction_amount\")\n", + " )\n", + "\n", + "\n", + "results = index.aggregate(agg_query)\n", + "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", + "\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's analyze a user's recent transactions to build a feature for fraud detection." + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idtimestamptransaction_idmerchant_iditem_nameamountcard_provider
0transactions:entry:txn_01211746312151txn_0121m_012Furniture1277.46AMEX
1transactions:entry:txn_00261746214951txn_0026m_016Clothing1166.55DISCOVER
2transactions:entry:txn_01021746168151txn_0102m_021Gaming Console697.55VISA
3transactions:entry:txn_00091745919751txn_0009m_022Ride Share259.34DISCOVER
4transactions:entry:txn_01981745797351txn_0198m_005Furniture1042.48AMEX
5transactions:entry:txn_01681745462551txn_0168m_014Groceries1450.52AMEX
6transactions:entry:txn_00821745228551txn_0082m_017Streaming Subscription1320DISCOVER
7transactions:entry:txn_01161744810951txn_0116m_024Clothing350.8DISCOVER
8transactions:entry:txn_00861744670551txn_0086m_005Ride Share528.52VISA
\n", + "
" + ], + "text/plain": [ + " id timestamp transaction_id merchant_id \\\n", + "0 transactions:entry:txn_0121 1746312151 txn_0121 m_012 \n", + "1 transactions:entry:txn_0026 1746214951 txn_0026 m_016 \n", + "2 transactions:entry:txn_0102 1746168151 txn_0102 m_021 \n", + "3 transactions:entry:txn_0009 1745919751 txn_0009 m_022 \n", + "4 transactions:entry:txn_0198 1745797351 txn_0198 m_005 \n", + "5 transactions:entry:txn_0168 1745462551 txn_0168 m_014 \n", + "6 transactions:entry:txn_0082 1745228551 txn_0082 m_017 \n", + "7 transactions:entry:txn_0116 1744810951 txn_0116 m_024 \n", + "8 transactions:entry:txn_0086 1744670551 txn_0086 m_005 \n", + "\n", + " item_name amount card_provider \n", + "0 Furniture 1277.46 AMEX \n", + "1 Clothing 1166.55 DISCOVER \n", + "2 Gaming Console 697.55 VISA \n", + "3 Ride Share 259.34 DISCOVER \n", + "4 Furniture 1042.48 AMEX \n", + "5 Groceries 1450.52 AMEX \n", + "6 Streaming Subscription 1320 DISCOVER \n", + "7 Clothing 350.8 DISCOVER \n", + "8 Ride Share 528.52 VISA " + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Example: Get recent transaction history for a user\n", + "user_id = \"u_016\"\n", + "user_filter = Tag(\"user_id\") == user_id\n", + "\n", + "# Regular search query for transactions, sorted by timestamp\n", + "query = FilterQuery(\n", + " return_fields=[\"transaction_id\", \"timestamp\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", + " filter_expression=user_filter,\n", + " num_results=10\n", + ").sort_by(\"timestamp\", asc=False)\n", + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
daydaily_transactionsdaily_total
017446705511528.52
117448109511350.8
2174522855111320
3174546255111450.52
4174579735111042.48
517459197511259.34
617461681511697.55
7174621495111166.55
8174631215111277.46
\n", + "
" + ], + "text/plain": [ + " day daily_transactions daily_total\n", + "0 1744670551 1 528.52\n", + "1 1744810951 1 350.8\n", + "2 1745228551 1 1320\n", + "3 1745462551 1 1450.52\n", + "4 1745797351 1 1042.48\n", + "5 1745919751 1 259.34\n", + "6 1746168151 1 697.55\n", + "7 1746214951 1 1166.55\n", + "8 1746312151 1 1277.46" + ] + }, + "execution_count": 37, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Calculate transaction frequency and spending patterns\n", + "user_filter = Tag(\"user_id\") == user_id\n", + "\n", + "# Using Redis aggregation functions to work with dates\n", + "agg_query = (\n", + " AggregationQuery(str(user_filter))\n", + " .load(\"@timestamp\")\n", + " .apply(ts=\"format('%s', @timestamp)\")\n", + " .apply(day=\"SUBSTR(@ts, 0, 10)\")\n", + " .group_by(\n", + " \"@day\",\n", + " reducers.count().alias(\"daily_transactions\"),\n", + " reducers.sum(\"amount\").alias(\"daily_total\")\n", + " )\n", + " .sort_by(\"@day\")\n", + ")\n", + "\n", + "results = index.aggregate(agg_query)\n", + "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", + "\n", + "pd.DataFrame(results)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's analyze transaction patterns by geographic location." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
merchant_idtransaction_counttransaction_totals
0m_0171352.43
1m_0111515.01
2m_0121588.93
3m_0222607.55
4m_0201835.78
5m_0212998.04
6m_00511000.78
7m_01921130.5
8m_00411420.3
9m_00931738.16
10m_00721788.59
11m_00251922.51
12m_01432060.54
13m_00332105.87
14m_01343129.86
15m_00843357.49
16m_02433394.42
17m_01654135.16
18m_00654463.77
19m_02344858.47
\n", + "
" + ], + "text/plain": [ + " merchant_id transaction_count transaction_totals\n", + "0 m_017 1 352.43\n", + "1 m_011 1 515.01\n", + "2 m_012 1 588.93\n", + "3 m_022 2 607.55\n", + "4 m_020 1 835.78\n", + "5 m_021 2 998.04\n", + "6 m_005 1 1000.78\n", + "7 m_019 2 1130.5\n", + "8 m_004 1 1420.3\n", + "9 m_009 3 1738.16\n", + "10 m_007 2 1788.59\n", + "11 m_002 5 1922.51\n", + "12 m_014 3 2060.54\n", + "13 m_003 3 2105.87\n", + "14 m_013 4 3129.86\n", + "15 m_008 4 3357.49\n", + "16 m_024 3 3394.42\n", + "17 m_016 5 4135.16\n", + "18 m_006 5 4463.77\n", + "19 m_023 4 4858.47" + ] + }, + "execution_count": 38, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Group transactions by latitude/longitude ranges to identify geographic clusters\n", + "# For simplicity, we'll round lat/lon to the nearest whole number and group\n", + "from redisvl.query.filter import GeoRadius, Geo\n", + "\n", + "\n", + "geo_filter = Geo(\"location\") == GeoRadius(-71.730465, 27.584806, 1000, \"mi\")\n", + "\n", + "agg_query = (\n", + " AggregationQuery(str(geo_filter))\n", + " .group_by(\n", + " \"@merchant_id\", \n", + " reducers.count().alias(\"transaction_count\"),\n", + " reducers.sum(\"amount\").alias(\"transaction_totals\")\n", + " )\n", + " .sort_by(\"@transaction_totals\", max=20)\n", + ")\n", + "\n", + "results = index.aggregate(agg_query)\n", + "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", + "\n", + "pd.DataFrame(results)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Cleaning Up Redis Resources\n", + "\n", + "When you're done, it's good practice to clean up your Redis resources." + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 39, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Clean up by deleting the index\n", + "# Uncomment the line below when you're ready to delete the index\n", + "index.delete(drop=True)\n", + "client.flushall()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "In this notebook, we've demonstrated how to:\n", + "\n", + "1. Load transaction data into Redis\n", + "2. Create vector embeddings for semantic search\n", + "3. Perform various search operations:\n", + " - Exact match filtering\n", + " - Vector similarity search\n", + " - Full text search\n", + " - Using search patterns for record linkage tasks\n", + "4. Execute complex aggregation queries\n", + " - Analyzing spending patterns by user\n", + " - Looking at transaction volumes over time\n", + " - Analyzing geographic transaction patterns\n", + "\n", + "These capabilities make Redis and RedisVL powerful tools for building real-time feature stores that can support fraud detection, personalization, and analytics applications." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.2" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/python-recipes/feature-store/resources/transactions_200.json b/python-recipes/feature-store/resources/transactions_200.json new file mode 100644 index 00000000..22264ef5 --- /dev/null +++ b/python-recipes/feature-store/resources/transactions_200.json @@ -0,0 +1,2402 @@ +[ + { + "transaction_id": "txn_0001", + "user_id": "u_002", + "merchant_id": "m_009", + "item_name": "Headphones", + "amount": 1154.59, + "currency": "USD", + "timestamp": "2025-05-02T10:42:31Z", + "lat": 27.584806, + "lon": -71.730465, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0002", + "user_id": "u_013", + "merchant_id": "m_018", + "item_name": "Dinner", + "amount": 501.64, + "currency": "USD", + "timestamp": "2025-05-11T13:42:31Z", + "lat": 28.831898, + "lon": -104.441434, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0003", + "user_id": "u_008", + "merchant_id": "m_006", + "item_name": "Laptop", + "amount": 1359.33, + "currency": "USD", + "timestamp": "2025-05-10T01:42:31Z", + "lat": 46.087128, + "lon": -102.099503, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0004", + "user_id": "u_011", + "merchant_id": "m_024", + "item_name": "Gaming Console", + "amount": 157.54, + "currency": "USD", + "timestamp": "2025-05-11T22:42:31Z", + "lat": 27.226349, + "lon": -115.753846, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0005", + "user_id": "u_010", + "merchant_id": "m_014", + "item_name": "Concert Ticket", + "amount": 718.0, + "currency": "USD", + "timestamp": "2025-04-23T18:42:31Z", + "lat": 45.108103, + "lon": -79.409905, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0006", + "user_id": "u_017", + "merchant_id": "m_016", + "item_name": "Hotel Stay", + "amount": 1232.8, + "currency": "USD", + "timestamp": "2025-04-16T12:42:31Z", + "lat": 37.649799, + "lon": -77.058366, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0007", + "user_id": "u_001", + "merchant_id": "m_013", + "item_name": "Bicycle", + "amount": 654.41, + "currency": "USD", + "timestamp": "2025-04-20T19:42:31Z", + "lat": 40.320478, + "lon": -85.333857, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0008", + "user_id": "u_017", + "merchant_id": "m_022", + "item_name": "Shoes", + "amount": 564.91, + "currency": "USD", + "timestamp": "2025-05-12T10:42:31Z", + "lat": 45.336203, + "lon": -86.563822, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0009", + "user_id": "u_016", + "merchant_id": "m_022", + "item_name": "Ride Share", + "amount": 259.34, + "currency": "USD", + "timestamp": "2025-04-29T09:42:31Z", + "lat": 38.300888, + "lon": -72.732838, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0010", + "user_id": "u_013", + "merchant_id": "m_019", + "item_name": "Book", + "amount": 29.38, + "currency": "USD", + "timestamp": "2025-04-18T18:42:31Z", + "lat": 27.115775, + "lon": -87.557584, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0011", + "user_id": "u_012", + "merchant_id": "m_008", + "item_name": "Clothing", + "amount": 800.34, + "currency": "USD", + "timestamp": "2025-05-11T05:42:31Z", + "lat": 40.33122, + "lon": -66.476, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0012", + "user_id": "u_018", + "merchant_id": "m_016", + "item_name": "Plane Ticket", + "amount": 234.87, + "currency": "USD", + "timestamp": "2025-04-20T21:42:31Z", + "lat": 35.667998, + "lon": -80.404291, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0013", + "user_id": "u_001", + "merchant_id": "m_005", + "item_name": "Gaming Console", + "amount": 293.8, + "currency": "USD", + "timestamp": "2025-05-02T20:42:31Z", + "lat": 44.173333, + "lon": -84.271951, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0014", + "user_id": "u_014", + "merchant_id": "m_001", + "item_name": "Coffee", + "amount": 928.29, + "currency": "USD", + "timestamp": "2025-04-22T00:42:31Z", + "lat": 33.521475, + "lon": -118.850244, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0015", + "user_id": "u_001", + "merchant_id": "m_018", + "item_name": "Clothing", + "amount": 114.86, + "currency": "USD", + "timestamp": "2025-04-14T04:42:31Z", + "lat": 40.242235, + "lon": -122.310061, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0016", + "user_id": "u_009", + "merchant_id": "m_016", + "item_name": "Smartphone", + "amount": 1127.47, + "currency": "USD", + "timestamp": "2025-04-29T12:42:31Z", + "lat": 34.937718, + "lon": -122.620049, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0017", + "user_id": "u_014", + "merchant_id": "m_003", + "item_name": "Groceries", + "amount": 561.3, + "currency": "USD", + "timestamp": "2025-05-03T18:42:31Z", + "lat": 42.858736, + "lon": -76.164098, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0018", + "user_id": "u_017", + "merchant_id": "m_013", + "item_name": "Dinner", + "amount": 462.71, + "currency": "USD", + "timestamp": "2025-04-15T19:42:31Z", + "lat": 43.538835, + "lon": -107.110697, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0019", + "user_id": "u_012", + "merchant_id": "m_005", + "item_name": "Dinner", + "amount": 1462.52, + "currency": "USD", + "timestamp": "2025-04-21T17:42:31Z", + "lat": 27.86959, + "lon": -114.3255, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0020", + "user_id": "u_014", + "merchant_id": "m_002", + "item_name": "Groceries", + "amount": 684.84, + "currency": "USD", + "timestamp": "2025-05-12T21:42:31Z", + "lat": 33.388952, + "lon": -73.477522, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0021", + "user_id": "u_010", + "merchant_id": "m_014", + "item_name": "Laptop", + "amount": 242.07, + "currency": "USD", + "timestamp": "2025-04-29T22:42:31Z", + "lat": 45.913472, + "lon": -94.14629, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0022", + "user_id": "u_011", + "merchant_id": "m_017", + "item_name": "Coffee", + "amount": 820.46, + "currency": "USD", + "timestamp": "2025-04-28T01:42:31Z", + "lat": 30.397703, + "lon": -90.929251, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0023", + "user_id": "u_009", + "merchant_id": "m_017", + "item_name": "Streaming Subscription", + "amount": 308.89, + "currency": "USD", + "timestamp": "2025-04-30T01:42:31Z", + "lat": 31.349166, + "lon": -121.504568, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0024", + "user_id": "u_007", + "merchant_id": "m_017", + "item_name": "Hotel Stay", + "amount": 1138.38, + "currency": "USD", + "timestamp": "2025-05-13T01:42:31Z", + "lat": 29.659919, + "lon": -91.25178, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0025", + "user_id": "u_020", + "merchant_id": "m_011", + "item_name": "Headphones", + "amount": 515.01, + "currency": "USD", + "timestamp": "2025-04-20T20:42:31Z", + "lat": 28.203223, + "lon": -72.35043, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0026", + "user_id": "u_016", + "merchant_id": "m_016", + "item_name": "Clothing", + "amount": 1166.55, + "currency": "USD", + "timestamp": "2025-05-02T19:42:31Z", + "lat": 42.130318, + "lon": -75.962909, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0027", + "user_id": "u_011", + "merchant_id": "m_013", + "item_name": "Streaming Subscription", + "amount": 1261.31, + "currency": "USD", + "timestamp": "2025-05-08T09:42:31Z", + "lat": 39.634021, + "lon": -123.654283, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0028", + "user_id": "u_011", + "merchant_id": "m_023", + "item_name": "Concert Ticket", + "amount": 187.86, + "currency": "USD", + "timestamp": "2025-05-11T13:42:31Z", + "lat": 33.655497, + "lon": -97.275834, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0029", + "user_id": "u_011", + "merchant_id": "m_012", + "item_name": "Hotel Stay", + "amount": 1156.58, + "currency": "USD", + "timestamp": "2025-04-29T03:42:31Z", + "lat": 44.475051, + "lon": -112.836735, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0030", + "user_id": "u_013", + "merchant_id": "m_002", + "item_name": "Shoes", + "amount": 566.13, + "currency": "USD", + "timestamp": "2025-05-02T11:42:31Z", + "lat": 44.904586, + "lon": -113.857037, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0031", + "user_id": "u_012", + "merchant_id": "m_003", + "item_name": "Gym Membership", + "amount": 187.88, + "currency": "USD", + "timestamp": "2025-05-02T11:42:31Z", + "lat": 37.06791, + "lon": -88.883813, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0032", + "user_id": "u_013", + "merchant_id": "m_002", + "item_name": "Concert Ticket", + "amount": 585.69, + "currency": "USD", + "timestamp": "2025-05-09T04:42:31Z", + "lat": 36.173155, + "lon": -79.595479, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0033", + "user_id": "u_008", + "merchant_id": "m_021", + "item_name": "Gaming Console", + "amount": 402.54, + "currency": "USD", + "timestamp": "2025-04-25T01:42:31Z", + "lat": 38.506396, + "lon": -104.044904, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0034", + "user_id": "u_017", + "merchant_id": "m_019", + "item_name": "Ride Share", + "amount": 509.08, + "currency": "USD", + "timestamp": "2025-05-11T17:42:31Z", + "lat": 29.37294, + "lon": -122.845091, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0035", + "user_id": "u_005", + "merchant_id": "m_016", + "item_name": "Dinner", + "amount": 68.93, + "currency": "USD", + "timestamp": "2025-04-20T16:42:31Z", + "lat": 28.397242, + "lon": -73.512899, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0036", + "user_id": "u_020", + "merchant_id": "m_002", + "item_name": "Gaming Console", + "amount": 758.65, + "currency": "USD", + "timestamp": "2025-04-18T03:42:31Z", + "lat": 43.583139, + "lon": -107.254806, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0037", + "user_id": "u_011", + "merchant_id": "m_024", + "item_name": "Clothing", + "amount": 345.23, + "currency": "USD", + "timestamp": "2025-05-11T02:42:31Z", + "lat": 45.918092, + "lon": -107.464638, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0038", + "user_id": "u_012", + "merchant_id": "m_014", + "item_name": "Smartphone", + "amount": 912.55, + "currency": "USD", + "timestamp": "2025-04-22T14:42:31Z", + "lat": 41.170477, + "lon": -112.502033, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0039", + "user_id": "u_017", + "merchant_id": "m_006", + "item_name": "Bicycle", + "amount": 1260.26, + "currency": "USD", + "timestamp": "2025-04-30T20:42:31Z", + "lat": 36.996129, + "lon": -73.176784, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0040", + "user_id": "u_010", + "merchant_id": "m_017", + "item_name": "Headphones", + "amount": 159.33, + "currency": "USD", + "timestamp": "2025-04-17T15:42:31Z", + "lat": 38.079073, + "lon": -107.800637, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0041", + "user_id": "u_017", + "merchant_id": "m_003", + "item_name": "Clothing", + "amount": 612.59, + "currency": "USD", + "timestamp": "2025-04-24T15:42:31Z", + "lat": 27.662753, + "lon": -90.939557, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0042", + "user_id": "u_017", + "merchant_id": "m_012", + "item_name": "Groceries", + "amount": 742.36, + "currency": "USD", + "timestamp": "2025-05-05T11:42:31Z", + "lat": 47.310652, + "lon": -67.337246, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0043", + "user_id": "u_008", + "merchant_id": "m_007", + "item_name": "Headphones", + "amount": 608.46, + "currency": "USD", + "timestamp": "2025-05-12T16:42:31Z", + "lat": 25.262947, + "lon": -85.93962, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0044", + "user_id": "u_002", + "merchant_id": "m_008", + "item_name": "Shoes", + "amount": 1049.74, + "currency": "USD", + "timestamp": "2025-05-05T20:42:31Z", + "lat": 32.300431, + "lon": -107.612217, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0045", + "user_id": "u_007", + "merchant_id": "m_017", + "item_name": "Laptop", + "amount": 1124.25, + "currency": "USD", + "timestamp": "2025-05-07T06:42:31Z", + "lat": 36.181491, + "lon": -99.971276, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0046", + "user_id": "u_013", + "merchant_id": "m_018", + "item_name": "Hotel Stay", + "amount": 588.18, + "currency": "USD", + "timestamp": "2025-04-22T14:42:31Z", + "lat": 45.842571, + "lon": -107.366423, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0047", + "user_id": "u_004", + "merchant_id": "m_005", + "item_name": "Coffee", + "amount": 692.86, + "currency": "USD", + "timestamp": "2025-04-28T17:42:31Z", + "lat": 38.243205, + "lon": -122.989491, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0048", + "user_id": "u_010", + "merchant_id": "m_012", + "item_name": "Hotel Stay", + "amount": 588.93, + "currency": "USD", + "timestamp": "2025-05-11T18:42:31Z", + "lat": 27.537447, + "lon": -75.877372, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0049", + "user_id": "u_019", + "merchant_id": "m_022", + "item_name": "Book", + "amount": 1105.52, + "currency": "USD", + "timestamp": "2025-04-15T06:42:31Z", + "lat": 29.942826, + "lon": -101.711559, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0050", + "user_id": "u_005", + "merchant_id": "m_016", + "item_name": "Groceries", + "amount": 1277.78, + "currency": "USD", + "timestamp": "2025-05-13T13:42:31Z", + "lat": 28.901584, + "lon": -77.946765, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0051", + "user_id": "u_002", + "merchant_id": "m_004", + "item_name": "Groceries", + "amount": 332.51, + "currency": "USD", + "timestamp": "2025-04-25T10:42:31Z", + "lat": 47.487041, + "lon": -85.644273, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0052", + "user_id": "u_002", + "merchant_id": "m_005", + "item_name": "Coffee", + "amount": 694.7, + "currency": "USD", + "timestamp": "2025-05-09T22:42:31Z", + "lat": 28.223914, + "lon": -121.611648, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0053", + "user_id": "u_007", + "merchant_id": "m_008", + "item_name": "Streaming Subscription", + "amount": 1178.24, + "currency": "USD", + "timestamp": "2025-04-29T05:42:31Z", + "lat": 41.118064, + "lon": -118.814622, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0054", + "user_id": "u_010", + "merchant_id": "m_011", + "item_name": "Clothing", + "amount": 521.6, + "currency": "USD", + "timestamp": "2025-05-04T00:42:31Z", + "lat": 44.551653, + "lon": -120.638602, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0055", + "user_id": "u_004", + "merchant_id": "m_008", + "item_name": "Gym Membership", + "amount": 355.54, + "currency": "USD", + "timestamp": "2025-04-20T00:42:31Z", + "lat": 44.022969, + "lon": -78.470695, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0056", + "user_id": "u_014", + "merchant_id": "m_018", + "item_name": "Furniture", + "amount": 559.06, + "currency": "USD", + "timestamp": "2025-05-08T05:42:31Z", + "lat": 45.316075, + "lon": -73.144982, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0057", + "user_id": "u_012", + "merchant_id": "m_011", + "item_name": "Laptop", + "amount": 1201.71, + "currency": "USD", + "timestamp": "2025-04-22T04:42:31Z", + "lat": 43.212439, + "lon": -86.496966, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0058", + "user_id": "u_013", + "merchant_id": "m_024", + "item_name": "Plane Ticket", + "amount": 916.96, + "currency": "USD", + "timestamp": "2025-04-24T19:42:31Z", + "lat": 45.642569, + "lon": -81.460941, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0059", + "user_id": "u_013", + "merchant_id": "m_021", + "item_name": "Headphones", + "amount": 291.56, + "currency": "USD", + "timestamp": "2025-04-28T12:42:31Z", + "lat": 35.256912, + "lon": -96.109774, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0060", + "user_id": "u_013", + "merchant_id": "m_001", + "item_name": "Ride Share", + "amount": 1094.44, + "currency": "USD", + "timestamp": "2025-04-17T02:42:31Z", + "lat": 41.139183, + "lon": -78.754213, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0061", + "user_id": "u_006", + "merchant_id": "m_004", + "item_name": "Groceries", + "amount": 1499.87, + "currency": "USD", + "timestamp": "2025-04-17T23:42:31Z", + "lat": 27.433476, + "lon": -116.967174, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0062", + "user_id": "u_012", + "merchant_id": "m_002", + "item_name": "Ride Share", + "amount": 544.92, + "currency": "USD", + "timestamp": "2025-05-03T00:42:31Z", + "lat": 34.857944, + "lon": -87.97746, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0063", + "user_id": "u_018", + "merchant_id": "m_004", + "item_name": "Book", + "amount": 433.91, + "currency": "USD", + "timestamp": "2025-04-16T13:42:31Z", + "lat": 45.794633, + "lon": -66.991026, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0064", + "user_id": "u_015", + "merchant_id": "m_008", + "item_name": "Laptop", + "amount": 144.89, + "currency": "USD", + "timestamp": "2025-05-08T23:42:31Z", + "lat": 39.863625, + "lon": -69.926886, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0065", + "user_id": "u_013", + "merchant_id": "m_012", + "item_name": "Plane Ticket", + "amount": 1231.32, + "currency": "USD", + "timestamp": "2025-05-08T03:42:31Z", + "lat": 34.297999, + "lon": -99.90915, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0066", + "user_id": "u_001", + "merchant_id": "m_014", + "item_name": "Groceries", + "amount": 1205.4, + "currency": "USD", + "timestamp": "2025-05-10T05:42:31Z", + "lat": 29.191642, + "lon": -108.53245, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0067", + "user_id": "u_013", + "merchant_id": "m_010", + "item_name": "Headphones", + "amount": 1311.19, + "currency": "USD", + "timestamp": "2025-05-08T14:42:31Z", + "lat": 43.989609, + "lon": -101.166057, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0068", + "user_id": "u_018", + "merchant_id": "m_023", + "item_name": "Hotel Stay", + "amount": 1313.17, + "currency": "USD", + "timestamp": "2025-04-30T12:42:31Z", + "lat": 39.338714, + "lon": -78.511394, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0069", + "user_id": "u_001", + "merchant_id": "m_022", + "item_name": "Groceries", + "amount": 465.63, + "currency": "USD", + "timestamp": "2025-05-05T06:42:31Z", + "lat": 35.247421, + "lon": -95.145348, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0070", + "user_id": "u_003", + "merchant_id": "m_005", + "item_name": "Plane Ticket", + "amount": 1000.78, + "currency": "USD", + "timestamp": "2025-04-20T07:42:31Z", + "lat": 30.33704, + "lon": -75.901229, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0071", + "user_id": "u_019", + "merchant_id": "m_007", + "item_name": "Clothing", + "amount": 856.17, + "currency": "USD", + "timestamp": "2025-04-25T06:42:31Z", + "lat": 42.067385, + "lon": -69.542388, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0072", + "user_id": "u_007", + "merchant_id": "m_003", + "item_name": "Gaming Console", + "amount": 68.88, + "currency": "USD", + "timestamp": "2025-04-20T23:42:31Z", + "lat": 47.219621, + "lon": -83.324229, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0073", + "user_id": "u_002", + "merchant_id": "m_005", + "item_name": "Bicycle", + "amount": 152.42, + "currency": "USD", + "timestamp": "2025-05-11T15:42:31Z", + "lat": 45.098806, + "lon": -77.76401, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0074", + "user_id": "u_020", + "merchant_id": "m_013", + "item_name": "Book", + "amount": 994.55, + "currency": "USD", + "timestamp": "2025-04-16T08:42:31Z", + "lat": 40.788461, + "lon": -67.492872, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0075", + "user_id": "u_003", + "merchant_id": "m_002", + "item_name": "Hotel Stay", + "amount": 232.59, + "currency": "USD", + "timestamp": "2025-04-16T07:42:31Z", + "lat": 28.262526, + "lon": -83.05635, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0076", + "user_id": "u_012", + "merchant_id": "m_017", + "item_name": "Shoes", + "amount": 1060.16, + "currency": "USD", + "timestamp": "2025-05-01T16:42:31Z", + "lat": 34.863386, + "lon": -113.326341, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0077", + "user_id": "u_001", + "merchant_id": "m_019", + "item_name": "Software License", + "amount": 1065.32, + "currency": "USD", + "timestamp": "2025-04-21T14:42:31Z", + "lat": 43.213399, + "lon": -69.274885, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0078", + "user_id": "u_015", + "merchant_id": "m_021", + "item_name": "Streaming Subscription", + "amount": 942.25, + "currency": "USD", + "timestamp": "2025-04-20T06:42:31Z", + "lat": 36.505616, + "lon": -102.44351, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0079", + "user_id": "u_012", + "merchant_id": "m_021", + "item_name": "Gym Membership", + "amount": 1251.08, + "currency": "USD", + "timestamp": "2025-04-19T18:42:31Z", + "lat": 48.944844, + "lon": -86.169174, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0080", + "user_id": "u_007", + "merchant_id": "m_006", + "item_name": "Smartphone", + "amount": 565.47, + "currency": "USD", + "timestamp": "2025-04-18T19:42:31Z", + "lat": 27.323892, + "lon": -77.47086, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0081", + "user_id": "u_006", + "merchant_id": "m_004", + "item_name": "Software License", + "amount": 113.89, + "currency": "USD", + "timestamp": "2025-04-19T07:42:31Z", + "lat": 34.208669, + "lon": -116.123841, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0082", + "user_id": "u_016", + "merchant_id": "m_017", + "item_name": "Streaming Subscription", + "amount": 1320.0, + "currency": "USD", + "timestamp": "2025-04-21T09:42:31Z", + "lat": 36.940728, + "lon": -105.7124, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0083", + "user_id": "u_009", + "merchant_id": "m_011", + "item_name": "Camera", + "amount": 268.8, + "currency": "USD", + "timestamp": "2025-05-09T10:42:31Z", + "lat": 42.4911, + "lon": -75.243374, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0084", + "user_id": "u_012", + "merchant_id": "m_002", + "item_name": "Groceries", + "amount": 225.02, + "currency": "USD", + "timestamp": "2025-04-16T02:42:31Z", + "lat": 29.013773, + "lon": -84.218267, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0085", + "user_id": "u_018", + "merchant_id": "m_008", + "item_name": "Bicycle", + "amount": 1429.46, + "currency": "USD", + "timestamp": "2025-05-05T23:42:31Z", + "lat": 43.52706, + "lon": -95.817588, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0086", + "user_id": "u_016", + "merchant_id": "m_005", + "item_name": "Ride Share", + "amount": 528.52, + "currency": "USD", + "timestamp": "2025-04-14T22:42:31Z", + "lat": 37.11829, + "lon": -98.551549, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0087", + "user_id": "u_014", + "merchant_id": "m_024", + "item_name": "Camera", + "amount": 597.48, + "currency": "USD", + "timestamp": "2025-05-11T18:42:31Z", + "lat": 40.308861, + "lon": -73.842162, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0088", + "user_id": "u_011", + "merchant_id": "m_015", + "item_name": "Gaming Console", + "amount": 26.63, + "currency": "USD", + "timestamp": "2025-05-12T04:42:31Z", + "lat": 42.128628, + "lon": -103.544356, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0089", + "user_id": "u_008", + "merchant_id": "m_023", + "item_name": "Dinner", + "amount": 1043.94, + "currency": "USD", + "timestamp": "2025-04-19T19:42:31Z", + "lat": 36.426513, + "lon": -79.418672, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0090", + "user_id": "u_008", + "merchant_id": "m_002", + "item_name": "Plane Ticket", + "amount": 194.37, + "currency": "USD", + "timestamp": "2025-04-24T15:42:31Z", + "lat": 29.855991, + "lon": -84.724093, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0091", + "user_id": "u_014", + "merchant_id": "m_021", + "item_name": "Laptop", + "amount": 525.54, + "currency": "USD", + "timestamp": "2025-04-16T02:42:31Z", + "lat": 47.299312, + "lon": -108.388258, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0092", + "user_id": "u_018", + "merchant_id": "m_010", + "item_name": "Headphones", + "amount": 522.26, + "currency": "USD", + "timestamp": "2025-05-11T18:42:31Z", + "lat": 27.834422, + "lon": -116.544803, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0093", + "user_id": "u_020", + "merchant_id": "m_016", + "item_name": "Bicycle", + "amount": 633.55, + "currency": "USD", + "timestamp": "2025-04-22T11:42:31Z", + "lat": 26.538749, + "lon": -113.070724, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0094", + "user_id": "u_008", + "merchant_id": "m_011", + "item_name": "Shoes", + "amount": 1106.98, + "currency": "USD", + "timestamp": "2025-05-08T19:42:31Z", + "lat": 34.502058, + "lon": -103.173666, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0095", + "user_id": "u_004", + "merchant_id": "m_007", + "item_name": "Ride Share", + "amount": 139.01, + "currency": "USD", + "timestamp": "2025-05-08T15:42:31Z", + "lat": 33.836531, + "lon": -112.978767, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0096", + "user_id": "u_014", + "merchant_id": "m_021", + "item_name": "Gaming Console", + "amount": 1393.99, + "currency": "USD", + "timestamp": "2025-04-25T03:42:31Z", + "lat": 47.788627, + "lon": -82.551373, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0097", + "user_id": "u_002", + "merchant_id": "m_004", + "item_name": "Gym Membership", + "amount": 1420.3, + "currency": "USD", + "timestamp": "2025-04-27T00:42:31Z", + "lat": 34.034171, + "lon": -75.039266, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0098", + "user_id": "u_012", + "merchant_id": "m_012", + "item_name": "Headphones", + "amount": 503.01, + "currency": "USD", + "timestamp": "2025-05-05T21:42:31Z", + "lat": 27.906234, + "lon": -98.225217, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0099", + "user_id": "u_002", + "merchant_id": "m_007", + "item_name": "Gym Membership", + "amount": 477.54, + "currency": "USD", + "timestamp": "2025-05-01T13:42:31Z", + "lat": 28.640963, + "lon": -88.91177, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0100", + "user_id": "u_008", + "merchant_id": "m_018", + "item_name": "Smartphone", + "amount": 925.37, + "currency": "USD", + "timestamp": "2025-05-07T03:42:31Z", + "lat": 29.991755, + "lon": -110.038073, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0101", + "user_id": "u_018", + "merchant_id": "m_008", + "item_name": "Gym Membership", + "amount": 1366.36, + "currency": "USD", + "timestamp": "2025-05-06T20:42:31Z", + "lat": 29.725257, + "lon": -96.026107, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0102", + "user_id": "u_016", + "merchant_id": "m_021", + "item_name": "Gaming Console", + "amount": 697.55, + "currency": "USD", + "timestamp": "2025-05-02T06:42:31Z", + "lat": 35.472585, + "lon": -71.016793, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0103", + "user_id": "u_013", + "merchant_id": "m_015", + "item_name": "Coffee", + "amount": 1360.17, + "currency": "USD", + "timestamp": "2025-04-17T17:42:31Z", + "lat": 45.263733, + "lon": -101.799395, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0104", + "user_id": "u_008", + "merchant_id": "m_002", + "item_name": "Plane Ticket", + "amount": 1045.76, + "currency": "USD", + "timestamp": "2025-04-22T01:42:31Z", + "lat": 35.659498, + "lon": -88.648183, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0105", + "user_id": "u_010", + "merchant_id": "m_006", + "item_name": "Bicycle", + "amount": 359.98, + "currency": "USD", + "timestamp": "2025-04-14T00:42:31Z", + "lat": 35.013838, + "lon": -81.283235, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0106", + "user_id": "u_019", + "merchant_id": "m_013", + "item_name": "Dinner", + "amount": 99.45, + "currency": "USD", + "timestamp": "2025-04-23T22:42:31Z", + "lat": 41.661238, + "lon": -69.636419, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0107", + "user_id": "u_012", + "merchant_id": "m_023", + "item_name": "Concert Ticket", + "amount": 196.89, + "currency": "USD", + "timestamp": "2025-04-30T19:42:31Z", + "lat": 34.982872, + "lon": -114.702068, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0108", + "user_id": "u_020", + "merchant_id": "m_008", + "item_name": "Software License", + "amount": 1225.65, + "currency": "USD", + "timestamp": "2025-05-01T03:42:31Z", + "lat": 29.590469, + "lon": -89.543112, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0109", + "user_id": "u_007", + "merchant_id": "m_020", + "item_name": "Gaming Console", + "amount": 43.49, + "currency": "USD", + "timestamp": "2025-05-07T00:42:31Z", + "lat": 43.520622, + "lon": -87.8863, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0110", + "user_id": "u_011", + "merchant_id": "m_002", + "item_name": "Coffee", + "amount": 368.85, + "currency": "USD", + "timestamp": "2025-04-28T23:42:31Z", + "lat": 30.172612, + "lon": -113.864311, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0111", + "user_id": "u_014", + "merchant_id": "m_006", + "item_name": "Coffee", + "amount": 1471.73, + "currency": "USD", + "timestamp": "2025-04-21T10:42:31Z", + "lat": 26.217393, + "lon": -85.889865, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0112", + "user_id": "u_014", + "merchant_id": "m_005", + "item_name": "Streaming Subscription", + "amount": 1009.49, + "currency": "USD", + "timestamp": "2025-05-06T12:42:31Z", + "lat": 35.813565, + "lon": -116.584942, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0113", + "user_id": "u_013", + "merchant_id": "m_001", + "item_name": "Software License", + "amount": 733.8, + "currency": "USD", + "timestamp": "2025-04-25T07:42:31Z", + "lat": 35.364947, + "lon": -89.141154, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0114", + "user_id": "u_007", + "merchant_id": "m_008", + "item_name": "Gym Membership", + "amount": 1117.29, + "currency": "USD", + "timestamp": "2025-04-24T23:42:31Z", + "lat": 37.138476, + "lon": -66.617477, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0115", + "user_id": "u_006", + "merchant_id": "m_017", + "item_name": "Plane Ticket", + "amount": 352.43, + "currency": "USD", + "timestamp": "2025-05-08T17:42:31Z", + "lat": 27.266694, + "lon": -87.567325, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0116", + "user_id": "u_016", + "merchant_id": "m_024", + "item_name": "Clothing", + "amount": 350.8, + "currency": "USD", + "timestamp": "2025-04-16T13:42:31Z", + "lat": 44.216702, + "lon": -78.011064, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0117", + "user_id": "u_003", + "merchant_id": "m_009", + "item_name": "Ride Share", + "amount": 844.69, + "currency": "USD", + "timestamp": "2025-05-02T10:42:31Z", + "lat": 30.665674, + "lon": -115.214701, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0118", + "user_id": "u_007", + "merchant_id": "m_001", + "item_name": "Headphones", + "amount": 618.03, + "currency": "USD", + "timestamp": "2025-04-30T21:42:31Z", + "lat": 48.450893, + "lon": -119.881776, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0119", + "user_id": "u_011", + "merchant_id": "m_017", + "item_name": "Headphones", + "amount": 656.92, + "currency": "USD", + "timestamp": "2025-04-30T13:42:31Z", + "lat": 27.300297, + "lon": -97.848452, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0120", + "user_id": "u_005", + "merchant_id": "m_009", + "item_name": "Book", + "amount": 195.24, + "currency": "USD", + "timestamp": "2025-04-23T20:42:31Z", + "lat": 45.416751, + "lon": -117.251359, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0121", + "user_id": "u_016", + "merchant_id": "m_012", + "item_name": "Furniture", + "amount": 1277.46, + "currency": "USD", + "timestamp": "2025-05-03T22:42:31Z", + "lat": 48.388609, + "lon": -68.319947, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0122", + "user_id": "u_009", + "merchant_id": "m_019", + "item_name": "Bicycle", + "amount": 142.94, + "currency": "USD", + "timestamp": "2025-05-13T09:42:31Z", + "lat": 40.454142, + "lon": -103.888266, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0123", + "user_id": "u_015", + "merchant_id": "m_013", + "item_name": "Groceries", + "amount": 271.69, + "currency": "USD", + "timestamp": "2025-05-06T03:42:31Z", + "lat": 40.56738, + "lon": -104.724346, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0124", + "user_id": "u_005", + "merchant_id": "m_022", + "item_name": "Dinner", + "amount": 369.56, + "currency": "USD", + "timestamp": "2025-04-25T12:42:31Z", + "lat": 33.925645, + "lon": -114.66281, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0125", + "user_id": "u_013", + "merchant_id": "m_018", + "item_name": "Shoes", + "amount": 1032.48, + "currency": "USD", + "timestamp": "2025-05-13T13:42:31Z", + "lat": 41.964321, + "lon": -110.604425, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0126", + "user_id": "u_008", + "merchant_id": "m_022", + "item_name": "Software License", + "amount": 348.21, + "currency": "USD", + "timestamp": "2025-04-18T05:42:31Z", + "lat": 33.253319, + "lon": -77.253532, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0127", + "user_id": "u_001", + "merchant_id": "m_021", + "item_name": "Gaming Console", + "amount": 508.48, + "currency": "USD", + "timestamp": "2025-04-15T06:42:31Z", + "lat": 26.29976, + "lon": -122.674529, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0128", + "user_id": "u_009", + "merchant_id": "m_006", + "item_name": "Camera", + "amount": 277.54, + "currency": "USD", + "timestamp": "2025-04-15T21:42:31Z", + "lat": 46.908017, + "lon": -109.966729, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0129", + "user_id": "u_006", + "merchant_id": "m_021", + "item_name": "Ride Share", + "amount": 1175.93, + "currency": "USD", + "timestamp": "2025-04-20T10:42:31Z", + "lat": 36.385901, + "lon": -101.862841, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0130", + "user_id": "u_009", + "merchant_id": "m_023", + "item_name": "Book", + "amount": 1191.61, + "currency": "USD", + "timestamp": "2025-05-10T14:42:31Z", + "lat": 30.572501, + "lon": -79.398727, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0131", + "user_id": "u_003", + "merchant_id": "m_012", + "item_name": "Plane Ticket", + "amount": 1446.89, + "currency": "USD", + "timestamp": "2025-05-01T22:42:31Z", + "lat": 47.576226, + "lon": -97.47794, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0132", + "user_id": "u_020", + "merchant_id": "m_002", + "item_name": "Smartphone", + "amount": 362.56, + "currency": "USD", + "timestamp": "2025-04-23T23:42:31Z", + "lat": 43.443214, + "lon": -110.714027, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0133", + "user_id": "u_002", + "merchant_id": "m_020", + "item_name": "Software License", + "amount": 1369.37, + "currency": "USD", + "timestamp": "2025-04-27T20:42:31Z", + "lat": 48.443181, + "lon": -116.386725, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0134", + "user_id": "u_008", + "merchant_id": "m_001", + "item_name": "Headphones", + "amount": 113.24, + "currency": "USD", + "timestamp": "2025-04-20T20:42:31Z", + "lat": 37.099213, + "lon": -92.823366, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0135", + "user_id": "u_010", + "merchant_id": "m_021", + "item_name": "Hotel Stay", + "amount": 489.81, + "currency": "USD", + "timestamp": "2025-04-27T01:42:31Z", + "lat": 44.339275, + "lon": -86.365949, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0136", + "user_id": "u_007", + "merchant_id": "m_002", + "item_name": "Bicycle", + "amount": 1323.15, + "currency": "USD", + "timestamp": "2025-04-17T01:42:31Z", + "lat": 45.744648, + "lon": -107.780895, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0137", + "user_id": "u_010", + "merchant_id": "m_005", + "item_name": "Dinner", + "amount": 1229.72, + "currency": "USD", + "timestamp": "2025-04-30T02:42:31Z", + "lat": 28.033281, + "lon": -115.147526, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0138", + "user_id": "u_008", + "merchant_id": "m_012", + "item_name": "Coffee", + "amount": 927.33, + "currency": "USD", + "timestamp": "2025-05-11T09:42:31Z", + "lat": 42.097804, + "lon": -122.800478, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0139", + "user_id": "u_020", + "merchant_id": "m_013", + "item_name": "Book", + "amount": 985.31, + "currency": "USD", + "timestamp": "2025-05-06T11:42:31Z", + "lat": 40.248008, + "lon": -76.23493, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0140", + "user_id": "u_014", + "merchant_id": "m_008", + "item_name": "Gaming Console", + "amount": 884.5, + "currency": "USD", + "timestamp": "2025-04-30T13:42:31Z", + "lat": 29.864244, + "lon": -104.826713, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0141", + "user_id": "u_019", + "merchant_id": "m_023", + "item_name": "Ride Share", + "amount": 1366.72, + "currency": "USD", + "timestamp": "2025-04-23T01:42:31Z", + "lat": 33.858566, + "lon": -90.105201, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0142", + "user_id": "u_015", + "merchant_id": "m_016", + "item_name": "Bicycle", + "amount": 102.06, + "currency": "USD", + "timestamp": "2025-05-08T02:42:31Z", + "lat": 28.462585, + "lon": -98.776259, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0143", + "user_id": "u_012", + "merchant_id": "m_013", + "item_name": "Camera", + "amount": 967.88, + "currency": "USD", + "timestamp": "2025-05-04T20:42:31Z", + "lat": 32.884941, + "lon": -111.848094, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0144", + "user_id": "u_006", + "merchant_id": "m_020", + "item_name": "Groceries", + "amount": 396.84, + "currency": "USD", + "timestamp": "2025-04-25T05:42:31Z", + "lat": 28.28319, + "lon": -90.22848, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0145", + "user_id": "u_013", + "merchant_id": "m_024", + "item_name": "Hotel Stay", + "amount": 1382.3, + "currency": "USD", + "timestamp": "2025-05-11T01:42:31Z", + "lat": 42.73012, + "lon": -87.270315, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0146", + "user_id": "u_006", + "merchant_id": "m_008", + "item_name": "Smartphone", + "amount": 1294.97, + "currency": "USD", + "timestamp": "2025-04-29T05:42:31Z", + "lat": 33.099471, + "lon": -85.020744, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0147", + "user_id": "u_018", + "merchant_id": "m_003", + "item_name": "Dinner", + "amount": 1462.78, + "currency": "USD", + "timestamp": "2025-04-16T02:42:31Z", + "lat": 43.588713, + "lon": -67.228994, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0148", + "user_id": "u_019", + "merchant_id": "m_014", + "item_name": "Plane Ticket", + "amount": 494.3, + "currency": "USD", + "timestamp": "2025-04-28T12:42:31Z", + "lat": 37.84442, + "lon": -90.129741, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0149", + "user_id": "u_006", + "merchant_id": "m_014", + "item_name": "Book", + "amount": 1167.11, + "currency": "USD", + "timestamp": "2025-05-09T02:42:31Z", + "lat": 25.44261, + "lon": -122.931084, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0150", + "user_id": "u_013", + "merchant_id": "m_003", + "item_name": "Plane Ticket", + "amount": 1075.13, + "currency": "USD", + "timestamp": "2025-05-09T17:42:31Z", + "lat": 28.176308, + "lon": -86.449934, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0151", + "user_id": "u_008", + "merchant_id": "m_007", + "item_name": "Streaming Subscription", + "amount": 666.33, + "currency": "USD", + "timestamp": "2025-05-12T05:42:31Z", + "lat": 26.680286, + "lon": -111.39278, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0152", + "user_id": "u_018", + "merchant_id": "m_005", + "item_name": "Plane Ticket", + "amount": 1104.57, + "currency": "USD", + "timestamp": "2025-04-22T13:42:31Z", + "lat": 42.805229, + "lon": -90.308427, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0153", + "user_id": "u_004", + "merchant_id": "m_014", + "item_name": "Coffee", + "amount": 694.76, + "currency": "USD", + "timestamp": "2025-04-15T15:42:31Z", + "lat": 42.818599, + "lon": -74.853567, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0154", + "user_id": "u_005", + "merchant_id": "m_015", + "item_name": "Concert Ticket", + "amount": 802.42, + "currency": "USD", + "timestamp": "2025-04-20T01:42:31Z", + "lat": 28.357563, + "lon": -111.068094, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0155", + "user_id": "u_003", + "merchant_id": "m_009", + "item_name": "Software License", + "amount": 463.11, + "currency": "USD", + "timestamp": "2025-05-03T11:42:31Z", + "lat": 30.63969, + "lon": -84.48075, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0156", + "user_id": "u_012", + "merchant_id": "m_004", + "item_name": "Streaming Subscription", + "amount": 773.57, + "currency": "USD", + "timestamp": "2025-04-30T11:42:31Z", + "lat": 25.338644, + "lon": -120.319205, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0157", + "user_id": "u_003", + "merchant_id": "m_022", + "item_name": "Bicycle", + "amount": 979.02, + "currency": "USD", + "timestamp": "2025-04-21T18:42:31Z", + "lat": 34.514126, + "lon": -102.671221, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0158", + "user_id": "u_015", + "merchant_id": "m_020", + "item_name": "Hotel Stay", + "amount": 835.78, + "currency": "USD", + "timestamp": "2025-04-23T01:42:31Z", + "lat": 25.929198, + "lon": -78.622702, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0159", + "user_id": "u_004", + "merchant_id": "m_004", + "item_name": "Camera", + "amount": 589.17, + "currency": "USD", + "timestamp": "2025-04-21T11:42:31Z", + "lat": 45.137628, + "lon": -93.296632, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0160", + "user_id": "u_018", + "merchant_id": "m_024", + "item_name": "Hotel Stay", + "amount": 1443.17, + "currency": "USD", + "timestamp": "2025-05-07T06:42:31Z", + "lat": 28.102495, + "lon": -83.485704, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0161", + "user_id": "u_020", + "merchant_id": "m_019", + "item_name": "Gym Membership", + "amount": 1283.26, + "currency": "USD", + "timestamp": "2025-05-11T02:42:31Z", + "lat": 47.372771, + "lon": -73.897498, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0162", + "user_id": "u_003", + "merchant_id": "m_009", + "item_name": "Bicycle", + "amount": 120.46, + "currency": "USD", + "timestamp": "2025-04-26T06:42:31Z", + "lat": 40.977427, + "lon": -77.550807, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0163", + "user_id": "u_011", + "merchant_id": "m_004", + "item_name": "Groceries", + "amount": 699.56, + "currency": "USD", + "timestamp": "2025-05-07T21:42:31Z", + "lat": 43.794934, + "lon": -66.623264, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0164", + "user_id": "u_018", + "merchant_id": "m_024", + "item_name": "Dinner", + "amount": 1353.77, + "currency": "USD", + "timestamp": "2025-04-22T13:42:31Z", + "lat": 40.184985, + "lon": -73.553351, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0165", + "user_id": "u_006", + "merchant_id": "m_007", + "item_name": "Streaming Subscription", + "amount": 1176.3, + "currency": "USD", + "timestamp": "2025-04-24T09:42:31Z", + "lat": 32.550425, + "lon": -118.573451, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0166", + "user_id": "u_015", + "merchant_id": "m_013", + "item_name": "Streaming Subscription", + "amount": 1050.55, + "currency": "USD", + "timestamp": "2025-04-30T14:42:31Z", + "lat": 32.736235, + "lon": -80.030938, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0167", + "user_id": "u_018", + "merchant_id": "m_019", + "item_name": "Dinner", + "amount": 1101.12, + "currency": "USD", + "timestamp": "2025-04-21T06:42:31Z", + "lat": 27.5808, + "lon": -84.386804, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0168", + "user_id": "u_016", + "merchant_id": "m_014", + "item_name": "Groceries", + "amount": 1450.52, + "currency": "USD", + "timestamp": "2025-04-24T02:42:31Z", + "lat": 42.344951, + "lon": -116.958872, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0169", + "user_id": "u_005", + "merchant_id": "m_008", + "item_name": "Headphones", + "amount": 153.22, + "currency": "USD", + "timestamp": "2025-04-18T20:42:31Z", + "lat": 47.384434, + "lon": -104.403664, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0170", + "user_id": "u_014", + "merchant_id": "m_010", + "item_name": "Headphones", + "amount": 374.23, + "currency": "USD", + "timestamp": "2025-04-19T14:42:31Z", + "lat": 42.994319, + "lon": -72.553889, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0171", + "user_id": "u_012", + "merchant_id": "m_014", + "item_name": "Smartphone", + "amount": 396.59, + "currency": "USD", + "timestamp": "2025-05-04T06:42:31Z", + "lat": 32.054904, + "lon": -85.132955, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0172", + "user_id": "u_015", + "merchant_id": "m_003", + "item_name": "Hotel Stay", + "amount": 330.74, + "currency": "USD", + "timestamp": "2025-04-14T19:42:31Z", + "lat": 29.042974, + "lon": -73.84691, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0173", + "user_id": "u_005", + "merchant_id": "m_011", + "item_name": "Plane Ticket", + "amount": 428.5, + "currency": "USD", + "timestamp": "2025-04-26T13:42:31Z", + "lat": 33.555802, + "lon": -110.347239, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0174", + "user_id": "u_006", + "merchant_id": "m_001", + "item_name": "Gym Membership", + "amount": 758.12, + "currency": "USD", + "timestamp": "2025-05-13T11:42:31Z", + "lat": 45.606891, + "lon": -111.128267, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0175", + "user_id": "u_006", + "merchant_id": "m_010", + "item_name": "Gym Membership", + "amount": 452.62, + "currency": "USD", + "timestamp": "2025-05-11T09:42:31Z", + "lat": 42.423968, + "lon": -110.307526, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0176", + "user_id": "u_008", + "merchant_id": "m_001", + "item_name": "Smartphone", + "amount": 549.6, + "currency": "USD", + "timestamp": "2025-05-07T17:42:31Z", + "lat": 46.006509, + "lon": -99.448649, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0177", + "user_id": "u_007", + "merchant_id": "m_015", + "item_name": "Plane Ticket", + "amount": 38.08, + "currency": "USD", + "timestamp": "2025-04-20T05:42:31Z", + "lat": 28.546376, + "lon": -115.57471, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0178", + "user_id": "u_002", + "merchant_id": "m_003", + "item_name": "Streaming Subscription", + "amount": 700.0, + "currency": "USD", + "timestamp": "2025-04-23T08:42:31Z", + "lat": 41.048977, + "lon": -69.073116, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0179", + "user_id": "u_006", + "merchant_id": "m_023", + "item_name": "Shoes", + "amount": 1309.75, + "currency": "USD", + "timestamp": "2025-05-09T18:42:31Z", + "lat": 25.594526, + "lon": -83.796834, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0180", + "user_id": "u_007", + "merchant_id": "m_021", + "item_name": "Clothing", + "amount": 902.73, + "currency": "USD", + "timestamp": "2025-04-29T03:42:31Z", + "lat": 42.842689, + "lon": -90.572094, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0181", + "user_id": "u_006", + "merchant_id": "m_007", + "item_name": "Gym Membership", + "amount": 1180.13, + "currency": "USD", + "timestamp": "2025-04-22T11:42:31Z", + "lat": 25.453649, + "lon": -74.808704, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0182", + "user_id": "u_020", + "merchant_id": "m_016", + "item_name": "Dinner", + "amount": 692.55, + "currency": "USD", + "timestamp": "2025-05-01T17:42:31Z", + "lat": 45.090152, + "lon": -77.454249, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0183", + "user_id": "u_019", + "merchant_id": "m_016", + "item_name": "Book", + "amount": 1320.78, + "currency": "USD", + "timestamp": "2025-05-05T01:42:31Z", + "lat": 33.906415, + "lon": -82.116657, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0184", + "user_id": "u_010", + "merchant_id": "m_016", + "item_name": "Shoes", + "amount": 58.12, + "currency": "USD", + "timestamp": "2025-04-18T07:42:31Z", + "lat": 44.630922, + "lon": -69.218753, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0185", + "user_id": "u_015", + "merchant_id": "m_008", + "item_name": "Hotel Stay", + "amount": 1282.87, + "currency": "USD", + "timestamp": "2025-05-08T15:42:31Z", + "lat": 38.628489, + "lon": -89.03014, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0186", + "user_id": "u_004", + "merchant_id": "m_021", + "item_name": "Hotel Stay", + "amount": 300.49, + "currency": "USD", + "timestamp": "2025-05-04T10:42:31Z", + "lat": 37.60722, + "lon": -75.899307, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0187", + "user_id": "u_006", + "merchant_id": "m_009", + "item_name": "Shoes", + "amount": 432.81, + "currency": "USD", + "timestamp": "2025-04-15T16:42:31Z", + "lat": 42.932298, + "lon": -113.147075, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0188", + "user_id": "u_001", + "merchant_id": "m_010", + "item_name": "Dinner", + "amount": 394.58, + "currency": "USD", + "timestamp": "2025-04-15T09:42:31Z", + "lat": 26.964914, + "lon": -105.076442, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0189", + "user_id": "u_014", + "merchant_id": "m_021", + "item_name": "Concert Ticket", + "amount": 1281.12, + "currency": "USD", + "timestamp": "2025-04-19T12:42:31Z", + "lat": 32.450397, + "lon": -96.881517, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0190", + "user_id": "u_013", + "merchant_id": "m_005", + "item_name": "Plane Ticket", + "amount": 1413.99, + "currency": "USD", + "timestamp": "2025-04-25T08:42:31Z", + "lat": 41.126811, + "lon": -102.944164, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0191", + "user_id": "u_001", + "merchant_id": "m_014", + "item_name": "Plane Ticket", + "amount": 912.33, + "currency": "USD", + "timestamp": "2025-05-09T11:42:31Z", + "lat": 30.321647, + "lon": -85.547224, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0192", + "user_id": "u_019", + "merchant_id": "m_006", + "item_name": "Gym Membership", + "amount": 806.33, + "currency": "USD", + "timestamp": "2025-04-21T13:42:31Z", + "lat": 36.445358, + "lon": -74.416482, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0193", + "user_id": "u_010", + "merchant_id": "m_010", + "item_name": "Gym Membership", + "amount": 586.87, + "currency": "USD", + "timestamp": "2025-04-15T21:42:31Z", + "lat": 26.537991, + "lon": -120.527613, + "card_provider": "VISA" + }, + { + "transaction_id": "txn_0194", + "user_id": "u_009", + "merchant_id": "m_003", + "item_name": "Dinner", + "amount": 330.0, + "currency": "USD", + "timestamp": "2025-04-21T01:42:31Z", + "lat": 48.171873, + "lon": -66.261223, + "card_provider": "DISCOVER" + }, + { + "transaction_id": "txn_0195", + "user_id": "u_017", + "merchant_id": "m_002", + "item_name": "Groceries", + "amount": 429.52, + "currency": "USD", + "timestamp": "2025-04-29T11:42:31Z", + "lat": 38.11713, + "lon": -108.200041, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0196", + "user_id": "u_018", + "merchant_id": "m_007", + "item_name": "Hotel Stay", + "amount": 1402.06, + "currency": "USD", + "timestamp": "2025-04-24T05:42:31Z", + "lat": 48.421763, + "lon": -109.737807, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0197", + "user_id": "u_009", + "merchant_id": "m_014", + "item_name": "Software License", + "amount": 751.62, + "currency": "USD", + "timestamp": "2025-04-29T08:42:31Z", + "lat": 31.832108, + "lon": -82.46585, + "card_provider": "MASTERCARD" + }, + { + "transaction_id": "txn_0198", + "user_id": "u_016", + "merchant_id": "m_005", + "item_name": "Furniture", + "amount": 1042.48, + "currency": "USD", + "timestamp": "2025-04-27T23:42:31Z", + "lat": 32.759203, + "lon": -92.10264, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0199", + "user_id": "u_020", + "merchant_id": "m_024", + "item_name": "Smartphone", + "amount": 178.21, + "currency": "USD", + "timestamp": "2025-04-15T01:42:31Z", + "lat": 40.125231, + "lon": -90.035775, + "card_provider": "AMEX" + }, + { + "transaction_id": "txn_0200", + "user_id": "u_009", + "merchant_id": "m_007", + "item_name": "Bicycle", + "amount": 1054.41, + "currency": "USD", + "timestamp": "2025-04-15T08:42:31Z", + "lat": 42.611433, + "lon": -109.501848, + "card_provider": "AMEX" + } +] \ No newline at end of file