Merge pull request #3 from beekeeper-studio/feat/return-cursor

add ability to return a cursor instead of the results

Author: not-night-but

docs/usage/core-concepts.md

@@ -50,7 +50,16 @@ The main class that ties everything together. It provides a simple API to execut
 
 ```typescript
 const queryLeaf = new QueryLeaf(mongoClient, 'mydatabase');
+
+// Execute a query and get all results as an array
 const results = await queryLeaf.execute('SELECT * FROM users');
+
+// Or use a cursor for more control and memory efficiency
+const cursor = await queryLeaf.executeCursor('SELECT * FROM users');
+await cursor.forEach(user => {
+  console.log(`Processing user: ${user.name}`);
+});
+await cursor.close();
 ```
 
 ### DummyQueryLeaf
@@ -61,6 +70,13 @@ A special implementation of QueryLeaf that doesn't execute real MongoDB operatio
 const dummyLeaf = new DummyQueryLeaf('mydatabase');
 await dummyLeaf.execute('SELECT * FROM users');
 // [DUMMY MongoDB] FIND in mydatabase.users with filter: {}
+
+// Cursor support works with DummyQueryLeaf too
+const cursor = await dummyLeaf.executeCursor('SELECT * FROM users');
+await cursor.forEach(user => {
+  // Process mock data
+});
+await cursor.close();
 ```
 
 ## Relationship Between SQL and MongoDB Concepts
@@ -101,14 +117,31 @@ QueryLeaf uses specific naming conventions for mapping SQL to MongoDB:
 
 ## Execution Flow
 
+QueryLeaf supports two main execution methods:
+
+### Standard Execution
+
 When you call `queryLeaf.execute(sqlQuery)`, the following happens:
 
 1. The SQL query is parsed into an AST
 2. The AST is compiled into MongoDB commands
 3. The commands are executed against the MongoDB database
-4. The results are returned
+4. All results are loaded into memory and returned as an array
+
+This is simple to use but can be memory-intensive for large result sets.
+
+### Cursor Execution
+
+When you call `queryLeaf.executeCursor(sqlQuery)`, the following happens:
+
+1. The SQL query is parsed into an AST
+2. The AST is compiled into MongoDB commands
+3. For SELECT queries, a MongoDB cursor is returned instead of loading all results
+4. You control how and when results are processed (streaming/batching)
+
+This approach is more memory-efficient for large datasets and gives you more control.
 
-If any step fails, an error is thrown with details about what went wrong.
+If any step fails in either approach, an error is thrown with details about what went wrong.
 
 ## Extending QueryLeaf
 

docs/usage/examples.md

@@ -73,6 +73,37 @@ const usersInNY = await queryLeaf.execute(`
 `);
 ```
 
+### Using Cursors for Large Result Sets
+
+```typescript
+// Use cursor for memory-efficient processing of large result sets
+const cursor = await queryLeaf.executeCursor(`
+  SELECT _id, customer, total, items
+  FROM orders
+  WHERE status = 'completed'
+`);
+
+try {
+  // Process one document at a time without loading everything in memory
+  let totalRevenue = 0;
+  await cursor.forEach(order => {
+    // Process each order individually
+    totalRevenue += order.total;
+    
+    // Access and process nested data
+    order.items.forEach(item => {
+      // Process each item in the order
+      console.log(`Order ${order._id}: ${item.quantity}x ${item.name}`);
+    });
+  });
+  
+  console.log(`Total revenue: $${totalRevenue}`);
+} finally {
+  // Always close the cursor when done
+  await cursor.close();
+}
+```
+
 ### Array Element Access
 
 ```typescript
@@ -169,11 +200,11 @@ async function getUserDashboardData(userId) {
 }
 ```
 
-### Product Catalog with Filtering
+### Product Catalog with Filtering and Cursor-Based Pagination
 
 ```typescript
-// Product catalog with filtering
-async function getProductCatalog(filters = {}) {
+// Product catalog with filtering and cursor-based pagination
+async function getProductCatalog(filters = {}, useCursor = false) {
   const client = new MongoClient('mongodb://localhost:27017');
   await client.connect();
 
@@ -217,9 +248,52 @@ async function getProductCatalog(filters = {}) {
       query += ` LIMIT ${filters.limit}`;
     }
 
-    // Execute query
-    return await queryLeaf.execute(query);
+    if (filters.offset) {
+      query += ` OFFSET ${filters.offset}`;
+    }
+    
+    // Execute query with or without cursor based on preference
+    if (useCursor) {
+      // Return a cursor for client-side pagination or streaming
+      return await queryLeaf.executeCursor(query);
+    } else {
+      // Return all results at once (traditional approach)
+      return await queryLeaf.execute(query);
+    }
+  } catch (error) {
+    console.error('Error fetching product catalog:', error);
+    throw error;
+  } finally {
+    if (!useCursor) {
+      // If we returned a cursor, the caller is responsible for closing the client
+      // after they are done with the cursor
+      await client.close();
+    }
+  }
+}
+
+// Example of using the product catalog with cursor
+async function streamProductCatalog() {
+  const client = new MongoClient('mongodb://localhost:27017');
+  await client.connect();
+  
+  let cursor = null;
+  try {
+    const queryLeaf = new QueryLeaf(client, 'store');
+    // Get a cursor for large result set
+    cursor = await getProductCatalog({ category: 'Electronics', inStock: true }, true);
+    
+    // Stream products to client one by one
+    console.log('Streaming products:');
+    await cursor.forEach(product => {
+      console.log(`- ${product.name}: $${product.price} (${product.stock} in stock)`);
+    });
+  } catch (error) {
+    console.error('Error:', error);
   } finally {
+    // Close cursor if we have one
+    if (cursor) await cursor.close();
+    // Close client connection
     await client.close();
   }
 }

docs/usage/mongodb-client.md

@@ -46,13 +46,25 @@ await mongoClient.connect();
 Once you have a MongoDB client, you can create a QueryLeaf instance:
 
 ```typescript
-import { QueryLeaf } from 'queryleaf';
+import { QueryLeaf } from '@queryleaf/lib';
 
 // Create a QueryLeaf instance with your MongoDB client
 const queryLeaf = new QueryLeaf(mongoClient, 'mydatabase');
 
-// Now you can execute SQL queries
+// Execute SQL queries and get all results at once
 const results = await queryLeaf.execute('SELECT * FROM users');
+
+// For large result sets, use cursor execution for better memory efficiency
+const cursor = await queryLeaf.executeCursor('SELECT * FROM users');
+try {
+  // Process results one at a time
+  await cursor.forEach(user => {
+    console.log(`User: ${user.name}`);
+  });
+} finally {
+  // Always close the cursor when done
+  await cursor.close();
+}
 ```
 
 ## Connection Management
@@ -80,7 +92,7 @@ async function main() {
     // Create QueryLeaf instance
     const queryLeaf = new QueryLeaf(client, 'mydatabase');
 
-    // Execute queries
+    // Execute queries - get all results at once
     const users = await queryLeaf.execute('SELECT * FROM users LIMIT 10');
     console.log(`Found ${users.length} users`);
 
@@ -90,6 +102,23 @@ async function main() {
     );
     console.log(`Found ${products.length} electronic products`);
 
+    // For large result sets, use cursor execution
+    const ordersCursor = await queryLeaf.executeCursor(
+      'SELECT * FROM orders WHERE total > 1000'
+    );
+    try {
+      // Process results in a memory-efficient way
+      let count = 0;
+      await ordersCursor.forEach(order => {
+        console.log(`Processing order #${order.orderId}`);
+        count++;
+      });
+      console.log(`Processed ${count} high-value orders`);
+    } finally {
+      // Always close the cursor when done
+      await ordersCursor.close();
+    }
+    
   } catch (error) {
     console.error('Error:', error);
   } finally {

packages/lib/README.md

@@ -21,6 +21,7 @@
   - Array element access (e.g., `items[0].name`)
   - GROUP BY with aggregation functions (COUNT, SUM, AVG, MIN, MAX)
   - JOINs between collections
+  - Direct MongoDB cursor access for fine-grained result processing and memory efficiency
 
 ## Installation
 
@@ -47,6 +48,13 @@ const queryLeaf = new QueryLeaf(mongoClient, 'mydatabase');
 const results = await queryLeaf.execute('SELECT * FROM users WHERE age > 21');
 console.log(results);
 
+// Get a MongoDB cursor for more control over result processing and memory efficiency
+const cursor = await queryLeaf.executeCursor('SELECT * FROM users WHERE age > 30');
+await cursor.forEach((doc) => {
+  console.log(`User: ${doc.name}`);
+});
+await cursor.close();
+
 // When you're done, close your MongoDB client
 await mongoClient.close();
 ```
@@ -64,6 +72,13 @@ const queryLeaf = new DummyQueryLeaf('mydatabase');
 // Operations will be logged to console but not executed
 await queryLeaf.execute('SELECT * FROM users WHERE age > 21');
 // [DUMMY MongoDB] FIND in mydatabase.users with filter: { "age": { "$gt": 21 } }
+
+// You can also use cursor functionality with DummyQueryLeaf
+const cursor = await queryLeaf.executeCursor('SELECT * FROM users LIMIT 10');
+await cursor.forEach((doc) => {
+  // Process each document
+});
+await cursor.close();
 ```
 
 ## SQL Query Examples
@@ -85,6 +100,42 @@ SELECT status, COUNT(*) as count FROM orders GROUP BY status
 SELECT u.name, o.total FROM users u JOIN orders o ON u._id = o.userId
 ```
 
+## Working with Cursors
+
+When working with large result sets, using MongoDB cursors directly can be more memory-efficient and gives you more control over result processing:
+
+```typescript
+// Get a cursor for a SELECT query
+const cursor = await queryLeaf.executeCursor('SELECT * FROM products WHERE price > 100');
+
+// Option 1: Convert to array (loads all results into memory)
+const results = await cursor.toArray();
+console.log(`Found ${results.length} products`);
+
+// Option 2: Iterate with forEach (memory efficient)
+await cursor.forEach(product => {
+  console.log(`Processing ${product.name}...`);
+});
+
+// Option 3: Manual iteration with next/hasNext (most control)
+while (await cursor.hasNext()) {
+  const product = await cursor.next();
+  // Process each product individually
+  console.log(`Product: ${product.name}, $${product.price}`);
+}
+
+// Always close the cursor when done
+await cursor.close();
+```
+
+Features:
+- Returns MongoDB `FindCursor` for normal queries and `AggregationCursor` for aggregations
+- Supports all cursor methods like `forEach()`, `toArray()`, `next()`, `hasNext()`
+- Efficiently handles large result sets with MongoDB's batching system
+- Works with all advanced QueryLeaf features (filtering, sorting, aggregations, etc.)
+- Only available for read operations (SELECT queries)
+```
+
 ## Links
 
 - [Website](https://queryleaf.com)