Sample Store Component Documentation

Overview

The Sample Store Component is an ESP-IDF component designed for efficient storage and management of generic data samples (blobs) using a dedicated NVS partition. It provides a set-based organization system where users can define custom namespaces to categorize their data.

🎯 Key Features

• Set-based Organization: Store samples in user-defined sets (namespaces) for logical grouping
• Generic Data Storage: Store any data as blobs without knowledge of internal structure
• Automatic Retention Management: Configurable limits with event-driven retention control
• Iterator-based Navigation: Efficient forward/backward navigation through samples
• Sample Deletion: Delete oldest or newest samples to manage storage space
• Dedicated NVS Partition: Uses a separate partition to avoid conflicts with application data
• Thread-safe Operations: Safe for use in multi-threaded ESP-IDF applications

📋 Requirements

Hardware Requirements

• ESP32, ESP32-S2, ESP32-S3, ESP32-C3, or ESP32-C6 chip
• Sufficient flash memory for the dedicated NVS partition

Software Requirements

• ESP-IDF v4.4 or later
• CMake build system

Partition Table Requirements

The component requires a dedicated NVS partition separate from the default NVS partition. You must add this partition to your partitions.csv file:

# Name,            Type,     SubType,     Offset,      Size,      Flags
nvs,               data,     nvs,         ,            0x6000,
phy_init,          data,     phy,         ,            0x1000,
factory,           app,      factory,     ,            0x1F0000,
sample_store_nvs,  data,     nvs,         ,            0x600000,

⚠️ Important: The partition name must match the configuration in menuconfig (default: sample_store_nvs)

🔧 Configuration

The component can be configured through ESP-IDF's menuconfig system:

idf.py menuconfig

Navigate to: Component config → Sample Store Configuration

Available Configuration Options

Option	Default	Description
CONFIG_SAMPLE_STORE_PARTITION_NAME	"sample_store_nvs"	Name of the NVS partition to use
CONFIG_SAMPLE_STORE_MAX_SETS	5	Maximum number of sets to store
CONFIG_SAMPLE_STORE_MAX_SAMPLES_PER_DAY	1000	Maximum samples per set

Configuration Details

Partition Name (CONFIG_SAMPLE_STORE_PARTITION_NAME)

• Purpose: Specifies which NVS partition to use for sample storage
• Important: Must match an existing partition in your partitions.csv
• Default: "sample_store_nvs"

Maximum Sets (CONFIG_SAMPLE_STORE_MAX_SETS)

• Purpose: Controls how many different sets (namespaces) can exist simultaneously
• Behavior: When exceeded, oldest sets are automatically removed
• Range: 1-255 sets
• Considerations: More sets = more metadata overhead

Maximum Samples Per Set (CONFIG_SAMPLE_STORE_MAX_SAMPLES_PER_DAY)

• Purpose: Controls how many samples can be stored in each set
• Behavior: When exceeded, oldest samples in the set are automatically removed
• Range: 1-16,777,215 samples
• Note: The name "PER_DAY" is historical; it actually means "per set"

🚀 Quick Start

1. Add Component to Your Project

Copy the sample_store component directory to your project's components/ folder:

your_project/
├── components/
│   └── sample_store/
│       ├── CMakeLists.txt
│       ├── Kconfig
│       ├── sample_store.c
│       └── sample_store.h
├── main/
│   └── main.c
└── partitions.csv

2. Configure Partition Table

Edit your partitions.csv to include the sample store partition:

# Name,            Type,     SubType,     Offset,      Size,      Flags
nvs,               data,     nvs,         ,            0x6000,
phy_init,          data,     phy,         ,            0x1000,  
factory,           app,      factory,     ,            0x1F0000,
sample_store_nvs,  data,     nvs,         ,            0x600000,

3. Basic Usage Example

#include "sample_store.h"
#include "esp_log.h"

static const char* TAG = "APP";

void app_main(void)
{
    // Initialize NVS for default partition (metadata storage)
    ESP_ERROR_CHECK(nvs_flash_init());
    
    // Initialize sample store
    sample_store_handle_t store = NULL;
    ESP_ERROR_CHECK(sample_store_init(&store));
    
    // Write a sample to a set
    uint8_t data[] = {0x01, 0x02, 0x03, 0x04};
    esp_err_t err = sample_store_write_to_set(store, "sensors", data, sizeof(data));
    if (err == ESP_OK) {
        ESP_LOGI(TAG, "Sample written successfully");
    }
    
    // Read the sample back
    uint32_t min_counter, max_counter;
    if (sample_store_get_set_range(store, "sensors", &min_counter, &max_counter) == ESP_OK) {
        char hex_key[7];
        snprintf(hex_key, sizeof(hex_key), "%06lx", max_counter);
        
        size_t size = sizeof(data);
        uint8_t read_data[4];
        if (sample_store_read_from_set(store, "sensors", hex_key, read_data, &size) == ESP_OK) {
            ESP_LOGI(TAG, "Sample read successfully");
        }
    }
    
    // Cleanup
    sample_store_deinit(store);
}

📚 API Reference

For detailed API documentation, see doc/API_REFERENCE.md.

🔄 Storage Architecture

Set-based Organization

• Sets: User-defined namespaces (max 15 characters + null terminator)
• Samples: Stored as blobs with auto-incrementing hex keys (000001, 000002, etc.)
• Metadata: Stored in default NVS partition for persistence across reboots

Key Format

• Pattern: 6-character hexadecimal (%06x)
• Range: 000001 to ffffff (16,777,215 samples per set)
• Example: 000001, 0000ff, 123abc

Partition Usage

• Data Partition: Stores actual sample data (configured partition name)
• Metadata Partition: Stores set information and counters (nvs_default)

🎮 Event System

The component provides events for retention management:

typedef enum {
    SAMPLE_STORE_EVENT_PRE_OVERWRITE_SAMPLE = 0,  // Before deleting oldest sample
    SAMPLE_STORE_EVENT_POST_OVERWRITE_SAMPLE,     // After deleting oldest sample  
    SAMPLE_STORE_EVENT_PRE_OVERWRITE_SET,         // Before deleting oldest set
    SAMPLE_STORE_EVENT_POST_OVERWRITE_SET         // After deleting oldest set
} sample_store_event_type_t;

Event Handler Registration

void retention_handler(sample_store_event_type_t type, const char *set_name, 
                      const char *key, void *user_ctx) {
    // Handle retention events
    ESP_LOGW("APP", "Retention event: %d for set %s", type, set_name);
}

// Register the handler
sample_store_set_event_handler(store, retention_handler, NULL);

🔍 Advanced Usage

Iterator-based Navigation

sample_store_iterator_handle_t iterator;
uint32_t min_counter, max_counter;

// Get range for a set
sample_store_get_set_range(store, "sensors", &min_counter, &max_counter);

// Create iterator
sample_store_iterator_new(store, "sensors", min_counter, max_counter, &iterator);

// Navigate through samples
do {
    void *data;
    size_t size;
    uint32_t counter;
    
    if (sample_store_iterator_get_sample(iterator, &data, &size) == ESP_OK) {
        sample_store_iterator_get_current_counter(iterator, &counter);
        ESP_LOGI(TAG, "Sample %06lx: %zu bytes", counter, size);
    }
} while (sample_store_iterator_next(iterator) == ESP_OK);

sample_store_iterator_free(iterator);

Sample Deletion

// Delete oldest or newest sample only
sample_store_iterator_handle_t iterator;
sample_store_iterator_new(store, "sensors", min_counter, max_counter, &iterator);

// Delete oldest (positioned at start by default)
esp_err_t result = sample_store_iterator_delete_sample(iterator);

// Or go to newest and delete
sample_store_iterator_goto(iterator, max_counter);
result = sample_store_iterator_delete_sample(iterator);

sample_store_iterator_free(iterator);

⚠️ Important Considerations

Memory Management

• Iterator Data: Automatically managed, freed when iterator is destroyed
• Set Lists: Must be manually freed after sample_store_list_sets()
• Sample Data: User responsible for buffer management in direct reads

Thread Safety

• Component is thread-safe for concurrent operations
• Individual iterator instances are NOT thread-safe

Storage Limitations

• Set Names: Maximum 15 characters + null terminator
• Sample Size: Limited by NVS blob size limits (~508KB per entry)
• Total Storage: Limited by partition size configuration

Performance Considerations

• Retention Operations: May block briefly during automatic cleanup
• Iterator Operations: Optimized for sequential access
• Random Access: Use direct read functions for better performance

🐛 Troubleshooting

Common Issues

"Partition not found" Error

• Cause: NVS partition name in menuconfig doesn't match partitions.csv
• Solution: Ensure partition names match exactly

"No space left" Errors

• Cause: Partition size too small for configured limits
• Solution: Increase partition size or reduce storage limits

Retention Events Not Firing

• Cause: Event handler not registered or limits not reached
• Solution: Verify handler registration and configuration values

Debug Logging

Enable component logging for troubleshooting:

esp_log_level_set("sample_store", ESP_LOG_DEBUG);

📖 Additional Resources

• API Reference - Complete API documentation
• Examples - Full working example
• Component Configuration - Detailed configuration guide