From 7b04c279c551593b0992e9392f42f555e87836ab Mon Sep 17 00:00:00 2001 From: Vjeran Grozdanic Date: Mon, 29 Dec 2025 10:30:50 +0100 Subject: [PATCH 1/4] docs(encryption): Add docs for encrypted field usage --- .../backend/encrypted-fields/index.mdx | 197 ++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 develop-docs/backend/encrypted-fields/index.mdx diff --git a/develop-docs/backend/encrypted-fields/index.mdx b/develop-docs/backend/encrypted-fields/index.mdx new file mode 100644 index 00000000000000..befeb5fe683f7f --- /dev/null +++ b/develop-docs/backend/encrypted-fields/index.mdx @@ -0,0 +1,197 @@ +--- +title: "Encrypted Fields" +description: "Drop-in replacement Django fields for encrypting sensitive data in Sentry." +categories: + - backend + - encryption + - django +sidebar_order: 10 +--- + +# Encrypted Fields + +Encrypted fields provide transparent encryption for sensitive database fields in Sentry. They are drop-in replacements for standard Django fields that automatically encrypt data on save while maintaining backward compatibility with unencrypted already existing data. + +## Overview + +When dealing with sensitive data that cannot be stored in plain text, use encrypted field variants: + +- **`EncryptedCharField`**: Drop-in replacement for `CharField` +- **`EncryptedJSONField`**: Drop-in replacement for `JSONField` + +### Key Features + +- **Drop-in replacement**: Use exactly like regular Django fields +- **Encrypts on save**: Data is encrypted only when saving to the database +- **Reads plain text**: Can read unencrypted data for backward compatibility +- **Transparent decryption**: Works seamlessly with Django ORM, in memory field value is always decrypted (decryption happens during entry retrieval), so when you work with it, you can treat it as it is regular field + +## Usage + +Simply replace standard Django fields with their encrypted equivalents: + +```python {filename:models.py} +from django.db import models +from sentry.db.models.fields import EncryptedCharField, EncryptedJSONField + +class IntegrationConfig(models.Model): + name = models.CharField(max_length=100) + # Instead of: api_key = models.CharField(max_length=255) + api_key = EncryptedCharField() # DO NOT SET max_length as encrypted data is longer than plain text data!!! + + # Instead of: config = models.JSONField() + config = EncryptedJSONField() +``` + +That's it! The field will automatically encrypt data when saving and decrypt when reading. + +## Configuration + +### Encryption Method + +The encryption method is currently controlled globally via Sentry options: + +```python +# In sentry.conf.py or via Django admin +options.set('database.encryption.method', 'fernet') # or 'plaintext' +``` + +DO NOT CHANGE it, unless you have a really good reason to do it! + +### Key Management + +In production, encryption keys are managed via Kubernetes and mounted to pods as secrets. The keys are automatically loaded from the mounted directory. + +In `sentry/conf/server.py` there is a setting `DATABASE_ENCRYPTION_SETTINGS` which controls the the primary key id (key used for encrypting everything, while there might be multiple keys used for decryption), and key location directory. Each key is mounted as a separate file, where file name is used as a keyId. DO NOT change file names of existing keys. + + + - Encryption keys are managed via Kubernetes secret manager - Keys are mounted + to pods under a secure directory - Never commit encryption keys to version + control - Use different keys for different environments + + +## How It Works + +### Encryption on Save + +When you save a model instance: + +1. Data is encrypted using the configured encryption method +2. Encrypted data is wrapped with a format marker (e.g., `enc:fernet:key_id:data`) +3. The encrypted string is stored in the database + +### Decryption on Read + +When you read a model instance: + +1. The field checks if data is encrypted by looking for the format marker +2. **If encrypted**: Decrypts using the appropriate method based on the marker +3. **If not encrypted**: Returns the data as-is (backward compatibility) +4. Returns the decrypted value transparently + +### Backward Compatibility + +The fields are designed to work with existing unencrypted data: + +- Existing plain text data can be read without issues +- Data gets encrypted the next time it's saved +- No migration required to start using encrypted fields + +## Field-Specific Behavior + +### EncryptedCharField + +These work exactly like their Django counterparts but with automatic encryption: + +```python +from sentry.db.models.fields import EncryptedCharField + +class APIKey(models.Model): + key = EncryptedCharField(max_length=255) + # Use it like a regular CharField +``` + +### EncryptedJSONField + +For JSON data, `EncryptedJSONField` encrypts the entire JSON structure: + +```python +from sentry.db.models.fields import EncryptedJSONField + +class Integration(models.Model): + config = EncryptedJSONField() + # Store complex data structures securely +``` + +**Storage structure**: The encrypted data is wrapped in a JSON object for database compatibility: + +```json +{ + "sentry_encrypted_field_value": "enc:fernet:key_id:base64_encrypted_data" +} +``` + +This allows: + +- True jsonb storage in PostgreSQL +- Easy identification of encrypted vs unencrypted data +- Backward compatibility with unencrypted JSON + +## Security Considerations + +### When to Use Encrypted Fields + +Use encrypted fields for: + +- API keys and tokens +- Webhook secrets +- OAuth credentials +- Personal identifiable information (PII) +- Any sensitive data that should not be stored in plain text + +### Limitations + + + Encrypted fields **cannot** be used in: - WHERE clauses - ORDER BY statements + - GROUP BY statements - Database indexes - Full-text search - JSON path + queries (for `EncryptedJSONField`) + + +### Performance Considerations + +- **Encryption cost**: Minimal overhead on save operations +- **Decryption cost**: Happens on every field access +- **No caching**: Consider application-level caching for frequently accessed data +- **Query restrictions**: Cannot perform database-level filtering or sorting + +## Migration Strategy + +### Adding Encryption to Existing Fields + +To encrypt an existing field: + +1. **Create a migration** changing the field type: + +```python {filename:migrations/0001_encrypt_api_key.py} +from django.db import migrations +from sentry.db.models.fields import EncryptedCharField + +class Migration(migrations.Migration): + operations = [ + migrations.AlterField( + model_name='integrationconfig', + name='api_key', + field=EncryptedCharField(), + ), + ] +``` + +2. **Deploy**: The field will start reading existing unencrypted data +3. **Automatic encryption**: Data gets encrypted on next save +4. **No downtime**: Backward compatibility ensures smooth transition + + + You can switch to encrypted fields without downtime. The fields automatically + handle both encrypted and unencrypted data, encrypting values as they're + updated. + From 48c58e69df15e28f289a2134f4a2470df0b7fea9 Mon Sep 17 00:00:00 2001 From: Vjeran Grozdanic Date: Mon, 29 Dec 2025 13:41:00 +0100 Subject: [PATCH 2/4] move to application domains --- .../backend/{ => application-domains}/encrypted-fields/index.mdx | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename develop-docs/backend/{ => application-domains}/encrypted-fields/index.mdx (100%) diff --git a/develop-docs/backend/encrypted-fields/index.mdx b/develop-docs/backend/application-domains/encrypted-fields/index.mdx similarity index 100% rename from develop-docs/backend/encrypted-fields/index.mdx rename to develop-docs/backend/application-domains/encrypted-fields/index.mdx From 62f7e202191c9605d5ff0a12b7d7ad81077c9d0b Mon Sep 17 00:00:00 2001 From: Vjeran Grozdanic Date: Tue, 30 Dec 2025 13:48:27 +0100 Subject: [PATCH 3/4] update docs --- .../encrypted-fields/index.mdx | 333 ++++++++++++------ 1 file changed, 218 insertions(+), 115 deletions(-) diff --git a/develop-docs/backend/application-domains/encrypted-fields/index.mdx b/develop-docs/backend/application-domains/encrypted-fields/index.mdx index befeb5fe683f7f..e9d2c5c29f16d5 100644 --- a/develop-docs/backend/application-domains/encrypted-fields/index.mdx +++ b/develop-docs/backend/application-domains/encrypted-fields/index.mdx @@ -10,188 +10,291 @@ sidebar_order: 10 # Encrypted Fields -Encrypted fields provide transparent encryption for sensitive database fields in Sentry. They are drop-in replacements for standard Django fields that automatically encrypt data on save while maintaining backward compatibility with unencrypted already existing data. +Sentry provides encrypted database field types for storing sensitive data securely. The encryption system uses Fernet symmetric encryption with support for key rotation and backward compatibility. -## Overview +## [Available Field Types](#available-field-types) -When dealing with sensitive data that cannot be stored in plain text, use encrypted field variants: +### [EncryptedCharField](#encryptedcharfield) -- **`EncryptedCharField`**: Drop-in replacement for `CharField` -- **`EncryptedJSONField`**: Drop-in replacement for `JSONField` +A drop-in replacement for Django's `CharField` that encrypts text data. -### Key Features + + **Important**: Do not set the `max_length` property on `EncryptedCharField`. + The encrypted payload is larger than the original plaintext data. + -- **Drop-in replacement**: Use exactly like regular Django fields -- **Encrypts on save**: Data is encrypted only when saving to the database -- **Reads plain text**: Can read unencrypted data for backward compatibility -- **Transparent decryption**: Works seamlessly with Django ORM, in memory field value is always decrypted (decryption happens during entry retrieval), so when you work with it, you can treat it as it is regular field +```python +from sentry.db.models.fields.encryption import EncryptedCharField -## Usage +class MyModel(models.Model): + secret_token = EncryptedCharField() + api_key = EncryptedCharField(null=True, blank=True) +``` -Simply replace standard Django fields with their encrypted equivalents: +### [EncryptedJSONField](#encryptedjsonfield) -```python {filename:models.py} -from django.db import models -from sentry.db.models.fields import EncryptedCharField, EncryptedJSONField +A drop-in replacement for Django's `JSONField` that encrypts JSON data. -class IntegrationConfig(models.Model): - name = models.CharField(max_length=100) - # Instead of: api_key = models.CharField(max_length=255) - api_key = EncryptedCharField() # DO NOT SET max_length as encrypted data is longer than plain text data!!! +```python +from sentry.db.models.fields.encryption import EncryptedJSONField - # Instead of: config = models.JSONField() - config = EncryptedJSONField() +class MyModel(models.Model): + credentials = EncryptedJSONField(null=True, blank=True) + metadata = EncryptedJSONField(default=dict) ``` -That's it! The field will automatically encrypt data when saving and decrypt when reading. +## [Configuration](#configuration) -## Configuration +### [Encryption Method](#encryption-method) -### Encryption Method +The `database.encryption.method` option controls which encryption method to use: -The encryption method is currently controlled globally via Sentry options: +- `"plaintext"` - No encryption (default for development, base64-encoded only) +- `"fernet"` - Fernet symmetric encryption (production) ```python -# In sentry.conf.py or via Django admin -options.set('database.encryption.method', 'fernet') # or 'plaintext' +# In your Sentry options +options.set("database.encryption.method", "fernet") ``` -DO NOT CHANGE it, unless you have a really good reason to do it! +### [Fernet Keys](#fernet-keys) -### Key Management +Fernet encryption requires two settings in `DATABASE_ENCRYPTION_SETTINGS`: -In production, encryption keys are managed via Kubernetes and mounted to pods as secrets. The keys are automatically loaded from the mounted directory. +```python +DATABASE_ENCRYPTION_SETTINGS = { + "fernet_keys_location": "/path/to/keys/directory", + "fernet_primary_key_id": "key_2024_01" +} +``` -In `sentry/conf/server.py` there is a setting `DATABASE_ENCRYPTION_SETTINGS` which controls the the primary key id (key used for encrypting everything, while there might be multiple keys used for decryption), and key location directory. Each key is mounted as a separate file, where file name is used as a keyId. DO NOT change file names of existing keys. +- `fernet_keys_location`: Directory containing encryption key files +- `fernet_primary_key_id`: The key ID to use for encrypting new data - - - Encryption keys are managed via Kubernetes secret manager - Keys are mounted - to pods under a secure directory - Never commit encryption keys to version - control - Use different keys for different environments - +Keys are stored as Kubernetes secrets and mounted as files to pods that have access to the database. Each secret is mounted as a separate file in the keys directory, with the filename serving as the key ID: -## How It Works +``` +/path/to/keys/ +├── key_2023_12 +├── key_2024_01 # Current primary key +└── key_2024_02 +``` + +### [Generating Keys](#generating-keys) -### Encryption on Save +Generate a Fernet key using Python: -When you save a model instance: +```python +from cryptography.fernet import Fernet -1. Data is encrypted using the configured encryption method -2. Encrypted data is wrapped with a format marker (e.g., `enc:fernet:key_id:data`) -3. The encrypted string is stored in the database +key = Fernet.generate_key() +print(key.decode()) # Example: gAAAAABh... +``` -### Decryption on Read +## [Basic Usage](#basic-usage) -When you read a model instance: +```python +from sentry.db.models.fields.encryption import EncryptedCharField, EncryptedJSONField + +class TempestCredentials(models.Model): + client_id = models.CharField() + client_secret = EncryptedCharField() + metadata = EncryptedJSONField(default=dict) + +# Using the model +creds = TempestCredentials.objects.create( + client_id="my-client", + client_secret="super-secret-value", + metadata={"api_version": "v2", "scopes": ["read", "write"]} +) + +# Reading works transparently +print(creds.client_secret) # Prints: "super-secret-value" +print(creds.metadata) # Prints: {"api_version": "v2", "scopes": ["read", "write"]} +``` -1. The field checks if data is encrypted by looking for the format marker -2. **If encrypted**: Decrypts using the appropriate method based on the marker -3. **If not encrypted**: Returns the data as-is (backward compatibility) -4. Returns the decrypted value transparently +## [Migrations](#migrations) -### Backward Compatibility +### [Adding New Encrypted Fields](#adding-new-encrypted-fields) -The fields are designed to work with existing unencrypted data: +Add the field to your model and generate a migration: -- Existing plain text data can be read without issues -- Data gets encrypted the next time it's saved -- No migration required to start using encrypted fields +```python +class MyModel(models.Model): + api_key = EncryptedCharField(null=True, blank=True) +``` -## Field-Specific Behavior +```bash +sentry django makemigrations +sentry upgrade +``` -### EncryptedCharField +### [Converting Existing Fields](#converting-existing-fields) -These work exactly like their Django counterparts but with automatic encryption: +**Step 1**: Change the field type in your model: ```python -from sentry.db.models.fields import EncryptedCharField +# Before +class MyModel(models.Model): + api_key = models.CharField(max_length=255) -class APIKey(models.Model): - key = EncryptedCharField(max_length=255) - # Use it like a regular CharField +# After +class MyModel(models.Model): + api_key = EncryptedCharField() ``` -### EncryptedJSONField +**Step 2**: Generate and deploy the migration: + +```bash +sentry django makemigrations +sentry upgrade +``` + +The encrypted field will automatically: + +- Read unencrypted data as-is (backward compatibility) +- Encrypt new data on write +- Gradually encrypt existing data as records are updated -For JSON data, `EncryptedJSONField` encrypts the entire JSON structure: +**Step 3** (Optional): Force immediate encryption with a data migration: ```python -from sentry.db.models.fields import EncryptedJSONField +from sentry.new_migrations.migrations import CheckedMigration +from sentry.utils.query import RangeQuerySetWrapperWithProgressBar + +def encrypt_existing_data(apps, schema_editor): + MyModel = apps.get_model("myapp", "MyModel") + for instance in RangeQuerySetWrapperWithProgressBar(MyModel.objects.all()): + instance.save(update_fields=["api_key"]) + +class Migration(CheckedMigration): + is_post_deployment = True + + dependencies = [ + ("myapp", "0002_alter_mymodel_api_key"), + ] -class Integration(models.Model): - config = EncryptedJSONField() - # Store complex data structures securely + operations = [ + migrations.RunPython(encrypt_existing_data, migrations.RunPython.noop), + ] ``` -**Storage structure**: The encrypted data is wrapped in a JSON object for database compatibility: +## [Key Rotation](#key-rotation) -```json -{ - "sentry_encrypted_field_value": "enc:fernet:key_id:base64_encrypted_data" +To rotate encryption keys: + +1. Generate a new key and add it to the keys directory +2. Update `fernet_primary_key_id` to point to the new key +3. New/updated data will use the new key +4. Old data can still be decrypted with previous keys + +```python +# Before rotation +DATABASE_ENCRYPTION_SETTINGS = { + "fernet_keys_location": "/path/to/keys", + "fernet_primary_key_id": "key_2024_01" +} + +# After rotation +DATABASE_ENCRYPTION_SETTINGS = { + "fernet_keys_location": "/path/to/keys", + "fernet_primary_key_id": "key_2024_02" # New key } ``` -This allows: +Data will be gradually re-encrypted as records are updated. + +## [Important Notes](#important-notes) + +### [Querying](#querying) -- True jsonb storage in PostgreSQL -- Easy identification of encrypted vs unencrypted data -- Backward compatibility with unencrypted JSON +You **cannot** query encrypted field values directly: -## Security Considerations +```python +# This will NOT work +MyModel.objects.filter(secret="my-value") # Won't find encrypted data +``` -### When to Use Encrypted Fields +If you need to query by these fields, consider keeping a separate hash field: + +```python +class MyModel(models.Model): + secret = EncryptedCharField() + secret_hash = models.CharField(max_length=64, db_index=True) + + def save(self, *args, **kwargs): + if self.secret: + self.secret_hash = hashlib.sha256(self.secret.encode()).hexdigest() + super().save(*args, **kwargs) + +# Query by hash +MyModel.objects.filter(secret_hash=hashlib.sha256(b"my-value").hexdigest()) +``` + +### [What to Encrypt](#what-to-encrypt) Use encrypted fields for: -- API keys and tokens -- Webhook secrets -- OAuth credentials -- Personal identifiable information (PII) -- Any sensitive data that should not be stored in plain text +- API keys, tokens, and secrets +- Passwords and credentials +- OAuth tokens and refresh tokens +- PII when required by compliance -### Limitations +Don't encrypt: - - Encrypted fields **cannot** be used in: - WHERE clauses - ORDER BY statements - - GROUP BY statements - Database indexes - Full-text search - JSON path - queries (for `EncryptedJSONField`) - +- Data you need to query or filter on +- High-volume, low-sensitivity data +- Data already encrypted at rest + +### [Key Management](#key-management) -### Performance Considerations +- **Never commit keys to version control** +- Keys are stored as Kubernetes secrets and mounted to pods +- Use different keys for different environments +- Keep all historical keys - they're needed to decrypt old data +- Rotate keys periodically (recommended: annually) -- **Encryption cost**: Minimal overhead on save operations -- **Decryption cost**: Happens on every field access -- **No caching**: Consider application-level caching for frequently accessed data -- **Query restrictions**: Cannot perform database-level filtering or sorting +## [Troubleshooting](#troubleshooting) -## Migration Strategy +**Problem**: `ValueError: Fernet primary key ID is not configured` -### Adding Encryption to Existing Fields +**Solution**: Set `DATABASE_ENCRYPTION_SETTINGS["fernet_primary_key_id"]` in your configuration. -To encrypt an existing field: +--- -1. **Create a migration** changing the field type: +**Problem**: `ValueError: Encryption key with ID 'key_id' not found` -```python {filename:migrations/0001_encrypt_api_key.py} -from django.db import migrations -from sentry.db.models.fields import EncryptedCharField +**Solution**: Add the missing key file to the keys directory or update the configuration. + +--- + +**Problem**: Data is not being encrypted + +**Solution**: Verify `database.encryption.method` is set to `"fernet"`, not `"plaintext"`. + +--- + +**Problem**: Migration takes too long on large tables + +**Solution**: Use a post-deployment data migration with `RangeQuerySetWrapperWithProgressBar`. + +## [How It Works](#how-it-works) + +Encrypted data is stored with a marker prefix that identifies the encryption method: -class Migration(migrations.Migration): - operations = [ - migrations.AlterField( - model_name='integrationconfig', - name='api_key', - field=EncryptedCharField(), - ), - ] +``` +Plaintext: enc:plaintext:{base64_data} +Fernet: enc:fernet:{key_id}:{base64_encrypted_data} ``` -2. **Deploy**: The field will start reading existing unencrypted data -3. **Automatic encryption**: Data gets encrypted on next save -4. **No downtime**: Backward compatibility ensures smooth transition +The key ID in Fernet format enables key rotation—old data encrypted with previous keys can still be decrypted. - - You can switch to encrypted fields without downtime. The fields automatically - handle both encrypted and unencrypted data, encrypting values as they're - updated. - +For `EncryptedJSONField`, the encrypted value is wrapped in a JSON object: + +```json +{ + "sentry_encrypted_field_value": "enc:fernet:key_2024_01:gAAAAABh..." +} +``` + +This allows the field to distinguish encrypted from unencrypted data during migrations and maintain backward compatibility. From dc3363b50da2b95be6ea1ed4acbb5eea74d2b6a2 Mon Sep 17 00:00:00 2001 From: Vjeran Grozdanic Date: Fri, 2 Jan 2026 12:26:07 +0100 Subject: [PATCH 4/4] remove title --- .../backend/application-domains/encrypted-fields/index.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/develop-docs/backend/application-domains/encrypted-fields/index.mdx b/develop-docs/backend/application-domains/encrypted-fields/index.mdx index e9d2c5c29f16d5..38f3867b12d6b5 100644 --- a/develop-docs/backend/application-domains/encrypted-fields/index.mdx +++ b/develop-docs/backend/application-domains/encrypted-fields/index.mdx @@ -8,8 +8,6 @@ categories: sidebar_order: 10 --- -# Encrypted Fields - Sentry provides encrypted database field types for storing sensitive data securely. The encryption system uses Fernet symmetric encryption with support for key rotation and backward compatibility. ## [Available Field Types](#available-field-types)