Skip to content

Commit 7932589

Browse files
StuMasonStuMason
authored
feat: v1.1.0 - Automatic Background Sync with Full Audit Trail (#12)
* feat: v1.1.0 - Automatic Background Sync with Full Audit Trail ## Added **Automatic Background Sync** - Smart sync scheduler with APScheduler for automatic background syncing - Rate-limit-aware orchestration respecting Polar API limits (15-min and 24-hour windows) - Priority queue system for efficient multi-user sync: - CRITICAL: Users who haven't synced in 48h+ or have expiring tokens - HIGH: Active users, hasn't synced in 12h+ - NORMAL: Regular users, hasn't synced in 24h+ - LOW: Dormant users, hasn't synced in 7d+ - Comprehensive `SyncLog` model for complete audit trail of every sync operation - Consistent error classification with `SyncErrorType` enum covering: - Authentication errors (TOKEN_EXPIRED, TOKEN_INVALID, TOKEN_REVOKED) - Rate limiting (RATE_LIMITED_15M, RATE_LIMITED_24H) - API errors (API_UNAVAILABLE, API_TIMEOUT, API_ERROR) - Data errors (INVALID_RESPONSE, TRANSFORM_ERROR) - Internal errors (DATABASE_ERROR, INTERNAL_ERROR) - Post-sync analytics: Automatic baseline recalculation and pattern detection **Configuration** - `SYNC_ENABLED`: Enable/disable automatic syncing (default: true) - `SYNC_INTERVAL_MINUTES`: Sync cycle interval (default: 60) - `SYNC_ON_STARTUP`: Run sync immediately on startup (default: true) - `SYNC_MAX_USERS_PER_RUN`: Maximum users per sync cycle (default: rate-limit aware) - `SYNC_STAGGER_SECONDS`: Delay between user syncs (default: 5) **Database** - New `sync_logs` table with comprehensive fields for audit and debugging - Composite indexes for efficient querying by user, status, and error type Stu Mason + AI <me@stumason.dev> * feat(admin): Update dashboard with sync status, history and biosensing - Add sync scheduler status section with running state, next run time, 24h stats - Add recent sync history table showing last 10 sync attempts - Add biosensing data counts (SpO2, ECG, Temperature) - Add analytics counts (Baselines, Patterns) - Fix alertness scale display from /5 to /10 (Polar API uses 0-10 scale) - Fix migration: Remove invalid 'comment' params from create_index calls Stu Mason + AI <me@stumason.dev> * style: Format routes.py Stu Mason + AI <me@stumason.dev> * fix: Address PR review - update last_synced_at and fix API call counting Critical fixes from code review: - Update User.last_synced_at after successful sync (priority queue now works) - Fix API call counting: count data types with records, not record counts - Clarify rate limit tracking limitation (requires SDK changes) Note: APScheduler already in pyproject.toml, migration chain is correct. Stu Mason + AI <me@stumason.dev> * fix: Address additional PR review feedback - Add warning log when user not found after sync (prevents silent failure) - Track startup sync task and cancel on shutdown (fixes race condition) - Update .env.example with all new sync settings Note: "SQL injection" concern in review is incorrect - SQLAlchemy parameterizes all queries. The actual concern (XSS) is mitigated by Jinja2 auto-escaping. Stu Mason + AI <me@stumason.dev> * chore: Bump version to 1.1.0 and finalize CHANGELOG Stu Mason + AI <me@stumason.dev>
1 parent 0e5244f commit 7932589

File tree

13 files changed

+2240
-29
lines changed

13 files changed

+2240
-29
lines changed

‎.env.example‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,12 @@ DATABASE_URL=postgresql+asyncpg://polar:polar@postgres:5432/polar
1717
# JWT_SECRET=<secret-key>
1818

1919
# Sync settings
20-
SYNC_INTERVAL_HOURS=1
21-
SYNC_ON_STARTUP=true
22-
SYNC_DAYS_LOOKBACK=30
20+
SYNC_ENABLED=true # Enable/disable automatic background syncing
21+
SYNC_INTERVAL_MINUTES=60 # Minutes between sync cycles (default: 60)
22+
SYNC_ON_STARTUP=false # Run sync immediately on startup (default: false)
23+
SYNC_MAX_USERS_PER_RUN=10 # Max users per sync cycle (default: 10)
24+
SYNC_STAGGER_SECONDS=5 # Delay between user syncs (default: 5)
25+
SYNC_DAYS_LOOKBACK=30 # Days of history to sync
2326

2427
# Logging
2528
LOG_LEVEL=INFO

‎CHANGELOG.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,53 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
---
11+
12+
## [1.1.0] - 2026-01-13
13+
14+
### Added
15+
16+
**Automatic Background Sync**
17+
- Smart sync scheduler with APScheduler for automatic background syncing
18+
- Rate-limit-aware orchestration respecting Polar API limits (15-min and 24-hour windows)
19+
- Priority queue system for efficient multi-user sync:
20+
- CRITICAL: Users who haven't synced in 48h+ or have expiring tokens
21+
- HIGH: Active users, hasn't synced in 12h+
22+
- NORMAL: Regular users, hasn't synced in 24h+
23+
- LOW: Dormant users, hasn't synced in 7d+
24+
- Comprehensive `SyncLog` model for complete audit trail of every sync operation
25+
- Consistent error classification with `SyncErrorType` enum covering:
26+
- Authentication errors (TOKEN_EXPIRED, TOKEN_INVALID, TOKEN_REVOKED)
27+
- Rate limiting (RATE_LIMITED_15M, RATE_LIMITED_24H)
28+
- API errors (API_UNAVAILABLE, API_TIMEOUT, API_ERROR)
29+
- Data errors (INVALID_RESPONSE, TRANSFORM_ERROR)
30+
- Internal errors (DATABASE_ERROR, INTERNAL_ERROR)
31+
- Post-sync analytics: Automatic baseline recalculation and pattern detection
32+
33+
**Admin Dashboard**
34+
- Sync scheduler status section (running state, next run, 24h stats)
35+
- Recent sync history table with status, duration, records synced
36+
- Biosensing data counts (SpO2, ECG, Temperature)
37+
- Analytics counts (Baselines, Patterns)
38+
39+
**Configuration**
40+
- `SYNC_ENABLED`: Enable/disable automatic syncing (default: true)
41+
- `SYNC_INTERVAL_MINUTES`: Sync cycle interval (default: 60)
42+
- `SYNC_ON_STARTUP`: Run sync immediately on startup (default: false)
43+
- `SYNC_MAX_USERS_PER_RUN`: Maximum users per sync cycle (default: 10)
44+
- `SYNC_STAGGER_SECONDS`: Delay between user syncs (default: 5)
45+
46+
**Database**
47+
- New `sync_logs` table with comprehensive fields for audit and debugging
48+
- Composite indexes for efficient querying by user, status, and error type
49+
50+
### Fixed
51+
- Alertness scale display corrected from /5 to /10 (Polar API uses 0-10 scale)
52+
53+
---
54+
855
## [1.0.0] - 2025-01-13
956

1057
First stable release of polar-flow-server - a self-hosted health analytics server for Polar devices.
@@ -93,6 +140,7 @@ First stable release of polar-flow-server - a self-hosted health analytics serve
93140
- Database models for Polar data types
94141
- Sync service foundation
95142

143+
[1.1.0]: https://github.com/StuMason/polar-flow-server/compare/v1.0.0...v1.1.0
96144
[1.0.0]: https://github.com/StuMason/polar-flow-server/compare/v0.2.0...v1.0.0
97145
[0.2.0]: https://github.com/StuMason/polar-flow-server/compare/v0.1.0...v0.2.0
98146
[0.1.0]: https://github.com/StuMason/polar-flow-server/releases/tag/v0.1.0

‎alembic/versions/f7g8h9i0j1k2_add_sync_logs_table.py‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
"""Add sync_logs table for comprehensive sync audit trail.
2+
3+
Revision ID: f7g8h9i0j1k2
4+
Revises: e6f7g8h9i0j1
5+
Create Date: 2026-01-13 15:00:00.000000
6+
7+
"""
8+
9+
from collections.abc import Sequence
10+
11+
import sqlalchemy as sa
12+
from sqlalchemy.dialects.postgresql import JSON
13+
14+
from alembic import op
15+
16+
# revision identifiers, used by Alembic.
17+
revision: str = "f7g8h9i0j1k2"
18+
down_revision: str | None = "e6f7g8h9i0j1"
19+
branch_labels: str | Sequence[str] | None = None
20+
depends_on: str | Sequence[str] | None = None
21+
22+
23+
def upgrade() -> None:
24+
"""Create sync_logs table for complete sync audit trail."""
25+
op.create_table(
26+
"sync_logs",
27+
# Primary key
28+
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
29+
# User identification
30+
sa.Column(
31+
"user_id",
32+
sa.String(255),
33+
nullable=False,
34+
index=True,
35+
comment="User being synced (Polar user ID or Laravel UUID)",
36+
),
37+
sa.Column(
38+
"job_id",
39+
sa.String(36),
40+
nullable=False,
41+
index=True,
42+
comment="UUID for correlating logs across services",
43+
),
44+
# Timing
45+
sa.Column(
46+
"started_at",
47+
sa.DateTime(timezone=True),
48+
nullable=False,
49+
server_default=sa.func.now(),
50+
index=True,
51+
comment="When sync began",
52+
),
53+
sa.Column(
54+
"completed_at",
55+
sa.DateTime(timezone=True),
56+
nullable=True,
57+
comment="When sync finished (null if still running)",
58+
),
59+
sa.Column(
60+
"duration_ms",
61+
sa.Integer(),
62+
nullable=True,
63+
comment="Total sync duration in milliseconds",
64+
),
65+
# Status
66+
sa.Column(
67+
"status",
68+
sa.String(20),
69+
nullable=False,
70+
server_default="started",
71+
index=True,
72+
comment="Status: started, success, partial, failed, skipped",
73+
),
74+
# Error tracking
75+
sa.Column(
76+
"error_type",
77+
sa.String(50),
78+
nullable=True,
79+
index=True,
80+
comment="Categorized error type (token_expired, rate_limited_15m, etc.)",
81+
),
82+
sa.Column(
83+
"error_message",
84+
sa.Text(),
85+
nullable=True,
86+
comment="Human-readable error description",
87+
),
88+
sa.Column(
89+
"error_details",
90+
JSON(),
91+
nullable=True,
92+
comment="Full error context as JSON",
93+
),
94+
# Results
95+
sa.Column(
96+
"records_synced",
97+
JSON(),
98+
nullable=True,
99+
comment="Count of records synced per data type",
100+
),
101+
sa.Column(
102+
"api_calls_made",
103+
sa.Integer(),
104+
nullable=False,
105+
server_default="0",
106+
comment="Total API calls made during sync",
107+
),
108+
# Rate limit tracking
109+
sa.Column(
110+
"rate_limit_remaining_15m",
111+
sa.Integer(),
112+
nullable=True,
113+
comment="Remaining 15-min quota after sync",
114+
),
115+
sa.Column(
116+
"rate_limit_remaining_24h",
117+
sa.Integer(),
118+
nullable=True,
119+
comment="Remaining 24-hour quota after sync",
120+
),
121+
sa.Column(
122+
"rate_limit_limit_15m",
123+
sa.Integer(),
124+
nullable=True,
125+
comment="Max requests in 15-min window",
126+
),
127+
sa.Column(
128+
"rate_limit_limit_24h",
129+
sa.Integer(),
130+
nullable=True,
131+
comment="Max requests in 24-hour window",
132+
),
133+
# Analytics follow-up
134+
sa.Column(
135+
"baselines_recalculated",
136+
sa.Boolean(),
137+
nullable=False,
138+
server_default="false",
139+
comment="Whether baselines were updated after sync",
140+
),
141+
sa.Column(
142+
"patterns_detected",
143+
sa.Boolean(),
144+
nullable=False,
145+
server_default="false",
146+
comment="Whether pattern detection ran after sync",
147+
),
148+
sa.Column(
149+
"insights_generated",
150+
sa.Boolean(),
151+
nullable=False,
152+
server_default="false",
153+
comment="Whether insights were regenerated after sync",
154+
),
155+
# Context
156+
sa.Column(
157+
"trigger",
158+
sa.String(20),
159+
nullable=False,
160+
server_default="manual",
161+
comment="What triggered sync: scheduler, manual, webhook, startup",
162+
),
163+
sa.Column(
164+
"priority",
165+
sa.String(20),
166+
nullable=True,
167+
comment="Sync priority: critical, high, normal, low",
168+
),
169+
comment="Complete audit trail of every sync attempt",
170+
)
171+
172+
# Composite indexes for common queries
173+
op.create_index(
174+
"ix_sync_logs_user_started",
175+
"sync_logs",
176+
["user_id", "started_at"],
177+
)
178+
op.create_index(
179+
"ix_sync_logs_status_started",
180+
"sync_logs",
181+
["status", "started_at"],
182+
)
183+
op.create_index(
184+
"ix_sync_logs_error_type_started",
185+
"sync_logs",
186+
["error_type", "started_at"],
187+
)
188+
189+
190+
def downgrade() -> None:
191+
"""Drop sync_logs table and indexes."""
192+
op.drop_index("ix_sync_logs_error_type_started", table_name="sync_logs")
193+
op.drop_index("ix_sync_logs_status_started", table_name="sync_logs")
194+
op.drop_index("ix_sync_logs_user_started", table_name="sync_logs")
195+
op.drop_table("sync_logs")

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "polar-flow-server"
3-
version = "1.0.0"
3+
version = "1.1.0"
44
description = "Self-hosted health analytics server for Polar devices"
55
readme = "README.md"
66
authors = [

0 commit comments

Comments
 (0)