Skip to content

Commit 533dd35

Browse files
iamh2oiamh2o
authored
Merge pull request #151 from Daylily-Informatics/feature/analytics-mobile-backup-v2
Feature/analytics mobile backup v2
2 parents 7e15ccb + d7e5539 commit 533dd35

24 files changed

+3904
-2
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1594,3 +1594,4 @@ quadrantChart
15941594
*Last Updated: 2024-12-23*
15951595
*Generated for BLOOM LIMS v0.10.12*
15961596

1597+

‎analytics/README.md‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# BLOOM Analytics Dashboard Integration
2+
3+
This directory contains a loosely-coupled analytics solution for BLOOM LIMS and other Laboratory Information Systems (LIS). The integration uses **Metabase**, an open-source business intelligence tool, connected directly to the PostgreSQL database.
4+
5+
## Architecture
6+
7+
```
8+
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
9+
│ BLOOM LIMS │────▶│ PostgreSQL │◀────│ Metabase │
10+
│ (FastAPI) │ │ Database │ │ (Analytics) │
11+
└─────────────────┘ └─────────────────┘ └─────────────────┘
12+
│
13+
▼
14+
┌─────────────────┐
15+
│ SQL Views │
16+
│ (Analytics) │
17+
└─────────────────┘
18+
```
19+
20+
## Why Metabase?
21+
22+
| Feature | Benefit for Labs |
23+
|---------|------------------|
24+
| Open Source (AGPL) | Free, no vendor lock-in |
25+
| PostgreSQL Native | Direct connection to BLOOM database |
26+
| Visual Query Builder | Non-technical users can create reports |
27+
| Embeddable | Dashboard embedding in BLOOM UI |
28+
| Alerting | Automated notifications for thresholds |
29+
| API | Programmatic access to dashboards |
30+
31+
## Quick Start
32+
33+
### Option 1: Docker (Recommended)
34+
35+
```bash
36+
# Start Metabase with Docker
37+
cd analytics/metabase
38+
docker-compose up -d
39+
40+
# Access Metabase at http://localhost:3000
41+
```
42+
43+
### Option 2: Standalone JAR
44+
45+
```bash
46+
# Download and run Metabase
47+
./analytics/metabase/run_metabase.sh
48+
```
49+
50+
### Option 3: Conda Environment
51+
52+
```bash
53+
# If using conda, Metabase can run alongside BLOOM
54+
conda activate BLOOM
55+
./analytics/metabase/run_metabase.sh
56+
```
57+
58+
## Configuration
59+
60+
### Environment Variables
61+
62+
Add to your `.env` file:
63+
64+
```bash
65+
# Metabase Configuration
66+
METABASE_PORT=3000
67+
METABASE_DB_TYPE=postgres
68+
METABASE_DB_HOST=localhost
69+
METABASE_DB_PORT=5432
70+
METABASE_DB_NAME=bloom_lims
71+
METABASE_DB_USER=bloom_user
72+
METABASE_DB_PASS=your_password
73+
74+
# Optional: Embed dashboards in BLOOM
75+
METABASE_EMBED_SECRET=your-embed-secret-key
76+
```
77+
78+
## SQL Views for Analytics
79+
80+
The `sql_views/` directory contains PostgreSQL views optimized for analytics:
81+
82+
| View | Purpose | Key Metrics |
83+
|------|---------|-------------|
84+
| `v_sample_throughput` | Sample processing rates | Samples/day, TAT |
85+
| `v_workflow_bottlenecks` | Queue analysis | Wait times, backlogs |
86+
| `v_equipment_utilization` | Equipment usage | Utilization %, downtime |
87+
| `v_turnaround_times` | TAT analysis | Mean, median, percentiles |
88+
| `v_audit_activity` | User activity | Actions/user, peak times |
89+
| `v_object_counts` | Inventory | Counts by type/status |
90+
91+
### Installing Views
92+
93+
```bash
94+
# Run the analytics views installation script
95+
psql -d bloom_lims -f analytics/sql_views/install_views.sql
96+
```
97+
98+
## Sample Dashboards
99+
100+
Pre-built dashboard configurations are in `dashboards/`:
101+
102+
1. **Operations Overview** - Daily sample counts, active workflows
103+
2. **Turnaround Time Monitor** - TAT trends, SLA compliance
104+
3. **Equipment Dashboard** - Utilization, maintenance schedules
105+
4. **Quality Metrics** - Error rates, reprocessing trends
106+
5. **User Activity** - Audit logs, action summaries
107+
108+
## Portability to Other LIS
109+
110+
This analytics solution is designed to be portable:
111+
112+
1. **Standardized Views**: SQL views use common LIS terminology
113+
2. **Configurable Mappings**: `config/lis_mappings.yaml` maps BLOOM fields to standard names
114+
3. **API Abstraction**: Dashboard definitions reference abstract metrics
115+
4. **Documentation**: Each view includes comments for adaptation
116+
117+
### Adapting for Another LIS
118+
119+
1. Copy `analytics/` directory to your LIS project
120+
2. Edit `config/lis_mappings.yaml` to map your schema
121+
3. Modify SQL views to match your database structure
122+
4. Import dashboard configurations into Metabase
123+
124+
## Security Considerations
125+
126+
- Metabase connects read-only to the database (recommended)
127+
- Use a dedicated database user with SELECT-only permissions
128+
- Enable HTTPS for production deployments
129+
- Configure Metabase authentication (LDAP, SSO, etc.)
130+
131+
## Directory Structure
132+
133+
```
134+
analytics/
135+
├── README.md # This file
136+
├── metabase/
137+
│ ├── docker-compose.yml # Docker deployment
138+
│ ├── run_metabase.sh # Standalone runner
139+
│ └── metabase.db/ # Metabase config (gitignored)
140+
├── dashboards/
141+
│ ├── operations_overview.json
142+
│ ├── turnaround_times.json
143+
│ └── equipment_utilization.json
144+
├── sql_views/
145+
│ ├── install_views.sql # Main installation script
146+
│ ├── v_sample_throughput.sql
147+
│ ├── v_workflow_bottlenecks.sql
148+
│ └── ...
149+
└── config/
150+
└── lis_mappings.yaml # LIS field mappings
151+
```
152+
153+
## Troubleshooting
154+
155+
### Metabase can't connect to PostgreSQL
156+
- Ensure PostgreSQL is running: `pg_isready`
157+
- Check connection settings in Metabase admin
158+
- Verify firewall allows connection on port 5432
159+
160+
### Views not showing data
161+
- Run `SELECT * FROM v_sample_throughput LIMIT 5;` to test
162+
- Check that BLOOM has data in the database
163+
- Verify view permissions for Metabase user
164+
165+
## License
166+
167+
This analytics integration is released under the same MIT license as BLOOM LIMS.
168+

‎analytics/config/lis_mappings.yaml‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# BLOOM LIMS to Standard LIS Terminology Mappings
2+
# This file enables portability of analytics dashboards across different LIS systems
3+
#
4+
# Usage: Edit the 'source_column' values to match your LIS database schema
5+
# The 'standard_name' values should remain unchanged for dashboard compatibility
6+
7+
version: "1.0"
8+
lis_name: "BLOOM LIMS"
9+
database_type: "postgresql"
10+
11+
# Core object mappings
12+
objects:
13+
sample:
14+
source_table: "generic_instance"
15+
standard_name: "sample"
16+
columns:
17+
id: "uuid"
18+
display_id: "euid"
19+
name: "name"
20+
type: "btype"
21+
subtype: "b_sub_type"
22+
status: "bstatus"
23+
created_date: "created_dt"
24+
modified_date: "modified_dt"
25+
is_deleted: "is_deleted"
26+
properties: "json_addl"
27+
filters:
28+
- column: "super_type"
29+
values: ["content", "container"]
30+
31+
workflow:
32+
source_table: "generic_instance"
33+
standard_name: "workflow"
34+
columns:
35+
id: "uuid"
36+
display_id: "euid"
37+
name: "name"
38+
type: "btype"
39+
status: "bstatus"
40+
created_date: "created_dt"
41+
filters:
42+
- column: "super_type"
43+
values: ["workflow"]
44+
45+
workflow_step:
46+
source_table: "generic_instance"
47+
standard_name: "workflow_step"
48+
columns:
49+
id: "uuid"
50+
display_id: "euid"
51+
name: "name"
52+
type: "btype"
53+
step_type: "b_sub_type"
54+
status: "bstatus"
55+
filters:
56+
- column: "super_type"
57+
values: ["workflow_step"]
58+
59+
equipment:
60+
source_table: "generic_instance"
61+
standard_name: "equipment"
62+
columns:
63+
id: "uuid"
64+
display_id: "euid"
65+
name: "name"
66+
type: "btype"
67+
model: "b_sub_type"
68+
status: "bstatus"
69+
location: "json_addl->>'location'"
70+
serial_number: "json_addl->>'serial_number'"
71+
filters:
72+
- column: "super_type"
73+
values: ["equipment"]
74+
75+
# Relationship mappings
76+
relationships:
77+
lineage:
78+
source_table: "generic_instance_lineage"
79+
columns:
80+
id: "uuid"
81+
parent_id: "parent_instance_uuid"
82+
child_id: "child_instance_uuid"
83+
parent_type: "parent_type"
84+
child_type: "child_type"
85+
relationship_type: "relationship_type"
86+
created_date: "created_dt"
87+
88+
# Audit log mappings
89+
audit:
90+
source_table: "audit_log"
91+
columns:
92+
id: "uuid"
93+
table_name: "rel_table_name"
94+
record_id: "rel_table_uuid_fk"
95+
record_display_id: "rel_table_euid_fk"
96+
user: "changed_by"
97+
timestamp: "changed_at"
98+
operation: "operation_type"
99+
old_value: "old_value"
100+
new_value: "new_value"
101+
column_changed: "column_name"
102+
103+
# Status value mappings (normalize across different LIS)
104+
status_mappings:
105+
standard_statuses:
106+
pending: ["created", "pending", "queued", "waiting"]
107+
in_progress: ["in_progress", "processing", "active", "running"]
108+
complete: ["complete", "completed", "done", "finished"]
109+
failed: ["failed", "error", "rejected"]
110+
cancelled: ["abandoned", "cancelled", "canceled", "voided"]
111+
112+
bloom_to_standard:
113+
created: "pending"
114+
in_progress: "in_progress"
115+
complete: "complete"
116+
failed: "failed"
117+
abandoned: "cancelled"
118+
destroyed: "cancelled"
119+
active: "in_progress"
120+
121+
# Metric definitions for dashboards
122+
metrics:
123+
sample_throughput:
124+
description: "Number of samples processed per time period"
125+
calculation: "COUNT(*) WHERE status = 'complete'"
126+
127+
turnaround_time:
128+
description: "Time from sample receipt to completion"
129+
calculation: "modified_dt - created_dt WHERE status = 'complete'"
130+
unit: "hours"
131+
132+
queue_depth:
133+
description: "Number of items waiting in a queue"
134+
calculation: "COUNT(*) WHERE status IN ('pending', 'in_progress')"
135+
136+
completion_rate:
137+
description: "Percentage of samples successfully completed"
138+
calculation: "(completed / total) * 100"
139+
unit: "percent"
140+
141+
# Time zone configuration
142+
timezone:
143+
database: "UTC"
144+
display: "America/New_York" # Adjust for your lab's timezone
145+

0 commit comments

Comments
 (0)