feat: use hypercore for TA tables #327

joseph-sentry · 2025-07-16T18:48:14Z

Fixes CCMRG-1370

we're enabling Timescale's hypercore feature for the TA testrun hypertable and
the testrun summary and testrun branch summary CAs. We're doing this for the
space savings, since we're not really doing point queries on the testruns table.

We're segmenting by repoid and testid because we will almost always filter on
repo_id if we're running queries against any TA table, and I'm choosing to also
segment by test_id since we're likely to group by the test_id and it'd be likely
for multiple different tests to all run around the same time, which I suspect
would reduce the effectiveness of the compression for the computed_name and
test_id columns, since we'd have to store a bunch of different values instead
of just one or two for a given chunk.

linear · 2025-07-16T18:48:54Z

CCMRG-1370 Add columnar compression to TA hypertables and CAs

codecov · 2025-07-16T18:57:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.28%. Comparing base (45e74cc) to head (f3fef84).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #327   +/-   ##
=======================================
  Coverage   94.28%   94.28%           
=======================================
  Files        1247     1248    +1     
  Lines       45911    45916    +5     
  Branches     1455     1455           
=======================================
+ Hits        43289    43294    +5     
  Misses       2317     2317           
  Partials      305      305

Flag	Coverage Δ
sharedintegration	`40.00% <0.00%> (-0.02%)`	⬇️
sharedunit	`88.72% <100.00%> (+<0.01%)`	⬆️
workerintegration	`61.55% <ø> (ø)`
workerunit	`90.53% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codecov-notifications · 2025-07-16T18:58:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

codspeed-hq · 2025-07-16T18:58:42Z

CodSpeed Performance Report

Merging #327 will not alter performance

_{Comparing joseph/hypercore (f3fef84) with main (45e74cc)}

Summary

✅ 9 untouched benchmarks

rohan-at-sentry · 2025-07-17T14:39:20Z

@sentry review

ci-tests.docker-compose.yml

  timescale:
-    image: timescale/timescaledb-ha:pg14-latest
+    image: timescale/timescaledb:latest-pg14


docker-compose.yml

  timescale:
-    image: timescale/timescaledb-ha:pg14-latest
+    image: timescale/timescaledb:latest-pg14


libs/shared/shared/django_apps/ta_timeseries/migrations/0018_create_repo_summary_caggs.py

+            FROM ta_timeseries_testrun
+            WHERE branch IN ('main', 'master', 'develop')


libs/shared/shared/django_apps/ta_timeseries/migrations/0018_create_repo_summary_caggs.py

+            """
+            SELECT add_continuous_aggregate_policy(
+                'ta_timeseries_aggregate_hourly',
+                start_offset => '2 hours',
+                end_offset => NULL,
+                schedule_interval => INTERVAL '1 hour'
+            );


libs/shared/shared/django_apps/ta_timeseries/migrations/0019_refresh_repo_summary_caggs.py

+        RiskyRunSQL(
+            """
+            CALL refresh_continuous_aggregate(
+                'ta_timeseries_aggregate_hourly',
+                NOW() - INTERVAL '60 days',
+                NULL


libs/shared/shared/django_apps/ta_timeseries/migrations/0020_add_compression_policies.py

+            """
+            ALTER TABLE ta_timeseries_testrun SET (
+                timescaledb.enable_columnstore,
+                timescaledb.segmentby = 'repo_id, test_id',
+                timescaledb.orderby = 'timestamp DESC'
+            );
+            """,


we're enabling Timescale's hypercore feature for the TA testrun hypertable and the testrun summary and testrun branch summary CAs. We're doing this for the space savings, since we're not really doing point queries on the testruns table. We're segmenting by repoid and branch because we will almost always filter on repo_id if we're running queries against any TA table, and I'm choosing to also segment by branch because we'll be filtering by branch in some of the CAs and when we're reading raw data from the table in the API. For the orderby section I'm choosing to order by testsuite, classname, name then timestamp because i want to have similar tests grouped together in chunks so we can make the most of the compression and also because we're grouping by those fields in the computations for the CAs. the old image is 2 years old and is using timescale 2.10.2. We want to use the hypercore API so we have to have at least 2.18.0, so I'm updating the image we're using so CI works.

joseph-sentry · 2025-07-30T14:17:48Z

@sentry review

ci-tests.docker-compose.yml

  timescale:
-    image: timescale/timescaledb-ha:pg14-latest
+    image: timescale/timescaledb:latest-pg14


docker-compose.yml

  timescale:
-    image: timescale/timescaledb-ha:pg14-latest
+    image: timescale/timescaledb:latest-pg14


libs/shared/shared/django_apps/ta_timeseries/migrations/0024_add_compression_policies.py

+        RiskyRunSQL(
+            """
+            CALL add_columnstore_policy('ta_timeseries_testrun', INTERVAL '7 days');
+            """,
+            reverse_sql="CALL remove_columnstore_policy('ta_timeseries_testrun');",
+        ),


libs/shared/shared/django_apps/ta_timeseries/migrations/0024_add_compression_policies.py

+
+class Migration(migrations.Migration):


seer-by-sentry · 2025-07-30T14:28:01Z

libs/shared/shared/django_apps/ta_timeseries/migrations/0024_add_compression_policies.py

+            ALTER TABLE ta_timeseries_testrun SET (
+                timescaledb.enable_columnstore,
+                timescaledb.segmentby = 'repo_id, branch',
+                timescaledb.orderby = 'testsuite, classname, name, timestamp DESC'
+            );
+            """,
+            reverse_sql="ALTER TABLE ta_timeseries_testrun SET (timescaledb.enable_columnstore = false);",
+        ),
+        RiskyRunSQL(
+            """
+            CALL add_columnstore_policy('ta_timeseries_testrun', INTERVAL '7 days');
+            """,
+            reverse_sql="CALL remove_columnstore_policy('ta_timeseries_testrun');",


Potential bug: Migration may fail due to TimescaleDB version incompatibility with PostgreSQL 14 in the new Docker image, causing deployment failure.

Description: The migration attempts to enable TimescaleDB hypercore columnstore features and use functions like add_columnstore_policy() and parameters like timescaledb.enable_columnstore. These features require TimescaleDB 2.18.0+. However, the pull request changes the Docker image to timescale/timescaledb:latest-pg14. Recent TimescaleDB versions (2.20.0+) no longer officially support PostgreSQL 14. This incompatibility means the required TimescaleDB functions or parameters may not be available in the deployed environment, leading to ProgrammingError or similar database errors during migration execution. This will cause deployment failures and service outages.

Suggested fix: Verify the exact TimescaleDB version in timescale/timescaledb:latest-pg14 and ensure it supports both PostgreSQL 14 and the required hypercore columnstore features. If not, update the Docker image to a compatible TimescaleDB version that supports PostgreSQL 15+ or adjust the migration to features compatible with the existing environment.
_{severity: 0.9, confidence: 0.75}

_{Did we get this right? 👍 / 👎 to inform future reviews.}

joseph-sentry force-pushed the joseph/hypercore branch from 91a0c74 to 2965849 Compare July 16, 2025 19:19

This comment has been minimized.

Sign in to view

seer-by-sentry bot reviewed Jul 17, 2025

View reviewed changes

ci-tests.docker-compose.yml

Comment on lines 40 to +41

timescale:

image: timescale/timescaledb-ha:pg14-latest

image: timescale/timescaledb:latest-pg14

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Jul 17, 2025

View reviewed changes

docker-compose.yml

Comment on lines 146 to +147

timescale:

image: timescale/timescaledb-ha:pg14-latest

image: timescale/timescaledb:latest-pg14

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Jul 17, 2025

View reviewed changes

libs/shared/shared/django_apps/ta_timeseries/migrations/0018_create_repo_summary_caggs.py Outdated

Comment on lines 63 to 64

FROM ta_timeseries_testrun

WHERE branch IN ('main', 'master', 'develop')

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Jul 17, 2025

View reviewed changes

joseph-sentry force-pushed the joseph/hypercore branch from 2965849 to b7054a0 Compare July 24, 2025 19:01

joseph-sentry force-pushed the joseph/hypercore branch from b7054a0 to f3fef84 Compare July 30, 2025 14:10

joseph-sentry marked this pull request as ready for review July 30, 2025 14:17

joseph-sentry requested a review from a team as a code owner July 30, 2025 14:17

seer-by-sentry bot reviewed Jul 30, 2025

View reviewed changes

ci-tests.docker-compose.yml

Comment on lines 40 to +41

timescale:

image: timescale/timescaledb-ha:pg14-latest

image: timescale/timescaledb:latest-pg14

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Jul 30, 2025

View reviewed changes

docker-compose.yml

Comment on lines 146 to +147

timescale:

image: timescale/timescaledb-ha:pg14-latest

image: timescale/timescaledb:latest-pg14

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Jul 30, 2025

View reviewed changes

libs/shared/shared/django_apps/ta_timeseries/migrations/0024_add_compression_policies.py

Comment on lines +5 to +6

class Migration(migrations.Migration):

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Jul 30, 2025

View reviewed changes

		FROM ta_timeseries_testrun
		WHERE branch IN ('main', 'master', 'develop')

feat: use hypercore for TA tables #327

Are you sure you want to change the base?

feat: use hypercore for TA tables #327

Uh oh!

Conversation

joseph-sentry commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear bot commented Jul 16, 2025

Uh oh!

codecov bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codecov-notifications bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #327 will not alter performance

Summary

Uh oh!

rohan-at-sentry commented Jul 17, 2025

Uh oh!

This comment has been minimized.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

joseph-sentry commented Jul 30, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

seer-by-sentry bot Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joseph-sentry commented Jul 16, 2025 •

edited

Loading

codecov bot commented Jul 16, 2025 •

edited

Loading

codecov-notifications bot commented Jul 16, 2025 •

edited

Loading

codspeed-hq bot commented Jul 16, 2025 •

edited

Loading