Skip to content

Commit c28e8ed

Browse files
alcoalco
authored
Replace direct telemetry deps with the electric_telemetry library (#3412)
Extract telemetry code from Electric into a separate package. On top of that, the new package accepts additional user-provided periodic measurements and metric definitions for otel/prometheus exporters; both exporters now use the same list of metrics. One thing that's been lost in this move is the `stack_id` sentry tag inside the CallHomeRepoter process. Sentry is a dependency of Electric and there's no straightforward way of propagating it into CallHomeReporter which is a module in the electric_telemetry dependency now. I don't feel like it's a big loss because we set stack_id in the logger metadata for that process, as well as mention it in the process name and label. With this merged, we can delete the electric-sql/electric-telemetry repo.
1 parent 52be1e2 commit c28e8ed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+1810
-1477
lines changed

‎.buildkite/docker-image.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ steps:
1818
- export ELECTRIC_VERSION=$(git describe --abbrev=7 --tags --always --first-parent --match '@core/sync-service@*' | sed -En 's|^@core/sync-service@||p')
1919
- export SHORT_COMMIT_SHA=$(git rev-parse --short HEAD)
2020
- docker buildx build --platform linux/arm64/v8,linux/amd64 --push
21+
--build-context electric-telemetry=../electric-telemetry
2122
--build-arg ELECTRIC_VERSION=$${ELECTRIC_VERSION}
2223
-t $${ELECTRIC_IMAGE_NAME}:canary
2324
-t $${ELECTRIC_CANARY_IMAGE_NAME}:$${SHORT_COMMIT_SHA}
@@ -31,6 +32,7 @@ steps:
3132
- cd ./packages/sync-service
3233
- export ELECTRIC_VERSION=$(jq '.version' -r package.json)
3334
- docker buildx build --platform linux/arm64/v8,linux/amd64 --push
35+
--build-context electric-telemetry=../electric-telemetry
3436
--build-arg ELECTRIC_VERSION=$${ELECTRIC_VERSION}
3537
-t $${ELECTRIC_IMAGE_NAME}:$${ELECTRIC_VERSION}
3638
-t $${ELECTRIC_IMAGE_NAME}:latest

‎.changeset/eleven-pens-sell.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@core/electric-telemetry': patch
3+
'@core/sync-service': patch
4+
---
5+
6+
Extract telemetry code from Electric into a separate package, for easier modification and sharing of the telemetry code between Electric and Cloud.

‎.github/workflows/autoformat.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ jobs:
4848
elixir_formatting:
4949
strategy:
5050
matrix:
51-
package: [sync-service, elixir-client]
51+
package: [sync-service, elixir-client, electric-telemetry]
5252
name: Elixir formatting and linting
5353
runs-on: ubuntu-latest
5454
defaults:

‎.github/workflows/benchmarking.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ jobs:
6262
--push \
6363
--cache-from ${{ env.REGISTRY }}/electric:canary-builder \
6464
--cache-from ${{ env.REGISTRY }}/electric:pr-${{ github.event.issue.number }}-builder \
65+
--build-context electric-telemetry=../electric-telemetry \
6566
--build-arg BUILDKIT_INLINE_CACHE=1 \
6667
--tag ${{ env.REGISTRY }}/electric:pr-${{ github.event.issue.number }}-builder \
6768
--target builder \

‎.github/workflows/docker_image_smoketest.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,8 @@ jobs:
3636
- uses: docker/build-push-action@v6
3737
with:
3838
context: packages/sync-service
39+
build-contexts: |
40+
electric-telemetry=packages/electric-telemetry
3941
push: false
4042
load: true
4143
tags: 'electric-test-image'

‎.github/workflows/electric_telemetry_tests.yml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
name: Electric Telemetry CI
2+
3+
on:
4+
push:
5+
branches: ['main']
6+
paths-ignore:
7+
- 'website/**'
8+
- '**/README.md'
9+
pull_request:
10+
paths-ignore:
11+
- 'website/**'
12+
- '**/README.md'
13+
14+
permissions:
15+
contents: read
16+
17+
jobs:
18+
build_and_test:
19+
name: Build and test electric-telemetry
20+
runs-on: ubuntu-latest
21+
defaults:
22+
run:
23+
working-directory: packages/electric-telemetry
24+
env:
25+
MIX_ENV: test
26+
MIX_OS_DEPS_COMPILE_PARTITION_COUNT: 4
27+
steps:
28+
- uses: actions/checkout@v4
29+
30+
- uses: erlef/setup-beam@v1
31+
with:
32+
version-type: strict
33+
version-file: '.tool-versions'
34+
35+
- name: Cache electric-telemetry dependencies
36+
uses: actions/cache@v4
37+
with:
38+
path: packages/electric-telemetry/deps
39+
key: ${{ runner.os }}-electric-telemetry-deps-${{ hashFiles('packages/electric-telemetry/mix.lock') }}
40+
restore-keys: |
41+
${{ runner.os }}-electric-telemetry-deps-${{ hashFiles('packages/electric-telemetry/mix.lock') }}
42+
${{ runner.os }}-electric-telemetry-deps-
43+
44+
- name: Cache compiled code
45+
uses: actions/cache@v4
46+
with:
47+
path: |
48+
packages/electric-telemetry/_build/*/lib
49+
!packages/electric-telemetry/_build/*/lib/electric_telemetry
50+
key: "${{ runner.os }}-electric-telemetry-build-${{ env.MIX_ENV }}-${{ hashFiles('packages/sync-service/mix.lock') }}"
51+
restore-keys: |
52+
${{ runner.os }}-electric-telemetry-build-${{ env.MIX_ENV }}-${{ hashFiles('packages/sync-service/mix.lock') }}
53+
${{ runner.os }}-electric-telemetry-build-${{ env.MIX_ENV }}-
54+
${{ runner.os }}-electric-telemetry-build-
55+
56+
- name: Install dependencies
57+
run: mix do deps.get + deps.compile
58+
59+
- name: Compile
60+
# don't bail on compile warnings - let's get the results of the tests
61+
# and let the formatting task check for compilation warnings
62+
run: mix compile
63+
64+
- name: Run tests
65+
run: mix coveralls.json
66+
67+
- name: Upload coverage reports to CodeCov
68+
uses: codecov/codecov-action@ad3126e916f78f00edff4ed0317cf185271ccc2d
69+
with:
70+
token: ${{ secrets.CODECOV_TOKEN }}
71+
flags: elixir,unit-tests,electric-telemetry
72+
files: ./cover/excoveralls.json
73+
74+
- name: Upload test results to CodeCov
75+
uses: codecov/test-results-action@f2dba722c67b86c6caa034178c6e4d35335f6706
76+
if: ${{ !cancelled() }}
77+
env:
78+
DUMMY_COMMIT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}-dummy
79+
with:
80+
token: ${{ secrets.CODECOV_TOKEN }}
81+
fail_ci_if_error: true
82+
flags: elixir,unit-tests,electric-telemetry
83+
files: ./junit/test-junit-report.xml
84+
# Upload all PR test results to single branch - requires overriding branch and commit
85+
override_branch: ${{ github.event_name == 'pull_request' && 'codecov/pr-test-aggregation' || '' }}
86+
override_commit: ${{ github.event_name == 'pull_request' && env.DUMMY_COMMIT_SHA || '' }}

‎.github/workflows/elixir_tests.yml‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -95,26 +95,16 @@ jobs:
9595
${{ runner.os }}-sync-service-build-
9696
9797
- name: Install dependencies
98-
run: |
99-
mix deps.get
100-
mix deps.compile
101-
MIX_TARGET=application mix deps.compile
98+
run: mix do deps.get + deps.compile
10299

103100
- name: Compile
104101
# don't bail on compile warnings - let's get the results of the tests
105102
# and let the formatting task check for compilation warnings
106-
run: |
107-
mix compile
108-
MIX_TARGET=application mix compile
103+
run: mix compile
109104

110105
- name: Run tests
111106
run: mix coveralls.json --include slow --cover --export-coverage test
112107

113-
- name: Run telemetry tests
114-
env:
115-
MIX_TARGET: application
116-
run: mix coveralls.json --only telemetry_target --import-cover cover
117-
118108
- name: Upload coverage reports to CodeCov
119109
uses: codecov/codecov-action@ad3126e916f78f00edff4ed0317cf185271ccc2d
120110
with:

‎packages/electric-telemetry/.dockerignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_build/
2+
deps/
3+
test/

‎packages/electric-telemetry/.formatter.exs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Used by "mix format"
2+
[
3+
inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
4+
]

‎packages/electric-telemetry/.gitignore‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# The directory Mix will write compiled artifacts to.
2+
/_build/
3+
4+
# If you run "mix test --cover", coverage assets end up here.
5+
/cover/
6+
7+
# The directory Mix downloads your dependencies sources to.
8+
/deps/
9+
10+
# Where third-party dependencies like ExDoc output generated docs.
11+
/doc/
12+
13+
# Temporary files, for example, from tests.
14+
/tmp/
15+
16+
# If the VM crashes, it generates a dump, let's ignore it too.
17+
erl_crash.dump
18+
19+
# Also ignore archive artifacts (built via "mix archive.build").
20+
*.ez
21+
22+
# Ignore package tarball (built via "mix hex.build").
23+
electric_telemetry-*.tar
24+

0 commit comments

Comments
 (0)