proposal: new JS interstate object#446
Draft
Mikhail (oo-sleeper) wants to merge 9 commits intodevelopfrom
Draft
Conversation
It looks like all deprecations are only in test code. Pubsub V1 client we use in production source/target code is not deprecated (at least not yet and it's going to be supported only until July 31st 2026, see [this](https://github.com/googleapis/google-cloud-go/blob/main/pubsub/MIGRATING.md) migration guide to V2
* feat: Send heartbeat and alert events to webhook (#428) * Add changes for debug workflow * Separate target for filtered data. * Debug mode flag in HTTP target, which allows us to send original data alongside the transformed one. It's now possible to configure 3 different targets for each: transformed data, failed data and filtered data. By default target for filtered data is the `silent` one - it just acks data. * Precalculate rejection timestamp in HTTP target Before: -Send 2 headers: `Request-Timeout` and `Request-Timestamp`, which can be used by target server to determine if HTTP request should be rejected. Now: -Send 1 header `Rejection-Timestamp` which is precalculated in HTTP target based on configured client timeout and threshold. This allow us to simplify logic in targets (like GTMSS client). -To determine if a request is OK, target doesn't have to do any calculation, simply extract `Rejection-Timestamp` header and compare with a server's timestamp. -This commit also contains little refactoring of HTTP target creation + tests (no more `newHTTPTarget` function, using `HTTPTargetConfigFunction` directly). * Enable data only output for `stdout` target (#410) * enable data only output for stdout target * add missing newline * address comment & add test steps to README * Add context and complete the list of test commands * feat: Add hash function to JQ transformation (#411) * JQ command hash implementation is ready for review * fix linter errors * restrict num of parameters; support sha256 & md5; add tests * apply suggestions * remove duplicated tests * feat: Add golang linter to the project (#412) * fixing code to align with linter rules * fix majority of lint issues; left a few out - those seem bit more complex * fix tests * fix tests * implement suggestions * fix the code as per comments * revert slog introduction; would be done later * update lint cmd in Makefile to handle new linter * update multierror log output * make lint only runs linter; add instructions on how to install linter * Use golangci-lint action in CD workflow * Bump dependencies, use go 1.24 in CI/CD * Extend remove_nulls and add to JS configuration * Add option to pass literal JS script string * feat: Add hash function to JS transformation (#422) * hash func available in JS; code refactor; tests; bypass clean-up * linter errors fixed * support for hash salt JS config attribute; tests * fix tests; clean up the code a bit * fix tests * WIP: outlined the draft, need to define events * now need to figure where to place alerts collection * adjust webhook URL - it is passed in full from Terraform, so no need to hassle with it * tags are mandatory * fixing tags * fix interval attribute value * Add changes for debug workflow * Separate target for filtered data. * Debug mode flag in HTTP target, which allows us to send original data alongside the transformed one. It's now possible to configure 3 different targets for each: transformed data, failed data and filtered data. By default target for filtered data is the `silent` one - it just acks data. * Precalculate rejection timestamp in HTTP target Before: Send 2 headers: `Request-Timeout` and `Request-Timestamp`, which can be used by target server to determine if HTTP request should be rejected. Now: Send 1 header `Rejection-Timestamp` which is precalculated in HTTP target based on configured client timeout and threshold. This allow us to simplify logic in targets (like GTMSS client). To determine if a request is OK, target doesn't have to do any calculation, simply extract `Rejection-Timestamp` header and compare with a server's timestamp. This commit also contains little refactoring of HTTP target creation + tests (no more `newHTTPTarget` function, using `HTTPTargetConfigFunction` directly). * Enable data only output for `stdout` target (#410) * enable data only output for stdout target * add missing newline * address comment & add test steps to README * Add context and complete the list of test commands * feat: Add hash function to JQ transformation (#411) * JQ command hash implementation is ready for review * fix linter errors * restrict num of parameters; support sha256 & md5; add tests * apply suggestions * remove duplicated tests * feat: Add golang linter to the project (#412) * fixing code to align with linter rules * fix majority of lint issues; left a few out - those seem bit more complex * fix tests * fix tests * implement suggestions * fix the code as per comments * revert slog introduction; would be done later * update lint cmd in Makefile to handle new linter * update multierror log output * make lint only runs linter; add instructions on how to install linter * Use golangci-lint action in CD workflow * Bump dependencies, use go 1.24 in CI/CD * Extend remove_nulls and add to JS configuration * Add option to pass literal JS script string * feat: Add hash function to JS transformation (#422) * hash func available in JS; code refactor; tests; bypass clean-up * linter errors fixed * support for hash salt JS config attribute; tests * fix tests; clean up the code a bit * fix tests * export NewHclConfig for use in config service * Add OriginalData to models.Message * revert constants location * ticket suggestion * resolve header * code cleanup & expand monitoring test * fix linter * channel alerts from handleWrite * fix spelling * get a cooldown for alerts * small clean-up; add new test case for alerts * do header in prepareRequest * ensure only 1 alert is being sent * add e2e test for monitoring * resolving server conflicts * resolving server conflicts * resolving server conflicts * fix test unmarshal * check for expected alert * fix monitoring logs * add heartbeat test; update alert test * fix heartbeat test * debug monitoring test * fixing heartbeat e2e test * clean up * add explicit sleep to allow heartbeat events to appear * fixing heartbeat e2e test * fixing heartbeat e2e test * fixing heartbeat e2e test * fixing heartbeat e2e test * debug e2e heartbeat * debug e2e heartbeat * disable monitoring e2e tests for now * fix monitoring (still disabled) * enable monitoring e2e test 1 by 1 * fix monitoring e2e test * check if monitoring e2e test works now * check if monitoring e2e test works now * check if monitoring e2e test works now * correcting timeouts to detect heartbeats * correcting timeouts to detect heartbeats * small cleanup * address comments * attempt to run heartbeat docker for shorter period * attempt to run heartbeat docker for shorter period * attempt to run heartbeat docker for as long as in other tests * default monitoring to 5 mins instead of 60 * suggestions & add tests * fix test error * rework code to allow setup err resolution + test * fix tests * only reset monitoring if there was resolved setup error * add test to ensure monitoring is reset after setup error is resolved * fix config for new test * adding sleeps to ensure we capture heartbeat * fixing e2e test for alert -> heartbeat * fixing e2e tests * removed failing binary * removed failing binary * trying to test removed binary * trying to test removed binary * use correct hcl config * typo * test monitoring for not provided tags * marking tags as optional * addressing comments * suggestions * fix new test case * fixes * fix tests; update comments * reset suggestion * if after retries we still left with setup error - fail early * fix test expectation --------- Co-authored-by: Piotr Poniedziałek <pondzix@gmail.com> Co-authored-by: colmsnowplow <colm@snowplowanalytics.com> Co-authored-by: Piotr Poniedziałek <poniedzialek.piotr@gmail.com> * Metadata reporting POC * cleanup; adding the metadata implementation * suggestions * addressing comments * go mod tidy * update test * fixing unit test; adding e2e test for metadata_reporter * resolve lint issue * fix ObserverBuffer * debug nullpointer * debug nullpointer * fix nullpointer * cleanup * bypass typo fixes * update heartbeat & alert tests: improve readability * fix error: missed tag when refactored * add missing tags * address comments * fix test; remove comment * comments * only report invalid if those are successfully sent to failure target --------- Co-authored-by: Mikhail <mikhail.molotkov@snowplowanalytics.com> Co-authored-by: colmsnowplow <colm@snowplowanalytics.com>
* continuous flow of alerts unless resolved or deemed non-setup * remove interval check; rely on heartbeat tick
…f any which allows for more clarity and clear expectations
bbe4ac4 to
7612ad5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main idea here is to convert
to
so it is much clearer what interstate data could be, plus it allows for future improvements on the transformations in general
Note: this is a breaking change, because it is changes behaviour around how we handle arbitrary string as input to JS script
Used to be
now it must be