Skip to content

proposal: new JS interstate object#446

Draft
Mikhail (oo-sleeper) wants to merge 9 commits intodevelopfrom
new-js-interstate-poc
Draft

proposal: new JS interstate object#446
Mikhail (oo-sleeper) wants to merge 9 commits intodevelopfrom
new-js-interstate-poc

Conversation

@oo-sleeper
Copy link
Contributor

Main idea here is to convert

engineProtocol
	FilterOut    bool
	PartitionKey string
	Data         any
	HTTPHeaders  map[string]string

to

engineProtocol
	FilterOut    bool
	PartitionKey string
	Data         map[string]any
	HTTPHeaders  map[string]string

so it is much clearer what interstate data could be, plus it allows for future improvements on the transformations in general

Note: this is a breaking change, because it is changes behaviour around how we handle arbitrary string as input to JS script

Used to be

// x.Data is a 'string' type
function main(x) {
   let newVal = "Hello:" + x.Data;
   x.Data = newVal;
   return x;
}

now it must be

// x.Data is an 'object' (map[string]any) type
function main(x) {
   let newVal = "Hello:" + x.Data['Data'];
   x.Data['Data'] = newVal;
   return x;
}

It looks like all deprecations are only in test code. Pubsub V1 client we use in production source/target code is not deprecated (at least not yet and it's going to be supported only until July 31st 2026, see [this](https://github.com/googleapis/google-cloud-go/blob/main/pubsub/MIGRATING.md) migration guide to V2
* feat: Send heartbeat and alert events to webhook (#428)
* Add changes for debug workflow
* Separate target for filtered data.
* Debug mode flag in HTTP target, which allows us to send original data alongside the transformed one.
It's now possible to configure 3 different targets for each: transformed data, failed data and filtered data.
By default target for filtered data is the `silent` one - it just acks data.

* Precalculate rejection timestamp in HTTP target
Before:
-Send 2 headers: `Request-Timeout` and `Request-Timestamp`, which can be used by target server to determine if HTTP request should be rejected.

Now:
-Send 1 header `Rejection-Timestamp` which is precalculated in HTTP target based on configured client timeout and threshold. This allow us to simplify logic in targets (like GTMSS client).
-To determine if a request is OK, target doesn't have to do any calculation, simply extract `Rejection-Timestamp` header and compare with a server's timestamp.
-This commit also contains little refactoring of HTTP target creation + tests (no more `newHTTPTarget` function, using `HTTPTargetConfigFunction` directly).

* Enable data only output for `stdout` target (#410)
* enable data only output for stdout target
* add missing newline
* address comment & add test steps to README
* Add context and complete the list of test commands
* feat: Add hash function to JQ transformation (#411)
* JQ command hash implementation is ready for review
* fix linter errors
* restrict num of parameters; support sha256 & md5; add tests
* apply suggestions
* remove duplicated tests
* feat: Add golang linter to the project (#412)
* fixing code to align with linter rules
* fix majority of lint issues; left a few out - those seem bit more complex
* fix tests
* fix tests
* implement suggestions
* fix the code as per comments
* revert slog introduction; would be done later
* update lint cmd in Makefile to handle new linter
* update multierror log output
* make lint only runs linter; add instructions on how to install linter
* Use golangci-lint action in CD workflow
* Bump dependencies, use go 1.24 in CI/CD
* Extend remove_nulls and add to JS configuration
* Add option to pass literal JS script string
* feat: Add hash function to JS transformation (#422)
* hash func available in JS; code refactor; tests; bypass clean-up
* linter errors fixed
* support for hash salt JS config attribute; tests
* fix tests; clean up the code a bit
* fix tests
* WIP: outlined the draft, need to define events
* now need to figure where to place alerts collection
* adjust webhook URL - it is passed in full from Terraform, so no need to hassle with it
* tags are mandatory
* fixing tags
* fix interval attribute value
* Add changes for debug workflow
* Separate target for filtered data.
* Debug mode flag in HTTP target, which allows us to send original data alongside the transformed one.
It's now possible to configure 3 different targets for each: transformed data, failed data and filtered data.
By default target for filtered data is the `silent` one - it just acks data.

* Precalculate rejection timestamp in HTTP target

Before:

Send 2 headers: `Request-Timeout` and `Request-Timestamp`, which can be used by target server to determine if HTTP request should be rejected.

Now:

Send 1 header `Rejection-Timestamp` which is precalculated in HTTP target based on configured client timeout and threshold. This allow us to simplify logic in targets (like GTMSS client).

To determine if a request is OK, target doesn't have to do any calculation, simply extract `Rejection-Timestamp` header and compare with a server's timestamp.

This commit also contains little refactoring of HTTP target creation + tests (no more `newHTTPTarget` function, using `HTTPTargetConfigFunction` directly).

* Enable data only output for `stdout` target (#410)
* enable data only output for stdout target
* add missing newline
* address comment & add test steps to README
* Add context and complete the list of test commands
* feat: Add hash function to JQ transformation (#411)
* JQ command hash implementation is ready for review
* fix linter errors
* restrict num of parameters; support sha256 & md5; add tests
* apply suggestions
* remove duplicated tests
* feat: Add golang linter to the project (#412)
* fixing code to align with linter rules
* fix majority of lint issues; left a few out - those seem bit more complex
* fix tests
* fix tests
* implement suggestions
* fix the code as per comments
* revert slog introduction; would be done later
* update lint cmd in Makefile to handle new linter
* update multierror log output
* make lint only runs linter; add instructions on how to install linter
* Use golangci-lint action in CD workflow
* Bump dependencies, use go 1.24 in CI/CD
* Extend remove_nulls and add to JS configuration
* Add option to pass literal JS script string
* feat: Add hash function to JS transformation (#422)
* hash func available in JS; code refactor; tests; bypass clean-up
* linter errors fixed
* support for hash salt JS config attribute; tests
* fix tests; clean up the code a bit
* fix tests
* export NewHclConfig for use in config service
* Add OriginalData to models.Message
* revert constants location
* ticket suggestion
* resolve header
* code cleanup & expand monitoring test
* fix linter
* channel alerts from handleWrite
* fix spelling
* get a cooldown for alerts
* small clean-up; add new test case for alerts
* do header in prepareRequest
* ensure only 1 alert is being sent
* add e2e test for monitoring
* resolving server conflicts
* resolving server conflicts
* resolving server conflicts
* fix test unmarshal
* check for expected alert
* fix monitoring logs
* add heartbeat test; update alert test
* fix heartbeat test
* debug monitoring test
* fixing heartbeat e2e test
* clean up
* add explicit sleep to allow heartbeat events to appear
* fixing heartbeat e2e test
* fixing heartbeat e2e test
* fixing heartbeat e2e test
* fixing heartbeat e2e test
* debug e2e heartbeat
* debug e2e heartbeat
* disable monitoring e2e tests for now
* fix monitoring (still disabled)
* enable monitoring e2e test 1 by 1
* fix monitoring e2e test
* check if monitoring e2e test works now
* check if monitoring e2e test works now
* check if monitoring e2e test works now
* correcting timeouts to detect heartbeats
* correcting timeouts to detect heartbeats
* small cleanup
* address comments
* attempt to run heartbeat docker for shorter period
* attempt to run heartbeat docker for shorter period
* attempt to run heartbeat docker for as long as in other tests
* default monitoring to 5 mins instead of 60
* suggestions & add tests
* fix test error
* rework code to allow setup err resolution + test
* fix tests
* only reset monitoring if there was resolved setup error
* add test to ensure monitoring is reset after setup error is resolved
* fix config for new test
* adding sleeps to ensure we capture heartbeat
* fixing e2e test for alert -> heartbeat
* fixing e2e tests
* removed failing binary
* removed failing binary
* trying to test removed binary
* trying to test removed binary
* use correct hcl config
* typo
* test monitoring for not provided tags
* marking tags as optional
* addressing comments
* suggestions
* fix new test case
* fixes
* fix tests; update comments
* reset suggestion
* if after retries we still left with setup error - fail early
* fix test expectation

---------

Co-authored-by: Piotr Poniedziałek <pondzix@gmail.com>
Co-authored-by: colmsnowplow <colm@snowplowanalytics.com>
Co-authored-by: Piotr Poniedziałek <poniedzialek.piotr@gmail.com>

* Metadata reporting POC
* cleanup; adding the metadata implementation
* suggestions
* addressing comments
* go mod tidy
* update test
* fixing unit test; adding e2e test for metadata_reporter
* resolve lint issue
* fix ObserverBuffer
* debug nullpointer
* debug nullpointer
* fix nullpointer
* cleanup
* bypass typo fixes
* update heartbeat & alert tests: improve readability
* fix error: missed tag when refactored
* add missing tags
* address comments
* fix test; remove comment
* comments
* only report invalid if those are successfully sent to failure target

---------

Co-authored-by: Mikhail <mikhail.molotkov@snowplowanalytics.com>
Co-authored-by: colmsnowplow <colm@snowplowanalytics.com>
* continuous flow of alerts unless resolved or deemed non-setup
* remove interval check; rely on heartbeat tick
…442)

* Add ability to set individual credentials for a PubSub target (closes #441)

* Rename CredentialsJSONPath to CredentialsPath
…f any which allows for more clarity and clear expectations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants