What problem does your feature solve?
New application development that needs to run ingestion pipeline will tend to implement(repeat) many similar parsing routines to transform network data into derived payment data models that Horizon ingestion does internally.
Horizon has many routines that are package private, internal to ingestion which parse and derive payments from stellar tx meta, so, they can't be re-used externally as-is.
What would you like to see?
Follows design proposal for reusable transformer functions for a common model
A few of these private ingestion parsing routines in Horizon exposed as pure(no side effects) functions and package public so they can be re-used by external apps to derive payment model quickly in their ingestion pipelines.
We limit focus on derived models to those considered ubiquitous and therefore most widely accepted such as a payment.
The presence of public functions for parsing payment model can encourage adoption of proposed streaming ingestion pipeline of which operators within a stream can invoke these functions to include payments info in-stream.
Create new pure function and model for payment in the shared ingest package. Horizon internal ingestion should not be changed or refactored as part of this effort, later we may decide different.
-
transactionOperationWrapper.IsPayment() -> ingest.HasPayment(op xdr.Operation) bool
-
for all operations that qualified as payment from HasPayment(), pull up the 'details' parsing routines of these operations from transactionOperationWrapper.Details() and fold into -> ingest.DerivePayment(tx xdr.Transaction, op xdr.Operation) []PaymentEvent:
- payment
- strictsend/strictreceive
- createaccount/mergeaccount
- invokehostfn - SAC events for mint/xfer/burn/claw
-
Payment info modeled as an event in struct:
type PaymentEvent struct {
ToAmount: "10.0000000"
ToAsset: "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
FromAmount: "10.0000000"
FromAsset: "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
From: "GANFZDRBCNTUXIODCJEYMACPMCSZEVE4WZGZ3CZDZ3P2SXK4KH75IK6Y"
To: "GANFZDRBCNTUXIODCJEYMACPMCSZEVE4WZGZ3CZDZ3P2SXK4KH75IK6Y"
Timestamp: "1723153694"
}
-
Unit test coverage on all new functions added to ingest package
Visualization of where the payment transform function fits in the larger CDP design for data pipeline:
What alternatives are there?
we don't publicize the routines, external apps use horizon code as reference, and will clone from this to their codebase as they choose to re-implement.
What problem does your feature solve?
New application development that needs to run ingestion pipeline will tend to implement(repeat) many similar parsing routines to transform network data into derived payment data models that Horizon ingestion does internally.
Horizon has many routines that are package private, internal to ingestion which parse and derive payments from stellar tx meta, so, they can't be re-used externally as-is.
What would you like to see?
Follows design proposal for reusable transformer functions for a common model
A few of these private ingestion parsing routines in Horizon exposed as pure(no side effects) functions and package public so they can be re-used by external apps to derive payment model quickly in their ingestion pipelines.
We limit focus on derived models to those considered ubiquitous and therefore most widely accepted such as a payment.
The presence of public functions for parsing payment model can encourage adoption of proposed streaming ingestion pipeline of which operators within a stream can invoke these functions to include payments info in-stream.
Create new pure function and model for payment in the shared ingest package. Horizon internal ingestion should not be changed or refactored as part of this effort, later we may decide different.
transactionOperationWrapper.IsPayment()->ingest.HasPayment(op xdr.Operation) boolfor all operations that qualified as payment from
HasPayment(), pull up the 'details' parsing routines of these operations fromtransactionOperationWrapper.Details()and fold into ->ingest.DerivePayment(tx xdr.Transaction, op xdr.Operation) []PaymentEvent:Payment info modeled as an event in struct:
Unit test coverage on all new functions added to
ingestpackageVisualization of where the payment transform function fits in the larger CDP design for data pipeline:
What alternatives are there?
we don't publicize the routines, external apps use horizon code as reference, and will clone from this to their codebase as they choose to re-implement.