Skip to content

Commit a22385f

Browse files
danjoadanjoa
committed
extracted data federation
1 parent 1ebff7d commit a22385f

File tree

1 file changed

+12
-166
lines changed

1 file changed

+12
-166
lines changed

‎guides/integration/calesi.md‎

Lines changed: 12 additions & 166 deletions
Original file line numberDiff line numberDiff line change
@@ -843,47 +843,6 @@ While everything just works nicely when mocked in-process and with a shared in-m
843843
844844
![XTravels Fiori details view showing a travel requests, with the flights data missing](assets/xtravels-bookings-.png)
845845
846-
The log shows bulk requests – the Fiori client desperately trying to fetch the missing customer data. Scrolling the list would repeat this endlessly:
847-
848-
<span style="font-size:63%">
849-
850-
```js
851-
[odata] - POST /odata/v4/travel/$batch
852-
[odata] - > GET /Travels(ID=4133,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
853-
[odata] - > GET /Travels(ID=4132,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
854-
[odata] - > GET /Travels(ID=4131,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
855-
[odata] - > GET /Travels(ID=4130,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
856-
[odata] - > GET /Travels(ID=4129,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
857-
[odata] - > GET /Travels(ID=4128,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
858-
[odata] - > GET /Travels(ID=4127,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
859-
[odata] - > GET /Travels(ID=4126,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
860-
[odata] - > GET /Travels(ID=4125,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
861-
[odata] - > GET /Travels(ID=4124,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
862-
[odata] - > GET /Travels(ID=4123,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
863-
[odata] - > GET /Travels(ID=4122,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
864-
[odata] - > GET /Travels(ID=4121,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
865-
[odata] - > GET /Travels(ID=4120,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
866-
[odata] - > GET /Travels(ID=4119,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
867-
[odata] - > GET /Travels(ID=4118,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
868-
[odata] - > GET /Travels(ID=4117,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
869-
[odata] - > GET /Travels(ID=4116,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
870-
[odata] - > GET /Travels(ID=4115,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
871-
[odata] - > GET /Travels(ID=4114,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
872-
[odata] - > GET /Travels(ID=4113,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
873-
[odata] - > GET /Travels(ID=4112,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
874-
[odata] - > GET /Travels(ID=4111,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
875-
[odata] - > GET /Travels(ID=4110,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
876-
[odata] - > GET /Travels(ID=4109,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
877-
[odata] - > GET /Travels(ID=4108,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
878-
[odata] - > GET /Travels(ID=4107,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
879-
[odata] - > GET /Travels(ID=4106,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
880-
[odata] - > GET /Travels(ID=4105,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
881-
[odata] - > GET /Travels(ID=4104,IsActiveEntity=true) { '$select': 'Customer', '$expand': 'Customer($select=ID,Name)' }
882-
```
883-
</span>
884-
885-
We see there are specific implementions required, to actually integrate remote services at runtime. We deep dive into one possible solution for that next.
886-
887846
888847
889848
@@ -938,6 +897,10 @@ The querying API is the most powerful and closest to the use cases of data-centr
938897
939898
### Querying Remote Data
940899
900+
> [!tip] CAP-level Querying -> agnostic to databases & protocols
901+
> We work with **database-agnostic** and **protocol-agnostic** [CQL queries](../../cds/cql) both for interacting with the local database as well as for querying remote services. In effect, we got a fully generic solution for replication, i.e., it works for **_any_** remote service that supports OData, or HCQL.
902+
903+
941904
### Delegating Queries
942905
943906
Value helps are common use cases where delegation of requests is needed, which we implemented like this in `srv/travel-service.js` for the `Customers` entity:
@@ -990,142 +953,25 @@ this.after ('SAVE', Travels, ({ Bookings=[] }) => Promise.all (
990953
991954
### Generic Data Federation
992955
993-
**Motivation** – Displaying external data in lists commonly requires fast access to that data. Relying on live calls to remote services per row is clearly not an option, as that would lead to poor performance, excessive load on server, and a nightmare regarding resilience. Instead, we somehow need to ensure that all required data is available locally, so that it can be accessed fast and reliably by UIs, using good old SQL JOINs.
994-
995-
#### Federated Consumption Views
996-
997-
The xtravels app uses a simple data replication solution that automatically replicates data for all [consumption views](#consumption-views) tagged with `@federated`:
998-
999-
```cds :line-numbers=4
1000-
@federated entity Customers as projection on S4.A_BusinessPartner { ... }
1001-
```
1002-
1003-
If a remote service is detected, CAP turns these entities into tables to serve as local persistence for replicated data (line 9 in the code below).
956+
Displaying external data in lists commonly requires fast access to that data. Relying on live calls to remote services per row is clearly not an option, as that would lead to poor performance, excessive load on server, and a nightmare regarding resilience. Instead, we somehow need to ensure that all required data is available locally, so that it can be accessed fast and reliably by UIs, using good old SQL JOINs.
1004957
1005-
> [!tip] Stay Intentional -> <i>What, not how!</i> -> Minimal Assumptions
1006-
>
1007-
> By tagging entities with `@federated`, you declare your intention - **_what_** you want - without assuming **_how_** to implement it. This lets CAP runtimes or your own solutions choose the best implementation for each environment, which may differ between development, testing, and production environments, allowing evolution without disruption.
1008-
1009-
1010-
#### Generic Implementation
1011-
1012-
Here's the complete code, placed in file `srv/data-federation.js`:
958+
In the [`@capire/xtravels`](https://github.com/capire/xtravels) app we accomplished that with a simple, yet quite effective generic data replication solution. In essence that implementation boils down to these lines of querying remote services:
1013959
1014960
::: code-group
1015-
```js:line-numbers [srv/data-federation.js]
1016-
const PROD = process.env.NODE_ENV === 'production' /* eslint-disable no-console */
1017-
const cds = require ('@sap/cds')
1018-
const feed = []
1019-
1020-
// Collect all entities to be federated, and prepare replica tables
1021-
PROD || cds.on ('loaded', csn => {
1022-
for (let e of cds.linked(csn).entities) {
1023-
if (e['@federated']) {
1024-
let srv = remote_srv4(e)
1025-
if (is_remote(srv)) {
1026-
e['@cds.persistence.table'] = true //> turn into table for replicas
1027-
feed.push ({ entity: e.name, remote: srv })
1028-
}
1029-
}
1030-
}
1031-
})
1032-
1033-
// Setup and schedule replications for all collected entities
1034-
PROD || cds.once ('served', () => Promise.all (feed.map (async each => {
1035-
const srv = await cds.connect.to (each.remote)
1036-
srv._once ??=! srv.on ('replicate', replicate)
1037-
await srv.schedule ('replicate', each) .every ('3 seconds')
1038-
})))
1039-
1040-
// Event handler for replicating single entities
1041-
async function replicate (req) {
1042-
let { entity } = req.data, remote = this
961+
```js [srv/data-federation.js]
962+
const remote = await cds.connect.to (each.remote)
963+
```
964+
:::
965+
```js
1043966
let { latest } = await SELECT.one `max(modifiedAt) as latest` .from (entity)
1044967
let rows = await remote.run (
1045968
SELECT.from (entity) .where `modifiedAt > ${latest}`
1046969
)
1047970
if (rows.length) await UPSERT (rows) .into (entity); else return
1048-
console.log ('Replicated', rows.length, 'entries', { for: entity, via: this.kind })
1049971
}
1050-
1051-
// Helpers to identify remote services, and check whether they are connected
1052-
const remote_srv4 = entity => entity.__proto__._service?.name
1053-
const is_remote = srv => cds.requires[srv]?.credentials?.url
1054972
```
1055-
:::
1056-
1057-
Let's have a closer look at this code, which handles these main tasks:
1058-
1059-
1. **Prepare Persistence** – When the model is `loaded`, before deploying it to the database, we collect all `@federated` entities to be replicated, check whether their respective services are remote, and if so, turn them into tables for local replicas (line 11).
1060-
1061-
2. **Setup Replication** – Later when all services are `served`, we connect to each remote one (line 20), register a handler for replication (line 21), and schedule it to be invoked every three seconds (line 22).
1062-
1063-
3. **Replicate Data** – Finally, the `replicate` handler implements a simple polling-based data federation strategy, based on `modifiedAt` timestamps (lines 28-32), with the actual call to remote happening on line 29.
1064-
1065-
> [!tip] CAP-level Querying -> agnostic to databases & protocols
1066-
> We work with **database-agnostic** and **protocol-agnostic** [CQL queries](../../cds/cql) both for interacting with the local database as well as for querying remote services. In effect, we got a fully generic solution for replication, i.e., it works for **_any_** remote service that supports OData, or HCQL.
1067-
1068-
1069-
#### Test Drive
1070-
1071-
Let's see the outcome in action: to activate the above data federation code, edit `srv/server.js` file and uncomment the single line of code in there like this:
1072-
1073-
::: code-group
1074-
```js :line-numbers [srv/server.js]
1075-
process.env.NODE_ENV || require ('./data-federation')
1076-
```
1077-
:::
1078-
1079-
Restart the Xtravels app, and see these lines in the log output:
1080-
1081-
```zsh
1082-
Replicated 49 entries { for: 'sap.capire.xflights.Supplements', via: 'hcql' }
1083-
Replicated 44 entries { for: 'sap.capire.xflights.Flights', via: 'hcql' }
1084-
Replicated 727 entries { for: 'sap.capire.s4.Customers', via: 'odata' }
1085-
```
1086-
1087-
The S/4 Business Partner service in terminal 1 shows the incoming OData request(s):
1088-
1089-
```zsh
1090-
[odata] - GET /odata/v4/s4-business-partner/A_BusinessPartner {
1091-
'$select': 'BusinessPartner,PersonFullName,LastChangeDate',
1092-
'$filter': 'LastChangeDate gt 2024-12-31'
1093-
}
1094-
```
1095-
1096-
While the xflights service in terminal 2 shows its incoming HCQL requests like that:
1097-
1098-
```zsh
1099-
[hcql] - GET /hcql/data/ {
1100-
SELECT: {
1101-
from: { ref: [ 'sap.capire.flights.data.Flights' ] },
1102-
columns: [
1103-
{ ref: [ 'ID' ], as: 'ID' },
1104-
{ ref: [ 'date' ], as: 'date' },
1105-
{ ref: [ 'departure' ], as: 'departure' },
1106-
{ ref: [ 'arrival' ], as: 'arrival' },
1107-
{ ref: [ 'free_seats' ], as: 'free_seats' },
1108-
{ ref: [ 'modifiedAt' ], as: 'modifiedAt' },
1109-
{ ref: [ 'airline', 'icon' ], as: 'icon' },
1110-
{ ref: [ 'airline', 'name' ], as: 'airline' },
1111-
{ ref: [ 'origin', 'name' ], as: 'origin' },
1112-
{ ref: [ 'destination', 'name' ], as: 'destination' }
1113-
],
1114-
where: [
1115-
{ ref: [ 'modifiedAt' ] },
1116-
'>',
1117-
{ val: '2026-01-28T17:38:28.929Z' }
1118-
]
1119-
}
1120-
}
1121-
```
1122-
1123-
Finally, open the Fiori UI in the browser again, and see that customer data from S/4 as well as flight data from xflights is now displayed properly, thanks to the data federation implemented above.
1124-
1125-
![XTravels Fiori list view showing tarvel requests, now with customer names again.](assets/xtravels-list.png)
1126-
1127-
![XTravels Fiori details view showing a travel requests, now with flight data again.](assets/xtravels-bookings.png)
1128973
974+
[Learn more about CAP-level Data Federation in the dedicated guide.](data-federation){.learn-more}
1129975
1130976
### On-demand Replication
1131977

0 commit comments

Comments
 (0)