feat: Ingest API

[imaretic]

About the Contributor

This PR is posted on behalf of EVS Broadcast Equipment.

Type of Contribution

This is a: Feature

State of the work offered

• Feature development is considered done, offered as an addition to the public codebase more or less as-is. (comments and discussion welcome, but the main development work is already complete).
• The new code is of an additive nature, with no known breaking changes.
• For the same reason, the risk of regressions is considered low.
• The implementation follows established patterns.
• The maintenance burden of the new code is considered small, as the interfaces of the new code are located at the ingest side of the Sofie Core, which has historically proven itself stable.
• On the topics of added complexity, new abstractions or duplication (code, logic, responsibility, code branching): this feature does introduce a new alternative path of dataflow on top of existing ones. It will by nature add duplication of maintenance work when the ingest data models or workflows change. It also increases the footprint of the existing REST API, meaning increased work if this ever undergoes technological evolutions or code pattern refactoring.

The feature - What, why and how

An HTTP REST API for creating and updating rundown data.

This allow standard web clients to connect to Sofie to serve rundowns and rundown content, taking on the role that traditional NRCSes have in existing workflows. Sofie is already well covered on traditional workflows with the MOS workflows. This addition of HTTP REST API represents a step in the direction of open and web-centric systems which will further enable integrations and smooth workflows.

The implementation follows established patterns and is implemented in two separate parts: an OpenAPI definition and one part in the existing REST API implementation. A summary of the changes is listed at the end of this document.

Example use cases

Example 1: HTTP-based NRCS/CMS

• A traditional Sofie topology matching MOS-based NRCSes. Unidirectional flow.
• This use case represents a step towards the latest generation newsrooms, embracing web standards and HTTP-based networking.
• This model also allows for simpler* gateways which non-HTTP-based systems can use to convert into HTTP/REST communication for Sofie integration.

*not comparable to the full blown DDP-based PeripheralDevice gateways.

Example 2: Headless Sofie

• For self-op or other use cases where tailor-made frontends are desired.
• This use case demonstrates how this new Ingest API in combination with existing User Actions API and the Live Status gateway close the loop and enables Sofie to run as a headless engine/backend.

Topologies

• The API is not designed to allow multiple systems/clients to connect and collaborate on the same content. There's no concepts of versions, locks, guaranteed order or atomic operations. All operations will execute individually and naively. Last command wins.
• There is no session or other knowledge about the connected clients, still any number of clients can operate at the same time.
• This means that clients either should be logically separated to operate on different content, or be managed through a middleware that resolves clashes and conflicts.

Design, considerations and choices made

RESTful principles are followed

The verbs GET, POST, PUT, DELETE are used.

Each of Playlists, Rundowns, Segments and Parts are made available as resources, following this pattern:

/ingest/{studioId}/playlists/{playlistId}/rundowns/{rundownId}/segments/{segmentId}/parts/{partId} where, for example, /ingest/{studioId}/playlists is a collection and /ingest/{studioId}/playlists/{playlistId} is a resource in the Playlists collection.

Calling DELETE on a resource will always also remove any resources owned by the deleted resource. Calling PUT on a collection will replace all resources of the type under that collection - i.e. when an array of e.g. Segments is received for a Rundown, if there are any already existing Segments within the Rundown that are not included in the received array then those Segments will be deleted. In this way, the entire contents of a Rundown can be replaced by sending a new array of Segments via a single PUT.

The playlistId , rundownId , segmentId , and partId can be any unique ID of the external system's choosing, so long as they are consistent for a particular resource.

20x Response codes

20x OK means the request itself is valid and the system goes off doing the asynchronous ingest operation - which can later fail, with no way of communicating this back to the external system.

There is an opt-in mechanism for Blueprints validation of payloads during processing of the request. If specified, a Blueprint function can do synchronous validation before Core responds on the request. A failed validation in this Blueprints-hook results in a 40x error.

Even after a successful validation by Blueprints, the Core ingest job will perform asynchronously and might fail in silence after having reported 20x back to the external client.

Playlists lifespans in relation to Rundowns

• Playlists can't explicitly be created. They get created as a side-effect of creating new Rundowns. This follows the existing ingest logic where Rundowns sharing a playlistExternalId are merged into one Playlist, and Rundowns with unique playlistExternalIds live in discrete Playlists.
• Therefore, there are GET and DELETE methods for Playlists, but no POST or PUT.

The Resync mechanism

MOS allows Sofie to request an update of all rundown data. This user-facing feature is intended to allow users to refresh the rundown data if they observe something is outdated or wrong.

The same mechanism is implemented for this HTTP Ingest API:

1. Upon Rundown creation, the external system describes an URL which Sofie can use to request all rundown data to be updated.
2. On rundown resync, Sofie will do a simple HTTP POST request to this endpoint.
3. The external system is then supposed to PUT the entire rundown/all rundowns to Sofie to refresh all content.
4. There's no session or tracking of this naive (yet simple/effective) process.

ExternalIds

With exception of the studioId. the external system creates and owns all IDs in the /ingest/{studioId}/playlists/{playlistId}/rundowns/{rundownId}/segments/{segmentId}/parts/{partId} structure. These IDs map to externalId in the Sofie database.

For the external system to be able to control Sofie (Use case 2, "Headless Sofie"), it needs to be able to do rundown playout operations in an efficient way. Currently, the User Actions API expects internalIds for its operations. This could have meant that the external system would need to query back all content it has created, in order to create and store a map of the relation between external and internal IDs in memory, to be able to send any playout command, which wouldn't be pleasant to work with as a consumer of the APIs.

For this reason, we have simplified this by allowing either internalId or externalId to be used interchangeably on the User Actions API. This greatly simplifies/enables the described Use case 2, "Headless Sofie".

Rundown, Segment, and Part models

Rundowns, Segments, and Parts are modelled to match common public properties and database structure.

The Part payload property is un-typed, and will match the Blueprint's expectation to incoming content needed to create Parts with Pieces.

The structure allows for a closed loop where Blueprints define the schemas, which external systems use to generate the requests with typed payload, which ultimately gets synchronously parsed and validated during ingest by Blueprints again.

Security and auth is excluded from the scope of this PR

Following existing patterns/not unique for this Ingest API addition.To be dealt with elsewhere in the stack or in a more global matter.

Summary of changes

Design: packages/openapi/

• Addition called "Ingest API" as extension to the existing OpenAPI package.
• API description and formal schemas for the purpose of design, documentation and API exchange with third parties.
• The existing OpenAPI workflow is followed, catering for Swagger generation of docs, JSON schemas/types and example server/client suitable for automated CI tests.

Implementation: meteor/server/api/rest/v1/ingest.ts

• Follows the established pattern of a serverAPI class to be used in a registerRoutes() factory.
• Consumed by ./index.ts through the established register*Router(sofieAPIRequest).
• Interacts with collections and ingest operations (runIngestOperation()) in the same way as the existing ingestMosDevice.
• Rundown operations over this API are named httpIngest (rundownSource type httpIngest), which can be confusing considering the existing non-REST "http ingest api" and rundownSource type "http".
• No formal connection to the accompanying OpenAPI package. The implementation is hand-crafted to match the described API - no type safety or guarantees between the OpenAPI contract and the implementation.
• Blueprints validation interface: validatePartPayloadFromAPI method, described here: packages/blueprints-integration/src/api/studio.ts

Side-effect: packages/shared-lib/src/peripheralDevice/ingest.ts

• New properties added to the ingest interfaces
• All new properties typed as optional, so by nature no breaking changes

Side-effect: Job worker playout implementation

See the paragraph on "ExternalIds"

Changes to lookup of playlists, rundowns, segments and parts in the following places:

– meteor/server/api/rest/v1/playlists.ts
– meteor/server/security/rundownPlaylist.ts
– packages/job-worker/src/playout/model/implementation/PlayoutRundownModelImpl.ts
– packages/job-worker/src/playout/model/implementation/PlayoutSegmentModelImpl.ts
– packages/job-worker/src/playout/lock.ts

Testing

• I have added one or more unit tests for this PR
• I have updated the relevant unit tests
• No unit test changes are needed for this PR

Status

• PR is ready to be reviewed.
• The functionality has been tested by the author.
• Relevant unit tests has been added / updated.
• Relevant documentation (code comments, system documentation) has been added / updated.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 1.02587% with 2219 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
meteor/server/api/rest/v1/ingest.ts	0.00%	1727 Missing and 1 partial ⚠️
meteor/server/lib/rest/v1/ingest.ts	0.00%	272 Missing and 1 partial ⚠️
meteor/server/api/rest/v1/playlists.ts	0.00%	145 Missing ⚠️
meteor/server/api/rest/v1/typeConversion.ts	0.00%	51 Missing ⚠️
meteor/server/api/ingest/actions.ts	4.76%	20 Missing ⚠️
meteor/server/api/rest/v1/index.ts	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[jstarpl]

Considering that this is intended to be a part of the REST API, I think it would make sense to use restApi instead of httpIngest, thus avoiding any confusion.

[jstarpl]

I don't think we need this in release53?

[jstarpl]

Using regular global fetch, I think this would be a different mock?

[jstarpl]

Should this handle ENOTFOUND as well?

[imaretic]

That has been added. The idea here is to nicely log most common error messages.

[Julusian]

I'm not sure whether adding many of these properties here is a good idea.
As far as sofie is concerned these properties don't do anything, so why should they be at this level instead of inside of the payload?

[imaretic]

The idea here is to move all properties that have a direct 1:1 mapping with database fields to the top level of the object, and keep only blueprint-specific or custom logic properties inside the payload object.

For example:

• The float and autoNext properties are fields of the Part model and should be at the top level (just like already existing name or rank properties).
• In contrast, a property like partType, which may be used by blueprints to generate pieces but isn't part of the database schema, belongs inside the payload object.

[Julusian]

It might be best to have a workshop to discuss this. @nytamin can you arrange that?

My concern is that these properties don't do anything unless the blueprints decide that they do. So it feels weird to tell a gateway (with the types) that these properties exist, when it is likely that they will be ignored, and at the same time to tell blueprints that these properties exist when for most ingest sources they won't ever be set.

Presumably these values are added as they are the ones you want, so I am worried that if we start this then more will be added over time morphing this interface into a simplified version of the segment/part we store in the db, with an unclear definition of what can be expected to be set when working in blueprints.
Potentially encouraging the parsing of the actual nrcs data to be done in gateways instead of blueprints (leaving blueprints to re-do a bunch of the parsing and fixup any bad assumptions

The name/rank properties are different in that they are used by sofie-core for something (even if that is only logging like name is)

---

I should probably add this part somewhere else, but it is relevant to this discussion. (Sorry, this didnt occur to me until I was looking into this again just now)
In the MutableIngestSegment and related types, these properties become required to have a value (by providing defaults if they are not defined), which feels wrong in another way, as now we have lost that the nrcs/api did not provide these values. If they stay, I think that preserving them being undefined is important.

Additionally, MutableIngestSegment has getters for these properties but not setters, and they are not updated by defaultApplyIngestChanges meaning that changes to these values for existing parts/segments/rundowns will not be propogated through to blueprints.

[jesperstarkar]

Due to vacations, this will be idle from EVS' side for a couple more weeks before we come back to resolving this pending decision.

[Julusian]

I don't think it is the responsibility of the playout model to handle the api using ambiguous ids.
Instead you could add a getSegmentByExternalId to make this lookup explicit

[imaretic]

Moved to the API layer.

[Julusian]

same problem here as with getSegment

[imaretic]

Moved to the API layer.

[Julusian]

I think it would be best to avoid having these fuzzy checks out of core logic, and keep it inside of the api implementation.
There are a few places where this is done in this PR. It might require a separate/explicit query to convert an externalId to _id, but this should be safe to do as it is a stable function for converting an externalId to an _id

[imaretic]

This has been moved to API layer.