refactor tedge flows api

[reubenmiller]

Is your refactoring request related to a problem? Please describe.

Refactor the tedge flows interface and data types.

Describe the solution you'd like

The following items should be refactored to improve the overall tedge flows concept:

Item 1: Remove onConfigUpdate function

Remove the onConfigUpdate() function from the flow concept. Whilst the underlying functionality is useful, it is not a common case yet, and we might be able to solve it in a different way in the future. Removing this interface reducing the size of the public API for the initial launch of the feature.

Item 2: Replace Timestamp type with the JavaScript Date object - Done

Replace the Timestamp type with the JavaScript native Date object.

The type should be replaced in the following areas:

• In the onInterval function, the first argument
• In the message.timestamp, and rename the property to message.time

Below shows an example of the function signatures with the change in the timestamp data type.

export function onMessage(message: Message, config: any) {
    console.log(`Message received at: ${message.time}`);
    return [];
}

export function onInterval(time: Date, config: any) {
    return [];
}

Item 3: Replace config with context in functions

Change the function signatures of onMessage and onInterval to rename the config argument to context, however context is an object which should include a config property (which contains the step's configuration). Below shows an example of the context object and its relationship to the "config" property.

Example Context object

{
    config: {
        foo: "bar"
    }
}

Below shows an example of the renamed function signatures which use the new "context" type.

export function onMessage(message: Message, context: any) {
    console.log(`Message has context: ${context.config}`);
    return [];
}

export function onInterval(time: Date, context: any) {
    return [];
}

Item 4: Rename payload_raw to payload in the message data object

In the message type, rename payload_raw to payload (overwriting the previous string type used in payload). This will prevent duplicating the data being passed to the flows. JavaScript helper functions can be provided in the meantime to make it easier to encode and decode payloads to and from the JavaScript Uint8Array type.

export interface Message {
  time?: Date; // Previously a custom Timestamp object, but now a JavaScript Date
  topic: string;
  payload?: Uint8Array; // Previously was a string, and was duplicated in payload_raw
  retain: boolean;
}

Describe alternatives you've considered

Additional context

[didier-wenzek]

Item 1: Remove onConfigUpdate function

Okay. However, better to do this once flow Context implemented, the plan being to use the context to store measurements units or other metadata instead of an updated config.

Item 2: Replace Timestamp type with the JavaScript Date object

This makes sense. However, I think a better place for the time property is the context, not the message itself.

• In practice, no scripts attached a timestamp to the transformed messages
• And the runtime has to inject a timestamp to each intermediate message, just in case a downstream step will use it.

export function onMessage(message: Message, context: Context) {
    console.log(`Message received at: ${context.time}`);
    return [];
}

export function onInterval(context: Context) {
    console.log(`Messages aggregated at: ${context.time}`);
    return [];
}

Item 3: Replace config with context in functions

Why not. However, what would be the recommended syntax for default config values?

The following is working but not really nice:

export function onMessage(message, { time, config, state }) {
    let { threshold = 42 } = config;
}

Item 4: Rename payload_raw to payload in the message data object

We can then enhance the messages with builtin functions such as toString or toJson.

retain: boolean;

I disagree adding a retain flag to messages. It makes sense to have such protocol aware properties, but I would put those apart in a protocol or connector object. Mainly because, these properties are really specific (here to publish a message over MQTT) while a flow might have diverse sources and sinks (e.g a command).

export interface Message {
  topic: string;
  payload: Uint8Array;
  protocol: MQTT | Command
}

export interface MQTT {
     retain: boolean,
     qos: QoS,
}

export interface Command {
     command: string
     interval: number,
}

[reubenmiller]

Why not. However, what would be the recommended syntax for default config values?

Though since the context will be providing some additional functionality (in the future), we'll anyway generally not do object destructuring in the function signature, so it will just follow something very similar to your example:

export function onMessage(message, context) {
    const { threshold = 42 } = context.config;
    context.setState('foo', 'bar')
}

I disagree adding a retain flag to messages. It makes sense to have such protocol aware properties, but I would put those apart in a protocol or connector object. Mainly because, these properties are really specific (here to publish a message over MQTT) while a flow might have diverse sources and sinks (e.g a command).

This ticket doesn't include all of the API changes. I chose not to put the changes regarding the retain flag just yet, as I wanted the design to stabilize first, and then I'll create a ticket for it, but it does go in the direction that you proposed (but the idea is to call the property transportFields which is a map which different transports like MQTT can store their specific info within it).

[reubenmiller]

This makes sense. However, I think a better place for the time property is the context, not the message itself.

In practice, no scripts attached a timestamp to the transformed messages
And the runtime has to inject a timestamp to each intermediate message, just in case a downstream step will use it.

Though having the timestamp when the message was received on the message is more natural, even though it is only used as read only information. The context object is more directed at providing context for the overall flow.

[didier-wenzek]

This makes sense. However, I think a better place for the time property is the context, not the message itself.
In practice, no scripts attached a timestamp to the transformed messages
And the runtime has to inject a timestamp to each intermediate message, just in case a downstream step will use it.

Though having the timestamp when the message was received on the message is more natural, even though it is only used as read only information. The context object is more directed at providing context for the overall flow.

The provided timestamp is not the ingestion time but the processing time. With MQTT, the former is unknown. This is why I'm inclined to move that timestamp to the context.