Skip to content

Support incremental surface updates #284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Support incremental surface updates #284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
73b9903
feat: Support incremental surface updates
jacobsimionato Sep 7, 2025
012152d
chore: Apply automated fixes and formatting
jacobsimionato Sep 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions examples/simple_chat/lib/main.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ class _ChatScreenState extends State<ChatScreen> {
giving exactly one response for each user message.
Your responses should contain acknowledgment
of the user message.

When controlling the UI, you can add, update and replace surfaces.
- Use 'add' to create new surfaces.
- Use 'update' to make small changes to existing surfaces.
- Use 'replace' to completely change the content of a surface.
''',
catalog: CoreCatalogItems.asCatalog(),
onSurfaceAdded: _handleSurfaceAdded,
Expand Down
17 changes: 10 additions & 7 deletions examples/travel_app/lib/main.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ class _TravelPlannerPageState extends State<TravelPlannerPage> {
_conversation.removeWhere(
(m) => m is AiUiMessage && m.surfaceId == surfaceId,
);
case SurfaceUpdated(:final surfaceId, :final definition):
case SurfaceChanged(:final surfaceId, :final definition):
final index = _conversation.lastIndexWhere(
(m) => m is AiUiMessage && m.surfaceId == surfaceId,
);
Expand Down Expand Up @@ -375,7 +375,6 @@ to the user.
of time, the budget, preferred activity types etc.

Then, when the user clicks search, you should update the surface to have
<<<<<<< HEAD
a Column with the existing inputGroup, an itineraryWithDetails. When
creating the itinerary, include all necessary `itineraryEntry` items for
hotels and transport with generic details and a status of `choiceRequired`.
Expand Down Expand Up @@ -424,11 +423,15 @@ update existing content.
- Adding surfaces: Most of the time, you should only add new surfaces to the conversation. This
is less confusing for the user, because they can easily find this new content
at the bottom of the conversation.
- Updating surfaces: You should update surfaces when you are running an
iterative search flow, e.g. the user is adjusting filter values and generating
an itinerary or a booking accomodation etc. This is less confusing for the user
because it avoids confusing the conversation with many versions of the same
itinerary etc.
- Updating surfaces: You should update surfaces when you want to make small
changes to an existing surface, for example, when you are running an
iterative search flow, e.g. the user is adjusting filter values and
generating an itinerary or a booking accomodation etc. This is less confusing
for the user because it avoids confusing the conversation with many versions
of the same itinerary etc.
- Replacing surfaces: You should replace surfaces when you want to completely
change the content of a surface, for example, when you are starting a new
search or moving to a different part of the conversation flow.

When processing a user message or event, you should add or update one surface
and then call provideFinalOutput to return control to the user. Never continue
Expand Down
2 changes: 1 addition & 1 deletion packages/flutter_genui/IMPLEMENTATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ This layer is responsible for all communication with the generative AI model.

This is the central nervous system of the package, orchestrating the state of all generated UI surfaces.

- **`GenUiManager`**: The core state manager for the dynamic UI. It maintains a map of all active UI "surfaces", where each surface is represented by a `UiDefinition`. It provides the tools (`addOrUpdateSurface`, `deleteSurface`) that the AI uses to manipulate the UI. It exposes a stream of `GenUiUpdate` events (`SurfaceAdded`, `SurfaceUpdated`, `SurfaceRemoved`) so that the application can react to changes. It also acts as the `GenUiHost` for the `GenUiSurface` widget.
- **`GenUiManager`**: The core state manager for the dynamic UI. It maintains a map of all active UI "surfaces", where each surface is represented by a `UiDefinition`. It provides the tools (`addOrUpdateSurface`, `deleteSurface`) that the AI uses to manipulate the UI. It exposes a stream of `GenUiUpdate` events (`SurfaceAdded`, `SurfaceChanged`, `SurfaceRemoved`) so that the application can react to changes. It also acts as the `GenUiHost` for the `GenUiSurface` widget.
- **`ui_tools.dart`**: Contains the `AddOrUpdateSurfaceTool` and `DeleteSurfaceTool` classes that wrap the `GenUiManager`'s methods, making them available to the AI.

### 3. UI Model Layer (`lib/src/model/`)
Expand Down
63 changes: 51 additions & 12 deletions packages/flutter_genui/lib/src/core/genui_manager.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import 'ui_tools.dart';

/// A sealed class representing an update to the UI managed by [GenUiManager].
///
/// This class has three subclasses: [SurfaceAdded], [SurfaceUpdated], and
/// This class has three subclasses: [SurfaceAdded], [SurfaceChanged], and
/// [SurfaceRemoved].
sealed class GenUiUpdate {
/// Creates a [GenUiUpdate] for the given [surfaceId].
Expand All @@ -42,10 +42,10 @@ class SurfaceAdded extends GenUiUpdate {
}

/// Fired when an existing surface is modified.
class SurfaceUpdated extends GenUiUpdate {
/// Creates a [SurfaceUpdated] event for the given [surfaceId] and
class SurfaceChanged extends GenUiUpdate {
/// Creates a [SurfaceChanged] event for the given [surfaceId] and
/// [definition].
const SurfaceUpdated(super.surfaceId, this.definition);
const SurfaceChanged(super.surfaceId, this.definition);

/// The new definition of the surface.
final UiDefinition definition;
Expand Down Expand Up @@ -119,7 +119,7 @@ class GenUiManager implements GenUiHost {
if (event is! UiActionEvent) throw ArgumentError('Unexpected event type');
final stateValue = valueStore.forSurface(event.surfaceId);
final eventString =
'Action: ${jsonEncode(event.value)}\n'
'Action: ${jsonEncode(event.value)}\\n'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The change from \n to \\n appears to be a mistake. The double backslash escapes the newline character, which means a literal \n string will be sent to the model instead of a newline. This is unlikely to be interpreted as a line break by the AI. The original single backslash \n was correct for creating a newline character within the string.

Suggested change
'Action: ${jsonEncode(event.value)}\\n'
'Action: ${jsonEncode(event.value)}\n'

'Current state: ${jsonEncode(stateValue)}';
_onSubmit.add(UserMessage([TextPart(eventString)]));
}
Expand Down Expand Up @@ -164,19 +164,58 @@ class GenUiManager implements GenUiHost {
/// If a surface with the given ID does not exist, a new one is created.
/// Otherwise, the existing surface is updated.
void addOrUpdateSurface(String surfaceId, JsonMap definition) {
final uiDefinition = UiDefinition.fromMap({
'surfaceId': surfaceId,
...definition,
});
final action = definition['action'] as String? ?? 'replace';
final notifier = surface(surfaceId); // Gets or creates the notifier.
final isNew = notifier.value == null;
notifier.value = uiDefinition;

if (isNew) {
final uiDefinition = UiDefinition.fromMap({
'surfaceId': surfaceId,
...definition,
});
notifier.value = uiDefinition;
genUiLogger.info('Adding surface $surfaceId');
_surfaceUpdates.add(SurfaceAdded(surfaceId, uiDefinition));
} else {
genUiLogger.info('Updating surface $surfaceId');
_surfaceUpdates.add(SurfaceUpdated(surfaceId, uiDefinition));
switch (action) {
case 'replace':
final uiDefinition = UiDefinition.fromMap({
'surfaceId': surfaceId,
...definition,
});
notifier.value = uiDefinition;
genUiLogger.info('Replacing surface $surfaceId');
_surfaceUpdates.add(SurfaceChanged(surfaceId, uiDefinition));
case 'update':
final currentDefinition = notifier.value!;
assert(
definition['root'] == currentDefinition.root,
'The root widget ID must be the same for update actions.',
);
final newWidgetsList = definition['widgets'] as List<Object?>;
final newWidgetsMap = <String, JsonMap>{};
for (final widget in newWidgetsList) {
final typedWidget = widget as JsonMap;
newWidgetsMap[typedWidget['id'] as String] = typedWidget;
}
Comment on lines +195 to +200
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The code in this block uses several direct casts (as) on the definition map, which is populated from AI-generated content. This could lead to runtime exceptions if the AI model provides a malformed response that doesn't adhere to the expected schema. To make the code more robust, consider using type checks (is) and conditional casts (as?) to gracefully handle potential data inconsistencies.

For example:

final newWidgetsList = definition['widgets'];
if (newWidgetsList is! List) {
  // Handle error, e.g., throw an ArgumentError or log a warning.
  return;
}
for (final widget in newWidgetsList) {
  if (widget is! JsonMap) {
    // Handle error
    continue;
  }
  final id = widget['id'];
  if (id is! String) {
    // Handle error
    continue;
  }
  newWidgetsMap[id] = widget;
}


final updatedWidgets = {
...currentDefinition.widgets,
...newWidgetsMap,
};

// TODO(andrewkolb): Prune orphaned widgets.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The TODO comment highlights a significant issue. The current implementation for the update action merges new widgets but does not remove widgets that are no longer referenced in the widget tree. This will cause an accumulation of "orphaned" widgets in the UiDefinition, leading to memory bloat and potential performance degradation over time. This should be addressed to ensure the long-term stability of the application.

final uiDefinition = UiDefinition.fromMap({
'surfaceId': surfaceId,
'root': currentDefinition.root,
'widgets': updatedWidgets.values.toList(),
});
notifier.value = uiDefinition;
genUiLogger.info('Updating surface $surfaceId');
_surfaceUpdates.add(SurfaceChanged(surfaceId, uiDefinition));
default:
throw ArgumentError('Invalid action: $action');
}
}
}

Expand Down
25 changes: 20 additions & 5 deletions packages/flutter_genui/lib/src/core/ui_tools.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,23 @@ class AddOrUpdateSurfaceTool extends AiTool<JsonMap> {
'action': S.string(
description:
'The action to perform. You must choose from the available '
'actions. If you choose the `add` action, you must choose a '
'new unique surfaceId. If you choose the `update` action, '
'you must choose an existing surfaceId.',
'actions.\n'
'- `add`: Creates a new surface. You must choose a new, '
'unique `surfaceId`.\n'
'- `update`: Updates an existing surface by adding or '
'replacing individual widgets. This is efficient for small '
'changes, as it preserves the rest of the widget tree. The '
'`root` widget ID must be the same as the original '
'surface.\n'
'- `replace`: Replaces the entire content of an existing '
'surface. This is for when the entire UI needs to be '
'changed.',
enumValues: [
if (configuration.actions.allowCreate) 'add',
if (configuration.actions.allowUpdate) 'update',
if (configuration.actions.allowUpdate) ...[
'update',
'replace',
],
],
),
'surfaceId': S.string(
Expand All @@ -50,7 +61,9 @@ class AddOrUpdateSurfaceTool extends AiTool<JsonMap> {
'root': S.string(
description:
'The ID of the root widget. This ID must correspond to '
'the ID of one of the widgets in the `widgets` list.',
'the ID of one of the widgets in the `widgets` list. '
'For `update` actions, this must be the same as the '
'original surface.',
),
'widgets': S.list(
items: catalog.schema,
Expand Down Expand Up @@ -78,6 +91,8 @@ class AddOrUpdateSurfaceTool extends AiTool<JsonMap> {
Future<JsonMap> invoke(JsonMap args) async {
final surfaceId = args['surfaceId'] as String;
final definition = args['definition'] as JsonMap;
final action = args['action'] as String;
definition['action'] = action;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This line mutates the definition map, which is part of the input args. While it may work here, modifying input parameters is generally considered bad practice as it can lead to unexpected side effects and make the code harder to reason about. A cleaner approach would be to create a new map that includes the action property, leaving the original definition map unchanged.

For example:

    final newDefinition = {
      ...definition,
      'action': action,
    };
    onAddOrUpdate(surfaceId, newDefinition);

onAddOrUpdate(surfaceId, definition);
return {'surfaceId': surfaceId, 'status': 'SUCCESS'};
}
Expand Down
2 changes: 1 addition & 1 deletion packages/flutter_genui/lib/src/facade/ui_agent.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ class UiAgent {
);
_addMessage(message);
onSurfaceDeleted!.call(update);
} else if (update is SurfaceUpdated) {
} else if (update is SurfaceChanged) {
final message = AiUiMessage(
definition: update.definition,
surfaceId: update.surfaceId,
Expand Down
83 changes: 69 additions & 14 deletions packages/flutter_genui/test/core/genui_manager_test.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
// found in the LICENSE file.

import 'package:flutter_genui/flutter_genui.dart';
import 'package:flutter_genui/src/primitives/simple_items.dart';
import 'package:flutter_test/flutter_test.dart';

void main() {
Expand Down Expand Up @@ -55,28 +56,29 @@ void main() {
expect(manager.surfaces['s1']!.value!.root, 'root');
});

test('addOrUpdateSurface updates an existing surface and fires '
'SurfaceUpdated', () async {
test('addOrUpdateSurface with "replace" action updates an existing surface '
'and fires SurfaceChanged', () async {
final oldDefinition = {
'root': 'root',
'widgets': [
{
'id': 'root',
'widget': {
'text': {'text': 'Old'},
'Text': {'text': 'Old'},
},
},
],
};
manager.addOrUpdateSurface('s1', oldDefinition);

final newDefinition = {
'action': 'replace',
'root': 'root',
'widgets': [
{
'id': 'root',
'widget': {
'text': {'text': 'New'},
'Text': {'text': 'New'},
},
},
],
Expand All @@ -86,16 +88,69 @@ void main() {
manager.addOrUpdateSurface('s1', newDefinition);
final update = await futureUpdate;

expect(update, isA<SurfaceUpdated>());
expect(update, isA<SurfaceChanged>());
expect(update.surfaceId, 's1');
final updatedDefinition = (update as SurfaceUpdated).definition;
expect(updatedDefinition.widgets['root'], {
'id': 'root',
'widget': {
'text': {'text': 'New'},
},
});
expect(manager.surfaces['s1']!.value, updatedDefinition);
final changedUpdate = update as SurfaceChanged;
expect(
((changedUpdate.definition.widgets['root']! as JsonMap)['widget']
as JsonMap)['Text'],
{'text': 'New'},
);
expect(manager.surfaces['s1']!.value, changedUpdate.definition);
});

test('addOrUpdateSurface with "update" action updates an existing surface '
'and fires SurfaceChanged', () async {
final oldDefinition = {
'root': 'root',
'widgets': [
{
'id': 'root',
'widget': {
'Text': {'text': 'Old'},
},
},
{
'id': 'child',
'widget': {
'Text': {'text': 'Child'},
},
},
],
};
manager.addOrUpdateSurface('s1', oldDefinition);

final newDefinition = {
'action': 'update',
'root': 'root',
'widgets': [
{
'id': 'root',
'widget': {
'Text': {'text': 'New'},
},
},
],
};

final futureUpdate = manager.surfaceUpdates.first;
manager.addOrUpdateSurface('s1', newDefinition);
final update = await futureUpdate;

expect(update, isA<SurfaceChanged>());
expect(update.surfaceId, 's1');
final changedUpdate = update as SurfaceChanged;
expect(
((changedUpdate.definition.widgets['root']! as JsonMap)['widget']
as JsonMap)['Text'],
{'text': 'New'},
);
expect(
((changedUpdate.definition.widgets['child']! as JsonMap)['widget']
as JsonMap)['Text'],
{'text': 'Child'},
);
expect(manager.surfaces['s1']!.value, changedUpdate.definition);
});

test('deleteSurface removes a surface and fires SurfaceRemoved', () async {
Expand All @@ -105,7 +160,7 @@ void main() {
{
'id': 'root',
'widget': {
'text': {'text': 'Hello'},
'Text': {'text': 'Hello'},
},
},
],
Expand Down
1 change: 1 addition & 0 deletions packages/flutter_genui/test/core/ui_tools_test.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ void main() {
);

final args = {
'action': 'add',
'surfaceId': 'testSurface',
'definition': {
'root': 'rootWidget',
Expand Down
Loading