File tree Expand file tree Collapse file tree 1 file changed +9
-7
lines changed Expand file tree Collapse file tree 1 file changed +9
-7
lines changed Original file line number Diff line number Diff line change @@ -1568,13 +1568,15 @@ message ModelSequenceBatching
15681568 //@@
15691569 repeated State state = 5 ;
15701570
1571- //@@ .. cpp:var:: bool generative_sequence
1572- //@@
1573- //@@ The sequence batcher is expecting the sequence to be generative. A
1574- //@@ generative sequence is initiated by single request, the sequence
1575- //@@ batcher expects the same request to be "rescheduled" by the model if
1576- //@@ the sequence is continuing.
1577- bool generative_sequence = 6 ;
1571+ //@@ .. cpp:var:: bool iterative_sequence
1572+ //@@
1573+ //@@ Requests for iterative sequences are processed over a number
1574+ //@@ of iterations. An iterative sequence is initiated by a single
1575+ //@@ request and is "rescheduled" by the model until completion.
1576+ //@@ Requests for inflight requests will be batched together
1577+ //@@ and can complete independently. Note this feature
1578+ //@@ requires backend support. Default value is false.
1579+ bool iterative_sequence = 6 ;
15781580}
15791581
15801582//@@
You can’t perform that action at this time.
0 commit comments