You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use `getstream.video.rtc.AudioTrack` if you need to publish audio using PyAV, this class ensures that `recv` paces audio correctly every 20ms.
348
+
Use `getstream.video.rtc.AudioTrack` if you need to publish audio using PyAV, this class ensures that `recv` paces audio
349
+
correctly every 20ms.
322
350
323
351
- Use `.write()` method to enqueue audio (PcmData)
324
352
- Use `.flush()` to empty all the enqueued audio (eg. barge-in event)
@@ -347,8 +375,10 @@ This prevents mistakes related to handling audio with different formats, sample
347
375
348
376
### Testing
349
377
350
-
Many of the underlying APIs change daily. To ensure things work we keep 2 sets of tests. Integration tests and unit tests.
351
-
Integration tests run once a day to verify that changes to underlying APIs didn't break the framework. Some testing guidelines
378
+
Many of the underlying APIs change daily. To ensure things work we keep 2 sets of tests. Integration tests and unit
379
+
tests.
380
+
Integration tests run once a day to verify that changes to underlying APIs didn't break the framework. Some testing
381
+
guidelines
352
382
353
383
- Every plugin needs an integration test
354
384
- Limit usage of response capturing style testing. (since they diverge from reality)
@@ -442,11 +472,13 @@ metrics.set_meter_provider(
442
472
start_http_server(port=9464)
443
473
```
444
474
445
-
You can now see the metrics at `http://localhost:9464/metrics` (make sure that your Python program keeps running), after this you can setup your Prometheus server to scrape this endpoint.
475
+
You can now see the metrics at `http://localhost:9464/metrics` (make sure that your Python program keeps running), after
476
+
this you can setup your Prometheus server to scrape this endpoint.
446
477
447
478
### Profiling
448
479
449
-
The `Profiler` class uses `pyinstrument` to profile your agent's performance and generate an HTML report showing where time is spent during execution.
480
+
The `Profiler` class uses `pyinstrument` to profile your agent's performance and generate an HTML report showing where
481
+
time is spent during execution.
450
482
451
483
#### Example usage:
452
484
@@ -456,6 +488,7 @@ from vision_agents.core import User, Agent
456
488
from vision_agents.core.profiling import Profiler
457
489
from vision_agents.plugins import getstream, gemini, deepgram, elevenlabs, vogent
- Stops profiling when the agent finishes (on `AgentFinishEvent`)
480
514
- Saves an HTML report to the specified output path (default: `./profile.html`)
481
515
482
-
You can open the generated HTML file in a browser to view the performance profile, which shows a timeline of function calls and where time is spent during agent execution.
483
-
516
+
You can open the generated HTML file in a browser to view the performance profile, which shows a timeline of function
517
+
calls and where time is spent during agent execution.
484
518
485
519
### Queuing
486
520
@@ -498,21 +532,23 @@ You can open the generated HTML file in a browser to view the performance profil
498
532
499
533
### Video Frames & Tracks
500
534
501
-
- Track.recv errors will fail silently. The API is to return a frame. Never return None. and wait till the next frame is available
502
-
- When using frame.to_ndarray(format="rgb24") specify the format. Typically you want rgb24 when connecting/sending to Yolo etc
535
+
- Track.recv errors will fail silently. The API is to return a frame. Never return None. and wait till the next frame is
536
+
available
537
+
- When using frame.to_ndarray(format="rgb24") specify the format. Typically you want rgb24 when connecting/sending to
538
+
Yolo etc
503
539
- QueuedVideoTrack is a writable/queued video track implementation which is useful when forwarding video
504
540
505
-
506
541
### Loading Resources in Plugins (aka "warmup")
507
-
Some plugins require to download and use external resources like models to work.
542
+
543
+
Some plugins require to download and use external resources like models to work.
508
544
509
545
For example:
510
546
511
547
-`TurnDetection` plugins using a Silero VAD model to detect voice activity in the audio track.
512
548
- Video processors using `YOLO` models
513
549
514
550
In order to standardise how these resources are loaded and to make it performant, the framework provides a special ABC
515
-
`vision_agents.core.warmup.Warmable`.
551
+
`vision_agents.core.warmup.Warmable`.
516
552
517
553
To use it, simply subclass it and define the required methods.
518
554
Note that `Warmable` supports generics to leverage type checking.
@@ -551,12 +587,10 @@ class FasterWhisperSTT(STT, Warmable[WhisperModel]):
551
587
# This method will be called every time a new agent is initialized.
0 commit comments