Skip to content

Commit c927c2e

Browse files
SamYuan1990SamYuan1990
authored
demo for no stream vad asr with flutter (#2705)
Signed-off-by: SamYuan1990 <[email protected]>
1 parent 5aa8a95 commit c927c2e

36 files changed

+2228
-0
lines changed

‎flutter-examples/non_streaming_vad_asr/.gitignore‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Miscellaneous
2+
*.class
3+
*.log
4+
*.pyc
5+
*.swp
6+
.DS_Store
7+
.atom/
8+
.build/
9+
.buildlog/
10+
.history
11+
.svn/
12+
.swiftpm/
13+
migrate_working_dir/
14+
15+
# IntelliJ related
16+
*.iml
17+
*.ipr
18+
*.iws
19+
.idea/
20+
21+
# The .vscode folder contains launch configuration and tasks you configure in
22+
# VS Code which you may wish to be included in version control, so this line
23+
# is commented out by default.
24+
#.vscode/
25+
26+
# Flutter/Dart/Pub related
27+
**/doc/api/
28+
**/ios/Flutter/.last_build_id
29+
.dart_tool/
30+
.flutter-plugins
31+
.flutter-plugins-dependencies
32+
.pub-cache/
33+
.pub/
34+
/build/
35+
36+
# Symbolication related
37+
app.*.symbols
38+
39+
# Obfuscation related
40+
app.*.map.json
41+
42+
# Android Studio will place build artifacts here
43+
/android/app/debug
44+
/android/app/profile
45+
/android/app/release

‎flutter-examples/non_streaming_vad_asr/README.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Real-time speech recognition by non streaming and VAD
2+
3+
This APP supports the following platforms:
4+
5+
- macOS (tested)
6+
7+
## Getting Started
8+
9+
Follow these steps to download and set up the required models to run the demo successfully.
10+
11+
### 1. Select a non-streaming model
12+
13+
Choose one of the following non-streaming ASR models:
14+
15+
#### Code Available Models:
16+
- **whisper**: Whisper base model
17+
- **senseVoice**: SenseVoice multilingual model (supports Chinese, English, Japanese, Korean, Cantonese)
18+
- **parakeet-tdt**: NeMo transducer-based parakeet-tdt model
19+
20+
#### Model Download Links:
21+
- **whisper**: https://huggingface.co/csukuangfj/sherpa-onnx-whisper-base
22+
- **senseVoice**: https://huggingface.co/csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2025-09-09
23+
- **parakeet-tdt**: https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8
24+
25+
### 2. Download VAD Model
26+
27+
Download the VAD (Voice Activity Detection) model from:
28+
https://huggingface.co/csukuangfj/vad
29+
30+
Place the VAD model file (e.g., `silero_vad.onnx`) in the `assets` directory.
31+
32+
### 3. Configure the Model in Code
33+
34+
#### Step 3.1: Update Model Selection
35+
Edit `lib/non_streaming_vad_asr.dart` and set the model type:
36+
37+
```dart
38+
Future<sherpa_onnx.OfflineRecognizer> createOfflineRecognizer() async {
39+
final type = 2; // 0: whisper, 1: senseVoice, 2: parakeet-tdt
40+
final modelConfig = await getOfflineModelConfig(type: type);
41+
final config = sherpa_onnx.OfflineRecognizerConfig(model: modelConfig);
42+
return sherpa_onnx.OfflineRecognizer(config);
43+
}
44+
```
45+
46+
#### Step 3.2: Update Asset Configuration
47+
Edit `pubspec.yaml` and add the appropriate asset directory for your chosen model:
48+
49+
```yaml
50+
flutter:
51+
assets:
52+
- assets/
53+
- assets/whisper/ # For whisper model
54+
# - assets/senseVoice/ # For senseVoice model (uncomment when using)
55+
# - assets/nemo_transducer/ # For parakeet-tdt model (uncomment when using)
56+
```
57+
58+
### 4. Directory Structure Setup
59+
60+
#### For whisper model:
61+
```
62+
./assets/
63+
├── whisper/
64+
│ ├── base-decoder.onnx
65+
│ ├── base-encoder.onnx
66+
│ └── base-tokens.txt
67+
└── silero_vad.onnx
68+
```
69+
70+
#### For senseVoice model:
71+
```
72+
./assets/
73+
├── senseVoice/
74+
│ ├── model.int8.onnx
75+
│ └── tokens.txt
76+
└── silero_vad.onnx
77+
```
78+
79+
#### For parakeet-tdt model:
80+
```
81+
./assets/
82+
├── nemo_transducer/
83+
│ ├── encoder.int8.onnx
84+
│ ├── decoder.int8.onnx
85+
│ ├── joiner.int8.onnx
86+
│ └── tokens.txt
87+
└── silero_vad.onnx
88+
```
89+
90+
### 5. Advanced Configuration (Optional)
91+
92+
#### Modify Model Configuration:
93+
You can edit `lib/offline_model.dart` to customize the model configuration, such as model size and quantization settings.
94+
95+
#### Adjust Audio Recording Settings:
96+
In `lib/non_streaming_vad_asr.dart`, you can modify the VAD configuration:
97+
98+
```dart
99+
_vad = sherpa_onnx.VoiceActivityDetector(
100+
config: _vadConfig,
101+
bufferSizeInSeconds: 30 // Adjust based on your needs
102+
);
103+
_buffer = sherpa_onnx.CircularBuffer(capacity: 30 * 16000);
104+
```
105+
106+
### 6. Run the Application
107+
108+
Use the following command to run the app:
109+
110+
```bash
111+
flutter run -d macos
112+
```
113+
114+
## Troubleshooting
115+
116+
- Ensure all model files are placed in the correct directories
117+
- Check that `pubspec.yaml` includes the correct asset paths
118+
- Verify the model type in `non_streaming_vad_asr.dart` matches your chosen model
119+
- Make sure to delete unnecessary files to reduce app size
120+
121+
## Notes
122+
123+
- The VAD model is required for all non-streaming ASR models
124+
- Model performance may vary depending on hardware capabilities
125+
- Adjust buffer sizes and VAD parameters based on your specific use case

‎flutter-examples/non_streaming_vad_asr/analysis_options.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# This file configures the analyzer, which statically analyzes Dart code to
2+
# check for errors, warnings, and lints.
3+
#
4+
# The issues identified by the analyzer are surfaced in the UI of Dart-enabled
5+
# IDEs (https://dart.dev/tools#ides-and-editors). The analyzer can also be
6+
# invoked from the command line by running `flutter analyze`.
7+
8+
# The following line activates a set of recommended lints for Flutter apps,
9+
# packages, and plugins designed to encourage good coding practices.
10+
include: package:flutter_lints/flutter.yaml
11+
12+
linter:
13+
# The lint rules applied to this project can be customized in the
14+
# section below to disable rules from the `package:flutter_lints/flutter.yaml`
15+
# included above or to enable additional rules. A list of all available lints
16+
# and their documentation is published at https://dart.dev/lints.
17+
#
18+
# Instead of disabling a lint rule for the entire project in the
19+
# section below, it can also be suppressed for a single line of code
20+
# or a specific dart file by using the `// ignore: name_of_lint` and
21+
# `// ignore_for_file: name_of_lint` syntax on the line or in the file
22+
# producing the lint.
23+
rules:
24+
# avoid_print: false # Uncomment to disable the `avoid_print` rule
25+
# prefer_single_quotes: true # Uncomment to enable the `prefer_single_quotes` rule
26+
27+
# Additional information about this file can be found at
28+
# https://dart.dev/guides/language/analysis-options

‎flutter-examples/non_streaming_vad_asr/lib/info.dart‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
// Copyright (c) 2024 Xiaomi Corporation
2+
import 'package:flutter/material.dart';
3+
import 'package:url_launcher/url_launcher.dart';
4+
5+
class InfoScreen extends StatelessWidget {
6+
@override
7+
Widget build(BuildContext context) {
8+
const double height = 20;
9+
return Container(
10+
child: Padding(
11+
padding: const EdgeInsets.all(8.0),
12+
child: Column(
13+
crossAxisAlignment: CrossAxisAlignment.start,
14+
children: <Widget>[
15+
Text('Everything is open-sourced.'),
16+
SizedBox(height: height),
17+
InkWell(
18+
child: Text('Code: https://github.com/k2-fsa/sherpa-onnx'),
19+
onTap: () => launch('https://k2-fsa.github.io/sherpa/onnx/'),
20+
),
21+
SizedBox(height: height),
22+
InkWell(
23+
child: Text('Doc: https://k2-fsa.github.io/sherpa/onnx/'),
24+
onTap: () => launch('https://k2-fsa.github.io/sherpa/onnx/'),
25+
),
26+
SizedBox(height: height),
27+
Text('QQ 群: 744602236'),
28+
SizedBox(height: height),
29+
InkWell(
30+
child: Text(
31+
'微信群: https://k2-fsa.github.io/sherpa/social-groups.html'),
32+
onTap: () =>
33+
launch('https://k2-fsa.github.io/sherpa/social-groups.html'),
34+
),
35+
],
36+
),
37+
),
38+
);
39+
}
40+
}

‎flutter-examples/non_streaming_vad_asr/lib/main.dart‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
// Copyright (c) 2024 Xiaomi Corporation
2+
import 'package:flutter/material.dart';
3+
4+
import './non_streaming_vad_asr.dart';
5+
import './info.dart';
6+
7+
void main() {
8+
runApp(const MyApp());
9+
}
10+
11+
class MyApp extends StatelessWidget {
12+
const MyApp({super.key});
13+
14+
@override
15+
Widget build(BuildContext context) {
16+
return MaterialApp(
17+
title: 'Next-gen Kaldi flutter demo',
18+
theme: ThemeData(
19+
colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
20+
useMaterial3: true,
21+
),
22+
home: const MyHomePage(title: 'Next-gen Kaldi with Flutter'),
23+
);
24+
}
25+
}
26+
27+
class MyHomePage extends StatefulWidget {
28+
const MyHomePage({super.key, required this.title});
29+
30+
final String title;
31+
32+
@override
33+
State<MyHomePage> createState() => _MyHomePageState();
34+
}
35+
36+
class _MyHomePageState extends State<MyHomePage> {
37+
int _currentIndex = 0;
38+
final List<Widget> _tabs = [
39+
NoStreamingAsrVAdScreen(),
40+
InfoScreen(),
41+
];
42+
@override
43+
Widget build(BuildContext context) {
44+
return Scaffold(
45+
appBar: AppBar(
46+
title: Text(widget.title),
47+
),
48+
body: _tabs[_currentIndex],
49+
bottomNavigationBar: BottomNavigationBar(
50+
currentIndex: _currentIndex,
51+
onTap: (int index) {
52+
setState(() {
53+
_currentIndex = index;
54+
});
55+
},
56+
items: [
57+
BottomNavigationBarItem(
58+
icon: Icon(Icons.home),
59+
label: 'Home',
60+
),
61+
BottomNavigationBarItem(
62+
icon: Icon(Icons.info),
63+
label: 'Info',
64+
),
65+
],
66+
),
67+
);
68+
}
69+
}

0 commit comments

Comments
 (0)