Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
df33082
Add NeMo model support + Refactor codebase (#2)
titu1994 Jul 25, 2023
40cc09a
updating the run_whisper script to work with recent changes
Vaibhavs10 Jul 26, 2023
8640704
add jiwer & librosa to requirements
Vaibhavs10 Jul 26, 2023
611b27d
fix: load_dataset -> load_data
Vaibhavs10 Jul 26, 2023
baafd88
Manifest based Scoring (#3)
titu1994 Jul 31, 2023
89c9ed3
Add XL and XXL RNNT and CTC models (#5)
nithinraok Aug 2, 2023
574f893
Add Speechbrain toolkit (#4)
Adel-Moumen Aug 3, 2023
a5f8268
up
Vaibhavs10 Aug 29, 2023
f184963
up
Vaibhavs10 Aug 29, 2023
5c60dbb
up
Vaibhavs10 Aug 29, 2023
868d771
up
Vaibhavs10 Aug 29, 2023
6b493e5
Merge pull request #6 from huggingface/loop_over_transformers_models
Vaibhavs10 Aug 29, 2023
27c4e42
up
Vaibhavs10 Aug 29, 2023
0f9c861
add hubert models
Vaibhavs10 Aug 29, 2023
6063541
add data2vec models
Vaibhavs10 Aug 29, 2023
320ed6d
add wavlm models
Vaibhavs10 Aug 29, 2023
581ec3e
remove non en models
Vaibhavs10 Aug 29, 2023
5658abf
update hubert models
Vaibhavs10 Aug 29, 2023
4fdc80f
remove wavlm models
Vaibhavs10 Aug 29, 2023
de89533
streaming -> False
Vaibhavs10 Aug 30, 2023
6707cc3
indentation fix -> evals
Vaibhavs10 Aug 30, 2023
cb3fa07
final eval configurations
Vaibhavs10 Aug 30, 2023
05c3c85
add MMS models:
Vaibhavs10 Aug 31, 2023
398a393
initial commit - rtf calculation script
Vaibhavs10 Sep 2, 2023
2594106
up
Vaibhavs10 Sep 2, 2023
7f11b87
Initial commit - RTF script
Vaibhavs10 Sep 2, 2023
fa39250
update rtf evals w/ all models
Vaibhavs10 Sep 2, 2023
d96dafc
update rtf evals
Vaibhavs10 Sep 4, 2023
9bdd19f
update rtf evals w/ batching
Vaibhavs10 Sep 4, 2023
ecd653c
disclaimer
Vaibhavs10 Sep 4, 2023
2123ed9
Update README.md
titu1994 Sep 5, 2023
0ee9789
Fix issue in NeMo eval for parallel runs overwriting the cache dir
titu1994 Sep 5, 2023
d6eae2a
add rtf and remaining nemo models
Sep 7, 2023
47a56bd
update rtf
Sep 7, 2023
6401ce6
Add doc strings
Sep 7, 2023
82a29ae
comments
Sep 7, 2023
8857427
Merge pull request #7 from huggingface/add_remaining_nemo_models
titu1994 Sep 7, 2023
cbd8b9b
Update the leaderboard URL (#8)
ofirkris Sep 11, 2023
9a596ac
Add support for `ctranslate2` whisper models (#10)
chainyo Sep 11, 2023
3f30227
add nvidia parakeet asr models
Dec 31, 2023
9a14e91
Merge pull request #16 from huggingface/add_parakeet_models
titu1994 Dec 31, 2023
23489be
remove calc rtf
Feb 7, 2024
42533fe
add short form rtf
Feb 7, 2024
7366879
remove expected keys
Feb 7, 2024
4090e61
remove iterator
Feb 7, 2024
7081b93
small changes
Feb 7, 2024
d61bf72
Update transformers/run_eval.py
sanchit-gandhi Feb 7, 2024
0356f5d
clean
Feb 7, 2024
37858e1
Merge remote-tracking branch 'origin/short-rtf' into short-rtf
Feb 7, 2024
e71813e
fp16 eval
Feb 7, 2024
a7ac57d
Add nemo aed models (#21)
nithinraok Feb 8, 2024
d59499c
canary_run_script
Feb 8, 2024
1b66532
Merge pull request #22 from huggingface/canary_run_script
nithinraok Feb 8, 2024
4eb6a16
add nemo chunk rtf cal
stevehuang52 Feb 9, 2024
8172ff6
refactor
stevehuang52 Feb 9, 2024
f913864
Merge pull request #23 from stevehuang52/add_nemo_chunk_infer
nithinraok Feb 9, 2024
60d2561
Add LICENSE (#24)
sanchit-gandhi Apr 11, 2024
36dd49b
added pnc flag for canary nemo asr eval
KunalDhawan May 15, 2024
5c03c1f
Merge pull request #25 from KunalDhawan/fix_canary_pnc
nithinraok May 16, 2024
0fb7a1a
un-weight rtfx
Jun 20, 2024
fbf860a
use sorted ds
Jul 3, 2024
03e7c79
pipeline -> model api
Jul 3, 2024
392fba2
update nemo inference and include RTFx
Jul 9, 2024
659bd15
update_nemo_branch
Jul 23, 2024
6b129b1
remove previous rtf cal scripts
Jul 23, 2024
086a537
swap the check file for faster iterations
Jul 23, 2024
4b03c48
move to sorted datasets
Jul 23, 2024
c8e9cc9
Merge pull request #20 from sanchit-gandhi/short-rtf
sanchit-gandhi Jul 29, 2024
2c3b482
update nemo inference and include RTFx
Jul 9, 2024
1fe34d2
update_nemo_branch
Jul 23, 2024
f08d3aa
remove previous rtf cal scripts
Jul 23, 2024
f87bf08
swap the check file for faster iterations
Jul 23, 2024
443571b
move to sorted datasets
Jul 23, 2024
687ca5d
minor clean up and warmup upto 4 batches
Jul 29, 2024
97cee60
update with remote
Jul 29, 2024
20a7543
Merge pull request #28 from nithinraok/update_nemo_infer
nithinraok Jul 29, 2024
48ddc44
Use faster-whisper model names instead of hardcoded HF names
jordimas Jul 30, 2024
6551c74
Switch to hf-audio/esb-datasets-test-only-sorted dataset
jordimas Jul 30, 2024
38a851a
updates
Aug 5, 2024
c70cb04
Merge pull request #29 from jordimas/ct2-model-names
sanchit-gandhi Aug 5, 2024
b979546
Remove Common Voice Dataset
jordimas Aug 5, 2024
25c2180
finalise
Aug 7, 2024
d0a5ec4
Merge pull request #32 from sanchit-gandhi/static-kv-2
sanchit-gandhi Aug 7, 2024
72d38d3
Merge pull request #30 from jordimas/ct2-newdataset
sanchit-gandhi Aug 7, 2024
a9a8808
update faster whisper for rtfx
Aug 7, 2024
b5334ae
remove calc rtf
Aug 7, 2024
e4aeb4f
finish faster whisper
Aug 7, 2024
336069a
update speechbrain script
Aug 7, 2024
904e29f
finish speechbrain
Aug 7, 2024
db75efa
fix indentation
Aug 7, 2024
ca8e7a2
Merge pull request #34 from sanchit-gandhi/update-other-libs
sanchit-gandhi Aug 9, 2024
b33bdf9
[transformers] from common voice from data2vec
Aug 9, 2024
4ea93a2
Merge pull request #35 from sanchit-gandhi/remove-cv-from-data2vec
sanchit-gandhi Aug 9, 2024
1734b3b
NeMo RTFx updates (#33)
galv Aug 9, 2024
9079e98
update trmfs scripts (#37)
sanchit-gandhi Aug 12, 2024
fe50cf0
[readme] update for rtfx (#36)
sanchit-gandhi Aug 14, 2024
1053c19
add CrisperWhisper model (#39)
LaurinmyReha Oct 29, 2024
7878c93
Add UsefulSensors Moonshine benchmark (#43)
njeffrie Nov 20, 2024
003f525
[Ready] Add whisper TensorRT-LLM (#42)
yuekaizhang Nov 23, 2024
b4fff33
fix whisper .en model (#46)
yuekaizhang Nov 29, 2024
a471a82
Adds Phi-4-Multimodal
Jan 26, 2025
e7fba76
Adding largescale ASR model for speechbrain (#49)
TParcollet Feb 27, 2025
f7b9566
Add requirements for Phi evals
Mar 4, 2025
0123743
Merge pull request #51 from freewym/work
Deep-unlearning Mar 4, 2025
16bf928
Rearrange requirement files
Mar 9, 2025
ebb27b1
Create docker runner
Mar 9, 2025
422defb
Fix reference to deprecated _asdict()
Mar 9, 2025
94d174c
Failover logic
Mar 9, 2025
e540248
Use model full name
Mar 9, 2025
933a9be
Ignore cache folder
Mar 9, 2025
c97c87b
Update readme.txt
Mar 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 137 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# speechbrain
*/pretrained_models/*
*/model_checkpoints/*
*/results/*

# Docker cache mountpoint
.huggingface
201 changes: 201 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.

"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:

(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.

You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright [2024] [The HuggingFace Inc. team]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Loading