Newer linux and newer python libraries result in notably slower computation

[adantart]

It's weird and I want to share it here, since it could generate more problems or secondary effects in other systems.

I have a t2.2xlarge instance in Amazon with same configuration when launching the instance ... step by step, exactly the SAME configuration, same storage HD (30Gb gp2 100 IOPS), same networking, same everything.

• Software Image (AMI): Canonical, Ubuntu, 22.04 LTS, amd64 jammy
• t2.2xlarge
• Storage (volumes): 1 volume(s) - 30 GiB (30Gb gp2 100 IOPS)
• (all rest by default)

I do exactly the same commands at the beginning until the first test with Whisper. Same commands, same updates, same mp3, same everything ;-)

sudo apt-get update
sudo apt-get -y install python3-pip ffmpeg
pip3 install git+https://github.com/openai/whisper.git
export PATH=$PATH:/home/ubuntu/.local/bin
wget {my mp3}
whisper --model medium file.mp3

NOTE: first I do the last command to download the model, and then a second command, that is the one I take the execution time

In the first instance I launched 2 months ago, I got 4 minutes 5 seconds of my MP3 to be "Whispered"

Now, I've followed (again, sorry to repeat) the same launch protocol, same configuration, same command lines ... and I got 6 minutes 50 seconds.

After several hours comparing everything, the only different thing that Amazon setup differently is the AMI, and therefore some python librearies installed during the "whisper installation command":

Feature	The old one (the fastest)	The new one (slowest)
Linux AMI	ubuntu-jammy-22.04-amd64-server-20230115	ubuntu-jammy-22.04-amd64-server-20230208
huggingface-hub	0.12.0	0.12.1
numpy	1.24.1	1.24.2
python-apt	2.4.0	2.4.0+ubuntu1
setuptools	66.1.1	67.3.2
transformers	4.26.0	4.26.1
typing_extensions	4.4.0	4.5.0
ubuntu-advantage-tools	27.12	8001
Execution time of the test (just whisper, no download time)	4' 05"	6' 50"

How is possible than a "newer" AMI, with "newer" python libraries leads to a notably slower computation ?

Thank you in advance !

[glangford]

It seems pretty unlikely that such a big difference in execution time (>50% slower) is the result of these library changes. Such a big performance drop would have been noticed before.

This instance type is CPU only (no GPU present, correct me if I'm wrong). Given the big disparity, could the difference have something to do with the number of threads used? Just a guess. You could try changing your command line to use --threads to something appropriate. Something else running on the instance?

Note also that if you installed whisper two months ago with that command, you would be installing from the main branch at that time compared to now. Probably better in general to install a named release (eg. pip install "openai-whisper==20230124").

[glangford]

It looks like these T2 instances are also "burstable" and I'm not sure how to compare even identical configurations over time. The performance offered depends on accumulated CPU credits. So it's possible that you will get better performance by sitting idle for a while before running whisper to earn enough credits -

T2 instances are Burstable Performance Instances that provide a baseline level of CPU performance with the ability to burst above the baseline. The baseline performance and ability to burst are governed by CPU Credits. T2 instances accumulate CPU Credits when they are idle, and consume CPU Credits when they are active.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html

[adantart]

Hi @glangford

1. Yes, t2 instances are CPU. It's not about threads, or other argument since I have done the tests with all the arguments defined, just in case the installation of whispers could be different and have different default values.

2. Very interesting the "burstable" concept of Amazon instances, but it's not the case either. I've cloned the "old" instance (the "fastest") to have the same linux version (Ubuntu Jammy 20230115), and it's also fast. So your theory about "burstable" is not the answer here, since with instances just created, happens the same.

3. In both systems, the openai-whisper = 20230124 (I reinstalled in the new one to have the same whisper version).

Until now, although it's not an answer I like ... it seems the "linux version" (maybe some drivers) or the python libraries is the difference and the reason.

Please, tell me anything I can do in the systems to try to catch this.

As far as I know now ... I recommend the linux the Ubuntu Jammy 20230115 since for me, it's responding 40% faster than any other newer system

[glangford]

Until now, although it's not an answer I like ... it seems the "linux version" (maybe some drivers) or the python libraries is the difference and the reason.

I thought I was on to something with burstable...oh well. :)

So there are some interesting performance complaints on 22.04, for example here - I don't use this distribution but maybe this is the next most likely place to look??

https://askubuntu.com/questions/1406263/ubuntu-22-04-runs-too-slow
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1973434

[adantart]

Well, both systems are 22.04 ... the change is just the "version".
I'm not "complaining" about the new system, I can work with the old one since I have the AMI (image)
It's not about "22.04" is slow...

This post is about "the last version of 22.04" runs whisper 40% slower than the version of january. (in the exactly the SAME machine, SAME configuration, SAME commands, SAME installation)

But I wanted to post here because maybe somebody wants to investigate further.

;-)