Apple's SpeechAnalyzer Outperforms Whisper Small in English Benchmark
Apple's on-device SpeechAnalyzer now offers a compelling alternative to Whisper for developers building iOS and macOS apps that require long-form transcription, at least for English read speech.
Reporting from 1 source: GIGAZINE.
Inscribe's development team benchmarked Apple's on-device SpeechAnalyzer against OpenAI's Whisper models (Tiny, Base, Small) and the legacy SFSpeechRecognizer using the LibriSpeech dataset. SpeechAnalyzer recorded a word error rate of 2.12% on clean speech and 4.56% on noisy speech, beating Whisper Small's 3.74% and 7.95% respectively. It also processed audio roughly three times faster per second than Whisper Small. The testing was limited to English read speech, and real-world conditions like accents or meetings may differ.
The benchmark used the LibriSpeech dataset, which includes 2,620 clean read speech samples and 2,939 noisy samples, and measured word error rate (WER). SpeechAnalyzer achieved a WER of 2.12% on clean speech and 4.56% on noisy speech, compared to Whisper Small's 3.74% and 7.95% and the legacy SFSpeechRecognizer's 9.02% and 16.25%.
All processing ran on a Mac with an M2 Pro and 32GB of RAM, with no data sent to external servers. SpeechAnalyzer also processed one second of audio in about one-third the time of Whisper Small. Inscribe noted that the Mac was running development workloads during the test, so times may vary, and they plan to retest on an idle machine.
The tests were limited to English read speech. Inscribe said it will prioritize SpeechAnalyzer for supported languages in its app and fall back to Whisper for others.
Synthesized by Yomimono from the 1 cited source below, including Japanese-language reporting where cited, then editorially reviewed before publishing.