Can a computer analyze audio quicker than real time playback?

Question

So let’s say that your computer is transcribing audio (of someone speaking) to text. Because it’s looking at the digital values of the audio, does it “render” the transcription quicker than the time it takes to play it in real time? I would imagine that it is not “listening” like a human would, rather it processes it digitally. Am I right in this assumption?

The same question would apply to analyzing video.

My confusion is: When playing audio back at a faster rate, the words become unclear, so how does the computer compensate for that? Excuse me if I am missing something basic and fundamental here.

---

Edit: When I use the term “real time” in this question, I don't mean at the time of recording, and then transcribing in real time. Rather, I mean playback at 1x speed (or real time playback speed). It seems some people didn't catch what I meant.

Answer 1

Yes. Absolutely.

Algorithms can process data as fast as they can read them and get them through the CPU.

If data is on disk, for example, a modern NVMe can read at 5+ GB/s which is much faster than bit-rates normally used to store voice data. Of course, the actual algorithm being applied can be more or less complex, so we cannot guarantee it will be processed at the maximum read speed but there is nothing inherent that limits such analysis to be in real-time speed.

The same principle applies to video but that requires much more throughput due to the huge amount of data in such files. That obviously depends on resolution, frame-rate and complexity of the analysis. It is actually difficult to perform sophisticated video analysis in real-time because analysis is almost always done on decompressed video, so the processor must have time to decode and analyze in a short period of time and keep data flowing so that by the time some analysis is done, the next block of video is already decoded and in memory. This is something I worked on for almost a decade.

When you playback video faster, words are unclear to you but the data is exactly the same. The speed at which audio is being processed does not affect the ability of the algorithm to understand it. Software knows exactly how much time each audio sample represents.

Answer 2

I'd go a bit further than the current answer, and would like to contradict the idea that the computer is somehow "playing back" the files at all. That would imply that processing is necessarily a strictly sequential process, starting from the beginning of the file and working your way towards the end.

In reality, most audio processing algorithms will be somewhat sequential - after all, that's how sound files are meant to be interpreted when playing them for human consumption. But other methods are conceivable: For example, say you want to write a program that determines the average loudness of a sound file. You could go through the whole file and measure the loudness of each snippet; but it would also be a valid (although maybe less accurate) strategy to just sample some snippets at random and measure those. Note that now the file isn't "played back" at all; the algorithm is simply looking at some data points that it chose by itself, it is free to do so in any order it likes.

This means that talking about "playing back" the file isn't really the right term here at all - even when the processing does happen sequentially, the computer isn't "listening" to sounds, it is simply processing a dataset (audio files aren't really anything other than a list of recorded air pressure values over time). Maybe the better analogy isn't a human listening to the audio, but analyzing it by looking at the waveform of the audio file:

In this case, you aren't at all constrained by the actual time scale of the audio, but can look at whatever part of the waveform you want for however long you want to (and if you are a fast enough, you can indeed "read" the waveform in a shorter time than playing the original audio would take). Of course, if it's a very long waveform printout, you might still have to "walk" for a bit to reach the section you are interested in (or if you are a computer, seek to the right position on the hard drive). But the speed that you're walking or reading isn't intrinsically linked to the (imaginary) time labels on the x-axis, i.e. the audio's "real-time".

Answer 3

Your core question is this:

“Can a computer analyze audio quicker than real time playback?”

Other great answers here but here is — what I consider — to be a very commonplace real-world example of computers analyzing audio faster than real-time audio playback…

Converting an audio CD to MP3 files on a modern computer system is always faster than real-time playback of the audio on that CD.

It all depends on the speed of your system and hardware, but even 20-ish years ago converting a CD to MP3 files was always faster than real-time playback of the CD audio.

So, for example, how can a 45 minute audio CD can be converted to MP3 in less than 45 minute? How could that occur if the computer was constrained by audio playback limits? It’s all data on the data side, but constrained to human levels on playback.

Think about it: A computer is reading the raw audio data from a CD at a speed faster than normal audio playback and running an algorithm against it to convert the raw audio into a compressed audio data format.

And when it comes to transcribing text from audio, it’s a similar digital analysis process but with different output. A far more complex process than just transcoding audio from one format to another, but still it’s another digital analysis process.

---

^{PS: To the seemingly endless stream of commenters who want to point out that pre-1995 PCs could not encode MP3s faster than real time… Yes, I know… That is why I qualify what I posted by saying “…on a modern computer system…” as well as stating “…but even 20-ish years ago…” as well.}

^{The first MP3 encoder came out on July 7, 1994 and the .mp3 extension was formally chosen on July 14, 1995. The point of this answer is to explain at a very high level that on modern PCs the act of analyzing audio quicker than real time playback already exists in a way we all use: The act of converting an audio CD to MP3 files.}

Answer 4

Computers don't experience audio the same way we do.

Recordings played at a higher speed become incomprehensible for humans because we're receiving more data than we can process. Our bodies and brains have limits and once the "data rate" is exceeded slightly we start picking up only parts of what is being said. Turn the speed up and it becomes gibberish.

A computer doesn't experience this phenomenon because its perception of time in the recording isn't based on actual time that has passed, but on the amount of data processed. A computer will never read data from disk faster than it can process it¹, so it's never overloaded. Data rate always matches the processing speed perfectly.

---

¹ Unless it's told to do so by a buggy program, but this applies to every single computer question.

Answer 5

In spite of the good answers here, I'm going to have to go for a solid

It depends

Some algorithms depend on brute-force processing power. The more processing power you've got, the more processing (or the more accurate processing) you can do. We're at a point now where most audio processing is no longer resource-limited. Video processing is still resource-limited though, as can be seen by the continuing state-of-the-art in gaming.

After that though, the issue you have with real-time processing is latency - in this case the delay between you saying something and the computer putting the text up. All processing algorithms have some delay, but anything based on Fourier transforms is especially limited by this. By a mathematical theorem, the lower the frequency you want to be able to recognize, the more data you need to spot it, and hence the longer the delay before the computer gives you a result. So you do hit a point where it doesn't matter how fast you can do the maths, you're always at least that far behind.

The challenge for real-time processing is to find a sweet spot where you can get reasonably effective processing and have the delay relatively imperceptible for users. This is always a trade-off between lower delays and higher quality results, and the optimal algorithm for this can be a matter of personal taste as anything else.

And in the extreme case, some algorithms simply cannot be run in real time. Some very effective filtering algorithms exist which require the data to be run backwards, for example. These can give very good results for post-processing recorded data, but of course are utterly impossible to run with real-time data.

Answer 6

Since you asked about converting speech to text....

My confusion is: When playing audio back at a faster rate, the words become unclear, so how does the computer compensate for that?

I'm not sure, but it sounds like you may be thinking that the process of speech recognition (or real-time audio processing in general) works like this:

1. Capture audio into bytes (done by microphone and analog-to-digital converter)
2. Convert bytes into "internal audio"
3. Listen to "internal audio"

But that's not what happens. Once the audio is converted into bytes, everything is handled by signal processing algorithms, and those signal processing algorithms can run as fast as the CPU can manage.

Here's an example of how a super-simple speech-to-text system might work. It will have two threads, one audio acquisition thread and one signal processing thread. The audio acquisition thread runs the following steps in a loop:

1. Capture audio into bytes (done by microphone and analog-to-digital converter)
2. Store those bytes in a buffer (a dedicated part of RAM)

This process produces bytes at a fixed rate, namely the sample rate multiplied by the number of bytes used to store each sample. For example, a speech-to-text system might measure the sound wave 8000 times per second and use two bytes to store each of those measurements, so it adds 16000 bytes (96000 bits) of audio data to the buffer per second.

While that's happening, the signal processing thread is doing the following in a loop:

1. Take a group of bytes from the buffer
2. Run a signal processing algorithm on those bytes
3. Guess which phoneme (speech sound), if any, that group of bytes represents

The size of each group of bytes, and the amount of time taken to process those bytes, vary depending on which algorithm is being used. For example, suppose the signal processing algorithm is designed to operate on half a second of audio at a time. It'll wait until it finds at least 8000 bytes (0.5 seconds of audio) in the buffer, then move those 8000 bytes from the buffer into another part of memory and run, say, a Fourier transform on them, or feed them into a neural network, or so on.

Whatever the signal processing thread does with those 8000 bytes, if it finishes in less than half a second, that's fine; it can just wait until there are 8000 bytes available in the buffer again and start over. Of course, if it takes more than half a second, that's a problem, because the audio processing thread adds data to the buffer faster than the signal processing thread can remove it. When the buffer gets full, it has no choice but to start throwing away audio data. So the designers of real-time audio processing algorithms go to great lengths to make sure their algorithm can process half a second (or whatever) of audio in less than half a second (or whatever) of real time, so that the buffer doesn't fill up.

In other words, in any system that processes audio in real time (including speech recognition and many others), the actual processing has to be capable of working quicker than real time playback so that it doesn't exhaust the system's resources.

---

Note: I work for a speech recognition company, although everything I've said above is pretty generic information about real-time audio processing and not specific to the product I work on.

Answer 7

Current time & speed have no meaning to processing recordings of past signals. The recording itself has timing info.

The performance limits on processing signals are CPU number-crunching and/or how fast the CPU can read that data from memory, disk, network, or wherever it's coming from. For audio, often that's much faster than 1 signal-second per wall-clock-second, but you certainly can run an inefficient and/or computationally-intensive algorithm on an ancient slow computer and only go at 1 signal-second per minute, and get the identical output file to running on a modern CPU at 10 signal-seconds per second.

If that source is an analog->digital converter that's converting a mic input in real-time, that's what sets the speed limit to 1 signal-second per real-world second¹, not the CPU itself.

The key concept is that the passage of real-world wall-clock time has zero meaning or relevance to how a computer processes digitally sampled audio data; the data comes with its own timing information.

It's often useful to be able to do something at least as fast as real-time, meaning you can keep up with the outside world instead of having to save it to disk and process offline, but other than that you're just doing math on the samples from time t=123.456s.

---

Analogy: looking at printouts of weather records

If you have some printouts of weather records from the past 7 days, how can you recognize recognize freeze and thaw cycles, and shifting wind patterns, if you turn the pages and read faster than 1 day of data per real-time day? The obvious answer is "what? Why would I have to read that slow, and what does that even have to do with analyzing recorded data?" That's exactly the case for a computer doing any kind of processing on recorded signals of any kind (video, audio, key-stroke records from a keyboard keylogger, etc.)

Computers don't "experience" recorded audio in any way other than looking at the digital samples. For us, seeing lists of numbers would be meaningless. Same for computers; they don't truly understand anything. (Just like you don't feel cold or hot from looking at a table of past temperatures.)

If you program a set of steps to follow, they will do so. It's up to programmers to find useful sets of steps to make computers run on audio data. If that set of steps happens to have a useful result then running it to get that result can be useful. Like finding the peak sample value over some interval, normalizing the volume, filtering out some frequency range, or something more complex like outputting a sequence of text characters (speech->text), or a compressed representation of the audio (e.g. mp3, AAC, Opus, or FLAC).

At no point do any of this "mean" anything to the computer. It's all just numbers that represent sound, but the CPU itself doesn't know or care what they represent. It's just doing addition, multiplication, compare+branch to run different code depending on the data, and stuff like that. (i.e. running machine code, with numbers in registers and memory).

Speech-recognition is just a special case of what you can do to analyze a signal. It's not fundamentally different from compressing the audio file into a .zip, .flac, .mp3, or .opus, as far as the CPU and OS are concerned. Obviously the algorithms you'd use are much different, but nothing in them depends on real-time, CPU frequency, or whatever, and will give the same result no matter how fast or slow the CPU is.

I assume you understand that the faster your computer, the faster it can ZIP a hundred megs of data, but you still get the exact same output file. This is totally normal, right? Speech->text or other audio or video processing is the same.

---

All the file formats contain timing information (or you supply that separately). In most files (like simple .wav) there isn't a separate explicit timestamp stored for each sample, but there is one piece of metadata for the whole file that tells you the sample rate was (for example) 48kHz, so the computer knows that every 48,000 samples make up one second of recorded time. That's equivalent, and lets the computer know what time each part of the audio recording corresponds to.

It makes zero difference how slow or fast the computer is. e.g. you could run an audio processing program on a computer from 20 to 40 years ago (assuming it could load enough of it into RAM to run your algorithm at all), and it might take hours or days to do something to 2 minutes of audio, vs. 10 seconds for a modern computer.

The number-crunching speed vs. real-time, processing time vs. signal time, is just the performance number. It's not fundamentally different from timing how long it takes to zip (or zstd) compress a file: you can report the results as seconds per input megabyte. With audio and video input files, you could measure the input size in samples, or uncompressed-bytes, and report the speed in MB of audio crunched per second. Or you can measure in time and report a ratio of how fast you processed vs. real-time. e.g. audio compressors often report the speed they achieved as a ratio of signal-time / processing-time. i.e. how much faster or slower they were than playback at 1x.

If you run a benchmark on a video game, often you'll see the frames fly by at crazy high speed. (Or very low). Depending on the game, it's simulating game-time much faster than normal, because you told it to benchmark so it doesn't limit its speed to 1 game-second per real-second. It's simulating more than 1 game-second of game-time per real-second. If a car in the game goes off a jump and is in the air for 90 frames (1.5 game seconds at a standard 60Hz), that would still be true regardless of how fast or slow the benchmark was running. (Real games usually don't have their simulation time based on a fixed frame-rate, not since old 2d days, but pretend you had a simple game engine that always had to run at 60Hz to do real-time.)

---

For more about digital sampling, see Monty Montgomery's digital sampling show-and-tell 25 min video lecture / demo, and part2. The motivation was to explain why 96kHz / 24-bit audio is not perceptibly better than standard 44.1kHz / 16-bit, especially if correctly dithered, but it's a great intro from scratch to digital sampling concepts. (Monty famously developed the Ogg/Vorbis open-source audio codec and contributed to Opus, the current best quality audio compressor.) He even uses some real physical oscilloscopes including an analogue one in the demo, showing signals turning into digital and back to analogue, to prove that digital sampling works.

Seeing that signals can turn into a sequence of numbers may help understand how computers can process them.

(Of course this is all done with real-time processing, so IDK if it will help you grok the concept that time has no meaning for signal processing unless you're specifically doing real-time processing where real-now = signal-now.)

---

Footnote 1: It will take less than 1 CPU-second to process that second of audio data, so the CPU can spend some time in low-power sleep while it waits for more data to build up in a buffer. Otherwise your CPU will sometimes get behind and drop parts of the audio input to catch up, or simply just get behind, depending on the design of the software that manages input and output in a real-time or offline style.

If you care about real-time usage, you want the CPU to be faster so that even in the worst case, it doesn't fall behind real-time. (And also, your algorithm has to be designed not to need "look-ahead" at future parts of the signal. In a real-time use case, you can't get access to those without buffering for as many seconds as you want to look ahead.) Other than that, there's no fundamental difference (in terms of how the CPU works) between running signal processing in real-time vs. offline on already-recorded files.

I know you didn't ask about real-time processing, but it's interesting to note that the major difference between real-time vs. offline signal processing is that real-time introduces the possibility of the CPU "getting too far behind" in processing, if you try to make it run slow code. This maybe helps make clear that unlike a human brain, it always takes as much CPU time as it needs per second of signal, not processing differently depending on speed.

(You could write a program that switches to a simpler faster algorithm when it's getting behind, but that's not what I mean.)

Answer 8

Computers can adjust the speed audio data comes at them

The difference between computers and brains is that computers can generally change the speed at which they handle audio. Brains are limited in that respect.

While brains can't do anything to slow down audio if it's played too fast, computers can read and process an audio file slower if they need to. They can also read and process an audio file quicker if they're able to.

Computers break up programs into very small, correctly performed subtasks

Computers deal with bits: ones and zeroes. They have built into them a set of very small subtasks like adding together two numbers that are made up of a certain number of bits.

As long as the subtasks like addition are performed correctly, you'll get the same correct results no matter how quickly they're performed.

Then we combine those subtasks into whole programs that do things like audio processing. So what happens to the speed of our programs when the speed of those subtasks increases? The speed of our programs will increase.

As is commonly known, computers are much, much faster at addition than brains are.

Subtask speedups are program speedups

Computers started out being very slow.

Since our audio processing programs are made up of correctly-performed subtasks and since computers can process audio data at whatever speed they can handle, computers will start out processing audio much more slowly than playback speed.

When the subtasks speed up with newer computers, the audio processing programs speed up. We now have computers that can do most audio processing tasks faster than playback speed.

Answer 9

The computer will be constrained by its own hardware, mainly CPU when processing the audio (perhaps GPU if for some reason that would be used). This could be real-time speed (1x), slower than that (if the computer couldn't process it so quick, it may require more time e.g. 1.5x to process it, for instance video rendering often takes a long time, more than it would playing each part), and also could do so quicker than that.

Now, the part that I feel is missing from the other answers is that the computer will probably need to know what is the normal speed. It all depends on how it determines words, but it probably takes into account the length of each phoneme, for which it would note how long it was spoken (e.g. a 'phoneme' that only lasted 1ms was probably background noise). If you had an audio file, it will simply look at the timing information it has, and process the file as soon as it can finish it. If you are providing as input an audio much quicker than real time, it will probably need to be told that it is being played at a certain rate.

Answer 10

One thing to consider as well is how a human processes audio versus a computer. For a human, audio is a sequence of vibrations which can only be processed in real-time by the brain through reading these vibrations in the ear. Thus, the brain learns to and is designed in some ways to handle audio at the speed at which it is produced.

As for a computer, we digitize audio into a format which the computer can read - a sequence of amplitudes as a list. Unlike the vibrations we recorded this data from, a simple list can be read at whatever speed the computer can retrieve that data from memory / the hard drive and process it in the CPU. Of course speed depends on the operations used and the hardware used. If we store the audio on a floppy disk and use a microcontroller to process the audio it may be even slower than the human brain at decoding those signals. As for modern hardware, it is likely that we can process this data (for simple operations) at a speed many hundreds of times faster than a computer.

It also depends on the operation performed. Brains are brilliant at taking vibrations and determining the corresponding frequencies in that signal, for our brains to process and convert to music or sounds. It is also very good at recognising words within an audio signal. For computers, its strengths come from any operations which can be vectorised and have a solution in P - volume calculations, Fourier transforms, etc. are all very quick for computers to calculate. Word recognition is still a very hard problem, and so takes longer - often computers send the data to a server (such as Google's TTS) to process and return its answer back. In this case processing time is limited by internet bandwidth, and it is likely that humans will be quicker, even if the voice signal is already known.