5

I want to know how I can compare the quality of different voice products, in order that it gives the best audio quality rating for calls through the internet in average network conditions.

How exactly can I benchmark this? Are there benchmarks available on the internet?

7
  • Can anyone provide a real benchmark?
    – Jader Dias
    Jul 28, 2011 at 20:13
  • I don't see how this question isn't subjective. . .
    – surfasb
    Jul 28, 2011 at 23:53
  • @surfasb believe me. There is a way of measuring call quality.
    – Jader Dias
    Jul 29, 2011 at 11:57
  • Oh I believe it. It still is subjective because there is far far more factors that lie in call quality besides protocol. Deciding call quality based on protocol is like comparing monitors at Best buys, taking no account into signal quality, lighting, calibration, etc etc.
    – surfasb
    Aug 2, 2011 at 5:49
  • Even if you believe that you asked an objective question, no-one gave a benchmark or any kind. I think that makes it subjective in the end.
    – soandos
    Aug 2, 2011 at 17:04

6 Answers 6

2
+50

The article Best Free Video Conference Software Review grades video and voice conferencing software for audio and video. It gives the following grades for audio :

Skype : 7 (The audio is very loud. The audio quality is also very good with very little to no feedback.)
GoogleTalk : 8 (The audio is very good with little feedback)
Live Messenger : 4 (The audio stutters sometimes and has clipped speech. It also echos at times. )

1
  • Excelent! It is kinda subjective, but at least they put numbers.
    – Jader Dias
    Jul 29, 2011 at 12:00
6

The quality of each program will depend on the connection of each person involved in the conversation, as well as the available hardware - not limited to, but including:

  • Microphone (a poor quality microphone will sound even worse after being compressed and decompressed)
  • Poor speakers
  • Lower end sound cards / chips can reduce audio quality.

If these factors are not a consideration (IE, each person involved has good quality hardware, stable connection, etc) then your best bet would be Skype.

6
  • can anyone find a source for this information?
    – Jader Dias
    Sep 23, 2010 at 0:13
  • 4
    No source, but Skype is simply years ahead in terms of software development concerning the protocol stability and audio compression. They've proven over the past, oh, TEN years that they've got their act together. Oct 5, 2010 at 13:51
  • 1
    When I was testing voice communications recently, I came across a post indicating Skype uses twice the bandwidth for the same quality sound (older codecs?). GTalk with video to wireless on a train was very good quality.
    – BillThor
    Oct 6, 2010 at 16:00
  • 2
    Buddy of mine has parents who spend time in Russia, Egypt and Dubai and he has tried em all but always comes back to Skype.
    – Chris
    Oct 6, 2010 at 16:09
  • About the quality of the microphone. If you can, use a USB microphone or webcam with audio integrated. I have a cheap Logitech webcam and the sound is amazing. My old microphones had too noise. Jul 29, 2011 at 1:46
4

Skype requires a lot of ports open to work. Protocols are all closed. It violates my mostly closed firewall setup severely.

GTalk uses very few ports, and uses open protocols. It works well with my mostly closed firewall setup. It will interwork with your own servers, so you can setup your own border server if needed.

I haven't used Live Messager as I mostly use Linux.

I would expect no significant call quality differences between any of the three services. My choice would be based more on security considerations such as firewall requirements.

I tested GTalk to my wife when she was on a train using wireless. Quality was fine for both voice and video.

6
  • 1
    VoIP over wireless sucks, because most VoIP protocols use UDP, UDP has no retransmission and wireless has a high drop rate.
    – Jader Dias
    Oct 6, 2010 at 16:49
  • 2
    GTalk is TCP from what I've seen on my firewall.
    – BillThor
    Oct 7, 2010 at 2:26
  • all protocols that I know use both TCP (for command messages) and UDP (for audio packets). Do you mean GTalk is TCP-only?
    – Jader Dias
    Jul 28, 2011 at 20:12
  • @Jader - I'm using VOIP over wireless with video and no problem here. You must have bad equipment or interference.
    – hookenz
    Jul 28, 2011 at 22:46
  • Yes, as far as I can tell GTalk runs entirely on TCP using the Jabber protocol. You may run into problems if your ISP buffers too much data on their networks. Recent research is showing that using cheap memory to buffed data can severely degrade service. Try shaping you traffic below 100% of your bandwidth.
    – BillThor
    Jul 28, 2011 at 22:53
2

This is a fairly subjective question - I can only provide subjective responses :-) but I've used all three (Skype, GTalk, WLM) a fair amount.

Skype's SILK codec is vastly improved over previous iterations but for me - even with over 2 Mbps of upstream bandwidth - it still takes a while to scale up to use all the available bandwidth, then it seems to want to lower in quality at the drop of a hat (or lost packet).

GTalk's quality could in theory be excellent; their plugin supports:

  • PCMA
  • PCMU
  • G.722
  • GSM
  • iLBC
  • Speex

but in real life I've only ever experienced the pleasure of a 16 Kbps Speex call. :-( Windows Live Messenger is pretty much the same quality wise - Skype edges it. However, Google Plus' Hangout (which uses the latest Vidyo codecs) is (at the moment at least) really good quality and pretty much on a par with Skype for pure audio quality. Looking forward to testing more as times goes on.

Skype is also quite tolerant (so it seems, from a weekend of testing in a hotel with overpriced wifi) of 3G connections, at least with a VPN, HSDPA and an acceptable level of contention on the cell tower. to get a call for more than 3 seconds without almost 100% dropped packets I had to VPN through the connection (as expected, as I'm on the middle tier for my mobile broadband package it deprioritises Skype traffic). I had very acceptable latency of about 200-250ms; much better than I thought. Throughput was fairly steady too.

So at the moment, Skype pips it for call quality - but I very much expect Google/Vidyo to surpass that quality, particularly given the SQ and PQ of calls I've seen over standard broadband using the standard Vidyo-branded service.

2
  • In the end it is a matter of choosing the best codec. I haven't seen a codec benchmark though.
    – Jader Dias
    Jul 29, 2011 at 11:59
  • With Skype now supporting Opus, and with Opus set to become the next mainstream audio conferencing codec, we're due for a vast improvement in audio quality in the coming months and years. Preliminary test / example Opus files and streams at remarkably low bitrates already yield incredible sound quality, easily on a par with AAC+ but with additional advantages relevant to low latency duplex communication. Feb 18, 2013 at 17:39
1

This would be difficult to benchmark without sophisticated audio equipment. Most consumer level equipment, from the speakers to the headphones to the microphones, are very low quality, providing only enough data to basically transmit recognizable human speach.

You would need an accurate sound sample conveying a wide range of possible sounds in different frequencies and with different types of stops (consonant sounds), and you'd need a system that has a very good sound card electronically isolated from the rest of the system to prevent sample contamination. You would need very high quality, reference tuned inputs and outputs (microphones and speakers) and then a recording device that can compare the resulting sound output to the reference input and highlight any differences.

This becomes very expensive a long time before it becomes practical.

The cheap way to do this is to use the same computer and the same people to rehearse a set selection of dialog and rate it subjectively on clarity and comprehension.

Or, you can just go online and read what others who have more money and more time have done.

0

I think the best strategy to do this would be able using the same technique used to test telephony links. Here you have both conversation sides, called A (the caller) and B (the called one). The technique consists on inject on A side a wide and linear range of acceptable frequencies (for example from 3k to 15kHz) and analyze how this range of frequencies are heard on B side (on telephony and data links this are made by network analyzers). The best link would give exactly the same wide and linear levels of transmitted signal on B side (this is theorical, on phisical links you will never get this enviroment). I dont know if there is already a software solution to test it, but it wolud be great to have some solution. This solution would be a software that generates a specific (and fixed) bandwidth signals, with all components on the same level on the transmiter side; and a frequency component reader on the reception side. The aplication compares the diferences between transmited and received signals and give you a "benchmark note" to the link (and consequently to the voip solution software used). My recomendation is (if you dont find any specific software that does that) is use Matlab to generate a linear audio source, transmit it over the voip link channel, receive and analyze it on the frequency domain. This would be a start to a benchmark software.

1
  • PS: Im assuming that you want to test the voice software, codec and the transport protocol versus the audio quality (in other words, you want to test the "voice channel"). In order to test the final quality of a voice transmission, another variables hould be considerated, such as microphone and speakers quality.
    – Diogo
    Aug 4, 2011 at 17:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .