TEST METHODOLOGY
The network tests in the Netherlands were conducted from 28.01. until 09.02.2016. As in Germany, Austria, Switzerland, the UK and Spain, P3, connect’s partner for network benchmarks, sent out measurement cars to perform the tests. For the benchmark of the Dutch mobile networks, four cars covered a total distance of 7,600 kilometres through big and small Dutch cities and the connecting routes between them. The route is shown on the first page of this report. In total, the test cars drove through cities that accommodate about 4.3 million residents. Each car was equipped with eight Samsung Galaxy S5 and four Samsung Note 4 smartphones. The eight Galaxy S5 Cat 4 devices were used for the voice tests, the four Note 4 belonged in the Cat 6 category and were used for data measurements. Whenever possible, the smartphone’s firmware matched the network operators’ genuine ones. Where no operator-specific firmware was avail able the most recent Samsung firmware was used.
Smartphone telephony and data
Data and voice services were executed with separate smartphones. Telephony was tested mobile-to-mobile from one car to the other. In order to ensure realistic testing conditions, background data traffic was generated on the smartphones during a voice call. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. All network operators offer 4G capable subscriptions for anyone interested in a high performance data network. To take the high share of LTE into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones needed to switch (“fall back”) to 2G or 3G when they were connected to the LTE service before the call could be completed using circuit switched fall back (CSFB). Data services were tested with four separate Samsung smartphones in the cars. As a first measurement task, several popular live web pages were browsed using the built-in smartphone browser. The web pages had been selected previously according to the Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for such testing purposes was used. In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the internet.
In addition, the peak data performance was tested in uplink and downlink direction by assessing the amount of data that could be transferred within a 10 second time period. While the 10 seconds peak throughput test focuses on network capability, the YouTube tests add a more end user centric view to the test curriculum. Here, videos with standard definition (SD, 360p, 2.7 MB, 30 seconds) and high definition (HD, 720p, 11.9 MB, 30 seconds) were played using the YouTube player on the smartphones.
Route and Samples
In big and small cities the cars followed pre-defined routes. Altogether more than 50,000 speech samples were logged per operator. In order to collect them, a call of a typical duration of 70 seconds was established during which eight speech samples were exchanged alternately (four each from mobile A to mobile B and vice versa). About half of them were collected in 4G preferred to 4G preferred mode, while the other half in 4G preferred to 3G preferred. For the data benchmark about 38,000 sessions per operator were established. About 55 percent of the samples were obtained in big cities while the remaining 45 percent were collected in small cities and on connecting routes.
Benchmarking at the edge of the technically possible
Previously, the usual approach was to conduct mobile network drive tests with the measurement devices mounted in the car and connected to antennas in a roof box mounted on the vehicle. In accordance with ETSI TR 102 581 an attenuation of the roof antenna signals of -12 decibel was used to compensate for the antenna gain (+5 dB) and to simulate a typical attenuation (-7dB) of a mobile phone used indoor. With modern smartphones having their orthogonal MiMo antenna arrangements used for HSPA and LTE and with the use of more intelligent antennas changing their directivity and tuning in relation to the signals received, a new approach is better suited to reflect real life conditions. This new approach is a highly optimized mounting case for the smartphones – the so-called P3 Antenuatr. It is designed with unique materials selected for their specific absorption rate at high frequencies between 800 MHz and 2.6 GHz. By choosing the right materials in the right places, an attenuation rate typical for indoor measurements can be achieved without altering the antennas of the smartphones. The validity of this approach, for which a patent is pending, has been confirmed by comparing measurements between unaltered phones. The legacy approach with antennas in a roof box and attenuators has been further verified by comparing TIS (total isotropic sensitivity) and TRP (total radiated power) according to the CTIA test plan for over the air measurements in the Testlab owned by WEKA Media Publishing, the parent company of connect. These measurements confirmed an attenuation rate very close to the wanted -7 dB from LTE band 20 (800 MHz) to LTE Band 7 (2.6 GHz). The variation is even smaller than with external antennas connected over wideband attenuators. Three of these antenna boxes were mounted in each of the cars in the rear side and in the rear windows. One Antenuatr was responsible for 4G preferred to 3G preferred voice measurements, while the second was used for 4G preferred to 4G preferred voice measurements. The last Antenuatr was responsible for the data measurements. Real live testing confirmed the advantage of using the differently polarized MiMo antennas, as this increases the likeliness of achieving maximum data rates due to the higher linear independence of the antenna signals.
Conclusion
Hannes Rügheimer, connect author
In the 2016 installment of the P3 connect Mobile Benchmark for the Netherlands, the overall winner is T-Mobile. This operator managed to achieve a clear lead in the voice category and a slight advantage over its pursuer KPN in the data scoring.
In the voice discipline both networks showed close to perfect stability, with more than 99 percent success ratios in all scenarios. T-Mobile wins thanks to faster call setup and a slightly better speech quality. Vodafone and Tele 2 score almost as well in this category, despite different weaknesses: As Tele2 lacks HD voice, it achieves a lower qua lity result, while Vodafone has slight disadvantages regarding call setup duration and stability.
Regarding data, T-Mobile leads closely, with KPN following close by, and Vodafone and Tele2 (which is still working on its 4G rollout) still doing very well. It must however be emphasized how close the total ratings of all four candidates are. This is a clear indicator of the superior quality of the Dutch networks in comparison to those in many other countries. We applied exactly the same scoring standards here as in our other network benchmarks. And while for example in the 2015 Germany benchmark the resulting grades varied from “Very good“ to “Sufficient“, all four candidates in the Netherlands were able to obtain the treasured top grade.
Reactions
We asked the four Dutch mobile operators to comment on the results of the P3 connect mobile benchmark for the Netherlands. Here are their responses.
“We at T-Mobile Netherlands are very proud and happy with the results of the P3 Mobile Benchmark in the Netherlands. This remarkable outcome is the proof of our network strategy which has started back in 2012 with the spectrum auction and has continued with the successful deployment of a completely new SRAN network. T-Mobile Netherlands has been ranked the best tested network in all categories of service (voice, data) in a very competitive landscape, where all operators have developed high quality networks. We will continue focusing on best customer experience based on our network leadership.”
Lampros Iskos
Director Technology T-Mobile Netherlands
“The KPN mobile network covers more than 99% of the Dutch population and Dutch territory and therefore has the best degree of coverage in the Netherlands. This P3 test – carried out mainly in the big cities of the Netherlands – also shows that the KPN network is one of the best in the world. We are very proud on this position. But the KPN network is for the benefit of the whole of the Netherlands, not just for the cities. And that is not immediately visible in this test. Furthermore, every test is a snapshot; satisfied customers are the most important indicator for us. Overall our customers are very satisfied, as can be seen from recent independent customer satisfaction surveys and from our Net Promoter Score (NPS), which is higher than that of other operators. Above all, we aspire to provide our millions of customers with an exceptional user experience so that every day, thanks to KPN, they are free to do what they enjoy, anytime, anywhere.”
Jacob Groote
Executive Vice President Mobile Networks KPN
“Great to see the Dutch mobile networks belonging to the top echelon worldwide with only small differences between the networks. Vodafone continues to drive for excellence in all of its services. Room for improvement can be found indoors, especially with the enhanced focus on sustainable buildings which makes penetration for mobile signals a lot harder. In response to this trend, we introduce not only 4G calling this year (Voice over LTE or VoLTE), but will also be the first telecom provider in the Netherlands to launch WiFi calling (Voice over WiFi or VoWiFi). By combining a 4G signal (with better penetration capabilities for homes) and a WiFi signal (which originates indoors), our customers will experience much better quality of their indoor mobile calls, even if they live in a house where mobile signals are disrupted by, for example, dense construction materials.”
Eben Albertyn
Chief Technology Officer Vodafone Netherlands
“We are extremely proud that we have been able to close the network quality gap on our competitors so quickly, offering excellent speed and coverage, so our customers can enjoy their 4G mobile data carefree. However, we are not done yet, and we continue to build out our network at full pace, expecting to reach nationwide population coverage in just a couple of months. These fantastic results are a real credit to the team, and a true testament to their extraordinary efforts. We now can’t wait for the next test …”
Francois Mairey
Chief Technology and Information Officer Tele2 Netherlands