The P3 connect Mobile Benchmark in Spain took place from 7th to 28th October 2016. All samples were collected between 8am and 10pm. The network tests covered larger cities, smaller towns and connecting roads. The combination of test areas had been selected to provide a significant series of test results covering the Spanish population. The areas chosen for the 2016 test account for more than 11 million people, or 23.8 per cent of the total population of Spain.

P3 conducted the tests with four drive-test cars, equipped with arrays of Samsung Galaxy S5 Cat 4 smart- phones (Voice) and Samsung Galaxy S7 Cat 9 smartphones (Data) for the simultaneous measurement of voice and data services. 

Voice testing

Two smartphones per operator in each car were used for the voice tests, setting up test calls from one car to another. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POL- QA wideband algorithm. All Spanish network operators offer 4G capable subscriptions. To take the high share of LTE into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones in most cases needed to switch (“fall back”) to 2G or 3G when they were logged in to the LTE (so called “circuit-switched fall back“ or CSFB).

In order to account for typical smartphone use scenarios during the voice tests, background data traffic was generated in a controlled way through random injections of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results. 

Data testing

Data performance was measured using one smartphone per operator in each car. The radio access techno- logy was set to LTE preferred mode in order to reflect the customer experience. The web tests accessed web pages according to the widely recognized Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for testing purposes was used.

In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that could be transferred within a 10 seconds time period. Another discipline was the playback of Youtube videos. It took into account that Youtube dynamically adapts the video resolution to the available band- width. So, in addition to the success ratios, start times and playouts without interruptions, the Youtube measurements also determined the average video resolution.

All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 percent of the total results. 

Routes and Samples

The test routes are shown on page 1 of this report. In the 14 big cities and 26 smaller towns indicated, the cars had to follow predefined routes. Altogether, the four test cars covered more than 11,300 kilometres, of which approximately 4800 km led through the big cities, while 6500 km were covered in smaller towns and on connecting roads. The test routes also included the islands Tenerife and Gran Canaria. 

Performance indicators and rating

The score weighting reflects both the geographical distribution of Spain‘s population and the ranking of usage scenarios. Therefore, 600 of the total of 1000 maximum points were assigned to the cities – 240 maximum points refer to the voice results and 360 maximum points reflect the data results. For the towns and the roads, a maximum of 200 points each is available. In both categories, the possible maximum is 80 points in the voice, and 120 points in the data category. The table on page 2 shows the percentage of maximum points that each operator has achieved in each discipline. 

To those readers who are familiar with the outcome of 2015‘s P3 connect Mobile Benchmark in Spain, this year‘s results do not present many surprises: 2016‘s overall ranking looks exactly like the one 12 months ago. Vodafone is the clear winner, both in the voice and data categories. And, like all candidates, the winner was able to improve over last year‘s points. The biggest improvement could be observed in the data category, however Vodafone‘s score also grew in the voice category.

Movistar also manages to keep last year‘s position and ranks as a strong second candidate. Like the winner Vodafone, Movistar improves both in the voice and data categories.

Its constant competitor Orange finishes only 13 points behind and even drew level with Movistar in the voice category. The third rank results from an overall weaker performance in the data category. Spain‘s smallest operator, Yoigo, brings up the rear. However, it improved visibly in the voice discipline but basically remained static in the data category. In the big cities, Yoigo achieved fair results. In smaller towns and on connecting roads though, it showed much room for improvement.

In an international comparison, the Spanish networks still score below the best operators in Switzerland, Austria and Germany. However, Vodafone España does better than its cousins in some of the other European countries. At the other end of the range, Yoigo is on a similar level as Telefónica‘s German O2 network. 

With excellent data performance, EE is clearly ahead of the competition in this category. So last year’s winner is still one of now two top operators in the UK. If EE had not lost some ground in the voice discipline, it might still be the uncontested number one. Regardless, EE customers enjoy one of the UK’s best mobile networks. 

Compared to 2015, Vodafone hit the biggest score improvement – which reflects the advances of its mobile network. Vodafone ranks best in the voice discipline and stays narrowly ahead of EE’s overall results in the cities and towns. In spite of some room for improvement on roads, Vodafone is one of the UK’s best operators.    

Two test category top-spots helped put Three in third position overall this year – which is also their respective rank in the voice and data categories. In the voice discipline, Three scores best on connecting roads and also achieves good results in smaller towns. Adding its nice data performance, Three deserves the overall grade „good“. 

In comparison to 2015’s bench- mark, O2 also achieves a considerable improvement in its score and KPIs. Scoring best in the voice tests conducted in London and ranking second in the voice discipline overall, the operator might consider putting additional efforts into strengthening and enlarging its data service. 

The results of 2016‘s mobile network benchmark in Spain ended up with the same ranking as one year before. This, however is not carved in stone. Upcoming improvements like the further extension of 4G coverage, the ongoing network evolution with technologies like “4G Advanced“ and the introduction of Voice over LTE (VoLTE) may very well change the picture.

P3 communications is well prepared to accompany this development on the Spanish market. In this context, we are already preparing to include these new technologies and enhancements in next year‘s testing. Furthermore, P3 is also preparing for future challenges starting with new communications applications like autonomous driving and smart cities (see below), ranging all the way up to completely new technologies like the upcoming super-fast 5G networks. However, with 5G‘s approach of assigning different network capabilities to different use cases, an extended approach to network testing will also be required. This will call for concepts like advanced analytics and big-data technology. 

Extensions to the test routines

Readers interested in taking part in our continuous efforts to evaluate the performance of mobile networks can do so by downloading the “U get“ wireless performance rating app – see details on the right.

This crowdsourcing approach will give us valuable additional insights into the user experience and operational performance of the Spanish mobile networks in the near future.

Autonomous Driving

Smart Cities