METHODOLOGY

The tests in Germany took place from October 22nd to ­November 2nd, 2020, the tests in Austria from October 12th to 19th, and the tests in Switzerland from October 21st to 31st, 2020. For each country, connect‘s partner for the network measurements, umlaut, used four vehicles for drive testing on the chosen routes.

Each car carried a total of six smartphones. One Samsung Galaxy S10 per operator was used for the voice tests, another smartphone per operator took the data measurements: In two of the cars this were Samsung Galaxy S10, set to “4G preferred“, in the two other cars Galaxy S20+ with the setting “5G preferred“. In Switzerland, in accordance with the operators instead of the S20+ the Samsung S10 5G was used – again with “5G preferred“.

In addition to the drive tests in each country a walk test team took measurements by foot – ­visiting so-called “areas of in­terest“ with a strong visitor frequency like train stations, airport terminals, coffee shops, mu­seums and also local public transport. Part of the schedule of the walk tests were also rides on long distance trains. The walk test teams also utilised Galaxy S10 smartphones for the voice tests and Galaxy S20+ for the data tests (in Switzerland again: S10 5G) set to “5G preferred“ mode. The smartphones were installed on trolleys and backpacks with ­additional strong batteries.

The devices used each operator’s current firmware version. If such software was not available, the according regional “open market“ firmware was used.

Logistics

All drive tests and walk tests were done between 8 am and 10 pm. During the drive tests, two cars were present in the same cities, but on different routes to avoid any interference of one car’s measurement by the other car’s.

In Germany, the measurements included 21 larger cities and 25 smaller towns, while the walk tests frequented ten cities. The measurement cars drove a total of approx. 10,379 kilometers. With their drive tests, they covered a population of about 14.6 million which equals about 17.7 per cent of the inhabitants of Germany.

In Austria, the drive tests covered 9 big cities and 17 smaller towns, the walk test team visited seven cities. Here, the vehicles covered about 5,280 km equalling 3.1 million inhabitants or 35.1 per cent of the population.

In Switzerland, the drive tests covered 17 big cities and 36 smaller towns, the walk tests took place in eight cities. The test route in Switzerland was about 6,720 km long, equalling about 2.3 million inhabitants or approximately 27.5 per cent of the population. For the definition of the test routes umlaut generates four independent plans, from which connect then randomly chooses one.

Voice telephony 

Voice services were measured with the smartphones perfor- ming calls alternating between the measurement cars (“mobile to mobile“). The walk test teams called a stationary (smartphone) counterpart for all voice tests.

Background data traffic was transmitted by one of the smartphones simultaneously in order to reflect a realistic usage scenario.

As part of the tests we also evaluate the so-called MultiRAB (Multi Radio Access Bearer) Connectivity. This value denominates whether data connectivity is available during the phone calls. Audio quality was assessed by using the HD-Voice capable POLQA wide band scoring. All devices were configured in “VoLTE preferred” mode.

Data connectivity

To assess cellular data performance, top websites (according to the Alexa ranking) were dynamically downloaded. Additionally a static website was tested, the industry standard ETSI (European Telecommunications Standards Institute) Kepler reference page.

HTTP downloads and uploads were conducted with 5 MB and 2.5 MB files, simulating small file transfers. The networks’ peak performance 
was tested with a seven second download and upload of a single, very large file.

Youtube measurements consider the “adaptive resolution“ feature of this video platform und thus assess the success ratio, the time until the playback starts as well as the videos‘ average resolution.

Crowdsourcing

The results of the crowdsourcing analysis performed by umlaut contribute 20 per cent to the total score. For this purpose, in all three countries samples collected from mid-May until end of October 2020 were evaluated. 

For Germany, a total of approx. 2 billion single measurement values from about 401,300 users was analysed. This represents about 99.6 per cent of the country‘s built-up area.

For Austria, umlaut considered approx. 707 million values from a total of about 58,200 users, representing a 100 per cent of the built-up area in cities and 92.1 per cent of the built-up area outside of them.

The figures for Switzerland: Approx. 33,400 users contributed 177 million samples. This covers about 97 per cent of the built-up area in Switzerland.

The data base for these analysis is obtained by more than 1000 popular apps. They log in the background whether there is a network connection, which ­mobile network technologies are available and what download data rate and latency can be achieved – provided that the user has agreed to this completely anonymous data collection beforehand. These values are gathered every 15 minutes and transmitted once a day to umlaut‘s servers. The reports only comprise a small number of bytes so that they do not put a substantial strain to the users‘ data volumes.

Quality of Broadband Service

For the assessment of the Coverage Excellence, umlaut ­applies 
a grid of 2 x 2 km tiles (so-called evaluation areas or EAs) over the test area. For each tile, a minimum number of users and measurement values must be available. For the evaluation, umlaut awards one point if the considered network provides 4G or 5G coverage in an EA. Another point is awarded to a candidate for each competitor who provides a smaller or no share of broadband usage. In a country with three contenders, a candidate can thus reach up to three points per tile: one for providing broadband coverage and up to two addit­ional ones for “beaten“ compe­titors. The assessment then ­relates the obtained points to the total possible points for Coverage Excellence.

In addition, we con­sider the Time on Broadband. It reveals how often a single user had 4G or 5G reception in the observation period – independent from the EAs in which the samples were obtained. In order to calculate this, umlaut puts the number of samples with 4G/5G coverage into relation to the total number of all samples. Coverage Excellence and Time on Broadband results each provide 50 per cent of the points for the Quality of Broadband Service. Important: The percentages determined for both parameters ­reflect the respective degrees of fulfillment. They do not correspond to the percentage of 4G/5G coverage of an area or population.

Data rates and Latencies

Additionally, umlaut investigates the Data rates and Latencies 
that were actually available to each user. The examination of these parameters is independent from the EAs and thus concentrates on the experience of each single user. Samples which were for ­instance obtained via WiFi or with the smartphone‘s flight mode being active, are filtered from the data pool before further analysis.

In order to take the fact into ­account that many mobile phone tariffs limit data ­rates, umlaut has defined speed classes which are corresponding to particular ­­applications: For Basic Internet, a minimum of 2 Mbps must be met. HD Video requires 5 Mbps. And for UHD Video the minimum is 20 Mbps.

In order for a sample to count as valid, a minimum amount of data must have been transmitted within a 15 minute period. The same principle also applies to the assignment of a data packet‘s latency to the according application-based classes: Roundtrip times up to 100 ms are sufficient for OTT Voice, 50ms and faster qualify a sample for Gaming.

In the assessment, umlaut assigns the data rate and latency observed in a sample to one of these performance classes. Then, Basic Internet accounts for 60 per cent of the Data Rate score, HD Video for 30 per cent and UHD Video for 10 per cent The Latency score incorporates OTT Voice with a share of 80 per cent, Gaming with a share of 20 per cent.

conclusion

There were no surprises in the rankings this year again – the rank order of the operators 
in all three tested countries remains stable for the third year in a row. This does not mean, however, that not every single network operator is making ­extreme efforts to improve its ­position. This leads to the ­pleasant result that we can ­conform very clear improvements across the entire test field in Germany and that Telefónica now also joins the ranks of the providers rated “very good“. 

In the neighbouring countries, which were already at a very high level before, this time there are no serious increases to be seen. But there have been pleasing improvements in details – such as the improvement in mobile phone coverage in trains on the whole and particularly pronounced among the third-placed providers in each country.

In Germany, Deutsche ­Telekom is the overall winner for the tenth time in a row. The Bonn-based company was able to achieve a slight year-on-year increase in both voice and data discipline, and even a significant improvement in the crowdsourcing category. In addition, our individual analyses show that Deutsche Telekom is also clearly ahead in terms of 5G expansion. Vodafone in second place was also again able to improve in all three test disciplines compared to the previous year. The biggest surprise, however, came from Telefónica/O2. After the Munich-based provider had already ­improved significantly in the ­previous year, this year it is finally able to catch up with the overall field. If this trend continues, we can expect exciting rank fights in the future.

In Austria, Magenta continues the now also established ranking order and achieves the overall victory for the third time in a row as well as the top mark “outstanding“ for the second time. A clear lead regarding the 5G roll-out in the Alpine Republic has very probably also a share in this. 

A1 Telekom achieves a very good second place – in the overall ranking as well as in the 5G assessment. The Hutchison brand Three, once again receives the ­overall mark “very good“, although with a few points less than in the previous year. Con­sidering the fact that it is the smallest provider in Austria, Three can be particularly proud of the results of our 5G analyses.

In the hotly contested Switzerland, Swisscom also manages to outperform its strong rival Sunrise for the third time in a row. While the two providers are on a par in the data ranking, Swisscom can win the tight race in the disciplines of Voice and Crowdsourcing. With the best will, we cannot find a winner ­between Swisscom and Sunrise in terms of the 5G roll-out. Salt was able to improve slightly in comparison to the previous year and also shows great ­potential in the 5G area.

However, the biggest winners in all countries are the customers – as they benefit from very good to outstanding mobile networks.