Evaluation of Quality of Service in 4th Generation (4G) Long Term Evolution (LTE) Cellular Data Networks

The area of study proposed in this paper is the evaluation of the Quality of Service (QoS) provided by a Long Term Evolution (LTE) cellular data network. It is aimed to address the gap between the technical capabilities and the QoS experienced by the user. The analysis is based on the live data that are collected on commercial cellular network and compared against broadband wireless LAN. A methodology for QoS assessment based on a relatively small set of Key Performance Indicators (KPIs) is devised. The measured data are used to verify the proposed methodology.


Introduction
Data services are changing our life in a profound way. Cellular providers make Internet connectivity available anywhere and anytime. This allows for instantaneous access to social networks, employment Intranet, academic environments, shopping, Internet browsing, entertainment etc. From the user perspective, it is important that regardless of the access platform, there is a guarantee of the QoS with respect to the experience. Cellular companies strive to improve service and provide better experience to their users. Research and development in various areas of cellular technologies has allowed for growth, and advanced development of cellular broadband services. . Cellular telecommunication services became a valid alternative of traditional broadband landline connection service. Currently deployed advanced cellular standard is 4G Long Term Evolution (LTE) which allows cellular companies to provide even more advanced services in an efficient manner. With the development of LTE, the speed of the data transmission has increased with respect to the mobile and fixed broadband. The LTE offers support for more services such as voice, data, video and multimedia. It is based on OFDM/OFDMA (Orthogonal Frequency Division Multiplexing / Orthogonal Frequency Division Multiple Access) which is well suited to achieve high peak data rates in high spectrum bandwidth and multipath fading channel. LTE has the capability to use packet data at higher bit rates. The usage of advanced access, and transmission techniques for both the transmission bandwidth and QoS of cellular networks have been improved. Table 1 shows LTE network parameters.
The networks of 4G LTE hold a promise of performance that is comparable or even better than the broadband services provided by landline Wi-Fi access. However, even though the technical limits are high, due to the complexities associated with the cellular data networks, it is a frequent case that a user never sees the top performance of the underlying technology. Key Performance Indicators (KPIs) help define the performance metrics of a network. They allow cellular operators to maintain their networks so that the users remain satisfied. KPIs are calculated from measurements of various network parameters. KPIs represent an equations based upon simple counters that are used to give a more meaningful measurement of performance. For example, a simple counter, such as the numbers of times the users connect to the network, has a little meaning. However, the ratio of successful connections and the number of connection attempts represents the success rate -which is an important indicator of the performance quality. It is a much better indication of the performance of the network. QoS is a measure of the network quality as it relates to the user experience. Some examples of the QoS parameters would be achieved with the success rate, average throughput, and throughput jitter. One may attempt to characterize QoS of a network is by using these three significant parameters. The area of study proposed in this paper is the evaluation of QoS in a cellular data network. At present, this is a problem that exists despite the fact there are large volumes of measured performance data collected on various nodes of cellular networks. There is still no unified approach that is endorsed by the community on how these data are to be analyzed, processed and presented.

QoS Methodology
The proposed method for QoS evaluation in a cellular network consists of three steps. In the first step, a set of controlled measurements is preformed. The second step the measurements are used for KPI calculations. In the last step the KPIs are compared with the KPI obtained from a broadband network. To illustrate the methodology, the measurements are collected on three different networks: Wi-Fi available in a common home environment (802.11g), Wi-Fi provided in a common office environment (802.11n), and commercial LTE cellular network. The data are checked for integrity, filtered, and processed. In all three networks, the following data services are evaluated: HTTP Web browsing, FTP download, FTP upload, video streaming, and connectivity delay (test ping).
The method that is explored provides KPIs of the QoS parameters within the network. The essential issue is that KPIs, such as the power levels, do not correlate directly with the user experience. The proposed method allows network operators to see how the data they gather from testing the network relates to the subjective users experience. Providers will use the data interpretation to adjust certain network parameters to better meet the users expected QoS.

Equipment setup
All necessary QoS measurements are collected using specialized software operating on commercial grade drive test equipment. The equpment setup, shown in Fig. 1 Analysis software: Gladiator G-Station software is a tool for efficient process-management, consolidation, integration, analysis, and correlation of data collected.

. The Data Collection
The collection software is set to measure the following data services: HTTP Web browsing, FTP download, FTP upload, video streaming, and ping. The test and relevant measurements are shown in Table 2. All tests are run in cycles, as illustrated in Fig. 2. Service-based tests are explained as following:  HTTP (HyperText Transfer Protocol) tests provide measurements as shown in Table 2. The measurements are collected during the period of time from when a HTTP request is sent through the network until the last byte of HTTP page is received.  FTP (File Transfer Protocol) tests provide measurements as listed in Table 2. The measurements are obtained for transferring large and small files.  Video streaming tests provide measurements as shows in Table 2. The measurements are collected during the period of time from when a request is sent to the network to the time when the video is received.  PING tests provide measurements as presented in Table 2. PING task perform network connectivity and performance using small payloads.2.2. Data processing In this step, data collected are processed in order to find major statistical measures, such as minimum, maximum, and average values for all collected KPIs.   Fig. 3. If the connection does not go through in 120 seconds it is considered as failed. The total size measured in (kB), it indicates the minimum, the maximum, and the mean download application throughput. The application throughput of HTTP for all three broadband networks is shown in Fig. 4. After the application throughput, the test demonstrates the download throughput for each website, measured in (kbps). The download throughput indicates the minimum, the maximum, and the mean download throughput. The results of HTTP for all three broadband networks are shown in Tables 4, 8, and 12. After the HTTP, the test cycle moves to a series of the FTP download and FTP upload measurements. Two file sizes are used: a large and a small file. A large file has size of 3.38MB.
The number of times to run is 1. Time out is 120 seconds. A small file has size of 141kB. The number of times to run is 1. Time out is 60 seconds. The test measures the FTP success and the failure of the downloading and uploading small and large files. It demonstrates the total time count of the FTP downloading and uploading small and large files, success, and success rate. The success rate of FTP for all three broadband networks is shown in Fig. 3. Besides the success rate, the test measures FTP application throughput of downloading small and large files. The application throughput of FTP for all three networks under test is shown in Fig. 4 Fig. 3. The number of times to run is 1. Time out is 60 seconds. The data size is 32 Bytes. The results of ping measurements for the networks are shown in Tables 7, 11, and 15.

Data Analysis
The objective of the data collection is the comparison of the user experience across different wireless networks. With the help of G-station software the measured data are imported and a set of various KPIs is calculated. Through the KPIs and the other measurements gathered from the tests, one may make a case by case assessment of a connection which provided good quality of service. In this study, the networks are compared using the following three basic QoS parameters; success rate, average throughput, and throughput jitter. Each one of the QoS parameters is an important component of the wireless cellular network performance [2]. The three QoS KPI parameters are explained as follows: 1) Success rate -is defined as the percentage of service attempts with respect to all service attempts. 2) Average throughput -is the average of all call throughput measurements obtained from the call start time to the call end time. It is measured in kilobits per second (kbps). 3) Jitter -is the standard deviation of the throughput over the period of a call. It is measured in kilobits per second/second (kbps). Based on these three QoS parameters, a scoring matrix like the one shown in Table 3 may be constructed. Each of the QoS parameters has a ranking from one to five. Each is based on different thresholds which may be used for QoS evaluation of the network. The thresholds are reflection of current practice.

Presentation of Results
Data Quality Index (DQI) is used to evaluate the Quality of Service (QoS) of cellular data networks [3]. The scoring system is from one to five, where one indicates the lowest quality, and five indicates the highest quality as demonstrated in Table 3. For example, the network may receive the highest score on the success rate parameters if 99% of the tests are setup successfully. The blocking probability of the network should be no more than 1 %. The QoS parameter score is presented as DQI is a weighted average of QoS parameters scores. The are weights associated with individual scores. DQI equation is shown below: A network with highest DQI score is perceived to be the one with the best user experience. The score is technology and network independent, which makes it suitable for QoS evaluations. Each KPI measurement is scored based on the range defined in Table 3.

Summary of results from home environment
The collecting of data began on the 23 rd of June 2011 and continued through June 24 th 2011. The test was scheduled to begin at 6:00PM and ended at 12:00PM. The home testing site is located in the suburbs of Melbourne, Florida. Melbourne is a coastal city located on the east coast of Central Florida.
The data were collected by using a laptop that was equipped with (JDSU) E6474A Wireless Network 114 Evaluation of Quality of Service in 4th Generation (4G) Long Term Evolution (LTE) Cellular Data Networks Optimization Software, and utilized a 802.11g wireless connection with a connectivity speed up to 54 Mbps. After the collection of the KPI measurements, the post processing was conducted using the Gladiator G-Station software. The analysis produced Tables 4 through 7:

Summary of Results from Office Environment
The collecting of data was scheduled on the 20 th of June 2011. The test was scheduled to begin at 11AM and ended at 6:30PM. The office testing site is located in Melbourne, Florida close to the Melbourne International Airport. The wireless access is 802.11n. The analysis produced Tables 8   through 11:   Table 8. HTTP measurements Table 9. FTP download and upload measurements

Summary of Results from Drive Test
The collection of the data was scheduled on the 21 st of May 2013. The drive test was began at 11:00AM and ended at 5:00PM. The drive test was performed in Melbourne Florida, and Palm Bay Florida Fig. 5. The two towns are adjacent to each other, and allow for a smooth flow of traffic.
The tests were conducted in suburban, urban, industrial, and commercial areas within the two towns. The analysis produced Tables 12 through 15:

The Comparison between home, office, and LTE drive test
The summary of results from the three broadband networks is shown in Table 16. The three QoS parameters success rate, average throughput, and jitter are compared. The weighted average of QoS parameters has a range between zero and one. The weight value depends of the importance of a given service to the end user. Therefore, weights vary depending on the type of service used. Customers using web services are not likely to notice the effects of high jitter as they would when watching a video streaming. When weights vary, the weight of video streaming is likely to remain high within all applications. The users that cannot establish a connection to the network are discouraged regardless of what kind of service they wish to use.
Based on the obtained DQI values for all three networks, commercial LTE cellular network (drive test) provides the best user experience and it has the highest DQI score. However, DQI value calculated for LTE networks approaches Wi-Fi home experience. This result makes LTE networks very competitive with traditional broadband services.

Conclusions
The paper demonstrates how to measure network performance by using the Key Performance Indicators (KPIs). KPI is an evaluation criterion that measures the QoS of a data network. KPIs are used as a basic unit of measurement for monitoring the QoS of the network. The research presented in this paper is addressing two points.
First: based upon the collected data, a set of meaningful measurements are proposed. These measurements are technology agnostic. Their main purpose is to capture the user's experience within the network's QoS. Second: based upon the collected measurements, a set of KPI's is calculated. The KPI's are used to determine a meaningful set of metrics that objectively quantify the user experience. Therefore, it allows an approach to compare different networks regardless of the underlying technology. Future work will allow comparison of performance between different network environments with other wireless data networks.