Effects of Results Based Financing Models on Data Quality Improvement in Benin on 2014

The performances generated by results based financing (RBF) raise questions about the quality of data used to measure them. This study was carried out to evaluate the RBF contribution to the quality improvement of maternal and child health data. Sixty seven health facilities are sampled randomly in three strata (RBF_PRPSS, RBF_PASS, No_ RBF) gathering health zones in Benin. The strata comparison with maternal and child health data for the first halves 2014 and 2011 revealed that timeliness and completeness have improved in the strata exposed to results based financing compared to unexposed (p <0.05). Compared to No_RBF stratum, accuracy deteriorated in RBF_PRPSS stratum, with a 52% decrease in the proportion of concordance reports between 2011 and 2014. In 2014, recorded discrepancies are largely under-reported in the RBF_PRPSS stratum, and over-reported in RBF_PASS stratum. The accuracy of National Health Information and Management System (SNIGS) data compared to RBF’s validated data was low, verification factors varying between 30% and 97%. In total, the data quality was globally poor in most areas in Benin and it is not improved by the results based financing models implemented between 2011 and 2014. There is a need to target data quality in RBF models and to use SNIGS as a unique system of reporting.


Introduction
Health statistics are vital in the implementation of interventions to improve health [1]. Their management is a concern for officials who assign them an important role in the selection and implementation of health interventions [2]. The World Health Organization [3] established that the health information system is one of the health system building blocks and therefore a key axis of improving population health. In order to increase the performance of the health system, including use and quality of health services, many countries opt for results-based financing (RBF) [4].
The RBF is a contractual approach in which the health care and service providers receive funders' resources proportionally to their services to the population [4]. This funding model has contributed to the increasing facility-based delivery rate, uptake of family planning, and preventive consultation of children in the first four weeks of life [5,6]. There is also improvement of the quality of care in hospitals [7], and antenatal services [6]. The RBF emphasizes a heavy use of data to calculate performance and RBF incentives for health facilities [4].
However, large deficits in data quality are reported in low-income countries where the RBF is mostly applied [8]. For instance in immunization programs, discrepancy between reported and actual coverage increased since GAVI started supporting health systems [9]. In Mozambique, one of the countries concerned, poor data quality affects each type of vaccine with a noted difference between the sheets score, records of service, and district reports [10].
The poor quality of statistical data was also raised in Benin [11,12] where RBF was established in 2012. The RBF implementation requires that data be verified before they are used for incentives payment to health facilities. Existing health data was not accurate in Benin and RBF implementers expected that the new financing model would be accompanied with a significant improvement in data quality [4]. Such improvements have been obtained in others countries [13,14].
Our study aimed to assess the contribution of the RBF models to health data quality improvement in Benin between 2011 and 2014. Specifically, this study seeks to measure RBF-related data improvement in health care utilization and accuracy of maternal and child health data.

Health System and Health Information System in Benin
The study took place in the health zones of the Republic of Benin, a West African country whose area is 114,763 square kilometers. Benin's population is estimated at 10,361,057 in 2014, with approximately 3.6% children under one year, and 24% women of childbearing age. About 61% of the population live in rural areas [15].
The health system organization of Benin is pyramidal with three levels: central, intermediate and operational. The operational level consists of 34 health zones. The health zone is a network of public and private health facilities organized around a reference hospital called zonal hospital.
The country's health information system is organized according to the health pyramid. Around the National Health Information and Management System (SNIGS), gravitate systems developed by specific programs (HIV/Aids, malaria, tuberculosis). The RBF program, targeting maternal and child health (MCH), has also developed its information system ( Figure 1). For all systems, data collection is first made in log books or timesheets at the health facilities. Log books are then transcribed in SNIGS reports and RBF reports. According to national guidelines, SNIGS reports sent to the zone office must be controlled for quality, while RBF reports must be verified in situ in health facilities by an external team [4]. After capture, the electronic versions of the reports are transmitted to senior levels (departmental directorate and ministry). In RBF_PRPSS (RBF model implemented by the Health System Performance Strengthening Project) areas, RBF data are directly transferred to the RBF platform.

Study Design and Data Collection
The study is based on cross-sectional data. It describes, compares and analyses strata exposed to RBF and unexposed stratum considered as "comparison group". Data was collected between June and September 2015 and covered the period January 2011 to June 2014. Health zones covered by the study were selected in three strata formed on the RBF model: The study populations were health facilities that offered MCH services regularly. Service utilization concerns the following MCH services: new antenatal care (ANC), facility-based delivery, new post-natal consultation (PNC), family planning (FP), Penta3 and measles for children under 12 months' immunization.
Criteria such as number of communes, location (north, south), similarity of accessibility, similarity of recent partners support, were used to match the health zones of three strata and thus form trio's zones. Then two trio of zones were random selected. Zones without functional hospital since 2011 were excluded because they may not have been eligible for RBF.
With resource constraints, 50% to 67% facilities were retained in each zone. In total, 67 health facilities were drawn by simple random selection. All MCH activities management tools of the first halves 2011 and 2014 have been used. The tools used for this study are adapted tools of the data quality self-assessment (DQS) of immunization set up by World Health Organization [16]. A specific form was used for data collection in monthly reports of maternal and children health, immunization reports, MCH daily count registers, SNIGS database for the zone office, monthly reports of RBF benefits and books or slips for transmission of the first halves 2011 and 2014 reports.

Variables aspects and analysis
MCH data accuracy is the main variable of interest. It is expressed by the correlation between (1) utilization numbers of each MCH service contained in the logbooks of the facilities and (2) the number of users contained in the final report at the exit point of the data to the national system ( Figure 1).
Others variables are timeliness and completeness of reporting.

Data Accuracy
Accuracy of data of the SNIGS database in zone office (2) with data of MCH registry in health facilities (1) was determined by calculating the Verification Factor (VF) and discrepancy rates. VF indicates the direction of the inaccuracy and discrepancy rate allows scale appreciating.
Accuracy was also determined between the RBF validated data (3) and those of SNIGS database in zone office (2) ( Figure 1).
These indicators were calculated for the ANC, facility-based delivery, the PNC, FP, Penta3 and Measles immunizations.

Completeness and timeliness of reporting
Completeness of reporting is the number of reports expected. It is obtained by dividing number of reports received for the period (numerator) by the number of expected reports (denominator) in the zone office.
The timeliness of reporting is the timely transmission (before the 05 th of the month) of all reports from the facilities to the zone office. It is calculated by dividing the number of reports received on time (numerator) by the number of expected reports (denominator).
The collected data were entered in forms designed in Microsoft Excel.

Descriptive analysis
Timeliness and completeness of reports transmitted were determined for each type of report in each facility. Facilities were grouped into "prompt" (if 100% reports submitted on time) and "not prompt" to assess the progress made between 2011 and 2014.
VF was calculated for each MCH service in each stratum then compared to quality threshold (target). That allows concluding to:  data concordance, if the 95% ≤VF≤105%;  data over-reporting when VF <95%  data underreporting when VF> 105% Discrepancy is considered high if the rate is superior to 5%.

Strata comparison
Difference-in-difference method was used for comparison [17]. The discordance's difference between 2011 and 2014 in each stratum was calculated, as well as the difference between each RBF stratum and No_RBF stratum.
The accuracy of SNIGS data with RBF validated data in 2014 was determined by calculating the difference of discrepancy rate. The 95% confidence interval and the chi2 test, with p-value < 0.05, were used.

Ethical Considerations
The study protocol was approved by the National Committee for Ethics in Health Research. The administrative and technical authorizations have been obtained from the authorities of the Ministry of Health and RBF projects.

Results
In the 67 selected health facilities in the health zones of the three strata, 124,199 MCH uses were reported for the period of the study in the SNIGS database of zone offices.

Timeliness in 2011 and 2014
According to the Table 1, the timeliness of reports transmission has increased by at least 14% by 2014 compared to 2011. It is estimated 62% in 2011 and 77% in 2014 in the No_RBF stratum, against 22% and 83% in the RBF_PASS the stratum.

Completeness of the activity reports in 2011 and 2014
Completeness of monthly transmitted reports (Table 1) increased from 91% in 2011 to 95% in 2014 in the No_RBF stratum. In the RBF_PASS stratum, it increased from 55% in 2011to 100% in 2014. Completeness increase in both strata exposed to the RBF is at least 15% higher than that of the No_RBF stratum (p <0.001).

Global data accuracy of each stratum
In 2014, over-reporting of data was nearly universal for all services in the RBF_PASS stratum, the VF increased from 55% to 91% (Table 2). The RBF_PRPSS stratum recorded underreporting for three services (ANC, Penta3 and Measles immunization). The concordance of the reports was noted in the No_RBF stratum.
Compared to those of 2011, discrepancies were significantly reduced in 2014 in all strata, except PNC in the No_RBF stratum (DD = 7%) and the RBF_PRPSS stratum (DD = 42%), and also the FP in the RBF_PASS stratum (DD = 43%).
The decrease of discrepancy of the ANC data in the RBF_PASS stratum is 19% higher than its decrease in the No_RBF stratum. The RBF_PASS stratum recorded discrepancy decreasing for five out of six services in 2014, against two services (delivery and measles) out of six in RBF_PRPSS stratum.

Distribution of the accuracy of MCH data
Between 2011 and 2014, the proportion of concordance of services reports has increased significantly in the No_RBF stratum (22% to 69%) and the RBF_PASS stratum (33% to 50%), while it has not changed significantly in the RBF_PRPSS stratum ( Table 3). The increasing of the proportion of concordance of services reports is the highest in the No_RBF stratum (47%).
At the same time, the RBF_PASS stratum recorded the highest increase (17%) of the proportion of service with over-reporting between 2011 and 2014. Relative to the No_RBF stratum, the proportion of service with underreporting has dropped less in the RBF_PRPSS stratum on the period.

Comparison of SNIGS data with the RBF data in 2014
Taking the RBF as a reference, the SNIGS data of 2014 in RBF_PASS stratum are consistent with RBF validated data for the ANC (VF = 95%), delivery (VF = 96%) and Penta3 (VF = 96%). In the RBF_PRPSS stratum, the over-reporting of SNIGS data was recorded for all services (VF <95%), except for the PNC ( Table 4).
The discrepancy of services data varies between 3% and 70% in the RBF_PRPSS stratum, against 4% and 100% in the RBF_PASS stratum. This discrepancy was 3% to 66% higher in the RBF_PRPSS stratum relative to the RBF_PASS stratum. The RBF_PASS stratum had most discrepancy only for the PNC data.

Quality of the data collection process
The data collection process has improved between 2011 and 2014 in all strata. The timeliness progressed most in the strata exposed to the RBF that had a very low rate in 2011 (22% for RBF_PASS stratum, and 49% for the RBF_PRPSS stratum). The RBF_PASS stratum recorded the strongest progress, almost quadrupling its' 2011 timeliness.
The completeness has improved significantly in strata exposed to the RBF, increasing less than 85% in 2011 to 100%. This improvement is significant compared to the findings of Glèlè et al [12]. Completeness level in the RBF strata is similar to that seen in Rwanda where the RBF has already been scaled nationwide [18].
In short, the indicators of MCH reports transmission are broadly higher in the strata exposed to the RBF than in unexposed stratum. All things being equal, this difference of the progress occurred concurrently with RBF implementation could be assigned to this intervention. These results are probably due to the increase of data demand that accompanies RBF implementation. Similar results were found earlier in Rwanda where the quality of health information improved with RBF implementation [18].

Accuracy of data collected for SNIGS
In 2014, the discrepancies were higher in the RBF_PASS stratum (between 14% for delivery and 50% for PF) and the RBF_PRPSS stratum (between 17% for measles immunization and 67% for PNC) than in the No_RBF stratum (between 1% for ANC and 15% for PNC). These discrepancies are higher than those noted by Kayode et al [19] in Greater Accra, with a variation of 0.15% for deliveries and 1.05% for all registered persons. In Mozambique, a lower discrepancy (below 10%) than ours has been reported for immunization data between the register of health centers and district report [10].
The low accuracy noted in this study corroborates the earlier findings of Glèlè et al [12] in Benin. Using LQAS method and a 5% threshold, they observed a discrepancy of over 25% of maternity and immunization data. Others similar findings of low accuracy of data regarding the immunization were made in several low-income countries between 2001 and 2004, and more recently in Côte d'Ivoire [8,20].
The accuracy improvement of service data between 2011 and 2014 seems more important in the No_RBF stratum than in strata exposed to RBF. The accuracy decreased for five services out of six in the RBF_PASS stratum, and three out of six in the RBF_PRPSS stratum. In 2014, the proportion of services with concordance of reported data has increased most in the No_RBF stratum. Adamki et al [21] have attributed such increasing of discrepancy of Penta3 data between 2011 and 2012 to the absence of data sources and the poor data transcription.
Given observations made during the data collection, the significant decrease in the overall levels and distribution of data accuracy in the RBF strata, compared to the No_RBF stratum could be explained by: • Summation errors in the logbooks and early compilation of data (before the end of the month): this practice is common of facilities that want to receive the RBF subsidies related to timeliness of reporting. These factors already noted in Ghana [19], would result from the low quality of the training of the data officers; • Failure of the intra-zone data quality control mechanisms: provisions for the data validation are not functional in the facilities ; • Use of different rules and tools for data transmission: one rule for the RBF, different from those for SNIGS and others programs. This creates additional workload, generates adjustment efforts to respect RBF directives, and finally creates confusion and inconsistency in the data.
The large discrepancy between the SNIGS data and the RBF validated data confirms the above. This discrepancy is greater than 3% to 66% in the RBF_PRPSS stratum relative to the RBF_PASS stratum. This finding corroborates the absence of the discrepancy reduction recorded for most services between 2011 and 2014 in the RBF_PRPSS stratum.
In sum, health data quality is poor in Benin. RBF implementation does not increase accuracy thus data quality systematically. However, the RBF improves the full and prompt availability of activity reports.
To resolve the poor quality of the data, Wagenaar et al [22] carried out in Mozambique an intervention on health information quality which led to a significant improvement in the data concordance. In Benin, satisfactory results were obtained in the immunization field with the regular use of the DQS tool [23]. Consequently, the tool used for this study, which is derived from the DQS method, could be used to improve data quality. It would be integrated to the RBF quality assessment and used at least quarterly to identify and correct deficiencies. Data quality would be also enhanced by the use of SNIGS as unique system of reporting to reduce workloads and errors. However, further analysis of the PRISM (Performance of Routine Information System Management) factors may be useful to focus on the determinants of inaccuracy such as the absence of an organizational culture based on use of quality data [24].

Limitation of the study
The data were collected in 67 selected health facilities. When the logbooks did not exist, the data were considered poorly reported, except for one health zone (Klouékanmè-Toviklin-Lalo), which RBF data were excluded. Assessment of progress in the data quality in the RBF_PASS stratum was measured with a single zone's data (health zone Bassila).
These limits do not reduce the quality and validity of the results obtained by random selection of facilities and a 330 Effects of Results Based Financing Models on Data Quality Improvement in Benin on 2014 rigorous and comprehensive data collection.

Conclusions
The results of this study confirm the complexity of any attempt to improve and maintain a high level of data quality. The implementation of the RBF has increased mostly the completeness and timeliness in exposed strata. The RBF has not generated greater accuracy.
The RBF models currently implemented in Benin do not improve the quality of data, especially since they do not target related indicators. This limit of Benin models reflects the inability of the RBF to consistently generate positive effects on non-target indicators. The results of the study suggest that the RBF models target data quality by regular use of the tool of this study, directly use of SNIGS data and contribute to the development of an organizational culture based on the rigorous operating of data.