LETTER


An Analytical-Predictive Model for Measuring the Efficiency and Effectiveness of Public Health Services



Neda Vitezić1, *, Antonija Petrlić1
1 University of Rijeka, Faculty of Economics and Business, Ivana Filipovića 4, 51000 Rijeka, Croatia


Article Metrics

CrossRef Citations:
0
Total Statistics:

Full-Text HTML Views: 891
Abstract HTML Views: 414
PDF Downloads: 219
ePub Downloads: 179
Total Views/Downloads: 1703
Unique Statistics:

Full-Text HTML Views: 537
Abstract HTML Views: 263
PDF Downloads: 182
ePub Downloads: 148
Total Views/Downloads: 1130



Creative Commons License
© 2020 Vitezić and Petrlić.

open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: https://creativecommons.org/licenses/by/4.0/legalcode. This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

* Address correspondence to this author at the University of Rijeka, Faculty of Economics and Business, Ivana Filipovića 4, 51000 Rijeka, Croatia; Tel: +38551355126; E-mail: neda.vitezic@efri.hr


Abstract

Background:

The main function of public health services is to improve people’s health and therefore, efficiency and effectiveness are constantly a subject of various world-wide research works. Today, in the era of digitalization, when numerous data are created and built, it is much easier to develop and implement a measurement system. It is possible to quickly use a wide variety of accurate and reliable data, aiming to create different measures that will help in the assessment and the decision-making process. For a long time, public health services have been facing a problem of finding an appropriate solution for measuring efficiency and effectiveness.

Objective:

The aim of this research is to find an appropriate analytical-predictive model for measuring efficiency and effectiveness of public health institutes. Public health is oriented to monitoring, analysis, and evaluation of the health of a population i.e., prevention activities. It is a complex and interdependent process of different realisation of services, programmes, and activities the results of which are sometimes visible only after a long period of time. Therefore, the results of their activities should be evaluated using an appropriate performance measurement system.

Methods:

The adjusted Balanced Scorecard (BSC) combined with the non-parametric Data Envelopment Analysis (DEA) technique is used to help identify the possibilities for improving the efficiency and effectiveness of public health service activities.

Results:

The result of this study is the proposed Analytical-Predictive Model (APE) that uses Balanced Scorecard combined with Data Envelopment Analysis to measure relative and technical efficiency as well as long-term effectiveness. The model used DEA as a benchmark for targets set in each perspective within the BSC. Using the BSC model, we selected the goals and common indicators for all DMUs, and using DEA, we identified relative efficiency of the DMUs. Efficient DMUs are considered a benchmark and used as targets for measuring effectiveness.

Conclusion:

This research has confirmed the appropriateness of the combination of BSC and DEA methods for measuring efficiency and effectiveness of public health institutions. To be able to measure and predict the long-term effectiveness of the activities and programmes, we had to combine the realised outputs and the set outcomes. The implementation of the APE model in the institutes of public health will contribute to the improvement of analysis, forecast, and optimisation of all their activities. The model is applicable to other public health institutions.

Keywords: Analytical-predictive model, BSC, DEA, Efficiency, Effectiveness, Public health services.



1. INTRODUCTION

Public health services are organised differently in the world, but their common task is improving populations’ health. Public health services are usually provided by institutes or agencies that organise professional activities for the improvement and protection of a population’s health as well as reducing mortality, premature deaths, and disabilities. The subject of this research is the prevention activities organised by the institutes of health services in the Republic of Croatia (hereinafter: Croatia). They are oriented to monitoring, analysis, and evaluation of the population’s health, which require goal setting, developing a planning system and implementing adequate measures. In other words, a proper performance measurement system, the role of which, besides monitoring and evaluation, should also be adjusted to communicate the extent to which various aspects of the health system meet the key objectives [1].

Today’s digital environment and the ‘Big Data Age’ provide new capabilities and organisational, managerial, and strategic benefits [2] that will contribute to a more accurate and comprehensive measurement system. These datasets that contain volumes of different unstructured, semi-structured and structured data are, besides other 'Vs', characterised by data from different sources, data accuracy, and reliability (veracity) that examine the cost-benefit of data collecting (value) [3, 4]. It has been acknowledged that Big Data analytics has an important influence on improving the efficiency and quality of health care delivery [5]. Manyika et al. [6], point out that properly applied data analytics helps cut costs by approximately $300 million annually in the health care industry. Additionally, it also helps improve lifestyle management. Accordingly, Wang et al. [2] point out the following potential operational benefits of big data analytics in health care: it improves workflow efficiency, monitors quality, improves costs and outcomes, and reduces the time for extraction of information from research studies on large databases.

The health service measurement system is very complex because of various data expressed through different measures. Due to a wide range of inputs and outputs, a comparison between units is sometimes problematic and even impossible. At the same time, since the introduction of the concept of ‘new public management’ to the public sector, the measurement of efficiency and effectiveness has become almost mandatory. Public health institutes are also under pressure to provide their services as efficiently as possible. This means achieving the highest quality while operating rationally. Due to sufficient evidence of inefficiency in the health care system across EU countries [7], the efficiency of health care services is often subject of research [8-13]. However, research related to the efficiency of preventive public health services, which is of great importance for society as a whole, is very scarce. Public health services in Croatia are carried out through various programmes and projects organised at the national level along with twenty county-level institutes. According to the Healthcare Act adopted in 2008 [14], the activities of public health institutes include: microbiology, ecology, epidemiology of communicable and chronic non-communicable diseases, promotion of health, school medicine, addiction prevention, mental health, and public health care. Some of the important roles of the institutes are the evaluation of the population’s health, health care needs, and the results of health care services, as well as the introduction of health programmes and their coordination at the national and international level.

All these activities require a developed measurement system for efficiency and effectiveness assessment. Data Envelopment Analysis (DEA) introduced by Charnes, Cooper, and Rhodes [15] is very often used in health care for measuring relative efficiency [16-26]. DEA is a mathematical programming method for evaluating the relative efficiency of different units operating in similar conditions and the conversion of inputs into outputs. It identifies the sources and the level of inefficiencies and shows how some of the inputs and outputs can be improved to become relatively efficient for an inefficient unit. DEA could be used together with the Balanced Scorecard (BSC) and some studies found great potential in their integration [27, 28], especially when quantitative and qualitative data are used to obtain a comprehensive performance and efficiency management system [29]. BSC has some advantages, like improvement of the management information system, emphasis on the relationship between quantitative and qualitative indicators, and relatively few selected measures that can be checked at any time [30]. The BSC could not identify inefficiency in the use of materials and human resources. However, in combination with DEA, limitations such as lack of possibility to select the best measures and insufficient dynamics for simultaneous control could be avoided.

Today, health systems are still in the initial stage of performance measurement and much more needs to be achieved in order to improve their effectiveness [1]. Only a unified and comprehensive system that provides an information base for monitoring the efficiency and effectiveness of the results and set goals, enables the use of contemporary tools and methods. Therefore, every healthcare system should have set financial and non-financial efficiency indicators that lead to greater effectiveness measured through the quality improvement of services provided, and thus better health care policies at the national level.

Combining DEA with BSC in this research aims to develop an analytical, but also a predictive model for purposes of effectiveness. The main goal of any public health service is long-term quality improvements, so that the primary measure is effectiveness of a combination of outputs and set outcomes. The conceptual model proposes an adequate measurement system for the assessment of efficiency and effectiveness in preventive health service institutes.

2. ADEQUACY OF THE MEASUREMENT SYSTEM FOR EFFICIENCY AND EFFECTIVENESS ASSESSMENT

The variety of programmes and projects carried out by institutes of public health in Croatia makes efficiency measurement complex. The problem arises due to the diversity of ‘input’, ‘output’ and ‘outcome’ expressions. Inputs are generally referred to as physical, human, and financial resources. In public sector organisations, most measures are based on an input-oriented perspective, usually expressed in terms of cost, budget, and stuffing tools [31]. Comparison can be made with similar types of organisations or units using specific measures such as sampling cost, cost of analysis, cost per programme, unit cost per employee, resources used per programme, etc. Any change in these performance measures reflects the ‘effectiveness’ with which the organisation uses its resources. However, this provides little information about the operational process within the organisation, like the total cost for a single medical analysis, distribution of costs between units, overhead cost assignment, etc.

Outputs in the public sector are oriented to the satisfaction of their beneficiaries and generally cannot be traded in the market. From the aspect of health services, outputs refer to all prevention activities expressed as various analyses, samples, programmes, treatments, vaccination, etc. They are easily measured in quantitative terms, such as: the number of samples taken, number of blood analyses, number of educational programmes or educational events held, number of obese children, price of some prevention programmes, etc. Organisations usually use the calculated output/input ratio to measure how efficiently an organisation converts its inputs into outputs. This provides very little information on the effectiveness or the value of these outputs.

The problem arises when results need to be measured qualitatively or when it is necessary to express ‘outcome’. Nearly all outcomes of public health prevention activities are visible in the long run. Their goals are related to quality improvements and satisfaction of their beneficiaries, considering that public health prevention outputs should be constructed as quality/effectiveness metrics. For example, by increasing the number of vaccinated children, some diseases like measles will diminish or disappear in the long run, or more frequent measurement of sea pollution will have an impact on the purity and quality of the sea. Thus, the ultimate goal of public health services is the quality improvement for the beneficiaries, so the first necessary step is to set up the desired target values that need to be achieved in the long run. This is only possible with the measurement of technical efficiency, i.e. establishment of whether the same effects could be achieved with fewer people, in a shorter period of time, with less material etc.

Efficiency measurement sometimes uses financial inputs aiming to help answer questions about whether the output could be produced less expensively; e.g., could we have the same laboratory analysis quality when the cost per unit is reduced. To meet the beneficiaries’ needs, ensure responsible management of resources and the staff, and efficiently organise business processes, a set of various efficiency and effectiveness measures should be developed and implemented.

Although some systems move faster than others in adopting performance indicators, many countries, even those with advanced data systems, have difficulties linking practical performance to outcomes due to limitations in data availability and their linking [32]. Like in all other public systems, efficiency measurement of public health systems is a challenging task that includes conceptual and technical problems. Some of the problems highlighted by Kattel et al. [33] that can be directly linked to the Croatian public health system are the wide range of beneficiaries and difficulties in defining goals, as well as a static measurement as opposed to dynamic processes. Besides, the database system, as a precondition for creating an adequate measuring system in today’s Big Data Era, has an important role. Regarding the Croatian public health system, this phenomenon is still characterised by three main factors that need to be improved: lack of control, information integrity, and transparency [34].

Comparability is another problem, because of the heterogeneity of public health services. The issue that arises in the evaluation of efficiency and effectiveness is the comparability of inputs and outputs. Comparison is possible when monetary inputs or outputs are used, such as revenue, total costs, material costs and using common quantitative measures like the number of employees, working hours, education hours, etc. Sometimes adjustments are necessary because of the technological process; e.g., a department takes blood for testing, but some blood tests are performed in another department. In that case, costs are allocated in accordance with the time needed for each activity or a method is applied to eliminate variable pricing or differential cost structure in health care institutes.

3. METHODS AND RESULTS

3.1. The Usefulness of the BSC DEA Model for Predictive Purposes

The advantages of the BSC model have been recognised in the health system for a long time. It is used in the health care system worldwide as a management tool that can help organisations effectively implement strategies [35], or develop practical conceptualisation, mainly for hospitals [36-39]. Regardless of the advantages, BSC could not identify inefficiency in the use of material and human resources. For that reason, its combination with DEA can help avoid these limitations.

Public health services in Croatia are managed through various programmes and projects organised at the national level along with twenty county-level institutes. This research was carried out at the Teaching Institute of Public Health in one of the Croatian counties. The primary tasks of the Institute are monitoring, analysis, and evaluation of the population’s health, as well as planning, proposing, and implementing measures for the preservation and enhancement of the population’s health. Development of a model which the management of the Institute will use for the research of performance and prediction analysis starts with DEA applicability and, later, with BSC development and implementation.

A detailed analysis of the process was conducted first, regarding the organisation and processes in each of the 12 departments encompassing total 48 units (DMUs). Inputs and outputs were analysed for each department, and the departments differ according to their orientation to the patients they treat, or to a broader population in which they conduct prevention activities (prevention of epidemics, the examination of production workers, or food distribution workers). Some departments monitor and analyse the environmental factors and are not in direct contact with the population, and some also conduct diagnostic tests. Therefore, both inputs and outputs are different, and timing is also different, i.e. some activities provide results immediately or in a short period of time (vaccination, microbiological analyses), and some programmes are conducted for several years (different prevention programmes and early detection of diseases). Thus, in this phase of the research, the easiest way to test DEA was using comparable quantitative and financial data. An input-oriented model (BCC) was used, which is more appropriate than CCR due to variable return to scale permission. The BCC model evaluates the efficiency of DMUo (o= 1, 2, ..., n) by solving the following linear programme [40-42]:

Moreover, we took the BCC model for the period of three years (2014-2016), because the current health policy fosters rationality through reduced expenditures, but the management also has more control over the inputs.

INPUTS: OUTPUT:
     - working hours
     - number of employees
     - material costs
     - investment in equipment
     - realised and collected revenues

Some departments, for example, Health Ecology, operate in the market; it was thus interesting to analyse payments through collected revenues. We found that, at the level of all 48 DMUs, a third was relatively efficient in revenue collection. The projection to the efficient frontier was determined for inefficient DMUs as well as types of return to scale (RTS - increasing, constant, and decreasing). Most DMUs had a growing return to scale, which means for the decision-making process, the inputs should be increased for possible achievement of greater revenues.

To confirm the applicability of DEA to each of the departments, we chose the Department of Health Ecology that covers 12 DMUs and provides health prevention services regarding different environmental factors (air, food, water, soil…). We continued applying the BCC model and selecting inputs and outputs that were reliable and comparable for all units within the department. In this case, we used the following inputs and outputs:

INPUTS: OUTPUTS:
     - total costs
     - number of samples
     - total revenues
     - number of analyses

Inputs and outputs were highly correlated i.e., the correlation between total costs and total revenues was approximately 0.97, between the number of samples and the total revenues, it was approximately 0.70, between total costs and number of analyses, approximately 0.61, while between the number of samples and number of analyses, it was approximately 0.53. With the calculation of return, we established that only four units out of 12 had an increasing return, but only three units (Biological Monitoring and Exposure Unit, Ecotoxicology Unit, and Waste Control and Waste Water Unit) with an increasing scale showed the possibility to improve efficiency by scaling up their activities. A projection that enables movement of an inefficient unit to an efficient frontier was also calculated.

By testing DEA at the level of the institute and the chosen Department of Ecology, we confirmed that DEA is applicable with the selected common inputs and outputs. The next step was to create BSC for the entire Institute and its departments. We started with the mission and vision and selected four core perspectives. The first and the crucial one is the perspective of beneficiaries and other stakeholders (local community, suppliers, investors...) because of whom the institute exists. Continuous investments in the staff will encourage improvements and innovation, thus increasing internal efficiency and ensuring more financial funds for strengthening of the quality of services. Fig. (1) represents the strategic map with the perspectives and strategic objectives of the institute.

Based on the strategic map of the institute, all departments have developed their own BSC and strategy map based on the results obtained by DEA. For example, the Department of Health Ecology develops the map shown in Table 1.

The strategic map of the Department of Health Ecology contains in total eleven goals under four perspectives. Hence, it is not entirely identical to the strategic map of the Institute of Public Health. It is evident that the strategic map of the Department of Health Ecology does not have the “U4” user perspective goal from the Institute’s maps, which refers to the expansion of the public health service market. Hence, although the Institute’s departments work together in order to fulfill the common set of goals, their mission and vision, each department has its own particular function and therefore, not all can, nor they should have the same goals, which also includes a strategic map. They develop their maps on the basis of the set institute map, but by adopting the indicators that were considered the most important and feasible to follow. More precisely, the differences between the departments’ maps lie in the “U” and “L” perspective, while the “F” and “P” are the same for all departments. For example, the Department of School and University Medicine, whose primary goal is health care protection of school children, the youth and students, and the Department of Social Medicine, whose primary goal is to safeguard and promote the health of all residents and visitors. They will not have the “L2” objective: “modernisation of equipment and investment in supporting infrastructure (learning perspective goal). Their focus in the learning perspective will be on the development of innovative models and on the enhancement of external and internal education. By contrast, the Department of Epidemiology and Health Ecology, whose main task is testing various samples, will incorporate it due to the importance of equipment modernisation. Hence, deciding which goals and related indicators to include was a long-term process and subject to group meetings with the leaders and members of each department (in this case, the Department of Health Ecology, the Department of Controlling and the Board of Directors of the Institute).

Fig. (1). Strategic map of the Institute of public health.

Table 1. The strategic map of the department of health ecology.
Mission: To preserve and promote the health of all residents and visitors in the Primorsko-Goranska County through the protection of all environmental components.
Vision: Improving health and sustainable development through monitoring and analysis of environmental factors.
PER. GOALS MEASURES
U U1 Increase customer satisfaction through the quality of services provided • customer satisfaction rating (survey; 1-5)
• number of current complaints / number in the previous year
• number of services provided / funds planned
• number of services provided / previous period
U2 Increase the total number of environmental components’ testing • number of current tests / number in the previous year
• number of current tests / planned number of tests
• number of references / number of elaborates
U3 Increase the number of new types of services • number of new services / total number of services
F F1 Provide sufficient financial resources to provide quality services • average value (in HRK) per team of the Croatian Health Insurance Fund (HZZO)
• the share of income of the Croatian Health Insurance Fund (HZZO) in total revenues
F2 Increase revenues from services on the market • share of market-realised revenues in total revenues
• amount of invoiced services / total number of employees in the provision of services
F3 Ensure financial sustainability • collected revenues / expenses
• collected revenues / invoiced revenues
P P1 Certify, accredit, and integrate new management systems • number of implemented management systems / number of planned management systems (ISO/IEC 17025, ISO 9001, ISO 14001)
• number of accredited methods / total number of methods
P2 Improve the efficiency of internal processes with new organisational and IT solutions • number of services provided / effective hours of work
• maintenance cost method / total revenues from that method
• number of justified complaints / total number of complaints about service provided
L L1 Adoption and dissemination of knowledge • number of specialised educations / total number of employees
• number of planned educations / number of realised educations
• number of internal educations / number of external educations
L2 Modernisation of equipment and investment in supporting infrastructure • value of new equipment / total revenues
L3 Increasing innovative solutions • number of innovations (methods, procedures, analysis, processes ...)
• number of published professional and scientific papers
(Source: Own compilations)

3.2. The Conceptual Analytical-predictive Model for Efficiency and Effectiveness Measurement (APE)

BSC is created using DEA results about relative efficiency and benchmark units, which help the heads of the departments to set achievable short-term and long-term objectives. According to the strategic map created, heads of each department within the Institute could measure the efficiency of daily or monthly resource use . By measuring the efficiency with a different set of ratios for each perspective, the head of each department could easily get reports and analyse results and predict the degree of achieved targets . For example, the Department of Ecology identified DMUs that could achieve the same level with less costs and fewer employees. Projections to the effective frontier were used as target size in the BSC and prompted appropriate actions that will have an impact on the quality of public health (for example, department reorganisation). Therefore, the so-called APE model for efficiency and effectiveness measurement is presented in Fig. (2).

Fig. (2). Analytical-predictive model for efficiency and effectiveness measurement.

The model will be explained based on the Department of Health Ecology which covers 12 DMUs and provides prevention services for people’s health regarding environmental factors. The predictive analytical efficiency model consists of several phases.

In the first phase, we analysed relative efficiency using DEA for the period from 2014 to 2016. The selected inputs are total costs, number of samples and outputs, the number of analyses and total revenues; all highly correlated. The objective was to analyse which of the 12 DMUs was the efficient one in terms of the number of analyses and revenue realisation. They all perform in the same environment with appropriate equipment and technology. Using the DEA, we identified that three DMUs were efficient during the entire three-year period: for quality control of outdoor air, for the control of drinking water and natural water, and for the control of physical environmental factors.

In the second phase, BSC was created for the Department of Health Ecology with four perspectives. DEA results are used as a benchmark or targets from the perspectives of beneficiaries and other stakeholders, and from the point of view of financial management. Heads of the departments set targets until the end of 2020, attempting to measure operational efficiency on a monthly and yearly basis.

The third phase encompasses the analysis of the results of 2017 to discover whether the set targets (for example, increase in the number of analyses for 10 per cent and increase in revenues for 5 per cent) would be achieved by 2020. The results will be checked again using the DEA and the targets will be corrected if necessary. The BSC DEA model uses information technology and programmes that enable everyday evidence. Heads of departments can correct operational goals and predict the degree of fulfillment of the set long-term goals. This helps control operational efficiency and ensures long-term goals. Measurement of public health is important for the overall quality of the health care system and any improvements in operational efficiency will lead to long-term effectiveness.

CONCLUSION

Today’s orientation to digitalisation will undoubtedly have an impact on the methods, instruments, skills, and organisations used in the health care system. The health care system, in general, is oriented to meeting the expectations of its beneficiaries, aiming to ensure a healthy life for everyone. In this paper, we presented a conceptual model for efficiency measuring in an institute of public health. Their mission is preservation and promotion of health of citizens and visitors through the more efficient supply of health services. Therefore, efficiency measurement is crucial for achieving their mission and vision and, consequently for long-term improvement of the quality of life of all citizens of the county.

The predictive analytical model proposed in this paper is Balanced Scorecard combined with Data Envelopment Analysis to measure relative and technical efficiency as well as strategic effectiveness. The model used DEA as a benchmark for targets set in each perspective within the BSC. Operational efficiency is measured by indicators chosen for each objective along with initiatives. The results need to be correlated with target sizes to determine the degree of reaching the final goal set for several years in advance. Implementation of the APE in the institutes of public health will contribute to the improvement of analysis, forecast, and optimisation of all their activities and achievement of higher decision-making quality.

However, we are aware that the implementation of the model in a real case may have limitations. They are primarily related to the development of the information system as well as human resource ability and willingness to change. Secondly, there are some limitations in the methods used. DEA is a useful tool for assessing efficiency and identifying needs for improvement in similar organisations or units. In our case, when applied to a single institution, there is a limit in the number of DMUs, which also limits the number of inputs and outputs. So far, many studies have confirmed the usefulness of BSC, but some have criticised it. However, in our case, the management supports the BSC measurement system, and implementation problems may arise from different public health policy solutions (board member or organisational changes, sources of financing, etc.)

LIST OF ABBREVIATIONS

BSC  = Balanced Scorecard
DEA  = Data Envelopment Analysis
EU  = European Union
DMUs  = Decision-Making Units
BCC  = Banker, Charnes, and Cooper i.e., variable returns to scale
CCR  = Charnes, Cooper, and Rhodes i.e., constant returns to scale
RTS  = Return To Scale
DEP  = Department
HZZO  = Croatian Health Insurance Fund
APE  = Analytical-Predictive Model

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

Not applicable.

HUMAN AND ANIMAL RIGHTS

Not applicable.

CONSENT FOR PUBLICATION

Not applicable.

AVAILABILITY OF DATA AND MATERIALS

The data supporting the findings of the article is available from the corresponding author [N.V.], upon reasonable request and permission of the Public Health Institute from whom the data were obtained.

FUNDING

This work has been fully supported by the Croatian Science Foundation under the project ‘A Model for Measuring the Efficiency of Public Health Services’ (IP-2014-09-8235).

CONFLICT OF INTEREST

The authors declare no conflict of interest, financial or otherwise.

ACKNOWLEDGEMENTS

Declared none.

REFERENCES

[1] Smith PC, Mossialos E, Papanicolas I, Leatherman S, Eds. Performance measurement for health system improvement: experiences, challenges and prospects 2009.
[2] Wang Y, Kung L, Wang WYC, Cegielski CG. An integrated big data analytics-enabled transformation model: Application to health care. Inf Manage 2018; 55(1): 64-79.
[3] Merritt-Holmes M. Big Data & analytics: The DNA to a successful implementation in 2016 [Online]. 2016. Available from: http://dataanalytics.report/view-resource.aspx?id=2892
[4] Zhang J, Yang X, Appelbaum D. Toward effective big data analysis in continuous auditing. Account Horiz 2015; 29(2): 469-76.
[5] Murdoch TB, Detsky AS. The inevitable application of big data to health care. JAMA 2013; 309(13): 1351-2.
[6] Manyika J, Chui M, Brown B, et al. Big Data: The Next Frontier for Innovation, Competition, and Productivity The McKinsey Global Institute 2011. Available from: https://bigdatawg.nist.gov /pdf/MGI_big_data_full_report.pdf
[7] Medeiros J, Schwiers C. Efficiency estimates of health care systems, Economic Papers 549. Luxembourg: Publications Office of the European Union. 2015. Available from: http://ec.europa.eu /economy_finance/publications/economic_paper/2015/pdf/ecp549_en.pdf
[8] Anselmi L, Lagarde M, Hanson K. The efficiency of the local health systems: investigating the roles of health administrations and health care providers. Health Econ Policy Law 2018; 13(1): 10-32.
[9] Cylus J, Papanicolas I. Using data envelopment analysis to address the challenges of comparing health system efficiency [Special Issue]. health care policy. Res Alloc Finance Sustain 2015; 8(S2): 60-8.
[10] Grigoli F, Kapsoli J. Waste not, want not: The efficiency of health expenditure in emerging and developing economies. Rev Dev Econ 2017; 22(1): 384-403.
[11] Herwartz H, Schley K. Improving health care service provision by adapting to regional diversity: An efficiency analysis for the case of Germany. Health Pol 2018; 122(3): 293-300.
[12] Hollingsworth B. The measurement of efficiency and productivity of health care delivery. Health Econ 2008; 17(10): 1107-28.
[13] Worthington AC. Frontier efficiency measurement in health care: A review of empirical techniques and selected applications. Med Care Res Rev 2004; 61(2): 135-70.
[14] Healthcare Act. 2008.Official Gazette of the Republic of Croatia No 150 [Online] Available from: https://narodne-novine.nn.hr /clanci/sluzbeni/2008_12_150_4097.html
[15] Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. Eur J Oper Res 1978; 2(6): 429-44.
[16] Asandului L, Roman M, Fatulescu P. The efficiency of healthcare systems in Europe: A data envelopment analysis approach. Procedia Econ Finance 2014; 10: 261-8.
[17] Coppola MN, Ozcan YA, Bogacki R. Evaluation of performance of dental providers on posterior restorations: Does experience matter? A Data Envelopment Analysis (DEA) approach. J Med Syst 2003; 27(5): 445-56.
[18] Khushalani J, Ozcan YA. Are hospitals producing quality care efficiently? An analysis using dynamic network Data Envelopment Analysis (DEA). Socioecon Plann Sci 2017; 60: 15-23.
[19] Kohl S, Schoenfelder J, Fügener A, Brunner JO. The use of Data Envelopment Analysis (DEA) in healthcare with a focus on hospitals. Health Care Manage Sci 2019; 22(2): 245-86.
[20] Kaitelidou D, Katharaki M, Kalogeropoulou M, et al. The impact of economic crisis to hospital sector and the efficiency of Greek public hospitals. EJBSS 2016; 4: 111-25.
[21] Lacko R, Hajduová Z, Gábor V. Data envelopment analysis of selected specialized health centresand possibilities of its application in the terms of slovak republic health care system. J Health Manag 2017; 19(1): 144-58.
[22] Moreno P, Lozano S. Super SBI dynamic network DEA approach to measuring efficiency in the provision of public services. Int Trans Oper Res 2019; 25(2): 429-756.
[23] Stefko R, Gavurova B, Kocisova K. Healthcare efficiency assessment using DEA analysis in the Slovak Republic. Health Econ Rev 2018; 8(1): 6.
[24] Nistor CS, Ștefănescu CA, Crișan AR. Performance through efficiency in the public healthcare system - A DEA approach in an emergent country. Economic studies babes-bolyai university 2017; 62(1): 31-49.
[25] Martini G, Berta P, Mullahy J, Vittadini G. The effectiveness-efficiency trade-off in health care: The case of hospitals in Lombardy, Italy. Reg Sci Urban Econ 2014; 49: 217-31.
[26] Sherman HD. Hospital efficiency measurement and evaluation. Empirical test of a new technique. Med Care 1984; 22(10): 922-38.
[27] Eilat H, Golany B, Shtub A. Constructing and evaluating balanced portfolios of R&D projects with interactions: A DEA based methodology. Eur J Oper Res 2006; 172(3): 1018-39.
[28] Rouse P, Putterill M, Ryan D. Integrated performance design: insights from an application in aircraft maintenance. Manage Account Res 2002; 13(2): 229-48.
[29] Kádárova J, Durkáčová M, Teplická K, Kádár G. The Proposal of an Innovative Integrated BSC-DEA Model. Procedia Econ Finance 2015; 23: 1503-8.
[30] Hemati M, Danaei A, Shahhosseini M. An empirical study to measure the relative efficiency and strategic planning using BSC-DEA. Manage Sci Lett 2012; 2(4): 1109-22.
[31] Boland T, Fowler A. A system perspective of performance management in public sector organisations. Int J Pub Sect Manage 2000; 13(5): 417-46.
[32] Braithwaite J, Hibbert P, Blakely B, et al. Health system frameworks and performance indicators in eight countries: A comparative international analysis. SAGE Open Med 2017; 52050312116686516
[33] Kattel R, Cepilovs A, Drechsler W, Kalvet T, Lember V, Tõnurist P. Can we measure public sector innovation? A literature review Ragnar nurkse school of innovation and governance 2013; tallinn, estonia [Online] Available from: https://bit.ly/2DbaS8u
[34] Pristaš I, Ivanković D, Valentić M, Golubović S, Pleše B, Brkić M, et al. Culture of health information in croatia. Social ecology. Journal for environmental thought and sociological research 2017; 26(1-2): 13-21.
[35] Bisbe J, Barrubés J. The balanced scorecard as a management tool for assessing and monitoring strategy implementation in health care organizations. Rev Esp Cardiol (Engl Ed) 2012; 65(10): 919-27.
[36] Chang W, Tung Y, Huang C, Yang M. Performance improvement after implementing the balanced scorecard: A large hospitals experience in taiwan. Total Qual Manage 2008; 19(12): 1257-8.
[37] Lovaglio PG, Vittadini G. The balanced scorecard in health care: A multilevel latent variable approach. J Model Manag 2012; 7(1): 38-58.
[38] Prakash V, Annapoorni D. Performance evaluation of public hospitals in Tamil Nadu: Dea approach. J Health Manag 2015; 17(4): 417-24.
[39] Yilmaz N, Erdem R. Balanced scorecard applications in health care. J Int Health Sci Manage 2015; 1(1): 53-9.
[40] Banker RD, Cooper WW, Seiford LM, Zhu J. Returns to the Scale in DEA 2011.
[41] Cooper WW, Seiford LM, Tone K. Data envelopment analysis, a comprehensive text with models, applications, references and dea-solver software Second Edition. 2007.
[42] Zhu J. Quantitative models for performance evaluation and benchmarking 2014.