Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs

,


INTRODUCTION
• Key Performance Indicators (KPIs) represent a quantifiable measure of achieving goals set by an organization, both operational and strategic. Generally speaking, companies have different KPIs depending on their priority criteria. KPIs can also be established arbitrarily but, to be useful, they must meet the following requirements1: • Quantifiability -KPIs must be presented in the form of numbers.
Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs • Practicality -they integrate well with current business processes.
• Directionality -they help to determine whether companies are improving.
• Operations -they can be related to the practical context to measure an effective change.
The four leading indicator typologies are 1 : 1. General indicators -measure the amount of work in the process.
2. Quality indicators -evaluate the quality of the process output based on certain standards.
4. Service or time indicators -they measure the response time from the start of the process until its conclusion.
Thanks to a set of KPIs, it is possible to evaluate the performance of a clinical engineering service (CES).
This paper discusses the current status of benchmark indicators within the field of clinical engineering. The paper focuses on the evaluation and optimization of the medical equipment repair and maintenance activities of a CES by applying the Pareto principle to KPIs to focus on main critical activities.

STATE OF THE ART
According to Cohen et al., KPIs represent the process of comparing business performance levels to identify opportunities to improve. The results provided an idea of what should be changed and how. However, comparisons have met with limited success due to poor and inconsistent definitions of the parameters measured and lack of quality data available. In the first phase, it was necessary to identify basic indicators that are applied to any healthcare facility. They must be uniquely defined and consequently calculated by the same method so that they can be compared. Afterward, it was possible to start to build other indicators that will be different depending on the specific needs and the technical-economic information that everyone possesses. For this reason, the primary purpose of this paper was to detail some of the quantitative performance and cost benchmark indicators that have been historically used in medical equipment maintenance and repair and to make recommendations on how the clinical engineering profession can develop good quality, useful and meaningful benchmarks. The general characteristics of a useful benchmark indicator are2: • well defined; • objective; • measurable; • based on current knowledge experience; and • valid.
The study by Bassem et al. aims to evaluate CES's performance at the University of Cairo, Egypt, using quantitatively measured parameters to allow comparison and improvement objective.3 In addition to the parameters proposed by Cohen et al., considered insufficient, Bassem proposed new indicators. Data were collected by 10 hospitals, corresponding to different healthcare organizations. 2,3 These data were subsequently analyzed by a software tool, providing a score for each CES. The first step was to decide what exactly to evaluate and monitor. Some of the studies reported by Bassem adopted a survey technique as CES directors were asked to select from a list of proposed performance indicators that could be used for performance measurement benchmarking. Their response revealed three mainstream performance indicators. Other indicators had to be added and measured to evaluate the performance of the other services. They used additional essential indicators that should be involved to get an increased accurate evaluation. The results indicated an average gap of 67% between the performance of the CES and the reference they have identified, considered the ideal target. 3 According to the Tiwari study, service performance on medical equipment serviced by external suppliers is assessed. 4 The performance indicators of CES are first defined according to the needs and benefits required in the specific hospital structure and are then categorized and finally measured as indicated below.
1. The definition of KPIs considering the opinion of experienced staff.
2. The categorization of KPIs into four groups.
3. The measurement of KPIs.
The Tiwari study is an example of an outsourced CES, and the key to success is the measurement of performance to quantify the expected benefits. 4

METHODOLOGY
This study, conducted by the University Campus Bio-Medico of Rome (UCBM), where there are about 6,000 pieces of biomedical equipment for the year 2016. Before going into the explanation, and then in the calculation of the KPIs, it is vital to understand what type of data have been used. The CES utilizes an equipment management software, in which all the data are collected and related to the inventory number of biomedical equipment: the description, the serial number, the manufacturer, the purchase value, the intervention priority, therefore any information useful to characterize a specific piece of equipment. From the mentioned software, further information can be obtained relating to work orders and the schedule for preventive maintenance. In the first case, the CES takes care of entering all the work orders executed. In the second case, the CES takes care of inserting, within a schedule, all the equipment and the corresponding deadlines for preventive maintenance, in order to record the periodicity with which it is required to carry out maintenance. This approach maximizes effectiveness and efficiency in technical management and ensures economic and technical control of maintenance. It has the objective of providing operational and decision-making support for further optimizing the processes related to registry/inventory management. In this way, from this software, it is possible to obtain categorized data, from which it is possible to calculate the KPIs. Starting from these data, the results are obtained through the use of "Matlab," an environment for numerical calculation and statistical analysis, which also includes the programming language. It allows the calculation of the KPIs considered here. The following paragraph will report the explanation, and the subsequent calculation, of the identified and measurable KPIs.
The following KPIs are used and calculated: 1. Uptime. This denotes the time the biomedical equipment has been working for over one year; downtime is its complementary statistic and denotes the state of a not operational system. This may be due to failure, preventive maintenance, or other causes. The measurement is carried out in absolute values or percentage. Uptime is particularly important for all machines where stability and availability are fundamental. Through Uptime, efficiency can be deduced: a high Uptime indicates that the equipment is well configured, while a low Uptime could mean instability of the equipment. To get more evidence on the critical issues, this KPI calculation involves the initial use of data from all the equipment from which one or more work orders have taken place. Also, all devices that have not undergone a work order are then considered and has always been functional; a maximum Uptime value will appear. The Uptime calculation, represented by a percentage, is carried out by first calculating the downtime: the work orders corresponding to each inventory number of the equipment are considered and, consequently, the duration given by the sum of all the work orders for that specific inventory is calculated. It is then divided by the number of days within a year to indicate, the percentage of the number of days that a specific piece of equipment remained inoperative relative to the total period. 4 The formulas used are the following: 2. MTTR (Mean Time To Repair). This denotes the Time to Restore (TTR) expected value, where the TTR is the time interval where the equipment is unavailable due to a failure. The MTTR includes the time for diagnosis, the arrival of the maintenance technician, the arrival of the component(s) to be replaced, and the actual repair. It is a useful parameter for evaluating the effectiveness of the CES in terms of the logistic organization. The calculation of this coefficient involves data from all equipment on which a work order has occurred involving corrective maintenance or functional verification. The MTTR is calculated according to the work orders corresponding to the equipment's inventory and the duration given by the sum of the times to repair in all the work orders, or that specific item is calculated. This is referred to as TTR and it is then divided by the total number of work orders within a year. 4 The formula used is: 3. MTBF (Mean Time Between Failures). This term corresponds to the average time interval between two successive failures (TBF) and indicates the frequency with which a failure can occur. This is mainly a reliability parameter used to indicate the probability that equipment operating under certain conditions will retain, after a predetermined time, the functional capacities for which it was built. The calculation of this coefficient involves data from all equipment on which a work order has occurred that involved corrective maintenance or functional verification. The work orders correspond to each inventory number of the equipment and consequently the total time that elapses between the start date of one work order and the start date of another. This would be the TBF. The TBFs from all work orders are added together and then divided by the total number of work orders within one year.4 The inverse parameter, defined as "frequency of failure," indicates the rate at which technicians must carry out maintenance. The formulas used are the following: 4. COSR (Cost Of Service Ratio). COSR is an economic parameter that represents the sustainability of costs. It is calculated as the ratio between the overall maintenance cost and the purchase cost, assessed through a percentage measure.4 The calculation of the overall COSR is carried out by proceeding in three phases outlined below.
1. Equipment with maintenance contracts, to which maintenance cost (if any) has been added the cost of the pieces of spare parts.
2. Equipment with only spare parts and without maintenance contracts.
3. Company cost of all the personnel working in the CES.
The formula used is the following: 5. PPM (Percentage Preventive Maintenance). This term expresses the overall number of preventive maintenance events or carried out within the deadline, divided by the total planned preventive maintenances within a year expressed as a percentage. 4 This calculation is made for each piece of equipment based on the future expiration date of preventive maintenance and the scheduled maintenance frequency. This makes it possible to derive the previous preventive maintenance expiration date, which is compared with the date of the beginning of the maintenance carried out on each piece of equipment, allowing us to understand if the maintenance has been carried out before or after the deadline. The PPM calculation is also necessary in light of the accreditation manual of the Hospitals of the Joint Commission International. 5 According to this manual, "all medical equipment and technologies are regularly subjected to inspections, maintenances and calibrations and these activities are documented in the appropriate registers. The staff ensure that all medical equipment and technologies operate at acceptable levels and safely for operators." When there is a need to manage the maintenance of many pieces of equipment, it is necessary to adopt criteria allowing priority. The equipment is not all critical in the same way, so it is necessary to distinguish the critical equipment on which the patient's life depends from the less critical ones for which the priority level is lower. There is, therefore, an issue to solve: if the technician receives two maintenance requests at the same time, the technician must be able to evaluate what the priority request is. To do this, a risk assessment is carried out, which is done with objective criteria and not left to free interpretation. In our case, the criticality analysis is carried out based on the assignment of five scores, respectively relating to five categories of equipment criticality (Equipment Management Program Mayo Clinic 6 has taken up this method). According to these criteria, equipment that should be excluded from the plan can still be included if requested by a certification body or if scheduled by the manufacturer for periodic maintenance or calibrations. The formulas of PPM used are the following:

RESULTS AND DISCUSSION
Firstly, the first three indicators are analyzed, where possible, by applying the Pareto principle, one of the most used tools in quality management. The Pareto principle, also known as the 80/20 rule, took its name from Vilfredo Pareto, an Italian engineer who, in 1906, observed that the distribution of wealth in his country had an unequal distribution: 20% of the population owned 80% of the wealth.7 By expanding this idea to other areas and concepts, an empirical theory has been formulated which is respected in the majority of cases (this means that the distribution referred to it must be a very numerous distribution). The Pareto diagram's construction, based on this principle, shows that 20% of the causes produce 80% of the effects.
The advantages that derive from the application of the Pareto principle and diagrams are: • to help to break down big problems into smaller problems and to establish what are the main factors causing them; • to help to focus on the most important causes in order to establish priorities, using the available time more effectively; • to help to link causes with effects; and • to support in evaluating the improvement based on an analysis of the situation before and after the application of the corrective action.
To identify critical issues, the data relating to the first three indicators, namely Uptime, MTTR and MTBF, are analyzed by applying the Pareto diagrams, showing that, where applicable, 20% of the causes produce 80% of the effects. Considering all the medical equipment, it is possible to concentrate efforts only on 20% of causes to obtain a significantly better result. However, the analysis of data through the Pareto law is not always possible, but to extend its application, it expanded to 30% of causes. However, where the percentage is more than 30%, its application is not considered significant. This analysis makes it possible to focus on a small group of medical devices and understand the correlations between them (type of equipment, manufacturer). Identifying these characteristics allows the discovery of the main critical issues present within the health structure and what action is required. This will pertain to 80% of the overall behavior. On the other hand, the COSR and PPM indicators help analyze the data and suggest distribution models that enable focus on the most critical equipment.
The graph of Uptime only considered those devices on which one or more work orders took place, which involved a value of Uptime <100%. It is precisely on these parameters that the analysis of Pareto is carried out. Focusing will be on the downtime rather than on the Uptime to immediately highlight any critical issues. Overall, we note distribution of data mainly concentrated around 100%, while only a smaller percentage corresponds to a value of Uptime less than 100%, mainly distributed between 20% and 30%. Figure 1 shows the average value calculated, which is very high and aligned well with similar measurements noted in the bibliography.
The MTTR graph shows a data distribution mainly concentrated in a range between 0 and 1000 hours (42 days), while only a smaller part, the initial one, shows an increase in the number of hours that goes up to 1800 hours. As shown in Figure 2, this reflects the average value calculated.
Since MTBF is the time between failures, the calculation is performed if there are, for each inventory item, at least two failures, therefore two work orders within a year.
The data analysis focuses on the frequency of faults, and it is carried out using Pareto diagrams. The MTBF graph ( Figure 3) shows data distribution mainly concentrated in a range between 0 and 832 hours (35 days), while only a smaller part, the initial one, shows an increase in the number of hours that rises to 1354 hours.
The COSR is calculated by paying attention to equipment with COSR> 0% to highlight any critical issues more efficiently. A histogram has been constructed ( Figure 4) from these devices: the COSR trend has been highlighted through 0.5% intervals and the number of devices is then reported, having the corresponding COSR. The COSR trend resembles a Poisson distribution, in fact the data are distributed bell-shaped around a value belonging to the COSR range between 1.5 and 2%, corresponding to which we have 115 devices in 2015 and 124 devices in the year 2016. The trend over the two years is, in fact, the same. However, the highest histogram bar remains the one with COSR> 10% and will be analyzed later in detail for the analysis of critical issues. As can be seen, the COSR values are quite low at around 1%, but personnel costs must be added to this value, so overall it is around 4%.
The data analysis for the PPM is carried out through the use of histograms ( Figure 5) that highlight the equipment on which preventive maintenance took place in advance    (negative numerical value of the days) and late (positive numerical value of the days) concerning the scheduled deadline. If the number of days is 0, the preventive maintenance took place on the same day as the planned date. It should be noted that the maintenance carried out in advance is greater than the delayed since the initial section of the graph is greater than the final one.
If we want to summarize in a single value what has just been shown, the value of the Uptime, MTTR, MTBF, COSR, and PPM is reported expressing the measure according to its mean value and standard deviation. Each KPI is calculated for 2015 and 2016, as follows (Equipment that does not have work orders is also included in the Uptime calculation, resulting in 100% Uptime).
As shown earlier, the Tiwari study, which shows numerical calculations and graphs of identified KPIs, assesses the performance of medical equipment serviced by external suppliers; the CES examined in this work, on the other hand, is a predominantly internal CES, so a first difference is immediately apparent. In detail, however, we note that the numerical value of Uptime and COSR obtained from the two studies is comparable; the calculation of MTTR was carried out individually for each month by Tiwari, so it is clear this type of comparison is inconsistent. Also, because some work orders last more than one month; finally, in order to be able to compare the PPM, a clarification is needed, that is, it is necessary to take into account that, in the UCBM Polyclinic, the system revolves around a risk classification that guides the professionals of the CES to act according to different priorities. It is clear that comparing the two numerical values, they are different, but, taking this into account, it would then have been more significant to take as a reference the value of the calculated PPM for priority equipment, at 93.7%. Comparison with the study by Tiwari et al is shown in Table 1.
Starting with the results obtained critical issues are analyzed. In the case of the Uptime, MTTR, and MTBF, this analysis is conducted globally as there is a correlation between them and the level of the numerical calculation performed. They are calculated starting from the durations of the work orders that are carried out during a year. It is therefore important to concentrate on the equipment for which this phenomenon is most evident and this is possible with the analysis carried out using the Pareto diagrams, which are found to be applicable only in those cases where 20% or 30% of the causes have produced 80% of the effects. Therefore, making a detailed analysis for each of these KPIs, a global analysis is deduced, identifying the equipment that more frequently falls into 20% or 30% of the causes. Also, in the case of COSR, the critical issues present are analyzed and made possible by focusing on the equipment for which COSR is more than 10%. Finally, the same reasoning is carried out for the PPM, which, regarding preventive maintenance, focuses on the type of intervention priority, such as equipment of priorities I, II, III.
For example, it is reported as an average across the fleet of pressure such as therapy units for that particular manufacturer of equipment, considered from the criticality analysis of the Uptime, MTTR and MTBF (Table 2). This paper summarizes medical equipment repair and maintenance benchmark indicators that can be used. Therefore, the clinical engineering profession must develop and use indicators that accurately reflect the true costs and quality of medical equipment repair and maintenance.
The way to analyze the results obtained is, when possible, using Pareto diagrams. They help to break down the big problems into smaller problems and to determine which are the main factors that cause them; to help to focus on the most important causes and to set priorities, using the time available more effectively help to link the causes with the effects. This methodology makes it possible to have precise information on the critical equipment that will then be replaced or repaired correctly, which will be taken through work experience and information. Performance measurement of clinical engineering departments in hospitals using these indicators will get more accurate and fairer performance evaluation. We will be able to find the real reasons for failure and improve performance. Further analysis may be required to better define creating a standard and substantive performance evaluation benchmarks and solve it.