Sepsis is a potentially life-threatening condition in which initial signs of disease can be difficult for healthcare professionals to interpret. Approximately 6% of patients admitted to hospital emergency departments and around 30% of patients that are admitted to ICU wards have signs of sepsis.1,2 Sepsis arises when the host response to infection causes organ dysfunction in the patient and is persistently associated with a high mortality of 20%-30%. Globally, sepsis affects about 30 million people every year.3 The condition of some of these patients will deteriorate to septic shock (ie, the inability to sustain normal blood pressure), which is associated with an even higher mortality rate of 30%-40%.2 Also, patients admitted to the ICU for reasons other than sepsis are at high risk of developing sepsis during their ICU stay. For example, during the COVID-19 pandemic, it was established in Wuhan that sepsis was the most frequently observed complication among adult inpatients who died during hospitalization with confirmed COVID-19.4
Timing is critical when treating sepsis. Early diagnosis of sepsis has been shown to reduce delays in treatment, increase appropriate care, and reduce mortality.5–8 Early intervention with fluids, antibiotics, source control, and organ-supportive measures will improve patient outcomes.
A recently published review of 37 cost-of-illness studies published between 2005 and 2015 found that the mean hospital cost per patient stay for sepsis ranged between $13 292 (€10 909) and $75 015 (€61 567), using the January 22, 2021, exchange rate of €/$ = 0.82).9 The medians of the mean reported hospital costs indicated higher cost for survivors ($34 855 [€28 607]) compared with nonsurvivors ($20 537 [€16 855]). Also, the hospital cost for severe sepsis and septic shock was higher ($46 834 [€38 438]), compared with septicemia ($20 740 [€17 022]). Previous studies have shown that the timing of sepsis treatment has an impact on sepsis mortality. However, the cost impact of early detection and treatment of sepsis is less known.10
Despite updated diagnostic criteria, early detection of sepsis is still complicated by the lack of reliable biomarkers.11 A large number of biomarkers have been evaluated clinically as possible prognostic markers for sepsis; procalcitonin and C-reactive protein have been most frequently investigated.12 Other biomarkers include molecules related to vascular endothelial damage, coagulation, and vasodilation; cytokine/chemokines; heparin-binding proteins; acute-phase proteins; and many others.12–14 None of these, alone or in combination, have demonstrated high enough sensitivity or specificity to be routinely used in clinical practice.12 In this context, there exists significant room for improvement for alternative diagnostic approaches.
In practice, sepsis remains a clinical diagnosis made by combining information from physical examinations with laboratory data and information from monitoring devices. This procedure is time consuming, subjective, and heavily dependent on the skills and experience of the physician. These manual routines therefore increase the risk of a prolonged time to diagnosis.
Much of the same data, and more, that are now processed to make a sepsis diagnosis can be continuously interpreted by a machine learning prediction algorithm. This type of algorithm has a large potential to improve early detection of sepsis and thereby improve patient outcomes.
AlgoDx AB (Stockholm, Sweden) has developed NAVOY® Sepsis,15 a sepsis prediction algorithm with excellent predictive properties, based on variables routinely collected at ICUs. This algorithm has been validated in a prospective randomized clinical trial (ClinicalTrials.gov identifier: NCT04570618; manuscript in preparation) and is CE-marked as Software as a Medical Device, designed for commercial use in European ICUs.
In this study, we investigated the cost-saving potential of a machine learning algorithm such as NAVOY® Sepsis for early detection of sepsis in intensive care units (ICUs) in Sweden. Since the evaluation of costs and cost-effectiveness of novel diagnostic tools depend on multiple factors such as the definition of sepsis, the diagnostic tool currently used, and the sensitivity and specificity of the diagnostic tool,16 we aimed at developing a health economic model that would enable us to simulate the cost-saving potential of a sepsis prediction algorithm in various contexts.
A health economic model with a decision tree structure (Figure 1) has been developed to capture short-term (≤1 year) and long-term (>1 year) effects of sepsis. Short-term parameters for the model included incidence of suspected sepsis in the ICU, sensitivity and specificity of the predictive method (algorithm) or diagnostic method (current practice), the probability of septic shock and in-hospital mortality (affected by time to sepsis detection and time to adequate antibiotic therapy), readmission rate after discharge, and first-year mortality after discharge. The direct effect of introducing a new prediction algorithm was modeled by the time to sepsis detection, keeping the time to adequate antibiotic therapy the same across different predictive or diagnostic algorithms. Long-term parameters for subsequent years included long-term survival and long-term consequences.
The model was informed by the available literature. Base case inputs are summarized in Table 1. The relevant patient group for this assessment was all adult patients who were not diagnosed with sepsis at the time of admission. At the beginning of the model, sepsis incidence in ICUs was identified by distributing patients to either the “sepsis” or “no sepsis” health state (Figure 1). In the model, the incidence of sepsis in ICUs was defined as the proportion of patients without sepsis who develop sepsis after admission, estimated to be 14.1% in the base case in accordance with the findings of Sakr et al.2 Patients admitted with sepsis were therefore not included in the model. These patients can develop a secondary sepsis infection during the ICU stay,17 and a prediction algorithm would certainly apply to this group of patients. For the purpose of the analysis, however, we focused only on the patients without sepsis at diagnosis.
The sensitivity and specificity of the respective screening tool divided patients further as either “true positives” or “false negatives” for patients that develop sepsis, or “true negatives” or “false positives” for patients that do not develop sepsis during their ICU stay (Figure 1). In Sweden, sepsis screening tools are not routinely used in clinical practice, and therefore the Swedish model base case evaluates the sepsis prediction algorithm in comparison with sepsis diagnosis. Other comparators are tested in scenario analyses (eg, the illness severity risk scores [SOFA]11 and the National Early Warning Score [NEWS2]).24 NEWS2 has been endorsed for use by NHS England and NHS Improvement in acute and ambulance settings to identify acute deterioration, including sepsis.25
The potential of an algorithm to improve early detection of sepsis was integrated in the model through the risk of septic shock and in-hospital mortality. To model these time-dependent variables, the sum of 2 components (mean hours to sepsis detection and mean hours to empiric antibiotic from detection) was used to estimate the total hours to treatment. In the model base case, we assumed that the mean time to empiric antibiotic from diagnosis is 0 hours (ie, ICU patients receive treatment immediately when sepsis is diagnosed by healthcare professionals). This is not necessarily in line with the timing of empiric antibiotic administration seen in clinical practice but will not affect the comparison since the same assumption is made for both comparison arms. As mentioned above, the NAVOY® Sepsis algorithm has been validated in a prospective randomized clinical trial, showing that the algorithm detects sepsis up to 3 hours earlier than onset of sepsis in accordance with Sepsis-3 criteria.11 In practice, patients are not diagnosed at the same time as the Sepsis-3 criteria are fulfilled; it can take hours before the condition is discovered and diagnosed. The use of a sepsis prediction algorithm does not imply that the patient will be diagnosed with sepsis on the sole basis of an alert from the algorithm. However, it is reasonable to assume that if a suspicion of sepsis can be made 3 hours sooner than it is currently, the whole process can be accelerated in at least the same amount of time (ie, -3 total hours to treatment compared with diagnosis for current practice). Hence, in the model base case, true positives screened with the sepsis prediction algorithm were assumed to receive treatment 3 hours earlier than “true positives” for current practice. For the “false negatives,” the appropriate treatment was assumed to be delayed by 3 extra hours (ie, +3 hours to treatment compared with diagnosis for current practice). Since there is uncertainty with regard to the time to correct diagnosis in clinical practice, scenario analyses were conducted where the time to diagnosis is varied in the -4 hour to -1 hour interval.
The risk of developing septic shock was assumed to be a continuous variable determined by the total hours to treatment compared with diagnosis for current practice. In the model base case, the risk of septic shock in current practice was estimated to be 56.6% based on the findings by Sakr et al.2 It is arbitrarily assumed that the risk of septic shock for a 3-hour earlier detection is around 40% and that the risk of septic shock per extra hour of delayed treatment thereafter increases linearly by approximately 5.5 percentage points per hour ([0.566-0.4]/3 = 0.055). Scenario analyses were performed with other risks for septic shock, as well as testing for a constant risk for septic shock, independent of time to treatment.
The risk for in-hospital mortality was also a continuous variable determined by the total number of hours to treatment compared with diagnosis for current practice. The time-dependent in-hospital mortality was extrapolated from the results of Ferrer et al,10 and different risk equations were estimated for patients who develop or who do not develop septic shock. The risk equations were estimated in 2 steps. In step 1, a linear programming function was set with restrictions to solve for the percentage of patients who have died at different time points (hours) after diagnosis. In step 2, we used the point estimates from step 1 to fit parametric curves that describe the relationship between mortality (y) and hours from diagnosis to antibiotic treatment (t) for patients with septic shock (y = 0.4281e0.0272t) and without septic shock (y = 0.0052t). This means that the sepsis-related mortality within the first hour after diagnosis is 42.8% and 0.5% for patients with septic shock and without septic shock, respectively. A 3-hour earlier detection, which was used in the base case for the algorithm, would mean that the in-hospital mortality rate for patients with septic shock is 39.5% (0.4281e0.0272(-3)). In scenario analyses, alternative methods to estimate in-hospital mortality are tested (eg, a 7.6% linear decrease in survival for septic shock per extra hour of delayed treatment, as reported by Kumar et al,5 and a constant mortality risk of 33% for patients with septic shock and 22% for sepsis patients with no septic shock, independent of time to treatment, as reported by Lengquist et al19). For patients without sepsis, no assumptions of in-hospital survival were made due to the diversity of other diagnoses in ICUs. This will not impact the results since the proportion of patients without sepsis is the same for both comparison arms.
Due to a lack of data that distinguish patients with and without septic shock with regard to readmissions during the first year, we conservatively assumed the same readmission rate for the 2 categories of patients. Zilberberg et al20 estimate the readmission rate independent of sepsis severity to be 20.6%, which is used here. To consider mortality the first year after discharge for patients with septic shock, we used the estimate from Karlsson et al,18 who estimate the first-year post-discharge mortality at 17.5%.
In years subsequent to the first year, long-term survival and long-term consequences were considered. Patients without sepsis were assumed to have the same long-term survival as the general population (which does not impact the incremental survival since it is the same for both arms), whereas relative risks on the general population’s long-term survival were applied for patients who enter the model with sepsis. The relative risks of long-term survival applied in the analysis are 5.5 in years 1 to 5, 3.1 in years 6 to 10, and 1.0 from year 11 onward, in accordance with Linder et al.23 Scenario analyses are conducted where the long-term survival for patients with sepsis is assumed to be the same as for the general population. Long-term consequences were included for patients with sepsis and applied after the first year. Long-term consequences (with associated frequencies) included impaired kidney function (14.1% of patients), amputation (8.5%), depression (2.8%), and post-traumatic stress disorder (9.9%).22
The number of ICU and hospital ward days were estimated based on the findings of Ferrer et al.10 We used their presented medians and standard deviations to provide γ distributed simulations of ICU and hospital ward days. We assumed that the shortest length of ICU stay belongs to sepsis patients without septic shock, reflecting the proportion of patients without septic shock presented in Ferrer et al10 and vice versa for patients with septic shock. We thereafter calculated the mean number of ICU and hospital days for each group separately. For sepsis patients with septic shock, the number of ICU and hospital ward days were estimated to be 7.4 days and 17.0 days, respectively. The corresponding numbers for patients with sepsis but without septic shock were estimated to be 2.0 days and 5.7 days, respectively. For patients without sepsis, the number of ICU days for true negatives was estimated to be 1.0 days in accordance with Swedish Intensive Care Registry data21 and 2.0 days for “false positives,” reflecting the number of ICU days for sepsis patients without septic shock.10 The number of hospital ward days for these groups of patients were assumed to be the same as the number of days for sepsis patients without septic shock (ie, 5.7 days). The total cost for ICU and hospital ward for each comparator arm was calculated by multiplying the number of days with a unit cost for each ward type (Supplementary Table S1).26 The cost for the sepsis prediction algorithm was based on the cost per patient divided by the incidence of sepsis resulting in the cost to detect 1 sepsis patient with the technology.
Calculations were conducted on a per-patient level (average cost per patient) and on a country level (total potential cost impact for Sweden). Both deterministic and stochastic analyses were conducted. The stochastic analyses were based on 1000 simulations using information about each variable’s mean, standard error, and statistical distribution (eg, γ, β, log-normal); details are provided in Table 2 and Figure 2.27 The cost calculations were conducted using the euro as the common currency, as it facilitates cross-country comparisons (1 Sk = €0.098 using the exchange rate as of December 14, 2020). We used the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist when writing our report.28
Deterministic Base Case Analysis
Using the methodology described above, we quantified the potential financial and medical advantages using a predictive algorithm such as NAVOY® Sepsis over the current clinical practice for sepsis diagnosis in Sweden. The results are illustrated in Table 3.29,30 From the model, we calculated average total costs for both diagnostic procedures and incremental costs (cost savings) for an ICU patient. The aggregated cost savings were derived by multiplying the average results with the yearly number of patients admitted to ICUs. In Sweden, the yearly attendance to ICUs is estimated to be 45 000 patients.21 Of these, approximately 36 900 patients do not have sepsis upon admission.2 Setting the cost per patient per ICU day with the sepsis prediction algorithm to €90, for example, would render a per-patient cost of €146, considering an average of 1.62 days in the ICU and €1037 per sepsis patient detected (average days in the ICU divided by the sepsis incidence).
Under these assumptions, a machine learning algorithm that can detect sepsis 3 hours before current practice will reduce the cost per ICU patient by 0.5%. The total cost per patient with such an algorithm is €16 436, and the cost per patient for current practice is €16 512. The potential cost savings per patient is thus €76, and the aggregated yearly cost saving for the Swedish healthcare system is €2 798 915.
The largest cost savings are due to a shorter average length of stay in the ICU (0.16 days shorter for an algorithm like NAVOY® Sepsis compared with current practice), resulting in a cost saving of 8.9% (€10 322 vs €11 331) per patient related to ICU hospitalization. The shorter length of stay in ICU for the sepsis prediction algorithm compared with current practice results in 5860 fewer ICU days per year on an aggregated national level. In addition to the reductions in resources used, faster detection also implies reduced in-hospital mortality, resulting in 356 lives saved per year in Sweden alone.
Deterministic Scenario Analyses
Scenario analyses were run for the categories of model parameters presented in Table 4. The most sensitive parameters are the incidence, sensitivity, and specificity inputs and the probability of septic shock with the prediction algorithm. Aggregated cost savings would be about 11 times greater (1033%) if the ICU sepsis incidence is 30% (compared with 14.1% in the model base case). Assuming a sensitivity and specificity for the comparator based on NEWS2, as presented by Mellhammar et al30 (sensitivity 0.84, specificity 0.37), there is an aggregated yearly cost saving of €84 956 039, which constitutes approximately a 30-fold increase (2935%) compared with the base case (current practice). If the proportion of septic shock is set to 30% (40% in model base case) when sepsis detection is set 3 hours earlier, the cost savings would be 7 times greater (603%). Assuming that the proportion of septic shock is time independent and the same as for current practice, no cost savings would be achieved. The model is also sensitive to the time to sepsis detection. If the time to sepsis detection for the prediction algorithm is set 1 hour earlier (4 hours instead of 3 hours), the cost savings would be 4 to 5 times greater (329%). If the time to sepsis detection for the prediction algorithm is less than 2 hours earlier, no cost savings would be achieved.
Considering that sepsis detection in clinical practice does not concur with the time as the Sepsis-3 criteria are fulfilled, there is most likely a gap between the earliest time when the diagnosis is possible to establish (in accordance with the criteria) and the time when the diagnosis is established in practice, and this gap will add to the total time to treatment and thus increase the cost savings. In other words, the potential effect the prediction algorithm could have on time to treatment is most likely underestimated. The sensitivity analyses also show the results to be sensitive to changes in the cost and length of stay in ICU and hospital ward. Increasing the length of stay and cost in ICU and hospital ward by 25% would increase the cost savings by 377%. Decreasing the length of stay and cost in ICU and hospital ward by 25% would result in no cost savings.
Stochastic Sensitivity Analysis
To investigate how the cost estimates for the different scenarios were distributed, we conducted stochastic analyses based on 1000 individual simulations for the 11 most influential parameters, including the base case (Supplementary Table S2). When conducting the stochastic analysis, we kept each individual parameter of interest constant at its mean and allowed all other parameters to vary stochastically. By doing this, we were able to investigate the impact the parameter had on the overall uncertainty (vs the base case). We also estimated the incremental cost-effectiveness ratios (ICER = [Total Cost (NAVOY® Sepsis) – Total Cost (clinical practice)]/[QALYs (NAVOY® Sepsis) – QALYs (standard of care)]) and evaluated the percentage of the simulations below a cost per quality-adjusted life-year (QALY) threshold of €20 000 and €50 000, respectively. The results are detailed in Figure 2 as box plots and as descriptive statistics.
Figure 2 shows that there is cost saving per patient in all scenarios except when (1) sepsis accounts for less than 10% of all ICU admissions without sepsis at admission, (2) detection of sepsis is only 1 hour faster with NAVOY® Sepsis vs current clinical practice, (3) sensitivity is increased to 84% (NEWS2) and specificity is reduced to 78.5% (SOFA), (4) the prevalence of septic shock is the same (56.6%) despite 3 hours earlier detection, and (5) the unit costs for ICU and hospital ward is reduced by 25%. There are 2 scenarios in which NAVOY® Sepsis is unattractive from a cost-effectiveness perspective (>€20 000/QALY): when detection is only 1 hour faster and when it is assumed that faster detection does not impact the prevalence of septic shock.
The consequence of a test’s specificity is clearly illustrated in Figure 2; when the specificity of the comparator in clinical practice is reduced from 78.5% (Sepsis-3 criteria) to 48% (SOFA) and to 37% (NEWS2), the cost saving increased from €76 to €2170 and €2283, respectively. Likewise, if the specificity of NAVOY® Sepsis is reduced from 85.1% to 78.5% (Sepsis-3), the cost saving per patient of €76 will instead result in a cost increase of €287 per patient.
In this study, the potential cost savings and cost-effectiveness achieved by a machine learning algorithm forecasting the onset of sepsis were calculated using a novel health economic model developed for this purpose. The model shows that cost savings can be achieved for the healthcare sector in terms of factors such as reduced in-hospital mortality and shorter ICU length of stay. The model base case, with conservative assumptions about sepsis incidence and time to treatment, shows that by using a sepsis prediction algorithm that can detect sepsis 3 hours earlier than current practice, the average ICU length of stay can be reduced by 0.16 days. This might sound like a small amount, but it aggregates to a total of 5860 fewer ICU days per year on a national level. In addition to reducing costs, this would reduce the ICU occupancy rate and perhaps relieve some of the stress for the ICU staff. The base case model result also shows a potential of saving 356 lives per year in Sweden, almost 1 life per day. This can be compared to approximately 200 people in Sweden being killed in traffic each year.31 The Swedish Transport Agency has estimated the value of a “statistical life” to approximately €4 million,32 which is equivalent to a value of a quality-adjusted life-year (QALY) of €235 200.33
Strengths and Limitations
The results are sensitive to the sepsis incidence, which, as discussed by Bray et al,16 depends, among other things, on how sepsis is diagnosed in the clinic. A broad definition of sepsis leads to a higher incidence, and a prediction algorithm would thus have a larger potential of reducing costs in situations where a broader definition is used. Internationally, the Sepsis-3 criteria11 are acknowledged as the gold standard when diagnosing sepsis, but these criteria are not widely used in practice. Lengquist et al19 showed that sepsis is underdiagnosed in Swedish ICUs, indicating that a sepsis prediction algorithm has a high potential of contributing to identifying patients with sepsis. Further, patients admitted with sepsis are not included in the model even though they can develop a secondary sepsis infection during the ICU stay.17 This means that a sepsis prediction algorithm would have a potential of reducing costs also for this group of patients, adding to the total cost savings presented in this study. The results are also sensitive to the time to diagnosis, which in turn affects time to treatment. Since sepsis is underdiagnosed in practice,19 it is reasonable to assume that a sepsis prediction algorithm has the potential to substantially shorten the time to treatment compared with current practice. Also, the algorithm was developed on a large database of critically ill patients. The model base case was set to be conservative in relation to both incidence and time to diagnosis, to avoid exaggerating the positive effects of the use of a sepsis prediction algorithm like NAVOY® Sepsis.
Whereas previous research has primarily focused on interventions for sepsis treatment,34 our study is the first that has investigated the potential cost impact and cost-effectiveness of implementing a machine learning algorithm for earlier detection of sepsis. Most of the studies reviewed by Higgins et al34 were cost-effectiveness studies with a positive incremental cost-effectiveness ratio (extra benefits at extra costs), indicating that interventions for sepsis rarely are cost saving. In the present study, we also assessed the probabilistic cost-effectiveness of the machine learning algorithm using various alternative assumptions regarding sepsis incidence rates, rates of sensitivity and specificity, hours of faster sepsis detection, prevalence of septic shock, and unit costs of an ICU stay. Generally, we found that NAVOY® Sepsis was a dominant alternative (cost saving and more effective) compared with current clinical practice. We found that NAVOY® Sepsis was not cost-effective (threshold >€20 000) in 2 of the 11 tested scenarios; when the detection was only 1 hour faster instead of 3 hours, and when the prevalence of septic shock at 3-hour faster detection was 56.6% instead of 40%. Since the former assumption must be regarded as highly conservative and the latter is unrealistic, as it implies no gains from earlier detection, we regard these results as indicators that NAVOY® Sepsis has a high probability of being a cost-effective technology.
This study shows that future clinical studies of prediction algorithms such as NAVOY® Sepsis need to be powered to provide stable results of specificity. The reason is that “false alarms” have an impact on ICU days and thereby unnecessary ICU resources for false-positive patients. Running model simulations of a 78.5% specificity instead of 85.1% reduced the cost offsets per patient for ICU from €650 to €1009. Thus, a more specific instrument is more important than a more sensitive one as it has a greater impact on resource consumption. The base case for the model is based on international data sources (eg, Ferrer et al10 and Sakr et al2), which might not be directly applicable to a Swedish setting. This fact is accounted for in the sensitivity analyses, where alternative scenarios for parameters that could differ between countries (eg, incidence, length of stay, in-hospital mortality, unit costs) were accounted for. Our analysis indicates that different countries’ definitions of sepsis could have a substantial impact on incidence and thereby the cost-saving potential of introducing an algorithm, whereas differences in unit costs will have a minor impact. Also, the results of this study apply to Sweden, a relatively small, technologically advanced country. The benefits of a sepsis prediction algorithm would naturally apply to countries other than Sweden but on a larger or smaller scale, depending on the population size. The implementation of a prediction algorithm, however, would require systems for electronic data capture, which is not yet available in all countries.
Our health economic model shows that a machine learning prediction algorithm that can find septic patients 3 hours earlier than what is possible today can save lives, reduce time spent in the ICU, and reduce costs for society and the healthcare sector. The value of saving over 350 lives adds up to a total of €1.4 billion, based on the Swedish Transport Agency’s estimate of a statistical life.32 Internationally, the benefits of a sepsis prediction algorithm would be similar to those for Sweden but on a larger or smaller scale, depending on population size.
The sensitivity analyses show that the prevalence of septic shock has a large impact on the results, and there is little found in the literature on the possibility of reducing the septic shock rate by early detection of sepsis. Further research on the relationship between the key events in the management of sepsis, such as time of sepsis detection in clinical practice, time of fulfillment of Sepsis-3 criteria, time to treatment, and development of septic shock, would be valuable to develop an even more granular model. Future research should also focus on patient categories with high levels of inflammation, such as burns, trauma, and pancreatitis, where sepsis is known to be very difficult to diagnose and where a prediction algorithm could be of immense value.
The Swedish Institute for Health Economics (IHE), where OE and JH are employed, has received compensation from AlgoDx AB to conduct the research upon which this manuscript is based. AlgoDx AB is also sponsoring an ongoing study for prospective evaluation of the algorithm used in this manuscript, and FS’s institution is reimbursed for costs related to performing that study. IP and JS are shareholders of AlgoDx AB.
OE designed the model together with JH, performed the computations, drafted the manuscript, discussed the results, and contributed to the final manuscript. JH designed the model together with OE, drafted the manuscript, discussed the results, and contributed to the final manuscript. FS was involved in discussions and interpretations, drafted parts of the manuscript, discussed the results, and contributed to the final manuscript. IP was involved in discussions and interpretations, supervised the work, drafted the manuscript, discussed the results, and contributed to the final manuscript. JS conceived the idea, participated in planning and design of the study, supervised the work, discussed the results, and contributed to the final manuscript.
The research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The research and writing of the manuscript were funded by AlgoDx AB.