Randomized comparisons to establish the relative safety or tolerability of many competing interventions are scarce. In the presence of such a paucity of evidence, an approach to estimating relative treatment effects in the absence of head-to-head trials is to utilize an “adjusted” indirect comparisons approach. However, these methods are underused and difficult to understand. Formal statistical descriptions of this approach have been given elsewhere [1] but below we provide a non-technical account using analogies to predicting results in sports matches, which we hope makes the ideas accessible to those who may feel intimidated by technical descriptions. In addition, we illustrate how an indirect comparison is performed using a clinical example.

Many readers may have considered an indirect comparison when trying to predict the outcome in an event such as a soccer tournament and may have been among the approx 30,000,000 people who watched the 2010 Soccer World Cup, which we will now revisit. Let’s assume that we want to predict the last game of group E – a match between the Netherlands and Cameroon. What we do instinctively is find a common comparator, e.g., a team that both the Netherlands and Cameroon have competed against recently, and compare the outcomes. Since both Cameroon and the Netherlands have played against Denmark, we can predict which team would likely to be a winner based on the scores of their games against Denmark (Fig. 1). Cameroon lost to Denmark and Denmark lost to the Netherlands, so we predict the Netherlands will win and, in this case our prediction was correct. The Netherlands beat Cameroon 2 to1.

Fig. (1) To predict the match between the Netherlands and Cameron we can use the scores of the previous games between Cameroon and Denmark and Netherlands and Denmark.

Many factors can compromise validity of indirect comparisons of health interventions. First, an indirect comparison must respect the randomization to ensure balance in patient characteristics across groups. Comparing results of similar treatment arms from different clinical trials is strongly discouraged and can produce biased and overly precise estimates [2] since this does not respect randomization. Hence, comparisons should be based on treatment differences within each randomized trial. Second, the results of trials used in an indirect comparison should be homogeneous. In particular, all factors that affect the response to a treatment, i.e. effect modifiers, must have a similar distribution across the entire set of trials that are being compared. This implies that all considered trials should have similar study designs, populations and outcomes [2, 3].

In the above soccer prediction, we based the analysis on the scores of each of the matches, which is analogous to respecting the randomization in a clinical trial setting (i.e., we did not count all victories by Cameroon and all victories by the Netherlands). Since the Netherlands played Denmark only once and similarly, Cameroon played Denmark only once, we cannot evaluate the homogeneity of the results (which we can do if there are multiple trials of the same comparisons). However, both the Netherlands and Cameroon played Denmark under similar circumstances. The games were very close in calendar time, weather was similar, and although the matches were played in different cities, none of the teams had a home advantage. These are the types of assessments that are required to address the validity of the indirect comparisons [2].

Indirect comparisons in health care are not performed commonly despite all of their potential benefits, as systematic reviews and meta-analyses often focus on trials that compare one intervention with placebo. We hypothesized that if we could show that indirect comparisons were useful in predicting soccer winners, despite all the difficulties involved in such predictions, it might popularize the technique and increase acceptability and use of indirect comparisons and similar extended methodologies. With such an objective in mind we embark on a further exploration of the analogy.

We decided to use the first round results from the soccer world cups of 2006 and 2010, which are readily available from the official FIFA website [4]. We restricted our attention to the first round because the rules of the competition were suitable for this exercise. In the first round, teams are divided into groups of four teams, each team in a group plays once against each other team in that group, and ties are permitted. Of the six games played in each group, we predicted the last two games (Fig. 2). Since the earlier games can provide more than one common comparator, two predictions can be made about the same match. For example, the prediction of the Netherlands and Cameroon match can be based on how both teams fared against Japan or against Denmark. We elected to combine the two predications by averaging them.

Fig. (2) Network of teams and matches in Group E of the 2010 World Soccer Cup. The line connecting the teams represents the games played.

This is an illustration of how we did the comparisons. To obtain the first prediction, we considered that the Netherlands beat Japan by one goal and Cameroon lost to Japan by one goal. Based on this indirect comparison, we assigned a score of 1- (-1) = 2 to the match between the Netherlands and Cameroon. For the second prediction, we considered that the Netherlands beat Denmark by two goals and Cameroon lost to Denmark by one goal. Hence, we assigned to the match between the Netherlands and Cameron a score of 2 - (- 1) =3. After combining the two predictions by averaging, we predicted Netherlands will win.

Overall, we made 32 predictions in a similar fashion and found that the agreement between the predictions and the actual results was 59.4% while the expected agreement by chance was 37.3% (i.e., assuming no correlation between results). This result gives us a kappa of 0.35 and a p-value of 0.0025 (Table 1), which suggests a fair agreement between our predictions and the actual results.

Table 1

Agreement Among Indirect Comparison Predictions and Observed Match Results

Clinical Example

Let’s assume that we want to compare the tolerability of gabapentin, an anticonvulsant, and nortriptyline, an antidepressant. These two medications are commonly used for treatment of neuropathic pain. We arbitrarily selected two randomized parallel studies for this comparison. The first study evaluated gabapentin and placebo [5]; the second one evaluated nortriptyline and placebo [6]. The number of subjects in each treatment arm and the number of subjects who withdrew from the studies due to adverse events can be seen in Table 2.

Table 2

Description of Direct Comparison Studies Included in Clinical Example and Risk of Withdrawing Due to Adverse Events

To assess the tolerability, we calculated the risk of withdrawing from the studies due to adverse events in each arm and then the risk difference in each trial. To determine how gabapentin compares with nortriptyline, all that is needed is to subtract the risks from the two placebo trials (Table 3). This indirect estimate suggests that the risk of discontinuing medication because of the adverse events is 1.1% lower for nortriptyline than for gabapentin, but the difference is not statistically significant.

Table 3

Direct and Indirect Comparison Results

In the clinical setting in addition to the placebo control trials there could be head-to-head trials. In this circumstance one can assess the agreement between the direct and indirect estimates and even combined the indirect and indirect estimates. We found a randomized trial that directly compared the tolerability of nortriptyline and gabapentin [7]. The doses of nortriptyline and gabapentin were similar to the doses used in the placebo controlled trials and the duration of the follow up was similar as well as. This head-to-head study found that the risk of discontinuing medication because of adverse events was 2.6% lower with gabapentin, but it was not statistically significant, Table 2.

The direction of the indirect estimate is in opposite direction to the estimate of the head-to-head trial, but the difference between these two estimates is small (3.7%) and not statistically significant (Table 3). Therefore, we consider both the direct and the indirect estimates to be in agreement.

It is crucial to emphasize that a proper evaluation of the safety of two treatments should be based on a systematic search of the literature and not on an arbitrary selection of studies. The purpose of this exercise is only to illustrate how the indirect comparison is performed and not to conclude that one medication has a better tolerability profile than another.

Predicting individual soccer results is notoriously difficult [8, 9]. Although, the strength and skills of the team undoubtedly play a role, other variables such as the play of chance, team spirit, personal problems of the players and arbitration decisions (referees are inconsistent and could be biased) have an important role and are difficult to measure or to predict [10]. The low scores typically associated with a soccer match mean the play of chance has a more pronounced effect than in other, high-scoring, sports such as basketball [11]. Therefore, it is not surprising that the observed agreement in predictions of such capricious outcomes as world cup soccer, was lower than the agreement between direct and indirect evidence in the health field, which has been found to be good or very good [12-14].

You pay a price when you perform indirect comparisons, indirect estimates are less precise than direct estimates [1]. This can be appreciated from Table 3, in which the confidence interval of the indirect comparison estimate is much wider than the confidence interval of the head-to-head estimate.

Often, in the context of comparing health interventions, more than a pairwise comparison is needed. For example, suppose you want to compare the safety profile of all the medications for the treatment of neuropathic pain on the market against each other. In this case, extended meta-analytic methods are used. These methods include mixed treatment comparisons, network meta-analysis and a multiple treatment meta-analysis.

These techniques permit performing simultaneous comparisons of all treatments in the network and often use a bayesian framework, not because bayesian input (prior information) is needed, (in fact the priors are specified as non–informative), but because of the flexibility of the software used to perform bayesian analyses. The multiple-treatment comparison techniques can deal with trials that have multiple arms by accounting for the correlation due to those multiple arms; they can combine direct and indirect evidence and assess the inconsistency - the disagreement between the direct and indirect evidence; and they can also provide a probabilistic ranking of treatments [15-17]. The threats to the validity of the estimates in this scenario apply to each one of the indirect estimates, so the more complex the network, the more room there is for biased predictions. Efforts to facilitate access to these methods have been made by creating software that not only is friendlier, but that also allows a more efficient and transparent way to check the results of such analyses [18].

We illustrated the method of adjusted indirect comparisons by applying it to a topic of interest to many readers, namely the prediction of soccer match outcomes and to a clinical example. We found that even in circumstances in which the signal, i.e. strength of the team, can easily be diluted or even lost, there is an agreement between the predictions of the indirect comparisons and the actual results. We hope that this example will increase the acceptability of indirect comparisons and extended methodologies and widen the basis of health care decisions to a broader range of evidence instead of just the evidence from direct placebo-based comparisons.

For those who may be inclined to interpret this paper as suggesting a betting system rather than simply giving an accessible illustration of a methodology relevant to comparisons of health interventions, we suggest reading some additional references before betting [19].

MSC, DF, and VL were responsible for study concept, study design, data collection, data interpretation, and preparation of the manuscript.

AS was responsible for data analysis, data interpretation, and preparation of the manuscript.