The Open Drug Safety Journal


ISSN: 1876-8180 ― Volume 4, 2013

Can Soccer Predictions Boost the Use of Indirect Comparison to Assess the Safety Of Drugs?

M Soledad Cepeda*, 1, Daniel Fife1, Victor Lobanov1, Alex Sutton2
1 Johnson & Johnson Pharmaceutical Research and Development, 1125 Trenton Harbourton Rd, Titusville, NJ 08560, USA
2 Department of Health Sciences, University of Leicester, 2nd Floor (Room 214e), Adrian Building, University Road, Leicester LE1 7RH, UK


Randomized comparisons to establish the relative safety of competing interventions are scarce. An approach to estimate relative treatment effects in the absence of head-to-head trials is to utilize indirect comparisons and extended meta-analytic techniques, but these techniques are underused.

We provide a non-technical account using analogies to predicting results in sports matches, which we hope makes these methods more accessible. The ideas in this example may be familiar to readers who may have considered an indirect comparison when trying to predict the outcome of a sports match. We used the first round of the soccer world cups of 2006 and 2010. In addition, we provide a simplified clinical example.

We found that even in circumstances in which the signal, i.e. strength of the team or the relatively safety of the medication, can easily be diluted or even lost, there is a fair agreement between the predictions of the indirect comparisons and the actual results.

We hope that this example will increase the acceptability of indirect comparisons and extended methods and widen the basis of health care decisions to a broader range of evidence instead of just the evidence from direct placebo-based comparisons.

Keywords: Indirect comparison, agreement between direct and indirect estimates, meta-analytic techniques.

Article Information

Identifiers and Pagination:

Year: 2011
Volume: 2
First Page: 25
Last Page: 28
Publisher Id: TODSJ-2-25
DOI: 10.2174/1876818001102010025

Article History:

Received Date: 8/10/2010
Revision Received Date: 19/1/2011
Acceptance Date: 23/1/2011
Electronic publication date: 10/3/2011
Collection year: 2011

© Cepeda et al.; Licensee Bentham Open.

open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License ( which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.

* Address correspondence to this author at the Johnson & Johnson Pharmaceutical Research and Development, 1125 Trenton Harbourton Rd, Titusville, NJ 08560, USA; Tel: 609 730 2413; Ext: 30001; Fax: 609 730 7927; E-mail:


Randomized comparisons to establish the relative safety or tolerability of many competing interventions are scarce. In the presence of such a paucity of evidence, an approach to estimating relative treatment effects in the absence of head-to-head trials is to utilize an “adjusted” indirect comparisons approach. However, these methods are underused and difficult to understand. Formal statistical descriptions of this approach have been given elsewhere [1] but below we provide a non-technical account using analogies to predicting results in sports matches, which we hope makes the ideas accessible to those who may feel intimidated by technical descriptions. In addition, we illustrate how an indirect comparison is performed using a clinical example.

Many readers may have considered an indirect comparison when trying to predict the outcome in an event such as a soccer tournament and may have been among the approx 30,000,000 people who watched the 2010 Soccer World Cup, which we will now revisit. Let’s assume that we want to predict the last game of group E – a match between the Netherlands and Cameroon. What we do instinctively is find a common comparator, e.g., a team that both the Netherlands and Cameroon have competed against recently, and compare the outcomes. Since both Cameroon and the Netherlands have played against Denmark, we can predict which team would likely to be a winner based on the scores of their games against Denmark (Fig. 1). Cameroon lost to Denmark and Denmark lost to the Netherlands, so we predict the Netherlands will win and, in this case our prediction was correct. The Netherlands beat Cameroon 2 to1.

Fig. (1)

To predict the match between the Netherlands and Cameron we can use the scores of the previous games between Cameroon and Denmark and Netherlands and Denmark.

Many factors can compromise validity of indirect comparisons of health interventions. First, an indirect comparison must respect the randomization to ensure balance in patient characteristics across groups. Comparing results of similar treatment arms from different clinical trials is strongly discouraged and can produce biased and overly precise estimates [2] since this does not respect randomization. Hence, comparisons should be based on treatment differences within each randomized trial. Second, the results of trials used in an indirect comparison should be homogeneous. In particular, all factors that affect the response to a treatment, i.e. effect modifiers, must have a similar distribution across the entire set of trials that are being compared. This implies that all considered trials should have similar study designs, populations and outcomes [2, 3].

In the above soccer prediction, we based the analysis on the scores of each of the matches, which is analogous to respecting the randomization in a clinical trial setting (i.e., we did not count all victories by Cameroon and all victories by the Netherlands). Since the Netherlands played Denmark only once and similarly, Cameroon played Denmark only once, we cannot evaluate the homogeneity of the results (which we can do if there are multiple trials of the same comparisons). However, both the Netherlands and Cameroon played Denmark under similar circumstances. The games were very close in calendar time, weather was similar, and although the matches were played in different cities, none of the teams had a home advantage. These are the types of assessments that are required to address the validity of the indirect comparisons [2].

Indirect comparisons in health care are not performed commonly despite all of their potential benefits, as systematic reviews and meta-analyses often focus on trials that compare one intervention with placebo. We hypothesized that if we could show that indirect comparisons were useful in predicting soccer winners, despite all the difficulties involved in such predictions, it might popularize the technique and increase acceptability and use of indirect comparisons and similar extended methodologies. With such an objective in mind we embark on a further exploration of the analogy.


We decided to use the first round results from the soccer world cups of 2006 and 2010, which are readily available from the official FIFA website [4]. We restricted our attention to the first round because the rules of the competition were suitable for this exercise. In the first round, teams are divided into groups of four teams, each team in a group plays once against each other team in that group, and ties are permitted. Of the six games played in each group, we predicted the last two games (Fig. 2). Since the earlier games can provide more than one common comparator, two predictions can be made about the same match. For example, the prediction of the Netherlands and Cameroon match can be based on how both teams fared against Japan or against Denmark. We elected to combine the two predications by averaging them.

Fig. (2)

Network of teams and matches in Group E of the 2010 World Soccer Cup. The line connecting the teams represents the games played.

This is an illustration of how we did the comparisons. To obtain the first prediction, we considered that the Netherlands beat Japan by one goal and Cameroon lost to Japan by one goal. Based on this indirect comparison, we assigned a score of 1- (-1) = 2 to the match between the Netherlands and Cameroon. For the second prediction, we considered that the Netherlands beat Denmark by two goals and Cameroon lost to Denmark by one goal. Hence, we assigned to the match between the Netherlands and Cameron a score of 2 - (- 1) =3. After combining the two predictions by averaging, we predicted Netherlands will win.


Overall, we made 32 predictions in a similar fashion and found that the agreement between the predictions and the actual results was 59.4% while the expected agreement by chance was 37.3% (i.e., assuming no correlation between results). This result gives us a kappa of 0.35 and a p-value of 0.0025 (Table 1), which suggests a fair agreement between our predictions and the actual results.

Table 1

Agreement Among Indirect Comparison Predictions and Observed Match Results

Clinical Example

Let’s assume that we want to compare the tolerability of gabapentin, an anticonvulsant, and nortriptyline, an antidepressant. These two medications are commonly used for treatment of neuropathic pain. We arbitrarily selected two randomized parallel studies for this comparison. The first study evaluated gabapentin and placebo [5]; the second one evaluated nortriptyline and placebo [6]. The number of subjects in each treatment arm and the number of subjects who withdrew from the studies due to adverse events can be seen in Table 2.

Table 2

Description of Direct Comparison Studies Included in Clinical Example and Risk of Withdrawing Due to Adverse Events

To assess the tolerability, we calculated the risk of withdrawing from the studies due to adverse events in each arm and then the risk difference in each trial. To determine how gabapentin compares with nortriptyline, all that is needed is to subtract the risks from the two placebo trials (Table 3). This indirect estimate suggests that the risk of discontinuing medication because of the adverse events is 1.1% lower for nortriptyline than for gabapentin, but the difference is not statistically significant.

Table 3

Direct and Indirect Comparison Results

In the clinical setting in addition to the placebo control trials there could be head-to-head trials. In this circumstance one can assess the agreement between the direct and indirect estimates and even combined the indirect and indirect estimates. We found a randomized trial that directly compared the tolerability of nortriptyline and gabapentin [7]. The doses of nortriptyline and gabapentin were similar to the doses used in the placebo controlled trials and the duration of the follow up was similar as well as. This head-to-head study found that the risk of discontinuing medication because of adverse events was 2.6% lower with gabapentin, but it was not statistically significant, Table 2.

The direction of the indirect estimate is in opposite direction to the estimate of the head-to-head trial, but the difference between these two estimates is small (3.7%) and not statistically significant (Table 3). Therefore, we consider both the direct and the indirect estimates to be in agreement.

It is crucial to emphasize that a proper evaluation of the safety of two treatments should be based on a systematic search of the literature and not on an arbitrary selection of studies. The purpose of this exercise is only to illustrate how the indirect comparison is performed and not to conclude that one medication has a better tolerability profile than another.


Predicting individual soccer results is notoriously difficult [8, 9]. Although, the strength and skills of the team undoubtedly play a role, other variables such as the play of chance, team spirit, personal problems of the players and arbitration decisions (referees are inconsistent and could be biased) have an important role and are difficult to measure or to predict [10]. The low scores typically associated with a soccer match mean the play of chance has a more pronounced effect than in other, high-scoring, sports such as basketball [11]. Therefore, it is not surprising that the observed agreement in predictions of such capricious outcomes as world cup soccer, was lower than the agreement between direct and indirect evidence in the health field, which has been found to be good or very good [12-14].

You pay a price when you perform indirect comparisons, indirect estimates are less precise than direct estimates [1]. This can be appreciated from Table 3, in which the confidence interval of the indirect comparison estimate is much wider than the confidence interval of the head-to-head estimate.

Often, in the context of comparing health interventions, more than a pairwise comparison is needed. For example, suppose you want to compare the safety profile of all the medications for the treatment of neuropathic pain on the market against each other. In this case, extended meta-analytic methods are used. These methods include mixed treatment comparisons, network meta-analysis and a multiple treatment meta-analysis.

These techniques permit performing simultaneous comparisons of all treatments in the network and often use a bayesian framework, not because bayesian input (prior information) is needed, (in fact the priors are specified as non–informative), but because of the flexibility of the software used to perform bayesian analyses. The multiple-treatment comparison techniques can deal with trials that have multiple arms by accounting for the correlation due to those multiple arms; they can combine direct and indirect evidence and assess the inconsistency - the disagreement between the direct and indirect evidence; and they can also provide a probabilistic ranking of treatments [15-17]. The threats to the validity of the estimates in this scenario apply to each one of the indirect estimates, so the more complex the network, the more room there is for biased predictions. Efforts to facilitate access to these methods have been made by creating software that not only is friendlier, but that also allows a more efficient and transparent way to check the results of such analyses [18].


We illustrated the method of adjusted indirect comparisons by applying it to a topic of interest to many readers, namely the prediction of soccer match outcomes and to a clinical example. We found that even in circumstances in which the signal, i.e. strength of the team, can easily be diluted or even lost, there is an agreement between the predictions of the indirect comparisons and the actual results. We hope that this example will increase the acceptability of indirect comparisons and extended methodologies and widen the basis of health care decisions to a broader range of evidence instead of just the evidence from direct placebo-based comparisons.

For those who may be inclined to interpret this paper as suggesting a betting system rather than simply giving an accessible illustration of a methodology relevant to comparisons of health interventions, we suggest reading some additional references before betting [19].


MSC, DF, and VL were responsible for study concept, study design, data collection, data interpretation, and preparation of the manuscript.

AS was responsible for data analysis, data interpretation, and preparation of the manuscript.


M. Soledad Cepeda, Daniel Fife, Victor Lobanov are employees of Johnson & Johnson Pharmaceutical Research & Development.

Alex Sutton works in the Department of Health Sciences, University of Leicester.


Track Your Manuscript:


"Open access will revolutionize 21st century knowledge work and accelerate the diffusion of ideas and evidence that support just in time learning and the evolution of thinking in a number of disciplines."

Daniel Pesut
(Indiana University School of Nursing, USA)

"It is important that students and researchers from all over the world can have easy access to relevant, high-standard and timely scientific information. This is exactly what Open Access Journals provide and this is the reason why I support this endeavor."

Jacques Descotes
(Centre Antipoison-Centre de Pharmacovigilance, France)

"Publishing research articles is the key for future scientific progress. Open Access publishing is therefore of utmost importance for wider dissemination of information, and will help serving the best interest of the scientific community."

Patrice Talaga
(UCB S.A., Belgium)

"Open access journals are a novel concept in the medical literature. They offer accessible information to a wide variety of individuals, including physicians, medical students, clinical investigators, and the general public. They are an outstanding source of medical and scientific information."

Jeffrey M. Weinberg
(St. Luke's-Roosevelt Hospital Center, USA)

"Open access journals are extremely useful for graduate students, investigators and all other interested persons to read important scientific articles and subscribe scientific journals. Indeed, the research articles span a wide range of area and of high quality. This is specially a must for researchers belonging to institutions with limited library facility and funding to subscribe scientific journals."

Debomoy K. Lahiri
(Indiana University School of Medicine, USA)

"Open access journals represent a major break-through in publishing. They provide easy access to the latest research on a wide variety of issues. Relevant and timely articles are made available in a fraction of the time taken by more conventional publishers. Articles are of uniformly high quality and written by the world's leading authorities."

Robert Looney
(Naval Postgraduate School, USA)

"Open access journals have transformed the way scientific data is published and disseminated: particularly, whilst ensuring a high quality standard and transparency in the editorial process, they have increased the access to the scientific literature by those researchers that have limited library support or that are working on small budgets."

Richard Reithinger
(Westat, USA)

"Not only do open access journals greatly improve the access to high quality information for scientists in the developing world, it also provides extra exposure for our papers."

J. Ferwerda
(University of Oxford, UK)

"Open Access 'Chemistry' Journals allow the dissemination of knowledge at your finger tips without paying for the scientific content."

Sean L. Kitson
(Almac Sciences, Northern Ireland)

"In principle, all scientific journals should have open access, as should be science itself. Open access journals are very helpful for students, researchers and the general public including people from institutions which do not have library or cannot afford to subscribe scientific journals. The articles are high standard and cover a wide area."

Hubert Wolterbeek
(Delft University of Technology, The Netherlands)

"The widest possible diffusion of information is critical for the advancement of science. In this perspective, open access journals are instrumental in fostering researches and achievements."

Alessandro Laviano
(Sapienza - University of Rome, Italy)

"Open access journals are very useful for all scientists as they can have quick information in the different fields of science."

Philippe Hernigou
(Paris University, France)

"There are many scientists who can not afford the rather expensive subscriptions to scientific journals. Open access journals offer a good alternative for free access to good quality scientific information."

Fidel Toldrá
(Instituto de Agroquimica y Tecnologia de Alimentos, Spain)

"Open access journals have become a fundamental tool for students, researchers, patients and the general public. Many people from institutions which do not have library or cannot afford to subscribe scientific journals benefit of them on a daily basis. The articles are among the best and cover most scientific areas."

M. Bendandi
(University Clinic of Navarre, Spain)

"These journals provide researchers with a platform for rapid, open access scientific communication. The articles are of high quality and broad scope."

Peter Chiba
(University of Vienna, Austria)

"Open access journals are probably one of the most important contributions to promote and diffuse science worldwide."

Jaime Sampaio
(University of Trás-os-Montes e Alto Douro, Portugal)

"Open access journals make up a new and rather revolutionary way to scientific publication. This option opens several quite interesting possibilities to disseminate openly and freely new knowledge and even to facilitate interpersonal communication among scientists."

Eduardo A. Castro
(INIFTA, Argentina)

"Open access journals are freely available online throughout the world, for you to read, download, copy, distribute, and use. The articles published in the open access journals are high quality and cover a wide range of fields."

Kenji Hashimoto
(Chiba University, Japan)

"Open Access journals offer an innovative and efficient way of publication for academics and professionals in a wide range of disciplines. The papers published are of high quality after rigorous peer review and they are Indexed in: major international databases. I read Open Access journals to keep abreast of the recent development in my field of study."

Daniel Shek
(Chinese University of Hong Kong, Hong Kong)

"It is a modern trend for publishers to establish open access journals. Researchers, faculty members, and students will be greatly benefited by the new journals of Bentham Science Publishers Ltd. in this category."

Jih Ru Hwu
(National Central University, Taiwan)

Browse Contents

Webmaster Contact:
Copyright © 2021 Bentham Open