Clinical Trials, Economic Evaluation and Simulation

- supporting the economic evaluation of a clinical trial by simulation

by Simon Taylor, Tillal Eldabi and Ray Paul

The aim of the economic evaluation of a clinical trial is to assess the impact of new forms of treatment across different patient groups. A major problem of economic evaluation is the volume of data that can be potentially collected from a clinical trial. Limitations on resources for this task means that health economists must make informed decisions as to what data to collect so that they can successfully perform their evaluation. This article discusses how simulation can support this kind of decision making by helping health economists develop a deeper understanding of the relationships involved in a clinical trial.

-oo0oo-

The adjuvant breast cancer trial

Tamoxifen is a drug in pill form that interferes with the activity of oestrogen. It has been used for nearly twenty years to treat patients with advanced breast cancer. More recently it has also been used as adjuvant (additional) treatment following primary treatment for early stage breast cancer. However, it is unclear whether or not this treatment may be effectively augmented with other forms of adjuvant therapy such as chemotherapy and ovarian suppression. To make this assessment a clinical trial called the Adjuvant Breast Cancer (ABC) Trial was set up in the United Kingdom with the goal of investigating the relationship between these different forms of adjuvant therapy. The ABC Trial involved around six thousand patients from representative groups and fourteen different possible treatments. The Trial also involved a group of clinicians (to perform the trial) and a group of health economists from the Health Economics Research Group (HERG) at Brunel University (to perform the economic evaluation).

The main objective of adjuvant therapy for early breast cancer is to prolong survival while maintaining a high quality of life. The principle aim of the Trial was to assess the value of combining alternative forms of adjuvant therapy for early breast cancer. More formally the Trial aims to assess the value of adding cytotoxic chemotherapy and/or ovarian suppression to prolonged adjuvant tamoxifen in order to treat pre/perimenopausal women with early breast cancer, and cytotoxic chemotherapy to prolonged adjuvant tamoxifen in order to treat post menopausal women with early breast cancer. The possible treatment alternatives are listed in Table 1. Clinicians assess patients according to menopausal state and initial treatment suitability. Patients who are pre/peri menopausal may be initially treated with either tamoxifen, tamoxifen with chemotherapy, or tamoxifen with ovarian suppression. Patients who are post menopausal may only be initially treated with tamoxifen. To investigate the effect of further treatment, clinicians have specified treatment plans that are randomly selected based on initial treatment. For example, a pre/perimenopausal patient who has been initially treated with tamoxifen and chemotherapy may be randomised to having either subsequent treatment with ovarian suppression or no subsequent treatment. The Trial aimed to include four thousand pre/perimenopausal and two thousand postmenopausal women. The clinical end-points of the trial are overall, and relapse-free survival, for five years.

To perform assessment in an adjuvant therapy clinical trial, information must be gathered concerning the perceived quality of life of a patient during a particular health state (pre/post treatment for example). This is translated into a quality of life utility weighting which makes it possible for health economists to assess the different available treatments in a clinical trial on the basis of quality adjusted life years (QALY) and life years gained, as well as additional cost per QALY gained and the additional cost per life year gained.

The acquisition of data on perceived quality of life is typically performed by health economists interviewing patients who have received treatment. Given that the data collection resources available to the health economists are limited, the acquisition of quality of life data for every patient experiencing every health state is unrealistic. Decisions must be made as to which patients to interview and when they should be interviewed. How, given that much of the quality of life data is unknown, can health economists decide which patients to interview?

Operational research and quality of life

A typical operational research study is motivated by the search for a solution to a problem. One example might be How many cashiers should I expect to have on duty during lunch time in my bank? The study is then performed by modelling what is known about the bank and incrementally experimenting with the model to discover more about the sensitivity of relationships between factors within the bank system (customers, cashiers, etc). Results from the study are considered with regard to the real system and the number of cashiers are set as appropriate. Another possible result from the study is that the manager has a better understanding of how his or her bank operates after being involved in the development of a model of the bank and in analysis of experimental results. One might argue on the basis of this that a deliverable from a typical operational research study is better decision making through a deeper understanding of a problem.

In the case of the ABC Trial the problem is one of attempting to predict what quality of life data should be collected to evaluate the effectiveness of the treatments of the ABC Trial. If the adjuvant treatment of breast cancer is the system and the problem is one of attempting to assess the relationships between the factors in the Trial and the quality of life for the various patient cohorts, then one might take an operational research approach to the problem.

The rest of this article discusses the experiences, and the software, that resulted from taking this view of this problem.

The development of ABCSim

The decision to model the economic factors of the Trial came from an initial meeting between the health economists from HERG and members of the Centre for Applied Simulation Modelling, also from Brunel University. A review of previous attempts to predict patient survival in adjuvant treatment clinical trials indicated that either a critical review (non-modelling) approach, decision trees, or markovian analysis had been used. Discrete event simulation was chosen over these for the following reasons. There are clear definitions of disease and its progression, definable benefits and risks with different modes of therapy, and competing alternative approaches. Events can be clearly identified. These include the administration of the adjuvant treatments, their adverse effects, recurrence (including local, regional and metastatic relapse), and survival. The likelihood of these events are influenced by particular treatments and the overall effect of each event can be influenced by concurrent aspects of complicated cost and quality of life factors. For example, a patient may still be suffering from the symptoms of toxicity associated with chemotherapy when she experiences a recurrence. The toxicity of the chemotherapy could effect the severity of the effects of cancer recurrence. Simulation also makes possible the simple identification and tracking of various patient groups, a requirement made by the health economists to support their analysis. The technique was therefore chosen as each of the other alternatives had some or other deficiency in terms of the above points.

The main role of simulation in this project was therefore identified as modelling the perceived relationships between economic factors in the treatment of adjuvant breast cancer while providing a suitable experimental platform from which the health economists could investigate relationships between factors active in the ABC Trial. The health economists performing the analysis of the trial data therefore required a computer model of the Trials economic factors that could be easily used to support understanding as to how quality of life might be influenced by the ABC Trial factors. However, as representation of the Trial factors involved up to a thousand input variables and the health economists were considered to be experts in their own field and not simulation, it was decided that the simulation model would be augmented by a user interface that facilitated data entry and output analysis. The result of this was the ABCSim package.

Model development

The development of the simulation model that was the heart of the ABCSim package involved regular meetings between health economists and simulationists to develop a conceptual model. As previously mentioned, the identifiable sequences of events that patients experienced aided the modelling process. Effectively these were the initial treatment, the selection and application of subsequent treatment, and then possible recurrence of cancer. To begin modelling this system, simple flow diagrams were used rather than a computer simulation package. It had been the experience of the simulation team that initial model development, based strongly on the ease of communication and understanding between the involved groups, was sometimes hindered by the complex graphics of contemporary simulation software. Initial discussions often flowed better by using pencil and paper than mouse and screen.

As a better understanding of the problem developed, the simulation team introduced a more ‘rigorous’ modelling tool; the informal flow charts were redeveloped into activity cycle diagrams. After several different versions were produced, discussed and superseded (as mutual understanding of the system prospered) the final version of the conceptual model was produced. The model was divided into the selection of treatment (the randomisation) and the modelling of cancer recurrence. To illustrate the development of the model, we restrict ourselves to the selection of treatment. As can be seen in Figure 1, the treatment pathways consist of an alternative sequence of circles (queues) and boxes (activities). The queues in the model are an artefact of the activity cycle diagram notation and have no effect in the model. They remain in the model to keep the semantics of the modelling technique consistent and as a future basis for extending the model (to possibly consider resource allocation). The activities can be summarised as follows:

• Administration is zero duration and represents the choice of treatment according to patient type (pre/peri menopausal or post menopausal) and symptoms.
• The various P/P activities are also of zero duration and represent the commitment of a patient to a particular treatment path.
• All other activities represent the actual treatment, the treatment choice, duration and treatment success. The ordering of activities depends on the treatment plan. Ordering is important as the effects of possible treatments are not commutative.

Figure1: Activity cycle diagram showing treatment paths

It might be noted that the number of treatment paths identified in Figure 1 is greater than the options for treatment outlined in Table 1. This difference arose during the development of the conceptual model. During the modelling of the treatment paths, discussions between the groups focussed on obtaining a clear picture of exactly what a treatment path meant. One observation was that the ordering of treatment was crucial. Another was a clarification of what treatment paths were available. After discussing the implications of each path, it became clear that other paths were available (leading to fourteen instead of the original ten). It could be argued that this would have occurred irrespective of the approach taken to the modelling of cancer treatment. However, it was the view at the time that the use of simple explanatory devices to clearly illustrate discussions allowed this problem to be identified early in the modelling process.

The conceptual model was translated into a simulation package. The choice of package was based principally on cost and the ease of building the user interface. These factors led to the simulation being developed using Simul8, a simulation package from Visual Thinking International, Strathclyde, Scotland. This package was suitable as it was inexpensive (when compared to other simulation software) and the user interface was relatively easy to implement via a link to Visual Basic. Figure 2 shows part of the model translated into Simul8. As can be seen in figure 2, Simul8 features have been used to implement the conceptual model. Each of the symbols of the networked model are, in Simul8 terms, work centres that take a work item (in this case a patient) and make changes to the attributes of the work item. A patient was modelled by using many attributes such as the progress of cancer and life expectancy. The duration of each work centre was taken from the known information on the length of each treatment. The actual effect that each work centre had on the attributes of the patients processed corresponded to the particular treatment type. In both the case of the conceptual model and the Simul8 model, once a patient had passed through the treatment paths, the patient moved onto the recurrence part of the model.

Figure 2: Simul8 treatment path model

As previously mentioned, the data entry and analysis requirement of the ABCSim package led to the development of a user interface. This is shown in Figure 3. It was decided by the health economists that the experimentation would be by comparing test patient cohorts (with changes in the range of input factors) (alternative 1) against a control (alternative 2). The main screen of the package represents a summary of the current simulation results (the number of patients of each alternative involved in each treatment) but can be used to display results such as average cost, average life years gained, average quality of life, and average quality of life adjusted years, as well as the number of patients per treatment path and the duration of the experiment. As indicated by the menu bars in Figure 3, the wide range of input variables could also be input from this user interface. For information, the major input factors were age group distribution, distribution of treatment path selection, side effects in relation to chemotherapy (toxicities), in-trial menopausal distributions, recurrence and recurrence type probability, remission duration and distribution, costs, and quality of life adjusted years (utilities).

Figure 3: ABCSim user interface

The ABCSim package was in use at the time of writing and was being successfully used as indicated by the health economist team to infer further information concerning the various elements of the Trial and data collection requirements.

Summary

In this article, we have introduced simulation modelling as a useful tool in supporting data gathering in the ABC Trial by fostering a deeper understanding of how the Trial’s factors interact.

As a postscript, it is important to make note of the main theme of this article. Simulation, or any of the wide range of available operational research techniques, are not the answer to all questions and problems. The real power of these lie in their ability to support decision making by providing those with expertise in the problem domain with the opportunity to explore their problem through the development of a flexible, adaptable, visible model. The model does not, in itself, provide answers to the problems associated with the domain. Instead, it should be seen as a negotiation and exploration tool to be used by those in the domain to develop an informed understanding of their problem. Only through such understanding can stakeholders move towards informed and justifiable decision-making based on the analysis of cost and effectiveness of planned decisions.

For the interested reader

Taylor S J E, Eldabi T, Macredie R D, Paul R J, Brown J and Karnon J (1998) ‘Economic Evaluation of Adjuvant Breast Cancer Treatment Using Simulation Modelling’. In the Proceedings of the 1998 Medical Sciences Simulation Conference. San Diego, California. 11-14 January 1998. The Society for Computer Simulation International, San Diego, pp 42-47.

Readers wishing to know more about breast cancer might want to consult the web site www.breastcancerinfo.com

The ABCSim project was funded by medical research grant G9437850.

SIMON TAYLOR is the Chair of the Simulation Study Group of the UK Operational Research Society. He is a lecturer in the Department of Information Systems and Computing and is a member of the Centre for Applied Simulation Modelling, both at Brunel University, UK. He was previously part of the Centre for Parallel Computing at the University of Westminster. He has an undergraduate degree in Industrial Studies (Sheffield Hallam), an MSc in Computing Studies (Sheffield Hallam) and a PhD in Parallel and Distributed Simulation (Leeds Metropolitan). His main research interests are distributed interactive simulation and applications of simulation in health care. He is an advisor to the ABCSim project. He has also been known to occasionally tread the boards.

TILLAL ELDABI is a Research Assistant, and Secretary to the Centre for Applied Simulation Modelling, at Brunel University, UK. He received a BSc in Econometrics and Social Statistics from the University of Khartoum, and an M.Sc. in Simulation Modelling from Brunel University. His doctoral research is in aspects of simulation and health management and the intervention of simulation in health. He is working on applying simulation modelling to health services and health economics, looking at the possibilities of identifying standard methodologies for health simulation. He was the main developer of the ABCSim package.

RAY PAUL is Professor of Simulation Modelling in the Department of Information Systems and Computing at Brunel University. He holds a BSc in Mathematics, and an MSc and PhD in Operational Research, from Hull University. Ray has over 200 journal, book and conference publications in information systems and operational research. He is director of both the Centre for Living Information Systems Thinking (LIST), and the Centre for Applied Simulation Modelling (CASM), at Brunel. Ray acts as a consultant to various government departments, software companies, and commercial companies in the tobacco and oil industries.

First published to members of the Operational Research Society in OR Insight October - December 1998