Epidemiological study design in gastroenterology and how to avoid them 

What makes us want to conduct a piece of research? Have we seen something unusual, or all too common, in the clinic? Is there something that is a major issue of clinical or public health importance? Has there been particular media interest in a specific disease and/or treatment? Is there a call for research funding into a specific topic?

When something piques our interest, it is crucial to find what other research on the topic has already been done and what is underway. This knowledge helps to determine whether or not we should pursue our interest. We may wish to apply the ‘FINER’ criteria to our work—is our study feasible (see mistake 8), interesting, novel, ethical (see mistake 9) and relevant?¹ 

If we begin with the assumption that our initial thought is highly unlikely to be novel, then the logical first step is to find out what has already been done on the topic, what the results were, and in what setting or populations was it performed. Doing such due diligence will generate ideas regarding the appropriate methods of enquiry and potentially identify limitations in previous research that could be improved on. Essentially, a thorough search of the existing literature and knowledge of ongoing research in the field of interest allows a broad area of interest to be refined into a specific and well-formulated question (see mistake 2).²

Frequently, people assume there are only two steps to a literature review—gathering information and then synthesising it—but there is more to it than that (figure 1). Crucial to a good scientific literature review (systematic or otherwise) is to critically analyse the literature before synthesising it.³ Only by critiquing the literature can you assess its validity and context in terms of your specific research question. This is increasingly important in the world of science and medicine, characterised as it is by information overload and the proliferation of journals, some of which have questionable peer-review processes. By identifying gaps and weaknesses in the research that has gone before, you are more likely to be able to refine your own study question into a relevant and answerable hypothesis. 

It is not always necessary to perform a systematic review, although doing so, particularly if you are embarking on a larger programme of primary research, can be extremely beneficial and potentially justify publication in its own right.⁴ Remember that not all research is published in paper form, so knowing who is doing what (perhaps from conference attendance) means that you can find out more information and also see what the current trends are in your research field.⁵
 

Spending a great deal of work, time and resources on a research project before the question of data analysis is raised (as may commonly occur during abstract submission deadline season) is extremely detrimental to its chances of success. By not prioritising the formulation of the precise research question, it can become obvious that the data collection was incomplete, or that the question cannot be answered without the need for additional work that may or may not be feasible (e.g. the patients filling a questionnaire, etc.) (figure 2). 
                                   
Framing a clear, well-defined research question is key to a successful research project. Such a question should be generated in the early stages of the research project and provide the rationale for most of the decisions subsequently made during the study design process. Clear research questions in epidemiological study design are often framed using a PICO format, with a clearly defined patient population, intervention (or exposure), comparison and outcome.⁶ Only once a research question has been framed, can a clear plan for the study design, data collection and analysis be completed (further information can be found in an article we wrote examining this “mistake” from a Young GI angle⁶). 

The prevailing school of thought in the teaching and practice of medicine across Europe is that medicine should be evidence based.^7,8 Indeed, this seems rather intuitive—who would not want to be treating their patients, or treated as a patient, on the basis of the best available knowledge at the time?

Ubiquitous within the teaching and scholarship of evidence-based medicine (EBM) is the evidence hierarchy.9 Most people will be familiar with this as a pyramidal representation of different study designs, with the ‘lowest’ form of evidence provided by case reports and case series at the base, moving up through various epidemiological study designs, case–control studies, cohort studies, and randomized controlled trials (RCTs), with the top of the pyramid culminating in systematic reviews and meta-analyses. Following our literature review and research question refinement, we determine we need to conduct primary research (thus removing the option of a systematic review or meta-analysis) and are left with RCTs as the ‘gold standard’ in research. 

Trying to push our research into the highest tier of the EBM pyramid on no other basis than believing it will therefore provide ‘better quality evidence’ is a foolish way to embark on research.¹⁰ Indeed, we recommend that you examine the question and choose the appropriate study design. Each epidemiological study design has its merits and its weaknesses, and each is better suited to answering certain types of question.¹¹ In particular, recent advances in causal inference methodology have removed some of the previous postulated weaknesses of observational studies not being able to answer causal questions.¹² Arguably, a well-conducted causal observational study will provide better evidence of a real world application than an RCT, conducted as they usually are on a highly sampled population (see mistake 7). 

In addition, there are several pertinent research questions that can only be answered by eliciting patient or physician preferences and viewpoints. Although this article is considering quantitative epidemiological study designs only, we cannot stress strongly enough that the findings of a well-conducted qualitative study are just as important in adding to our evidence base if the question being asked is best answered by a study of this nature. As EBM pioneer David Sackett so eloquently said more than 20 years ago, “Each method should flourish because each has features that overcome the limitations of others when confronted with questions they cannot reliably answer.”13 Precisely! 

Defining key data items such as primary exposure or intervention, primary outcome and covariates is clearly important for the data collection and analysis process.¹⁴ If the parameters are not clearly defined nonuniform and/or unsystematic data collection results, and the data generated will be uninformative, or worse, may lead to incorrect inference (bluntly speaking: garbage in = garbage out). This point is especially important if multiple individuals are to be involved in the data collection process. 

While this may be less of an issue if the data item is an objective measurement (e.g. CRP levels), even if a time frame is defined (e.g. CRP levels after 3 months of treatment), the situation becomes more challenging when more objective findings need to be subjectively evaluated (e.g. severity of mucosal inflammation demonstrated by colonoscopy), and especially when the entire evaluation is based on subjective information (e.g. wellbeing or quality of life [QoL]). The utility of subjective measurements should not be discounted however. There is a clear shift to including patient reported outcome measures (PROMs) and patient reported experience measures (PREMs) alongside ‘hard’ endpoints to allow full assessment of healthcare quality.¹⁵

Using previously validated scores and questionnaires (e.g. from the International Household Survey Network (IHSN) Question Bank [https://qbank.ihsn.org/]) is a way to ensure the scientific value of any data collected and goes some way to meet reviewers’ expectations for publication. Reviewing the relevant literature and consulting an expert in the field as part of the protocol preparation process may help to identify appropriate instruments and scoring systems. Of note, some may require permission for use. Using standardized instrumentation is also beneficial in other ways: a clear methods section becomes easier to write, and the possibility that the work can be reproduced and replicated is increased.^16,17

In the age of electronic records and large datasets, it is important to keep the focus on the predefined research question and to be careful with any additional analysis that may be performed. One of the pitfalls to avoid is blindly running statistical tests across all parameters in a large dataset, hoping to stumble across a significant finding (again, this is more likely to occur when time pressured and is sometimes referred to as ‘fishing’ or ‘p-hacking’).¹⁷ 

The reason why running numerous statistical tests can be a mistake is both logical and mathematical.18 The basis for many statistical hypothesis tests is evaluating the chance of obtaining the observed results compared with what would have been expected (e.g. based on the general population or the study’s control group), with a commonly used threshold of 5% for stating that the observed data are significantly different to the expected (this is viewed by many as a mistake in itself,19 but further discussion is beyond the scope of this article). In other words, there is a 5% chance of getting results that appear significantly different from the expected ones, even though the reality is that there is no significant difference (this is a type-1 error). The more statistical tests we run, the more these 5%s start to add up, and, therefore, when multiple comparisons are performed the threshold for determining significance is made more conservative. The Bonferroni correction is often used in this regard, though other methods also exist.²⁰

Of note, certain projects have more than one outcome. In these cases, the primary outcome and any secondary outcomes should be clearly predefined, and the analysis should be performed accordingly. Finally, important discoveries can still be made outside those predicted in the analysis plan or even by the original question. In these cases, it is important to realize whether the findings are indeed a true signal or an artefact from multiple comparisons. Additional confirmatory studies, focusing on the new question, may be needed. 

Fundamental to conducting a good study is the awareness of sample size (or study size).²¹ If an effect truly exists then the study should be performed on a sufficiently large sample to detect it. Too small a sample size can also increase the likelihood of generating a false-positive effect because smaller sample sizes can be more prone to certain biases (see mistake 7). 

What the sample size should be for any particular study depends on a number of parameters. There are several pieces of knowledge that are fundamental to all sample size calculations (figure 3). 

The first consideration is the prevalence of the disease in the unexposed population, which may be known from previously conducted research or from other similar populations. There is a level of guestimation involved in determining this parameter, but basing it on prior knowledge is, of course, beneficial. The second factor is the minimum effect size, that is, the clinically relevant difference that we wish to detect (e.g. a reduction of bilirubin levels of 1mg/dL or 0.1mg/dL). The smaller the difference to be detected, the larger the sample size required. The third parameter is the significance level (or alpha value) of the study, which is the probability of rejecting a null hypothesis given that it is true (type-I error) and is frequently set at 5%. The final component when calculating sample size is the power of a study, the probability that it will detect an effect when there is one to be detected—power is frequently set at 80% or 90%. 

With these parameters known to us or chosen, the required sample size can be calculated. Other factors that may need to be taken into consideration include the ratio of controls to cases and the likelihood of loss to follow-up in the patient population. The more complicated the study design, the more complicated the calculation of sample size becomes. Whilst there are several statistical packages available that allow the calculation of sample sizes (some opensource and others proprietary),²²  as with so many aspects of study design it is best to ensure you have the correct skills mix and support for your project from the outset. Consulting or collaborating with a statistician at this stage of project development can be extremely beneficial. 

In practice, sample size considerations are often based as much, if not more, on resource implications as they are on the purely mathematical components mentioned above. If we know there is a finite number of cases, an adjustment to other parameters will have to be justified to allow the study to go ahead, such as increasing the numbers of controls to cases, or increasing the effect size to be detected. Conducting a study on too many cases (perhaps by wishing to detect a very small effect size) may be considered a waste of resources and, therefore, the study runs a higher risk of not being funded (see mistake 8). 

The avoidance of bias is one of the cornerstones of good epidemiologic study design. Bias has been descrided as “…the deviation of results or inferences from the truth or processes which lead to such deviation.”²³ There are a huge number of defined biases and it is not possible to discuss every single one of them here. Broadly, bias in epidemiological studies can be categorized into selection bias and misclassification (information bias; see table 1). 

Selection bias arises when there is a systematic difference in the characteristics of people within a study compared with those who are not. This can include differences or errors in those enrolled in the study in the first place compared with the target population (sampling bias), through the inclusion of an inappropriate control population (e.g. choosing hospital controls who have a particular condition), and through following up participants differently dependent on their exposure status (attrition bias or follow-up bias). 

Misclassification is a deviation in the measurement of one or more factors of interest in a study. Misclassification may affect all groups equally (nondifferential misclassification) or, perhaps more frequently, it may be more likely to occur in one group than another (differential misclassification, or information bias). Information bias includes observer bias (where a study team member differentially records observations based on known exposure or outcome status), recall bias (where a participant is more or less likely to remember accurately an exposure based on their outcome status) and reporting bias (where a participant is more or less likely to reveal or suppress certain information, particularly when there may be stigma attached to it [e.g. alcohol use]). 

We also include confounding in this section. There is debate as to whether confounding is strictly speaking a type of bias because it does not arise from a systematic error in the design of a piece of research, but rather from the nonrandom distribution of risk factors in the population being studied. A confounder is a third variable that is associated both with the disease and exposure of interest in any given study. If a confounding variable has been measured then it is usually possible to adjust for it in the analytical stages of the study through stratification or modelling. The study design can also be adjusted to minimize or remove confounding by restricting the study population or matching the population based on certain known (or suspected) confounding factors. If this is not possible or appropriate, careful thought as to the likely confounding variables in any study will ensure that appropriate data have been collected to allow adjustment at the analysis stage. 

Resources for research are often limited, especially for those in the early stages of their career. Research that requires recruitment of patients is often orders of magnitude more expensive than research that can be performed on data collected via other mechanisms. Of course, although funding is a crucial point, it is not the only one, and once a research question has been framed, and a protocol has been finalized, it is important to evaluate whether the proposed project is feasible or not. 

At times, a protocol that on paper seems perfectly tailored to answer a research question is not practicable and will be impossible to execute in the real world. In such cases, the protocol and research question should be revisited, and a more practical approach considered. Although this may be frustrating because plenty of time and work will have already been invested in the project, it is vital to recognize unfeasible projects, and identify them as ‘nonstarters’. This course of action may ultimately be more helpful to the researcher’s professional development than the alternative of starting a neverending project. 

Many factors may render a project unfeasible. Insufficient protected time for carrying out the research, and an insufficiently framed research question are pitfalls that early-stage researchers should avoid. Finding good mentors and experienced collaborators (or those with special attributes relevant to the project, such as access to a unique subset of patients) can provide tremendous support—with their input it may be possible to turn an unfeasible project into a project that can be successfully completed. 

Overlooking ethical and regulatory aspects is no less important a mistake than any of the others already considered, but such considerations are not unique to epidemiological studies. Studies recruiting participants or using data obtained from humans (most epidemiological studies) have specific ethical considerations that were enshrined in the Declaration of Helsinki in 1964 and are updated periodically at the World Medical Association General Assembly.²⁴ Different considerations are required for studies involving animals (including ensuring the welfare of animals and the principles of replacement, reduction and refinement). Ethical conduct, in clinical practice and in research, should always be front and centre in a clinician’s mind. 

A study protocol may be well designed to answer the research question but fall short of meeting ethical and regulatory requirements. For example, collecting 50 ml of blood may be sufficient to run all the tests needed for a new study on necrotizing enterocolitis in preterm babies, but such a blood draw may be considered unethical in this population. This point also corresponds with issues of feasibility (see mistake 8), but it is to be hoped that such a request would not be approved by the local review board for ethical reasons before empirical data indicated the feasibility challenge. 

Ethical and regulatory aspects should, therefore, be integrated into the study design thought process, so that the end result is a protocol that would meet ethical expectations, receive the necessary approvals and meet publishing standards. For studies involving humans, the informed consent process, risk/benefit ratio, conflicts of interest, adherence to good clinical practice (GCP), and a clear distinction between routine clinical practice and study-related procedures are a partial list of the elements that need to be addressed when a new application for study approval is submitted. Specific considerations regarding the use of data are covered in the EU’s General Data Protection Regulation (GDPR) and most ethics applications now also require data management and processing considerations.²⁵

Of course, different regions may have different regulatory requirements, and these can also vary based on the role of the researcher in the study staff. For early-stage investigators, it is important to be aware of the requirements that have to be met both on the personal and protocol-associated level. In addition to the often-required formal training, having a good mentor and consulting colleagues may prove invaluable when trying to overcome any hurdles being faced for the first time. 

For many of the mistakes discussed above, our opinion is that early collaboration and consultation with other colleagues and experts is the best course of action. This includes, but is not limited to, working with other clinicians, epidemiologists, statisticians, triallists, ethicists, methodologists, librarians, clinical coders, patients and public representatives. Indeed, trying to go it alone is invariably a mistake because collaboration and consultation generate the best, and most enjoyable, research, allowing us to question each other’s positionality, to refine our thinking, and design the best study available to answer our question. For certain questions, the magnitude of what we hope to achieve can only be attained through multisite, perhaps multicountry, collaboration. Fostering a good network from an early stage of your career, building relationships, and not being afraid to drop a potential collaborator an email or DM, can all lead to fruitful research opportunities and ultimately better designed and conducted studies.