forked from MathiasHarrer/Doing-Meta-Analysis-in-R
-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy path08-publication_bias.Rmd
142 lines (88 loc) · 7.84 KB
/
08-publication_bias.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
# Publication Bias
![](publicationbias.jpg)
The **file-drawer** or **publication bias** problem is based on the fact that studies with big effect sizes **are more likely to be published** than studies with small effect sizes [@rothstein2006publication].
The studies with low effect sizes might never get published, and therefore cannot be integrated into our meta-analysis. This leads to **publication bias**, as the pooled effect we estimate in our meta-analysis might be higher than the **true effect size** because we did not consider the missing studies with lower effects due to the simple fact that they were never published.
Although this practice is gradually changing [@nelson2018psychology], whether a study is published still heavily depends on the **statistical significance** ($p<0.05$) of its results [@dickersin2005publication]. For any sample size, a result is more likely to become **statistically significant** if its **effect size is high**. This is particularly true for **small studies**, for which very large effect sizes are needed to attain a statisitcally significant result.
## Detecting publication bias {#smallstudyeffects}
Various methods to assess and control for publication bias have been developed, but we will only focus on a few well-established ones here. According to Borenstein et al. [@borenstein2011]. The model behind the most common small-study effects methods has these core **assumptions**:
1. Because they involve large commitment of ressources and time, **large studies are likely to get published**, weather the results are significant or not
2. Moderately sized studies are at **greater risk of missing**, but with a moderate sample size even moderately sized effects are likely to become significant, which means that only some studies will be missing
3. Small studies are **at greatest risk** for being non-significant, and thus being missing. Only small studies with a very large effect size become significant, and will be found in the published literature.
In accordance with these assumptions, the methods we present here particularly focus **on small studies with small effect sizes, and wheather they are missing**.
### Funnel plots
The best way to visualize whether **small studies with small effect sizes are missing** is through **funnel plots**.
We can generate a funnel plot for our `m_re` meta-analysis output using the `funnel()` function in `metafor`:
```{r,echo=FALSE,message=FALSE}
library(metaforest)
df <- curry
m_re <- rma(d, vi, data= df)
```
```{r}
funnel(m_re, xlab = "Hedges' g")
```
The **funnel plot** basically consists of a **funnel** and two **axes**: the y-axis showing the **Standard Error** $SE$ of each study, with larger studies (which have a smaller $SE$) plotted **on top of the y-axis**; and the x-axis showing the **effect size** of each study.
Given our assumptions, and in the case when there is **no publication bias**, all studies would lie symmetrically around our pooled effect size (the vertical line in the middle), within the form of the funnel. When **publication bias is present**, we would assume that the funnel would look asymmetrical, because only the small studies with a large effect size very published, **while small studies without a significant, large effect would be missing**.
We see from the plot that in the case of our meta-anlysis, there is no obvious publication bias among the smaller studies, but there are a few moderately-sized studies that fall outside of the funnel shape. These are the outliers we investigated earlier.
We can also display the name of each study using the `studlab` parameter.
### Egger's test
Remember that we can formally test for funnel asymmetry, by predicting the effect size from its standard error. For this, we can simply apply the `regtest` function to our existing random-effects meta-analysis, and the SE will be used as a predictor:
```{r}
regtest(m_re)
```
The result is non-significant, so there is no evidence for asymmetry.
### Contour-enhanced funnel plots
An even better way to inspect the funnel plot is through **contour-enhanced funnel plots**, which help to distinguish publication bias from other forms of asymmetry [@peters2008contour]. Contour-enhanced funnels include colors signifying the significance level into which the effects size of each study falls (p < .05, p < .01, and p < .001). We can plot such funnels using this code:
```{r,message=FALSE,warning=FALSE}
funnel(m_re, level=c(90, 95, 99), shade=c("white", "gray55", "gray75"), refline=0)
```
We can see in the plot that while **some studies have statistically significant effect sizes** (the gray areas), others do not (white background). We see that the significant studies are primarily on the right side of the line.
### Testing for funnel plot asymmetry using Egger's test
**Egger's test for funnel plot asymmetry** [@egger1997bias] quantifies the funnel plot asymmetry and performs a statistical test:
```{r}
regtest(m_reg)
```
The function uses regression to test the relationship between the observed effect sizes and the standard error of the effect sizes. If this relationship is significant, that might indicate publication bias. However, asymmetry could have been caused by other reasons than publication bias. In this case, there is no significant asymmetry.
### Duval & Tweedie's trim-and-fill procedure {#dant}
**Duval & Tweedie's trim-and-fill procedure** [@duval2000trim] is also based the funnel plot and its symmetry/asymmetry. When **Egger's test is significant**, we can use this method to estimate what **the actual effect size would be had the "missing" small studies been published**. The procedure **imputes** missing studies into the funnel plot until symmetry is reached again.
```{block,type='rmdinfo'}
**The trim-and-fill procedure includes the following five steps** [@schwarzer2015meta]:
1. Estimating the number of studies in the outlying part of the funnel plot
2. Removing (trimming) these effect sizes and pooling the results with the remaining effect sizes
3. This pooled effect is then taken as the center of all effect sizes
4. For each trimmed/removed study, a additional study is imputed, mirroring the effect of the study on the left side of the funnel plot
5. Pooling the results with the imputed studies and the trimmed studies included
```
The **trim-and-fill-procedure** can be performed using the `trimfill` function in `metafor`. For this example, we will first introduce some intentional publication bias:
```{r}
df_bias <- df
# Identify the 20 effect sizes with the smallest effect sizes
small_effects <- order(df_bias$d)[1:20]
# Of these small effect sizes, find the ones with the biggest sampling variance
big_variance <- order(df_bias$vi[small_effects], decreasing = TRUE)
# Delete these studies:
delete_these <- small_effects[big_variance[1:10]]
df_bias <- df_bias[-delete_these, ]
```
Now, we re-do some of the analyses:
```{r}
# Fit random-effects model
m_bias <- rma(d, vi, data = df_bias)
# Test for publication bias is now significant:
regtest(m_bias)
# Carry out trim-and-fill analysis
m_taf <- trimfill(m_bias)
m_taf
```
We see that the procedure identified and trimmed **16 studies** `(with 16 added studies)`). The overall effect estimated by the procedure is $g = 0.1480$.
Let's compare this to our initial results.
```{r}
m_re$b[1]
```
The initial pooled effect size was $g = 0.2393$, which is substantially larger than the bias-corrected effect. In our case, if we assume that **publication bias** was a problem in the analyses, the **trim-and-fill procedure** lets us assume that our initial results were **overestimated** due to publication bias, and the "true" effect when controlling for selective publication might be $g = 0.1480$ rather than $g = 0.2394$. Of course, we intentionally introduced the bias in this case.
We can also create **funnel plots including the imputed studies**:
```{r}
# Draw funnel plot with missing studies filled in
funnel(m_taf)
```
<br><br>
---