Reliability of and correlations between attention networks

Note: The ggplot2 wiki is no longer maintained, please use the ggplot2 website instead!

Data and vector graphic PDFs attached at bottom, code & hi-res PNG images included in text. This data comes from in an in-progress manuscript (MacLeod’, Lawrence*, McConnell’, Eskes*, Klein* & Shore’; ’ = affiliated with McMaster University, * = affiliated with Dalhousie University). For publishing purposes, the graphics are developed in greyscale, although in cases where full-colour printing is available, it might be advisable to colour the 95% CIs plotted below to highlight/distinguish them from the data points.

These graphics display data from 15 published and unpublished experiments, involving a total of 1128 human participants. Each experiment employed a popular experimental psychology task called the Attention Network Test (ANT). The ANT requires participants to respond to visual stimuli on a computer screen, and both response time (RT) and error rate (ER) are measured. Within each participant and each measure (RT & ER), the ANT computes a score reflecting the efficiency of each of 3 proposed “Networks” of attention, Alerting, Orienting, and Executive Control. Thus each person has 6 scores of interest (2 measures × 3 networks). Raw data from the 15 experiments was obtained to answer two questions:

1. How reliable is each score?
2. Do the scores correlate?

Reliability

The question of reliability is addressed in figure 1, mapping experiment to the y-axis (for ease of printing/reading the tick labels), observed reliability to the x-axis, and experiment size to point size. The 95% confidence interval of the weighted mean of points within each panel is plotted by a semi-transparent vertical band. A black vertical line marks the zero point on the x-axis and the panels are bounded in width between -1 and 1. A legend describing the mapping of point size to experiment size is omitted as this information is described elsewhere in the manuscript.

This graphic suggests that the Executive scores are reasonably reliable in both RT & ER, but all other scores suffer from poor reliability.

Code:

#start the layers by specifying the base layer with 
# data from which facets will be determined
relplot_base = ggplot(
	data = rels_byexp
)

#specify the facets relplot_facets = facet_grid( net~dv ) #plot reliability estimates from each experiment relplot_points = geom_point( data = rels_byexp , mapping = aes( x = value , y = experiment , size = N ) , alpha = .5 ) #plot a vertical band representing the 95% confidence # interval of the weighted mean of points in each panel relplot_ci = geom_rect( data = rels_weighted , mapping = aes( xmin = lower , xmax = upper , ymin = 0 , ymax = 16 ) , alpha = .5 ) #plot a vertical line indicating zero relplot_zero = geom_vline( data = rels_weighted , mapping = aes( x = zero ) , size = .4 ) #name the x-axis (adding space), set limits, breaks, etc relplot_x = scale_x_continuous( name = '\nReliability' , limits = c(-1,1) , breaks = c(-.5,0,.5) , labels = c('-.5','0','+.5') , expand = c(0,0) ) #name the y-axis (adding space) and set expansion to c(0,0) # to ensure we don't see the top/bottom of rect plotted by # relplot_ci relplot_y = scale_y_discrete( name = 'Study\n' , expand = c(0,0) ) #get rid of major/minor gridlines (at co-author's request) # and remove the size legend relplot_tweaks = opts( panel.grid.major = theme_blank() , panel.grid.minor = theme_blank() , legend.position = 'none' ) #layer all the plot objects together relplot = relplot_base+ relplot_facets+ relplot_points+ relplot_zero+ relplot_ci+ relplot_x+ relplot_y+ relplot_tweaks

#save the plot to pdf & png for this page ggsave(relplot, file='fig1.pdf',height=8,width=6) ggsave(relplot, file='fig1.png',height=8,width=6)

Correlation

The question of correlation is addressed in figure 2, containing a modified correlation matrix. The diagonal contains variable labels for the correlation matrix. Right-of-diagonal panels represent the data split by study, organized in an analogous manner as in figure 1. Left-of-diagonal panels contain the weighted mean correlation across the 15 studies, and whether this mean may be considered significantly different from zero is mapped to the color of the text, white for significant, black for non-significant (panels are otherwise filled with a 50% grey to equate contrast across significance).

This graphic describes small but significant correlations amongst the network scores, both with and between measures (RT & ER), suggesting that the networks are not independent, at least as measured by the ANT. Indeed, the true magnitude of these correlations may be greater than that observed by the current data, as the reliability of a measure places an upper limit on the magnitude of its associated correlations and we previously observed less than perfect reliability in these scores. Using Spearman’s (1907) correction, for example, the correlation between Alerting and Executive networks measured in error rate is -.97.

Code:

#start the layers by specifying the base layer with 
# data from which facets will be determined
corplot_base = ggplot(
	data = cors_byexp
)

#specify the facets
corplot_facets = facet_grid(
	dv1net1~dv2net2
)

#plot the correlation estimates from each study
# Note: the data frame 'cors_byexp' only contains
# dv1net1~dv2net2 combinations that specify the
# right-of-diagonal cells, letting us plot other
# information on the diagonal and left-of-diagonal
# cells. Important for this is that the full set of
# factor levels are specified for the data frame, even
# though some combinations are not used.
corplot_points = geom_point(
	data = cors_byexp
	, mapping = aes(
		x = value
		, y = experiment
		, size = N
	)
	, alpha = .5
)

#plot a vertical band representing the 95% confidence 
# interval of the weighted mean of points in each panel
corplot_ci = geom_rect(
	data = cors_weighted
	, mapping = aes(
		xmin = lower
		, xmax = upper
		, ymin = 0
		, ymax = 17
	)
	, alpha = .5
)

#plot a vertical line indicating zero
corplot_zero = geom_vline(
	data = cors_zero
	, mapping = aes(
		x = x
	)
	, size = .4
)

#in each left-of-diagonal panel, plot a square
# the size of the panel to serve as a background
# for the weighted means text
corplot_means_background = geom_rect(
	data = cors_weighted_text
	, mapping = aes(
		xmin = xmin
		, xmax = xmax
		, ymin = ymin
		, ymax = ymax
	)
	, colour = 'grey50'
	, fill = 'grey50'
)

#plot the text describing the weighted means
corplot_means = geom_text(
	data = cors_weighted_text
	, mapping = aes(
		x = x
		, y = y
		, label = printable
		, colour = nsig
	)
	, size = 10
)

#in each diagonal panel, plot a square the size
# of the panel to serve as a background for the
# variable labels
corplot_diag = geom_rect(
	data = cors_diag
	, mapping = aes(
		xmin = xmin
		, xmax = xmax
		, ymin = ymin
		, ymax = ymax
	)
	, colour = 'white'
	, fill = 'white'
)

#plot the variable labels
corplot_diag_text = geom_text(
	data = cors_diag
	, mapping = aes(
		x = x
		, y = y
		, label = dv1net1
	)
	, size = 5
)

#specify the x-axis limits, breaks, and expansion
corplot_x = scale_x_continuous(
	limits = c(-1,1)
	, breaks = c(-1,0,1)
	, expand = c(0,0)
)

#set the y-axis expansion to c(0,0) so you don't
# see the top of the rectangle created by corplot_ci
corplot_y = scale_y_discrete(
	expand = c(0,0)
)

#set the colour scale to white and black
corplot_colour = scale_colour_manual(
	values = c('white','black')
)

#get rid of gridlines, axis text/ticks/titles, 
# size legend and strip labels
corplot_tweaks = opts(
	panel.grid.major = theme_blank()
	, panel.grid.minor = theme_blank()
	, axis.title.y = theme_blank()
	, axis.text.y = theme_blank()
	, axis.title.x = theme_blank()
	, axis.text.x = theme_blank()
	, axis.ticks = theme_blank()
	, legend.position = 'none'
	, strip.background = theme_blank()
	, strip.text.y = theme_blank()
	, strip.text.x = theme_blank()
)

#layer all the plot objects together
corplot = corplot_base+
	corplot_facets+
	corplot_points+
	corplot_ci+
	corplot_zero+
	corplot_means_background+
	corplot_means+
	corplot_diag+
	corplot_diag_text+
	corplot_tweaks+
	corplot_x+
	corplot_y+
	corplot_colour

#save the plot to pdf & png for this page
ggsave(corplot, file='fig2.pdf',height=6,width=6)
ggsave(corplot, file='fig2.png',height=6,width=6)