Dyslexia and Academic Confidence

The position to date (July 2016); 

166 good datasets have now been received in total, thus forming the datapool.

The dataset generated from each eQNR reply has been received as a .csv file so in the first instance these have been collected together into a ‘master’ Excel spreadsheet.

SPSS will be used later for a more detailed interrogation of the data but for the moment, Excel has been an entirely satisfactory tool to enable me to reflect on the data as a whole and to execute an initial analysis which has provided interesting results, indeed, the results that had been hoped for:

Introduction

The research groups referred to in the analysis summary below have been labelled carefully so that I will know what I’ve done at this stage for when I come to analyse the data in more detail later and write it all up more thoroughly. However to make it easier to follow… :

In the eQNR, respondents either declared that they had dyslexia or they didn’t. Hence forming two primary groups:

• Respondents who declared their dyslexia are designated Research Group: DI
• Respondents who didn’t declare dyslexia are designated Research Group: ND.  This group includes a few respondents who declared other learning challenges.

Two subgroups from research group: ND have been established:

• Respondents who presented a Dyslexia Index, Dx < 400, are designated Research Group: ND-400. Metrics from these respondents have been used to create the profile diagram background means for the 6 Likert scales measured in the eQNR as the group of students who are not dyslexic. Details about the reasons for using this boundary point are reported in the post ‘the data is settling… ‘ elsewhere in the StudyBlog.
• Respondents who presented a Dyslexia Index, Dx > 600, are designated Research Group: DNI. This is the research group that is of particular interest as it represents students who are presenting a dyslexic profile (by virtue of their responses generating a Dyslexia Index, Dx > 600) but who have not declared, disclosed, identified dyslexia as a learning challenge. Details about the reasons for using this boundary point are reported in the same elsewhere but also below.

Another subgroup, from research group: DI, has also been established:

• Respondents who presented a Dyslexia Index, Dx > 600, are designated Research Group: DI-600. Metrics from these respondents have been used to create the profile diagram background means for the 6 Likert scales measured in the eQNR as the group of students who are dyslexic.  The reasons for this are discussed elsewhere (as indicated above) but also more is said about this below. The underlying reason though, is it is felt important that as far as possible, metrics aside from the Academic Behavioural Confidence (ABC) that is the main focus of the research scrutiny, are kept as close as possible between the main research groups of interest, this now being research group: DNI and research group: DI-600 as both of these groups present mean Dx values that are not significantly different (details about how this was established are reported below).

So in summary, where a research group has been created as a subgroup of either of the main research groups DI or ND, these are designated by the parent research group they’ve come from with a suffix indicating the Dyslexia Index boundary point that has been used to separate these respondents out. The exception to this is research group: DNI, the group of primary interest.

Now as reported in an earlier post, a fourth research group has emerged which represents a kind of overlap between the two primary research groups ND and DI and this is respondents who presented a Dyslexia Index, 400 < Dx < 600 which is interesting. However for the moment, inspecting data in this group has been parked although I will return to this later.

In analysing the stats, summarized below, where Student’s T-test has been used, the version applied has been to test independent means between groups assuming equal population variances. More work will be conducted later on additional tests that are part of this (for example, testing for equal population variances) and this will be done through SPSS when the data has been exported to that application.

Additionally, a one-tailed test has been used as I am interested in whether one ‘test’ statistic is greater than the other rather than just different (for which a two-tailed test would be appropriate).

Where effect size has been reported below, I am using at present, Cohen’s ‘d’ but of course even a raw difference between means is one measure of effect size, as is a correlation coefficient. All of this will be explored in more detail later and reported.

The Summary  Table of results so far is available here (opens in a new window) in which the rows of data in pale blue are the ones of greatest interest, that is, summary data for research groups: DNI and DI-600.

So in summary, this is what has emerged so far:

• students in research group(RG): DI presented a Dyslexia Index (Dx) in the range: 357 < Dx < 922 with a mean Dx = 653, median Dx = 676 and 95% confidence interval for the population mean Dx of 623.41 < Dx < 682.51.  n = 68.

• research group: DI presented an Academic Behavioural Confidence (ABC) score in the range: 26 < ABC < 92 with a mean ABC = 58.45, median ABC = 56.79 and 95% confidence interval for the population mean ABC of 54.77 < ABC < 62.12. n = 68.

• the subgroup of research group: DI which comprises students presenting Dx > 600, has been used to generate the mean values for the 6 psychometric scales that were measured in the eQNR and these were applied to the profile diagrams as the background reference against which each respondent’s profile could be compared. More details about the processes used for this are reported in a separate blog-post. This subgroup is labelled DI-600, n=46.

• in this subgroup, DI-600, which I am treating as students who are most definitely presenting a dyslexic profile, the mean Dx = 723.6 and the mean ABC = 57.45 with a 95% CI of 53.09 < ABC < 61.82. n = 46. This subgroup will be used for statistical comparison with the ‘test’ research group: DNI since both research groups will comprise students with Dx > 600.

• students in research group: ND presented a Dyslexia Index in the range: 100 < Dx < 910 with a mean Dx = 454. However, it is from this datapool that the ‘test’ research group: DNI has been established – that is, students presenting a dyslexic profile who are otherwise not declaring or disclosing dyslexia. Applying a consistent criteria for creating the subgroup, DNI, that is, datasets presenting Dx > 600, the mean Dx = 690.8, median Dx = 653.1 and 95% confidence interval for the population  mean Dx is 655 < Dx < 727.

• this research group: DNI presented an ABC score in the range 45 < ABC < 89 with a mean ABC = 64.84, median ABC = 61.7 and 95% confidence interval for the population  mean ABC of 58.75 < ABC < 70.93.

• so in comparison with research group: DI-600, this research group: DNI presents a mean ABC that is over 7 points higher although with a confidence interval range of 12.18 this is substantially wider than that for research group: DI-600, at 8.73 which suggests a greater variability.

• the subgroup of research group: ND which comprises students presenting Dx < 400, has been used to generate the mean values for the 6 psychometric scales and these were also applied to the profile diagrams as a background reference. This subgroup is labelled ND-400.

• in this subgroup, ND-400, which I am treating as students who are most definitely NOT presenting a dyslexic profile, the mean Dx = 300.6 and the mean ABC = 72.19 with a 95% CI of 68.46 < ABC < 75.92. n = 43.

• the normal distribution curves for the estimated background population distributions for each research group: DI-600, DNI and ND-400 are shown below:

Effect size results:

On the basis of these figures, Cohen’s ‘d’ effect sizes for the differences in mean ABC for the three research groups: DI-600, DNI and ND-400 have been calculated using using weighted means (degrees of freedom) pooled sample standard deviations (eg: Cumming, 2012, p156) as the calculating process which generated these results:

• Cohen’s ‘d’ effect size difference for Academic Behavioural Confidence between RGs: ND-400 <-> DI-600  =  1.0606

• Cohen’s ‘d’ effect size difference for Academic Behavioural Confidence mean score between RGs: DNI <-> DI-600  =  0.5082  and it is this result that is the really interesting one as it appears to be indicating that there is a medium effect size (Sullivan & Feinn, 2012) – which is very gratifying as it is the result I had hoped the research data would generate and which forms the backbone of this complete project.

• These effect sizes are readily apparent from the figure (above) broadly indicated by the extent of overlap between the three graphs.

Student’s t-test results:

Even though I am converted to the measure of effect sizes and confidence intervals as more relevant when reporting research results in education and social sciences research, I have also run a conventional t-test to look for a significant difference between the 2 independent Academic Behavioural Confidence sample means of research groups DI and DNI.

For a quick result, I used one of the many online t-test calculators here: http://www.socscistatistics.com/tests/ which generated the results listed below and in the summary table at the end of this section.  I have been able to confirm these t-test results in my own Excel spreadsheet of the complete datapool using Excel’s built-in t-test function.

• t-value = -1.79069, p-value = 0.039153 using a one-tailed test.  Since p-value 0.01 < p < 0.05, this result is significant at the 5% level but not quite highly significant at the 1% level.

• The result obtained using Excel’s in-built t-test function generated a p-value = 0.039179 – the slightly higher (i.e. less significant) result I attribute to different calculation processes between Excel and the calculation engine at www.socscistatistics.com – what would the Student, (William Sealy Gosset (right)) have to to say about this I wonder?

So it is also pleasing to record this as an additional indicator that the ABC mean value for research group: DNI is statistically significantly higher than the ABC mean value for research group: DI-600. Together with the effect size calculation this is beginning to indicate that students with an unknown dyslexic profile present a significantly higher Academic Behavioural Confidence than students with a known dyslexic profile. Cool 🙂

An alternative perspective on analysing these data:

However, it occurred to me that although the result above looks pleasing, I am keen to avoid falling into the trap of just reporting the data analysis that appears to be providing the results hoped for. To this end, I felt that in order to justifiably conclude that there is a difference in ABC between research groups: DI and DNI I need to be more confident that in other respects, the students’ data results are broadly similar – at least in relation to the mean Dyslexia Index of each group.

At the moment, I have used a somewhat arbitrary discriminator for the boundary point Dx = 600 where a student whose Dyslexia Index is higher than this I am considering as pretty definitely presenting a dyslexic profile. So research group: DNI is comprised of students from research group: ND who present a Dx > 600 taken from this datapool where the Dyslexia Index ranges from a very undyslexic Dx = 100.16 to an astonishingly high value of Dx = 910.20.

As reported above, the mean Dx for this subgroup (DNI) is Dx = 690.8 so in order to be able to more properly compare these students’ mean ABC to the mean ABC for students in research group: DI-600 – the dyslexic control group – this group (DI-600) needs to present a mean Dx that is not significantly different/higher than the mean Dx for research group: DNI.  However for research group: DI-600, the mean Dx = 723.6 which at face value at least is certainly higher (cf: 690.8) but is it significantly higher as to therefore imply that the two groups are less than equal in terms of their mean Dyslexia Indicies?

To test this, I reverted to the independent means t-test as a quick way to establish whether there is a significant difference between the Dx means of research group: DI-600 and research group: DNI.

• research group: DNI, mean Dx = 690.8;  research group: DI-600, mean Dx = 723.59

• Student’s t-test assuming two independent samples, equal population variances: p-value = 0.05077 which is just > 0.05, but only just, which when taken at face value does satisfy my judgment criteria that there is no statistically significant difference between the means (Dx) for research groups: DNI and DI-600.

• In keeping with my ‘conversion’ to effect size reporting, I used the quick effect size calculator at www.socscistatistics.com to calculate the effect size between the two means. This generated an effect size of 0.4586 which is close to the small/medium boundary.

This leaves me a little uneasy, so I will explore the impact of adjusting the catchment for the subgroup of research group:DI-600 so that the mean Dx is not only NOT significantly different from the mean Dx for research group: DNI, but which also generates an effect size that is at most, ‘small’ (that is, Cohen’s d < 0.2 (Sullivan & Feinn, 2012)) and this will be established by adjusting the boundary value Dx and then using the t-test to determine if the significant difference between the means has been eradicated:

1st attempt:

For the first attempt, I have adjusted the boundary value to Dx = 500. That is, the new subgroup of research group: DI comprises datasets that present a Dx > 500.  This research subgroup is identified as DI-500.

• research group: DNI, mean Dx = 690.8; research group: DI-500, mean Dx = 694.05

• Student’s t-test generates p-value = 0.4444 which is > 0.05, in fact it is miles away from 0.05 and so is most definitely NOT significant at the 5% level. Well I’m not surprised as the two Dx values are very close.

• Cohen’s ‘d’ effect size for Academic Behavioural Confidence between research group: DNI and research group: DI-500  =  0.4739.

So the effect size for ABC is still close to ‘medium’ when using this adjusted research subgroup: DI-500 as the ‘control’ group.

However, with a p-value in the t-test of 0.4444, one might say that this is very not significant at the 5% level and that an alternative boundary point might therefore be sought that produces a t-test p-value that is closer to the 0.05 critical point.

2nd attempt:

To try to establish this, the t-test has been applied repeatedly to the mean Dx for research group: DNI and a subgroup of research group: DI to try to find a more appropriate boundary value for Dx that is between Dx = 500 and Dx = 600 in research group: DI. A few trials indicated that Dx = 580 could be the cut-off point.

• research group: DNI, mean Dx = 690.8; research group: DI-580, mean Dx = 710.71

• Student’s t-test generates p-value = 0.1736 which is still p > 0.05 and hence not indicating a significant difference.

Will there be much impact on re-setting the boundary Dx value to Dx = 580 on the effect size analysis for Academic Behavioural Confidence?

• Cohen’s ‘d’ effect size difference for Academic Behavioural Confidence mean scores between RGs: DNI <-> DI-580  =  0.5077.

Conclusion

We are left with the conclusion that the differences in effect size between using research subgroup DI-500 or research subgroup DI-580 in comparison to using research group: DI-600 data are very small.

My view is that this exercise in trying to get the best, that is, most statistically substantiated result has been very worthwhile and with results for the effect size working out to be broadly similar in all cases surely this adds weight to the robustness of the analysis process and which suggests that my original boundary Dx = 600 is a pretty good one. So at this stage, I am planning to leave it as this. I will be reporting this exercise in data-tinkering more fully in the final write-up as I am hoping that by doing this, I am demonstrating a good awareness of some of the peculiarities of stats analysis.

• So at this early stage of the data analysis process it is nevertheless reasonable to conclude that there is an effect size of about 0.5 between the mean Academic Behavioural Confidence of students with a known dyslexic profile and those presenting an unknown dyslexic profile based on their respective Dyslexia Index profile measures obtained through this research’s data collection process. How satisfying.