4.1. Phase One: Automated Testing
Most previous research has used ‘Bobby’ evaluation software for automated testing (Kane et al, 2007). The program is now owned by IBM (IBM, 2009a) and known as ‘Rational Policy Tester’. The commercial licence fee was beyond the resources of this project so Fujitsu Web Accessibility Inspector (Fujitsu, 2009) (FWAI) Version 5.11 was chosen instead. This was the only freeware automated evaluation tool listed by the W3C (W3C, 2006) which is capable of producing reports in a Comma–Separated Values (.csv) format — a requirement for the data analysis stage.
The program was configured to test each homepage against WCAG Version 1.0. During this study, WCAG Version 2.0. became an official W3C recommendation (W3C, 2008). However, to enable the results of this study to be directly comparable to previous research, Version 2.0. was not used.
Using the method outlined in Section 3.2.1, each test candidate was loaded into the Firefox 3 web browser (Mozilla Europe, 2009) to ensure that the URL was valid and that the site itself was functioning. Some sites failed to load and some URLs pointed towards an invalid address — these test candidates were not evaluated.
Each functional homepage URL was copied and pasted into FWAI and the test procedure was initiated. The program output was then imported into the author’s spreadsheet software and compiled into a single data set in preparation for analysis.
Each site was archived for repeatability purposes using Free Download Manager (Free Download Manager, 2009). Every publicly available resource at the time of evaluation was downloaded, including HTML, CSS and image files for every page found at the domain URL. Unlike previous research (Kane et al, 2007; pp.151), archived sites were not subjected to algorithmic evaluation since, due to limited resources, the second phase of the research could provide no other way than for participants to evaluate the live sites. The author is happy to provide archive copies of the analysed websites to other researchers upon request.
4.2. Phase Two: Questionnaire Survey
The questionnaire was originally developed using Google Forms (Google, 2009), however given the subject matter of the research it was of paramount importance that the questionnaire was designed for high accessibility. Viewing the form in the non–graphical browser WebbIE (WebbIE, 2009) determined that the questionnaire may have been completely unintelligible to some users. Due to the fashion in which form controls had been labelled by the Google software, the questions and most of the answers were not displayed at all by WebbIE (See Appendix 4 for a screen shot).
Due to these shortcomings, the researcher developed a manually–coded alternative using semantic XHTML for the page structure and CSS to control visual presentation. During development, the webpages were frequently checked using a variety of validation engines including the W3C Markup Validation Service (W3C, 2009) and TotalValidator (TotalValidator.com, 2009) to maintain control over automatically–verifiable coding and accessibility issues.
Once a functional form had been prepared, the questionnaire was then submitted to the Web Access Team at the Royal National Institute for Blind People (RNIB) (RNIB, 2009) with a speculative request for feedback regarding any major anticipated accessibility problems. The response received was far more generous than expected and enabled the researcher to minimise accessibility issues by a significant margin. A copy of the assistance received from the RNIB is included in Appendix 5.
Manual development and the integration of high accessibility required more time than had been scheduled for research instrument development. However, by publishing a highly–accessible survey electronically, the questionnaire had far greater potential to reach individuals who may have found self–administration of a traditional paper–based survey impossible on their own. Unfortunately, the amount of time available for data collection and analysis was impacted to some extent by these allowances.
Although not all questions were compulsory, many of the questions required answers in order for the collected data to be useful. Similarly, some questions if answered would then require an additional response. For example, if the participant indicated that they had a disability, they would then be prompted to indicate their categorisation of the condition or alternatively provide a ‘no answer’ response.
PHP scripts were developed to catch the absence of required responses, or illogical answers. Upon validation failure, rather than being submitted the questionnaire was re–displayed to the participant with problems highlighted using colour and indicative text. Acceptable responses were preserved to prevent user–fatigue and reduce non–response levels. If the user chose not to re–submit the form, none of the responses entered would have been collected, and the participant’s right to withdraw at any time would have been preserved.
PHP scripting was also used to filter out malicious text entries which could have been used by a hacker to manipulate the website or database for nefarious purposes (Hewson et al, 2003; p.121). Upon successful data validation and sanitisation, a PHP script was used to insert the responses into a MySQL (Sun Microsystems, Inc., 2009a) database. Once the survey had been closed, the data was exported from MySQL into OpenOffice Calc (OOC) (Sun Microsystems, Inc., 2009b) format to be prepared for analysis.
Once the data had been imported, OOC was used to codify and count the data. Rating scales used in the questionnaire automatically produced ordinal data (Oates, 2006; p.247) when submitted by the online form. However, an omission by the researcher necessitated codification of the remaining quantitative data, so care was taken to use a consistent range of codes (Cornford and Smithson, 2006; p.131). The imported data was then quickly scanned for erroneous responses (Cornford and Smithson, 2006; p.131). PHP form handling routines ensured that all questionnaires were completed before being stored and so no incomplete responses had to be removed.
The answers to multiple–choice questions were originally stored in single columns in MySQL because there was insufficient development time to implement a more elegant solution. Where multiple responses had been given, these were automatically ‘exploded’ into separate columns for the benefit of analysis using the OOC ‘text–to–columns’ feature.
Discrete data was generated by the first research phase for which appropriate statistical measures were used to find the central tendencies (Oates, 2006; p.254) and distribution of the data. Bivariate comparisons were drawn between certain data such as disability and task difficulty ratings, in an attempt to highlight broad correlations. The sample size was not considered large enough to make the calculation of correlation coefficients (Oates, 2006; p.259) or significance levels (Oates, 2006; p.260) worthwhile.
Qualitative textual data obtained from the open questions was processed using the data visualisation tools provided by the IBM Many Eyes website (IBM, 2009b). Free–text from each question was in turn cleaned up (by removing spelling mistakes and needless punctuation) and then the relevant column was copied and pasted into the dataset upload tool. Once uploaded to the website, the data was first processed using the ‘Wordle’ data visualisation algorithm, which depicts the words which appear the most frequently in the dataset as the largest. When out of context many words have neutral meanings and so the most prominent words were then entered into the ‘Word Tree’ data visualisation algorithm to uncover more specific meanings.