3. Methodology

3.1. Introduction

The primary research conducted for this study was divided into two phases. Phase One provided an automated assessment of UK SME websites using computer software. Exploiting the benefits of a multi–method approach (Saunders et al, 2003; p.99), Phase Two then attempted to mitigate some of the shortcomings of automated testing (DRC, 2004; p.31) by complementing it with user evaluation.

The study was cross–sectional (Saunders et al, 2003; p.96) and analysed the phenomenon of website accessibility at a particular point in time. Employing a deductive approach (Saunders et al, 2003; p.86), the work is based upon the theory that the majority of websites have failed accessibility testing in previous studies and so SME websites may follow a similar trend.

The 2004 DRC study conducted user evaluation on 100 websites with 50 users (DRC, 2004; p.8) and Petrie et al used 15 participants to test 25 websites (Petrie et al, 2005; p.10). Within the resource limitations of this study, it was decided that automatic evaluation of 30 websites and a minimum of 30 user evaluation respondents would be feasible.

3.2. Phase One: Automated Survey

3.2.1. Research Design

Phase One was informed by an etic epistemology, relying upon extrinsic concepts and categories that have meaning for scientific observers (Lett, n.d.) and was based upon the positivist (Oates, 2006; p.283) research paradigm. The research performed an objective investigation on an ordered and regular (Oates, 2006; p.283) world and sought facts of a quantitative nature.

The research performed document analysis on a randomised sample of SME websites, evaluating them against all WCAG Version 1.0. (W3C, 1999a) checkpoints which can be automatically tested. WCAG comprises 14 Guidelines divided into 65 checkpoints. These checkpoints are subsequently organised into Priority 1, Priority 2 and Priority 3. A site which meets all Priority 1 checkpoints is termed A–compliant and represents the bare minimum a site must achieve to be considered accessible. If a site meets both Priority 1 and 2 checkpoints, it is termed AA–compliant and if it satisfies all checkpoints it will be termed AAA–compliant. Due to a lack of accessibility evaluation skills, the emphasis of the research fell upon checkpoint violations, which can be automatically tested. Checkpoint warnings (which indicate that human evaluation is required) were not considered to any great extent.

As Robson (2007; p.28) states, document analysis can include electronic documents such as websites which in this case are considered a primary source of data (Coombes, 2001; p.55). Robson adds confidence to the validity and reliability of this research approach, stating that: carrying out an analysis of the document has no effect on the document itself (Robson, 2007; p.29).

3.2.2. Selection Approach

The research sample will be drawn from the UK population of SMEs, defined by BERR (2009) as having a workforce of fewer than 250 employees. There were approximately 4.7 million SMEs registered in the UK in 2007 (BERR, 2007; Table 2), including nationalised bodies. This is the most recent data available and does not take into account the current economic slowdown. This data does nothing to suggest how many of these SMEs currently operate a website. Therefore this resource did not provide an adequate basis for the sample selection process.

Instead, it was decided to use the directory service provided by the Federation of Small Businesses (FSB) (FSB, 2009a), which lists businesses alphabetically by profession. It was decided that the use of a search–engine based stratified sampling (Oates, 2006; p.97) approach akin to the work of Williams and Rattray (2003; pp.713) might be problematic. With only verified SMEs listed by the FSB, the directory service was considered to provide an appropriate sampling frame (Oates, 2006; p.95–96).

Probabilistic Random sampling (Oates, 2006; p.96) was applied to sequential stages of the directory hierarchy to draw the profession of a candidate at random. The 'True Random Number Generator' provided by Random.org (Random.org, 2009a) was configured to generate a number between 1 and 26 inclusive. The corresponding letter of the alphabet selected a candidate from the first stratum, which was the first letter of the profession listed in the directory.

The list of professions displayed under this letter was then copied into the Random.org 'List Randomiser' (Random.org,, 2009b). The profession displayed at the top of the randomised list was then selected. The directory service does not alphabetise organisations by name and since the research was only concerned with SMEs who have websites, each entry in the result set was viewed in turn starting from the top. The first organisation listed under this profession with a functioning website was evaluated. The procedure then began again from the start.

3.2.3. Data Collection

Due to time restrictions and limited resources, it was decided that the unit of analysis (Aves, 2003d; slide 5) should include only the homepage of each participant and that this would be sufficiently representative of the whole site (Kane et al, 2007; pp.150). In the event that the homepage used HTML frames, the entire frameset used on the homepage was tested.

Once a candidate site successfully opened in the researcher's web browser and was seen at face value to be functional, the full URL was then copied into the evaluation software and tested. Contrary to more complex outcomes used by the e–Government Unit (2005; p.23) the small sample size justified a simple pass/fail outcome, with a single instance of an error failing the whole page. This approach provided an indication of which issues require attention from the website designer and whether the site would present severe accessibility barriers to a disabled user.

3.3. Phase Two: Questionnaire Survey

3.3.1. Research Design

A website which is found to comply with the W3C guidelines could still create accessibility issues in practice, for example, the DRC study found that:

The number of Checkpoint warnings or instances of warnings do not relate statistically to any of the user evaluation measures (DRC, 2004; p.31)

Therefore, the second phase of research used an analytical research (Saunders et al, 2003; p.282) approach. This approach attempted to understand how users actually experience a website and whether WCAG checkpoint violations impact upon accessibility in practice. Data was collected via an online questionnaire which randomly selected a site from the Phase One sample for each participant to evaluate.

Questionnaires are particularly suitable for gathering information from a disparate geographical location (Denscombe, 2005; p.145). The Internet has made this approach highly cost–effective, which to some extent mitigates the typically low response rate.

A self–administered survey reduces the scope for researcher influence by removing the influence of being observed, otherwise known as the Hawthorne effect (Oates, 2006; p.204). The possibility that the researcher may inadvertently influence responses through variations in voice intonation or body language (Oates, 2006; p.221) is also removed. Furthermore, participants are less likely to give answers which they believe are 'correct' when completing questionnaires themselves (Oates, 2006; p.221). However, the researcher acknowledges that shortcomings of this approach include the inability to support participants, opening up the possibility that they may respond flippantly or fail to complete the survey.

Oates (2006; p.228) illustrates the low tedium threshold of online survey participants. The length of the survey was carefully tempered with the intention of discouraging non–participation and was laid out in an attractive format with consistent navigation and content themes. The researcher's college and parent University logos were placed in the masthead to convey a sense of authority (Cornford and Smithson, 2006; p.119). Questions were logically grouped and ordered and attention was given to phrasing to ensure participants were capable of providing reliable responses (Robson, 2007; p.79).

Respondent demographic data was collected using closed questions, allowing participants to answer such straightforward questions quickly (Oates, 2006; p.223). Open questions were used to provide a point of comparison with user experience ratings, lending a degree of construct validity (Oates, 2006; p.227) to the research instrument. The collection of a small amount of qualitative data was also an acknowledgement of the social model of disability, which as Adam and Kreps illustrates is: &ellip;interpretivist in that it looks towards the interpretations of disabled people, as to their experiences (Adam and Kreps, 2006; pp.211).

A small pilot was conducted with 6 participants responding. Most of the feedback was positive with a few minor procedural and semantic improvements suggested. From the responses gathered, the survey was considered to have sufficient content validity (Oates, 2006; p.227) to provide a valid indication of user experience. Problems encountered with the technical implementation are discussed in Section 4.2.

The chosen research approach allowed the participants to use their own computer and assistive technologies, bringing logistical savings to the project and making the research a more valid measure of real–world experiences. Figures from the 2004 DRC study support this approach: 78&37; of the user panel completed their tasks at home with the remaining 22&37; performing their tasks under laboratory conditions. The differences in participant performance were found to be statistically undifferentiated between all groups (DRC, 2004; p.26).

3.3.2. Sample Selection Approach

The small sample size in this study would have made it impossible to draw any meaningful comparisons between disabled and non–disabled groups from a representative population sample. Disabled people and blind people in particular are considered the most disenfranchised web users (DRC, 2004; p.27). Therefore, the sample selection was purposefully skewed towards disabled user groups by using non–probabilistic sampling techniques (Oates, 2006; p.97). However, a homogeneous (Saunders et al, 2000; p.175) sample could have been misleading since it is possible that a website may simply be very difficult for both disabled and non–disabled groups to navigate. Any non–disabled responses therefore formed a control group, allowing for comparisons to be drawn between user experiences.

Charitable organisations were purposively sampled (Oates, 2006; p.98), meaning that they were deliberately picked to represent a range of disability causes, but with particular emphasis on those representing visual and auditory impairments. A total of 25 organisations were approached and asked to complete the survey (see Appendix 2 for a specimen email). They were also asked to partake in raising awareness of the research by promoting it in their newsletter for example. It was the intention that with the endorsement of recognised organisations this snowball sampling technique would add credibility to the study (Oates, 2006; p.98), encouraging participation. The charity employees and their contacts therefore became the sampling frame. Requests for participants were also posted on a small number of web forums orientated towards individuals with disabilities.

Responses offering help were received from 8 organisations, with 3 organisations preferring not to get involved. The remaining organisations which did not respond were sent polite reminders after 1 week. This approach yielded a further 4 responses expressing interest in participation.

3.3.3. Data Collection

The questionnaire was published online, a copy of which can be seen on the questionnaire page. Part one of the questionnaire was designed to collect indirectly topic related (Oates, 2006; p.94) respondent data which was used in the analysis to identify sampling bias. The second part of the questionnaire was a key–task test (Krug, 2006; p.144) which collected qualitative data about each participant's impression of their test website. The respondent was asked to perform two tasks which any user visiting the website would be likely to perform: establish what products or services are offered, and find a telephone number. Following advice from Cornford and Smithson (2006; p.117), this part of the questionnaire used questions based upon a research instrument used by (Petrie et al (2005; p.14).

Respondents were asked to rate their experience of completing each task on a scale (Oates, 2006; p.224) ranging from '1' ('extremely easy') to '7' ('extremely hard') and were then asked to comment on what they felt influenced their experience. To triangulate the answers given to the tasks, respondents were also asked to rate their experience of site navigation, how confident they would be using the site again and to what extent they felt that the site took their needs into account.

3.3.4. Research Ethics

Participants were assured of their anonymity, and no personally identifiable information (for example, participant name or computer IP address) was collected. Participants were informed of their right to withdraw at any time and were not harassed, nor induced to take part (Aves, 2004; slides 7 and 8). The identity of which organisations failed to reply or refused to help raise awareness of the survey will not be revealed.

Back to top