The role of automated accessibility testing
Posted in Accessibility on: Wednesday, September 21, 2005 by Paul Boag
Many tools on the market automate the process of checking for website accessibility. However, there are some serious question marks over the value of such tools.
There are many different reasons why automated checkers have limited value. However, the forthcoming Government Guidelines on Accessibility provide a very neat summary:
"…automated tools are like spell checkers – they look for obvious problems within a web page, and then generate a list of possible problems. They cannot give a straightforward statement of whether your website meets certain accessibility standards. The list of possible problems needs to be interpreted by an experienced person and matched against what your site is actually doing. There is a substantial list of accessibility issues (at least 50%) that cannot be assessed by current automatic tools…"
Subjective decision making
The Government Guidelines on Accessibility show us that automated checking alone cannot be trusted. Computers are great at answering questions with yes/no answer. They are not so good at making subjective decisions. So for example a computer can easily tell you if an image has an associated alt tag but it cannot tell you if that alt tag is really descriptive of the image or not. As is stated above at least 50% of the WAI guidelines require subjective decision-making and so require a manual check.
It is this need for manual checking that undermines the primary, timesaving benefits of automated tools. An automated checker can scan a page and give it the "all clear" but you still need to visit that page to ensure it conforms to the subjective checkpoints.
Can even the automated checks be trusted?
It is also important to question the reliability of checks made by automated tools. I believe that practically all of the checks made by accessibility checkers also need to be checked manually. This is because automated tools are based on certain assumptions. The algorithm that the tools uses to assess a website are entirely dependent on the developers own interpretation of guidelines which are often entirely subjective.
Web accessibility can be very complicated. There is much discussion about the interpretation of guidelines and often it is only the end user that can decide whether a site is accessible or not.
When an automated tool flags up an error, it is the developer’s interpretation of the checkpoint that is being tested, and not necessarily the checkpoint itself. It is important when using automated testing tools to have an informed opinion on all web accessibility issues in order to be able to verify results.
Some accessibility issues are not covered by WCAG guidelines
It is possible to create a website that complies with WCAG guidelines and still presents accessibility barriers. A site that has text that is not fixed in size but scales between "1pt" and "4pt" technically meets Web Content Accessibility Guidelines. It will incidentally pass through most automated testing tools. Yet it would not only make the site inaccessible to disabled people, it would be inaccessible to most people. Ironically, the only people likely to be able to use the site without altering their browser settings would be screen reader users who would not be affected by text size. So while measuring accessibility using WCAG guidelines is undeniably the best starting point, there is more to accessibility than a list of checkboxes.
There is still a place for automated testing
So is there no place for automated tools? Well personally, I cannot bring myself to claim they are redundant. After all, my first tentative step into the world of accessibility was to use Bobby. If it had not been for that automated checker I could well have been put off by the intimidating WAI checkpoints. Surely, if all you do is check your site using an automated tool then this is still better than doing nothing at all. The danger is that you never move beyond that and recognise that web accessibility is a much more complex and subjective than a set of automated checkpoints.
My thanks to Ian Dunmore of Public Sector Forums and Grant Broome from the Shaw Trust for their contribution to this post.
4 Comments
Comments are for the discussion of this post. If you have other questions / comments then post them to the forum or send me an email
Although they can be redundant and tedious, I agree that autocheckers for Accessibility are better than nothing. You can also try the checker Cindy Says, which is more direct than some others. I believe one should just be as knowledgeable as possible on the subject when validating for Accessibility, that way he can more easily go through the manual checks and confirm the automated ones. Of course, the best method would be having several impaired users go through the site with different user agents and input devices.
To be honest I am not convinced there is such a thing as the “best method”. The trouble is that there are so many types of disability, so many platforms and so many devices that its impossible to test on everything. I think all you can do is take a long hard look at your users and make decisions against that. It is a very difficult subject with no clear cut answers
The one thing I find a little annoying with the automated checkers is the inability of them to work out what is an image and what is not.
So when it comes back and tells you not to use Pixels, and you check the line, it can turn out to be an Image..!
Whilst those of use experienced in the use of this checker can simply ignore this as a blip, I know a number of people who find the output from these automated checkers confusing enough, let alone when it tells them not to use Pixels for an Image width and height.
In an ideal world we would all have the time and resources built into projects to check for a wide range of devices and users, but for most, it is based on experience, and the shared knowledge we gain over time.
Who can afford to spend nearly £1000 on a speech reader to check a site..? I know I can’t, and rely on the timed trails on a number of them to try out sites and try and remember how it worked best.
Although image how this impacts on the very people who need the software just ot use their computers…maybe it is time this specialist software was massively reduced in cost…surely the Bill Gates of this world could spare some funds to reduce the cost to diabled people in using IT, afterall it helps everyone of us who are disabled in so many ways.
Yes the automated checkers can only do what they are programmed to but there are so many different situations that only manual check can help, cause we can’t foresee everything.