UX Research | Accessibility
Improving the accessibility of interacting with research papers on arXiv.org through in-depth UX research
👨🦼
🧑🏻🦽
👨🏼🦽
👩🦯
🧑🏼🦯
👨🦯
👩🦼
🧑🏻🦼
👨🏽🦼
👨🦼 🧑🏻🦽 👨🏼🦽 👩🦯 🧑🏼🦯 👨🦯 👩🦼 🧑🏻🦼 👨🏽🦼
Project Type
Research Project
Duration
Aug 2022 - Present
Team
4 UX Researcher
My Role
UX Researcher
PROJECT OVERVIEW
Research Objective
🎯 Improving accessibility and user experience for users with assistive technologies on arXiv.org
About arXiv: arXiv is an open-access repository of electronic preprints and postprints (known as e-prints) approved for posting after moderation, but not peer review. In scientific publishing, arXiv has played an important role in open access for over 30 years by removing financial, institutional, and geographic barriers to research.
What’s the problem?
The research content hosted by arXiv is not fully accessible to everyone due to disabilities and other barriers. This matters because a significant proportion of people have reading and visual disabilities, it is important to our community that arXiv is as open as possible, and if science is to advance, we need wide and diverse participation.
Who did we study?
Our research took two forms, a quantitative survey and a series of qualitative interviews.
We undertook to research the experience of people with a variety of disabilities and other barriers as it relates to accessing research articles, on arXiv and beyond.
We also gathered input from experts in different fields: accessibility researchers; writers of web standards for accessibility, Math, and PDF; TeX and LaTeX experts; developers of screen reader and other assistive technologies; scholars of accessibility law; and science communicators.
What did we find so far?
RESEARCH PROCESS
QUANTITATIVE RESEARCH
Survey
Demographics
We invited two groups of people to take this survey: researchers who directly rely on assistive technology, and professors who assist their students in accessing the research and data they need. Out of a pool of 275 volunteers we had a response rate of 18% and received a total of 53 individual responses.
Our respondents were primarily frequent arXiv users. 58% use arXiv daily and 25% weekly.
Respondents come from various, but not all, geographic regions. Europe had the highest representation at 44%, followed by Asia at 23%, North America at 21%, South America at 8%, and Africa at 4%. We continue to work towards greater global representation in our surveys
Respondents represent a variety of research fields:
Survey Data Collection
The assistive technology reported in use by our respondents are:
Access to Research Output
Our respondents heavily depend on access to research, but users of assistive technology report they only have access to 38% of the research they need without assistance.
Our respondents heavily depend on access to research, with 89% saying that research is completely or somewhat essential to their professional work:
We next asked what level of access respondents have today, without requiring assistance from others. Overall the numbers were high: 89% report having access to all or most of the research they need without assistance. However, those numbers look quite different if the respondent uses assistive technology, with only 30% reporting access to all papers without assistance:
What are the preferences and barriers related to accessing content on arXiv?
Barriers to access
Survey respondents agree that PDF formatting is the biggest barrier. The main reason reported for not submitting accessible papers is a lack of understanding around requirements.
When asked what the biggest barriers were to accessing papers, PDF formatting topped the list:
We asked respondents the reasons that stopped them from submitting accessible papers. For users who do not use assistive technology the top reason was not knowing what an accessible paper requires, while for users of assistive technology, it was that they consider their papers to be accessible already.
Preferred format
Respondents who use assistive technology preferred HTML, while those who didn't preferred PDF. Use of specific types of assistive technology, such as screen magnification and color and contrast remediation, correlated with a strong preference for HTML.
When asked what their preferred format would be if well formatted HTML was available, 67% still indicated PDF would be preferred; among assistive technology users a small majority of 55% would prefer HTML:
Interestingly, respondents with specific barriers indicated a preference for HTML: those who need to adjust font size, color, or contrast; all barriers which indicate a vision impairment.
Also preferring HTML are professors who help students with translation or by describing images and charts.
Participants: 44 individuals, including include researchers with reading disabilities or other access barriers who use arXiv, professors who help students with disabilities, researchers whose focus is on various fields of accessibility, experts on standards for web content, and leaders in LaTeX, MathML, and other languages critical to the success of this project.
Recruitment: accessibility survey, two accessibility mailing lists, direct invitations, and word of mouth
Interview method: semi-structured interviews with individuals or small groups using video conferencing tools, and in two cases in person
Demographics
Fields of research: physics, math, statistics, computer science, legal, and regulatory. A number of participants who serve on various W3C boards related to accessibility and web standards.
Career stage: PhD students, professors, and researchers working in industry
Disability affected their access to research: blindness, dyslexia, ADHD, and movement disability
QUALITATIVE RESEARCH
Interviews
What are the user needs and challenges when interacting with research papers?
Transcripts were broken down into observations, then documented following Atomic Research principles by mapping each observation to a semantic layer to facilitate discovery.
During analysis we looked for similarities, grouped them into themes, and asked ourselves ‘what are the opportunities for improvement within this feedback?’.
Themes were also compared to data from the survey and analyzed for disparities and correlations.
User Journey Map
What are the typical steps and actions when interacting with arXiv?
To evaluate the user journey we sorted feedback into five primary user goals: Find Research, Read Research, Participate in Scholarship, Prepare my Document, and Submit.
When analyzing each of these steps we asked: “when it comes to accessing research, was the experience of the participant positive or negative?”
Each bar in this positive/negative bar chart represents one experience shared by a participant, while the height of each bar represents it’s impact.
It is immediately clear that Read Papers elicited the most observations, and most are negative experiences of high impact.
Key Insights from Research Synthesis
The PDF format as a barrier.
The benefits of HTML as a format.
Skepticism on the potential for real change.
arXiv has a role to play in improving the accessibility of research papers.
HTML is just the beginning.
Additional Analysis: HTML vs. PDF
SYNTHESIS
Affinity Diagram
What are some common themes we discovered in the research?
The level of accessibility of research papers is low, and we cannot claim to have achieved truly open science while those with disabilities are barred from equivalent access.
Based on provided feedback, we have rated how well we expect the PDFs that arXiv generates now will compare to the well-formatted HTML that our plan will generate to see how they score on a number of criteria.
arXiv assessment of HTML and PDF
Scale: 0 = non-functional, 1 = OK, 2 = good, 3 = excellent.
Based on our user research the step our community wants arXiv to take is clear: offer well formatted, accessible HTML alongside existing sources
