Usability evaluation of travel booking sites

I worked with a partner to conduct a usability evaluation and expert review of two travel booking websites. We conducted usability testing with 8 participants on CheapOair.com and 8 participants on Kayak.com from June 24-28, 2020.

Methodology

Between-subjects study (participants tested CheapOair or Kayak; not both)
Evaluations were conducted using CheapOair's and Kayak’s live, public-facing websites
Sessions were conducted both in-person and remotely using Zoom
Two moderators followed the same facilitator's guide and used an interrupted task-based protocol

Read facilitator's guide

Participants

My partner and I conducted usability evaluations with 16 total participants:

8 CheapOair evaluators; 8 Kayak evaluators
9 men; 7 women
Various ages (ranging from late 20s to mid 60s)
All had traveled in recent memoryThose who had traveled since the COVID-19 outbreak in March 2020 indicated that they drove rather than flew
Had not used the site they were evaluating in past 3 months

Many participants mentioned familiarity with travel booking websites:

For flights: Expedia, Google Flights, Hipmunk (now defunct), Hopper, Momondo, Orbitz, Priceline, Skyscanner, specific airlines (Allegiant, Delta, Southwest)
For accommodations: Airbnb, Priceline, VRBO
For rental cars: Turbo

Rating severity of usability problems

The following 1-3 rating scale was used to rate the severity of usability problems:

1 = Cosmetic problem only: need not be fixed unless extra time is available on project
2 = Minor usability problem: fixing this should be given low priority
3 = Major usability problem: important to fix, so should be given high priority

Rating effort to address usability problems

The following 1-3 rating scale was used to rate the perceived effort required to address usability problems:

1 = Minor effort (e.g., minor fixes to styling or simple interactions)
2 = Moderate effort (e.g., changes to page structure or more complex interactions)
3 = Major effort (e.g., incorporating new data sources, significant changes to IA or interactions)

Expectations

In general, researchers expected Kayak to outperform CheapOair. A few reasons include:

CheapOair has many calls to action on their homepage; Kayak has two
CheapOair displays alternate dates and airports in search results by default; Kayak sticks closer to user’s entered search parameters
CheapOair's search button does not provide obvious user feedback; Kayak's search button provides subtle (but more obvious) feedback
CheapOair has duplicative options for limiting flights by number of connection(s); Kayak provides clear separation between search parameters and filters

Findings

In reality, tasks across the two sites proved equally difficult for participants
Uncovered nearly the same number of usability issues with Kayak (20) as with CheapOair (21)
Participants were most comfortable performing tasks when the steps necessary to accomplish the tasks were similar to other travel booking sites they had used in the past

Read full report