User Research for Taxonomy Part 6: Analyze Your Findings

The following is an excerpt from an ebook I wrote with Rachel Price, User Research for Taxonomy Design. We wrote it as a companion piece to a presentation we gave at Taxonomy Bootcamp 2016, because we think there’s a lack of good information out there about generative qualitative research for taxonomy. In our book, we break it down into simple steps so you can do your own research.

While there are many ways to analyze data, we’re going to use this space to dive into our favorite method: thematic analysis. Thematic analysis is the process of finding useful or pertinent themes in a set of qualitative data. By working through observations to see what themes arise, we can identify patterns of behavior and information needs which we can use to make evidence-based decisions.

Choose your level of rigor

We use three different methods:

  1. A fast and loose collective brain dump
  2. A somewhat-structured closed card sort of notes and observations
  3. A rigorous open card sort of findings
  1. Collective Brain Dump (fast and loose)

Good for:

  • Identifying large and obvious themes quickly.
  • A very small study (3-5 people)
  • Subjects that you’re very familiar with (little translation or interpretation needed)

It works best with as many researchers as possible, because you are relying on everyone’s individual notes and memories to generate content for the whiteboard.

Avoid if:

  • This is an area that you and your team don’t understand well
  • You’re very close to the subject and need to worry about injecting your own biases into the findings
  • You need to have well-documented evidence for your conclusions

How to do it:

  1. Gather your individual notes and re-familiarize yourself with what happened during the interview sessions (hopefully you all took good ones)
  2. Get 2-3 members of the research team in a room together.
  3. Discuss what you remember from the sessions. What patterns emerged? What stuck out? What was particularly surprising, or predictable?
  4. Write these themes on the board, and flesh out your thesis statements about each theme. A thesis statement includes:
    1. The observation, or finding (what happened).
    2. The implication (what should be done, given what happened).

Time investment: 1-2 hours, depending on the size of the study.

  1. Closed card sort style (somewhat structured)

Good for:

  • A study with a strong set of objectives
  • Five or more participants
  • Research where you have good notes from the sessions
  • A domain with which you’re fairly familiar
  • Situations where you need a good record of which observations answered which questions

It works best with 2-3 people working together on analysis.

Avoid if:

  • After doing research, you’re not sure your objectives were the right ones, or you worry they might be limiting you
  • It’s a very contentious topic, and stakeholders might disagree with the primary research questions
  • You’re doing something brand new, and you need big ideas

How to do it:

  1. Gather your detailed notes from the sessions — possibly even transcriptions if you have them
  2. Create note cards out of your notes, with one observation per note card..
  3. Write your original objectives or primary research questions on post-its and put those up on the wall/board as your headings.
  4. Map your observation note cards to each of those objectives, creating clusters of note cards under each post-it heading. What are the answers to your research questions?

This process goes much quicker than option 3 when you’re working within a familiar domain, because your findings shouldn’t require a lot of interpretation. This is an especially useful way to go about analyzing when you’re in the middle of a series of projects dealing with the same domain and similar audiences.

Time investment: 1-2 hours for a small study (4-6 participants), up to 4 hours for a larger study (8-12 participants).

  1. Open card sort style (most rigorous, most reliable)

This is a true ‘Thematic Analysis,’ in a way that wouldn’t make actual social scientists mad.

Good for:

  • Truly generative research, when you need fundamental ideas about how something works
  • When you’re trying to identify a problem out of a morass of patterns of behavior
  • A study that had particularly surprising results
  • When you need really solid evidence for any conclusions you might make
  • Situations where you need to ensure as much objectivity as you can

It works best with 3-4 people doing the analysis.

Avoid if:

  • Your time is limited
  • You don’t have enough people who are familiar with the sessions
  • Your notes aren’t very good

How to do it:

  1. Get your team together in a room with lots of empty wall space. Turn your detailed notes into observation notecards, with one observation per notecard. Note: This process requires very accurate notes.
  2. Take your observations and start clustering them on the walls, like an open card sort. Put like notes together, and see what themes arise out of the clusters.
  3. Identify a single thesis statement for each theme. A thesis statement includes:
    1. The observation, or finding (what happened).
    2. The implication (what should be done, given what happened).

This method is a lot of work, but with huge pay-off.

Time investment: 30 minutes per participant (this will change depending on your experience and how many participants there actually were). We assume it takes at least 6 hours for a study of 12 people.