CA Health Data Project

Asthma Data Visualization Product Development

Sacramento Health Data Ambassadors, in partnership with IDMLOCO, are developing a website that visualizes emergency department visits for asthma by county and zip code.

An Iterative, Agile Process

Our goals for this project are to only demonstrate best practices for effective communication through data visualization, but to also document an iterative, agile software development process. In other words, the process is just as important as the end product. Users of the website will be able to download raw indicator data, manipulate geographical displays, and obtain other contextual information. By focusing on collaboration, we hope to create more durable products, and engage CHHS so that they can learn too, about how to integrate this concept into their own technology work.

View the Final Product

Sprints

In agile development, weekly or biweekly sprints are the frequency at which new, working features are released for the purposes of validating with users. At the end of each sprint, a standup meeting (or call) should occur to reflect, adjust, and iteratively refine the product roadmap as a better understanding of user needs emerges across all stakeholders. Because the process involves iteratively determining features as the product is built, it is not advisable to specify interim deliverables at the outset of the project.

Completed

Sprint 1: Monday, June 29 to Friday, July 10 (2 weeks)

  • Research asthma data and stakeholder needs/use cases
  • Brainstorm 3-5 written data visualization concepts with wireframe examples
  • Initial development exploration to determine feasibility of data visualization concepts

Slide Deck 1st Mockup 2nd Mockup

Sprint 1 Feedback Summary

Pre-Sprint Survey

Prior to commencing site development, we surveyed participants in the Sacramento Health Data Roundtable. Among other questions, the survey asked for their reaction to the proposed asthma data visualization site.

The Survey

Sprint 1 Interviews

During Sprint 1, we interviewed three health/asthma data stakeholders to learn what they would find valuable in an asthma data visualization site. We showed them our initial wireframe concept for the site and asked for their reaction. Summaries of those interviews are below.

Local public health officer

  • Loves the wireframe
  • ED visit data is heavily skewed by certain populations using the ER as their physician - will need to include careful language explaining this
  • From a superuser perspective, data should be meticulously labeled and cite sources, and be at the finest geographical level possible
  • Would like to see overlay with risk factors - outdoor air quality, indoor air quality (smoking, mold)

State team responsible for measuring and reducing asthma

  • Very positive on the wireframe - “looks great, user friendly, easy to read” - they have all the data, but they know their site isn’t beautiful
  • In addition to population behaviors, ED data is skewed by different hospital policies as well. ED visits are in between prevalence (only available from survey data) and hospitalization.
  • Would like to be able to filter map by ethnicity
  • Would like to see overlay with risk factors - three have strong evidence behind them: PM, ozone, and smoking
  • Should be able to provide us with detailed data: multiple years of county data broken down by age and ethnicity, single year of ZIP code data broken down by age and ethnicity - but may be delayed by key staff vacation

State public health officer responsible for data and statistics

  • Also loves the wireframe
  • ED visits should be interpreted as a failure of treatment
  • Site should be useful to asthma sufferers, not just the general public and the local health agencies:
    • Having first attack: link to signs and symptoms, risk factors
    • Sufferers without regular care: discussion of the purpose and value of care, how to get it
    • Providers: link to best practices in asthma care
  • Needs a “so what” at the bottom. What can you do with this information? What will you do with this information? Option to contact us to tell us what you’ve done with this information.
Completed

Sprint 2: Monday, July 13 to Friday, July 24 (2 weeks)

  • Perform major feature design for up to two data visualization concepts
  • Perform major feature development for one data visualization concept
  • Possible design and development work for additional concepts if feasible under project budget and timing constraints

1st Iteration 2nd Iteration

Completed

User Testing: Monday, July 27 to Friday, August 7 (2 weeks)

  • Convene user testing group to collect user feedback
  • Distill feedback and validate/refine product roadmap

User Testing Session

User Testing Period 1 Summary

User testing period 1, from July 27 through August 7, 2015, consisted of three parts: a recorded user testing session, a survey asking interested stakeholders to rank proposed additions to the site, and an open-ended request for email feedback.

User Testing Session

Conducted with an interested member of the general public, the recorded user testing session is available to watch in its entirety here.

Ranking of Proposed Additions

We asked interested stakeholders to rank the following proposed additions to the website, from most to least valuable.

  • Data Expansion - Risk Factor Overlays: Acquire external risk factor data sets (ozone and particulate matter emissions and smoking have been suggested) and incorporate into data portal. Allow users to toggle these risk factors on the map tool to visualize interplay between asthma ED visits and risk factors.
  • Data Expansion - Trends: Acquire asthma ED visit data for all available years and incorporate into data portal. This will allow users to understand the trajectory of asthma ED visits at the statewide and local levels.
  • Data Expansion - Ethnicity: Acquire ethnic breakdowns (available by county) of asthma ED visits and incorporate into data portal. Will enable additional map options to visualize data. Will also enable additional visualizations such as small multiples for rapidly comparing a large number of geographies.
  • Advanced Analysis: Custom statistical analysis/comparisons/data processing to illustrate concepts like potential paths to achieving the CA 2022 0-17 goal (which counties/zips need to improve and by how much). Analyses would be guided by policy experts within our stakeholder group to ensure they are appropriate, helpful and accurate.
  • Advanced Sharing: Implement social and email sharing features for the overall asthma site and within individual visualizations. This will allow users to input settings into site features like the map tool and share a link via email/social to that specific version of the map. Useful for sharing insights and key messages around the data.
  • Advanced Content Management: Explore API/news feed connections with related asthma content providers to automate updates with relevant information from trusted sources. Explore integration opportunities with existing CaliforniaBreathing.org content management system to plan for long term sustainability.
  • Map Improvements: Your suggestions for making the map tool useful and functional for your purposes. We want it to be a tool that a range of users will enjoy using, and even need. What have we not offered or considered?
  • Customized Next Steps: Position the site as a jumping-off point for various user personas by adding links and information customized to their needs. User personas may include new asthma sufferers (links to symptoms and general advice), reporters (asthma expert contact information), and researchers (links to additional asthma-related data).

Numerical rankings from the six respondents (or respondent groups) are summarized in the table below. Other than respondent D, who was the outlier, all respondents ranked adding ethnicity and trend data as #1 and #2. Respondents C and E, who work specifically with asthma data, were very skeptical of risk factor overlays, indicating they were “too simplistic” or “might lead to erroneous conclusions”. Respondent F did not submit a survey, but indicated a strong preference for advanced sharing in a phone call.

Feature Average Ranking
Data Expansion - Ethnicity 2.40
Data Expansion - Trends 2.80
Advanced Sharing 4.00
Map Improvements 4.25
Customized Next Steps 4.50
Data Expansion - Risk Factor Overlays 4.80
Advanced Content Management 5.00
Advanced Analysis 5.75

Open-Ended Feedback

We also asked a number of stakeholders for open-ended feedback. Here follows a summary of a conference call and relevant excerpts from responding emails, anonymized to the respondents’ roles.

Health Data Project Participants, July 30 conference call

  • May be too much text at the top, want to see the map immediately
  • Would be interesting to see data by hospital area. Failing that, just showing hospital locations could be interesting. Ask CA Breathing team about possibilities.
  • Advanced sharing options highly desirable: ability to share your version of the map, possibly embed in your site. Ability to send a "datagram" to your legislator.
  • Slide show describing status of specific areas, or labels on the map, to tell more of the story. Option to leave comments regarding specific areas could be interesting. Map labels could show ethnic breakdown of selected area.
  • Histogram needs more explanation.
  • Once we receive other stakeholders' priorities, we'll summarize and share.

Interested Member of the General Public, August 10 email

Just fyi, ALAC has a lot if this data. They use it to compile their state of the air reports. My rankings are below. I also sat with Ash yesterday and walked through the site, and it is recorded. I made some suggestions on how to improve the map and analysis tools. I think audience is key for this: how would a researcher, a political/policy staffer, and advocate, a community member use this. I thought some "helper" visualizations might be cool and assist lay audiences with understanding the tool better.

Helper visualizations example rocklin.opengov.com (you need to view in private mode or clear your browser cookies if you’ve ever visited an OpenGov site):

Evaluation Expert, August 10 email

  • Typo - 2012 Asthma Emergecy Department – missing the “n”
  • Typo - 2012 ED Vistits – an extra "t”
  • Love, love, love the download feature!
  • I suggest an explanation as to what “context” means – I do not understand what the graph is representing or saying. Maybe an axis descriptor would help?

Asthma Epidemiologist, August 10 email

1) The CB website is printed at the top of the page, but that's confusing because it makes it seem like it's part of their program website.

2) We thought it would be more clear if everywhere that the 2022 target is mentioned, that it could be change to LGHC (or spelled out) target so it's totally clear what target is being referred to.

3) The 'context' section could use a little bit of explanation of what you are seeing below. Perhaps it will help just to label the LGHC target, or maybe it'll be more clear when the whole site is built, but right now it seems like it needs more context (no pun intended).

4) My understanding is that there is only a LGHC target for kids 0-17, so what is the target referring to when someone selects the other age ranges?

5) It would be nice for there to be some notes or other metadata to give context to the data (e.g., the source) - especially when the data are downloaded.

[Reasons for low rankings of two development options]

7) advanced analysis (mostly because I'm unclear about what this really means and what it would look like)

8) risk factor overlays - this is something we try to stay away from as an epidemiologist because it might lead to erroneous conclusions; also, this is already available (not as overlay, but the information) on the Tracking website.

State Public Health Analytics Manager, August 10 email

Great work! The overview is visually simple, easy for the lay reader to understand. In the right side text box, the work “Visits” is misspelled, although you’ve probably already picked this up.

The map is also visually simple and easy to interact. I admit I don’t get the Context section, what’s your y axis? Perhaps your most useful tool is the link to the CaliforniaBreathing.org site for the state Asthma Plan, I think this will be very helpful to a wide variety of audiences!

Suggestion: I would think it may be helpful, if the data is accessible, to add # of primary care asthma treatment sites (as a potential alternative to ED asthma care per county) to accentuate lower cost access opportunities.

Keep up the great work!

Completed

Sprint 3: Monday, August 10 to Friday, August 21 (2 weeks)

  • Perform feature refinement, product development, and bug fixes
  • New feature development is not included in this sprint unless feasible under project budget and timing constraints

3rd Mockup

Completed

Sprint 4: Monday, August 24 to Friday, September 4 (2 weeks)

  • Perform minor feature refinement
  • Extensive testing across devices and platforms and bug fixes identified through end client review

3rd Iteration

Completed

User Testing: Monday, September 7 to Friday, September 11 (1 week)

  • Convene user testing group to collect user feedback
  • Distill feedback and define minor feature improvements

User Testing Period 2 Summary

User Testing Period 2, from September 7 through September 11, 2015, consisted of an open-ended request for feedback on the largely completed asthma site. Emailed responses, anonymized to the respondents’ roles, are copied below.

Health Data Project Participant, September 10 email

Bravo! I've been spending some time reviewing the site since you sent it. Amazing to see this all coming together. I really like the curated stories and the ability forfolks to easily share them. Quick question: Is it possible to share the actual slide on Twitter as a photo, so it shows up when you tweet?

It was really fascinating to see the mis-classifications of of AI/AW as another race. Some really good insights for folks coming to the site overall to make sense of the data.

A couple things -

1) In the intro paragraph I think it might be good mention the interactive map itself so people can search. I know we have it on the left. But, maybe a line and a link to the map to reinforce the interactive nature. I think it's helpful to reinforce that folks can search on their own as well.

2) The zip code functionality does not seem to be working. When you switch over to zip codes nothing is showing up. Also, we talked about having a search tool, so you can type in your zip code or county. Is that possible to add?

3) I see ethnicity data in the overview, but the option is not available in the interactive map. Wasn't possible to add?

4) At the end of the overview slides, do we want to have one more with the source of this information, or as a footnote on one of the slides?

5) Finally, how about adding something to the footer to give CHHS some love for helping create the app in an agile fashion? Could say something like this...

A product of the California Health Data Project, a statewide effort funded by the California HealthCare Foundation’s Free the Data initiative in partnership with CHHS.

Probably too long I know. But, want to highlight them.

Also, I nudged the state folks to review the site and get back to us feedback.

Great job!!!

State Public Health Analytics Manager, September 10 email

Huge ‘home run’ from my point of view! Well-designed, intuitive navigation, interesting and informative! Thanks for sharing.

Health Data Project Participant, September 11 email

I think this is looking fantastic – an excellent model of how a curated data story can be told, then shared via Twitter, etc. Congrats to all of you on a great product and great process leading us here! It’s a good model not only for LGHC but for other types of health data storytelling. To that end, when it’s ready, I’d love to share this with the counties and grantees we’re working with on improving county health departments’ capacity to tell stories with data.

I saw XX’s comments, which makes sense to me. Seems like from my perspective only some additional minor tweaks are needed. A few edits/Qs:

  • For the focused, zoomed-in maps, is it possible to darken the names of cities, in order to help orient the user to where they are. That could be useful for the LA area map, for example.
  • Related to that, for the Imperial Valley map, can you also show a bit of SD County as a contrast?
  • Just want to double check: Are the terms we’re using for racial/ethnic groups okay? If they’re consistent with what’s on the CDPH dataset, we should be fine. I just didn’t know, for example, if we say Blacks or African American.
  • Not certain, but I wonder if that trend graph may have more impact if you focus on just one age group (perhaps 5-17?), in order to eliminate some lines and make it look less cluttered. See what you think…
  • Also, you note American Indians, but I don’t see them appearing on the graph.
  • The slideshow story seems to end somewhat abruptly. Not sure about this, but what do you think about re-casting some of the info in “where do I go from here” into a final slide?
  • For some reason, the zip code-level interactive map wasn’t working for me. It wasn’t showing any data.
  • If someone wanted to, could they embed the entire slideshow onto their website or blog? Maybe that’s understandably not feasible at this late stage (the idea just came to mind for me).
Completed

Sprint 5: Monday, September 14 to Friday, September 25 (2 weeks)

  • Perform minor edits and bug fixes
  • Final delivery to end client

4th Iteration