User research topline Sprint 3, March 2021

User research topline

Sprint 3, March 2021

Executive Summary

We conducted research to explore these design hypotheses around the concept of personalized dashboards for NRM G&A users. Here’s what we learned in answer to the following overarching research questions:

1. Will the natural customization enforced by the fact that different roles and individuals are assigned to pay attention to different things result in the right level of role-based customization?

   Yes, but there are limitations. People hold multiple roles and play different roles on different proposals / grants / agreements, which is a vote against dashboards designed for specific roles and a vote for personalization based on what an individual may be tasked to watch or manage. However, not all of the relationships that would be useful for surfacing relevant information are actually in the existing database (for example, when someone wants to watch an agreement when they’re covering for someone else in their region).

2. Does the landing page we’ve designed successfully guide people to the action(s) they need to take within an acceptable number of clicks

   Yes and no. It was helpful to surface actions that could be taken on an agreement, but we need to design to support multiple actions that may be possible across multiple roles at any given time. And while the flow we’ve designed for folks to get from the quick view to the actions they need to take would be helpful, how we labeled those actions sometimes didn't make sense.

3. Do the list views we’ve designed have the potential to meet people’s needs? What’s missing? What’s on there now that’s not useful?

   There’s potential, but a lot of room for improvement. We validated that certain pockets of information we suspected were valuable actually are, like tracking status and seeing agreements relevant to an individual. But, we identified many ways to improve how this information is structured and presented. This research helped us identify user needs to guide and prioritize work on the next iteration.

User stories to de-risk in the E&I

We recommend which user stories should go straight into the vendor backlog and which of these the 18F should explore in the near term including low /med/high priority rating. Priority is based on a rough estimation of the scope and severity of the impact to users and on how well exploring it it will allow us to test the technical limitations of the existing database.

High priority

• As a Program Manager / Budget Approver / Grant Technician / Grants Specialist, I need direct paths to complete actions specific to my role on a particular proposal, grant, or agreement when my action is required to move it forward.
• As a G&A user, I need to see at a glance whether a proposal, grant, or agreement that I’m assigned to requires action by me or by someone else so that I can either complete the action or raise it to someone else’s attention.
• As a G&A user, I need to be able to see proposals, grants, and agreements that I’m not assigned to but that I’m interested in so that I can cover for my colleagues when they’re gone, and make sure things don’t fall through the cracks.
• As a G&A user, I need to be able to see a proposal, grant, or agreement’s status, or where it’s at in the process, so that I can decide whether and how to act.

Medium priority

• As a G&A user, I need to see proposals, grants, and agreements that are created by me so that I can focus my attention and make sure they move forward.
• As a G&A user, I need to see grants, and agreements that I’m assigned to so that I can focus my attention and make sure they move forward.
• As a technician, I need a direct path to contact who’s responsible for moving a proposal forward so that I can make sure it doesn’t get stuck.

Low priority

• As a G&A user, I need to be able to distinguish at a glance items that are due soon from those that are overdue so that I can be sure I address what’s most urgent first.
• As a G&A user, I need to be able to filter so that I can see proposals and agreements by various states of pending in a single view.

User stories for the vendor backlog

• As a Program Manager / Budget Approver / Grant Technician / Grants Specialist, I need tasks specific to my role on a particular proposal or agreement to be written in language that I understand.
• As a G&A user, I need to be able to indicate which proposals, grants, and agreements that I’m not assigned to that I’d like to be notified about so that I can cover for my colleagues when they’re gone, and make sure things don’t fall through the cracks.
• As a G&A user, I need to be able to modify my selections re: proposals, grants, and agreements that I’m not assigned to but that I’d like to be notified about so that I can remove them from my list when they’re no longer relevant to me.
• As a G&A user, I need the information on my region, unit, and forest that’s embedded in agreement numbers to be scannable so that I can quickly find what I need.

Insights and recommendations

Overall impressions

People generally remarked positively on the look and feel of both prototypes and on the apparent pace of the team’s progress:

“I like what you’re doing, it’s going to be way more user friendly than what we currently have in NRM.” - Program Manager

“Looks like you did a whole lot of work in a couple weeks.” - Budget Approver

“First off, I like it.” - Grants Specialist looking at mocks

“I do like this very much, it would be great if you could have this tomorrow as an update, thank you!” - Grants Specialist looking at mocks

“I like the filter … the filter is perfect… The search is picking everything up, I like that a lot” - Program Manager looking at Django admin app

Much of the critical feedback we received was around details that detracted from the mocks’ verisimilitude, typically related to folks seeing actions that it seemed like they could complete but that they wouldn’t actually do given their role.

Showing people agreements “assigned” to them is helpful, and they also need to track items beyond what’s formally assigned to them.

Different roles generally care about grants and agreements in different statuses. For example, we talked to a tech who is primarily involved when an agreement is pending, whereas a program manager is focused on post-award management tasks for agreements that’ve been executed.

People pick up other people’s work which has an impact on how people think about “my agreements.” These kinds of “covering” relationships may not be reflected in the database and may be hard to surface.

Program managers, Grants Specialists, and Budget Approvers talked about having 100-300 items in their active agreements list at any given time, but they also watch or have access to agreements for their entire region, forest, and/ or unit, in which case the list of active agreements could be many more hundreds or even thousands of rows long.

Recommendations

• Provide a way to organize records based on whether a user is responsible for or is merely watching/monitoring a grant or agreement.
• Explore the usefulness and feasibility of providing a way to assign others to watch and/or work on your agreements
• Explore the usefulness and feasibility of allowing users to customize preferences on whether / how they get notifications for grants and agreements they’re “watching” or covering on.
• Consider how we can make long lists of 100+ agreements more navigable (e.g. filters, list search, sortable columns) based on how people using G&A need to navigate them (e.g. an agreement with a certain cooperator, an agreement with a certain status)

A quick path to multiple actions from the list page would be useful.

In our mocks, we guided people to act in two ways: an action column in the due soon table, and via links to multiple actions from the quick view.

It’s common that more than one action may be needed on a given agreement at a time. For example, quarterly invoices may need approval at the same time a performance report is due, but we did not design for this in the action column.

Several people were distracted by seeing what appeared to be routes completing actions that they aren’t personally responsible for or can’t personally complete given their role (e.g., upload invoices is always handled by ASC). But, at least one person commented it would be useful to quickly see what actions were needed so that they could remind the person who’s holding things up.

A few people noted that the notice that funds were expiring and the prompt to begin closeout would address a known pain point. People also noted that a quick path from an active agreement to a pending modification to that agreement would be particularly useful.

People expected that clicking on an action in the action column or in the quick view would take them to the place in the existing G&A app where they could complete that action. For example, people expected review invoices would take them to the attachments tab in the current application and open the specific invoice attachment requiring approval.

Recommendations

• Develop a template that can support folks choosing a quick path to multiple actions from the list page (could consider relying on the hover over quick view for this).
• Consider giving folks a quick way to get details on who and how to contact the person responsible for an action needed to move an agreement forward to the next status.
• Explore how to gather and surface info to users on actions needed that are required by them vs. required by others to complete
• Refine action microcopy to match users own language

The timing for when something is “due soon” or requires urgent action depends on the type of action needed.

In general 90 days for what is due soon met people’s expectations. Exceptions: Program Managers are supposed to certify payment within 7 days of receipt, and invoices are supposed to be approved in under 14 days; people get multiple notices that an agreement is expiring at 90, 45, and 30 days before the expiration date.

One person said that using the word “soon” implies that it could be put off, and two people commented that “due now” and “overdue” or “late” would be more helpful. Another person commented that showing the number of days until an item was due and the number of days an item was overdue would be more valuable than a due date.

Recommendations

• Explore visually distinguishing actions due soon and a hierarchy to prioritize them.
• Explore distinct visual indicators for due soon and overdue items.
• Continue research on what items have formal due dates and if there are items that don’t have formal due dates, but need notification of inaction (e.g. no invoices submitted in past 6 months)

When it comes to people feeling confident in what action to take, “status” is queen.

While people generally liked the concept of being able to see in one place what needed their attention, due soon as a status on equal footing with where an agreement was in the pipeline (active, pending, closed, etc,) was not intuitive.

We observed a pattern of folks noticing what’s due soon and immediately wanting to understand the status--”where it’s at in the process”-- of a due soon item in order to decide what to do next. While the actions listed in the action column in the due soon table we designed hint at this status, it was not explicit enough for folks to feel confident moving forward from there.

Recommendations

• List items that are due soon with those that don’t currently require action (eliminate due soon as a separate page or table).
• Provide clear filters that help folks filter by status in language that they understand.
• Include a column for status in the default view.
• Explore how status and/or notifications are different for more defined workflows. (pre-award) vs. longer more collaborative workflows (post-award management).

People want status information to be persistent and explicit.

People need status as one of the first few columns they see by default. They also need to be able to filter by status.

We tried a few different ways to filter by status in our prototypes. In the Django admin app, we allowed people to filter by an agreement status, surfacing some of the statuses in the current database; In the mocks, we used different pages and tables to filter things into a few high-level status categories: active agreements, pending agreements and modifications, draft agreements, closed agreements, and rejected agreements.

In using the Django admin app, people appreciated the agreement status filters. But there were more filters than folks expected to see. Several people commented that they tend to think of their work as falling into a few higher level categories by which they’d want to filter: executed, pending, and closed. One participant pointed out that having such fine grained filters without the ability to group them or select more than one would make it hard to find pending agreements in particular. Several of the filters describe different phases of “pending,” and you’d have to know precisely which state it was in in order to find it.

In using our mocks, people did not readily understand that the different pages were effectively filtering agreement status.

People assumed that active agreements would all be outgoing funds because that’s all they can see in the current system. People also had different ideas of which agreements they were seeing in “active agreements;” some people accurately assumed that the agreements in their active agreement list were associated with their name, while others assumed they were active agreements for their region, forest, or unit.

Some of the statuses that folks rely on did not map to what we showed in either our mocks or the Django Admin app, such as:

• received the templates
• received the proposal from program
• accepted and approved the proposal
• agreement is completed ready to go for signature
• agreement linked to work plan
• pending execution
• out for signature
• Several people said that expired agreements that should be closed may slip through the cracks because until they are closed, they’d show up as “active.” One person suggested that showing items with a “pending closeout” status as a possible solution to this problem.

Recommendations

• Explore how to bucket status filters under high level categories: active, pending, expired, closed, draft
• Allow people to apply more than one filter at a time
• Refine microcopy and label filters in language users understand (rather than what’s in the database)

People need to see agreement number first, then cooperator name and project title.

People felt that the first few columns in the tables we prototyped are the most useful, but they have an easier time using them if they were in a different order. People want to see agreement number as the first column, since it’s the only true unique identifier. There is also a lot of information encoded into the agreement number that folks scan for, particularly to get a quick read on whether an agreement is within their region or forest.

When looking at the Django Admin app, we observed people squinting and leaning in close to look at the agreement number column.

Recommendations

• Increase the font size for items in the list view and explore hyphenating or otherwise breaking up agreement numbers to make them more scannable. Proposed order for the first 3 columns of a list of agreements: 1. Agreement number, 2. Cooperator name, 3. Agreement title

• Assess the technical feasibility and conduct a card sort or similar activity to arrange the additional columns people may want to see upon landing. Upon logging in, we heard that the users expect to see:

  • Status
  • Start date
  • Expiration date/end date
  • Agreement amount
  • Available funds
  • Program manager

  ...And additional columns they want to see if they scrolled right include:

  • If there’s a pending modification
  • Execution date
  • Program Manager
  • Whether an agreement is tied to a master agreement
  • Shorthand code
  • Whether an agreement has incoming funds / outgoing funds / or no fund
  • Funding status (from FMMI)
  • Date last performance report
  • Invoices pending
  • Date of last invoice

Search needs to be front and center

We tried two different approaches to search in the mocks and in the Django admin app. The mocks included a tab for “all agreements” that on hover, listed a few search options: search all agreements, and agreements by region. People expected to be able to search but did not always find the search links. More than one person asked “can I search?” Folks who found the search felt it would be helpful to have a quick path to search within their unit and forest, in addition to agreements by region and search all agreements. One program manager who talked about having lists of hundreds of agreements said that search is their primary means of finding what they need.

The Django admin app included a search bar at the top of the page that allowed folks to search by project title, description, cooperator name, agreement number or application ID, and people generally were drawn to it. We saw several examples of where the results returned were broader than people expected, since it wasn’t always clear how results were relevant to their search criteria, but we also saw folks using the filters to narrow down search results. At least one person tried including a % or “wildcard” in the search box because that is how the legacy system works.

Recommendations

• Prioritize search in the page hierarchy
• Consider grouping search and filter so that folks can better understand how search and filter interact to show them only what they want and need
• Explore how to indicate why results were returned / where search terms appear in search results

Appendix: Methods

Semi structured interviews with 5 participants: 2 program managers and 2 grants specialists and 1 budget approver. Participants were asked to think aloud while completing specific tasks using two different design artifacts: Mocks (tailored to the needs of program managers, grants specialists, budget approvers) and Django Admin interface (screenshots | live site).

Design hypotheses tested

H1: We believe that giving people individualized view into grants and agreements that are relevant to them will make it easier and faster for them to address issues that are waiting on their action.

H2: We believe that an overarching information architecture that organizes grants and agreements around the category status and then time/due date will help people find what they need to do more quickly.