Colin Brooks ↰

Senior Developer @ Whitney Museum

Answering the question “what’s on today?”

Screenshot of a section of a webpage, with 6 exhibitions with images on the left, and a vertical list of events for the day in chronological order on the right.
Today at the Whitney on 12/3/2017.

A few months ago we launched a new visit planning feature on whitney.org. Varyingly referred to internally as “Today at the Whitney” or “What’s on Today”, this feature came about after a conversation with a number of our colleagues in Visitor Services around our Plan Your Visit and Events calendar pages. They brought up how difficult it can be for visitors to get a good view “at a glance” of what all is happening at the Whitney on a given day, and wondered if there was a way we could better serve this need on whitney.org. To anyone who’s visited museums and their websites, this is likely a familiar problem.

Ultimately the solution we came up with did three important things:

  1. Drove more people to our event pages
  2. Increased traffic to our ticketing platform
  3. Noticeably boosted our overall ecommerce conversion rate, despite including zero ecommerce focused features or calls to action.

At the same time we also discovered a new and interesting method for messaging upcoming events, and revamped a portion of our public WiFi login process. We set out to improve the visit planning experience on single page of whitney.org, and ended up supporting the broader goals of getting more visitors committed to visiting the museum, and orienting them once they’re here.

Considering the problem

The core challenge for us was, how do you show both numerous long running exhibitions AND brief special events that only happen at certain times on certain days, without overwhelming the viewer with information? A traditional calendar might work well for daily events, but with exhibitions that run for months, how do you best mix the two and still have it make sense “at a glance”? Especially when the distinction between a special exhibition and an event, meaningful in terms of museum programming, might often be unimportant to the average visitor who is simply interested in anything that’s available for them to do.

There are a number of examples of other museums that have attempted to solve this very real problem, but given our own particular blend of programming and distinctive design style, we thought we’d need to break a bit from our peers and generate our own solution.

The questions we really needed to answer were 1) what’s the minimum amount of information that a future visitor needs while still providing context, and 2) what sort of layout and functionality supports a simple path to discovery? Keeping the amount of content to a minimum was incredibly important, as the whole concept would still live in parallel with the existing fleshed-out exhibitions and events sections of whitney.org. And ultimately, those sections would likely better serve someone with a more specific idea of what they were looking for. What we were after was a starting point for someone to think about what they would do during their visit.

With that in mind, what we kept coming back to was: what would someone physically hand you if we could design and produce a museum guide for the exact day you were planning to visit, or even while at the museum? The paper guide experience is an extremely familiar one, and seemed to offer the closest equivalent to the experience we were aiming for: not a deep dive, but something to guide you as you moved (or planned to move) through the museum.

Using paper as inspiration

Right around when we started this project, I happened to visit the Philadelphia Museum of Art, where I ran across this handout:

Diptych of the front and back of a paper pamphlet listing all the tours and programs at the Philadelphia Museum of Art today.
“Today” at the Philadelphia Museum of Art handout.

While it only covered the day’s events and a small selection of the exhibitions on view, this was by far the most helpful piece of messaging around events I can recall encountering at a museum. It’s incredibly straightforward, clearly organized chronologically, and gives someone (like me) who does not tend to seek out information about events a solid impression of options I wouldn’t otherwise have considered. Printing physical paper drags along all kinds of complications, but the idea, translated to digital, felt very close to something that could work for the Whitney.

Putting it together with exhibitions

Armed with the understanding that a chronological list (not surprisingly) is very helpful for parsing events at a glance, the remaining challenge was how to mix it with a presentation of our current exhibitions and any other potential content. Focusing on this feature as a visit planning tool, we decided to limit our display of exhibitions to those that had a physical presence at the museum. This meant stripping out our online-only content, our then-current billboard, and any other exhibitions that did not strictly have a presence on a floor within the building that a visitor could expect to see. This limited the content we would need to showcase, lessening the risk of the feature becoming overwhelming.

At the same time, through rough experimentations to more polished explorations with our colleagues in Graphic Design, we jettisoned earlier thoughts of also including hours and dining information. Exhibitions and events offered enough of an organizational challenge, and adding anything else pushed us into territory where it seemed like we would have to hide some amount of the content behind user interactions…defeating the original goal of having everything “at a glance.”

With our content scope settled, we determined how much information to include about both kinds of programming, and put it all together. What we ended up with was a sort of side-by-side presentation of exhibitions and events, with the former arranged in an image-centric masonry grid, and the latter in a vertical list that allowed for the clear display of times. This solution felt appropriate because it addressed the critical pieces of what each content type needed to get across: events happen a specific time, and exhibition images are helpful (even if not perfectly representative) in terms of grabbing someone’s attention, and giving them at least a slight sense of what a show might entail.

Benchmarking right from launch

In a first for whitney.org and our team, we did not launch a major feature for 100% of our audience: “Today at the Whitney” went out with a hard soft-launch for 90% of our web visitors, with 10% receiving the previous version of the Visit page, sans new planning tool. We did this purely so we would have a benchmark to compare against, and see what effects the new feature might have.

While this might sound reminiscent of our approach to A/B testing, it’s different in the critical fact that “Today at the Whitney” was always going to ship. This was not a comparison between two competing options: the feature was going out, and we would iterate it on it going forward. But by releasing it to just the majority of our audience rather than all of it (via Google Optimize), we allowed ourselves the chance to better understand the product we had built.

Measuring success

With “Today at the Whitney” released through Google Optimize, we had 3 metrics chosen to observe:

  1. Traffic to exhibition pages
  2. Traffic to event pages
  3. Traffic to our ticketing site

One of the more interesting questions raised by a feature like this, is whether success should be reflected in an increase or a decrease to the expanded views of the represented content. Is “Today at the Whitney” more successful if it sends fewer people to our exhibition pages, because they no longer need to navigate to the richer views to get a sense of what’s on, or is it less successful because they neglected to take a deeper dive? I think you can make an argument for both options, which is a huge part of why it felt so important to measure any changes to these metrics post-launch. Both reflected strategies might have their positives and negatives, but we ought to know which one we’re shifting towards.

Changing behavior through supportive content

We ran our Google Optimize experiment for 3 weeks after “Today at the Whitney” was launched for 90% of our audience, and covered approximately 25,000 user visits to our Visit page. In terms of user traffic to exhibition and event pages, “Today at the Whitney” decreased the conversion rate by ~3.5% to exhibitions, and increased it by ~17% to events. It also increased the conversion rate to our ticketing site by ~8.3%, and our overall ecommerce ($$$) conversion rate by ~3.5% over the period.

The slight drop in traffic to our exhibition pages seemed reasonable in light of our speculation about certain users only wanting a “glance” of what was on instead of a deeper dive. The huge increase in the traffic to our event pages on the other hand was a more delightful surprise, and suggests that this feature is serving a need that wasn’t previously being met elsewhere in the user journey. Perhaps most intriguing however was the bump in traffic to our ticketing site, and the subsequent purchase rate. “Today at the Whitney” has no callout to ticketing, so any change in these numbers isn’t due to some new button or link: it’s due to users simply being more interested in purchasing tickets.

Moving the needle on ecommerce while trying to provide a better visit planning experience is an ideal outcome for a feature like this. Through building a display of content to support the user experience on our site, we ultimately impacted our broader goal of trying to get more visitors into the museum.

A surprise second product and what’s next

At the same time we were working on “Today at the Whitney”, we were (at first) unrelatedly updating our public WiFi login flow. After enough time staring at both projects, and given some particularly tight technical constraints around WiFi login, there was a bit of a light bulb moment where we decided to try incorporating a simple listing of the days upcoming events into the screen a user would see immediately after connecting. And as it turned out, this felt quite natural, and in many ways turned into the sleeper success of this project.

Additionally, since the initial launch we’ve made a number of improvements to “Today at the Whitney”, including turning it into a “Tomorrow at the Whitney” feature once the museum has closed for the day. Going forward we may want to expand it again to include some concept of weekend planning, or draw on stronger links to our general events calendar. The challenge will continue to be how much can we include, while keeping the “at a glance” view understandable and readable for the average user.

The strategy of launching a major feature, even one limited to a certain section of a single page, to a portion of our audience and benchmarking the remainder, was vital to understanding “Today at the Whitney.” That strategy will not always be appropriate or viable for all of our releases, but when possible, and in conjunction with user testing, A/B testing, and Google Analytics, it’s something we’ll continue to do in the future.

[Read this on Medium]

Experimenting with what works

Drawing of anthropomorphic A and B holding hands while a W watches on at a distance.

A lot has changed about whitney.org over the last year. This includes the entire platform underpinning the site, and a number of major usability and user interface improvements, from reworked navigation, to new mobile experiences for audio and video, to the incorporation of outside voices in our exhibition content. And with growing distance from the complexities of launching a new website, our data-related work has been picking up steam as we’ve been able to devote more time and mindshare to it, which in turn has begun to more deeply impact our design thinking and decision making processes. A major aspect of that impact is in an increasing ability to reevaluate our assumptions, and to better understand how visitors are actually interacting with us…rather than just how we might think they are.

In the beginning…

With whitney.org being rebuilt from the ground up, there was the opportunity (and requirement) to reconsider what data we ought to be collecting and analyzing through Google Analytics (GA) and similar tools. I joined the Whitney Digital Media team part way through this project, and so initially my primary goal was simply to get a handle on what was being tracked, and where we had major gaps.

Previously I had interned with the Museum of Modern Art on a data analytics and ingestion project and I hoped to build on that effort at the Whitney. Google Analytics can give you a lot just from incorporating the tag on your website, but to really understand our audience we needed to make sure we were also tracking important interactions, like use of our navigation, audio and video plays, and various call-to-action (CTA) clicks that wouldn’t necessarily be visible in the default pool of data. Without that tracking, we wouldn’t have a baseline to compare to when we were ready to start considering more major design and structural changes. Looking forward meant getting our house in order, today.

In addition to putting together the pieces for an interaction-baseline in Google Analytics, I was also interested in reviewing how accurate our GA-reported ecommerce numbers were compared to the organization’s internal accounting. This necessitated reaching out to colleagues in other departments. And while those conversations were brief, they have been incredibly important in framing our work. Seeing how our numbers compare to their numbers gives context to the fuzzy reporting of Google Analytics, and helps us to know what of level of confidence to ascribe to our figures…which is vital to separating the signal from the noise.

From passive data to A/B testing

The next stage in the Whitney’s digital data lifecycle has been to start investigating why our numbers are what they are. It’s one thing to know that we got 100 clicks on a button last week, it’s another to tease apart why it was 100 and not 50, or 200, or 10,000. To that end, we’ve begun to evaluate more and more of our core digital content through A/B testing. Altering certain elements of pages and considering those changes in relation to basic metrics like session duration, pageviews, and ecommerce transactions has been hugely helpful in terms of figuring out what aspects of our design and layout are effective at achieving their intended purpose. It has also been a gradual process, characterized by a growing willingness to experiment.

We began by making small changes to individual elements using Google Optimize: experimenting with the color of a few CTA elements, shifting the placement of a few links (and adding others), all in the hope of driving more conversions. As we became more familiar with the process of building and running tests, we started experimenting with larger aspects of the site, including a wholesale replacement of the footer, and a number of tests designed to start teasing out the effects of utilizing different kinds of content to promote exhibitions (or broadly, the institution) on our homepage. In the case of the latter, we’ve varyingly given users videos of artworks, images of artworks, experiential videos of exhibition installations, and images of people in those exhibitions.

The question of what content works is incredibly important to us as the museum’s Digital Media department. It’s our job to determine how to best reach new and returning audiences, and a major driver of our ability to do that is the media we choose to forefront. A/B testing is one more tool we can use to help make those decisions.

The conventional web wisdom tends to be that video will result in higher user engagement than still images. Video is also inherently more expensive than images, so the choice to go with one over the other doesn’t come without a few tradeoffs. In the effort to set a baseline for our both our data and any data-driven strategy, we wanted to test the conventional wisdom against our specific circumstances as a mission-driven arts institution. Given the added cost, we wanted to be sure that we understood the implications of each approach, and the metrics they affect.

And while it’s still too early to say what the definitive value difference may or may not be for us, the initial results have been interesting, and continue to warrant further investigation, and suggest new avenues to explore for homepage content.

Incorporating user feedback

Closely connected with our push into A/B testing, we’ve been working to get more qualitative feedback on various aspects of our platforms. This has included both quick in-person terminology surveys, and an incredibly rigorous two-part in-depth usability study of whitney.org run by an outside consulting firm. Both kinds of testing have been extremely useful, while also demonstrating why it can be worth it ($$$) to work with experts in the field. In-house user testing has made a lot of sense for us as small and focused endeavors, while the external usability study brought a level of expertise and specialized tooling we would never otherwise have had access to. There’s much more to say on both, but that deserves its own post.

It’s worth mentioning that all of our user testing efforts at some level remind us of what we aren’t doing as well as we’d like. And while it would be easy to discouraged by that, I think it’s important not to be. There is no end state in usability work. There is no point at which everyone will convert, everyone will understand our UI, and everyone will have a positive experience the first time, and every time after. Rather our goal is the constant evolution of our products, reflecting the changing needs and expectations of our audience.

T H E F U T U R E, and moving to product development in tandem with evaluation

So what comes next for our data efforts? Likely, it’s wrapping together all the helpful experiences we’ve had over the last year into a more consistent, repeatable process. In effect, moving to something akin to usertest driven development, where new features are planned alongside user-focused evaluatory measures. Whether that means A/B testing aspects of design or functionality, or small scale in-house user surveys, or even larger external evaluation, our actions should depend on the nature of the feature and be determined in advance, rather than only as a post-launch reactionary measure.

TL;DR

  1. Just because we measured something doesn’t mean we measured what we thought we did.
  2. User testing is useful in all kinds of forms.
  3. User-test driven development is good development.

[Read this on Medium]

How Many Hoppers?

An up to date count of the number of works by Edward Hopper currently on display at the Whitney Museum of American Art.

“Hey could you give me the numbers on that again?”

Interning with MoMA’s Digital Media department is great. Building a dashboard to track metrics across the museum is really complicated. Both of these statements are true, which is the kind of authoritative certainty I strive for in data analytics. This summer my job has been to create a dynamic dashboard that pulls in data from sources all over MoMA, that ideally updates automatically.

On its face, building a dashboard might sound like a straightforward task, composed of a) collecting the data, and b) visualizing it. But in practice it’s been much more complicated, requiring a significant amount of both human and technical collaboration. Figuring out where various metrics are coming from, if they are what they should be, and how to wrangle them into a dashboard requires answering a lot of questions with often imperfect solutions.

Part 1: Data audit — “What’s out there?”

Being new and legitimately having no idea how MoMA kept track of anything, there was no reasonable way to start this project without first talking to the people who knew their data best. MoMA had already identified the types of metrics it wanted to track, but the nitty gritty of how to generate them accurately and efficiently was something I needed to sort out. I began by sitting down with individuals from different departments and asking about what metrics they were interested in, what services they used, what their workflow looked like, what things worked well, and what things they’d like to do differently.

Having these conversations with people from retail, social media, email marketing, membership, management information, IT, collections technology, and others, was vital to the project for two primary reasons:

  1. There would be no technical way to create the metrics I needed without knowing where they had to come from.
  2. Without these conversations I wouldn’t be able to even start to answer the sorts of questions that have come up constantly throughout this project, and I’d be sending a dozen emails a day asking how numbers were calculated.

What those early conversations allowed me to do was be more efficient when I did have to bother people. They also meant that I had a better idea of what was possible to track, and critically at what point in the data-flow I should try to track from.

By the way, when you think you’re done you’re with this you’re not: something else will always come up.

Phase 2: Get the data — or SaaS (Spreadsheets as a Solution)

Of course the obvious question for all of this is how do you get data into the dashboard in the first place? How do you marshal it all together in a consistent and interoperable enough form where that’s possible? It’s a lot easier said than done when you start reaching dozens of separate services, and feel your brain start to melt thinking about how to combine your InstagramTumblrTwitterFacebookYouTubePeriscope, and email subscriber engagements segmented by month, post, impression, or number of followers.

“Data” is great, but the practice of getting it cleanly, consistently, and quickly for multiple people can be a real headache when it’s spread so widely. This is part of the impetus for MoMA to build a unified data-dashboard. To create a single place where KPIs and other important metrics can be observed by multiple people, over time. Where there is enough information to be useful at a high level even if it isn’t necessarily the absolute truth, and the reasons for any inaccuracies are clearly documented.

It’s important to point out that this has been by far the longest step in the process. The state of data on the web in 2016 is one of APIs and a gauntlet of new vendor-specific dashboards you need to login to once a month, remind yourself where they hid the export button and date selection, and dump the data to a spreadsheet. For a place like MoMA this presents a kind of sweet spot of organizational difficulty: enough data and the savvy to utilize it within its channel, but enough distinct data sources that it becomes a monumental task to assemble everything into a bigger aggregated picture of what’s going on.

Grid of 12 screenshots of various dashboards and data sources.
A small smattering of MoMA data sources. There are dozens more.

It’s great that (in a vacuum) so many of the museum’s data sources provide access to their content or analytics online. But as the number of services MoMA and other similar institutions use continues to creep upwards, these sorts of silo’ed check-each-service solutions start becoming really unwieldy.

I think it can be tempting, having reached this point, to do one of the following:

  1. Buy a service that sells itself on integrating all your other services. This will inevitably pose its own complications, and will require yet another service in the future to better integrate the integrating services.
  2. Develop something in-house that will require highly technical (and expensive) maintenance for the duration of its existence.
  3. Say screw it to the whole thing and stick with your current inefficient workflow.

What I’ve tried to do with MoMA is split the difference between those choices, hopefully taking some of the benefits of each.

In my mind the core problem in sharing data across any like-minded medium-sized institution is making the data available at a level that is accessible from a range of technical perspectives. And I mean that both with respect to the technologies themselves and the expertise of the people utilizing them. It might be great to break your social media data out in a SQL database, or to say hey, here’s the API keys, have at it. But that doesn’t work for everyone’s level of technical know-how, even if it might work with your Tableau or Qlik integration. Building something that will last requires building something that can evolve alongside needs and expectations. So rather than focus on creating the perfect dashboard, I’ve worked to get MoMA’s data feeding into spreadsheets.

That may sound a little antiquated, but the reality is that spreadsheets are already in just about everyone’s workflow who deals with data. Getting data into Google Sheets means that nearly anyone can do something with it, from charting within the sheet, to pivot tabling it somewhere else, to importing it into just about any other data-dealing software imaginable (including in MoMA’s case, Tableau). However, what makes Google Sheets in particular an appropriate tool for building consistency across sources, is Google Apps Script.

Google Apps Script allows you with minimal fuss to write JavaScript that interacts with a corresponding Google Sheet. In turn that allows for the possibility of connecting to the APIs of the various data sources you’re trying to do something with. For MoMA that means things like our ticketing system, email service, social media channels, and our app downloads tracking service among others can all be setup such that each night a Google Apps Script goes out to each API, requests data from the previous day, and then inserts that back into a Google Sheet. And with Google Apps Script triggers, that happens entirely automatically with no human input. From there it can be brought in to visualization software like Tableau or Google Data Studio, or simply shared to someone else and they can do whatever they in turn want to do with it.

Admittedly, “here’s the solution, just write some JavaScript and couple it all to a dashboard in Tableau” is itself fairly technical, but on the scale of what it could be I think this is reasonable. Google provides sample code for a variety of services that without too much trouble that can be massaged into connections to others. And on a personal level, I think the reality is that as more of the systems we interact with shift to RESTful APIs this kind of institutional knowledge will have to grow.

What this sort of middle ground of technical automation and integration allows for is an easy source of consistent data. With proper documentation, this opens up metrics to more people without having to deal with access to the specific services themselves, and allows for them to answer more of their own data questions. The technical cost is in setting up the API connections in Google Apps Script, but once that’s done there should be much less upkeep necessary than for a more end-to-end from-source-to-viz solution.

TL;DR

The vast majority of the work in building a dashboard for MoMA’s digital properties has been in figuring out why the numbers are what they are, and how to get them into a consistent and useable format. Updating them automatically on a schedule has been icing on the cake. This is a project that has taken me across the museum and a good chunk of the internet. And it’s become abundantly clear that there is no substitute for personal conversations.

I’ll end with a few top tips for anyone undertaking a similar journey through data:

  1. Talk to people.
  2. Don’t assume you’ll remember what someone patiently explained to you; please write it down.
  3. Accept imperfection. Lots of it.

And if I could give those tips a second time, I would do that too.

[Read this on Medium]

Neuron

Mixed media enlargement of an imagined neuron, incorporating generative audio and video, projection mapping with Kinect and OpenCV, chicken wire, stainless steel, and acrylic.

Recursion lesson & survey

An experiment in teaching about the concept of recursion to new audiences using an animated Sierpinski triangle. Learners read a brief sequence of paragraphs describing recursion while the animation played alongside, then answered a brief survey on the topic.

Spheres II

Continued exploration of deconstructing spheres, limited color, galaxy-inspired movements and shapes, and generative growth.

Whitney Teens Open Studio sign in

A smarter sign in form for Whitney Museum teen events. Parses acronyms and commonly mis-entered information, aggregates data, and pushes it to a central Google Sheet.

Deconstructing Shells II

Early exploration of deconstructing spheres, galaxy-inspired movements and shapes, and generative growth.