Written by: Cassie Zhou, Sam Baker, and Matt Grunwald

This post will discuss tactics and methodologies employed by our product management team to build Player Levels into our existing product. 

Every product feature at Skillz starts with identifying a core problem or opportunity at the heart of our players’ experience. One of our most recent opportunities had the product team at Skillz appealing to a type of gamer known as an “Achiever”. These are players that enjoy collecting achievements, sometimes regardless of rewards or benefits. When these types of players have something to chase, they are more likely to be retained and engaged with the system. In seeking to create a meaningful experience for all of our users in this context, we landed on a product feature idea: Player Levels. 

To develop features like Player Levels, we often exercise our left brain, using tactics like A/B testing, financial projections, engagement models, and user segmentation. In many instances though, we run into challenges that numbers alone can’t easily solve. What is the best visual layout? How can we use animations to create an engaging experience that sparks excitement? Delivering the right experience to the right user requires us to exercise both sides of our brain, thinking both analytically and creatively. 

This post will discuss tactics and methodologies employed by our product management team to build Player Levels into our existing product. 

Writing a Product Requirements Document

Once a user problem has been identified, the next step is to write a rock-solid Product Requirements Document (PRD) which outlines the problem statement, key product features, a vision of future versions, engineering cost estimates, and a list of impacted stakeholders. A well thought-out and easily digestible PRD is essential to ensure the business, marketing, and product teams at Skillz are aligned, since they are responsible for approving the vision of every new product feature.

For our Player Levels PRD, we addressed the following key questions:

1. What KPIs are we aiming to improve with this feature?
2. How can we create a level and progression curve that addresses those KPIs?
3. Should players be awarded for their previous play on our system, or should everyone start from a clean slate?

After discussing internally and researching other successful games, we decided that retention would be our primary goal for this feature. Clear and visible progress in a game can be a strong motivator for players to keep coming back. Player Levels would surface regular achievements in the form of in-game level ups and thereby emphasize the players’ sense of progression. Furthermore, we felt it was important to reward players who had been active on our system for many years. We decided early on to employ the concept of “grandfathering” and backfill the levels earned by players in their previous play.

Joy/User Experience (UX)

Once a PRD is finalized, reviewed, and approved by the Skillz business teams, our focus shifts to creating wireframes and thinking about the user flow. We make sure we engage our left and right brain to plan a feature.

Right Brain: One tactic our team uses is “dueling whiteboards,” where the team (including PMs, designers, and analysts) splits into random groups and brainstorms how to solve the problem at hand. Once all brainstorming is complete, each team presents their ideas to the others. This allows participants to work on their presentation and persuasion skills while simultaneously thinking critically about perspectives different from their own to uncover the best of the proposed solutions. 

Left Brain: Skillz often approaches the UX through asking “how many games into the experience” is this user? Are they on their 10th game or their 2,000th? To better understand how users at different stages might interact with our feature, we pull existing player behavior data from our database and build multiple player behavior models. We’re mainly focused on how behavior patterns differ or coincide depending on the number of games a user has previously played. 

With historical data in mind, we set out to evaluate our First Time User Experience (FTUE) for Player Levels. Concurrent messages would overwhelm a user, so we tuned the level curve to complement existing features and optimize FTUE. Therefore, our instinct was to introduce the very first level-up at the same time a new player segment received the existing “welcome to our rewards program” message. 

Getting things built

After finalizing product features and high fidelity designs, we document the content in a product specification deck (PSD) that includes our overarching goals, functionality, and design. We then work with the engineering team to iterate the specification based on technical limitations, feature changes, and UX improvements. 

As we built Player Levels, we realized we had underestimated the amount of work required to grandfather users. While grandfathering users would reward player loyalty, it wouldn’t improve retention if players were entering the system already quite close to the maximum achievable level and this would make the feature a lot less impactful. With this in mind, we decided to remove grandfathering from the feature.

Rollout

After a feature has been built, we typically roll it out in a few test games to evaluate its performance and see if anything needs to be tuned before it’s enabled for all the games on our system.

While multivariate tests for features is always our preference, when you have a social platform where players communicate with each other frequently, only giving a feature as exciting as Player Levels to half of the users becomes unappealing. As a result, we shipped this particular  feature without multivariate testing and instead evaluated on a pre and post-release basis of game performance. This rollout structure made the analytics a bit more difficult to interpret as we had to take into account live ops and other changes within the games, especially since the main metric we were trying to evaluate was retention. 

An important learning from our initial rollout of Player Levels was that users were reaching higher levels so quickly that it devalued the sense of achievement. In response, we again tuned our level curve with the goal of making it achievable, but more meaningful when users hit the max level. After implementing this optimization, we gradually rolled out the feature to all the games on our platform.  

Surprises and Success

One pleasant surprise from our rollout of Player Levels was an unexpectedly large increase in cross-promotion, which signifies a Skillz user installing another game from one of our 20,000+ developers. 

This win resulted from an element we included in Player Levels late in development. The “Add a New Game” component consists of a button and silhouette of a blank game icon meant to direct players to our website where they can download more Skillz games. We thought that this would be an non-intrusive way to let players know that there were other games on the Skillz platform. Through our analysis, we found that players who were engaged with multiple Skillz games tend to have much higher retention. Positioning this prompt within the Player Levels feature led to a massive increase in our website traffic and a double-digit increase in cross-promotion.

Wrapping up

Player Levels is one of our favorite features we’ve worked on, as it speaks directly to the Skillz mission of connecting players around the world in fair, fun and meaningful competition. It provides exciting experiences for seasoned players and helps our users find new games to play on our platform.

Some interesting and unexpected results:

1. The core Player Levels system has been very effective at encouraging player progression. We hit our primary goals as defined in the PRD.
2. Grandfathering was a concept we loved, but ultimately didn’t ship, and players still enjoyed the feature. Sometimes what you don’t build is most important. 
3. The “Add a Game” button was an idea our lead designer first suggested to help balance a visual, and as a team we found a way to make it truly additive to the overall impact of the feature.

As with any product feature, this one is still evolving. We are already considering a v2 of Player Levels with more ways to earn experience and rewards for exploring new games. This type of iteration and follow through is the heart of what we do at Skillz.