Why product analytics and percentiles are a great match

Data analytics can be tricky, especially when the numbers are based on averages.

In reality you might find that what’s happening in your product isn’t always reflected by averages, and conclusions may not always hold up to scrutiny!

A music streaming service can be used here as an example of analytics metrics that give insight into how people interact with your product.

For instance, if in reality, almost all people play just a few songs (say, around 3), and our metric of average songs played hovers between 2 and 3, then that metric would be accurate enough.

But what if most people play no songs, and a handful of people play hundreds of songs? That would skew our average, and let us believe that, on average, most people play a decent amount of songs.

This is why Stormly has always provided more insightful aggregation metrics, such as the median and distributions. This shows a more realistic distribution of metrics. For example, our Product Feature Analysis insight shows not only the average times a user interacted with a feature, but also a full distribution of that usage for each feature, as shown below.

From the above distribution you can see that most people didn’t play a song at all. Among all people that did play at least once, most of them played up to 4 songs. Only a small minority played more than 37 songs. This gives a different view than just knowing that on average 3.8 songs are played per user.

Today Stormly introduces the Statistics and Distributions Insight, that allows you to get a more accurate understanding of any metric. You can use it to look deeper at funnel conversion times, session length, times an event was performed, properties of events, or anything else.

In this example we’ll be looking at the playtime of the play song event, per user.

One thing to decide on before running your Insight, is if you want to know the playtime per user, among all users. This also includes users that didn’t play a song at all and hence, have a playtime of zero.

Or, if you want to get the playtime per user, only for users that did play a song. The former will give lower results for all metrics, while the latter will have a higher average for example.

In this case, we’ll choose to only analyze users that did play a song. We do so by picking the “Play Song” event in the top bar user filter, and picking “Has this event”.

Now we’ll click “Add Item” from the Insight questions, and choose “Event Properties”. Then we pick the “Play Song” event and the “Playtime” attribute.

The default for any event attribute is to summarize them as total per user. But in this case, we want the average playtime per user to be our metric. So we click the gear icon, and choose the average option, as show below:

Our average playtime per user will be distributed into 6 separate buckets. But if you have specific ranges for each bucket that you want your distribution to fall into, you can specify them using the custom bucket ranges option.

Once our results are displayed, we can see at the top the most important statistics.

Most metrics should be familiar, such as the range of minimum and maximum, the average and median. But it also shows the percentiles such as the 95th percentile under the P95 metric.

The 95th percentile here means that 95% of all users played a song that was 370 or less on average in playtime. Or said otherwise, only 5% of users had an average playtime per song of more than 370 seconds.

Likewise, for the 90th percentile, 90% of users played a song of 316 seconds or less on average. The same for all other percentiles, such as the 75th, 50th (also known as median) and the 10th percentiles.

The percentiles chart shows much more percentiles in a single chart, as shown below.

Another useful chart to look at, is the distribution of values. It divides our average playtime per user into six separate buckets. For example, we can easily see that the largest chunk (around 42% of our users) has an average playtime per song of 161 till 197 seconds.

It also shows that a much smaller percentage of users listen to longer songs of 5 minutes and more.

Seeing a distribution of values allows you to get a much clearer picture than just using averages. Patterns such as power usage, where a relatively small group of users contribute to most of the usage or revenue, can be easily discovered this way.

All of the charts and metrics discussed above are summarized over the full date range. But at the bottom of the Insight, you’ll find the minimum, maximum, average and all percentiles visualized over time. We do this automatically for day, week and month granularity.

If you’re trying to change your 95th percentile user behavior by making some changes to your product, you can now easily see if that works or not.

By comparing the percentiles since the day you’ve released the product update, with the 95th percentiles before that, you’ve got a clear picture of what’s happening.

More complex metrics

So far we’ve looked at the average playtime of a song per user, a relatively simple metric.

What if we want to look at funnel conversion time? Or session time, based on a customized definition where a session should last at least 2 hours, and where at least a song should have been added to a playlist? Or maybe even define a custom SQL metric that we want to get a closer look at.

To look at funnel conversion time, we first need to define a custom funnel property. Click on the “Segments / User Properties” menu item and choose “User Properties”. Select the funnel property, and define a funnel – In this case installing the app, playing a song and finally paying for the premium plan.

Now open the Statistics and Distributions insights again, and choose from “Defined User Properties” the payment funnel we’ve just created. From the “Funnel Times” option, select “Play song to Payment Completed”.

By default all users that started the funnel are used to analyze funnel times. That means that for users that didn’t make a payment, their funnel time from play song to payment is counted as zero. This is fine if that’s the metric we’re after, but if we want to know the conversion time only for people that did the payment, our averages would be lower than expected.

To work only with converted users, we need to set our main user filter like that. From the main user filter, choose “Defined User Properties”, then the payment funnel we’ve created and select “Payment is completed”.

Once we run our Insight and the results are in, we can see that the minimum conversion time to payment is around 10 minutes, and the maximum 57 days. We can also see that 90% of our users upgrade to the premium plan in 27 days or less, while only 10% of users take less than an hour to convert to payment.

From our distribution histogram, we can see that the largest chunk of users (around 68%) take between 6 days and 27 days to upgrade to premium.

The last charts show how cohorts over time, by the week users were first using the app, differ in their conversion time.

But that’s not all! With Stormly, you can use percentiles in any custom user and event calculations. This means that any Insight that supports those, such as forecasting and anomaly detection, can use any percentile as the metric to work with and get forecasts and more from those.

At Stormly, we are always looking for ways to improve our product and stay ahead of the competition. That’s why we’re so excited about our new Product analytics metrics percentiles and distributions feature. With this innovation, Stormly is the most accurate and flexible analytics product on the market. If you’re serious about taking your analytics to the next level, sign up today and see for yourself how Stormly can help you reach your goals.