In the first version of our product, we built a suite of analytics that mirrored much of what we had known: conversation completion rates, time spent on the bot, etc. Conversion metrics are often valued on completion of an action, and “time spent” is how we value good content on a website. However, now having looked at hundreds of bots and their performance, we are re-thinking these metrics.

Bots are a new and emerging field. The downside of being first in this field is that we value bots in the lens of how we’ve measured tactics in the past. People who are using bots now, are sort of like the boyfriend/girlfriend who brings the baggage from their past relationship into their new one. And we are no exception. Too many years spent with Google Analytics have left us with metrics hangover, asking bots the same questions, like: “How long were you out?”, or “Why did you leave that text hanging?”

For bots meant to entertain and provide content, like the Maroon 5 Facebook Bot or Marvel’s Spiderman Bot, these metrics still stand. If your bot is a standalone entity, then greater time spent on your bot and a high completion rate are good things. However, if your bot is meant to integrate with other products and be a catalyst for an action outside the bot, then this all changes.

What we’re finding is that bots that are really performing best are those that are gaining and providing information quickly, and helping users commit conversions or other actions. For some bots, we would like the user to spend more time so we can gain more information, but for other bots that are meant to be functional (e.g. get people to make an appointment), less time is optimal. Moreover, because a bot is dynamic, sometimes you need certain users to spend more time on your bot to learn from it and others to be quicker, so “time spent per user segment” and “type of conversation” are important.

This creates a very gray area where there is no perfect high or low—no metric that can be created for a sector, and no specific goals for companies in this space to measure themselves by. Because bots are more nuanced than advertisements or websites, we have to start getting more sophisticated about how we both measure and compare them.

So what’s next? What’s the solution? We’ve started working on a better way to measure bots. We’ll be unveiling a new metric at the end of the year.

We always love good feedback on what we’re working on. If you want to test out our new analytics, e-mail us at [email protected] and let us know.