How we were building the wrong product for the last 10 months

It took us almost 10 months to put our product in the hands of real users, then we discovered that we were building the wrong product and decided to make a dramatic shift.

When the first stable version of the product was ready I invited 40 subscribers on our waiting list to try it out. The users I invited were highly engaged, I had email correspondence with every one of them, they were very interested in trying the product and the perfect examples of our target audience.

Although I handpicked the best candidates for beta testing, only three (yes 3!) users actually completed the integration process and started sending data to and using it on a daily basis, less than 10%.

That was a big warning sign, those three users who completed the integration were very active and used the product regularly, they clearly got a significant value out of it, but the real problem was that I didn’t know what happened to the other 37, and how to fix it.

I assumed there were two possible problems:

  1. It’s too hard to complete the integration. 

    In order to integrate a developer needs to embed our event code snippets.
    This often requires from our users to create a developmental task, prioritize it, implement it and test it. Which means lots of work in order to integrate a new tool that doesn’t yet have any reputation.

  2. The motivation to integrate isn’t high enough.

    Since our potential users never saw the tool, they had a hard time understanding the exact value the tool will provide, and to see the value from it lots of work was required.

Here is an typical reply I got when asking one of our subscribers why he didn’t complete the integration.


I think startups should make big dramatic decisions only if they have a visible impact, rather than doing small optimizations (but that’s probably for another post).

Following this principle, I decided that starting immediately no one could sign Up to unless they first had a product demo with me on Skype.

My goal was to show the value of the tool to potential users in order to increase their motivation to integrate the tool.

I’m still continuing with the product demos and am currently sending 100 emails to beta waiting list subscribers every day inviting them for a demo, I use for demo booking which is great for corrdinating between diffrent time zones.

demo invite

I had about five demos per day and after a week I already found out what was the problem.

Product demos are extremely helpful, you can learn X1000 more from a user you talk to compared to a user who just signed up without any direct communication. Starting with the demos was one of my best decisions I took in

During the demos I demonstrated the product and explained what it takes to integrate with it.

The ability to see face expressions, voice fluctuations and even breaths taken, is extremely valuable and really helped me come to a conclusion that our plug & play concept might be very difficult to achieve.

Our initial goal was to focus on SaaS companies, instruct them how to transmit 5 basic events through the API (website visit, user signup, app visit, user cancel and user billed) so that we can automatically calculate 25 charts and metrics we believe every SaaS company should have.

During the product demos I found out that different companies have different use cases. Some have a cancellation option, while others don’t. Some have only a trial and paid subscription plans, others have a freemium model and so on.

What we were trying to do is fit our users into some kind of template of metrics and events, and fitting them into this template required plenty of work including embedding 5 events, importing historic data, adjustments, testing etc.

It just takes too much time until users start getting value from our product.

Also, I found out that the most companies don’t even want our template, they want to visualize data and events they already have and most of the time they already have all their events set up via Segment is a data hub that allows with a single integration to send them data, from there on you can tell Segment to which tools the data should be sent on fourth.

This means that companies want to create their own dashboards, visualize data that they’re already sending elsewhere to different tools. The problem is that they just don’t have an easy way to create “big-picture” dashboards based on this data.

I decided that we will give our users what they want, instead of forcing them to use something that we think they want. We are moving away from the plug & play concept into the do it yourself concept.

This will both simplify our integration and dramatically increase our potential market size, even coffee machines can now send events to and create their own dashboards!

We actually just started sending events every time we drink coffee or tea and created a simple dashboard for tracking our team’s drinking habits. A post about the fun process of setting this up is yet to come:


Fortunately, at the time I made the decision to focus our product on event based dashboards, our dev team just finished working on a new feature called custom widgets.

This feature originally was about letting our users add their own metrics on top of the 25 predefined metrics we already provide. But following our new concept we are no longer providing any pre-defined widgets and the custom widgets function will be used to fill the dashboards with any chart or metric a user decides on.


Moving Fast!

During the last weekend I thought about changing the focus of the product.

On Sunday we discussed with the development team the changes we needed to make to make this happen. During the week I had several product demos introducing the new concept, and on Thursday before the week was over I already saw our users working with custom widgets, some of them spending hours analysing their business with (!!).

Here is a little sneak peak of my weekly recap of these changes, part of a series of videos I’ve been doing to keep our company’s journey documented:

“Moving fast and breaking things” is exactly the culture I want to embrace in and optimise for.

The biggest advantage of the event-based dashboards approach is that whoever is already using (which is thousands of companies) can just plug into and start building their dashboards in seconds (Yes, that’s correct!).

What we could have done better?

As I mentioned at the beginning of the post, it’s been almost 10 months from when we started working on the product, until we gave it to someone to use. 10 months is a really long time in startup terms, and I am constantly thinking how we could have done this quicker.

Could we have found out that the plug & play concept is not going to work, before we developed 25 pre-defined charts and metrics?

Could we have gotten to the same conclusion just with two dashboards instead of three?

Could we have reached the same conclusion by spending less time on the design?

Sadly the answer to all of the above questions is yes.

We could have reached this turning point earlier.

Fortunately, the changes we decided on don’t require a complete re-design of our product, just some adjustments. Still I estimate that we did lose about 2 months.

I’m sharing this article in a hopes that it will inspire other entrepreneurs to think about the product they’re building and how they can put it in the hands of real users earlier.

That was our story of how we built the wrong product. If you have your own it’ll be great if you could share it here so we can all learn from each other.

  • Robert Hopman

    I’m doing research on best onboarding practices. A quick delivery of a WOW or Oh f*ck this is great, is one of the key elements of a good startup. Or even a good product. Because most devs are really in love with their code. The experience should be within 15 – 30 seconds of signing up, otherwise its too complicated and doesn’t motivate the user to keep using it. Best thing to do, is to interview 5 users every day what they are doing with your software. I’ll repeat that. Every day. At minimum. I’ve learned this after helping a lot of startups. The ones that succeed listen a lot to their customers. Looking forward to the next blog post!

    • Alex Flom

      So far my record was 7 per day, but like you said I am aiming for 3-5 per day. So far I think it is a great investment of my time, specially in this early stage.

  • Evgeny Zislis

    When talking to customers, its also important to remember that they don’t always tell you what you actually need to hear. I have recently read this little book called The Mom Test, it gives some great practical advice on how to improve your interviewing technique. Kudos for the really great blog post!

    • Alex Flom

      Thanks mate,
      I strongly agree with the saying “Entrepreneurs Innovate and Customers Validate”.
      Will check out the book :-)

  • Andy Cars

    Thank you Alex for sharing your story. If it’s ok with you I might refer to it in one of my workshops that I run on lean startup and business modeling. I think it holds some very valuable lessons. And as Evgeny mentioned, The Mom Test is a must read for anyone doing customer interviews / development.

    • Alex Flom

      Thanks Andy,
      Sure, it will be great if other people will learn from our experience.

      • Andy Cars

        Thanks Alex. I wish you all the best with future experiments and iterations.

  • Evgeni Stavinov

    The problem with giving users what they want, and “do it yourself” concept is that customer acquisition cost skyrockets. It’s a fundamental shift of the business model.

  • Predrag Radic

    Thank you Alex for this story, it’s useful and inspiring as all the previous :) Regarding “plug’n’play” VS “do it yourdelf”, it should not be such a tradeoff, and fundamental decision. I belive that custom modeling engine is essential feature, but predefined templates built on top of this engine are also. When testing some new software I always look for templates or pre-built setup to easy see fundamental features and experience what software do. If you have them, it should be big plus and increase acquisition rate. Also later, in real use, existence of (customizable!) templates for most used widgets should provide much better user experience. At last, what do you think about initial integration service provided by your side, which can follow up Skype conversation with your customer? It would increase your costs, but I suppose could bring customer service to next level, and provide you even much more insight into customer real use cases. Wish you all the luck and wisdom…

    • Alex Flom

      Thanks Predrag,

      I agree that we need to find the balance between providing some kind of template and at the same time not limiting the freedom of our users. :-)