Product Metrics Spikes

A friend pointed me to How to crack product metric questions in PM interviews. In essence, it discusses how does a service provider respond to a change in a metric or KPI (Key Performance Indicator).

A definition from the article:

Metric interview questions test if candidates can perform data analysis and select key metrics that matter most to the success of a product. Employers like Facebook and Google use these questions to evaluate critical thinking and communication skills.

There are two types of metric questions - one builds on the other

Metric definition questions	Metric change questions
What are the metrics that provide clarity on the health of a product or feature? What metrics matter??	Do you know what to do when a key product metric (e.g. traffic, revenue, engagement, etc.) is going up or down for no apparent reason.

Solving the Metric definition question

The article recommends the use the GAME method

Goals	Make sure you understand how the functionality helps the commercial goals of the company. 80/20 rule - make sure you're measuring the 80% - the stuff that matters!
Actions	What are key activities that indicate the functionality is achieving those goals? This requires good understanding of the product functionality. Make sure that DevOps doesn't define and implement these unsupervised!!
Metrics	Where in the value chain is a good place to position a measurement? Yes or No evaluation is a starting point, but is there another metric that is a precursor to the switch toggling from one condition to another? In web performance management ie website availability and uptime, then "What goes slow, goes down." (see the Webmetrics case study)
Evaluation	What is normal, good or bad about a metric reading? And are the trade-offs and limitations in having this figure? For example, seeing a drop in shopping basket conversions over a 24 hour period might be useless without understanding that there was 60 minute outage from your payment provider.

Solving the Metric Change question

The article provides its own methodology.

Define the metric change	Make sure you fully understand the metric that is being presented to you: What is it and what is it not? You may have to get into the weeds about how the metric is measured. See Actions above For example is 'Page Views' the total number of views or page views by unique users? What happens if the same user logs in and views the page on two devices or two different sessions, how is that logged?
Explore possible root-causes of the change	Use the MECE framework Mutually Exclusive and Collectively Exhaustive = Identify cause(s) that is independent of all others OR understand what the cause is definitely NOT (*). For a metric incident, the root-cause of the problem has to be internal OR external, as there is no other option. If you think there is an internal AND an external cause, then you need to do more research. Did one external condition cascade to an internal condition? Unfortunately, these can be super hard to run down!
Conclude	Make sure there aren't any horrible assumptions that you haven't validated / eliminated.

   
* For info on MECE:

Escalation and Communication

The article hardly touches on the critical, critical point of issue escalation and communication.

After you have done your initial assessment, you need to answer the question: Do I ring the alarm bell and to whom?

Judgement call: Let's assume that (a) the metric isn't a false positive and you've validated to some significant extent (b) the metric matters. So, how high do you escalate the issue?

Factors to consider:

• How many other business functions are impacted by the incident?
• Is that metric important to them?
• Can they do something about it, either to resolve it (eg some root-cause analysis) or to reconfigure their business function to accommodate the impact of the incident?

I recommend using this yard stick:

Level of concern	Action
Not a concern or quickly solved and implemented	If you are sure that impact is minimal and contained, then I recommend concentrating on solving the issue. You can report that the problem was resolved with minimal impact. Warning: if you get stuck in the resolution, then you quickly have to over-escalate the issue eg to 'Very concerning'
Somewhat concerning, but impact is contained	Keep researching until either you hit a brick wall or an unreasonable amount of time has elapsed
Very concerning	More business functions are required to investigate and you need to escalate it to a suitably high level to get resources devoted to it. When communicating, be clear to separate known symptoms vs known conclusions to date vs guestimates as to possible causes, articulate mitigating factors and (partial) resolutions that have been applied to date.
Ridiculously high	This is a hard one to call. You most probably know that something basic has gone wrong and that others have spotted it and are working on a solution (and may have even resolved it). However, that is a dangerous assumption. It's best to escalate it - even if it is likely to be a false positive.

The importance of communicating - and re-communicating

As soon as you communicate outside of your team, make sure that someone is knowingly and consistently appointed as Incident Manager (ie it could be you!). Note that if the problem rattles on, then you may need to appoint a Communications Manager in front of the Incident Manager to protect the Incident Manager from simply responding to communication enquiries rather than actually working on the resolution.

It's best also to alert other business functions how you will update them on progress and frequency of updates (even if there is no progress).

Use of incident support tools

There is a great danger in over-engineering the use of incident support tools. Over-engineering = it is too clumsy or laborious at the point of the crisis. If others aren't familiar with the tools, then they will still ask for email / telephone / instant messenger updates anyway!

Next step: Deep Dive into Root-Cause Analysis

Once you have done your initial assessment and then communicated your initial findings to relevant business function, it's time to dive deep to separate symptoms from root-cause. You may need access to multiple test and near-production systems(*) to validate theories.So do make sure these are functional before the crisis hits.

Do ask around your team (DevOps, Release Management, Testing, Engineering) for their input into symptoms and route cause. In many circumstances, something similar has been seen before.

Cycle back with colleagues and partners based on status and resolution - instant messenger is fab here!

(*) Near-production = an environment that mostly closely replicates your live or production environment

Multiple resolutions

There may be several resolutions available: there's a matrix of time horizon vs cost-benefit.

Multiple resolutions might be appropriate - and that's absolutely appropriate to pursue several simultaneously. eg roll-back recent enhancement; test proposed partial fix; kick back to engineering for a rework with new testing scenarios etc.

Product Managers: Now, Next, Future

Steve Johnson, one of the respected voices in Product Management, cleans up some of the confusion about product roles and titles in Product Management. He admits that he (like me) is fan of  the “product management triad” - a product or portfolio team responsible for strategic planning, release planning, and launch planning. Here's the link to the particular post and full description of these roles. 

Title	Scope	Comment
Product Strategy Manager	Future	Evaluates and prioritizes future products and the portfolio
Product Owner1	Next	Focuses on what’s next in terms of features and product options
Product Marketing Manager2	Now	Focuses on products that are currently available

¹ Steve prefers Product Planning Manager - yuk!

² Steve prefers Product Growth Manager - yuk!

I do like the time box approach to the focus of each of these roles. Of course, this is simplified. It's not as if the Product Strategy Manager slurps espressos all day and pays no attention to the last quarter's sales figures or the Product Marketing Manager isn't constantly assessing the competition's capabilities and figuring out how to 'out-product' their offerings.

I would also recommend studying this Product Taxonomy from Product Focus (providers of product management training and resources - and one of the sponsors of the Cambridge Product Management Network).

Here's my take on 'Who owns what' which dovetails with Product Strategy vs Product Owner vs Product Marketing

Portfolio - owned by Product Strategy Manager

Responsible for managing the portfolio. He / She isn't the only decision maker. This is best managed by Product Investment Committee - this is a beast in its own right - but this has representation from Sales, Marketing, Product, Engineering, Research plus some financial modelling expertise.

The Strategy Manager often takes on the long-term management of the core platform, as moving the platform forward requires care and evaluation, because key decisions may negatively impact capability of the products that are built on top of the core platform. ie mods to the core platform are expensive and time-consuming to get right.

Propositions - owned by Product Marketing

This is the realm of product marketing. Product Strategy may be involved if there is cross-subsidisation of prices between products (or product lines) and platforms.

This area gets super complicated when propositions vary between countries / region of the world. Different countries may provide functionality for free (or charge for free functionality!) for a whole variety of valid reasons (local commercial agreements, tax or legal reasons, history & politics, how core product development costs are charged to territories). This is matter is worthy of another blog post or even a book!

Products - owned by Product Owners

When considered in a portfolio, this is the domain of product owners or, if the product is significant enough, a manager in their own right.

How everyone else views the first-in-post product manager

CEO mutters that maybe it is time to hire a product manager.

Everyone else wonders, "Err, and how will they help?" And they get that uneasy feeling that the company is going to be less fun and more restrictive and more "NO!" Perhaps the first publication of the Employee Handbook had the same reaction - mildly menacing, but necessary for some (other) slightly dysfunctional part of the business, but not for me.

Below I augment the great story that Product Focus (a well-regarded provider of product management training) tell in their Product Leadership Workshops and, to some extent, is outlined in their publication, "What is the point of Product Management?"

Here's the history of a B2B software company and the welcome emergence of product management.

The small company was founded on technical innovation and good customer service to its early customers. Initial Product Management is executed by the CEO. The company continues to thrive, and the CEO's time gets further and further sub-divided, but the technology and the product remain his/her passion.

But a new Sales Director joins the company and complains, "We’re a technical-led company. We need to become customer-driven." And that sounded fine except every new contract required custom work. Contracts were signed with a dozen clients from a dozen market segments. Soon the costs to support all these different products started to escalate. The latest customer’s voice always dominated the product plans. The CEO concludes that “customer-driven” meant “being driven by the latest customer”, and this created short-termism to the frustration of Engineering with the creation of massive technical debt and the sense of favouritism towards particular Sales team(s) creating lots of subversive backdoors to decision-makers. The price list becomes a joke, as the Sales team somehow manages to get colossal discounts on big deals using the mantra of 'a strategic customer deal,' without regard to profit margin. 

When a board member declares, “We’ve become a sales-led company. We really need to start being market-driven,” a brand specialist is head-hunted away from a big consumer product company to be your VP of Marketing. As part of a re-branding initiative, she designs a new corporate logo with a new colour scheme for the web site, new collateral, and an updated trade show booth and a big expense budget for conferences. Hundreds of thousands of dollars are spent without any change in revenue.

Soon the CFO whispers to the CEO, “Don’t you think it’s time we started controlling costs?” So, the company becomes cost-driven and started cutting out all the luxuries like travel, technical support, bonuses, and employee appreciation. Everything had to be justified with a business case and the CFO was tough on any assumptions – the only things that got past him were small evolutions of existing products. Account management was lauded for winning incremental business, usually with small but exotic customisations which started to build technical debt and make upgrades to new product releases more and more painful. And, innovation stalls, engineering resignations escalates!

At this point, when Finance has gone too far, margins have suffered, and the future product pipeline looks thin and also-ran. The CEO steps back in and is sorely tempted to take the company back to its technology roots but he/she knows that this needs to be tempered with supporting existing customers and protecting existing revenue streams. So, a new, hybrid approach is required – so the company recruits a product manager.

The CTO and the Head of Engineering welcome the idea. Finally, the Product Manager is the go-to person to make sure his/her team build the right stuff and have a solid, defensible product strategy. This prevents his team/technology from being yanked around all the time OR working on the latest, crazy C-suite initiatives OR brainless product customisations for the big customers, that don't make sense and will become a mill stone around the neck of Customer Support in the future.

But the Head of Sales is also keen. Her team can’t answer the tough questions customers are asking. Finally, someone with lots of product expertise can answer technical questions without murdering customers currently in pre-sales with infinite complexities and CLOSE SALES! The perfect PM should answer customer RFPs (Request for Proposals); AND come out on sales trips whenever they’re needed; AND be the demo jockey; AND talk authoritatively about future product releases and when they will occur. They should also be able to write some technical whitepapers and do some decent competitive analysis.

Marketing, who if you’ll remember lost a lot of credibility with their splurge on branding, are also having problems with their collateral. Their vague product brochures and website content don’t seem to hold customers' interest and come in for a lot of criticism from prospects and internally. The Product Manager can explain the product once and for all, providing decent product messaging and propositions for the collateral. A product roadmap with some realistic dates would allow her to plan marketing spend, rather than a knee-jerk reaction to product releases or to cancel events sponsorship at the last minute due to another slippage in the product release date.

Finance is relieved to hear about Product Management arriving. Finally, there’ll be someone to hold accountable for whether each product is making any money. The CFO immediately starts working on a new, complex, business case spreadsheet that the Product Manager will be tasked with completing and comes up with detailed KPIs that the new product managers will measure for him!

By this stage, Account Management & Customer Support have almost lost the will to live. Every day, they writhe in agony trying to solve exotic customer problems which Testing and Engineering struggle to recreate, never mind solve. They see the new Product Management team as the last hope to solve the Sales madness of selling solutions that are further away from the core product offering and hanging Customer Support out to dry AGAIN. Otherwise, they are definitely throwing in the towel and quitting.

Conclusion

Everyone has their view of what the perfect product manager should be doing – it’s helping them do their job! In truth, Product Management should be greasing the wheels of all these business functions because they put the product's holistic success at the centre of what they do..

About Arthur
A Product Manager who enjoys taking new technologies and concepts to market. Fascinated by software, internet and mobile sectors. Start-up and scale-up specialist. Based in Cambridge, UK. For more information, see www.stream121.com.

The challenge of beta programmes

In a previous post, I looked at the Product Management & Product Marketing model that Redgate Software (a local Cambridge-based company) uses. In it, I identified a component, that didn't appear to be addressed: beta programmes. 

 

A quick diversion into pronunciation

• In the UK, it is pronounced 'Bee-ta'.
• In the US, it is pronounced 'Bay-ta'

It's a pronunciation that I found far, far difficult to surmount when I moved to the US and when I moved back. 'Tom-mart-to' and 'Tom-may-to' was easy!

What is the purpose of a beta programme?

The build team (Engineering, Testing, Product Management, Implementation, Technical Support) want the beta programme to technically validate the latest functionality - that it works outside the lab, in real customers hands and that bugs are flushed out and those bugs are validated in subsequent releases. The end goal of a solid product should not be compromised by time constraints.

• Marketing and Product Marketing want a case study and a quote for the press release - as soon as possible please.
• Sales, Sales Engineering, the CFO wants a big name brand to do reference case study that ignites the sales pipeline - ASAP as well.

Hmm, plenty of different objectives!

Why should a customer participate in a beta programme?

I'm sure that we have all participated in these CBT (Customer-Based Testing) programmes at some point: buggy software whereby the customer does the testing - seemingly to the shocked surprise of the vendor. This can be immensely damaging to the beta programme, the provider’s brand, the relationship between participant and the provider.

And, for the customer, it's a darned good reason not to participate in a beta programme: too much effort for no advantage (or indeed negative advantage).

For this reason, providers need to ensure that, during the beta, they have the Engineering bandwidth to rapidly fix issues that are identified. This, of course, is difficult to estimate exactly how much resource is required, but essential that this remains top priority. Unfortunately, Engineering’s core capability of building beautiful, well thought-out functionality does not match the issue + investigation + quick response cycle of beta programmes – expertise that is embodied in Support or Professional Services.

Valid reasons for customers to participate in a beta programme are:

• Access to latest version - see later for my experience with CDK Global
• High-profile within the vendor's organisation: ie access to Technical resources or insight into the vendor's strategic intentions.
• Cost-reduction in fees, perhaps for a limited period

Which business functions should participate in Beta Programmes?

There are many good and bad reasons why different parts of the organisation should participate in beta programmes.



Customers
Err, yeah, of course, but some customers have a different propensity towards beta programs.

Customer whales (ie big ticket customers) that that are currently in pre-sales should definitely not be part of a beta. It may destroy the sale altogether: experience with buggy software may leave a nasty impression in the mouth - even if it is exactly the functionality that they are begging for.

Getting a quote out of a whale's marketing department may take months, if at all.

Beta software is usually provided at a discount - possibly a significant discount - a discount which may never be recovered on full release and therefore full price - and too expensive a margin to surrender.


Existing customers are great for betas - they understand what the previous product does, users and the business buyer have a positive relationship with the beta provider. Technically and financially, the provider has been validated and approved. This certainly makes it easy to draw up a shortlist.

However, there may be enough 'baggage' in the account that existing customers may derail the beta at the outset. They may demand so-called 'fixes' (missing pieces of functionality that they perceive in the existing product) or their environment / use may not match the requirements of the beta functionality. They will want compensation for their participation - and this compensation may be more valuable for them then the business benefits of the beta software.

Some of the best beta programmes I have participated in have been with new business divisions of existing customers. Commercially an existing contract is easy to extend. The vendor sees new business and the customer has the assurance of existing proof elsewhere in their organisation.


New Customers
New, NEW customers can also be really good. They genuinely behave like new customers.
They generally have more operational flexibility - they haven't honed their own processes to your product yet - and recognise that they need to create new processes.

Existing customers may say, "Yes, we'll be on your beta - as long as the product is "better" and nothing changes." Well, if the beta product aims to cannibalise the existing product, then the beta product will be "different" - better in many respects, but different in others. These differences may be negative enough to existing customers that they outweigh the new advantages.

The issue may be become more apparent if the existing customer is now off-centre from the target market with the new product. Existing customers may feel that they are being shoved to one side by the relentless progress of the vendor - and this may give the existing customer on the beta programme a sour taste in the mouth.

Another advantage of new customers is that the beta programme doesn't have to tiptoe around Account Management or Sales and get their buy-in.


Testing
Testing needs to valid the software on its way out and also to validate any issue as reported. For this reason, it is often essential that Testing has access to an environment that has beta code on it - let's call it pre-production - they are interested in isolating the real-world issues in a lab environment. Messing around with the production server is not to be recommended as it may create serious degradation to the service.


Engineering
Well, yes, they created the software and they are responsible for delivering it and fixing it. They are critically interested in technical validation in real-world environmental.


Account Management
If an existing customer is involved, the participation of Account Management (or Sales or Professional Services or whatever function manages existing relationships) is undoubtedly required, particularly at the start and end of the beta programme.

This function is compensated with incremental revenue - quite rightly, they don't want to jeopardise their account with the unknown. So, account management need reassurance that their customers' participation in the beta will help their account.

If the beta programme runs in parallel with existing product, then Account Management and Customer Support continue to support the existing in-life product. If the beta programme cannibalises the existing product, then the beta programme manages and supports the customer.


Customer Support
This function is generally wary of beta programmes, as it is often increases the amount of first-line call handling that they are required to do.


Implementation / Professional Services Team
The participation of ProServe is dependent on the amount of customisation required for the beta customers. Development would like 'vanilla' customers with the least customisation. However, this is dependent on the interest of the existing customer base in the beta programme and, with gritted teeth, the beta programme may be obliged to accept a highly customised account into the beta.

The great hope of vendors is that the new version fixes lots of issues in existing accounts - helping Account Management / Sales (unlocking new revenue), Professional Services (less implementation grief) and Customer Support (fewer issues and better relationship). And yes, it should leapfrog the competition and trample all others in the marketplace.


Special Cases - Saturated Markets
Newly all of my product experience has been in small scale-ups. However, I did have one fascinating experience working for CDK Global (see case study). As the Release Director, I was also involved with the Beta Programme - admittedly at arm's length.

CDK Global's product was the first major release in 12 years. It is worth mentioning because CDK Global dominated their market with 60%+ Total Ownership. Their customers (car dealers) operated in a highly competitive market and any new advantage could significantly tip the market. So, getting on CDK Global's beta programme could be a game changer in the short- and mid-term. When a slot on the beta programme was offered to a shortlist of existing customers, each place was eagerly snapped up.

Conclusion

Beta Programmes are extremely important to software creation teams, but managing these diverse expectations can be equally difficult - particularly when the product launch (with all those expectations!) is dependent on it.