Generative AI: How do we measure success?

How to map both incremental and exponential AI initiatives to OKRs
May 10, 2023

The axiom I was taught in my early days in business was simple but stuck with me: "If you can't measure it, you can't manage it." To reach the potential of generative AI strategies, organizations must measure their success even in their early days to justify ongoing investment. While generative AI promises cost-savings or revenue gains, true measures of generative AI programs will also include innovation as well as human impacts (good and bad) in their scorecards.

Here, I'll cover different ways to measure success—from user satisfaction to the number of AI-generated ideas—and unpack the difference between incremental and exponential definitions of generative AI success. Finally, we'll discuss how to map generative AI projects to OKRs.

Please comment with your views, stories and facepalms—we'd love to hear from you!

Overview of OKRs and KPIs for Generative AI

For the uninitiated, OKRs (objectives and key results) are a goal-setting framework for organizations, teams, and individuals, focused explicitly on intended outcomes and key drivers (in contrast to KPIs which may measure the performance of a system, but which are not always measuring outcomes). OKRs are intended as an antidote to 'vanity' or 'churn' metrics which measure activity without mapping to meaningful results. For more on OKRs, project management software company Asana has a great guide.

Avoid getting distracted by 'shiny tech'—each incremental use case for generative AI you discover should relate to an OKR your company already values. If not, it's probably not worth investing in. Exponential use cases for generative AI may not map to existing OKRs, but they should at least connect to your company DNA in some way.

Incremental improvements to operations using generative AI

Generative AI has the potential to revolutionize how organizations operate. Using sophisticated algorithms and language learning models, businesses can automate or improve operations-related tasks such as customer service and financial analysis. This technology offers numerous advantages over traditional approaches, including cost savings and improved efficiency, by making data and systems 'conversational.' In a survey conducted in the first workweek of January 2023, less than six weeks from ChatGPT’s mass-market release, 29% of Gen Z workers reported that they already used the tool in the workplace

For example, instead of needing to know complex Excel formulas or delegate inquiries to a data team, an executive could quickly ask questions of their company's data as if they had an entry-level data scientist with them at all times. (The accuracy of this requires a lot of investment in good underlying data, like many generative AI use cases).

The incremental improvements generated by generative AI might go unnoticed initially, but will yield substantial benefits for organizations in the long run. “Could” is the key term here; poorly-implemented solutions might create more friction than benefit—for example, using ChatGPT on its own may not result in brand-aligned tone in messaging and can expose a company to liability through data breaches, inaccurate answers, or poor advice. “Organizations” is the other key term—what’s good for the organization may not always be good for its customers or teams—so they should be considered at least as much as any balance sheet wins. Gartner predicts that by 2025, 30% of outbound messages from corporations will be generated by AI—up from 2% in 2022. Will they meet your customers’ expectations?

Implementing generative AI programs into organizational culture and operations holds potential for improvement across all departments, but is not a substitute for critical thinking and empathy.

Companies often have incremental OKRs focused on objectives like:

  • Satisfaction of customers
  • Earning market share
  • Strong revenue and profitability

Right now, generative AI can contribute to incremental 'faster, better and cheaper' key results of company OKRs :

  • Improved code quality
  • Faster and more friendly customer service
  • Marketing and social media content for improved engagements
  • Cost reduction of rote tasks like copyediting

However, remember that you can't cost-cut your way to innovation. Will you use your AI-based savings to invest back into your business’ people and innovative R&D? Or just pocket it? Incremental operational improvements using generative AI may seem groundbreaking now, but will soon become a baseline in most industries. 

Exponential innovation using generative AI

Generative AI has the potential to produce unprecedented levels of innovation in how businesses operate and create new products, services, and experiences. In an era where nearly every company is at least asking about generative AI, it is becoming crucial for businesses to leverage generative AI technology not just to cost-cut, but to be competitive.

For example, generative AI, when combined with human critical thinking, can:

  • create content customized for a specific user or industry with generative AI can help organizations boost engagement and earn gratitude while also driving revenue;
  • identify new combinations of customer needs and features that can result in new offerings that might have been 'off the radar' of existing product leaders;
  • make previously unprofitable offerings may become viable (such as an on-demand tutor for students, or public health offerings to those who cannot afford or reach a doctor's visit); and
  • solve previously un-solvable problems may become solvable (as with AI-generated pharmaceuticals which are now being tested—Gartner predicts that 30% of drug discoveries in 2025 will be from generative AI).
Photorealistic AI rendering of a robot and professional woman of color sitting across table from each other, working on a drawing. The woman's wrist appears to be bionic due to AI rendering errors.
(I know, I know, robots again, but it is pretty cool that this was Firefly’s first shot at ‘human and robot collaborating on a drawing.’)

From music composition to video game design, generative AI can produce seemingly creative works when well-prompted (though it’s not creative in the same sense as humans). By creating audio or visual content tailored specifically to an individual’s tastes and preferences, organizations can craft experiences that are more enjoyable and engaging than ever before—Spotify is offering everyone their own private DJ, for example. In a sighting from the fashion industry, generative AI can synthesize limitless product photos where it may not have otherwise been financially viable for fashion companies to do photo shoots for every possible product color and size permutation, not to mention with models with a variety of physical appearances and ethnicities. Levi's controversially did this kind of photo synthesis with Lalaland.ai, raising concerns about digital blackface and further discrimination against models who are minorities. Adobe's Firefly suite of AI offerings allows even lay designers to quickly synthesize any number of scenes, allowing for a rapid decrease in the time from idea to mockup. And these tools can be applied in very customer-specific ways, as we've been seeing for a while with chatbots or 'try these glasses on my face' tools.

Some best practices should be taken into consideration when leveraging generative AI technology for maximum exponential impact:

  • build Digital Fluency, AI readiness, and data ethics skills among decision-makers so they understand the technology and the business models around it;
  • modernize the data supply chains within your company (otherwise, generative AI solutions will only be surface-level);
  • especially if you're innovating with your own AI models, ensure that data science teams have access to the right resources and decision-makers;
  • evolve your understanding of customer needs so that generative AI initiatives can deliver targeted solutions; and
  • encourage experimentation and pilot programs.

By following these best practices, organizations can harness the power of generative AI technology for exponential innovation and success.

Exponential objectives might look like:

  • Innovation of new products and business models
  • Creation of digital offerings
  • Identification of new investment opportunities

Generative AI can drive key results like:

  • Faster ideation and prototyping
  • Drastically decreased time-to-market for alpha and beta products
  • Synthesis of new ideas and research at unprecedented speeds and identification of new market trends

Quantitative and qualitative success

Generative AI programs have the potential to transform how organizations operate, allowing them to unlock new levels of efficiency and productivity or even create entirely new offerings.

Quantitative measures are probably the most convenient (and demanded). Cost savings are often an important metric for tracking success, of course. Generative AI can help reduce operational costs by automating repetitive tasks such as customer service or support inquiries or laborious editing tasks (turning things from first-person to third-person, for example, or writing customer case studies). It also has the potential to increase sales and revenue through improved customer engagement—provided the AI voice of the company is appropriate to the relationship with the customer. Additionally, generative AI can streamline back-end operations (like IT services, writing documentation, summarizing quantitative data in prose, or translating content).

Qualitatively, customer satisfaction is a key indicator of success when using generative AI programs—as is customer perception of the AI strategy. Investing in properly-managed generative AI solutions promises to pay off through increased customer loyalty and the long-term sustainability of your business model, while poorly-launched approaches can alienate customers and cause real-world harm. Employee satisfaction is another important qualitative measure—do your team members feel empowered by the use of generative AI? Are they happy with its results and able to offload less-satisfying work? Or do they just feel deeply anxious about their jobs? For both customers and team members, good, proactive messaging about what you are or are not going to do with generative AI is essential.

Measuring the success of early generative AI programs and pilots

Your generative AI program may be in its early days, but you can still demonstrate empirical progress.

Incremental pilot programs can identify areas that need improvement or optimization to see if generative AI can be helpful. Meanwhile, exponential pilot programs can help identify business model, pricing, or product opportunities and test their feasibility and user/market's interest while identifying potential ethical risks.

Measuring both the quantitative and qualitative success of a pilot generative AI program is a little different than other kinds of projects. Here are some key questions to consider:

  • Objectives: What are the objectives of this project? Is it incremental, or are you aiming for something more exponential? How do these objectives contribute to your overall OKRs—or change them?
  • Stakeholders & Users: Who are the users and stakeholders benefiting from AI in this program? Are there any groups that are underserved, negatively impacted, or have difficulty using it? Labor and talent impacts should absolutely be part of any pilot project's assessment—people need as much lead time as possible to update their skills or pivot in their careers.
  • Network Effects: Are you progressing towards network effects such as an increased number of data points or larger user numbers? Network effects usually have to occur prior to profit or revenue increases, so it's important to measure them early on to justify continued investment in long-tail strategies (read more in our piece on The Exponential Journey).
  • Lessons Learned: Have there been any lessons learned from running this pilot program that can be applied to other projects in the future?
  • Near-term benefits and risks: Did generative AI tech result in faster, better, and/or cheaper results than a human user? Were there any positive surprises (like new insights)? Or was the more friction or negative outcomes than the manual performance of tasks?

As you consider these questions, here are some additional areas to measure to really get at how the underlying tech is functioning (or how well you are prompting it):

Accuracy of tasks generated by AI: How successful were the models used at producing reliable output in comparison with human-generated results?

Time to completion is a key metric when tracking the success of generative AI programs. By automatically completing complex processes that would otherwise take humans significantly longer, businesses can save time and money—ultimately leading to improved customer experience and increased bottom-line performance. But only if the output is of acceptable quality.

The amount of fine-tuning needed for an acceptable result is also important when measuring the success of a generative AI strategy. This metric indicates how much effort is required from developers for the model's output to meet customers' expectations—if too much tweaking is necessary, then this could indicate that the model needs further training or adjustment.

For example, measuring the number of innovative ideas generated or scaled through generative AI can provide valuable insights into whether the investment in this technology has been worthwhile.

Other key indicators in applications of AI to creative or innovation fields include the number and quality of ideas created through generative AI compared with manual efforts, any increases or decreases in time between idea generation and execution, and any harms caused (or prevented) by using generative AI technology instead of a human-only process. Organizations should also survey customers and users to ascertain satisfaction with the results of Generative AI models compared to manual tasks.

Demonstrating generative AI success in the organization

Organizations that want generative AI's benefits (or to avoid being disrupted by them) have to have deep digital fluency while also working on their AI fluency. With these two competencies combined, leaders can invest in Generative AI and hold teams accountable—without stifling innovation. Otherwise, the 'incremental immune system' of most organizations will preclude generative AI. Digital Fluency and AI Fluency programs should especially emphasize the importance of robust data capabilities, how exponential initiatives should be managed differently than incremental ones, and realistic expectations (and ethical implications) of today's generative AI. At the same time, organizations should be appointing AI program directors, AI czars or other key point-people to help coordinate their efforts. (May Habib of Writer wrote an on-point post about a proto-job description for an AI program director worth checking out). 

Candid conversations about generative AI's impact on strategy, jobs, and customers will build trust and respect between business and tech stakeholders. Go for it—but measure your progress and impact so that you can be proud of what you create. 

This is one of a multi-part series on Questions to Answer in AI Strategy. MJ Petroni is a Digital Fluency and AI readiness speaker, author, and accelerator. You can read even more in the Digital Fluency Guide.

Comment and share on LinkedIn

Key Terms

Select any number of buttons on the left to see varieties of data sources available for analysis.