Metrics for a GenAI World

In today's rapidly evolving landscape, organizations are embracing Generative Artificial Intelligence (GenAI) to automate routine tasks and streamline processes. However, this shift also requires rethinking the way we measure organizational success. As we move away from traditional metrics (KPIs) focused on widgets produced or tickets closed, it's crucial to identify new metrics that align with our goals in a GenAI-driven world.

What gets measured gets changed

And no, we're not talking about quantum theory. In physics as well as in our organizations, as soon as we focus on something specific, we change how the system behaves. If we measure tasks completed, the tasks will get smaller, and more of them will be ticked off. Tasks get pushed into someone else's inbox. To make our businesses more predictable, we trained our employees to be risk-averse and follow strict processes. To minimize salary expenses, we outsourced as much as we could to offshore contractors and gig workers. We set up a large part of our organizations to be automatable.

Now, we have the technology to get rid of those humans. As Ben Evans said last year, GenAI is a bit like unlimited free interns (at the current state of development). It's increasingly cheap and fast to produce good enough results for routine tasks. I like to think about it as "mediocre is free now". Highly standardized processes can be fully automated with GenAI. This will lead to several major shifts in organizations. A vast reduction of low-skilled knowledge workers (e.g. customer support functions), maybe even a reverse-globalization in some contexts, as the remaining roles get located closer to the customer again. And we'll see a challenge in talent development; if companies don't need to hire junior workers (since GenAI is better than humans with little work experience) how will they make sure they have highly skilled talent to replace those retiring or leaving the company for other reasons?

Identifying New Metrics

To adapt to this new reality, organizations must begin by identifying new metrics that align with their objectives in a GenAI-driven world. These may include measures such as employee satisfaction, collaboration across departments, customer satisfaction, and the ability to tackle complex problems. By focusing on these areas, companies can harness the full potential of both human and artificial intelligence.

For example, a company might consider implementing metrics related to employee engagement, teamwork, and continuous learning, which would encourage employees to develop new skills and contribute more effectively to the organization's success.

How might you reward the employee who spends time (and budget) to unblock another team? How might you incentivize employees to expand their skillset or mentor others?

Traditional productivity-focused KPIs make this kind of work almost impossible.

Establishing new metrics around collaboration and continued learning will help mitigate the risk of misuse or over-reliance on GenAI in decision-making processes. Unchallenged, automated decision-making could lead to unintended consequences or a lack of skilled human oversight of critical aspects of a business. Organizations should maintain a human-centric approach to AI implementation and ensure that AI systems are transparent, explainable, and auditable.

How might you incentivize the use of GenAI, while encouraging employees to challenge decisions automated systems suggest to them?

The Rewards

Thinking about human and environmental factors in a profit-driven organization isn't new. B-Corporations already started to implement metrics that go beyond pure commercial stats. These organizations demonstrate that by focusing on factors such as employee well-being, customer satisfaction, and environmental impact, businesses can outperform their peers in terms of revenue, growth, stability, resilience, and sustainability. A white paper from B Lab suggests that B Corps outperform other businesses when it comes to these factors. From 2019 to 2021, B Corps were more likely to grow their revenue and their headcounts, and they were more resilient – with 95% remaining in business in 2023 compared to 88% of non-B Corp businesses.

With the shift to highly automated hyper-personalized products and services, a wider range of metrics – like B Corps model them – is essential not only from a sustainability and ethics perspective but also to ensure organizational resilience.

Personal Resilience

So what can you do, as an individual worried about the future of your career? GenAI represents a shift in technology that's different from innovations we've seen before. Whereas say the "cloud" simply represented a shift in where the computers sit (in-house vs offshore) and who owns them, or "mobile" changed how we access digital services (stationary vs anywhere), GenAI changes what we do.

To think about what that means for our careers we might be able to learn from GenAI itself. The transformer architecture (the T in GPT) is a key part of what enabled the AI revolution. Before the transformer, AI was fairly slow and specialized. Grossly simplified, machine learning had to compute a lot of possibilities, one at a time until it found the solution. Now with the transformer, it's a little like AI has fuzzy attention on everything and it only goes through all the detailed calculations once it's decided which parts of everything are relevant to the task at hand.

In the last two decades or so, our job roles have become more and more specialized. For example, when I studied design, my university – like most – offered one course in digital that educated "Interaction Designers"; digital all-rounders who have a broad understanding of technology, design, and research and could seamlessly switch between different skillsets. Now people have very specific job roles like UX designer, UX researcher, UI designer, UX developer, UI developer, CX designer,... and so on. These highly standardized roles and processes are a lot easier to automate. The easiest way to get replaced by GenAI is to stay deep in a micro-niche of a job role. Being able to recognize outliers, make connections across disciplines, and make sense of complex issues is one way to future-proof yourself.

How might you branch out to get an understanding of adjacent roles and skills to get more of that fuzzy attention on everything yourself?

What do you need to learn to collaborate with GenAI to do highly specialized, repetitive tasks?

The way forward

The adoption of GenAI presents an opportunity for organizations to reimagine their approach to work, measurement, and success. By shifting our focus from traditional KPIs to metrics that align with the evolving nature of human-AI collaboration, we can unlock new levels of innovation, creativity, and impact within our organizations. As you navigate this exciting new landscape, it's crucial to identify new metrics that accurately reflect the changing nature of work.

What questions are you holding right now?

_How might you help your team adapt to this new shift? _

And how might you future-proof your own career?

If this article made you think and you want to continue the conversation, please reach out.

I offer 1:1 coaching as well as facilitated workshops for teams to help you adapt to the new opportunities GenAI brings to how we work.

Becoming a Cyborg

What if we think of GenAI as a third hand or a second brain? Rather than competing with GenAI, we could also use it as an extension of ourselves. In this last section, I'll share some interesting GenAI solutions that empower us humans. I'll focus on ideas that are not simply focusing on automating boring tasks, but explore how we might collaborate with AI in useful ways.

AudioPen

One of the AI apps that I keep coming back to is AudioPen at its core it's an AI-powered voice recorder that automatically creates transcripts for you. And while it's definitely not the first or only audio-transcription service, it's doing some interesting things. Mainly it's making you think freely by doing less.

You record up to 15min of random babbling
AudioPen transcribes the voice note
It rewrites the note for clarity (removing uhms, fillers, repetitions)
You get to choose and fine-tune the tone of voice it uses
It creates sharable images for the summary
It integrates with Zapier to save your notes to Notion, Obsidian, or your second brain of choice.

In my experience, AudioPen's transcription and writing quality tops the likes of otter.ai and something about the simplicity of the interface, the single job of helping you make sense of fleeting thoughts, makes it so much more joyful and impactful to use.

What do you think? What's your favorite AI tool?

Some of the links in this article are affiliate links, I might – at no cost to you – earn a referral fee if you end up buying one of these products or services.