Consultants performed 23% worse when using AI: BCG study on GenAI

Artificial intelligence is advancing rapidly, and one of the most exciting frontiers is generative AI. Systems like GPT-4 and DALL-E 3 can now generate remarkably high-quality text, images, code, and more from simple prompts. A new BCG study on GenAI in collaboration with leading academics, including Harvard and MIT, provides incredible insights into these questions

  • The potential for generative AI to disrupt businesses and industries is massive.

But questions remain:

  • Where and how can generative AI create tremendous value?
  • What are the potential downsides if deployed poorly?
  • Just how competent are these systems today?

A fascinating new study from BCG in collaboration with leading academics provides incredible insights into these questions.

About the Research Study

  • Conducted with over 750 consultants from BCG worldwide
  • Tested generative AI on real-world business tasks:
  • Creative product innovation and ideation
  • Complex business problem solving using qualitative and quantitative data
  • Measured performance when using AI vs. not using AI

Key Finding 1: Massive Upside for Creative Ideation

When used for creative ideation and content generation, the AI delivered striking results:

  • Around 90% of participants created higher quality output when using the AI
  • On average, they achieved 40% better performance compared to those doing the task without AI
  • Even top performers boosted results by 17% with the AI over their own unaided work

The best approach was simply to use the AI’s output directly, without trying to edit or enhance it.

  • Editing the AI text reduced performance – quality dropped 17 percentile points for every 10% of divergence from the original.

For creative tasks, today’s AI often surpasses what even highly skilled professionals can produce on their own.

Key Finding 2: Major Risks of Overreliance

However, for complex business problem solving, the story was very different:

  • Participants performed 23% worse when using the AI versus doing the task unaided
  • Despite warnings, many accepted the AI’s wrong answers as valid
  • Providing basic training did not help, and actually resulted in even worse performance, potentially indicating overconfidence in one’s ability to apply the AI appropriately

Unlike narrowly defined content creation, synthesizing messy real-world data to make nuanced judgements remains firmly outside today’s AI capabilities.

Leaders must be exceedingly wary of overreliance risks on tasks requiring true comprehension beyond pattern recognition.

Key Finding 3: Creativity Tradeoffs

While the AI boosted individual creative output, some concerning tradeoffs emerged:

  • Collectively, ideas became 41% less diverse
  • 70% worried relying too much on AI could reduce their own creative abilities long-term

Leaders must remain vigilant that applying generative AI does not reduce workforce creativity over time.

Implications for Leaders

Realizing the massive potential of AI while avoiding pitfalls requires rethinking how it is deployed:

  • Invest heavily in internal data infrastructure to train models
  • Identify tasks where today’s AI excels, and reengineer workflows
  • Plan workforce strategies to leverage complementary human-AI capabilities
  • Continually test applications rigorously as AI progresses rapidly

The scale of disruption will depend on how wisely leaders implement AI. With the right strategies, generative AI can enable tremendous competitive advantage.

Key Takeaways from BCG study on GenAI

Critical insights for leaders:

  • Generative AI already excels at creative ideation beyond most professionals
  • But it still falls short on complex real-world reasoning
  • Overuse can reduce creativity diversity and individual abilities
  • Realizing massive benefits requires rethinking roles, strategies, and workflows

Leaders should:

  • Catalog tasks where AI excels and redesign workflows
  • Identify decisions still requiring human judgement
  • Develop data capabilities to train models
  • Begin realigning workforce strategies
  • Build rigorous testing regimes
  • Monitor for declines in workforce creativity
  • Make adaptivity a core competency
Creative IdeationComplex Problem Solving
Embrace AI AssistanceApply AI Critically
Don’t Waste Efforts EditingBeware Overreliance

Importance of the Sanctity of AI

With careful, adaptive strategies, leaders can realize incredible benefits from generative AI while mitigating risks. Success requires resisting hype and fear in order to integrate AI wisely.

The future is human and artificial intelligence working together. Leaders who embrace this future with vision and practicality can create tremendous value. Those who ignore or dismiss it place their organizations at existential risk.

Leave a Reply

Your email address will not be published. Required fields are marked *