Build speech AI apps with LeMUR and LLMs

The explosion of audio content has created an unprecedented opportunity for businesses. Organizations processing audio data continue growing rapidly, yet most struggle to extract meaningful intelligence from their conversations, meetings, and calls.

Here's the challenge: traditional speech-to-text gives you transcripts, but transcripts alone don't drive business decisions. You need intelligence—summaries that capture key points, answers to specific questions, and structured data that integrates with your existing systems.

This is where Large Language Models (LLMs) transform speech applications from simple transcription tools into intelligent business platforms. But connecting speech recognition to LLMs has historically required complex engineering and significant resources.

LeMUR helps ease this challenge, helping teams move more quickly to apply LLMs to speech data.

LeMUR: Applying LLMs to speech data

LeMUR is AssemblyAI's framework for applying Large Language Models (LLMs) to speech data, reducing the technical complexity of building speech AI applications. Instead of managing multiple APIs, handling audio preprocessing, and coordinating between speech and language models, LeMUR provides a single interface that takes your audio files and returns intelligent insights.

Here's how the architecture works:

LeMUR Architecture Flow:

Audio Input → Speech Recognition → Transcript Processing → LLM Analysis → Structured Output
     ↓              ↓                    ↓                 ↓              ↓
  Raw Audio    →  Text Transcript  →  Context Prep  →  Intelligence  →  Your Application

The framework handles three critical integration points that typically consume weeks of development time: audio quality optimization, transcript context management, and LLM prompt engineering. This lets your team focus on building features that differentiate your product rather than managing infrastructure complexity.

What makes LeMUR particularly powerful is its ability to maintain conversation context across long audio files while applying sophisticated reasoning to extract business-relevant insights. LeMUR can process up to 100 files or 100 hours of audio content in a single API call.

LeMUR core functions

LeMUR provides several core functions that address the most common speech intelligence use cases. Each function is designed to solve specific business problems while remaining flexible enough to adapt to various industries and applications.

Summarization

The summarization function transforms lengthy audio content into concise, actionable insights. Unlike generic text summarization, LeMUR's audio summarization capabilities understand conversation dynamics, speaker relationships, and business context.

For example, a 90-minute board meeting becomes a structured summary highlighting decisions made, action items assigned, and key discussion points—formatted specifically for executive review. Sales teams use this to quickly understand prospect calls without listening to entire recordings.

The business impact is immediate: executives report saving 4-6 hours weekly on meeting follow-ups, while sales teams can review 3x more prospect interactions in the same timeframe.

Question and answer

The Q&A function lets you ask specific questions about audio content and receive precise, contextual answers. This goes far beyond keyword search—LeMUR understands the semantic meaning of conversations and can answer complex questions about intent, sentiment, and outcomes.

Customer success teams use this to quickly identify why clients are churning by asking questions like "What concerns did the customer raise about our pricing?" or "How did the customer respond to our retention offer?" The system analyzes the entire conversation context to provide accurate, actionable answers.

Legal teams leverage Q&A to review depositions and client meetings, asking targeted questions about specific topics without manually reviewing hours of recordings.

Custom prompts

Custom prompts provide the flexibility to extract any specific information or insight from your audio content. This function essentially lets you program LeMUR to understand your unique business context and return precisely formatted results.

Healthcare organizations use custom prompts to extract structured medical information from patient consultations, automatically formatting results for electronic health records. Financial services firms extract compliance-related information from advisor-client meetings, ensuring regulatory requirements are consistently met.

The key advantage is customization without complexity—you define what you need in natural language, and LeMUR handles the technical implementation.

Extracting data

The data extraction function identifies and structures specific information from conversations, transforming unstructured audio into organized, searchable data. This capability enables call data extraction that integrates directly with existing business systems.

Common applications include extracting contact information from sales calls, identifying action items and deadlines from project meetings, and pulling key metrics from performance reviews. The extracted data can be automatically formatted for CRM systems, project management tools, or custom databases.

This function particularly benefits organizations processing high volumes of similar conversations—customer service centers extracting issue categories and resolution details, or recruitment firms identifying candidate qualifications and interview outcomes.

Use case-specific LeMUR workflows

Have a more specific use case you need for LeMUR? These workflows will help you get started quickly:

• Implement a sales playbook using LeMUR
• Pass context from previous LeMUR requests
• Generate meeting action items with LeMUR
• Segment a phone call using LeMUR
• Generate SOAP notes using LeMUR

Real-world applications

The combination of LeMUR's core functions creates powerful applications across multiple industries. Here's how leading organizations are implementing speech AI solutions:

Meeting intelligence applications

Use case comparison:

Meeting intelligence platforms like Otter.ai and Fireflies.ai demonstrate the market demand for speech AI applications. These platforms process millions of meetings monthly, providing summarization, action item tracking, and searchable conversation archives.

Organizations implementing meeting intelligence solutions report significant productivity gains: 65% reduction in meeting follow-up time, 45% improvement in project coordination, and 30% increase in action item completion rates.

The key success factor is customization—generic meeting summaries provide limited value, but summaries formatted for specific business processes drive measurable outcomes.

Sales intelligence applications

Sales intelligence platforms represent one of the fastest-growing applications of speech AI technology. Platforms like Gong and Chorus.ai have demonstrated how conversation analysis can dramatically improve sales performance.

LeMUR enables similar capabilities for organizations building internal sales intelligence tools:

• Deal analysis: Automatically extract competitor mentions, pricing discussions, and decision-making criteria from sales calls
• Coaching insights: Identify successful conversation patterns and areas for improvement across sales teams
• Pipeline intelligence: Track deal progression signals and risk factors mentioned during prospect conversations

Sales teams using conversation intelligence report 20% higher close rates and 15% shorter sales cycles. The technology pays for itself within months through improved deal conversion alone.

Customer service solutions

Customer service represents the largest opportunity for speech AI applications, with contact centers processing billions of interactions annually. LeMUR enables several high-impact use cases:

Call center implementations use LeMUR to automatically categorize support tickets, identify escalation triggers, and extract customer satisfaction indicators. This reduces manual quality assurance workload by 80% while improving consistency in issue identification.

Support automation examples include automatically routing calls based on conversation content, generating follow-up summaries for customer records, and identifying training opportunities for support agents.

The business case is straightforward: a 1,000-agent contact center can save $2.4M annually through improved efficiency and reduced quality assurance costs.

Enhanced speech recognition foundation

LeMUR builds on AssemblyAI's industry-leading speech recognition models:

Universal: Production-ready model with 27.4% faster inference time and up to 14.8% improvement in formatting accuracy. Supports 99 languages with automatic language detection and speaker diarization for 95 languages.

Slam-1: Prompt-based Speech Language Model offering up to 66% reduction in missed entities with domain-specific customization. Available in public beta for English with superior accuracy through contextual understanding.

Other resources

Video Tutorials:

• How to Create Speaker-Based Subtitles for Your Videos with AI: Learn advanced audio processing techniques
• Build an AI Voice Translator: Explore multilingual speech applications
• Build An AI Chat Bot In Java: Understand real-time speech processing implementation

These tutorials provide hands-on guidance for implementing advanced speech AI features, covering everything from basic transcription to complex multi-language applications.

Implementation Guides:

• Getting started documentation covers authentication, basic setup, and first API calls
• Advanced integration guides show how to connect LeMUR with popular business tools‍
• Industry-specific examples demonstrate proven implementation patterns

Final words

Speech recognition combined with Large Language Models transforms how organizations extract value from audio content. LeMUR eliminates the traditional barriers to building speech AI applications—technical complexity, integration challenges, and ongoing maintenance requirements.

The business case is clear: organizations implementing speech AI solutions report 40-60% improvements in productivity for audio-intensive workflows, with measurable returns typically achieved within 2-3 months. More importantly, these applications enable new business capabilities that weren't previously feasible.

Whether you're building meeting intelligence tools, sales analytics platforms, or customer service automation, LeMUR provides the foundation for transforming speech data into competitive advantage. Organizations implementing speech AI solutions gain competitive advantages through better customer understanding and data-driven decision making.