Claude Code realtime

Event-Driven Architecture Bottlenecking Under Load

Your application's event-driven architecture works fine in development but collapses under production load. Events pile up in queues, consumers can't keep pace with producers, and end users experience increasing delays in seeing updates, receiving notifications, or having their actions processed.

Event-driven architectures are powerful but introduce complexity that AI-generated code often doesn't handle correctly. The initial implementation works for single-digit users but fails at scale because it lacks consumer scaling, proper partitioning, dead letter queues, and backpressure mechanisms.

Symptoms include growing queue depths, increasing latency on event processing, out-of-memory errors on consumer processes, and eventually dropped events when queues hit their size limits.

Error Messages You Might See

Queue depth exceeding threshold: 50000 events pending Consumer lag increasing: 30s behind OOM killed: event-consumer process Event processing timeout after 30000ms Dead letter queue overflow: 10000 failed events
Queue depth exceeding threshold: 50000 events pendingConsumer lag increasing: 30s behindOOM killed: event-consumer processEvent processing timeout after 30000msDead letter queue overflow: 10000 failed events

Common Causes

  • Single consumer for all events — One process handles all event types sequentially instead of parallel consumers per event type
  • No consumer group scaling — The architecture doesn't support multiple consumer instances processing the same queue in parallel
  • Missing backpressure — Producers flood the queue faster than consumers process, with no mechanism to slow down producers
  • In-memory queue instead of persistent — Using an in-process event emitter that loses all events on restart and can't be distributed
  • No dead letter queue — Failed events are retried infinitely, blocking the queue for other events

How to Fix It

  1. Use a production message broker — Replace in-memory events with Redis Streams, RabbitMQ, or Apache Kafka for persistent, distributed event processing
  2. Scale consumers horizontally — Run multiple consumer instances with consumer groups so events are distributed across workers
  3. Implement dead letter queues — After 3-5 retries, move failed events to a dead letter queue for manual inspection instead of blocking the main queue
  4. Add backpressure — Monitor queue depth and slow down producers when queues exceed a threshold
  5. Partition by entity — Ensure events for the same entity are processed in order, but events for different entities can be processed in parallel

Real developers can help you.

Matthew Jordan Matthew Jordan I've been working at a large software company named Kainos for 2 years, and mainly specialise in Platform Engineering. I regularly enjoy working on software products outside of work, and I'm a huge fan of game development using Unity. I personally enjoy Python & C# in my spare time, but I also specialise in multiple different platform-related technologies from my day job. Mehdi Ben Haddou Mehdi Ben Haddou - Founder of Chessigma (1M+ users) & many small projects - ex Founding Engineer @Uplane (YC F25) - ex Software Engineer @Amazon and @Booking.com Richard McSorley Richard McSorley Full-Stack Software Engineer with 8+ years building high-performance applications for enterprise clients. Shipped production systems at Walmart (4,000+ stores), Cigna (20M+ users), and Arkansas Blue Cross. 5 patents in retail/supply chain tech. Currently focused on AI integrations, automation tools, and TypeScript-first architectures. zipking zipking I am a technologist and product builder dedicated to creating high-impact solutions at the intersection of AI and specialized markets. Currently, I am focused on PropScan (EstateGuard), an AI-driven SaaS platform tailored for the Japanese real estate industry, and exploring the potential of Archify. As an INFJ-T, I approach development with a "systems-thinking" mindset—balancing technical precision with a deep understanding of user needs. I particularly enjoy the challenge of architecting Vertical AI SaaS and optimizing Small Language Models (SLMs) to solve specific, real-world business problems. Whether I'm in a CTO-level leadership role or hands-on with the code, I thrive on building tools that turn complex data into actionable value. legrab legrab I'll fill this later Milan Surelia Milan Surelia Milan Surelia is a Mobile App Developer with 5+ years of experience crafting scalable, cross-platform apps at 7Span and Meticha. At 7Span, he engineers feature-rich Flutter apps with smooth performance and modern UI. As the Co-Founder of Meticha, he builds open-source tools and developer-focused products that solve real-world problems. Expertise: 💡 Developing cross-platform apps using Flutter, Dart, and Jetpack Compose for Android, iOS, and Web. 🖋️ Sharing insights through technical writing, blogging, and open-source contributions. 🤝 Collaborating closely with designers, PMs, and developers to build seamless mobile experiences. Notable Achievements: 🎯 Revamped the Vepaar app into Vepaar Store & CRM with a 2x performance boost and smoother UX. 🚀 Launched Compose101 — a Jetpack Compose starter kit to speed up Android development. 🌟 Open source contributions on Github & StackOverflow for Flutter & Dart 🎖️ Worked on improving app performance and user experience with smart solutions. Milan is always happy to connect, work on new ideas, and explore the latest in technology. Yovel Cohen Yovel Cohen I got a lot of experience in building Long-horizon AI Agents in production, Backend apps that scale to millions of users and frontend knowledge as well. ISHANTDEEP SINGH ISHANTDEEP SINGH Senior Software Engineer with 7+ years of experience in React, JavaScript, TypeScript, Next.js, and Node.js. I’ve also worked as a tech lead for startups, owning end-to-end technical execution including architecture, development, scaling, and delivery. I bring a strong mix of hands-on coding, product thinking, and technical leadership, and I’m comfortable building products from scratch as well as improving and scaling existing systems. Matthew Butler Matthew Butler Systems Development Engineer @ Amazon Web Services Daniel Vázquez Daniel Vázquez Software Engineer with over 10 years of experience on Startups, Government, big tech industry & consulting.

You don't need to be technical. Just describe what's wrong and a verified developer will handle the rest.

Get Help

Frequently Asked Questions

When should I switch from in-memory events to a message broker?

If your app has more than one server instance, needs event persistence across restarts, or processes more than 100 events per second, use a message broker like Redis Streams or RabbitMQ.

How do I monitor event processing health?

Track three metrics: queue depth (events waiting), consumer lag (how far behind consumers are), and processing time per event. Alert when any exceeds your SLA thresholds.

Related Claude Code Issues

Can't fix it yourself?
Real developers can help.

You don't need to be technical. Just describe what's wrong and a verified developer will handle the rest.

Get Help