How I Built a Fault-Tolerant Notification System with BullMQ

01Why Notifications Are Harder Than They Look

A notification system sounds simple at first: user action, enqueue a job, deliver the event. In production, retries, restarts, concurrency, and noisy traffic quickly make that model fragile.

• A failed delivery needs retries without sending the same notification twice.
• Queue or worker restarts should not lose in-flight jobs.
• Burst traffic should not flood a user's feed with repetitive events.
• A single bad job should not bring down the worker process.

02The Architecture

NotifyX uses a queue-driven pipeline where the API enqueues delivery work, BullMQ workers process it, Redis Pub/Sub fans it out in real time, and MongoDB persists notification history and read state.

• BullMQ provides durable jobs, retries, and scheduling.
• Redis Pub/Sub decouples worker execution from live socket delivery.
• MongoDB stores notification history and read state.

Client Action -> API -> BullMQ Queue -> Worker -> Redis Pub/Sub -> Socket.io -> Client
                                      |
                                      v
                                MongoDB Persist
                                      |
                                      v
                              Dead Letter Queue

03The Idempotency Problem

The hardest bug class was duplicate delivery after retries. A worker can begin processing, hit a transient failure, and then re-run the same logical event.

To prevent duplicates across retries and restarts, the design uses a fast Redis guard plus a persistent MongoDB uniqueness check.

• A single `SET key value NX EX ttl` atomically claims delivery *and* sets the expiry — no gap where a crash could leave a key without a TTL.
• MongoDB unique indexes remain the fallback when Redis state is lost.
• Together they protect against duplicate sends caused by retries, races, and restarts.

const idempotencyKey = `notif:delivered:${jobId}`;

// SET NX EX in one command — atomic claim + TTL. A separate
// setnx + expire pair leaks a permanent key if the process
// dies between the two calls.
const claimed = await redis.set(idempotencyKey, '1', 'NX', 'EX', 86400);

if (!claimed) return;

notificationSchema.index({ userId: 1, eventId: 1 }, { unique: true });

try {
  await Notification.create({ userId, eventId, ...data });
} catch (err) {
  if (err.code === 11000) return;
  throw err;
}

04Retry Logic with Exponential Backoff

Transient failures should not permanently lose notifications. BullMQ retries jobs with exponential backoff so short-lived outages can recover automatically.

notificationQueue.add('deliver', payload, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000,
  },
});

After the final failed attempt, the job is moved to a dead-letter queue for manual inspection and replay instead of being silently dropped.

05Rate Limiting

The system also protects itself from traffic spikes by tracking request volume in Redis sorted sets and enforcing sliding-window limits globally and per user.

async function checkRateLimit(userId) {
  const globalKey = 'ratelimit:global';
  const userKey = `ratelimit:user:${userId}`;
  const now = Date.now();
  const windowMs = 60 * 1000;

  const pipeline = redis.pipeline();
  pipeline.zremrangebyscore(globalKey, 0, now - windowMs);
  pipeline.zadd(globalKey, now, `${now}-${Math.random()}`);
  pipeline.zcard(globalKey);
  pipeline.expire(globalKey, 60);

  return pipeline.exec();
}

Sorted sets make the rate limit precise over a moving time window instead of resetting in bulk every minute.

06Batch Notifications

Repeated actions in a short window should feel like one grouped update, not a flood of alerts. NotifyX batches similar events for 30 seconds before delivering a single summary notification.

const batchKey = `batch:${userId}:${eventType}`;
const existing = await redis.get(batchKey);

if (existing) {
  const batch = JSON.parse(existing);
  batch.count += 1;
  batch.latestActor = actorId;
  await redis.setex(batchKey, 30, JSON.stringify(batch));
} else {
  await redis.setex(batchKey, 30, JSON.stringify({ count: 1, actorId }));
  await notificationQueue.add('deliver', payload, { delay: 30000 });
}

When the delayed job runs, it reads the final batch state and emits one concise notification instead of many repetitive ones.

07Real-time Delivery via Socket.io and Redis Pub/Sub

Online users should receive updates instantly, so workers publish delivery events to Redis channels and Socket.io servers forward those events to connected clients.

await redis.publish(`user:${userId}:notifications`, JSON.stringify(notification));

const sub = redis.duplicate();
sub.subscribe(`user:${userId}:notifications`);
sub.on('message', (channel, data) => {
  io.to(`user:${userId}`).emit('notification', JSON.parse(data));
});

This keeps workers independent from socket servers and scales cleanly across multiple server instances.

08What I'd Do Differently

• Add circuit breakers so a MongoDB outage doesn't let the queue grow without bounds.
• Track queue depth, dead-letter volume, and delivery latency in a metrics dashboard.
• Introduce priority queues for urgent alerts such as security notifications.