Performance Optimization

The database profiler captures slow operations into the system.profile capped collection. It has three levels: 0 (off), 1 (log operations slower than slowms), and 2 (log all operations). Setting level 1 with a threshold of 100ms is a common production strategy. The profiler adds overhead, so level 2 should only be used temporarily during debugging. Querying system.profile reveals the worst offenders by namespace, millis, and operation type.

// Enable profiling level 1 for queries > 100ms
db.setProfilingLevel(1, { slowms: 100 })

// Check current profiling level
db.getProfilingStatus()

// Query the profile collection
db.system.profile.find({ millis: { $gt: 100 } }).sort({ ts: -1 }).limit(10)

// Check most expensive queries
db.system.profile.find().sort({ millis: -1 }).limit(5)

Why it matters: Production profiling is one of the most powerful tools for identifying bottlenecks; knowing how to enable and query profile data shows you understand real-world performance troubleshooting.

Real applications: In production systems you enable profiling level 1 with an appropriate slowms threshold, then regularly query system.profile to find and fix slow operations before they impact users.

Common mistakes: Forgetting to set a slowms threshold (leaving it at the default 100ms), enabling level 2 profiling in live production without capacity planning, or not disabling profiling after debugging.

Projection limits which fields MongoDB returns in query results, reducing both network transfer and memory usage. Instead of fetching entire documents, you specify 1 for fields to include or 0 to exclude. A covered query — where all fields in the query and projection exist in an index — never touches the actual documents, making it extremely fast. Always project only the fields your application needs, especially in high-throughput APIs. The _id field is included by default and must be explicitly excluded with _id: 0.

// Include only name and email (exclude everything else)
db.users.find({ active: true }, { name: 1, email: 1, _id: 0 })

// Covered query: index on { active: 1, name: 1, email: 1 }
// MongoDB never reads actual documents — fastest possible query

// Exclude sensitive fields
db.users.find({}, { password: 0, creditCard: 0 })

Why it matters: Projection directly reduces query latency, memory usage, and network bandwidth—critical for API performance at scale. It's a must-know optimization that interviewers expect candidates to consider.

Real applications: Mobile apps and slow-network clients benefit significantly from projection; returning only needed fields can reduce payload by 50-80%, dramatically improving load times and reducing data plan overage charges.

Common mistakes: Not excluding sensitive fields, projecting entire nested objects when only one property is needed, or forgetting that _id is included by default and must be explicitly excluded with _id: 0.

Connection pooling maintains a pool of reusable TCP connections to MongoDB, avoiding the overhead of establishing a new connection for every database operation. The MongoDB driver manages the pool automatically with a default size of 100 connections per MongoClient instance. The critical rule is to create one MongoClient per application and reuse it — creating a new client per request is a major anti-pattern that exhausts file descriptors and causes timeouts. Pool size should be tuned based on concurrent request load and MongoDB's maxIncomingConnections.

// WRONG — creates new connection per request
app.get('/users', async (req, res) => {
  const client = new MongoClient(uri); // anti-pattern!
  await client.connect();
  // ...
});

// CORRECT — reuse single client
const client = new MongoClient(uri, { maxPoolSize: 50 });
await client.connect();
const db = client.db('myapp');
app.get('/users', async (req, res) => {
  const users = await db.collection('users').find().toArray();
  res.json(users);
});

Why it matters: Connection pooling is fundamental to building scalable applications; mismanaging connections is one of the most common performance anti-patterns that interviewers watch for.

Real applications: High-traffic systems with thousands of concurrent users depend on connection pooling to serve requests efficiently. Without it, connection exhaustion becomes the limiting factor and requests timeout.

Common mistakes: Creating a new MongoClient per request (causing connection pool exhaustion), not waiting for client.connect() before using the database, or closing the client after each request instead of at application shutdown.

The hint() method forces MongoDB to use a specific index, overriding the query planner. This is useful when the query planner chooses a suboptimal plan due to outdated statistics or data distribution. You can specify the index by name string or by key pattern object. Use hint({ $natural: 1 }) to force a collection scan. Always verify with explain() that the forced index actually improves performance. Running db.collection.reIndex() or refreshing statistics with analyze commands may resolve planner issues without hardcoding hints.

// Force specific index by key pattern
db.orders.find({ userId: "u123", status: "pending" })
  .hint({ userId: 1, status: 1 })
  .explain("executionStats")

// Force index by index name
db.orders.find({ createdAt: { $gt: new Date("2024-01-01") } })
  .hint("createdAt_1")

// Force natural (collection) scan
db.orders.find({ status: "pending" }).hint({ $natural: 1 })

Why it matters: Query optimization sometimes requires understanding when to override the planner; hint() shows you comprehend how to debug planner decisions and verify fixes.

Real applications: During emergencies, you may use hint() as a temporary workaround when the query planner chooses a suboptimal plan due to stale collection statistics or unusual data distribution.

Common mistakes: Hardcoding hints without verify they improve performance using explain(), adding hints to permanent code without plans to remove them once statistics are refreshed, or using hint() when reIndex() or analyze() would solve the root cause.

The WiredTiger storage engine (default since MongoDB 3.2) uses an in-memory cache to store recently accessed documents and index pages. By default it uses the larger of 50% of (RAM minus 1GB) or 256MB. When frequently accessed data fits in cache, reads are served from memory without disk I/O, dramatically improving throughput. The cache hit ratio can be monitored via db.serverStatus().wiredTiger.cache. A low hit ratio means MongoDB is constantly reading from disk. Optimizing your working set size and indexes reduces pressure on the cache.

// Check WiredTiger cache stats
const status = db.serverStatus();
const cache = status.wiredTiger.cache;
console.log({
  bytesInCache: cache["bytes currently in the cache"],
  maxCache: cache["maximum bytes configured"],
  pagesReadFromDisk: cache["pages read into cache"],
  pagesEvicted: cache["pages evicted by application threads"]
});

// Configure cache size in mongod.conf
// storage:
//   wiredTiger:
//     engineConfig:
//       cacheSizeGB: 4

Why it matters: The WiredTiger cache is the heart of MongoDB's performance; understanding how it works shows deep knowledge of MongoDB internals and memory management.

Real applications: In production, you monitor cache hit rates and tune cache size to ensure your working set (frequently accessed data) stays in memory, achieving millisecond response times without disk I/O.

Common mistakes: Setting cache size too small (causing evictions of hot data), not monitoring cache hit ratios to detect when working set doesn't fit, or assuming more RAM always helps without understanding working set size.

Using skip() for pagination is problematic at large offsets because MongoDB must scan and discard all preceding documents to reach the desired page. A skip of 10,000 means MongoDB reads 10,000 documents just to throw them away. The cursor-based pagination pattern (keyset pagination) solves this by using the last seen document's indexed field as a boundary in the next query — this is O(1) regardless of depth. Always combine limit() with a proper index and use cursor pagination for performance-critical lists.

// BAD — skip degrades as offset increases
db.products.find({ category: "electronics" })
  .sort({ _id: 1 }).skip(10000).limit(20) // reads 10020 docs!

// GOOD — cursor-based pagination
// First page
const page1 = await db.products
  .find({ category: "electronics" })
  .sort({ _id: 1 }).limit(20).toArray();

// Next page — use last _id from previous page
const lastId = page1[page1.length - 1]._id;
const page2 = await db.products
  .find({ category: "electronics", _id: { $gt: lastId } })
  .sort({ _id: 1 }).limit(20).toArray();

Why it matters: Pagination patterns directly affect query performance at large offsets; knowing cursor-based pagination shows you understand scalability and real-world pagination problems.

Real applications: Large result sets (like search results, leaderboards, or feed items) require efficient pagination; skip-based pagination becomes unusably slow at page 1000, while cursor-based pagination stays O(1).

Common mistakes: Using skip() without limits, not providing an indexed sort field, implementing offset-based pagination without considering query performance degradation, or not storing the cursor position between requests.

MongoDB has a 16MB document size limit. Overly large documents degrade performance because reads return more data than needed and updates rewrite more data than necessary. Unbounded arrays (growing arrays like comments, logs, events) are a major anti-pattern — they cause document bloat, hit the 16MB limit, and make array queries slower as they grow. Use the bucket pattern or separate collections for time-series or log data. Frequently accessed subdocuments that are always read together should be embedded; rarely-accessed or independently-queried data should be referenced.

// Anti-pattern: unbounded array
{
  _id: "post123",
  title: "...",
  comments: [ /* grows forever — problematic */ ]
}

// Better: bucket pattern for high-volume data
{
  _id: ObjectId(),
  postId: "post123",
  bucket: 1,
  comments: [ /* max 100 per document */ ],
  count: 100
}

// Or separate collection with reference
{ _id: "c1", postId: "post123", text: "...", createdAt: ISODate() }

Why it matters: Schema design decisions directly impact query efficiency and replication throughput; understanding document size and embedding strategies is crucial for scalable systems.

Real applications: Social media posts, blog comments, and time-series data grow unbounded—using separate collections or the bucket pattern prevents hitting the 16MB limit and maintains predictable performance.

Common mistakes: Embedding unbounded arrays that grow over time (comments, logs), storing entire related objects when only IDs are needed, or not considering update frequency when choosing between embedding and referencing.

Read preferences control which replica set member MongoDB routes read operations to. The default is primary (always read from primary). Setting secondaryPreferred routes reads to secondaries when available, offloading the primary and allowing it to focus on writes. nearest routes to the member with the lowest network latency. Analytics and reporting queries are ideal candidates for secondary reads since they're heavy but don't need the absolute latest data. Be aware of replication lag: secondaries may serve slightly stale data.

// Node.js driver read preference
const client = new MongoClient(uri, {
  readPreference: 'secondaryPreferred'
});

// Per-query read preference
const db = client.db('myapp');
const results = await db.collection('analytics')
  .find({ date: { $gte: startDate } })
  .withReadPreference(ReadPreference.SECONDARY)
  .toArray();

// Mongoose
const Model = mongoose.model('Log', schema);
await Model.find().read('secondary').exec();

Why it matters: Read preferences enable scaling read-heavy applications on replica sets; understanding when and how to use them shows knowledge of distributed systems and load balancing.

Real applications: Analytics queries, reporting dashboards, and batch exports can route to secondaries, offloading the primary to focus on writes and user-facing reads that need the latest data.

Common mistakes: Using secondary reads for consistency-sensitive operations without accounting for replication lag, not ensuring secondaries have indexes matching the queries, or routing all reads to secondaries when the application needs strong consistency.

Write concern specifies the level of acknowledgment MongoDB requires before confirming a write. w: 0 (fire-and-forget) is fastest but risks data loss. w: 1 (default) waits for the primary to acknowledge. w: "majority" waits for a majority of replica set members, providing the strongest durability guarantee but the highest latency. The j: true option additionally waits for the write to be journaled to disk. Bulk logging or analytics writes can safely use lower write concerns while financial transactions should use w: "majority", j: true.

// Fire-and-forget (highest throughput, risks loss)
await db.collection('pageviews').insertOne(
  { page: '/home', ts: new Date() },
  { writeConcern: { w: 0 } }
);

// Majority write (strongest durability, higher latency)
await db.collection('payments').insertOne(
  { amount: 1000, userId: 'u1' },
  { writeConcern: { w: 'majority', j: true } }
);

// Balanced: primary ack + journaled
await db.collection('orders').insertOne(order,
  { writeConcern: { w: 1, j: true } }
);

Why it matters: Write concern settings determine the durability-vs-latency trade-off; choosing the right level for each operation type shows understanding of reliability and performance trade-offs.

Real applications: Financial systems require w:"majority",j:true for payments, while analytics can use w:0 for page views to maximize throughput without risking critical data loss.

Common mistakes: Using identical write concerns for all operations, not considering that lower write concerns risk data loss if the primary crashes immediately after acknowledgment, or misunderstanding that j:true adds significant latency.

bulkWrite() sends multiple write operations to MongoDB in a single network round-trip, dramatically reducing overhead compared to individual operations in a loop. It supports ordered (sequential, stops on error) and unordered (parallel, continues on error) modes. Unordered bulk writes are faster because MongoDB can optimize their execution order. Use bulkWrite for data imports, batch updates, upsert operations, and any scenario where you process arrays of documents. The method returns a BulkWriteResult with counts of matched, modified, inserted, and deleted documents.

const operations = users.map(user => ({
  updateOne: {
    filter: { _id: user.id },
    update: { $set: { lastSeen: new Date() } },
    upsert: true
  }
}));

const result = await db.collection('users').bulkWrite(
  operations,
  { ordered: false } // unordered = faster
);

console.log({
  matched: result.matchedCount,
  modified: result.modifiedCount,
  upserted: result.upsertedCount
});

Why it matters: Batch operations are essential for high-throughput migrations and updates; knowing bulkWrite() vs. loops shows you understand performance optimization fundamentals.

Real applications: Importing 1M records, syncing data from external systems, or updating millions of documents per hour—bulkWrite() makes the difference between seconds and hours.

Common mistakes: Writing slow loops instead of using bulkWrite(), using ordered:true when unordered would be faster, or not measuring performance before/after to verify the batch operation is actually faster.

The N+1 problem occurs when you fetch N documents and then execute one additional query per document. In MongoDB this commonly happens when using references instead of embedding. The solution is to batch lookups using $in queries or to use the aggregation pipeline's $lookup stage for joins. In Mongoose, populate() performs batched lookups under the hood. Never loop and query individual documents — collect all IDs first, then fetch them in one query. This reduces N+1 queries to exactly 2 queries.

// BAD — N+1 problem
const orders = await db.collection('orders').find().toArray();
for (const order of orders) {
  order.user = await db.collection('users').findOne({ _id: order.userId }); // N queries!
}

// GOOD — batch lookup with $in
const orders = await db.collection('orders').find().toArray();
const userIds = orders.map(o => o.userId);
const users = await db.collection('users').find({ _id: { $in: userIds } }).toArray();
const userMap = new Map(users.map(u => [u._id.toString(), u]));
orders.forEach(o => { o.user = userMap.get(o.userId.toString()); });

Why it matters: The N+1 problem is a classic performance footgun that appears in real-world systems; recognizing and fixing it demonstrates mature database optimization skills.

Real applications: When fetching 1000 orders with user details, N+1 means 1001 queries (one per order plus one for the collection); batching reduces this to exactly 2 queries regardless of order count.

Common mistakes: Querying inside loops without realizing the performance impact, not using $in or $lookup, or using populate() without understanding that it still makes individual queries if not used correctly in aggregation pipelines.

A capped collection is a fixed-size circular buffer that automatically overwrites the oldest documents when full. Inserts are extremely fast because they always append to the end and never trigger space allocation. They maintain natural insertion order, making them ideal for logs, caches, audit trails, and real-time event streams. Capped collections support tailable cursors that block and wait for new documents, similar to tail -f on a log file. Limitations include no deletions, no document growth (updates cannot increase document size), and no sharding.

// Create capped collection (100MB, max 1 million docs)
db.createCollection("appLogs", {
  capped: true,
  size: 100 * 1024 * 1024, // 100MB in bytes
  max: 1000000
})

// Tailable cursor for real-time streaming
const cursor = db.collection('appLogs').find({}, {
  tailable: true,
  awaitData: true
});
for await (const doc of cursor) {
  console.log('New log:', doc);
}

Why it matters: Capped collections offer unique performance characteristics for streaming and logging; knowing when to use them shows understanding of specialized MongoDB features.

Real applications: Application logs, sensor data streams, audit trails, and real-time event processing benefit from capped collections' circular buffer and tailable cursor support for live streaming.

Common mistakes: Using capped collections for general data that needs deletion, trying to update documents in capped collections (updates cannot increase size), or not considering that capped collections cannot be sharded.

$lookup performs a left outer join between two collections, similar to SQL joins. It is powerful but expensive — it executes a separate query per input document by default. For better performance, filter documents with $match before $lookup to minimize the number of joins, and ensure the foreign collection has an index on the join field. MongoDB 5.0+ supports $lookup with pipeline and $match/$expr for correlated subqueries. In very hot paths, denormalizing (embedding) the looked-up data during write is often faster than $lookup at read time.

// ALWAYS $match before $lookup to reduce input documents
db.orders.aggregate([
  { $match: { status: "shipped", createdAt: { $gte: ISODate("2024-01-01") } } },
  {
    $lookup: {
      from: "users",
      localField: "userId",
      foreignField: "_id",
      as: "user"
    }
  },
  { $unwind: "$user" },
  { $project: { "user.password": 0 } }
])

// Index on users._id (already exists as primary key)
// Create index on orders.userId for the match stage
db.orders.createIndex({ userId: 1, status: 1, createdAt: -1 })

Why it matters: $lookup is powerful but expensive; understanding its performance profile and optimization strategies is critical for complex aggregations handling large datasets.

Real applications: Correlating orders with user details, products, and shipping information in a single query requires careful use of $lookup with upstream $match stages to minimize sub-queries.

Common mistakes: Using $lookup on unfiltered collections (every input document triggers a lookup), forgetting to ensure indexes on foreign join fields, or using $lookup at the start of a pipeline without upstream filtering stages.

Production MongoDB monitoring combines db.serverStatus() for real-time metrics, db.stats() for collection-level sizes, and the profiler for slow queries. Key metrics include: opcounters (reads/writes per second), connections (current vs. available), memory (resident vs. virtual), and replication lag. Tools like MongoDB Atlas provide built-in dashboards, alerting, and Performance Advisor. For self-hosted clusters, Ops Manager, Prometheus with mongodb-exporter, and Grafana dashboards are standard. Set up alerts for slow queries (>100ms), connection exhaustion, and replication lag exceeding 10 seconds.

// Real-time stats
db.serverStatus().opcounters    // reads, writes, commands per second
db.serverStatus().connections   // current, available, totalCreated
db.serverStatus().mem           // resident, virtual MB
db.serverStatus().repl          // replication info

// Collection stats
db.orders.stats()               // size, storageSize, nindexes

// Current operations (find long-running ops)
db.currentOp({ active: true, secs_running: { $gt: 5 } })

// Kill a specific operation
db.killOp(opId)

Why it matters: Production visibility and alerting prevent performance degradation from going unnoticed; knowing how to monitor key metrics shows you understand 24/7 operational reliability.

Real applications: Production systems use dashboards tracking opcounters (throughput), connections (saturation), memory (resource usage), and replication lag (consistency) with alerts for anomalies.

Common mistakes: Not setting up monitoring until after a production outage, only looking at aggregate metrics without investigating slow operations, or ignoring early warning signs like rising cursor timeouts or connection pool saturation.