The Aggregation Pipeline

When find() is not enough, and you need grouping, computed fields, joins, or multi-step transformations, you reach for the aggregation pipeline. A pipeline is an array of stages, and documents flow through the stages in order, each stage reshaping or filtering the stream. This CodeYourCraft lesson builds up a real analytics pipeline stage by stage.

The Mental Model

Think of a factory conveyor belt. Documents enter from the collection, each stage transforms them, and the output of one stage becomes the input of the next:

javascript

db.orders.aggregate([
  { $match: { ... } },   // filter early
  { $group: { ... } },   // summarize
  { $sort:  { ... } }    // order the results
])

$match and $project

$match filters documents exactly like a find() query, and should come as early as possible so later stages process fewer documents (and so indexes can be used). $project reshapes each document, keeping, dropping, or computing fields:

javascript

use shop
db.orders.aggregate([
  { $match: { status: 'completed', createdAt: { $gte: new Date('2026-01-01') } } },
  { $project: {
      customerId: 1,
      total: 1,
      month: { $month: '$createdAt' }
  } }
])

Inside expressions, a string starting with $ like '$createdAt' refers to a field of the incoming document.

$group: The Heart of Aggregation

$group collapses many documents into one per key, using accumulator operators:

javascript

db.orders.aggregate([
  { $match: { status: 'completed' } },
  { $group: {
      _id: '$customerId',                 // group key
      orderCount: { $sum: 1 },
      totalSpent: { $sum: '$total' },
      avgOrder: { $avg: '$total' },
      lastOrder: { $max: '$createdAt' }
  } },
  { $sort: { totalSpent: -1 } },
  { $limit: 10 }                          // top 10 customers
])

Common accumulators include $sum, $avg, $min, $max, $first, $last, and $push, which collects values into an array.

$unwind: Working with Arrays

$unwind outputs one document per array element, which is how you aggregate over arrays like order line items:

javascript

db.orders.aggregate([
  { $unwind: '$items' },
  { $group: {
      _id: '$items.sku',
      unitsSold: { $sum: '$items.qty' },
      revenue: { $sum: { $multiply: ['$items.qty', '$items.price'] } }
  } },
  { $sort: { revenue: -1 } }
])

$lookup: Joins, MongoDB Style

$lookup performs a left outer join against another collection, attaching matches as an array field:

javascript

db.orders.aggregate([
  { $lookup: {
      from: 'customers',
      localField: 'customerId',
      foreignField: '_id',
      as: 'customer'
  } },
  { $unwind: '$customer' },
  { $project: { total: 1, 'customer.name': 1, 'customer.email': 1 } }
])

Use $lookup sparingly on hot paths; if you join constantly, that is often a signal to embed the data instead, a topic we cover in the schema design lessons.

Other Stages Worth Knowing

$addFields adds computed fields without dropping the rest; $count emits a single count document; $facet runs several sub-pipelines at once (perfect for search results plus facet counts); $out and $merge write pipeline results to a collection, enabling materialized views for dashboards.

Performance Notes

Put $match and $limit as early as possible, project away large fields you do not need, and remember that a $match at the front can use indexes while a $match after a $group cannot. Each stage has a 100 MB memory limit; pass allowDiskUse: true for big jobs, or better, aggregate less data.

Key Takeaways

A pipeline is an ordered array of stages; each stage transforms the document stream.
$match early, $group to summarize, $sort and $limit to rank.
$unwind explodes arrays; $lookup joins collections.
'$field' strings reference document fields inside expressions.
Filter early for index usage, and use $merge for materialized views.

Next we step back from queries and think about structure: how to design schemas that fit the document model.