Sitemap, robots.txt and Open Graph Images

Titles and descriptions get a page indexed; sitemaps, crawl rules and social preview images decide how well it is discovered and shared. The App Router generates all three from code, so they stay in sync with your routes. After this lesson you will be able to generate sitemap.xml and robots.txt from data, produce Open Graph images per page, and set the base URL that ties them together.

File-Based Metadata Conventions

Placing a file with one of these names in a route segment turns it into metadata for that segment and its children:

| File | Produces | Static or dynamic | |---|---|---| | favicon.ico, icon.png, apple-icon.png | Icon <link> tags | Static file, or icon.tsx for generated | | opengraph-image.png or .tsx | og:image tags | Either | | twitter-image.png or .tsx | twitter:image tags | Either | | sitemap.xml or sitemap.ts | /sitemap.xml | Either | | robots.txt or robots.ts | /robots.txt | Either |

Static files are served as they are; the .ts/.tsx variants run at build time or on request.

Generating a Sitemap

Return an array of entries from app/sitemap.ts. The MetadataRoute.Sitemap type documents the accepted fields:

typescript

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const base = "https://example.com";
  const posts = await getAllPosts();

  const postEntries = posts.map((post) => ({
    url: `${base}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: "weekly" as const,
    priority: 0.7,
  }));

  return [
    { url: base, lastModified: new Date(), changeFrequency: "daily", priority: 1 },
    { url: `${base}/about`, changeFrequency: "monthly", priority: 0.5 },
    ...postEntries,
  ];
}

Sitemaps are limited to 50,000 URLs. For larger sites export a generateSitemaps function that returns [{ id: 0 }, { id: 1 }, ...]; Next.js then calls sitemap({ id }) for each and serves them at /sitemap/0.xml, /sitemap/1.xml and so on.

Crawl Rules With robots.ts

typescript

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow: ["/admin", "/api/"] },
      { userAgent: "GPTBot", disallow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Keep disallow for crawl budget, not security; a disallowed URL is still reachable. Pages that must stay out of search results should also send robots: { index: false } through the Metadata API.

Dynamic Open Graph Images

The sample at the top of this lesson generates a 1200 x 630 PNG per blog post from JSX. ImageResponse from next/og renders a subset of HTML and CSS (flexbox, absolute positioning, fonts, no grid) to an image. Guidelines:

Export size and contentType so Next.js can emit correct <meta> tags; an alt export adds og:image:alt.
Wrap content in a display: flex container; the renderer requires explicit flex layout.
Images are cached like any route: static when the data is, revalidated with the same rules otherwise.

The file is picked up automatically as og:image for the segment. A static opengraph-image.png in app/ provides a site-wide default.

Absolute URLs and metadataBase

Social platforms require absolute image URLs. Set metadataBase once in the root layout and Next.js resolves every relative metadata URL against it:

tsx

// app/layout.tsx
import type { Metadata } from "next";

export const metadata: Metadata = {
  metadataBase: new URL("https://example.com"),
  title: { default: "CodeYourCraft", template: "%s | CodeYourCraft" },
};

Common mistakes

Forgetting metadataBase, which produces relative og:image URLs that previews cannot load.
Using CSS grid in ImageResponse JSX; only the supported flexbox subset renders.

Key Takeaways

Metadata files (sitemap.ts, robots.ts, opengraph-image.tsx, icons) are generated from code in app/.
sitemap.ts returns typed entries; use generateSitemaps beyond 50,000 URLs.
robots.ts controls crawling but not access; use index: false metadata for pages that must not appear.
ImageResponse renders flex-based JSX to a PNG per route, with size and contentType exports.
metadataBase makes every metadata URL absolute, which social previews demand.

Next lesson: Environment Variables and next.config — configure secrets, public variables and the framework itself.