Next.js robots.txt

robots.txt is the oldest, plainest piece of SEO infrastructure a site can have — a text file at a well-known path telling crawlers what they're allowed to touch. Next.js gives it the same two-tier treatment as every other metadata file convention: write it as a literal static file for the common case, or generate it programmatically when your rules need to vary by environment, by data, or by anything else code can express that a static file can't.

Static robots.txt

User-Agent: *
Allow: /
Disallow: /private/

Sitemap: https://acme.com/sitemap.xml

Placed at the root of app, this is served exactly as written, following the Robots Exclusion Standard — no processing, no transformation, just a static file at the conventional /robots.txt path.

Generating robots.txt with Code

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: "/private/",
    },
    sitemap: "https://acme.com/sitemap.xml",
  };
}

This produces output identical to the static version above. The value of the generated form shows up the moment your rules need to reflect something dynamic — an environment variable gating whether staging environments should be fully disallowed, or a disallow list built from a database of paths rather than hand-maintained in a text file. Structurally, robots.js is a special Route Handler, cached by default unless it reaches for a Request-time API or explicit dynamic config — the same caching contract every metadata-file convention in this family shares.

Targeting Specific Crawlers

Passing an array of rule objects (instead of a single object) to rules lets you give different crawlers genuinely different instructions:

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "Googlebot",
        allow: ["/"],
        disallow: "/private/",
      },
      {
        userAgent: ["Applebot", "Bingbot"],
        disallow: ["/"],
      },
    ],
    sitemap: "https://acme.com/sitemap.xml",
  };
}

User-Agent: Googlebot
Allow: /
Disallow: /private/

User-Agent: Applebot
Disallow: /

User-Agent: Bingbot
Disallow: /

Sitemap: https://acme.com/sitemap.xml

Note that userAgent accepts either a single string or an array — an array applies the same rule set to multiple crawlers at once (as with Applebot and Bingbot sharing one blanket disallow above), rather than requiring a separate rule object per crawler when the rules are identical.

Non-Standard Directives — a Recent, Genuinely Useful Addition

Some crawlers (Seznam's Request-Rate, Yandex's Clean-param) support directives outside the official Robots Exclusion Standard. As of a fairly recent version, Next.js's robots.js supports these via an other field on any rule:

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/" },
      {
        userAgent: "SeznamBot",
        allow: "/",
        other: {
          "Request-Rate": "10/1m",
        },
      },
    ],
  };
}

User-Agent: *
Allow: /

User-Agent: SeznamBot
Allow: /
Request-Rate: 10/1m

Two behaviors worth being precise about: keys in other preserve their exact casing as you write them (Next.js doesn't normalize or reformat directive names), and array values emit one line per entry, each scoped correctly within that rule's own User-Agent block rather than bleeding into a different crawler's section.

Next.js does not validate any of this. Values in other pass through completely verbatim — there's no checking that Request-Rate is spelled correctly, formatted the way Seznam actually expects, or even a directive that any real crawler recognizes. If you're using this field, the burden of correctness is entirely on you; consult the specific search engine's own documentation for the exact syntax it expects, since Next.js won't catch a typo or malformed value here the way it would catch a TypeScript type error elsewhere.

The Robots Object Shape

type Robots = {
  rules:
    | {
        userAgent?: string | string[];
        allow?: string | string[];
        disallow?: string | string[];
        crawlDelay?: number;
        other?: Record<string, string | number | Array<string | number>>;
      }
    | Array<{
        userAgent: string | string[];
        allow?: string | string[];
        disallow?: string | string[];
        crawlDelay?: number;
        other?: Record<string, string | number | Array<string | number>>;
      }>;
  sitemap?: string | string[];
  host?: string;
};

Two details buried in that type worth calling out: sitemap accepts either a single string or an array, so a site with multiple sitemap files (perhaps split via generateSitemaps for a large product catalog) can list all of them here rather than being limited to one. And crawlDelay, while present in the type, is worth checking against your target crawler's actual support — not every major search engine respects it, even though the field exists and Next.js will happily emit it.

Version History

Version	Changes
v16.3.0	Added the other field for non-standard per-agent directives
v13.3.0	robots introduced

Key Takeaways

Aspect	Detail
Static form	app/robots.txt, served exactly as written
Dynamic form	app/robots.ts/.js, default-exporting a function returning a Robots object
Multiple crawlers, different rules	Pass an array to rules, one entry per crawler or crawler group
Non-standard directives	The other field — passed through verbatim, with zero validation
sitemap field	Accepts a string or an array — list multiple sitemap files if you have them
Caching	Cached by default, like every other metadata-file Route Handler

robots.txt is about as low-ceremony as web infrastructure gets, and Next.js's file convention doesn't try to make it more complicated than it needs to be. Reach for the static file until you have an actual reason — environment-specific rules, a dynamically generated disallow list, non-standard directives for a specific crawler — to generate it with code instead.