Custom & Headless Guide

Optimize Custom Stores for AI

Technical developer checklist for custom React, Next.js, Vue, or Laravel backends to optimize indexable API feeds and metadata schema.

1 Dynamic Robots.txt Endpoint

If you are using Next.js (App Router), create a app/robots.ts file to programmatically expose rules allowing crawler bots access to your product dynamic routes:

import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: ['GPTBot', 'PerplexityBot', 'ClaudeBot'],
      allow: ['/products/', '/api/catalog/'],
    },
    sitemap: 'https://yourstore.com/sitemap.xml',
  }
}

2 Server-Side Rendered (SSR) Schema.org Tags

Ensure schema.org JSON-LD scripts are parsed on the initial HTML server response (rather than client-side React rendering) so bots don't miss them.

// Next.js page metadata setup
export async function generateMetadata({ params }) {
  const product = await fetchProduct(params.id);
  return {
    title: product.name,
    other: {
      'product-schema': JSON.stringify({
        '@context': 'https://schema.org',
        '@type': 'Product',
        'name': product.name,
        'sku': product.sku,
        'offers': {
          '@type': 'Offer',
          'price': product.price,
          'priceCurrency': 'USD',
          'availability': product.inStock ? 'https://schema.org/InStock' : 'https://schema.org/OutOfStock'
        }
      })
    }
  }
}

3 Expose llms.txt route

For custom backends, place the generated llms.txt file in your application public assets folder (e.g. /public/llms.txt) so it is served statically, or set up a simple path redirect handling it.

4 Connect via PointNXT Developer API

Expose webhook endpoints to listen to inventory events or push products programmatically using PointNXT's lightweight REST APIs.