Sitemaps and Robots.txt

Two of the fastest SEO wins: a sitemap that search engines can crawl, and a robots.txt that tells them what to ignore. SEO Manager handles both per site.

XML sitemaps

Open SEO Manager on the site you are working on. The Sitemap Status panel shows when the sitemap was last built and where the file lives.

Automatic regeneration

Switch on Regenerate sitemap automatically and save. With it on:

The platform rebuilds the sitemap on a schedule.
The nightly job runs at 04:00 and covers the central site and every tenant.
The job is queued with no overlap and runs in the background, so it never stacks up.

Leave it off and the sitemap is only rebuilt when you trigger it manually.

Manual regeneration

From the platform server you (or your host) can run:

# Regenerate the central sitemap
php artisan seo:sitemap

# Regenerate the central sitemap and every tenant site
php artisan seo:sitemap --all-tenants

# Ignore the "auto-regeneration" setting and rebuild anyway
php artisan seo:sitemap --force

# List what would be built, without crawling
php artisan seo:sitemap --dry-run

Where the sitemap is written

Site	File
Central (landlord)	`sitemap/landlord/<site-title-slug>.xml`
Each tenant	`sitemap/tenants/<tenant-key>.<central-domain>.xml`

The file name is stable, so the URL does not change between runs.

Search-engine pinging

SEO Manager compares the new sitemap to the previous one. Only when the content actually changed does it ping Google and Bing, so you do not waste pings on unchanged sites. The sitemap is exposed at your site URL plus the file path, for example:

https://your-domain.com/sitemap/landlord/your-site.xml

Submit that URL in Google Search Console and Bing Webmaster Tools.

Tip

After the first generation, copy the exact path shown in the Sitemap Status panel — the file name follows your site title, so it is easy to guess wrong.

Robots.txt

Open SEO Manager, scroll to the Robots.txt panel. Whatever you save here is served as-is at:

https://your-site.com/robots.txt

Each site (central and every tenant) has its own robots.txt. The link View live robots.txt opens the current file.

Presets

Two one-click presets fill the editor for you:

Preset: allow all — the safe default:
```
User-agent: *
Allow: /
```

Preset: block AI bots — keeps AI crawlers out while allowing search engines:

User-agent: GPTBot
User-agent: CCBot
User-agent: ClaudeBot
User-agent: anthropic-ai
Disallow: /

Presets only fill the textarea — review the content and click Save Changes to apply it.

When robots.txt is empty

If you save an empty robots.txt, SEO Manager serves the default allow-all file instead of nothing, so your site is never accidentally blocked.

Warning

Do not Disallow: / on a live site unless you intend to remove it from search entirely. If you block a URL here, search engines stop crawling it within days.

Notes on multi-site behavior

On the central site, sitemap and robots settings are stored as central options.
On a tenant site, the same settings are stored per tenant.
The seo:sitemap --all-tenants command loops tenants one at a time and initialises tenancy for each, so a single tenant failing to build does not stop the rest. Failed tenants are logged and skipped.

Still stuck?

Our support team is ready to help you get set up.

Get support