Sitemap and robots.txt

Sitemap and robots.txt

Sitemap

OSeo_Sitemap::generate() produces a single XML file served at /sitemap.xml. The plugin registers a rewrite rule and query var so WordPress routes the path directly into OSeo_Sitemap::maybe_serve().

Key methods:

phpOSeo_Sitemap::get_url();      // canonical sitemap URL
OSeo_Sitemap::url_count();    // count of indexable URLs
OSeo_Sitemap::bust_cache();   // invalidate the generated XML cache
OSeo_Sitemap::flush_rewrite_rules_on_activate();

The sitemap pulls posts, pages, and public custom post types. It honors noindex on individual posts. Generation results are cached.

robots.txt

OSeo_Robots hooks the robots_txt filter at priority 99. The effective output is:

1. The custom robots content if set under robots.txt, or the WP default otherwise.
2. Plus AI-crawler rules built from the oseo_blocked_ai_crawlers option.

Direct file note: WordPress only filters the virtual robots.txt. If a static robots.txt file sits in the web root, the filter never runs. Delete the static file or the plugin's edits will not appear.

Built-in WP default

When you reset, the plugin reverts to the WordPress default, which respects the Discourage search engines setting under Settings > Reading:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://your-site/sitemap.xml

If the site is set to private, the default flips to Disallow: /.