AI crawler blocking and broken links

AI crawler blocking and broken links

AI crawler blocks

The plugin ships a curated list of AI crawler user agents in OSeo_Robots::AI_CRAWLERS:

• GPTBot - OpenAI GPTBot
• ChatGPT-User - OpenAI ChatGPT browse
• OAI-SearchBot - OpenAI SearchBot
• anthropic-ai - Anthropic AI
• ClaudeBot - Anthropic ClaudeBot
• PerplexityBot - Perplexity AI
• Bytespider - ByteDance / TikTok
• CCBot - Common Crawl
• omgili - Webz.io
• Diffbot - Diffbot
• ImagesiftBot - Imagesift
• Applebot-Extended - Apple AI (extended)

Under robots.txt > AI crawlers, tick the ones you want to block. The selection is stored in oseo_blocked_ai_crawlers and produces this in robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Note: blocking is voluntary on the crawler's side. Reputable bots honor it; malicious ones do not.

Broken link scanner

OSeo_Broken_Links::schedule_full_scan() queues a sweep of every public post's content for outbound links. The cron event oseo_broken_links_batch runs in batches via run_cron_batch() and checks each URL with wp_remote_head (timeout configurable). Responses are recorded with their status code.

Scanner status

php$status = OSeo_Broken_Links::scan_status();
// Returns: total, scanned, broken, queued, last_run

Re-checking a single link

phpOSeo_Broken_Links::run_recheck( $post_id, $url, $anchor );

Use this when you have fixed a broken URL and want to confirm before marking it resolved.

Trigger a scan manually

Click Scan now on the Broken Links screen (calls ajax_trigger_scan). Subsequent passes run on cron until the queue empties.