# robots.txt - sommer-co.com # Updated: 2026-10-08 # # STRUCTURE NOTE - read before editing: # Under RFC 9309 a crawler obeys ONE group only: its most specific user-agent # match. Groups are never merged. Every crawler we care about is therefore # stacked into the SINGLE group below, so all of them inherit every rule. # # Do NOT split any bot into its own "User-agent: X / Allow: /" block. That does # not grant access (absence of a Disallow already means allowed). It only # exempts that bot from every rule in this file. # To add a new bot: add one User-agent line to the stack below. Nothing else. Sitemap: https://www.sommer-co.com/sitemap_index.xml # --- Default --- User-agent: * # --- Major search crawlers --- User-agent: Googlebot User-agent: Bingbot # --- AI crawlers (all permitted) --- User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: Google-Extended User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: anthropic-ai User-agent: CCBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Bytespider User-agent: meta-externalagent # WordPress admin and login Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /xmlrpc.php # Comments, feeds, trackbacks Disallow: /trackback/ Disallow: */trackback/ Disallow: /feed/ Disallow: */feed/ Disallow: *?replytocom= # Internal site search Disallow: /search/ Disallow: /?s= Disallow: *?s= Disallow: /?search= Disallow: *?search= # Unpublished blog content Disallow: /blog/drafts/ # Not blocked on purpose. Google can crawl these today. Uncomment only after # checking that no URL in the pattern should rank or be cited: # Disallow: /tag/ # Disallow: /archive/ # Disallow: *.pdf$