Winbuzzer<p>Cloudflare Overhauls Web’s AI Rulebook with New Robots.txt ‘Content Signals’</p><p><a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Cloudflare</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/DataScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataScraping</span></a> <a href="https://mastodon.social/tags/Publishing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Publishing</span></a> <a href="https://mastodon.social/tags/GenerativeAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GenerativeAI</span></a></p><p><a href="https://winbuzzer.com/2025/10/06/cloudflare-overhauls-webs-ai-rulebook-with-new-robots-txt-content-signals-xcxwbn" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">winbuzzer.com/2025/10/06/cloud</span><span class="invisible">flare-overhauls-webs-ai-rulebook-with-new-robots-txt-content-signals-xcxwbn</span></a></p>
eupolicy.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
This Mastodon server is a friendly and respectful discussion space for people working in areas related to EU policy. When you request to create an account, please tell us something about you.
Administered by:
Server stats:
206active users
eupolicy.social: About · Profiles directory · Privacy policy
Mastodon: About · Get the app · Keyboard shortcuts · View source code · v4.4.3
#robotstxt
1 post · 1 participant · 0 posts today
NERDS.xyz – Real Tech News for Real Nerds<p><strong>Cloudflare launches Content Signals Policy to fight AI crawlers and scrapers</strong></p>
<p><a href="https://web.brid.gy/r/https://nerds.xyz/2025/09/cloudflare-content-signals-policy-ai-crawlers/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">web.brid.gy/r/https://nerds.xy</span><span class="invisible">z/2025/09/cloudflare-content-signals-policy-ai-crawlers/</span></a></p>
Twissell<p>Utilities such as <a href="http://serpstat.com/" rel="nofollow noopener" target="_blank">serpstat.com</a> see the URLs <a href="http://example.com/topic/42/mytitle/1" rel="nofollow noopener" target="_blank">example.com/topic/42/mytitle/1</a><br>
and <a href="http://example.com/topic/42/mytitle/3" rel="nofollow noopener" target="_blank">example.com/topic/42/mytitle/3</a> as two different pages with duplicate title meta tags.<br>
Is it possible to fix this without blocking all posts in the thread except the first one in robots.txt?<br>
How to get rid of this "ghosting duplication"<br>
Asking you guys here, because Google says title duplication is harmful for SEO.</p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Initiatives" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Initiatives</span></a> <br>AI’s free web scraping days may be over · Say hello to RSS’s younger, tougher brother <a href="https://ilo.im/166s9q" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/166s9q</span><span class="invisible"></span></a> </p><p>_____<br><a href="https://mastodon.social/tags/Web" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Web</span></a> <a href="https://mastodon.social/tags/Publishing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Publishing</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Blog" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Blog</span></a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Content</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/Payments" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Payments</span></a> <a href="https://mastodon.social/tags/RSL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RSL</span></a> <a href="https://mastodon.social/tags/RSS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RSS</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a></p>
eicker.news ᳇ tech news<p>A new <a href="https://eicker.news/tags/licensingstandard" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>licensingstandard</span></a>, <a href="https://eicker.news/tags/ReallySimpleLicensing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ReallySimpleLicensing</span></a> (<a href="https://eicker.news/tags/RSL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RSL</span></a>), aims to allow <a href="https://eicker.news/tags/webpublishers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>webpublishers</span></a> to set terms for <a href="https://eicker.news/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> companies using their content. Supported by major brands like Reddit and Yahoo, RSL builds upon the existing <a href="https://eicker.news/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a> protocol, enabling <a href="https://eicker.news/tags/publishers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>publishers</span></a> to specify <a href="https://eicker.news/tags/licensing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>licensing</span></a> and <a href="https://eicker.news/tags/royaltyterms" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>royaltyterms</span></a> for <a href="https://eicker.news/tags/AItraining" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AItraining</span></a> data. <a href="https://www.theverge.com/news/775072/rsl-standard-licensing-ai-publishing-reddit-yahoo-medium?eicker.news" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">theverge.com/news/775072/rsl-s</span><span class="invisible">tandard-licensing-ai-publishing-reddit-yahoo-medium?eicker.news</span></a> <a href="https://eicker.news/tags/tech" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>tech</span></a> <a href="https://eicker.news/tags/media" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>media</span></a> <a href="https://eicker.news/tags/news" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>news</span></a></p>
Hostvix<p>RSL is the missing layer for the AI era: set terms, get attribution, and get paid (per crawl or per inference). Open standard, collective leverage. If AI uses your work, it should respect your license. Time to take control.</p><p><a href="https://hostvix.com/rsl-a-new-standard-to-make-ai-pay-for-the-content-it-consumes/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">hostvix.com/rsl-a-new-standard</span><span class="invisible">-to-make-ai-pay-for-the-content-it-consumes/</span></a></p><p><a href="https://mastodon.social/tags/RSL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RSL</span></a> <a href="https://mastodon.social/tags/ReallySimpleLicensing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ReallySimpleLicensing</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/AIethics" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AIethics</span></a> <a href="https://mastodon.social/tags/AIsafety" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AIsafety</span></a> <a href="https://mastodon.social/tags/AIdata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AIdata</span></a> <a href="https://mastodon.social/tags/ContentRights" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ContentRights</span></a> <a href="https://mastodon.social/tags/Licensing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Licensing</span></a> <a href="https://mastodon.social/tags/OpenWeb" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenWeb</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Publishers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Publishers</span></a> <a href="https://mastodon.social/tags/Creators" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Creators</span></a> <a href="https://mastodon.social/tags/Attribution" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Attribution</span></a> <a href="https://mastodon.social/tags/PayPerCrawl" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PayPerCrawl</span></a> <a href="https://mastodon.social/tags/PayPerInference" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PayPerInference</span></a> <a href="https://mastodon.social/tags/RSS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RSS</span></a> <a href="https://mastodon.social/tags/WebStandards" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebStandards</span></a> <a href="https://mastodon.social/tags/DigitalRights" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DigitalRights</span></a> <a href="https://mastodon.social/tags/CollectiveLicensing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CollectiveLicensing</span></a> <a href="https://mastodon.social/tags/Fastly" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Fastly</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Reports" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Reports</span></a><br>The web has a new AI payment system · The RSL Standard sets rules for AI scraping fees <a href="https://ilo.im/166ryy" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/166ryy</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/Web" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Web</span></a> <a href="https://mastodon.social/tags/Publishing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Publishing</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Blog" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Blog</span></a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Content</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/Payments" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Payments</span></a> <a href="https://mastodon.social/tags/RSL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RSL</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/Findings" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Findings</span></a><br>AI bots and robots.txt · How websites use robots.txt to set AI crawling rules <a href="https://ilo.im/166la4" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/166la4</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bots</span></a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Content</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/UserAgents" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>UserAgents</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/Guides" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Guides</span></a><br>Who owns your content in the AI age? · When AI bots take your content without consent <a href="https://ilo.im/165tej" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/165tej</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/Consent" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Consent</span></a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Content</span></a> <a href="https://mastodon.social/tags/GitHub" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GitHub</span></a> <a href="https://mastodon.social/tags/AccessLogs" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AccessLogs</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Design" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Design</span></a> <a href="https://mastodon.social/tags/WebDesign" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDesign</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a></p>
Kevin Karhan :verified:<p><span class="h-card" translate="no"><a href="https://ohai.social/@lina" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>lina</span></a></span> np. </p><p>I think it's important to highlight such projects like <a href="https://cuiiliste.de" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">cuiiliste.de</span><span class="invisible"></span></a> and actually point people at it in the hopes that <span class="h-card" translate="no"><a href="https://mastodon.social/@ooni" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>ooni</span></a></span> and <span class="h-card" translate="no"><a href="https://mastodon.social/@citizenlab" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>citizenlab</span></a></span> add it to their <a href="https://github.com/citizenlab/test-lists/" rel="nofollow noopener" target="_blank">test lists...</a></p><ul><li>Personally I did add it to my <a href="https://github.com/greyhat-academy/lists.d/blob/cacf5b9fbd80affd34d760c50b0b7333def000bf/blocklists.list.tsv#L28" rel="nofollow noopener" target="_blank">blocklist lists</a> not as an endorsement [far from it!] but as a point of reference for <a href="https://infosec.space/tags/CUII" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CUII</span></a>... </li></ul><p>Maybe someday I (or someone else) can get around and make a <a href="https://infosec.space/tags/VirusTotal" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>VirusTotal</span></a> - style <a href="https://infosec.space/tags/URL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>URL</span></a> / <a href="https://infosec.space/tags/Domain" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Domain</span></a> checker that provides more than a mere <em>"is clean"</em> / <em>"is sus"</em> / <em>"is dangerous"</em> assessment but a more nuanced answers like:</p><ul><li><em>"This domain is blocked by CUII for alleged copyright infringement"</em> </li></ul><p>or</p><ul><li><em>"This domain belongs to <a href="https://infosec.space/tags/NSAbook" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>NSAbook</span></a>"</em> </li></ul><p>and allow for granular, informed decisions (i.e. allow accessing it as a client, but blocking the entire <a href="https://infosec.space/tags/ASN" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ASN</span></a> from accessing one's Servers and thus block non-consensual <a href="https://infosec.space/tags/Scrapers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Scrapers</span></a> that violate <a href="https://infosec.space/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTXT</span></a>.) </p><ul><li>I'm just shit at programming anything with a <a href="https://infosec.space/tags/GUI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GUI</span></a> but hey, I'm just a <a href="https://infosec.space/tags/Sysadmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Sysadmin</span></a> not a <a href="https://infosec.space/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Frontend</span></a> person...</li></ul>
Thom Zane<p>How do we get Crawl-delay into the RFC 9309 robots.txt standard? Big tech hides behind RFC 9309 as an excuse to ignore Crawl-delay because they want to crawl as fast as they want.</p><p><a href="https://daedal.io/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a> <a href="https://daedal.io/tags/sysadmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sysadmin</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Debates" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Debates</span></a><br>AI search engine fight · Cloudflare and Perplexity clash over crawling <a href="https://ilo.im/165wpr" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/165wpr</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/Perplexity" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Perplexity</span></a> <a href="https://mastodon.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Cloudflare</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/SearchEngine" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SearchEngine</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Christian Gudrian<p>“robots.txt” is like leaving your front door open with a “please don’t steal here” sign next to it. </p><p><a href="https://social.tchncs.de/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://social.tchncs.de/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a></p>
apfeltalk :verified:<p>Perplexity ignoriert robots.txt: Kontroverse um Daten-Scraping für KI-Training<br>Das Training großer Sprachmodelle beruht auf einer Vielzahl von Webdaten. Die Einhaltu<br><a href="https://www.apfeltalk.de/magazin/news/perplexity-ignoriert-robots-txt-kontroverse-um-daten-scraping-fuer-ki-training/" rel="nofollow noopener" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">apfeltalk.de/magazin/news/perp</span><span class="invisible">lexity-ignoriert-robots-txt-kontroverse-um-daten-scraping-fuer-ki-training/</span></a><br><a href="https://creators.social/tags/News" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>News</span></a> <a href="https://creators.social/tags/Apple" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Apple</span></a> <a href="https://creators.social/tags/Applebot" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Applebot</span></a> <a href="https://creators.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Cloudflare</span></a> <a href="https://creators.social/tags/Cybersecurity" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Cybersecurity</span></a> <a href="https://creators.social/tags/Datenanalyse" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Datenanalyse</span></a> <a href="https://creators.social/tags/Datensicherheit" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Datensicherheit</span></a> <a href="https://creators.social/tags/EthikInDerKI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>EthikInDerKI</span></a> <a href="https://creators.social/tags/KITraining" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>KITraining</span></a> <a href="https://creators.social/tags/KnstlicheIntelligenz" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>KnstlicheIntelligenz</span></a> <a href="https://creators.social/tags/OpenWeb" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenWeb</span></a> <a href="https://creators.social/tags/Perplexity" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Perplexity</span></a> <a href="https://creators.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a> <a href="https://creators.social/tags/Sprachmodell" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Sprachmodell</span></a> <a href="https://creators.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebScraping</span></a> <a href="https://creators.social/tags/WebseitenBetreiber" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebseitenBetreiber</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Reports" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Reports</span></a><br>Perplexity is using undeclared crawlers · The AI search engine tries to evade website no-crawl rules <a href="https://ilo.im/165vrc" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/165vrc</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/Perplexity" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Perplexity</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/AnswerEngine" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AnswerEngine</span></a> <a href="https://mastodon.social/tags/SearchEngine" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SearchEngine</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Jonathan Bailey<p>I asked ChatGPT about the recent copyright news. It rehashed my latest column and misconstrued the facts. But why was it on my site at all?</p><p><a href="https://www.plagiarismtoday.com/2025/07/23/chatgpt-ignores-robots-txt-rehashes-my-column/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">plagiarismtoday.com/2025/07/23</span><span class="invisible">/chatgpt-ignores-robots-txt-rehashes-my-column/</span></a></p><p><a href="https://mastodon.world/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.world/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ChatGPT</span></a> <a href="https://mastodon.world/tags/OpenAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenAI</span></a> <a href="https://mastodon.world/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/Trends" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Trends</span></a><br>Who’s crawling your site in 2025 · The most active and blocked bots and crawlers <a href="https://ilo.im/1652mx" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/1652mx</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bots</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/UserAgents" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>UserAgents</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Frontend</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Findings" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Findings</span></a><br>Most blocked SEO bots · Insights from ~140 million websites <a href="https://ilo.im/16439x" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/16439x</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bots</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Content</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Blog" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Blog</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Explorations" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Explorations</span></a><br>What would happen if I blocked big search? · Pros and cons of blocking major search engines <a href="https://ilo.im/163yb3" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/163yb3</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/SearchEngine" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SearchEngine</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Website" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Website</span></a> <a href="https://mastodon.social/tags/Blog" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Blog</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Frontend</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Tino Eberl<p><a href="https://mastodon.online/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a> nutzt Inhalte für das <a href="https://mastodon.online/tags/KI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>KI</span></a>-Training auch dann, wenn Urheber dem widersprechen. Das wurde nun offiziell bestätigt.</p><p>Laut Google <a href="https://mastodon.online/tags/Deepmind" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Deepmind</span></a> betrifft der Widerspruch nur bestimmte <a href="https://mastodon.online/tags/Konzernbereiche" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Konzernbereiche</span></a>. Wer seine Daten schützen will, muss die Seite komplett aus der <a href="https://mastodon.online/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a>-Suche entfernen. <a href="https://mastodon.online/tags/Verlage" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Verlage</span></a> und <a href="https://mastodon.online/tags/Webseitenbetreiber" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Webseitenbetreiber</span></a> sehen sich dadurch wirtschaftlich benachteiligt.</p><p><a href="https://www.golem.de/news/kuenstliche-intelligenz-google-trainiert-ki-auch-wenn-urheber-es-nicht-erlauben-2505-195928.html" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">golem.de/news/kuenstliche-inte</span><span class="invisible">lligenz-google-trainiert-ki-auch-wenn-urheber-es-nicht-erlauben-2505-195928.html</span></a></p><p><a href="https://mastodon.online/tags/Urheberrecht" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Urheberrecht</span></a> <a href="https://mastodon.online/tags/KITraining" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>KITraining</span></a> <a href="https://mastodon.online/tags/Gemini" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Gemini</span></a> <a href="https://mastodon.online/tags/Suchmaschinen" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Suchmaschinen</span></a> <a href="https://mastodon.online/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTXT</span></a> <a href="https://mastodon.online/tags/KITraining" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>KITraining</span></a> <a href="https://mastodon.online/tags/Kartellverfahren" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Kartellverfahren</span></a></p>
TrendingLive feeds
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.
Create accountLoginDrag & drop to upload