Recent searches

No recent searches

Search options

Only available when logged in.
eupolicy.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
This Mastodon server is a friendly and respectful discussion space for people working in areas related to EU policy. When you request to create an account, please tell us something about you.

Administered by:

Server stats:

201
active users

eupolicy.social: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.3

#backend

12 posts9 participants1 post today
Inautilo

#Development #Findings
AI bots and robots.txt · How websites use robots.txt to set AI crawling rules ilo.im/166la4

_____
#AI #Bots #Content #Website #UserAgents #RobotsTxt #Business #SEO #WebDev #Backend

Paul Calvano · AI Bots and Robots.txtThere’s been a lot of discussion lately around AI crawlers and bots, which are used to train LLMs and/or fetch content on behalf of their users. In the past few weeks I’ve seen blog posts about the amount of traffic from these crawlers, techniques and products to control how and what they can crawl, reports of misbehaving crawlers and more. Ironically, there’s even AI based services to mitigate AI crawler bots! Given how much interest there is, I thought I’d try and explore some HTTP Archive data to see how sites are using robots.txt to state their preferences on AI crawling.
Nikolay Robotoev 🥞🎙️

Here’s a bizarre observation that goes along with the “people are strange” argument, where by people I mean one person, and that person is me.
And the thing is that I like backend development. Which includes designing and improving and refactoring the architecture.

I did frontend development in the beginning of my career. Preparing resources, renaming, cropping and adding logic in a scripting language like Lua. Some people call this position _scripters_.
#programming #gamedev #backend

TrendingLive feeds
About