The New Reality of the Web: More Bots, Less Engagement

Traffic patterns on the web have changed dramatically in the last few years.  With the rise of generative AI and its insatiable appetite for data, harvesting bots and AI agents currently represent at least half of all web activity, with expected sharp increases.  For many information-rich websites, defending against harvesting bots has become a priority to maintain performance for human visitors.  Many information repositories are creating MCP gateways to control delivery of information to AI agents in addition to the optimization in place for human visitors.  Information seekers increasingly start with an AI service such as ChatGPT, Claude, Grok, Co-Pilot instead of search engines like Google and Bing.  These changes in web traffic and research workflows bring important implications for libraries.  How do we optimize our resources and services for this new reality?