Re: Dealing with bot traffic - what tools/services are you using?

Hi Shannon,

This is a massive problem across the memory institution sector. There is a group that meets fairly regularly and has hosted "solution showcase" webinars about different approaches (Anubis, WAF/bot detection (F5, Cloudflare), Turnstile, etc).    Arran Griffith (Lyrasis, Fedora Program Manager) generously leads and facilitates that group. It can be a shortcut to answers and is absolutely a community.

More here (and be sure to look at resources and links): https://wiki.lyrasis.org/display/cmtygp/Aggressive+AI+Harvesting+of+Digital+Resources

Also, I as noted, C4L has a dedicated slack channel on this topic.  See wiki page above for link.

We first suffered almost two years ago, and about 18 months ago it got so bad our comms colleague wrote it up as a story:
https://library.unc.edu/news/library-it-vs-the-ai-bots/

And we wrote it up in a more formal way here:
https://journal.code4lib.org/articles/18489


Best, 
Tim

-----Original Message-----
From: Code for Libraries <CODE4LIB_at_LISTS.CLIR.ORG> On Behalf Of Lucky, Shannon
Sent: Thursday, April 30, 2026 2:36 PM
To: CODE4LIB_at_LISTS.CLIR.ORG
Subject: [CODE4LIB] Dealing with bot traffic - what tools/services are you using?

[Some people who received this message don't often get email from shannon.lucky@usask.ca. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi all,

I am curious what methods folks are using to deal with aggressive AI harvesting on websites - particularly digital project sites. Many of our servers are being hammered with traffic that impacts our service delivery and the methods we have been using cannot keep up.

Specifically I am wondering who is using services like Cloudflare or implementing OS solutions like Anubis, or are you using something else? I'm gathering information about what services or methods are being using at academic libraries hosting DH/digital projects so we can look at investing in some kind of service or process solution.

What are you using? Are you happy with it? What kinds of costs are associated?)


Shannon Lucky, MLIS MA

she/her

Associate Librarian


University of Saskatchewan

University Library

Ph: 306-966-2740

ORCID 0000-0001-9134-8560<https://orcid.org/0000-0001-9134-8560>


I acknowledge that I live and work on Treaty 6 Territory and the Homeland of the Métis. We pay our respect to the First Nations and Métis ancestors of this place and reaffirm our relationship with one another.