Package isbot

zgo.at/isbot Index | Files | Directories

package isbot

import "zgo.at/isbot"

Package isbot attempts to detect HTTP bots.

A "bot" is defined as any request that isn't a regular browser request initiated by the user. This includes things like web crawlers, but also stuff like "preview" renderers and the like.

func Bot(r *http.Request) Result
func IPRange(addr string) Result
func UserAgent(ua string) Result
func (r Result) String() string

Package Files ¶

ip.go ip_ranges.go isbot.go user_agent.go

Constants ¶

const (
    NoBotKnown   = 0 // Known to not be a bot.
    NoBotNoMatch = 1 // None of the rules matches, so probably not a bot.
)

Not bots.

const (
    BotPrefetch      = 2 // Prefetch algorithm
    BotLink          = 3 // User-Agent contained an URL.
    BotClientLibrary = 4 // Known client library.
    BotKnownBot      = 5 // Known bot.
    BotBoty          = 6 // User-Agent string looks "boty".
    BotShort         = 7 // User-Agent is short of strangely formatted.
)

Bots identified by User-Agent.

const (
    BotRangeAWS          = 8  // AWS cloud
    BotRangeDigitalOcean = 9  // Digital Ocean
    BotRangeServersCom   = 10 // servers.com
    BotRangeGoogleCloud  = 11 // Google Cloud
    BotRangeHetzner      = 12 // hetzner.de
)

Bots identified by IP.

const (
    BotJSPhanton   = 150 // Phantom headless browser.
    BotJSNightmare = 151 // Nightmare headless browser.
    BotJSSelenium  = 152 // Selenium headless browser.
    BotJSWebDriver = 153 // Generic WebDriver-based headless browser.
)

These are never set by isbot, but can be used to send signals from JS; for example:

var is_bot = function() {
    var w = window, d = document
    if (w.callPhantom || w._phantom || w.phantom)
        return 150
    if (w.__nightmare)
        return 151
    if (d.__selenium_unwrapped || d.__webdriver_evaluate || d.__driver_evaluate)
        return 152
    if (navigator.webdriver)
        return 153
    return 0
}

func Is ¶

func Is(r Result) bool

Is this constant a bot?

func IsNot ¶

func IsNot(r Result) bool

IsNot is the inverse of Is().

func IsUserAgent ¶

func IsUserAgent(r Result) bool

IsUserAgent reports if this is considered a bot because of the User-Agent header.

func Prefetch ¶

func Prefetch(h http.Header) bool

Prefetch checks if this request is a browser "pre-fetch" request.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Link_prefetching_FAQ

type Result ¶

type Result uint8

func Bot ¶

func Bot(r *http.Request) Result

Bot checks if this HTTP request looks like a bot.

It returns one of the constants as the reason we think this is a bot.

This assumes that r.RemoteAddr is set to the real IP and does not check X-Forwarded-For or X-Real-IP.

Note that both 0 and 1 may indicate that it's *not* a bot; use Is() and IsNot() to check.

func IPRange ¶

func IPRange(addr string) Result

IPRange checks if this IP address is from a range that should normally never send browser requests, such as AWS and other cloud providers.

func UserAgent ¶

func UserAgent(ua string) Result

UserAgent checks if this User-Agent header looks like a bot.