Cloaking, Working with Bots

How the bot detection and cloaking work?

Bots are detected by the following mechanism:

  1. Check if IP in the internal database
  2. Check if there are bot signatures in UserAgent from the internal database
  3. Check if there isn't UserAgent
  4. Find IP in the user list
  5. Check if there are signatures in UserAgent from the user list.

The option “Action for bots” is available at all campaigns. If you choose “Let them go to streams” you can filter bots later at the streams.

How to set cloaking on website using Keitaro TDS?

We can use another website as a safe page. The code snippet is written on PHP, so the server of the website must provide PHP support.

  1. Go to page “Campaign > Integration Assistant”.
  2. Choose “Other > Keitaro API”.
  3. Copy the code and paste to website template above the tag “<html>”.
  4. Change Action in the section “BOT PROTECTION” to “Show text”, and field “Text” empty.
  5. Use any browser extension that changes the current UserAgent to see the safe page as a bot.

How bots can be detected?

1. By signature UserAgent. A visitor is checked by internal and user database of signatures.

2. By bot's IP database. A visitor is checked by internal and user base of IPs.

3. By empty UserAgent. All devices without UserAgent are bots.

Mechanisms of determining bots are described at page Settings.

How to update the database of bots?

At the page Maintenance > Geo DBs.

How to extend the list of bots IP?

At the page Settings open tab “Bots”. There are you can edit the additional list. The list may contain a single IP and CIDR.

78.12.31.1
92.12.1.0/24

Attention! Masks and the intervals are not supported!

What is the signature of bots?

It is a part of UserAgent. For example, for Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) the signature Googlebot.

How to add signature of bots?

Open the page Settings > Bots. You can set the signatures separated by commas:

signature1, signature2, ...

Empty UserAgent as a bot characteristic

Consider everyone whose UserAgent is empty as a bot.

How can anti-bot action be set?

Choose the necessary anti-bot action in campaign settings.

  • Show 404 Not Found - displays an empty page and “404 Not Found” status;
  • Redirect to URL - redirecting bots to a certian page;
  • Display text - displays a certain text to the bot;
  • Skip - the bots proceed to streams the same way as regular users.

How to catch bots by provider?

Database of providers are not supplied with Keitaro TDS because it is a serious violation of the Maxmind license (maintainer of GeoISP database).

You can buy license or find another way to get it. Enter the license key at the page Settings > Integration. Here is an instructions about manual connection of the file GeoISP.dat — manual_installation_and_update_of_maxmind_geo_isp.

After connecting the database you can use the filter “Provider” at the streams.