How to monitor and calibrate Bot Manager

After deploying Bot Manager, the next step is to understand what the data tells you and adjust the configuration to match your application’s traffic patterns. This guide walks you through reading logs, interpreting classifications, and calibrating thresholds and rules to reduce false positives while blocking real threats.

Prerequisites

Step 1: Enable full logging

Before you can calibrate Bot Manager, you need complete visibility into every request it evaluates. By default, Bot Manager only writes logs for requests that exceed the threshold. To observe all traffic—including legitimate requests—set the action to allow and enable internal logs during the observation phase.

In your Firewall, go to Functions Instances, select your Bot Manager instance, and update the JSON Args:

{
  "action": "allow",
  "internal_logs": 2,
  "log_tag": "bot_manager_calibration",
  "log_headers": [
    "accept",
    "accept-encoding",
    "accept-language",
    "content-type",
    "host",
    "referer",
    "user-agent",
    "x-forwarded-for",
    "x-request-id"
  ]
}

Step 2: Read logs in Real-Time Events

Real-Time Events is the fastest way to inspect individual Bot Manager decisions.

1. Access Azion Console > Real-Time Events.
2. Select the Functions data source.
3. Set the Time Filter to the period you want to analyze.
4. In the search bar, filter by your log tag:

log_tag = "bot_manager_calibration"

Each log entry is a JSON object. The key fields to focus on during calibration are:

Example log entry

{
  "action": "allow",
  "classified": "bad bot",
  "bot_category": "Bad Bot Signatures",
  "score": 14,
  "matched_rules": [10, 22],
  "azion_fingerprint": "5f852499202f7fa889157ce8b39a613b",
  "remote_addr": "203.0.113.42",
  "http_user_agent": "python-requests/2.28.0",
  "request_uri": "/api/v1/products",
  "geoip_country": "United States",
  "log_tag": "bot_manager_calibration"
}

Reading this log: A request from IP 203.0.113.42 using a Python HTTP library was scored 14 and classified as bad bot with category Bad Bot Signatures. Rules 10 and 22 fired. The action was allow because you’re in observation mode—once you set a threshold below 14, this request would be blocked.

Step 3: Interpret traffic classifications

Bot Manager assigns every request one of four classifications:

Understanding “under evaluation”

When a new fingerprint appears, Bot Manager needs up to 15 minutes to consolidate enough data to classify it confidently. During this window, requests are marked under evaluation. This is expected behavior—it prevents premature blocking of legitimate new visitors. If you see a high volume of under evaluation traffic, it may indicate:

• A large influx of new users (expected after a campaign or launch).
• Rotating IPs or User-Agents (a bot evasion technique worth investigating).

Good bot categories to recognize

If you see legitimate good bots being scored high, check whether their User-Agents are correctly recognized. You may need to add them to an exclusion rule.

Step 4: Analyze patterns with GraphQL

For aggregate analysis across large volumes of traffic, use the GraphQL API to query the botManagerMetrics dataset.

Query: traffic breakdown by classification and category

Access the GraphiQL Playground at https://api.azion.com/v4/metrics/graphql and run:

query {
  botManagerMetrics(
    filter: {
      tsRange: {
        begin: "2024-10-01T00:00:00"
        end: "2024-10-03T23:59:59"
      }
    }
    aggregate: {
      sum: requests
    }
    orderBy: [sum_DESC]
    groupBy: [classified, botCategory, action]
    limit: 10000
  ) {
    classified
    botCategory
    action
    sum
  }
}

This query returns the total request count grouped by classification, category, and action—giving you a clear picture of your traffic composition.

Query: top URLs impacted by bad bots

Use the botManagerBreakdownMetrics dataset to identify which endpoints are most targeted:

query {
  botManagerBreakdownMetrics(
    filter: {
      tsRange: {
        begin: "2024-10-01T00:00:00"
        end: "2024-10-03T23:59:59"
      }
    }
    aggregate: {
      sum: botRequests
    }
    groupBy: [requestUrl]
    orderBy: [sum_DESC]
    limit: 10
  ) {
    requestUrl
    sum
  }
}

If your login endpoint (/api/auth/login) or checkout path (/checkout) appears at the top, those are high-priority targets for stricter rules.

Step 5: Monitor with Real-Time Metrics dashboards

Real-Time Metrics provides pre-built dashboards for Bot Manager with two sections:

Overview section

Breakdown section

Step 6: Set and calibrate the threshold

The threshold parameter is the most critical calibration lever. It defines the maximum score a request can reach before Bot Manager executes the configured action.

How scoring works

Each static rule that fires adds points to the request score. The total score determines whether the request is blocked. A score of 0 means no rules matched; higher scores indicate more suspicious signals.

Choosing an initial threshold

Start with a conservative (high) threshold and lower it gradually as you gain confidence in the data:

Calibration workflow

1. Observe (days 1–3): Run with action: allow and internal_logs: 2. Collect score distributions.
2. Identify the score gap: Find the score range where bad bots cluster vs. where legitimate traffic sits. The threshold should sit between these two clusters.
3. Set threshold: Update JSON Args with your chosen threshold and a blocking action.
4. Monitor false positives: Watch for legitimate traffic being blocked. Check classified: legitimate entries with scores near your threshold.
5. Adjust: Lower the threshold if too many bad bots pass; raise it if legitimate users are blocked.

Example: setting the threshold after observation

After 48 hours of observation, your GraphQL data shows:

• Legitimate traffic: scores between 0–8
• Bad bots: scores between 15–30
• A small cluster of ambiguous traffic: scores 9–14

Set the threshold to 15 to block confirmed bad bots while allowing the ambiguous cluster to pass for further observation:

{
  "threshold": 15,
  "action": "deny",
  "log_tag": "bot_manager_production",
  "mode": "web",
  "dynamic_rules_tolerance": "soft",
  "reputation_network_lists": [],
  "disabled_static_rules": [],
  "log_headers": [
    "accept",
    "accept-encoding",
    "accept-language",
    "content-type",
    "host",
    "referer",
    "user-agent",
    "x-forwarded-for",
    "x-request-id"
  ]
}

Step 7: Tune static rules

Static rules are predefined detection patterns. Each rule has an ID and contributes a fixed number of points to the request score when matched. You can disable specific rules that generate false positives without removing them from the detection engine.

Identifying problematic rules

In Real-Time Events, filter for classified: legitimate entries that have a high score. Check the matched_rules array to identify which rule IDs are firing on legitimate traffic.

For example, if rule 22 consistently fires on your internal monitoring tool’s requests:

{
  "disabled_static_rules": [22]
}

Disabled rules still appear in disabled_matched_rules in the logs, so you can track their matches without affecting the score.

Step 8: Configure dynamic rules

Dynamic rules use behavioral analysis to detect anomalies that static rules may miss. They adapt to your application’s traffic baseline over time.

Dynamic rules parameters

Choosing tolerance

Enabling dynamic rules debug logs

During calibration, enable debug logs to understand dynamic rule decisions:

{
  "dynamic_rules_logs_enabled": true
}

Step 9: Exclude static assets from bot analysis

Bot Manager should not analyze requests for static files (images, CSS, JavaScript). These requests are never bots and add unnecessary processing overhead. Configure a Rules Engine rule to skip Bot Manager for static assets.

In your Firewall’s Rules Engine, create a rule with the following criteria:

• Criteria: Request URI does not match the regex:

\.(png|jpg|jpeg|gif|ico|css|js|svg|woff|woff2|ttf|eot|otf|webp|avif|mp4|webm|pdf)(\?.*)?$

• Behavior: Run Function > select your Bot Manager instance.

This ensures Bot Manager only runs on dynamic requests, reducing noise in your logs and improving accuracy.

Step 10: Protect high-value endpoints with stricter rules

Different endpoints have different risk profiles. Use multiple Bot Manager instances with different thresholds to apply stricter protection where it matters most.

To implement per-endpoint rules, create separate Rules Engine rules that run different Bot Manager instances based on Request URI criteria.

Step 11: Stream logs to external systems

For long-term analysis and SIEM integration, use Data Stream to forward Bot Manager logs to your observability stack.

1. Access Azion Console > Data Stream.
2. Click + Stream.
3. Configure the stream:
   • Source: Functions
   • Template: Functions Event Collector
4. Configure the Destination (Splunk, Datadog, Elastic, S3, or any HTTP endpoint).
5. Click Save.

Bot Manager logs will be forwarded in real time, enabling you to build custom dashboards, set alerts on spike detection, and correlate bot activity with application performance metrics.

Calibration checklist

Use this checklist after each calibration cycle:

• Observation mode ran for at least 24 hours with internal_logs: 2
• Score distribution analyzed via GraphQL botManagerMetrics
• Top impacted URLs identified via botManagerBreakdownMetrics
• Threshold set between the legitimate and bad bot score clusters
• Static rules producing false positives added to disabled_static_rules
• Dynamic rules tolerance validated for your traffic profile
• Static assets excluded from Bot Manager via Rules Engine
• High-value endpoints configured with stricter thresholds
• Production logs streaming to external SIEM or observability platform
• Real-Time Metrics dashboards reviewed weekly

Related resources