Real-Time Events best practices

A search over the records of past traffic is useful only when it comes back. Three mistakes stop it from coming back. A question asked too broadly reads more than the system reads for one answer, and fails without returning a row. A question asked of the wrong store returns nothing, because the record was never written there. A question asked too late returns nothing, because the record has already been removed. From the outside those three look alike: an empty result, or an error with no row behind it.

These practices apply to a search run in Azion Console and to a query sent to the Real-Time Events GraphQL API. The mechanism each one leans on is on How Real-Time Events works, and the value of every bound is on Limits.

The practices below narrow a query before its period is widened, read the data source that owns the question, choose between Real-Time Events, Real-Time Metrics, and Data Stream, and move records out before retention removes them.

Filter before widening the time range

The log database counts the rows a query reads, not the rows it returns. A period widened with no filter reads every record in it, and can reach the row-read bound while answering with almost nothing. One filter on a value the record carries cuts the read down before the period has to. The cost is that a filter set wrong hides the record you are looking for, so narrow on a value you are certain of: a request id, a host, or an HTTP status code.

This query reads the HTTP Requests records of one request id, inside a window around it:

The request id bounds the read before the period does, so widening the window costs little. Check it by running the query: a read inside the bound returns its rows, and one past it returns an error instead. To narrow a search on the same values in Azion Console, refer to Filter events.

Query the data source that owns the question

Each data source is a separate index, and a query reads the rows of the one it names and of no other. A question about a function answered from the HTTP Requests records reads one row for every request the workload served, while the Functions records hold one row for each request that invoked a function. The cost is that you have to know which product writes the record before you can ask for it. A question spanning two products therefore takes two queries, one per data source.

This query reads the Functions records, which carry the instances a request ran and the time they took:

The same question asked of workloadEvents reads a row for every request in the period, to answer about the few that ran a function. Check it by asking both: the narrower data source returns the same answer and reads less, which is what Data Scan measures. For the data source each product writes into, refer to Data sources, and for the fields of each dataset to Real-Time Events GraphQL API fields. To build a query around one record, refer to Investigate a request with the GraphQL API.

Choose the product that matches the shape of the question

Three products read the same traffic and answer different questions. Real-Time Events answers what happened to one request: it returns the records themselves, one row per event, carrying every field that event wrote. Real-Time Metrics answers how many, over time: it returns counters that are already aggregated, so no single request is visible in them. Data Stream answers neither by itself, and sends every record continuously to an endpoint you own, where the question is asked with your own tools.

The cost of the choice is that each product commits you to its shape. A counter cannot be opened to show the request behind it. A record search over the full retention window with no filter reaches the row-read bound rather than counting what it found. A stream answers nothing until the destination that receives it is running.

Check the choice against the question as you asked it: a single named request belongs here, a count over time to Real-Time Metrics, an answer read elsewhere to Data Stream.

Move records out before retention removes them

Real-Time Events keeps an event record for 7 days, which is 168 hours, and keeps an Activity History record for 2 years. At the end of that period the record is removed, whether or not anyone read it, and nothing recovers it afterwards. Retention is not applied backwards, so a record that has to outlive the window has to be leaving while it still exists. That is a Data Stream job, configured before the incident rather than after it.

The cost is a second system. A stream has an endpoint you run and storage you pay for, and a delivery the destination refuses is one more thing to watch: the Data Stream data source holds one record per delivery, with the status the endpoint answered with.

Check it in the destination rather than here: when the oldest record there falls inside the Real-Time Events retention window, nothing has been leaving. For both retention periods alongside every other bound, refer to Limits.

Related resources