Deduplicate webhook deliveries with KV Store

Skip a webhook event your function already handled, with one key per event ID in KV Store, written only after the work succeeds.

Last updated

View as Markdown Agent setup

You mark each webhook event that a function handled with a key in KV Store, and the function skips an event whose key already exists, from the Azion API and the function’s code. To read and write keys for other purposes, refer to Manage key-value data from a function.

A provider that does not see a delivery acknowledged sends the same event again, so one event can reach the function more than once. A key per event ID lets the function recognize a repeat and answer it without doing the work twice.

  1. The function reads the key for the event ID. A value means an earlier delivery of the event was handled, so the function answers 200 and does nothing else.
  2. Without a key, the function does the work the event triggers.
  3. When the work fails, the function answers 500 and writes no key, so the provider’s next delivery runs the work again.
  4. When the work succeeds, the function writes the key with an expiration and answers 200.

Prerequisites

  • KV Store enabled on your account. The product is in Preview and is not enabled by default, so request access through Technical Support.
  • A personal token for the API call that creates the namespace.
  • A function that receives the provider’s webhook and verifies its signature, such as the handler that Build a Stripe webhook handler with Functions builds. Each event the provider sends carries an ID that stays the same across its deliveries.

The examples use the namespace webhook-events, keys of the form event:<event-id>, and an expiration of 86400 seconds. Replace them with your own values.

Create the namespace

A namespace is created through the Azion API, and a function only opens one that already exists. A namespace cannot be renamed or deleted, and two names that differ only in case are two namespaces, so create it once, in lowercase. A name runs 3 to 63 characters of letters, numbers, the hyphen, and the underscore.

To create the namespace, send its name to the KV Store API:

Shell
01curl --request POST \
02 --url https://api.azion.com/v4/workspace/kv/namespaces \
03 --header 'Accept: application/json' \
04 --header 'Authorization: Token <personal-token>' \
05 --header 'Content-Type: application/json' \
06 --data '{"name": "webhook-events"}'

The API answers 201 with the namespace. The request is synchronous, and there is no provisioning state to poll:

{"name":"webhook-events","created_at":"...","last_modified":"..."}

The webhook-events namespace exists and is empty. For every field and error of the namespace API, refer to Namespaces.

Skip a delivery the function already handled

kv.get returns null for a key the namespace does not hold, so a null means the event is new. The function writes the key only after the work succeeds: a key written before a failed attempt marks the event as handled, and the provider’s retry is then skipped. expirationTtl removes the key after the seconds it names, so the namespace does not keep a key per event forever. Its minimum is 60 seconds.

To deduplicate deliveries, add this code to the function, after the signature check:

JavaScript
01const NAMESPACE = 'webhook-events';
02const KEY_TTL_SECONDS = 86400;
03
04// Replace with the work the event triggers, such as a publish to a queue.
05async function handleEvent(event) {
06 console.log(`Handling ${event.id}`);
07}
08
09export default {
10 async fetch(request, env, ctx) {
11 // Read the ID from where your provider puts it, once the signature is verified.
12 const event = await request.json();
13 const key = `event:${event.id}`;
14 const kv = await Azion.KV.open(NAMESPACE);
15
16 // A key for this event means an earlier delivery was already handled.
17 if ((await kv.get(key, 'text')) !== null) {
18 return Response.json({ received: true, duplicate: true });
19 }
20
21 try {
22 await handleEvent(event);
23 } catch (error) {
24 console.log(error.message);
25 return Response.json({ error: 'Event not handled' }, { status: 500 });
26 }
27
28 // Written only after the work succeeds, so a failed attempt is never marked as handled.
29 await kv.put(key, 'handled', { expirationTtl: KEY_TTL_SECONDS });
30 return Response.json({ received: true });
31 },
32};

Azion.KV is a global of the runtime, reached with no import line and no credential. Azion.KV.open throws NotFound: KV namespace "webhook-events" does not exist when the account holds no namespace with that name. Under azion dev, open accepts a name that belongs to no namespace, so test the function once it is deployed.

The first delivery of an event runs handleEvent and writes event:<event-id>. A later delivery of the same event, within one day, answers {"received":true,"duplicate":true} and does not run the work. A repeat that arrives after the key expires runs the work again, so set expirationTtl longer than the time your provider keeps retrying an event.

Each delivery reads one key, and each new event writes one. KV Store includes 100,000 keys read and 1,000 keys written per day before a charge applies, and it accepts 1 write per second to the same key. For every bound, refer to KV Store limits.

Next steps

How KV Store worksWhy a key written in one place can be missing elsewhere for up to 60 seconds.KV Store APIEvery method, option, and error of the client that reads and writes the keys.Build event-driven APIsA webhook producer that filters repeated events before it publishes them to a queue.Publish a message to Upstash QStash from a functionHand each new event to a queue as the work the function does.