Sign origin requests with HMAC

Serve a private S3-compatible bucket through a connector by signing each origin request with HMAC credentials.

Last updated

View as Markdown Agent setup

You can serve the objects of a private bucket through a connector that signs every request to the bucket’s S3-compatible endpoint, from Azion Console, the Azion CLI, or the API. The signing is HMAC, a setting of Origin Shield on a connector of type http. To accept only Azion’s addresses at your origin instead, refer to Allow Azion’s IP ranges at your origin.

The examples use the Azion Object Storage endpoint s3.us-east-005.azionstorage.net with the region us-east-005, and an object named hello.txt at the root of the bucket. Replace them with the endpoint, the region, and an object of your provider.

Select the interface you use. The prerequisites and every task on this page follow that choice.

Prerequisites

  • An application served by a workload, and the workload domain or a domain of your own that the workload answers on. To create them, refer to Applications quickstart.
  • A private bucket on an S3-compatible endpoint, and an access key and a secret key that can read the bucket. For the credentials of Azion Object Storage, refer to S3 compatibility.
  • curl, to verify the result.
  • The Azion CLI, installed and authorized.
  • The ID of your application.
  • A personal token and the ID of your application.

Create the signing connector

The connector reaches the endpoint over HTTPS and sends the endpoint’s own name as the Host header. Its path, /<bucket-name>, goes in front of every request path, so a request for /hello.txt reaches the endpoint as /<bucket-name>/hello.txt. With Origin Shield on, HMAC signs each of these requests with your credentials, using the aws4_hmac_sha256 type.

The CLI creates a connector from a JSON file. Save this body as connector.json, and replace <bucket-name>, <access-key>, and <secret-key> with your values:

JSON
01{
02 "name": "my-bucket-connector",
03 "type": "http",
04 "attributes": {
05 "addresses": [{ "address": "s3.us-east-005.azionstorage.net" }],
06 "connection_options": {
07 "transport_policy": "force_https",
08 "host": "s3.us-east-005.azionstorage.net",
09 "path_prefix": "/<bucket-name>"
10 },
11 "modules": {
12 "origin_shield": {
13 "enabled": true,
14 "config": {
15 "origin_ip_acl": { "enabled": false },
16 "hmac": {
17 "enabled": true,
18 "config": {
19 "type": "aws4_hmac_sha256",
20 "attributes": {
21 "region": "us-east-005",
22 "service": "s3",
23 "access_key": "<access-key>",
24 "secret_key": "<secret-key>"
25 }
26 }
27 }
28 }
29 }
30 }
31 }
32}

The file holds your secret key. Delete it after you create the connector.

To create the connector with the Azion CLI, pass the file and the type. The command needs --type even though the file names the type:

azion create connector --type http --file connector.json

The output carries the ID of the new connector:

Created Connector with ID <connector-id>

Record the ID: the next task passes it as <connector-id>. The connector exists in your account and signs each request it sends to the endpoint.

The API takes the connector as a JSON body. Save this body as connector.json, and replace <bucket-name>, <access-key>, and <secret-key> with your values:

JSON
01{
02 "name": "my-bucket-connector",
03 "type": "http",
04 "attributes": {
05 "addresses": [{ "address": "s3.us-east-005.azionstorage.net" }],
06 "connection_options": {
07 "transport_policy": "force_https",
08 "host": "s3.us-east-005.azionstorage.net",
09 "path_prefix": "/<bucket-name>"
10 },
11 "modules": {
12 "origin_shield": {
13 "enabled": true,
14 "config": {
15 "origin_ip_acl": { "enabled": false },
16 "hmac": {
17 "enabled": true,
18 "config": {
19 "type": "aws4_hmac_sha256",
20 "attributes": {
21 "region": "us-east-005",
22 "service": "s3",
23 "access_key": "<access-key>",
24 "secret_key": "<secret-key>"
25 }
26 }
27 }
28 }
29 }
30 }
31 }
32}

The file holds your secret key. Delete it after you create the connector.

To create the connector with the API, send a POST request to the connectors endpoint. Replace [TOKEN VALUE] with your personal token:

Shell
01curl --request POST \
02 --url https://api.azion.com/v4/workspace/connectors \
03 --header 'Accept: application/json' \
04 --header 'Authorization: Token [TOKEN VALUE]' \
05 --header 'Content-Type: application/json' \
06 --data @connector.json

The API answers 202 with "state": "pending" and the stored connector, whose id the next task passes as <connector-id>. In the response, attributes.modules.origin_shield.config.hmac.enabled reads true. With hmac.enabled as true and no hmac.config, the API refuses the request with 400 and 28014 Module configuration must be provided when 'enabled' is true.

To create the connector in Azion Console:

Open the Connectors page

Access Azion Console > Connectors.

Start a new connector

The Create Connector page opens.

Name the connector

In General, enter my-bucket-connector as the Name.

Select the HTTP type

In Connector Type, select HTTP.

Enter the endpoint as the address

Under Address Management, enter s3.us-east-005.azionstorage.net in Address, without a protocol or a port.

Send the endpoint's name in the Host header

In Host, enter s3.us-east-005.azionstorage.net.

Enter the bucket as the path

In Path, enter / followed by the name of your bucket.

Connect over HTTPS only

In Transport Protocol Policy, select Force HTTPS.

Turn on Origin Shield

In Modules, turn on Origin Shield.

Turn on HMAC

Turn on the switch of the HMAC section. Type shows aws4_hmac_sha256 and cannot change.

Enter the region and the service

In Region, enter us-east-005. In Service, enter s3.

Enter the credentials

In Access Key and Secret Key, enter the credentials that can read the bucket.

Select Create

The connector exists in your account and signs each request it sends to the endpoint.

Turning HMAC off later removes the stored credentials. To turn it back on, enter the access key and the secret key again.

Send requests to the connector

A connector receives no traffic until a rule on your application names it. This rule matches requests for /hello.txt in the request phase, and its Set Connector behavior sends them to the signing connector. Replace /hello.txt with the path of your object. If your application already has a rule that matches every path, keep this rule after it, so this rule decides the connector for the object. For more information, refer to Set Connector.

Keep the rule in a file: on a command line, the shell would expand ${uri}. Save this body as rule.json, and replace <connector-id> with the ID of the signing connector:

JSON
01{
02 "name": "send-object-to-bucket",
03 "active": true,
04 "criteria": [
05 [
06 {
07 "variable": "${uri}",
08 "conditional": "if",
09 "operator": "starts_with",
10 "argument": "/hello.txt"
11 }
12 ]
13 ],
14 "behaviors": [
15 {
16 "type": "set_connector",
17 "attributes": { "value": <connector-id> }
18 }
19 ]
20}

To create the rule with the Azion CLI, run this command. Replace <application-id> with the ID of your application:

azion create rules-engine --application-id <application-id> --phase request --file rule.json

The output carries the ID of the new rule:

Created Rules Engine with ID <rule-id>

The rule is active on your application, and it sends requests for /hello.txt to the signing connector.

Keep the request body in a file: on a command line, the shell would expand ${uri}. Save this body as rule.json, and replace <connector-id> with the ID of the signing connector:

JSON
01{
02 "name": "send-object-to-bucket",
03 "active": true,
04 "criteria": [
05 [
06 {
07 "variable": "${uri}",
08 "conditional": "if",
09 "operator": "starts_with",
10 "argument": "/hello.txt"
11 }
12 ]
13 ],
14 "behaviors": [
15 {
16 "type": "set_connector",
17 "attributes": { "value": <connector-id> }
18 }
19 ]
20}

To create the rule with the API, send a POST request to the request-phase rules of your application. Replace <application-id> with the ID of your application:

Shell
01curl --request POST \
02 --url https://api.azion.com/v4/workspace/applications/<application-id>/request_rules \
03 --header 'Accept: application/json' \
04 --header 'Authorization: Token [TOKEN VALUE]' \
05 --header 'Content-Type: application/json' \
06 --data @rule.json

The API answers 202 with "state": "pending", and the response returns the stored rule with its id and its order. The rule is active on your application, and it sends requests for /hello.txt to the signing connector.

To create the rule in Azion Console:

Open the application

Access Azion Console > Applications, and select your application.

Select the Rules Engine tab
Select + Rule
Name the rule

In General, enter send-object-to-bucket as the Name.

Select the request phase

In Phase, select Request Phase.

Set the criterion

Under Criteria, select the variable ${uri} and the operator starts_with, and enter /hello.txt as the argument.

Select the Set Connector behavior

Under Behaviors, select Set Connector.

Select the signing connector

In Connector, select my-bucket-connector.

Select Save

The rule appears in the Rules Engine tab, under the Request heading.

Verify the signed request

The check is the same whichever interface you used. A new connector and a new rule take several minutes to reach Azion’s distributed infrastructure, and data centers apply them at different times. Until then, a request can still reach the connector of another rule. Repeat the request until the endpoint answers. For more information, refer to Propagation.

Request the object through your workload. Replace <workload-domain> with the workload domain or your own domain:

curl -s -D - https://<workload-domain>/hello.txt

The endpoint answers 200 with the content of the object. These headers identify the storage endpoint as the source of the answer:

Text
01HTTP/2 200
02…
03content-type: text/plain
04…
05server: azion webserver
06x-amz-request-id: …
07…

To compare, request the same object from the endpoint directly, without a signature:

curl -s -D - https://s3.us-east-005.azionstorage.net/<bucket-name>/hello.txt

The endpoint refuses the unsigned request with 401, not 403:

Text
01HTTP/1.1 401
02Server: azion webserver
03…
04<Error>
05 <Code>UnauthorizedAccess</Code>
06 <Message>bucket is not authorized: <bucket-name></Message>
07</Error>

The workload returns the same 401 when HMAC is off on the connector. The connector signs each request to the private bucket, and your workload serves the object. For the fixes, refer to The storage endpoint returns 401 UnauthorizedAccess.

Next steps

Origin IP ACL and HMACHow Origin Shield protects the origin with an allowlist of Azion's addresses and with HMAC signing.Connector settingsEvery Origin Shield and HMAC field, its API name, its default, and its limits.Set the Host header and path prefix for an originChoose the Host header a connector sends and the path it prepends to every request.Troubleshoot ConnectorsFix a connector whose origin refuses requests or answers with an error.