Troubleshoot Workloads

This page lists the symptoms a workload shows on live traffic or in a refused request, each with its cause and its fix. The workload’s own symptoms open the page: responses that are missing or out of date, and refused domains, settings, deployments, and certificates. Sections for Certificate Manager, which issues and stores the workload’s certificates, and Custom Pages, which replaces its error responses, close it. A quoted refusal is the API’s message unless the entry names another source, and Azion CLI prints the same message inside Error: Failed to ... [...].

A workload answers 404 with Azion’s error page

Requests to the workload domain return HTTP/2 404 and Azion’s HTML error page instead of your application’s response.

The workload has no application to send the request to: no deployment binds one yet, or the deployment has not propagated, as The deployment explains. A workload with Workload Domain Allow Access off also answers 404 on its workload domain, while its own domains keep serving.

• Create the deployment: it names the application, as Workloads quickstart shows for each interface.
• Repeat the request after several minutes: a deployment reaches traffic only once it propagates.
• Check the access switch: the workload domain serves only while workload_domain_allow_access reads true.
• Read the headers: curl -sI https://<your-workload-domain>/get prints the status line and the x-azion-request-id header that identifies the request.

Once the deployment propagates, the same request returns 200 from your application.

A request receives the old configuration after a change

After you save a change to a workload or its deployment, some requests still get the previous behavior, or the answers alternate between old and new.

The change spreads across Azion’s distributed infrastructure one data center at a time, over several minutes, and propagation is best-effort. Each data center serves either the old configuration or the new one until all of them agree, as Propagation details.

• Wait several minutes before you test: an early request measures propagation, not the configuration.
• Send several requests, not one: repeat the request until the answers agree.
• Confirm the API stored the change: azion describe workload --workload-id <workload-id> --format json prints the workload as the API keeps it.

When every data center serves the new configuration, each request receives the answer that configuration defines.

A custom domain does not reach the workload

Requests to a hostname of your own do not reach the workload, while its workload domain answers.

A workload answers only the hostnames listed in its domains, and a client reaches it only after DNS sends the hostname to Azion. Both parts are needed, as Workload domain and custom domains explains.

• List the hostname on the workload: add it to domains, or as a Subdomain and Domain row of the Domains section, as Add a custom domain to a workload shows.
• Point the hostname at the workload domain: add a CNAME record at your DNS provider, or host the zone in Edge DNS, as Point a domain to a workload shows.
• Give resolvers time: visitors reach the workload once their resolvers pick up the new record.
• Use a production workload: a staging workload takes no custom domain.

The hostname then answers with the same content as the workload domain.

A domain is refused with This account is not allowed to use the following CNAMEs

Saving the workload’s domains fails with This account is not allowed to use the following CNAMEs: example.com., where the message names each refused domain.

The account has no permission for a domain in domains, and the API refuses the whole request, as the Workload settings errors show.

• Remove the domain the message names, and save the rest of the list.
• List only domains your account has permission for.

Azion CLI then prints Updated Workload with ID <workload-id>, and the workload answers on the domains it kept.

A custom hostname is refused on a staging workload

Adding a domain to a workload fails with Custom hostname is not available in the environment 'Staging Infrastructure'.

A staging workload, with infrastructure set to 2, takes no custom hostname, an azion.app hostname included. It answers only on its <id>.preview.azionedge.net workload domain.

• Test on the workload domain: send "domains": [] and reach the staging workload through its workload domain.
• Use a production workload for your hostname: a workload with infrastructure set to 1 accepts it.
• Preview your hostname before its DNS changes: map it to a production workload in your device’s hosts file, as Test an application through the hosts file shows.

The production workload then accepts the hostname, and the staging workload keeps serving on its workload domain.

An azion.app domain is refused

Adding an azion.app hostname, which Azion Console calls Azion Custom Domain, fails with one of three messages.

A workload holds at most one azion.app hostname, a name belongs to one workload at a time, and every entry must conform to RFC 1035, as the Workload settings errors show.

• Keep one azion.app hostname per workload: Duplicated usage of suffix in alternate_domains: 'azion.app'. refuses a second one, even under a different name.
• Choose a name no other workload holds: The custom hostname is not available. means another workload has it, so pick another name or remove it there first.
• List the full hostname, never a wildcard: The domain does not conform to the format defined in RFC 1035. refuses *.azion.app.

The update then stores the hostname with the case you sent, and Azion serves it over HTTPS with its SAN certificate, as Create an Azion custom domain shows.

The workload domain cannot be closed

Turning off Workload Domain Allow Access fails with When the workload hostname access is blocked, the workload requires alternate domains or domains.

A workload needs a hostname to answer on, so workload_domain_allow_access can be false only while domains holds at least one entry.

• Add your own hostname or an azion.app hostname first.
• Close the workload domain after your hostnames answer everywhere, as Workloads best practices explains.

The workload domain then answers 404, while your hostnames keep serving.

The infrastructure cannot be changed

An update that moves a workload between staging and production fails with The infrastructure cannot be changed after Workload creation.

The infrastructure value is fixed when the workload is created, and the Console help text for Infrastructure reads “Once this option is saved, it cannot be modified.”

• Create a second workload on the other infrastructure, with infrastructure set to 1 for production, and give it its own deployment.
• Leave infrastructure out of updates to the existing workload.

The new workload gets the workload domain suffix of its infrastructure, .map.azionedge.net on production, as Infrastructure explains.

A second deployment is refused

Creating a deployment fails with The maximum number of deployments allowed per workload is 1.

A workload holds one deployment, which names its application, firewall, and custom page set, so a change goes into that deployment, as The deployment explains.

• Find the existing deployment: azion list workload-deployment --workload-id <workload-id> prints its ID, and GET /v4/workspace/workloads/<workload-id>/deployments returns it.
• Edit it in Azion Console: change Application, Firewall, or Custom Page in Deployment Settings, then select Save.
• Or send a PATCH to /v4/workspace/workloads/<workload-id>/deployments/<deployment-id> with the changed keys of strategy.attributes.

The PATCH returns 202, and the change reaches every hostname of the workload once it propagates.

A port or HTTP version change is refused

An update to the workload’s protocols fails with Invalid choices for multiple choices field: [80, 8008, 8080, 8880]. or Missing required choices for multiple choices field: ['http1', 'http2'].

The first message lists the four HTTP ports a workload accepts. The second means versions holds http3 without both http1 and http2.

• Use only 80, 8008, 8080, or 8880 in http_ports, and the HTTPS ports Protocols and ports lists.
• Send ["http1", "http2", "http3"], or drop http3.
• Turn on HTTPS support before HTTP/3 support in Azion Console: HTTP/3 support requires HTTPS support.

Azion CLI then prints Updated Workload with ID <workload-id>. For the other settings refusals, such as a cipher suite outside 1 to 8, refer to the Workload settings errors.

A client fails the TLS handshake on an mTLS workload

A client gets no response: curl exits with code 56 after the server’s certificate request, and LibreSSL reports reason(1116), certificate required.

The workload has mTLS in enforce mode, and the client presented no certificate or one the Trusted CA did not sign. A client that sends no Server Name Indication (SNI) is also closed before a route of your application resolves.

• Present a certificate the Trusted CA signed: the Trusted CA is the certificate in mtls.config.certificate, and curl sends a client certificate with --cert and --key.
• Send SNI from every client: on an enforce workload, a connection without it never reaches your application, as mTLS lists.
• Admit clients while you test: permissive completes the handshake and leaves the decision to a firewall rule, as Verification modes describes.
• Wait for a mode change to propagate: until it does, some data centers answer with the previous mode.

The handshake then completes, and the request reaches the application. For the verbose curl output of a refused handshake, refer to the mTLS errors.

A certificate is refused in a workload setting

Binding a certificate to a workload fails with Invalid certificate type, MUST be an Edge Certificate. or Invalid certificate type, MUST be a Trusted CA.

Each certificate field takes one type. tls.certificate takes a server certificate, of type edge_certificate, and mtls.config.certificate a Trusted CA certificate, of type trusted_ca_certificate, as the mTLS errors show.

• Check the type: the TYPE column of azion list digital-certificate --details shows it.
• Put the server certificate in tls.certificate, or null for the Azion SAN certificate.
• Put the Trusted CA certificate in mtls.config.certificate.

Azion CLI then prints Updated Workload with ID <workload-id>, and the bound certificate reads active in Certificate Manager.

Certificate Manager

Certificate Manager requests Let’s Encrypt certificates for a workload’s domains, stores the certificates you upload, and reports the state of each one in its status field. For where it acts on a request, refer to How Workloads works.

A certificate stays pending

A Let’s Encrypt certificate keeps the pending or challenge_verification status, and the workload’s Digital Certificate field warns “This certificate is pending validation and HTTPS may not work until it’s validated”.

The challenge cannot validate one of the names. HTTP-01 needs every name to point to Azion, and DNS-01 needs an _acme-challenge record, as Domain validation explains.

• Check the names: a certificate requested from the workload form covers the workload’s Domains, so confirm each one is correct.
• For HTTP-01, point every name at Azion, alternative names included.
• For DNS-01 at another DNS provider, add the CNAME: _acme-challenge.<your-domain> with the value <your-domain>.letsencrypt.azion.com. In Edge DNS, Azion writes the record itself.
• After 48 hours pending, check the CNAME records again: retries then run every 3 hours, and once a day after 7 days, as Issuance time and retries lists.

After a later attempt validates, the certificate reads active while a workload names it, or inactive until then. For the records step by step, refer to Request a Let’s Encrypt certificate.

A certificate request is refused with This account cannot use certain common or alternative names

A Let’s Encrypt request or a certificate signing request (CSR) fails with This account cannot use certain common or alternative names because it does not have permission for their domain names.

A name on the request belongs to a domain the account has no permission for. An azion.app hostname is refused too, even one a workload of the account holds, as the Certificates errors show.

• Name only hostnames in domains your account has permission for, in common_name and alternative_names.
• Request no certificate for an azion.app hostname: with tls.certificate set to null, the workload presents the Azion SAN certificate, which covers the Azion Custom Domain and the workload domain.

An accepted request returns the certificate’s id with the pending status.

A certificate fails to issue

A Let’s Encrypt certificate reads failed, and the workload’s Digital Certificate field warns “This digital certificate failed and HTTPS cannot be used until the issue is resolved”.

Validation failed, and Azion records why in the certificate’s status_detail field. A typical value names the hostname whose record to check:

Read status_detail from any interface, then correct the record or the name it points to:

• Azion Console: in Certificate Manager, the error icon of the certificate carries it. To sign in, refer to How to access Azion Console.
• Azion CLI: azion describe digital-certificate --digital-certificate-id <certificate-id> --format json prints it beside status.
• API: GET /v4/workspace/tls/certificates/<certificate-id> returns it with the certificate.

A later attempt in the retry schedule can then issue the certificate, and an issued certificate in use reads active with status_detail at "".

An uploaded private key is refused

Uploading a server certificate fails with The provided private key is invalid. Please check the key and try again.

The API cannot read the private_key sent with the certificate, as the Certificates errors show.

• Send the key that matches the certificate.
• Send it in PEM, with its -----BEGIN and -----END lines and no passphrase, as Key algorithms and formats lists.
• Keep the key one JSON string in an API body, with each line break written as \n.

Azion CLI then prints Created Digital Certificate with ID <certificate-id>, and the certificate reads inactive until a workload names it.

Custom Pages

Custom Pages replaces the error responses of a workload with pages you choose, through a custom page set that the workload’s deployment names. For where it acts on a request, refer to How Workloads works.

A custom page set is refused with Ensure this field has at least 1 elements

Creating a custom page set fails with Ensure this field has at least 1 elements., or Azion Console refuses to save it with You must have at least one custom page code.

A set holds at least one page, and the request sent "pages": [], as the Custom page settings errors show.

• Add a page to pages: one page binds a status code to content on a connector, such as {"code": "404", "page": {"type": "page_connector", "attributes": {"connector": <connector-id>, "ttl": 0, "uri": "/html", "custom_status_code": 404}}}.
• Add a page code in Azion Console: the Page Codes table needs one row before you save the set.

Azion CLI then prints Created Custom Page with ID <custom-page-id>.

An error page does not appear

The connector’s error response reaches the client unchanged, even though a custom page set holds a page for its status code.

A workload uses a set only when its deployment names it in strategy.attributes.custom_page. A deployment without one reads "custom_page": null in GET /v4/workspace/workloads/<workload-id>/deployments.

• Name the set in the deployment: select it in the Custom Page field of Deployment Settings, send its ID in a PATCH of strategy.attributes.custom_page, or pass --custom-page when you create the deployment with Azion CLI.
• Check the status code the connector returns: a page acts only on the code it lists, such as "404".
• Wait for the change to propagate, and for the ttl of an edited page to expire.

The client then receives the page’s content in place, with the page’s custom_status_code and no Location header.

Related resources