Skip to content

Frequently asked questions#

Answers to the questions we are asked most often in support. If your question isn't here, or the steps don't solve your problem, contact support.

Site down, errors and timeouts#

My site returns 500, 502, 503 or 504 errors, or the logs show PHP-FPM 'seems busy'. What should I check?

These usually mean requests take too long or all PHP-FPM workers are busy. Timeouts along the request path:

  • CDN: 90 seconds, then a 503 (first byte timeout).
  • Cluster ingress: 60 seconds, then a 504.
  • PHP: max_execution_time defaults to 900 seconds (PHP_MAX_EXECUTION_TIME).

When an upstream times out first, nginx logs a 499. To investigate:

  1. Check logs in the logging dashboard or with lagoon logs --project <project> --environment <environment> --service nginx --follow (also php). Look for pm.max_children warnings, PHP fatals, failed database connections, slow requests and spikes from single IPs or user agents.
  2. For 500s, check that the latest deployment succeeded and no variable is missing.
  3. Set short timeouts on outbound HTTP calls, make anonymous pages cacheable and block abusive bots.
  4. Raise the ingress timeout with the proxy-read-timeout route annotation only for genuinely long requests; support changes the CDN timeout.

If it continues, contact support with the environment, time window and sample errors.

See also: request timeouts

How can I temporarily take a site offline or block all public traffic to an environment?

  1. Stop the environment's nginx service with the Lagoon API. Visitors then get an error instead of your application:

    Text Only
    lagoon raw --raw 'mutation { stopOrStartEnvironmentService(input: { project: "<project>" environment: "<environment>" serviceName: "nginx" state: STOP }) }'

    Run it again with state: START to bring the site back. If it still returns 503 afterwards, redeploy with lagoon deploy latest -p <project> -e <environment>.

  2. If you use the amazee.io CDN and need an edge block, an IP restriction or a static maintenance page instead, contact support.

  3. For a non-production environment you no longer need, delete the environment.

Tell support before you deliberately take a production site offline, so monitoring alerts are expected.

See also: 2.31.0

A container is in CrashLoopBackOff and my site returns 503. What should I check?

The container starts, then exits or fails its health checks. Check:

  1. The container logs (logging dashboard, the failed build log, or lagoon logs --project <project> --environment <environment> --service <service>) for the actual error, such as a PHP fatal or a Node.js start failure.
  2. Missing environment variables: many apps exit when configuration validation fails. Add one with lagoon add variable --project <project> --environment <environment> --scope runtime --name NAME --value VALUE, then redeploy.
  3. Missing build output, for example a Node.js app whose build artefacts were not produced.
  4. Memory: containers that reach their limit are killed and restarted. Look for out-of-memory messages.
  5. Slow startup: work done before the app listens (fetching remote data, warming caches) can fail the liveness probe. Move it out of startup.
  6. A bad build: redeploy the last known-good commit.

If the logs show no application error, or production is down, contact support with the environment, URL, start time and build link.

See also: rollouts

My site was briefly slow or down and recovered on its own. How do I find out whether it was a platform issue?
  1. Check status.amazee.io for an incident or maintenance at that time.
  2. Check your application and nginx logs in the logging dashboard for that window.
  3. Look at the pattern. Short failures (database connection refused, 503s) across several sites at the same moment usually point to a platform event such as managed database maintenance. A single slow site usually points to the application: slow queries, external APIs or exhausted PHP workers.
  4. A common application cause is queue workers calling a slow or unavailable external API. Each stuck retry ties up a PHP-FPM worker; once the pool is exhausted, requests fail with 503s and visitors who give up show as 499s. Set short timeouts on outbound calls, use backoff or a circuit breaker, and run queue workers separately from web PHP-FPM.

If you can't explain it, contact support with the URLs, start and end times with timezone, the exact error messages and whether several sites were affected.

See also: statuspage

Will my site have downtime during deploys or maintenance, and how do I get high availability for a launch?

During deployments Lagoon starts the new pod before removing the old one. During maintenance or node failures, a service running a single pod can stop before its replacement is ready, causing a short gap.

  • Application pods are scaled horizontally to meet traffic on both Cloud Basic and Cloud Pro.
  • Cloud Pro also uses Pod Disruption Budgets (PDBs), which make sites more robust against node restarts and maintenance.
  • Only services whose application supports it can run more than one pod: nginx, nginx-php, node and python. Redis can't; for basic and worker services it depends on what runs inside them.

Before a launch or migration, open a ticket with the production environments (both, if you use Active/Standby), the launch date and expected traffic so support can confirm autoscaling and PDBs. Make sure your pages are cacheable by the CDN and your application tolerates rolling restarts.

See also: cloud basic vs cloud pro

My site is down. What should I include in my support ticket?

A one-line "site is down" ticket slows diagnosis, because support first has to ask for the basics. Include:

  • the full URLs affected and the project and environment name
  • the exact error (copy it or add a screenshot) and the HTTP status
  • when it started, with timezone, and whether it is constant, intermittent or ongoing
  • where it fails from (everywhere, or one network)
  • recent deploys, DNS or configuration changes, with the build link
  • whether production is affected
  • curl -sv https://www.example.com/ output, the response headers (X-Cache, X-Served-By) for CDN sites, and a HAR file for browser-only issues

For production outages, check status.amazee.io and mark the ticket as urgent. See the ticket system for how to raise it.

See also: tickets

My development environment returns 502/503 or won't wake up after being idle. What can I do?

Development environments are idled after a period of inactivity (typically 4 hours, subject to change). Production environments are never idled. When you visit an idled environment, a waiting page runs a JavaScript check while it starts; requests made before the pods are ready can return 502 or 503, and Fastly may show a first-byte timeout.

  1. Wait a minute and reload in a real browser. Add a query string such as ?nocache=1 in case a cached waiting page is served.
  2. Wake it by SSH (lagoon ssh -p <project> -e <environment>) or by triggering a deployment.
  3. If external tools need to reach the environment, set disableRequestVerification: true on its routes in .lagoon.yml.
  4. If that isn't enough, ask support to disable idling for that environment. This uses more resources.

If an environment keeps returning 502 after it has fully started, check its container logs, then contact support with the environment, URL and times.

See also: idling

How do I restart, stop or start a single service (for example Solr, Redis, PHP or nginx)?

Use the Lagoon API through the Lagoon CLI. You only need your normal Lagoon permissions:

Text Only
lagoon raw --raw 'mutation { stopOrStartEnvironmentService(input: { project: "<project>" environment: "<environment>" serviceName: "<service>" state: RESTART }) }'

Use RESTART to roll a stuck service, STOP to scale it down and START to bring it back. <service> is the service name from your docker-compose.yml. A redeploy (lagoon deploy latest -p <project> -e <environment>) also restarts all services.

After restarting a search service, re-run indexing if needed (for example drush search-api:index). If the service keeps failing, or you can't redeploy production right now, contact support with the project, environment and service name.

See also: 2.31.0

SSL/TLS certificates#

My Let's Encrypt certificate is not being issued or renewed, or I got a certificate expiry or name-mismatch alert. What should I check?

amazee.io issues Let's Encrypt certificates automatically with cert-manager, using the HTTP-01 challenge only (DNS-01 is not supported). An expiry alert or a name-mismatch error means issuance is failing. Check:

  1. DNS for the hostname points to amazee.io, or to the CDN target if you use the amazee.io CDN, for IPv4 and (if configured) IPv6: dig +short www.example.com.
  2. The route doesn't have tls-acme: false in .lagoon.yml (true is the default).
  3. No CAA record blocks Let's Encrypt. letsdebug.net can check this.
  4. If the site is behind the amazee.io CDN (Fastly), LAGOON_FASTLY_SERVICE_ID is set for the project or environment (see the Fastly SAN question).
  5. Redeploy the environment after fixing DNS or configuration, then allow a few minutes.

If the domain is no longer used, remove it from .lagoon.yml and redeploy so renewal attempts and alerts stop. If every check passes and the certificate is still missing, contact support with the hostname and environment.

See also: invalid ssl

How do I install or renew a custom (non-Let's Encrypt) SSL certificate?

Let's Encrypt is the default and recommended option because it renews automatically. To use your own certificate:

  1. Open a ticket with the environment and hostnames.
  2. Send the certificate, the full intermediate chain (PEM, in chain order) and the private key through a secure, expiring link. Never paste keys into the ticket, and send any password separately.
  3. Before sending, check the key matches: openssl x509 -noout -modulus -in cert.pem | openssl md5 and openssl rsa -noout -modulus -in key.pem | openssl md5 must print the same value.
  4. Set tls-acme: false on the routes that use the custom certificate and deploy. Switching from true to false removes the previously generated Let's Encrypt certificate for that route, so agree the timing with support.
  5. Check the result: openssl s_client -connect www.example.com:443 -servername www.example.com.

Custom certificates are not renewed automatically. Send the replacement several weeks before expiry; expiry alerts continue until it is installed.

See also: certificate management

My site behind the amazee.io CDN (Fastly) shows 'Requested host does not match any Subject Alternative Names (SANs)' or other TLS errors at the edge, but the origin works. How do I fix it?

Fastly does not have a certificate for that hostname yet. The amazee.io CDN registers your domains with the Fastly service and uploads their certificates only where LAGOON_FASTLY_SERVICE_ID is set.

  1. Check the variable at organization, project or environment level, for example lagoon list v -p <project> -e <environment> --reveal.
  2. If it is missing, add it with build scope and redeploy. A project-level variable needs every environment in the project redeployed.
  3. Make sure the route uses tls-acme: true (unless you use a custom certificate) and that DNS points to the CDN target.
  4. After a successful deployment, wait a few minutes for the certificate to be issued and uploaded.

If you don't know your service ID, or the error remains after a redeploy, contact support with the hostnames and environment.

See also: fastly service

Do amazee.io's automatic Let's Encrypt certificates use keys of at least 2048 bits?

Yes. Certificates issued by cert-manager use RSA keys of 2048 bits or more. Check any hostname with:

Text Only
echo | openssl s_client -connect www.example.com:443 -servername www.example.com 2>/dev/null | openssl x509 -noout -text | grep 'Public-Key'

If the site is behind the CDN, this shows the edge certificate. For a custom certificate, the key size depends on how you generated it.

See also: certificate management

WAF and security#

My site is overloaded by bots, scrapers or a possible DDoS. How do I get traffic blocked or rate-limited, or request other WAF changes?
  1. Find the source in your router logs in the logging dashboard: top client IPs, user agents and paths (for example search pages with many query parameters). Requests for files like wp-config.php or .env are usually automated scanners, and are noise unless they cause load.
  2. Ask well-behaved crawlers to slow down with robots.txt, and make heavy endpoints cacheable.
  3. For abusive traffic, open a ticket with the IPs or user agents, paths, time window and the behaviour you want (block, rate limit or challenge). Support can add CDN or WAF rules for specific paths, user agents or IPs.
  4. With the amazee.io Advanced WAF, threshold blocking flags abusive IPs automatically. Self-service rule actions are block, allow and add signal; ask support for anything else, such as a challenge action. New rules normally start in log-only mode.

If the WAF is blocking legitimate production traffic, say so clearly in the ticket so it is handled as urgent.

See also: waf

Requests (for example Drupal admin saves, form POSTs or third-party calls) fail with HTTP 406. Is the WAF blocking them?

Usually yes. A 406 from the amazee.io Advanced WAF means the request was flagged (for example as XSS, SQLi or code injection) and blocked. It often affects Drupal admin saves with rich content (Layout Builder, Paragraphs, embeds, translations) and integrations calling from unfamiliar IPs.

Open a ticket with:

  • the full URL and method
  • the exact time with timezone
  • the source IP
  • the request ID from the error page or response headers, if shown
  • what you were doing and which environments are affected

Support reviews the signal and adds a narrowly scoped exception, for example for an admin path or a partner's published IP ranges. Don't keep retrying meanwhile: repeated flagged requests can get your IP blocked. Don't ask for the WAF to be disabled, or disable your editor's security features, as a workaround.

See also: waf

We want to run a penetration test or vulnerability scan against our site. What do we need to do?

Testing against your own applications is generally allowed, but you must give notice:

  • At least two weeks before any testing that may affect other customers or our services, create a support ticket.
  • At least four weeks before broader testing, such as load testing your application on amazee.io infrastructure or security testing of amazee.io infrastructure.

Include the testing vectors and scope, target websites, originating IPs, expected traffic volume (bandwidth and requests per second), start and end times, and a contact who can stop the test.

Without an agreed exception, the WAF or CDN may block or flag the scanner, and your results won't reflect your application. Support can add a temporary exception for the source IPs during the test window. Don't start until support confirms.

See also: infra testing

A visitor is being blocked (403 or connection errors) and we think the WAF is responsible. What should we do?
  1. If you have WAF dashboard access, search for the request around that time and check the list of flagged IPs. IPs that exceed thresholds are blocked for a configurable time (24 hours by default) and can be evicted from the list.
  2. For a false positive, an allow rule (by path, IP or user agent) is usually the fix. Monitoring and SEO tools can be allow-listed the same way. Allow rules take precedence over block rules.
  3. If you can't find the event, open a ticket with the URL, the client IPv4/IPv6 address, the exact time and timezone, and the error the user saw.

A browser ERR_SSL_PROTOCOL_ERROR is not a typical WAF block (those return 403 or 406). Mention it, because it may be a network or TLS issue.

See also: waf

Can the WAF protect my site while I prepare a Drupal or PHP security upgrade, and can you tell whether the vulnerability was exploited?

The amazee.io Advanced WAF supports virtual patches for known vulnerabilities, which can buy your team time to patch. Open a ticket with the advisory (CVE or Drupal SA), the affected sites and your upgrade timeline; support checks whether existing rules cover the attack and adds a targeted rule if needed. If you don't have the WAF, contact support to discuss options.

A WAF rule only reduces risk. Always apply the core or contrib update and redeploy as soon as possible.

To look for exploitation:

  1. In the logging dashboard, select your tenant and review router, nginx and PHP logs for the exposure window: unusual POSTs, unexpected paths, errors and spikes from single IPs.
  2. Check your application for unexpected files or admin users.
  3. If you use the WAF, ask support to check WAF signals for that period.

See also: waf

My origin still answers requests directly, bypassing the CDN/WAF. How do I lock it down?

Removing direct-to-origin routes from .lagoon.yml stops Lagoon creating those ingresses, but the origin can still answer for hostnames it serves. To make sure all traffic passes through the CDN and WAF, ask support to restrict origin access to the CDN. Include the project, environment and hostnames, plus a test command (for example curl -I against the origin) so the change can be verified afterwards.

See also: cdn

Our vulnerability scanner reports CVEs in the amazee.io-managed infrastructure. Who handles that?

Open a support ticket and attach the scanner export (CVE IDs, affected images or packages, scan date). amazee.io checks the findings against the managed infrastructure and plans any upgrades; support tells you about maintenance windows and confirms when patching is done.

Vulnerabilities inside your own containers (application dependencies, your Dockerfile layers, base image versions you pin) are yours to fix: rebuild on the latest Lagoon base images and update your dependencies. If you plan your own security testing against the platform, give the notice required by the testing policy.

See also: infra testing

PHP, Node.js and base images#

A PHP security release is out. Will my environments be patched automatically, and how do I check the running version?

No. Running environments are not patched in place. Updated Lagoon base images (uselagoon/php-<version>-fpm, php-<version>-cli and so on) are published once the upstream PHP fix is available, and progress on critical issues is posted on status.amazee.io. Then:

  1. If you pin image tags, update them to the release that contains the fix. If you build from your own derived base images, rebuild those first.
  2. Redeploy every environment you want patched, starting with a development environment: push a commit or run lagoon deploy latest -p <project> -e <environment>.
  3. Check the version in the container: connect with lagoon ssh -p <project> -e <environment> and run php -v, or look at the Drupal status report.

The PHP version shown in the dashboard can lag behind the running container; the container is the source of truth. If php -v still shows the old version after a clean deploy, look for a pinned or cached image tag before contacting support.

See also: php fpm

Have the Lagoon Node.js images been updated for the latest Node.js security release?

New releases of the Lagoon Node.js images (uselagoon/node-<version> and the -builder variants) pick up Node.js patch releases. Running containers are not updated automatically.

  1. Check the release notes at github.com/uselagoon/lagoon-images for the Node.js patch version.
  2. Redeploy your environments.
  3. Confirm with node -v in the container.

Use a currently supported major version (listed on the Node.js image page). Images for end-of-life versions stay available for compatibility only: they are no longer officially supported and usually stop being updated with the Lagoon release after the official end-of-life date, so they don't get security fixes.

See also: nodejs

How can I get notified when new Lagoon base images are released?
  • Watch github.com/uselagoon/lagoon-images on GitHub with Custom > Releases notifications. Every image release, including security fixes, is published there with release notes.
  • Subscribe to status.amazee.io for platform-wide incidents such as critical PHP vulnerabilities, where amazee.io posts progress and image availability.

A release does not update your environments until you redeploy them.

See also: statuspage

How do I change PHP settings such as php.ini values (opcache, upload or session settings)?

The Lagoon PHP images support three approaches, in order of preference:

  1. Use a documented PHP_* environment variable where one exists, for example PHP_MEMORY_LIMIT or PHP_MAX_EXECUTION_TIME. Add it with lagoon add variable or in the dashboard.
  2. Create your own fpm-pool config and set values with php_admin_value / php_admin_flag. Settings with the PHP_INI_SYSTEM changeable mode can't be set this way.
  3. Provide your own php.ini or php-fpm.conf in your Dockerfile. This is least preferred, because you stop receiving upstream changes to those files.

Changes take effect only after a deployment (lagoon deploy latest -p <project> -e <environment>), not on a running environment. Test on a development environment first, then check with php -i | grep <directive> inside the container.

See also: php fpm

Deployments and builds#

Deploys fail with 'Permission denied (publickey)', or our Git host changed its SSH hostname. What do I update?

Lagoon clones your repository with the Git URL stored on the project and a per-project deploy key.

  • Key missing: get it with lagoon get project-key --project <project> and add it as a read-only deploy key on the repository (on GitHub: Settings > Deploy keys). Each project has its own key, even if it shares a codebase. For private submodules or Composer packages, the key needs access to those repositories too.
  • Git host changed: Lagoon doesn't update the URL automatically. Update each project, for example lagoon update project -p <project> --git-url 'git@<new-host>:<org>/<repo>.git', then deploy a non-production branch to confirm cloning works.
  • Pushes no longer trigger deployments: check that the repository webhook still points to Lagoon.

Redeploy after fixing. If the error refers to a host other than your Git provider, or you need many projects updated, contact support with the build log.

See also: add project

Why are my builds queued (for example 'queued 4/8'), slow or not starting?

Each cluster runs a limited number of builds at once; further builds wait in a shared queue. 'Queued 4/8' means your build is 4th of 8 builds waiting on that cluster. You only see your own builds, so the others belong to other projects. Deploying many projects together makes most of them show as queued; they run in order.

  • Stagger large release deploys, and don't keep re-triggering deployments, which lengthens the queue.
  • Keep builds efficient: use the Docker layer cache, keep images small and avoid heavy pre- or post-rollout tasks such as large syncs.

A deployment stuck at New is different: it never reached the cluster. Contact support with the project, environment and build names if a build stays queued for more than about an hour, stays at New, or fails or is cancelled without an error in the log.

See also: slow builds

How do I remove a service (for example OpenSearch or Solr) from an environment, or change its type?

Run a one-off deployment that sets the service type to none and enables cleanup of removed services:

Text Only
lagoon deploy latest -p <project> -e <environment> \
  --buildvar LAGOON_FEATURE_FLAG_CLEANUP_REMOVED_LAGOON_SERVICES=enabled \
  --buildvar LAGOON_SERVICE_TYPES=<service>:none

Build variables apply to that deployment only (deployments triggered by webhooks don't receive them). You can also add them as build-scope variables so every deployment applies them.

  • Cleanup removes the service and its data, so take a snapshot or export first, and make sure your application no longer uses the service.
  • Once stable, remove the service from docker-compose.yml.
  • To change a type, update lagoon.type for the service in docker-compose.yml; the next deployment recreates it.

If the deployment fails or a volume is left behind, contact support.

See also: environment variables

My deployment fails with an ingress admission webhook error saying the host is already defined. Why?

The same hostname is usually defined twice in .lagoon.yml. A common cause is listing a domain as a plain route and again with wildcard: true: a wildcard route already creates both the root hostname and the *. ingress, so the second definition conflicts.

Remove the duplicate entry, keep the single wildcard: true route and redeploy. Wildcard certificates can't be issued with Let's Encrypt HTTP-01, so check the TLS settings for wildcard routes. If the redeploy still fails because an old ingress is stuck, contact support with the build link.

A production deploy fails with 'unable to create deploy task: environments must be on same deploytarget'. What's wrong?

The project usually still has an old Active/Standby setting: standbyProductionEnvironment points to an environment that no longer exists or is on a different cluster, so Lagoon refuses to create the deploy task. The running environment is not affected.

A project admin can clear it by following Disabling Active/Standby in the Lagoon docs, which sets standbyProductionEnvironment to an empty value with the updateProject mutation. Otherwise, contact support to clear it. Then trigger the deployment again.

See also: active standby

My deployment fails with 'deployment 'X' exceeded its progress deadline'. What does that mean?

The new pod for that service didn't become ready in time, so Lagoon kept the old one and failed the build. Read the section under the error in the build log: it includes the new container's startup logs and pod conditions.

Typical causes are application errors at startup, missing environment variables, failing health checks, stale build caches, or a storage volume that can't be mounted. For a search service such as Solr with a damaged volume, recreating the service and reindexing is often fastest. Redeploy after fixing. If the log shows no application error, contact support with the build link.

See also: rollouts

Deploying a new branch fails with 'exceed development limit'. What can I do?

Each project has a maximum number of development environments.

  1. Delete development environments you no longer need, in the dashboard or with lagoon delete environment -p <project> -e <environment>.
  2. Deploy again, for example lagoon deploy branch -p <project> -b <branch>.

If you regularly need more environments, ask support to review the project's limit. This may affect your plan.

My deployment or build failed. What should I do first?

Start with the build log rather than guessing from the symptom. In the dashboard, open the environment, go to Deployments and select the failed build. Look for the first real error. Common causes:

  • dependency authentication (private Composer or npm repositories, tokens, auth.json, COMPOSER_AUTH)
  • dependency resolution or version conflicts
  • network or registry timeouts
  • out-of-memory errors or time limits
  • a missing Lagoon variable
  • a failing post-rollout task (for example drush)

Try to reproduce it locally with the same branch. If you still can't find the cause, contact support with the link to the failed build.

See also: rollouts

A variables-only deployment fails (for example 'Unauthorized' or pods stuck in PodInitializing). Why?

A variables-only deployment reuses the images from the last full build of that environment. It fails if the environment has never had a successful full deployment, or if its last build failed.

Run a full deployment first with lagoon deploy latest -p <project> -e <environment> or a git push. Variables-only deployments should work after that. If the full deployment also fails, contact support with the build log.

A local build fails with 'pull access denied ... repository does not exist or may require docker login'. How do I fix it?

Your build tried to use an image that hasn't been built yet. In most Lagoon projects other services (for example nginx and php) inherit from the cli image, and BuildKit builds in parallel.

  1. Use Docker Compose v2.13.0 or later, which resolves this.
  2. If you already do, check that docker buildx ls shows the default docker builder, not a docker-container based one.
  3. As a workaround, build the cli image first: docker compose build cli, then docker compose build and docker compose up -d.

If the image really comes from a private registry, run docker login for it.

See also: docker compose yml

Domains, DNS and routes#

We get alerts that routes are 'missing DNS', or some domains no longer point at amazee.io. What should we do?

A route in .lagoon.yml points to a hostname whose DNS doesn't resolve to amazee.io. Let's Encrypt can't issue or renew its certificate, so it keeps raising certificate and monitoring alerts.

  • Domain still in use: point its DNS at amazee.io as described on the DNS page (a CNAME, or A/AAAA records for apex domains). Check with dig +short <domain>.
  • Domain retired: remove it from the routes section of .lagoon.yml (or from routes set through the API or dashboard) and redeploy the environment. If the domain is also on the CDN service, ask support to remove it there.

If you change autogenerated routes, use the documented routes: autogenerate: keys, because misspelled keys are ignored. If you can't tell which environment a hostname belongs to, or a route remains after a redeploy, contact support with the full list of hostnames.

See also: dns

How do I change a site's domain without TLS errors, and remove the old route afterwards?
  1. Add the new hostname to the route in .lagoon.yml with tls-acme: false and deploy, so the route exists before DNS moves.
  2. Point DNS for the new hostname to amazee.io.
  3. Once DNS resolves, set tls-acme: true and redeploy so Let's Encrypt can issue the certificate via HTTP-01. Check with curl -vI https://<new-domain>.
  4. Add redirects from the old domain if needed, for example with an nginx redirects map.
  5. Remove the old domain from .lagoon.yml. Removed routes are cleaned up only when the build variable LAGOON_FEATURE_FLAG_CLEANUP_REMOVED_LAGOON_ROUTES=enabled is set for the deployment, for example lagoon deploy latest -p <project> -e <environment> --buildvar LAGOON_FEATURE_FLAG_CLEANUP_REMOVED_LAGOON_ROUTES=enabled.

Check the build log to confirm the route was removed. If it remains, or the environment was already deleted, contact support with the hostnames and build link.

See also: lagoon yml

Which IP addresses do outgoing requests from my site come from, and can I allowlist them?

Outbound traffic from each cluster leaves through a small set of egress IPs shared by all workloads in the region. The per-region list is on the Egress IP page, and changes are announced on status.amazee.io whenever possible.

You can allowlist these IPs on third-party APIs or firewalls, but also use authentication, because other customers share the same IPs. To restrict incoming access to your environments (for example to your CI runners), contact support with the IP ranges and environments.

See also: egress ip

My site shows DNS_PROBE_FINISHED_NXDOMAIN. Is the platform down?

No. NXDOMAIN means the domain name doesn't resolve at all, which is a DNS issue at your DNS provider or registrar rather than the hosting.

  1. Check with dig <domain> and dig NS <domain>.
  2. Make sure the domain hasn't expired and the nameservers are correct.
  3. Make sure the CNAME or A/AAAA records point to amazee.io as described on the DNS page.

The environment's autogenerated URL should still work, which confirms the site itself is up.

See also: dns

How do I disable the autogenerated URLs on my environments?

In .lagoon.yml, set:

YAML
routes:
  autogenerate:
    enabled: false

To keep autogenerated routes for pull requests, add allowPullrequests: true. To re-enable them for a single environment, set autogenerateRoutes: true under environments.<name>. Commit and deploy.

Configure your custom routes first so the site stays reachable. Autogenerated URLs include the project and environment names, so don't put sensitive information in those names.

See also: lagoon yml

A domain disappeared from my environment after a deploy, and I use wildcard routes. How do I fix it?

Check whether .lagoon.yml defines the same domain twice, once plain and once with wildcard: true (and wildcardApex: false). That combination behaves inconsistently and can drop the domain. Define each domain once with wildcard: true; Lagoon then adds the apex for you:

YAML
routes:
  - nginx:
      - "example.com":
          wildcard: true
          tls-acme: false

Wildcard routes need a custom or CDN-provided certificate, because Let's Encrypt HTTP-01 can't issue wildcard certificates. Redeploy the affected environment. Only use wildcardApex: false if the apex is served by a separate route.

See also: lagoon yml

CDN and caching#

How do I control CDN caching for one path, or add redirects, custom VCL or a custom error page?
  • Per-path caching: Fastly honours the caching headers your application sends. For content that changes (for example a per-site favicon), send short lifetimes on that path, such as Surrogate-Control: max-age=60 for the CDN and Cache-Control: no-cache or a short max-age for browsers, or add a Surrogate-Key tag and purge it on change. Check what the CDN sees with curl -sI -H 'Fastly-Debug:1' https://www.example.com/favicon.ico.
  • Redirects: handle them in your application or nginx config (for example a redirects map). Use CDN-level redirects only when the origin can't handle the request.
  • Custom VCL, maintenance or error pages: these are made by amazee.io on request. Open a ticket with the exact change (for an error page, the HTML), the domains and when it should go live or be removed.

Projects in the same organization often share one Fastly service; each site can still be purged by its domain cache tag.

See also: cdn

How do I purge the Fastly cache for my site, and how do we get access?

Purges require a Fastly API token. If you don't have one, ask support for a token or Fastly user access, giving the people, service and whether you need purge-only access.

  • Purge one URL: curl -XPURGE -H 'Fastly-Soft-Purge:1' -H "Fastly-Key:$FASTLY_API_TOKEN" https://www.example.com/page
  • Purge by cache tag (surrogate key) through the Fastly API. amazee.io tags every response with its domain name, so you can purge one site without affecting others on the same service.

Prefer soft purges: stale objects can still be served if the origin is unhealthy. Purging by wildcard path isn't supported, so group content with cache tags. The Drupal Fastly and Purge modules purge cache tags automatically when content changes. Avoid purging the whole service, because several sites can share it.

See also: cdn

The CDN strips a cookie my application needs (for example a session or SSO cookie). How do I keep it?

The amazee.io CDN strips cookies it considers unnecessary and keeps session cookies. A cookie is kept if its name matches one of these patterns: wordpress_[a-z0-9_]+, [a-z0-9_]+_session, S?SESS[a-z0-9]+ or [A-Z]+_UUID (plus exceptions for some frameworks).

Check the exact name: app-session (hyphen) doesn't match [a-z0-9_]+_session. Rename the cookie to match, or open a ticket with the cookie name and path to have it added to the keep-list.

Any cookie that isn't stripped forces a cache miss, so set it only where needed, for example after login.

See also: cdn

Adding '?format=auto' or width parameters to image URLs does nothing on Fastly. Why?

Fastly Image Optimization is a separately licensed add-on and is not enabled by default. Without it, Fastly ignores image transformation parameters and serves the original file; WebP or AVIF conversion is not automatic.

To use it, contact support to discuss enabling it. Until then, generate responsive or WebP images in your application, for example with Drupal image styles.

See also: cdn

Databases#

A database dump, import or sync between environments fails. How do I fix the common errors?
  • Access denied; PROCESS privilege: add --no-tablespaces to mysqldump or mariadb-dump, or use drush sql-dump --extra-dump=--no-tablespaces.
  • max_allowed_packet, or Error 2013 (Lost connection during query): usually a row larger than the client's max_allowed_packet (64M by default in the CLI image). Set MARIADB_MAX_ALLOWED_PACKET (for example 256M) as a runtime variable and redeploy, and look for oversized cache rows you can clear.
  • MySQL 8 error 3105 (generated columns): dump with a client matching the server (MySQL mysqldump for MySQL 8), or use lagoon-sync.
  • "no alias record could be found": run drush sa in the cli container for real alias names (/ in branch names becomes -). Keep amazeeio/drupal_integrations installed and up to date, then run drush sql-sync @lagoon.<project>-<source> @self.
  • lagoon-sync "connection closed": update lagoon-sync and check lagoon ssh works for both environments.
  • Collation, DEFINER or sandbox errors: see the MySQL issues page.

If a deploy task fails on the dump step, include the build log link when you contact support.

See also: mysql

Monitoring, support and account#

What does amazee.io monitor for my site, can we get the alerts, and what if we put the site in maintenance mode?

amazee.io monitors your production routes (uptime and certificate expiry), but the alerts go to amazee.io support, not to you, and you can't configure your own destinations or thresholds. For your own alerting, use an external uptime service or an APM tool.

  • To monitor a specific path instead of the homepage, set monitoring-path on the route in .lagoon.yml, and check with support that it's used.
  • Before you deliberately take a site offline or enable maintenance mode, tell support the site and time window, or monitoring flags it and support may contact you.
  • For a long-term maintenance page, ask support to change the monitored URL.

See also: apm

Where can I find amazee.io security and compliance information for a security questionnaire?

amazee.io holds an ISO 27001 certificate, a SOC 2 Type 2 attestation and a CSA STAR Level 2 attestation. Most typical questions are answered by the documents in the amazee.io Trust Portal, so check those first.

If something is missing, open a support ticket with the specific questions. Minimum turnaround is 2 weeks for yes/no questionnaires and 4 weeks for open-ended ones. Answers about third-party services such as the WAF depend on the vendor and your configuration.

See also: questionnaires

Emails from my site are delayed or not arriving. Can amazee.io guarantee delivery?

No. The built-in outgoing email service is best-effort, with no delivery guarantee.

  1. Check your application logs to confirm the email was handed off.
  2. If it was, the delay is usually at the receiving mail server (greylisting or spam filtering).
  3. Make sure your sending domain has the SPF record from the email docs.

For critical or high-volume email, use a transactional email provider (SMTP or API) from your application. Support can look for a specific message if you send the Message-ID, time and recipient domain.

See also: sending email

How do I add a custom task to my project when the Lagoon CLI has no command for it?

The Lagoon CLI has no dedicated command for adding a custom task. Use lagoon raw, which sends any GraphQL query or mutation to the Lagoon API, to call the advanced task definition mutation shown in the Lagoon custom tasks documentation. You need the right project or organization permissions to create task definitions.

See also: custom tasks

How do I delete Lagoon projects I no longer need?

Organization owners can delete projects in their organization. If your role doesn't allow it, open a support ticket listing the exact project names and their organization; support confirms the list with you and deletes them.

Deleting a project removes all its environments, databases and files, so take the backups you need first (for example drush sql-dump and a files download). If the projects are covered by a contract or billing arrangement, mention it so it can be adjusted.

See also: rbac

Logs#

The logging dashboard shows 'Could not locate that index-pattern-field (id: @timestamp)', or I can't find my project's index. How do I fix it?
  1. Switch to the correct tenant (profile menu > Switch tenants). Index patterns are per tenant, and you may need to do this every time you log in.
  2. Make sure you are in a Lagoon group that has the project assigned. A group only gets a tenant if it has projects.
  3. Use the project-specific index pattern (for example container-logs-<project>-*), not a global one.
  4. If the field list is stale (often after an upgrade), go to Dashboards Management > Index patterns, open the pattern, refresh its fields and reload Discover.

If the project is missing from the list, the index is empty, or refreshing doesn't help, contact support with the project and index name and a screenshot.

See also: logging

How far back can I search logs, and how do I get older logs for an incident investigation?

Router, application and container logs are in the logging dashboard (OpenSearch). The current month and the previous month are searchable until about the last week of the current month, when the previous month's logs are deleted after a snapshot is taken. Snapshots are kept for 90 days.

You can filter router logs by host, request_uri, status and http_user_agent (useful for finding bots) and export the results. For older logs, open a ticket with the projects, date range and fields you need, and support restores the snapshot.

See also: logging

Access, SSH and users#

SSH fails with 'no access to -' even though my key works. Why?

Your SSH key is valid, but your Lagoon user has no role that grants SSH on that environment. Access comes from groups: a role applies only to projects assigned to that group.

  • The project may not be assigned to your group. An organization owner can associate the project with the group in the dashboard.
  • Your role may be too low. The Developer role has SSH access to development environments only; production needs Maintainer or Owner.

Run lagoon whoami to confirm which user your key maps to, then retry lagoon ssh -p <project> -e <environment>. Contact support if the group setup looks correct.

See also: rbac

A team member lost their two-factor authentication device, or I need to check roles before adding a developer. What should I do?
  • Lost 2FA device: open a support ticket from an existing authorised account, naming the affected user. Support can reset the second factor so they can enrol again.
  • Roles: an organization owner can see members and their roles in the dashboard (Organizations) and add a user to the right group with the needed role, for example Developer or Maintainer.

If you aren't sure who owns your organization, or can't see it, contact support.

See also: rbac

SSH to my environment times out when I use a hand-written ssh command (for example port 32222). Which host and port should I use?

Don't hand-craft ssh commands. Use the Lagoon CLI, which picks the right endpoint and port for your environment:

Text Only
lagoon ssh -p <project> -e <environment>

For scp, rsync or other tools that need the raw details, get the exact ssh command with lagoon ssh -p <project> -e <environment> --conn-string. Port forwarding is disabled.

See also: ssh

How long are Lagoon API tokens valid, and can I revoke one?

Lagoon-issued tokens are short-lived (about 10 hours; the expiry is in the token's exp claim) and currently can't be revoked, so a leaked token stays valid until it expires.

Instead of generating tokens manually, configure the Lagoon CLI once (lagoon config add ..., then lagoon config default) and run lagoon login; it uses your SSH key to request tokens for you. If you suspect your SSH key was compromised, remove it from your account in the dashboard and contact support.

See also: lagoon cli

Backups#

How do backups work, and how do I restore data (including after deleting an environment or when backups are missing)?

Lagoon backs up each database and persistent volume nightly. Daily backups are kept for 7 days and weekly backups for 6 weeks. You'll find them on the environment's Backups tab, or with lagoon list backups.

To restore:

  1. Click Retrieve on the backup (or lagoon retrieve), wait until it's ready, then Download it.
  2. Restoring isn't automatic: import the database dump or copy the files into the target environment yourself, for example over SSH or rsync.

If you deleted an environment, redeploy the branch to recreate it, then restore from its backups. Variables set only on the deleted environment must be added again.

If no backups are listed, the newest is several days old, or a retrieval stays pending, contact support with the project and environments. Meanwhile take a manual dump, for example lagoon ssh -p <project> -e <environment> -C 'drush sql-dump --gzip' > backup.sql.gz.

See also: backups

Storage and files#

A persistent volume (for example a database or Solr volume) is nearly full. How do we increase it?

Block storage volumes don't grow automatically and can only be increased, never shrunk. Their size comes from the lagoon.persistent.size label in docker-compose.yml.

  1. Check usage with df -h in the container and clean up data you don't need (old logs, caches, temporary files).
  2. Increase the label, for example lagoon.persistent.size: 20Gi, commit and redeploy. Request only what you need.
  3. For managed databases, or if the volume doesn't resize after a deploy, contact support with the project, environment, service and size.

The amazee.io team is usually notified as volumes approach their limit and will reach out. Bulk (NFS-type) file volumes don't need resizing, but you're billed for what you store.

See also: storage

The 'copy files between environments [drush rsync]' task warns it will overwrite data. Will it overwrite my database?

No. The task only syncs the public files directory (sites/default/files) from the source environment to the target. The warning is the standard drush rsync confirmation about files in the destination directory.

To copy the database, use the separate database sync task, or run drush sql-sync @source @self from the target environment's cli container. Back up first if the target's files matter.

See also: drush 9

I need to delete a persistent volume (for example to move a service to a newer storage class). Can I do it myself?

No. You can't delete persistent volumes directly. Open a support ticket with the project, environment, service and volume name; support deletes the volume and its deployment. Then redeploy so the volume is recreated, and restore data or rebuild indexes as needed.

If you only want to remove or change a service, try the one-off cleanup deployment described in the "remove a service" question first. Block volumes can be made bigger but never smaller.

See also: storage