Reloading :: tilegroxy Docs

Reload triggers

Reloading can occur in a few different situations:

If --hot-reload is enabled via the CLI, any changes to configuration will be detected and the system will be reloaded with the new configuration. Hot reloading occurs automatically within a few seconds of the configuration changing on the filesystem.
If a Secret configuration includes the TTL parameter, the system will be automatically reloaded without any changes to configuration when the TTL expires.
If a SIGHUP signal is received, the configuration will be read and the system reloaded with the changes. See Reload signal.

Configuration Changes

When reloading with a new/modified configuration, the new configuration is stored side-by-side with the previous configuration and swapped atomically. The exact timing of that change cannot be controlled but should occur within several seconds of configuration being read depending on the size of your configuration. Requests that are already in-flight while this change occurs will continue operating against the old configuration while requests coming in after the swap completes will use the new configuration.

Reloads never run at the same time, whether they come from a configuration change, a reload signal, or a secret. They are queued up to run serially. A reload requested while tilegroxy is still starting up is held until the server is ready and then applied. If several arrive during startup only the most recent is applied.

Warnings

Configuration changes that affect the core functionality of the service (for example changing the port of the HTTP server) still require a restart of the executable.

When reloading occurs, new connections will be established for configured datastores and caches. The previous connections are closed once in-flight requests using them have had time to complete, therefore the number of connections returns to its steady state instead of growing with each reload. Expect a temporary period after each reload where both the old and new connections are open.

Buffered analytics events are flushed before the previous configuration is released so a reload doesn’t lose the current batch.

Tilegroxy uses fsnotify to detect when configuration has changed. It is possible for the file change event to reach tilegroxy before the file has finished being written. Tilegroxy waits 1 second before applying changes to avoid this but if you are using a filesystem with especially slow writes it is possible tilegroxy will attempt to read the configuration too early. This will usually result in an invalid configuration which will be ignored but could result in operating against incorrect configuration.

Avoid using hot reload if you’re at risk of slow writes, especially if using YAML configurations.

Automatic configuration reloading will not work with remote providers (such as etcd) or certain file systems that do not support file notifications (such as NFS, SMB, or FUSE). Use a reload signal in those cases.

Supported Configuration

The following sections of the configuration can be reloaded:

Every other section, including server, is fixed when the process starts. Changes to unsupported sections will be ignored. To check whether a pending configuration change can be reloaded, use config diff.

Secrets

Secret has a watch option that, when set, can trigger a reload using the same mechanism described here. One key exception is that when a secret changes, the configuration itself will not change, even if the configuration has changed on disk or remotely. If you are using both secret reloading and configuration reloading, you may want to consider the timing of your changes to avoid excessive reloads.

Reload signal

Sending tilegroxy the SIGHUP signal re-reads the configuration from its original source and reloads it the same way a file change does. You may want to consider this method, rather than hot-reload, if you’ll be manually modifying configuration, to avoid reloading partially completed changes.

Unlike --hot-reload, this works with configuration pulled from a remote provider and on file systems without change notifications. Obviously the configuration will never change on reload if you use --raw-config.

Reload signals are only supported on Linux, macOS, and other Unix-like systems.

Shutdown

For cases where you need to terminate the tilegroxy instance (when changing a non-reloadable section of config, or cycling pods), tilegroxy supports graceful shutdown to ensure requests aren’t lost when properly orchestrated.

Shutdown behaviour

When tilegroxy receives SIGINT or SIGTERM it shuts down in a fixed order rather than exiting immediately. This gives batched analytics a chance to flush and connection pools a chance to close.

The phases run in this order:

Health readiness starts returning 503 while liveness keeps returning 204
Wait for DrainDelay
The HTTP server finishes the requests already in flight
The health server and its background checks stop
Entities close, which flushes analytics and releases caches, datastores, providers and the secret store
OpenTelemetry shuts down
Log file handles close

Every phase draws from a single deadline of ShutdownTimeout seconds. If the deadline passes or a second signal is sent, the remaining phases are skipped and an error is logged.

When ShutdownTimeout isn’t specified it defaults to Server.Timeout plus DrainDelay, which covers the two phases that consume it: the wait before draining and one full-length request finishing during the drain. A value at or below DrainDelay is rejected as it leaves no time to drain.

A value below Server.Timeout is allowed but means a slow request can outlive the budget.

Coordinating with your orchestrator

Your orchestrator’s grace period must be longer than the shutdown sequence it’s waiting on, otherwise the process is killed partway through and any analytics being flushed are lost. Three values need to stay in order:

preStop sleep (or DrainDelay)  <  ShutdownTimeout  <  termination grace period

Be careful to preserve this order when adjusting these values. Raising ShutdownTimeout without also raising the grace period means the process might be killed prematurely.

In Docker Compose the grace period is stop_grace_period which defaults to 10 seconds and therefore needs raising for the default budget. In Kubernetes it’s terminationGracePeriodSeconds. The Helm chart included with tilegroxy sets both that and a preStop hook for you.

A preStop hook runs to completion before the signal is sent, so its sleep and DrainDelay are additive. Use one or the other, not both. The Helm chart handles this by setting DrainDelay to 0 whenever its preStop hook is enabled.