Rate Limiting

Overview

This is useful for keeping one user from starving everyone else, keeping the total traffic to an expensive layer within what you’re willing to serve, and enforcing usage quotas on a tenant. Rate limiting is disabled by default.

Limits are checked once the request is authenticated, its layer is found, and it passes authorization and zoom checks, but before the cache is consulted. This means:

Cache hits are counted
Only the requested layer is checked. A layer rendered indirectly through the ref provider isn’t checked or counted separately
Requests generated by the seed command, test command, or health checks are never limited
TileJSON documents and the preview page itself are never limited.

This is not DDoS protection. Every rejected request still reaches tilegroxy and costs a lookup against the store. Use a CDN or WAF for that.

Policies

Rate limiting is configured in a top-level ratelimit block. Its name picks the store that holds the counters, and the redis and postgresql stores also need a datastore. Its policies parameter is a list of named sets of limits.

Each policy has an id and a list of rules, described below. Every layer is limited by exactly one policy, which it picks with its own ratelimit parameter. Layers that don’t pick one use the policy named by the top-level defaultratelimit, or the first policy if that isn’t set. Setting skipratelimit to true on a layer exempts it entirely.

A policy can also extend other policies by listing their IDs, which enforces their rules as well as its own. The rules aren’t copied: a layer using the extending policy shares counters with every layer using the policy it extends.

The following gives every user a burst of 50 tiles per second on every layer, and limits each tenant to 100,000 satellite tiles a month:

YAML
JSON

ratelimit:
  name: redis
  datastore: redis-0
  policies:
    - id: standard
      rules:
        - segment:
            - user
          limit: 50
          period: 1
    - id: premium
      extend:
        - standard
      rules:
        - segment:
            - tenant
          limit: 100000
          calendar: month
defaultratelimit: standard (1)
layers:
  - id: basemap (2)
    provider: ...
  - id: satellite
    ratelimit: premium (3)
    provider: ...
  - id: internal_debug
    skipratelimit: true (4)
    provider: ...

1	Limits every layer that doesn’t pick another policy.
2	`basemap` doesn’t pick a policy, so it gets `standard`.
3	`satellite` gets the monthly quota from `premium` and the burst limit from `standard`. The burst counter is the same one `basemap` uses, so a user can’t double their burst by spreading requests across both layers.
4	Never limited.

{
  "ratelimit": {
    "name": "redis",
    "datastore": "redis-0",
    "policies": [
      {
        "id": "standard",
        "rules": [
          {
            "segment": [
              "user"
            ],
            "limit": 50,
            "period": 1
          }
        ]
      },
      {
        "id": "premium",
        "extend": [
          "standard"
        ],
        "rules": [
          {
            "segment": [
              "tenant"
            ],
            "limit": 100000,
            "calendar": "month"
          }
        ]
      }
    ]
  },
  "defaultratelimit": "standard", (1)
  "layers": [
    {
      "id": "basemap", (2)
      "provider": { ... }
    },
    {
      "id": "satellite",
      "ratelimit": "premium", (3)
      "provider": { ... }
    },
    {
      "id": "internal_debug",
      "skipratelimit": true, (4)
      "provider": { ... }
    }
  ]
}

1	Limits every layer that doesn’t pick another policy.
2	`basemap` doesn’t pick a policy, so it gets `standard`.
3	`satellite` gets the monthly quota from `premium` and the burst limit from `standard`. The burst counter is the same one `basemap` uses, so a user can’t double their burst by spreading requests across both layers.
4	Never limited.

A policy may have no rules. Make an empty policy the default if only a few layers should be limited.

Policies that extend each other in a loop are rejected at startup. A policy reached through more than one extend is only checked once.

If a layer has its own policy, any layer that reaches it through ref or blend providers need that policy too.

Rules

Each rule allows limit requests per period. A request is rejected if any rule of its layer’s policy, including the policies it extends, is exceeded. Rejected requests aren’t counted against any rule, so a client that keeps retrying doesn’t use up its other limits.

Each counter is updated atomically, but checking a request’s rules and counting it are separate steps. Under heavy concurrency a few more requests than configured can get through.

Periods

A rule’s period is set with exactly one of period, for a rolling window, or calendar, for a window that follows the calendar.

period is the length of the window, in seconds. Counts are kept in fixed buckets of that length, and a request is checked against the current bucket plus the previous bucket weighted by how much of it is still within the window. For example, 15 seconds into a 60 second bucket, the estimate is the current count plus 45/60 of the previous one. This stops a client doubling its allowance across a bucket boundary.

calendar takes one of hour, day, week, month, or year, and the count resets at the start of each one in UTC. Weeks start on Monday. Use this for quotas that line up with a billing cycle. Other time zones and cycles anchored to a signup date aren’t supported.

Segments

A rule’s segment parameter decides how usage counters are segmented. For instance a rule of 100 requests per hour segmented by user and layer will allow each user to make 100 requests per layer per hour. Meanwhile, a rule applied to two layers that’s segmented by just user means the user has a shared pool of 100 requests over both layers.

Values for `segment`
Value	Description
`user`	The user ID established during authentication. If user IDs are only unique within a tenant, use `[tenant, user]`
`tenant`	The tenant ID established during authentication. See Serve Multiple Tenants
`ip`	The client IP address. IPv6 addresses sharing their first 64 bits count as one client, which `ipv6Prefix` can change.
`layer`	The ID of the requested layer, not the name from the URL. Every name matching a pattern layer shares one counter

Overrides

A rule’s overrides parameter gives some requests a different limit under the same rule, such as a higher quota for a large customer or an exemption for an internal service. Each override has its own limit and matches by user ID, tenant ID, IP range, or anonymous. An override with several conditions matches if any one of them does. The full list is in the reference.

Overrides are checked in order and the first match wins. Requests matching none get the rule’s own limit. Overrides only change the limit, not what’s counted.

YAML
JSON

ratelimit:
  name: redis
  datastore: redis-0
  policies:
    - id: default
      rules:
        - segment:
            - ip
          limit: 100
          period: 1
          overrides:
            - ips:
                - 10.0.0.0/8
              limit: -1 (1)
        - segment:
            - tenant
          limit: 100000
          calendar: month
          overrides:
            - tenants:
                - acme
                - globex
              limit: 5000000 (2)
            - tenants:
                - delinquent-co
              limit: 0 (3)
            - anonymous: true (4)
              limit: 1000

1	Requests from the internal network are never limited or counted by this rule.
2	These tenants get 50 times the default monthly quota.
3	A limit of 0 rejects every request, cutting this tenant off without touching authentication.
4	Requests without a tenant all share one counter, with a small quota of their own. `anonymous` only applies to rules segmented by user or tenant.

{
  "ratelimit": {
    "name": "redis",
    "datastore": "redis-0",
    "policies": [
      {
        "id": "default",
        "rules": [
          {
            "segment": [
              "ip"
            ],
            "limit": 100,
            "period": 1,
            "overrides": [
              {
                "ips": [
                  "10.0.0.0/8"
                ],
                "limit": -1 (1)
              }
            ]
          },
          {
            "segment": [
              "tenant"
            ],
            "limit": 100000,
            "calendar": "month",
            "overrides": [
              {
                "tenants": [
                  "acme",
                  "globex"
                ],
                "limit": 5000000 (2)
              },
              {
                "tenants": [
                  "delinquent-co"
                ],
                "limit": 0 (3)
              },
              {
                "anonymous": true, (4)
                "limit": 1000
              }
            ]
          }
        ]
      }
    ]
  }
}

1	Requests from the internal network are never limited or counted by this rule.
2	These tenants get 50 times the default monthly quota.
3	A limit of 0 rejects every request, cutting this tenant off without touching authentication.
4	Requests without a tenant all share one counter, with a small quota of their own. `anonymous` only applies to rules segmented by user or tenant.

Responses

A rejected request returns 429 Too Many Requests with a Retry-After header giving the number of seconds until the exceeded rule allows another request. Retry-After is left out when the rule’s limit is 0 or when headers parameter is none (disabled). The response body follows the error mode, using the RateLimited image and message.

The AlwaysOk error mode replaces the 429 with a 200, which most clients won’t recognize as a rejection.

Headers

By default, every response to a request that reached the check carries headers describing the rules that applied to it. The headers parameter of the ratelimit block controls this and can be one of:

standard: The RateLimit-Policy and RateLimit headers from version 11 of the IETF RateLimit header fields draft. This is the default.
min: No headers apart from Retry-After on a rejection. Use this if you’d rather not tell users what their limits are.
none: No headers at all, not even Retry-After.

Standard

A rule is identified by its policy ID, segment, and period joined by dots, such as standard.user.1s or premium.tenant.month. Multiple segments are joined with + and an empty segment is all. The same names are used in audit events and the ratelimit commands.

From the earlier satellite example, a user whose tenant has made 11,201 requests this month sees:

RateLimit-Policy: "standard.user.1s";q=50;w=1, "premium.tenant.month";q=100000;w=2678400
RateLimit: "standard.user.1s";r=49;t=1, "premium.tenant.month";r=88799;t=1209600

A rule with an unlimited override is left out of the headers for that request.

Auditing

The following let you see who is being limited and how much of a quota has been used:

Audit log: A rate_limited event naming the policy and rule is written to the audit log. Each instance writes at most one per counter per period, or per minute if the period is shorter, so a client hammering away doesn’t flood the log.
Metrics: When telemetry is enabled, rejections are counted per policy and in total, alongside failures of the store.
Traces: The check appears as a ratelimit span under the request.
Command: tilegroxy ratelimit status shows whether a user, tenant, or IP address is being limited and how much of each rule they’ve used. tilegroxy ratelimit reset sets a counter back to zero and tilegroxy ratelimit list finds rules by name, policy, layer, or segment.

Counters expire along with their period and can only be looked up for a given identity, not listed, so use analytics to find your heaviest users and for billing.

Alert on tilegroxy.ratelimit.error. By default a store outage means nothing is being limited, while tiles keep being served normally.

Deployment

High availability

Use the redis or postgresql store with more than one instance. Otherwise each instance gives out the full limit.

Every tile request, including cache hits, costs a round trip to the store, so keep it close to tilegroxy. When the store fails or exceeds timeout, onError decides whether the request is served or rejected. See the reference.

With several regions, either give each region its own store and divide limits by the number of regions, or share one store and accept the latency.

Behind a CDN, load balancer, or reverse proxy with a public address, add it to server.forwarded.trusted. Otherwise the client IP is the proxy’s and every client shares one counter on rules segmented by ip. Proxies on a private network are trusted by default.

Reloading

Rate limiting can be changed with a reload. Each rule’s counters belong to the policy that defines it, so they’re kept as long as that policy’s id and the rule’s segment and period stay the same. Changing a limit, an override, or which policy a layer uses takes effect immediately without losing counts. Changing a policy id, a rule’s segment or period, moving a rule to another policy, or changing the store’s own settings starts the affected counters over from zero.

During a rolling deployment that changes them, old and new instances count separately until the rollout finishes, so clients briefly get both allowances.

Security

A limit is only as good as the identity it counts by. Without authentication every request is anonymous, so a rule segmented by user or tenant is a configuration error when authentication is none.

User IDs, tenant IDs, and IP addresses are hashed before being used as store keys. The hash isn’t meant to hide them. See Security for more.