Skip to main content
This topic explains the changes in the Relay Proxy 9.0 release and how to migrate to that version. Version 9.0 includes breaking changes. The most significant are:
  • The Relay Proxy exports metrics, traces, and logs only through the OpenTelemetry Protocol (OTLP). Version 9.0 removes the Datadog, Stackdriver, and Prometheus integrations, and every metric and attribute name has changed. To learn more, read Metrics and tracing.
  • The log output format has changed. To learn more, read Logging.
  • Version 9.0 removes payload filtering. To learn more, read Payload filtering.
  • Version 9.0 removes several configuration options. A configuration file that still contains them prevents the Relay Proxy from starting. To learn more, read Configuration changes.
  • Docker images are distroless only, and are based on Debian 13 instead of Debian 12. The Alpine and i386 images and the -static-debian12- tags are no longer published. To learn more, read Docker images.
  • The Relay Proxy connects to LaunchDarkly using the protocol behind data saving mode, which requires network access to an additional LaunchDarkly host. To learn more, read Connection to LaunchDarkly.
Version 9.0 is also the Relay Proxy version that supports SDKs configured for data saving mode. Your SDKs do not need to change. SDKs that work with Relay Proxy 8.x continue to work with Relay Proxy 9.0.

Prerequisites

Before you migrate, we recommend that you:
  • Upgrade to the latest 8.x release first. Several features that 9.0 relies on, including the Redis timeout options, concurrent SDK keys, and the auto-configuration cache, were introduced during the 8.x release line. If you start from the latest 8.x release, the changes in this topic are the only changes you need to make. To find the latest release, visit Relay Proxy releases on GitHub.
  • Test the upgrade in a non-production environment.
  • Review the Relay Proxy 9.0.0 release notes.

Migration checklist

This checklist summarizes the changes you may need to make. Each item links to the section that explains it. To prepare a Relay Proxy 8.x deployment for 9.0:
  1. Remove the [Datadog], [Stackdriver], [Prometheus], and [Filters] sections, and the projKey, inlineUsers, exitOnError, and ignoreConnectionErrors options, from your configuration file. Read Removed configuration options.
  2. Remove the matching environment variables, such as USE_PROMETHEUS, LD_FILTER_KEYS_*, and EXIT_ON_ERROR. Read Removed configuration options.
  3. Configure OpenTelemetry export if you collect metrics, and rebuild your charts and alerts using the new metric names. Read Metrics and tracing.
  4. Update log parsers, alerts, and saved searches for the new log format. Read Logging.
  5. Allow outbound connections from the Relay Proxy to sdk.launchdarkly.com. Read Connection to LaunchDarkly.
  6. Switch any -alpine, -i386, or -static-debian12- Docker image tag, replace shell-based health checks, and check file permissions for the new user ID. Read Docker images.
  7. Replace any use of the expiringSdkKey field in tools that parse the status resource. Read Status resource.
  8. Review the new request body limit if your client-side SDKs send REPORT requests with very large contexts. Read Changed defaults.
  9. Update your imports and logger if you embed the Relay Proxy as a Go library. Read Go library.

Configuration changes

Version 9.0 removes, adds, and changes several configuration options.

How 9.0 handles removed options

The Relay Proxy treats a removed option differently depending on where you set it:
  • Configuration file: The Relay Proxy refuses to start. It logs an error such as unsupported or misspelled section "Prometheus" or unsupported or misspelled section "Environment", variable "projKey" and exits with status code 1.
  • Environment variable: For most removed variables, the Relay Proxy ignores the variable without logging a warning. For example, if you set USE_PROMETHEUS=true, the Relay Proxy starts normally but does not export Prometheus metrics. The exceptions are EXIT_ON_ERROR and IGNORE_CONNECTION_ERRORS. If you set either of them, the Relay Proxy logs this variable is no longer supported and refuses to start.
Check environment variables as well as configuration filesBecause the Relay Proxy ignores most removed environment variables, a deployment configured only through environment variables can start successfully and stop exporting metrics without any error. Review your environment variables before you upgrade.

Removed configuration options

Here are the options that 9.0 removes:

New configuration options

Version 9.0 adds the following configuration sections and options. The [OpenTelemetry] section controls metric, trace, and log export. It replaces the removed metrics sections: The Relay Proxy does not implement http/json. If you set http/json, the Relay Proxy logs a warning and sends http/protobuf instead. Your endpoint continues to work, because every OTLP/HTTP receiver accepts http/protobuf.
The Relay Proxy refuses to start with an unrecognized protocol valueIf OTLP export is on and OTEL_EXPORTER_OTLP_PROTOCOL is set to a value other than grpc, http/protobuf, or http/json, the Relay Proxy refuses to start. This includes http, which the 9.0 release candidates accepted. If you tested a release candidate with http, change it to http/protobuf. Check for this variable if your hosts set it globally for other services.
Other OpenTelemetry settings, such as OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, and OTEL_METRIC_EXPORT_INTERVAL, use the standard OpenTelemetry environment variables. They have no configuration file equivalent. The [Concurrency] section limits how many full data payloads the Relay Proxy sends to SDKs at the same time. This protects the Relay Proxy when many SDKs connect at once, for example after a deployment. The limit is off by default: The [Redis] section has three new options for mutual TLS: These options take effect only when TLS is on, either through REDIS_TLS or a rediss:// URL. The Relay Proxy reads the files at startup. Restart it after you rotate a certificate. To learn more about each option, read the Relay Proxy GitHub repository’s Configuration.

Changed defaults

The default for maxClientRequestBodySize (MAX_CLIENT_REQUEST_BODY_SIZE) changed from no limit to 5 MiB. This option limits the body size of REPORT evaluation requests from client-side SDKs. The Relay Proxy responds to a larger request with a 413 status code. You can raise the limit, for example to 10MiB, but you cannot turn it off.

Changed behavior

One-shot mode, which you turn on with exitAlways (EXIT_ALWAYS), changed in two ways:
  • The Relay Proxy waits for every environment to initialize, and exits with a non-zero status code if any environment failed. In 8.x, setting exitOnError made the Relay Proxy stop waiting after the first environment finished. If that environment succeeded, the Relay Proxy exited with status code 0, even if another environment failed.
  • initTimeout (INIT_TIMEOUT) must be greater than zero. If it is zero or negative, the Relay Proxy refuses to start.

Metrics and tracing

Version 9.0 replaces OpenCensus with OpenTelemetry. The Relay Proxy pushes metrics, traces, and logs over OTLP to any compatible receiver. It does not expose a metrics endpoint for scraping.

Send metrics to your monitoring system

To export telemetry, set USE_OTLP=true and point OTEL_EXPORTER_OTLP_ENDPOINT at an OTLP receiver. Here is an example:
Depending on the integration you used with 8.x, use one of these receivers:
  • Prometheus: Send metrics to the Prometheus OTLP receiver, available in Prometheus 2.47 and later with --web.enable-otlp-receiver, or to an OpenTelemetry Collector that exposes a Prometheus endpoint. Remove scrape jobs that target port 8031. If you send directly to Prometheus, set OTEL_TRACES_EXPORTER=none and OTEL_LOGS_EXPORTER=none, because Prometheus does not accept traces or logs. Pushed metrics have no up series. Prometheus cannot detect a stopped Relay Proxy the way it detected a failed scrape. Use the status resource or a metric staleness alert instead.
  • Datadog: Turn on OTLP ingestion in the Datadog Agent and send to it. Set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta. If you have not turned on OTLP log ingestion in the Agent, set OTEL_LOGS_EXPORTER=none. The DATADOG_TAG_* options are replaced by OTEL_RESOURCE_ATTRIBUTES.
  • Stackdriver (Google Cloud Monitoring): Send to an OpenTelemetry Collector configured with a Google Cloud exporter.
To learn more, read the Relay Proxy GitHub repository’s Metrics and Tracing.

Metric name changes

Version 9.0 renames every metric to follow the OpenTelemetry semantic conventions. It also removes the 8.x prefix options, and metric names in 9.0 are fixed. Here is how the 8.x metrics map to 9.0: If you send metrics to Prometheus, the 9.0 names appear as http_server_active_requests, launchdarkly_relay_requests_total, and http_server_request_duration_seconds.

Attribute changes

Metric attributes, which 8.x called tags, were also renamed. Values are no longer rewritten. In 8.x, every / in a value became _. In 9.0, values appear as received, and a missing value is not_provided instead of not-provided. Here is how the 8.x tags map to 9.0 attributes: Every metric also carries these resource attributes:
  • service.name, which defaults to ld-relay
  • service.instance.id, a unique ID for each Relay Proxy process
  • service.version
  • launchdarkly.relay.sdk.version
You can set your own with OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES.

New metrics

Version 9.0 adds metrics that have no 8.x equivalent:
  • Event forwarding: launchdarkly.relay.events.received.size, launchdarkly.relay.events.sent, launchdarkly.relay.events.sent.size, launchdarkly.relay.events.failed, launchdarkly.relay.events.dropped, and launchdarkly.relay.events.pending.
  • Status: The status resource is also available as metrics, such as launchdarkly.relay.status and launchdarkly.relay.environment.connection.state.count. Set OTEL_ENVIRONMENT_STATUS_METRICS=true to add a series for each environment. You can alert on these metrics instead of polling the status resource.
  • Initialization limits: launchdarkly.relay.init.* metrics, exported when you configure the [Concurrency] section.
  • Go runtime: Standard go.* metrics such as go.memory.used and go.goroutine.count.

Tracing

Version 9.0 exports distributed traces over OTLP. The Relay Proxy creates a span for each request, plus internal spans for authentication, data store reads, flag evaluation, and response writing. It accepts W3C traceparent headers from your applications. The Relay Proxy replaces the evaluation context in url.path span attributes with REDACTED. By default, the Relay Proxy records every trace. In 8.x, the default was 1 in 10,000 requests. To lower trace volume, set OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, or set OTEL_TRACES_EXPORTER=none. Here is an example that keeps 1% of traces:

Logging

The Relay Proxy now writes logs with Go’s structured logging package. The LOG_FORMAT and LOG_LEVEL options are unchanged, but the output format is different. Update any log parser, alert, or saved search that depends on the 8.x format. The JSON format has changed:
  • The timestamp field is now time, and the message field is now msg.
  • Timestamps use the host’s local time zone instead of always using UTC. Set TZ=UTC to keep UTC timestamps.
  • Details that 8.x wrote into the message are now separate fields. For example, the environment appears in an env field instead of an [env: ...] prefix in the message.
  • Many Relay Proxy messages have new wording.
Here is the same startup message in each version:
The text format is now key=value pairs. Here is an example:
Error messages still go to standard error, and all other messages go to standard output. When OTLP export is on, the Relay Proxy also sends its logs over OTLP. If you already collect the Relay Proxy’s standard output, or your receiver does not accept logs, set OTEL_LOGS_EXPORTER=none.

Connection to LaunchDarkly

Version 9.0 connects to LaunchDarkly using the same protocol as SDKs in data saving mode. Each environment first requests its data with a polling request, then opens a streaming connection that receives changes. If the streaming connection fails, the Relay Proxy falls back to polling. After a reconnect, the Relay Proxy receives only the changes it missed instead of the full data set. The LaunchDarkly hostnames have not changed, but 9.0 uses one more of them. Relay Proxy 8.x connects to sdk.launchdarkly.com only when big segments are on. Version 9.0 always connects to it.
Allow access to sdk.launchdarkly.comBefore you upgrade, make sure every Relay Proxy host can reach both stream.launchdarkly.com and sdk.launchdarkly.com, or the hosts you configured in streamUri and baseUri. If your egress proxy allows specific URL paths, also allow /sdk/stream, /sdk/poll, and /sdk/latest-all.
If you use the Federal or EU instance, the same applies to the baseUri and streamUri hosts for that instance.

SDK compatibility

Version 9.0 continues to serve every endpoint that 8.x serves. You do not need to change your SDKs. It also adds endpoints for SDKs in data saving mode:
  • Server-side SDKs: /sdk/stream and /sdk/poll
  • Client-side and mobile SDKs: /sdk/stream/eval and /sdk/poll/eval
To use data saving mode with the Relay Proxy, upgrade every Relay Proxy instance to 9.0 before you configure any SDK to use it. Relay Proxy 8.x does not serve these endpoints. SDKs might notice these differences:
  • Responses to authenticated SDK requests include an X-LD-EnvId header that contains the environment ID.
  • The ETag format for PHP SDK polling responses changed. The first request from each PHP SDK after the upgrade receives a full response. If 8.x and 9.0 instances serve the same SDKs during a rolling upgrade, conditional requests do not match across versions. This causes extra full responses, not errors.
  • If the Relay Proxy cannot connect because LaunchDarkly rejects an environment’s SDK key, it responds to SDK requests for that environment with a 401 status code, or 404 for JavaScript-based client-side SDKs, even when a persistent store holds data for that environment. It retries the key every expiredCredentialCleanupInterval, which defaults to one minute. Confirm that your SDK keys are valid before you upgrade.

Payload filtering

Version 9.0 removes payload filtering. The connection that 9.0 uses to receive data from LaunchDarkly does not support payload filters. If you use payload filters with Relay Proxy 8.x:
  • Remove the [Filters] sections and projKey options from your configuration file. Otherwise, the Relay Proxy refuses to start.
  • SDKs that request a filtered environment receive the full, unfiltered data for that environment. They do not receive an error. Expect larger payloads and more memory use in those SDKs.
  • The Relay Proxy no longer writes filtered data to a persistent store. If an SDK in daemon mode reads a filtered store prefix, change it to read the environment’s unfiltered prefix. Otherwise its data stops updating after you remove your last 8.x instance.
  • The status resource no longer lists filtered environments.
For environments that you configure manually, filtered SDK payloads can replace some uses of payload filtering. A filtered SDK payload associates one or more views with a non-default SDK key. When the Relay Proxy connects with that key, LaunchDarkly delivers only the flags and segments in those views. The Relay Proxy serves that filtered data to every SDK that uses the key. Filtered SDK payloads do not work with automatic configuration. Filtered SDK payloads are in beta, and are available only to certain customers on an Enterprise plan. If your use of the Relay Proxy depends on payload filtering and filtered SDK payloads do not meet your needs, continue to use the latest 8.x release.

Status resource

The status resource has one breaking change. Version 9.0 removes the expiringSdkKey field from each environment. Version 9.0 lists every SDK key for an environment in the sdkKeys array, and a key that is about to expire has an expiry field. Update any monitor that reads expiringSdkKey. Version 9.0 also adds the following to the status resource:
  • A status resource for each environment: /status/<environment ID or configured name> returns the status of one environment. With automatic configuration, /status/<project key>/<environment key> also works. These routes respond with 404 for an unknown environment and 503 for an environment that has not finished initializing.
  • The expect query parameter: Add a condition such as /status?expect=status=healthy. The Relay Proxy responds with 200 if the condition is true and 412 if it is false. A load balancer or Kubernetes probe can check health without parsing JSON.
  • The refusedEnvironments array: This lists auto-configured environments that the Relay Proxy received but could not set up, such as an environment with a malformed key set.
To learn more, read the Relay Proxy GitHub repository’s Service endpoints.

Persistent stores and rolling upgrades

Version 9.0 writes flag and segment data to Redis, DynamoDB, and Consul in the same format as 8.x. Because of this:
  • You can run 8.x and 9.0 instances against the same persistent store and prefix during a rolling upgrade.
  • SDKs in daemon mode read data written by 9.0 without any change.
  • 9.0 can read an auto-configuration cache written by 8.x.
If many Relay Proxy instances share one Redis database, we recommend setting REDIS_ATOMIC_UPSERT=true on all of them. This option is available in 8.23.0 and later. After the Relay Proxy receives data from LaunchDarkly, it serves SDKs from memory. It reads the persistent store only at startup, before data arrives from LaunchDarkly. A slow persistent store no longer delays streaming responses while the Relay Proxy is running.

Docker images

Version 9.0 publishes only distroless images. Distroless images contain the Relay Proxy and its runtime dependencies, without a shell or package manager. The 9.0 images use the Debian 13 distroless base. The 8.x distroless images used Debian 12. Here are the image tags for 9.0: The images are available for amd64, arm64, and armv7. The i386 image, all -alpine tags, and all -static-debian12- tags are no longer published for 9.0. The latest-static-debian12- tags continue to point to the latest 8.x release. Review these changes if you use Docker:
  • Image tags: If you pin an -alpine or -i386 tag, change to one of the tags above. If you pin a -static-debian12-nonroot or -static-debian12-debug-nonroot tag, change debian12 to debian13.
  • Health checks: The images have no shell, curl, or wget. Docker HEALTHCHECK commands, Docker Compose health checks, Kubernetes exec probes, and preStop hooks that run a shell command fail. Use HTTP probes against /status with the expect parameter instead. Don’t base a readiness probe on expect=status=healthy. The top-level status becomes degraded when an environment stays disconnected for longer than disconnectedStatusTime. During a LaunchDarkly outage, that probe would remove every pod from the Kubernetes Service at once. To learn more, read Health checks with the expect parameter.
  • User ID: The 8.x Alpine image ran as user ID 1000. The 9.0 images run as the distroless nonroot user, ID 65532. Make sure mounted files such as a configuration file or an offline mode archive are readable by that user, or set runAsUser and fsGroup in your Kubernetes security context.
The entry point, the default configuration file path /ldr/ld-relay.conf, and port 8030 are unchanged. Release binaries, Linux deb and rpm packages, and Windows builds are unchanged.

Go library

If you embed the Relay Proxy in a Go application:
  • The module path changed from github.com/launchdarkly/ld-relay/v8 to github.com/launchdarkly/ld-relay/v9. Update your imports.
  • relay.NewRelay, config.LoadConfigFile, config.LoadConfigFromEnvironment, and config.ValidateConfig take a *slog.Logger instead of ldlog.Loggers. config.LoadConfigFromEnvironmentBase no longer takes a logger.
  • Version 9.0 removes the configuration types for Datadog, Stackdriver, Prometheus, and payload filters. config.OpenTelemetryConfig and config.ConcurrencyConfig are new.
The minimum Go version is unchanged.

Roll back to 8.x

To roll back, deploy your previous 8.x version with your previous configuration. Because 9.0 uses the same persistent store format, 8.x can read data that 9.0 wrote. To learn more, read Upgrading the Relay Proxy.