- The Relay Proxy exports metrics, traces, and logs only through the OpenTelemetry Protocol (OTLP). Version 9.0 removes the Datadog, Stackdriver, and Prometheus integrations, and every metric and attribute name has changed. To learn more, read Metrics and tracing.
- The log output format has changed. To learn more, read Logging.
- Version 9.0 removes payload filtering. To learn more, read Payload filtering.
- Version 9.0 removes several configuration options. A configuration file that still contains them prevents the Relay Proxy from starting. To learn more, read Configuration changes.
- Docker images are distroless only, and are based on Debian 13 instead of Debian 12. The Alpine and i386 images and the
-static-debian12-tags are no longer published. To learn more, read Docker images. - The Relay Proxy connects to LaunchDarkly using the protocol behind data saving mode, which requires network access to an additional LaunchDarkly host. To learn more, read Connection to LaunchDarkly.
Prerequisites
Before you migrate, we recommend that you:- Upgrade to the latest 8.x release first. Several features that 9.0 relies on, including the Redis timeout options, concurrent SDK keys, and the auto-configuration cache, were introduced during the 8.x release line. If you start from the latest 8.x release, the changes in this topic are the only changes you need to make. To find the latest release, visit Relay Proxy releases on GitHub.
- Test the upgrade in a non-production environment.
- Review the Relay Proxy 9.0.0 release notes.
Migration checklist
This checklist summarizes the changes you may need to make. Each item links to the section that explains it. To prepare a Relay Proxy 8.x deployment for 9.0:- Remove the
[Datadog],[Stackdriver],[Prometheus], and[Filters]sections, and theprojKey,inlineUsers,exitOnError, andignoreConnectionErrorsoptions, from your configuration file. Read Removed configuration options. - Remove the matching environment variables, such as
USE_PROMETHEUS,LD_FILTER_KEYS_*, andEXIT_ON_ERROR. Read Removed configuration options. - Configure OpenTelemetry export if you collect metrics, and rebuild your charts and alerts using the new metric names. Read Metrics and tracing.
- Update log parsers, alerts, and saved searches for the new log format. Read Logging.
- Allow outbound connections from the Relay Proxy to
sdk.launchdarkly.com. Read Connection to LaunchDarkly. - Switch any
-alpine,-i386, or-static-debian12-Docker image tag, replace shell-based health checks, and check file permissions for the new user ID. Read Docker images. - Replace any use of the
expiringSdkKeyfield in tools that parse the status resource. Read Status resource. - Review the new request body limit if your client-side SDKs send
REPORTrequests with very large contexts. Read Changed defaults. - Update your imports and logger if you embed the Relay Proxy as a Go library. Read Go library.
Configuration changes
Version 9.0 removes, adds, and changes several configuration options.How 9.0 handles removed options
The Relay Proxy treats a removed option differently depending on where you set it:- Configuration file: The Relay Proxy refuses to start. It logs an error such as
unsupported or misspelled section "Prometheus"orunsupported or misspelled section "Environment", variable "projKey"and exits with status code 1. - Environment variable: For most removed variables, the Relay Proxy ignores the variable without logging a warning. For example, if you set
USE_PROMETHEUS=true, the Relay Proxy starts normally but does not export Prometheus metrics. The exceptions areEXIT_ON_ERRORandIGNORE_CONNECTION_ERRORS. If you set either of them, the Relay Proxy logsthis variable is no longer supportedand refuses to start.
Removed configuration options
Here are the options that 9.0 removes:New configuration options
Version 9.0 adds the following configuration sections and options. The[OpenTelemetry] section controls metric, trace, and log export. It replaces the removed metrics sections:
The Relay Proxy does not implement
http/json. If you set http/json, the Relay Proxy logs a warning and sends http/protobuf instead. Your endpoint continues to work, because every OTLP/HTTP receiver accepts http/protobuf.
Other OpenTelemetry settings, such as OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, and OTEL_METRIC_EXPORT_INTERVAL, use the standard OpenTelemetry environment variables. They have no configuration file equivalent.
The [Concurrency] section limits how many full data payloads the Relay Proxy sends to SDKs at the same time. This protects the Relay Proxy when many SDKs connect at once, for example after a deployment. The limit is off by default:
The
[Redis] section has three new options for mutual TLS:
These options take effect only when TLS is on, either through
REDIS_TLS or a rediss:// URL. The Relay Proxy reads the files at startup. Restart it after you rotate a certificate.
To learn more about each option, read the Relay Proxy GitHub repository’s Configuration.
Changed defaults
The default formaxClientRequestBodySize (MAX_CLIENT_REQUEST_BODY_SIZE) changed from no limit to 5 MiB. This option limits the body size of REPORT evaluation requests from client-side SDKs. The Relay Proxy responds to a larger request with a 413 status code. You can raise the limit, for example to 10MiB, but you cannot turn it off.
Changed behavior
One-shot mode, which you turn on withexitAlways (EXIT_ALWAYS), changed in two ways:
- The Relay Proxy waits for every environment to initialize, and exits with a non-zero status code if any environment failed. In 8.x, setting
exitOnErrormade the Relay Proxy stop waiting after the first environment finished. If that environment succeeded, the Relay Proxy exited with status code 0, even if another environment failed. initTimeout(INIT_TIMEOUT) must be greater than zero. If it is zero or negative, the Relay Proxy refuses to start.
Metrics and tracing
Version 9.0 replaces OpenCensus with OpenTelemetry. The Relay Proxy pushes metrics, traces, and logs over OTLP to any compatible receiver. It does not expose a metrics endpoint for scraping.Send metrics to your monitoring system
To export telemetry, setUSE_OTLP=true and point OTEL_EXPORTER_OTLP_ENDPOINT at an OTLP receiver.
Here is an example:
- Prometheus: Send metrics to the Prometheus OTLP receiver, available in Prometheus 2.47 and later with
--web.enable-otlp-receiver, or to an OpenTelemetry Collector that exposes a Prometheus endpoint. Remove scrape jobs that target port 8031. If you send directly to Prometheus, setOTEL_TRACES_EXPORTER=noneandOTEL_LOGS_EXPORTER=none, because Prometheus does not accept traces or logs. Pushed metrics have noupseries. Prometheus cannot detect a stopped Relay Proxy the way it detected a failed scrape. Use the status resource or a metric staleness alert instead. - Datadog: Turn on OTLP ingestion in the Datadog Agent and send to it. Set
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta. If you have not turned on OTLP log ingestion in the Agent, setOTEL_LOGS_EXPORTER=none. TheDATADOG_TAG_*options are replaced byOTEL_RESOURCE_ATTRIBUTES. - Stackdriver (Google Cloud Monitoring): Send to an OpenTelemetry Collector configured with a Google Cloud exporter.
Metric name changes
Version 9.0 renames every metric to follow the OpenTelemetry semantic conventions. It also removes the 8.x prefix options, and metric names in 9.0 are fixed. Here is how the 8.x metrics map to 9.0:
If you send metrics to Prometheus, the 9.0 names appear as
http_server_active_requests, launchdarkly_relay_requests_total, and http_server_request_duration_seconds.
Attribute changes
Metric attributes, which 8.x called tags, were also renamed. Values are no longer rewritten. In 8.x, every/ in a value became _. In 9.0, values appear as received, and a missing value is not_provided instead of not-provided.
Here is how the 8.x tags map to 9.0 attributes:
Every metric also carries these resource attributes:
service.name, which defaults told-relayservice.instance.id, a unique ID for each Relay Proxy processservice.versionlaunchdarkly.relay.sdk.version
OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES.
New metrics
Version 9.0 adds metrics that have no 8.x equivalent:- Event forwarding:
launchdarkly.relay.events.received.size,launchdarkly.relay.events.sent,launchdarkly.relay.events.sent.size,launchdarkly.relay.events.failed,launchdarkly.relay.events.dropped, andlaunchdarkly.relay.events.pending. - Status: The status resource is also available as metrics, such as
launchdarkly.relay.statusandlaunchdarkly.relay.environment.connection.state.count. SetOTEL_ENVIRONMENT_STATUS_METRICS=trueto add a series for each environment. You can alert on these metrics instead of polling the status resource. - Initialization limits:
launchdarkly.relay.init.*metrics, exported when you configure the[Concurrency]section. - Go runtime: Standard
go.*metrics such asgo.memory.usedandgo.goroutine.count.
Tracing
Version 9.0 exports distributed traces over OTLP. The Relay Proxy creates a span for each request, plus internal spans for authentication, data store reads, flag evaluation, and response writing. It accepts W3Ctraceparent headers from your applications. The Relay Proxy replaces the evaluation context in url.path span attributes with REDACTED.
By default, the Relay Proxy records every trace. In 8.x, the default was 1 in 10,000 requests. To lower trace volume, set OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, or set OTEL_TRACES_EXPORTER=none.
Here is an example that keeps 1% of traces:
Logging
The Relay Proxy now writes logs with Go’s structured logging package. TheLOG_FORMAT and LOG_LEVEL options are unchanged, but the output format is different. Update any log parser, alert, or saved search that depends on the 8.x format.
The JSON format has changed:
- The
timestampfield is nowtime, and themessagefield is nowmsg. - Timestamps use the host’s local time zone instead of always using UTC. Set
TZ=UTCto keep UTC timestamps. - Details that 8.x wrote into the message are now separate fields. For example, the environment appears in an
envfield instead of an[env: ...]prefix in the message. - Many Relay Proxy messages have new wording.
key=value pairs.
Here is an example:
OTEL_LOGS_EXPORTER=none.
Connection to LaunchDarkly
Version 9.0 connects to LaunchDarkly using the same protocol as SDKs in data saving mode. Each environment first requests its data with a polling request, then opens a streaming connection that receives changes. If the streaming connection fails, the Relay Proxy falls back to polling. After a reconnect, the Relay Proxy receives only the changes it missed instead of the full data set. The LaunchDarkly hostnames have not changed, but 9.0 uses one more of them. Relay Proxy 8.x connects tosdk.launchdarkly.com only when big segments are on. Version 9.0 always connects to it.
If you use the Federal or EU instance, the same applies to the baseUri and streamUri hosts for that instance.
SDK compatibility
Version 9.0 continues to serve every endpoint that 8.x serves. You do not need to change your SDKs. It also adds endpoints for SDKs in data saving mode:- Server-side SDKs:
/sdk/streamand/sdk/poll - Client-side and mobile SDKs:
/sdk/stream/evaland/sdk/poll/eval
- Responses to authenticated SDK requests include an
X-LD-EnvIdheader that contains the environment ID. - The
ETagformat for PHP SDK polling responses changed. The first request from each PHP SDK after the upgrade receives a full response. If 8.x and 9.0 instances serve the same SDKs during a rolling upgrade, conditional requests do not match across versions. This causes extra full responses, not errors. - If the Relay Proxy cannot connect because LaunchDarkly rejects an environment’s SDK key, it responds to SDK requests for that environment with a
401status code, or404for JavaScript-based client-side SDKs, even when a persistent store holds data for that environment. It retries the key everyexpiredCredentialCleanupInterval, which defaults to one minute. Confirm that your SDK keys are valid before you upgrade.
Payload filtering
Version 9.0 removes payload filtering. The connection that 9.0 uses to receive data from LaunchDarkly does not support payload filters. If you use payload filters with Relay Proxy 8.x:- Remove the
[Filters]sections andprojKeyoptions from your configuration file. Otherwise, the Relay Proxy refuses to start. - SDKs that request a filtered environment receive the full, unfiltered data for that environment. They do not receive an error. Expect larger payloads and more memory use in those SDKs.
- The Relay Proxy no longer writes filtered data to a persistent store. If an SDK in daemon mode reads a filtered store prefix, change it to read the environment’s unfiltered prefix. Otherwise its data stops updating after you remove your last 8.x instance.
- The status resource no longer lists filtered environments.
Status resource
The status resource has one breaking change. Version 9.0 removes theexpiringSdkKey field from each environment. Version 9.0 lists every SDK key for an environment in the sdkKeys array, and a key that is about to expire has an expiry field. Update any monitor that reads expiringSdkKey.
Version 9.0 also adds the following to the status resource:
- A status resource for each environment:
/status/<environment ID or configured name>returns the status of one environment. With automatic configuration,/status/<project key>/<environment key>also works. These routes respond with404for an unknown environment and503for an environment that has not finished initializing. - The
expectquery parameter: Add a condition such as/status?expect=status=healthy. The Relay Proxy responds with200if the condition is true and412if it is false. A load balancer or Kubernetes probe can check health without parsing JSON. - The
refusedEnvironmentsarray: This lists auto-configured environments that the Relay Proxy received but could not set up, such as an environment with a malformed key set.
Persistent stores and rolling upgrades
Version 9.0 writes flag and segment data to Redis, DynamoDB, and Consul in the same format as 8.x. Because of this:- You can run 8.x and 9.0 instances against the same persistent store and prefix during a rolling upgrade.
- SDKs in daemon mode read data written by 9.0 without any change.
- 9.0 can read an auto-configuration cache written by 8.x.
REDIS_ATOMIC_UPSERT=true on all of them. This option is available in 8.23.0 and later.
After the Relay Proxy receives data from LaunchDarkly, it serves SDKs from memory. It reads the persistent store only at startup, before data arrives from LaunchDarkly. A slow persistent store no longer delays streaming responses while the Relay Proxy is running.
Docker images
Version 9.0 publishes only distroless images. Distroless images contain the Relay Proxy and its runtime dependencies, without a shell or package manager. The 9.0 images use the Debian 13 distroless base. The 8.x distroless images used Debian 12. Here are the image tags for 9.0:
The images are available for amd64, arm64, and armv7. The i386 image, all
-alpine tags, and all -static-debian12- tags are no longer published for 9.0. The latest-static-debian12- tags continue to point to the latest 8.x release.
Review these changes if you use Docker:
- Image tags: If you pin an
-alpineor-i386tag, change to one of the tags above. If you pin a-static-debian12-nonrootor-static-debian12-debug-nonroottag, changedebian12todebian13. - Health checks: The images have no shell,
curl, orwget. DockerHEALTHCHECKcommands, Docker Compose health checks, Kubernetesexecprobes, andpreStophooks that run a shell command fail. Use HTTP probes against/statuswith theexpectparameter instead. Don’t base a readiness probe onexpect=status=healthy. The top-level status becomesdegradedwhen an environment stays disconnected for longer thandisconnectedStatusTime. During a LaunchDarkly outage, that probe would remove every pod from the Kubernetes Service at once. To learn more, read Health checks with theexpectparameter. - User ID: The 8.x Alpine image ran as user ID 1000. The 9.0 images run as the distroless
nonrootuser, ID 65532. Make sure mounted files such as a configuration file or an offline mode archive are readable by that user, or setrunAsUserandfsGroupin your Kubernetes security context.
/ldr/ld-relay.conf, and port 8030 are unchanged.
Release binaries, Linux deb and rpm packages, and Windows builds are unchanged.
Go library
If you embed the Relay Proxy in a Go application:- The module path changed from
github.com/launchdarkly/ld-relay/v8togithub.com/launchdarkly/ld-relay/v9. Update your imports. relay.NewRelay,config.LoadConfigFile,config.LoadConfigFromEnvironment, andconfig.ValidateConfigtake a*slog.Loggerinstead ofldlog.Loggers.config.LoadConfigFromEnvironmentBaseno longer takes a logger.- Version 9.0 removes the configuration types for Datadog, Stackdriver, Prometheus, and payload filters.
config.OpenTelemetryConfigandconfig.ConcurrencyConfigare new.