Crate relay_metrics

source ·
Expand description

Metric protocol, aggregation and processing for Sentry.

Metrics are high-volume values sent from Sentry clients, integrations, or extracted from errors and transactions, that can be aggregated and queried over large time windows. As opposed to rich errors and transactions, metrics carry relatively little context information in tags with low cardinality.

Protocol

Clients submit metrics in a text-based protocol based on StatsD. A sample submission looks like this:

endpoint.response_time@millisecond:36:49:57:68|d|#route:user_index|T1615889440
endpoint.hits:4|c|#route:user_index|T1615889440
endpoint.parallel_requests:25:17:42:220:85|g|#route:user_index|T1615889440
endpoint.users:3182887624:4267882815|s|#route:user_index|T1615889440

The metric type is part of its signature just like the unit. Therefore, it is allowed to reuse a metric name for multiple metric types, which will result in multiple metrics being recorded.

Metric Envelopes

To send one or more metrics to Relay, the raw protocol is enclosed in an envelope item of type metrics:

{}
{"type": "statsd", ...}
endpoint.response_time@millisecond:36:49:57:68|d|#route:user_index|T1615889440
endpoint.hits:4|c|#route:user_index|T1615889440
endpoint.parallel_requests:25:17:42:220:85|g|#route:user_index|T1615889440
endpoint.users:3182887624:4267882815|s|#route:user_index|T1615889440
...

Note that the name format used in the statsd protocol is different from the MRI: Metric names are not prefixed with <ty>: as the type is somewhere else in the protocol. If no metric namespace is specified, the "custom" namespace is assumed.

Optionally, a timestamp can be added to every line of the submitted envelope. The timestamp has to be a valid Unix timestamp (UTC) and must be prefixed with T. If it is omitted, the received time of the envelope is assumed.

Aggregation

Relay accumulates all metrics in time buckets before sending them onwards. Aggregation is handled by the Aggregator, which should be created once for the entire system. It flushes aggregates in regular intervals, either shortly after their original time window has passed or with a debounce delay for backdated submissions.

Warning: With chained Relays submission delays accumulate.

Aggregate buckets are encoded in JSON with the following schema:

[
  {
    "timestamp": 1615889440,
    "width": 0,
    "name": "d:custom/endpoint.response_time@millisecond",
    "type": "d",
    "value": [
      36.0,
      49.0,
      57.0,
      68.0
    ],
    "tags": {
      "route": "user_index"
    }
  },
  {
    "timestamp": 1615889440,
    "width": 0,
    "name": "c:custom/endpoint.hits@none",
    "type": "c",
    "value": 4.0,
    "tags": {
      "route": "user_index"
    }
  },
  {
    "timestamp": 1615889440,
    "width": 0,
    "name": "g:custom/endpoint.parallel_requests@none",
    "type": "g",
    "value": {
      "last": 25.0,
      "min": 17.0,
      "max": 42.0,
      "sum": 220.0,
      "count": 85
    },
    "tags": {
      "route": "user_index"
    }
  },
  {
    "timestamp": 1615889440,
    "width": 0,
    "name": "s:custom/endpoint.users@none",
    "type": "s",
    "value": [
      3182887624,
      4267882815
    ],
    "tags": {
      "route": "user_index"
    }
  }
]

Ingestion

Processing Relays write aggregate buckets into the ingestion Kafka stream. The schema is similar to the aggregation payload, with the addition of scoping information. Each bucket is sent in a separate message:

{
  "org_id": 1,
  "project_id": 42,
  "name": "endpoint.response_time",
  "unit": "millisecond",
  "value": [
    36.0,
    49.0,
    57.0,
    68.0
  ],
  "type": "d",
  "timestamp": 1615889440,
  "tags": {
    "route": "user_index"
  }
}

Macros

Structs

Enums

Type Definitions