Crate relay_metrics
source ·Expand description
Metric protocol, aggregation and processing for Sentry.
Metrics are high-volume values sent from Sentry clients, integrations, or extracted from errors and transactions, that can be aggregated and queried over large time windows. As opposed to rich errors and transactions, metrics carry relatively little context information in tags with low cardinality.
Protocol
Clients submit metrics in a text-based protocol based on StatsD. A sample submission looks like this:
endpoint.response_time@millisecond:36:49:57:68|d|#route:user_index|T1615889440
endpoint.hits:4|c|#route:user_index|T1615889440
endpoint.parallel_requests:25:17:42:220:85|g|#route:user_index|T1615889440
endpoint.users:3182887624:4267882815|s|#route:user_index|T1615889440
The metric type is part of its signature just like the unit. Therefore, it is allowed to reuse a metric name for multiple metric types, which will result in multiple metrics being recorded.
Metric Envelopes
To send one or more metrics to Relay, the raw protocol is enclosed in an envelope item of type
metrics
:
{}
{"type": "statsd", ...}
endpoint.response_time@millisecond:36:49:57:68|d|#route:user_index|T1615889440
endpoint.hits:4|c|#route:user_index|T1615889440
endpoint.parallel_requests:25:17:42:220:85|g|#route:user_index|T1615889440
endpoint.users:3182887624:4267882815|s|#route:user_index|T1615889440
...
Note that the name format used in the statsd protocol is different from the MRI: Metric names
are not prefixed with <ty>:
as the type is somewhere else in the protocol. If no metric
namespace is specified, the "custom"
namespace is assumed.
Optionally, a timestamp can be added to every line of the submitted envelope. The timestamp has
to be a valid Unix timestamp (UTC) and must be prefixed with T
. If it is omitted, the
received
time of the envelope is assumed.
Aggregation
Relay accumulates all metrics in time buckets before sending them onwards. Aggregation
is handled by the Aggregator
, which should be created once for the entire system. It flushes
aggregates in regular intervals, either shortly after their original time window has passed or
with a debounce delay for backdated submissions.
Warning: With chained Relays submission delays accumulate.
Aggregate buckets are encoded in JSON with the following schema:
[
{
"timestamp": 1615889440,
"width": 0,
"name": "d:custom/endpoint.response_time@millisecond",
"type": "d",
"value": [
36.0,
49.0,
57.0,
68.0
],
"tags": {
"route": "user_index"
}
},
{
"timestamp": 1615889440,
"width": 0,
"name": "c:custom/endpoint.hits@none",
"type": "c",
"value": 4.0,
"tags": {
"route": "user_index"
}
},
{
"timestamp": 1615889440,
"width": 0,
"name": "g:custom/endpoint.parallel_requests@none",
"type": "g",
"value": {
"last": 25.0,
"min": 17.0,
"max": 42.0,
"sum": 220.0,
"count": 85
},
"tags": {
"route": "user_index"
}
},
{
"timestamp": 1615889440,
"width": 0,
"name": "s:custom/endpoint.users@none",
"type": "s",
"value": [
3182887624,
4267882815
],
"tags": {
"route": "user_index"
}
}
]
Ingestion
Processing Relays write aggregate buckets into the ingestion Kafka stream. The schema is similar to the aggregation payload, with the addition of scoping information. Each bucket is sent in a separate message:
{
"org_id": 1,
"project_id": 42,
"name": "endpoint.response_time",
"unit": "millisecond",
"value": [
36.0,
49.0,
57.0,
68.0
],
"type": "d",
"timestamp": 1615889440,
"tags": {
"route": "user_index"
}
}
Macros
- Creates a
DistributionValue
containing the given arguments.
Structs
- Check whether the aggregator has not (yet) exceeded its total limits. Used for health checks.
- Any error that may occur during aggregation.
- Parameters used by the
AggregatorService
. - A collector of
Bucket
submissions. - An aggregation of metric values.
- Custom user-defined units without builtin conversion.
- A message containing a vector of buckets to be flushed.
- A snapshot of values within a
Bucket
. - A Bucket and its hashed key. This is cheaper to pass around than a (BucketKey, Bucket) pair.
- A message containing a list of
Bucket
s to be inserted into the aggregator. - A unique identifier for metrics including typing and namespacing.
- Iterator over parsed metrics returned from
Bucket::parse_all
. - An error returned when metrics or MRIs cannot be parsed.
- An error parsing a
MetricUnit
or one of its variants. - Service that routes metrics & metric buckets to the appropriate aggregator.
- Contains an
AggregatorConfig
for a specific scope. - A unix timestamp (full seconds elapsed since 1970-01-01 00:00 UTC).
Enums
- Aggregator service interface.
- The aggregated value of a metric bucket.
- Condition that needs to be met for a metric or bucket to be routed to a secondary aggregator.
- Time duration units used in
MetricUnit::Duration
. - Defines a field and a field value to compare to when a
Condition
is evaluated. - Units of fraction used in
MetricUnit::Fraction
. - Size of information derived from bytes, used in
MetricUnit::Information
. - The namespace of a metric.
- The type of a
BucketValue
, determining its aggregation and evaluation. - The unit of measurement of a metric value.
- Configuration value for
AggregatorConfig::shift_key
.
Type Definitions
- Type used for Counter metric
- Type of distribution entries
- A distribution of values within a
Bucket
. - Type used for Gauge entries
- Type used for set elements in Set metric
- A set of unique values.