This repo contains code to test compliance with various Prometheus standards.
The promql directory contains code to test compliance with the native Prometheus PromQL implementation.
The remote_write directory contains code to test compliance with the Prometheus Remote Write specification.
Additionally updated docs.
The motivation for this is that we can now set automated tests where in future we might want to provide docker images with exact set of test cases in versioned image.
Signed-off-by: Bartlomiej Plotka [email protected]
Based on the specification, here is the list of all the high-level cases that needs to be covered by the test suite. In all the cases, the content of the alerts, APIs, time series, are checked to be correct.
pending->firing->inactive.pending->inactive.pending or firing)pending alerts having changing annotation values (checked via API)firing and inactive alerts being sent when they first went into those states.firing alert being re-sent at expected intervals when the alert is active with changing annotation contents.inactive alert being re-sent at expected intervals up to a certain time and not after that.firing state (skipping the pending state) because of zero for duration.inactive state for both the cases where for duration is zero and non zero. Here we should test 2 cases: One where inactive alert was still being sent, hence should stop sending that. Two is the inactive alert was not being sent anymore.pending->firing->inactive while already having active alerts.for duration is non-zero and less than the evaluation interval, firing alert must be sent after the second evaluation of the rule and not before.ALERTS series from the rules above it in the same group.firing and pending.All the time comparison will be done within a certain acceptable delta and need not be exact.
The spec says the following:
The alert name from the alerting rule (HighRequestLatency from the example above) MUST be added to the labels of the alert with the label name as alertname. It MUST override any existing alertname label.
The statement above says that rule's name should override any existing alertname label. Does it mean that in templates $labels.alertname and .Labels.alertname values should behave in the same way?
One of the testcases expects template value to be equal to the existing alertname label:
https://github.com/prometheus/compliance/blob/c7c726de89973d77cb491faa1b32cfddf7dcde8a/alert_generator/cases/case_new_alerts_and_order_check.go#L254
But this looks controversial to what spec says.
The metrics-addr is going away in the next release (see breaking changes). It has been replaced by --set=service.telemetry.metrics.address
This is a pretty minimal import without too many changes, to get it moved before doing anything more.
The first commit imports it completely unchanged, the second one just fixes minimal things to adjust to the new repo location.
Signed-off-by: Julien Pivotto [email protected]
Noticed the OpenMetrics scrape validator tool was not directly linked from the readme, adding this for others to more easily discover.
Similar to the PromQL tests, I don't plan on building binaries and docker images in the CI and rather have a bunch of Go CLI commands in the docs that can be run to use the test suite.
What did you do?
We want to ensure OpenMetrics / Prometheus compatibility in the OpenTelemetry Collector. We have been building compatibility tests to verify the OpenMetrics spec is fully supported on both the OpenTelemetry Collector Prometheus receiver and PRW exporter as well as in Prometheus itself.
We used the OpenMetrics metrics test data available at https://github.com/OpenObservability/OpenMetrics/tree/main/tests/testdata/parsers
Out of a total of 161 negative tests in OpenMetrics, 94 tests pass (these tests are dropped) with an 'up' value of 0; 67 tests are not dropped and have an 'up' value of 1 and 22 tests have incorrectly ingested metrics.
In order to test Prometheus itself, we set up a metrics HTTP endpoint that exposes invalid/bad metrics from the OpenMetrics tests. We then configured Prometheus 2.31.0 to scrape the metrics endpoint.
What did you expect to see?
Expected result: The scrape should fail since the target has invalid metric and the appropriate error should be reported.
For e.g with following metric data: bad_counter_values_1 (https://raw.githubusercontent.com/OpenObservability/OpenMetrics/main/tests/testdata/parsers/bad_counter_values_1/metrics)
# TYPE a counter
a_total -1
# EOF
What did you see instead? Under which circumstances?
Current behavior: Scrape is successful. There are multiple bad test cases that are scraped successfully by Prometheus.
For example - Using bad_counter_values_1 (#5 listed below) does not show an error even though it is an negative counter value. According to OpenMetrics tests, this metric should not be parsed.
You can see no error has been reported and the scrape is successful.
Similar to bad_counter_values_1 test case, there are multiple bad test cases where the scrape is successful and metrics are ingested by Prometheus:
Environment
Darwin 20.6.0 x86_64
version=2.31.0
global:
scrape_interval: 5s
scrape_configs:
- job_name: "open-metrics-scrape"
static_configs:
- targets: ["localhost:3000"]
cc: @PaurushGarg @mustafain117
The remote write CI tests seem to always be failing, which is annoying, especially on unrelated PRs (e.g. for PromQL):
@tomwilkie would you be able to look into this and either fix or disable them?
When running the remote_write tests for the OpenTelemetry collector with the -race flag:
go test -race --tags=compliance -run "TestRemoteWrite/otel/.+" -v ./
The tests fail due to data races: https://gist.github.com/kirbyquerby/59e4e57d59ba131307fb3d9ac2a4e35d
This issue was found when adding these tests to the OpenTelemetry collector (open-telemetry/opentelemetry-collector-contrib/pull/5014), as the tests in the collector are run with -race.
