Empty result when step is larger than 300s #705
Comments
|
@arkbriar I am working on upgrading promxy to implement native histogram ([change],(#696) discussion) and this is based on latest prometheus (post nov 2024) I am facing the same issue but with very high step functions (like 3h). do find any resolution around this? |
@rishabhkumar92 It was the same on my side. High steps are much easier for reproducing the issue, though I was able to reproduce with my tiny query with step of 500s.
Solution, no. I found only a workaround which I posted in the description.
|
@arkbriar There were 2 key things mentioned above:
did you find any workaround for the second case? |
This is the trigger of the exact issue here: the |
|
@arkbriar i validated that delta was indeed set correctly but problem was due to max lookback limit. |
Sorry, I'm a bit confused. How the jump happens when t0 := t - b.delta
if b.valueType != chunkenc.ValNone && t0 > b.lastTime { |
|
I think I'm actually a bit lost on this error report. The subject seems to indicate that there are empty results with a large step (>300s); but I am able to get data with a 5000s step spread over the course of a day (equivalent upstream query Upstream 5000s There is also some mention of time alignment with grafana; which I played around with the start/end times but seemingly can't get it to return an empty dataset either. Is it possible to create a repro case using some public datasource? If not, could we attach some trace log output (which would include the query and data)? |
Sorry for the confusion... I'm pretty sure I can reproduce it with my VM setup since I scripted the MRE. Let me have a try with the open data source. |
I found I couldn't reproduce with the open datasource simply because it's a Proemtheus, which returns a result of aligned timestamps starting from the query's start so it avoids the problem. The branch will be correctly skipped if However, in the case I was and am able to reproduce, timestamps don't align. FYI, the following is the line from the trace log with all the values.
The query start is I think this is a VictoriaMetrics specific problem. Let me see if there's an open VM datasource. |
|
Reproduced with the public VM playground.
promxy:
server_groups:
- static_configs:
- targets: ["play.victoriametrics.com"]
path_prefix: /select/0/prometheus
scheme: https
http_client:
dial_timeout: 1s
tls_config:
insecure_skip_verify: true
Result from VM playground:
Result from promxy: empty.
Trace logs: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Seems the commit is missing from the forked prometheus
prometheus/prometheus#9946. Therefore, there's a high chance no data can be popped when a step is larger than 300s (which is the default look back delta).
An example:
start=1736607436000,end=1736780236000,step=1800sThe overall result from the query engine was empty. I traced through the code and found data was jumped because of this, where the delta is the default value
300000.A workaround to this is to align the
startandendtostep, which Grafana does. And the fact that Grafana always works while my program doesn't confused me a long time until I captured the HTTP request :(The text was updated successfully, but these errors were encountered: