Traces in a rolled-over index are unable to be hotlinked via /trace/{id}

[wiardvanrij]

Describe the bug

• Search a trace in your 'history' that should be in an index that is in a rollover'd index.
• You are able to find this via the search and click on it. This gives a working view of your trace at /trace/{your-id}
• Open a new tab
• Insert the link
• You get a 404 when directly going to the //trace/{your-id} page

When using a trace that is 'fresh' i.e. has not been in a rollover index, the /trace/{your-id} works.

Expected behavior
Directly 'hotlinking' to any /trace/{your-id} should work. Regardless if this lives in a roll-over'd index.

Version (please complete the following information):

• Jaeger version: 1.21

What troubleshooting steps did you try?
So many things, but your own tracing and debug lvls are empty.

Additional context
I'm aware of other issues before, I have searched them. The key difference here is:

• We can find the traces via search. It's merely based on the direct link to /trace/{your-id}

I think, but I got no evidence for this, that the direct page on /trace/{your-id} does not account for the flag es.use-aliases

[pavolloffay]

Without knowing your storage configuration it's impossible to understand whether this is a bug or misconfiguration. Could you please share all jaeger flags and rollover script configuration?

[wiardvanrij]

We are using Elasticsearch as storage. The flags regarding es.use-alias are implemented over all the components. We use jobs for rollovers and clean up. These come from Jaeger itself.
Basically nothing special, rollover on size.

I'm open to share more tomorrow but I'm somewhat confident that this ain't a misconfiguration as everything works fine. Traces can be found/searched for. It's merely not working for specific traces that are "older".

This gives an 404 when using the URL directly. Even though these can be found in ES. When just searching by date, these traces can be accessed. Again, this behavior is only for "older" traces.

Stats wise, we ingest about 350 billion spans per month. Or for the sane European, that's 350 miljard.

[pavolloffay]

The real configuration values would help here, but if you don't want to share that is fine.

Setting --es.max-span-age to match rollover's "lookback" (

jaeger/plugin/storage/es/esRollover.py

Line 54 in 7636530

print('lookback configuration:')

) should solve your issue.

[wiardvanrij]

Welp, this is awkward. That indeed fixed it. I still don't really understand how one could get a trace from the search UI part, but not from the URL. Anyhow, thank you so much!

[pavolloffay]

Let's keep this open and improve docs so the other people don't run into the same issue.

Where did you learn how to configure rollover? In the initial setup did you configure --es.max-span-age?

We should improve rollover docs and as well document this in --es.max-span-age.

Are you using operator as well?

[wiardvanrij]

I'm using the helm chart, but we've edited this quite some bit so this is not perse in line with upstream anymore.
So we use that rollover feature there: https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/values.yaml#L577

but instead of days, we use

    - name: CONDITIONS
      value: '{"max_size": "75gb"}'

And basically this works like a charm and you can also see your traces via the search (also on rollover'd indexes).

[albertteoh]

I still don't really understand how one could get a trace from the search UI part, but not from the URL

@wiardvanrij I think this is a good question and I'll attempt (because I didn't understand myself until looking over the code just now) to explain why this behaviour is the way it is. Feel free to correct me if I'm wrong @pavolloffay.

• Searching for traces uses the search UI's lookback time for both Trace ID discovery and fetching spans for that trace ID.

• Getting a trace by ID limits the index being searched to just the read alias and the corresponding span search to start from now - max-span-age.

If you're executing the lookback action as per the docs and the max-span-age doesn't match, it may mean that some spans were evicted from the read alias, causing the search by TraceID to fail because it's looking only at the read alias. If the spans were still in the read alias, the mismatch in max-span-age could lead to the query still missing the window of time where the spans existed:

It means that old data will not be available for search. This imitates the behavior of --es.max-span-age flag used in the default index-per-day deployment.