Jaeger UI is throwing error when RBAC security plugin is enabled on Opensearch

[Sidhartha-Biswal]

Hi,

We are getting the below error when trying to access Jaeger UI after RBAC security plugin is enabled on Opensearch.
Also, it shows "Internal server error" in the Jaeger Query logs. We are getting the exception as below. Could you please provide more explanation about this exception?

{
    "extra_data": {
        "caller": "app/http_handler.go:492",
        "stacktrace": "github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError	
		github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:492	
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices	
		github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:163	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2	
		github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:536	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3	
		go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:281	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP	
		go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:225	
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1	
		go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:83	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/gorilla/mux.(*Router).ServeHTTP	
		github.com/gorilla/mux@v1.8.1/mux.go:212	
github.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4	
		github.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6	
		github.com/gorilla/handlers@v1.5.1/compress.go:141	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/gorilla/handlers.recoveryHandler.ServeHTTP	
		github.com/gorilla/handlers@v1.5.1/recovery.go:78	
net/http.serverHandler.ServeHTTP	
		net/http/server.go:3142	
net/http.(*conn).serve	 n		net/http/server.go:2044"
    },
    "message": "HTTP handler, Internal Server Error",
    "metadata": {
        "container_name": "container1",
        "namespace": "ns1",
        "pod_name": "pod1"
    },
    "service_id": "service1",
    "severity": "error",
    "timestamp": "2024-09-02T06:32:57.718+00:00",
    "version": "1.0.0"
}

Please find the below screenshot.

Does Jaeger support this RBAC security plugin?

[dosubot[bot]]

Hey there, @Sidhartha-Biswal! I'm here to help you with your tech questions and bug squashing 🤖

I found a similar open issue and discussion that might be relevant to your problem:

1. Jaeger query to ES failed - Internal Server Error - [type=search_phase_execution_exception]: This discussion is still unsolved and was last updated on 2023-07-26 ^[1].
2. [Bug]: "HTTP handler, Internal Server Error","error":"search services failed..": This issue is open and was last updated on June 08, 2024. The suggested solution is to ensure all components are upgraded and to try pointing a new instance of query and UI to the backend ^[2].

Regarding your specific question about Jaeger supporting the RBAC security plugin for OpenSearch, I wasn't able to find a definitive answer in the repository. However, there are some general troubleshooting steps you can follow for diagnosing "Internal server error" in Jaeger Query logs:

1. Verify the sampling strategy: Ensure the correct sampling strategy is being used.
2. Use the logging reporter: Enable logging of spans by setting the environment variable JAEGER_REPORTER_LOG_SPANS to true.
3. Bypass the Jaeger Agent: Configure the Jaeger Client to send spans directly to the Collector.
4. Increase the logging in the backend components: Set the log level to debug for detailed debugging information.
5. Check the /metrics endpoint: Use the /metrics endpoint to check if spans for specific services were received ^[3][4].

Additionally, the Jaeger UI has specific error handling mechanisms for dealing with HTTP 400 errors from Elasticsearch/OpenSearch. The fetchTrace function throws an error on a status code of 400 or higher, capturing the status, status text, and error message from the response ^[5][6].

For more detailed error handling, the Jaeger backend processes and extracts detailed error information from HTTP responses, ensuring meaningful error messages are available for debugging and logging purposes ^[7][8].

If you haven't already, you might want to check if there are any specific environment variables or flags that need to be set in Jaeger to work with OpenSearch's RBAC security plugin. Unfortunately, I couldn't find specific information on this within the repository.

To continue talking to Dosu, mention @dosu.

[yurishkuro]

RBAC is authorization method, Jaeger only provides authentication via basic auth, token, or mTLS.

[Sidhartha-Biswal]

Hi @yurishkuro,

Thanks for the support.

So it means if we want to use Jaeger UI, we should not enable RBAC Security plugin on Opensearch?

Is the below "HTTP handler, Internal Server Error" expected when we enable RBAC security plugin?

When we are accessing the Jaeger UI we are getting the below exception in the Query logs. Could you please explain when this exception comes?

{
    "extra_data": {
        "caller": "app/http_handler.go:492",
        "stacktrace": "github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError	
		github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:492	
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices	
		github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:163	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2	
		github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:536	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3	
		go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:281	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP	
		go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:225	
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1	
		go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:83	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/gorilla/mux.(*Router).ServeHTTP	
		github.com/gorilla/mux@v1.8.1/mux.go:212	
github.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4	
		github.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6	
		github.com/gorilla/handlers@v1.5.1/compress.go:141	
net/http.HandlerFunc.ServeHTTP	
		net/http/server.go:2171	
github.com/gorilla/handlers.recoveryHandler.ServeHTTP	
		github.com/gorilla/handlers@v1.5.1/recovery.go:78	
net/http.serverHandler.ServeHTTP	
		net/http/server.go:3142	
net/http.(*conn).serve	 n		net/http/server.go:2044"
    },
    "message": "HTTP handler, Internal Server Error",
    "metadata": {
        "container_name": "container1",
        "namespace": "ns1",
        "pod_name": "pod1"
    },
    "service_id": "service1",
    "severity": "error",
    "timestamp": "2024-09-02T06:32:57.718+00:00",
    "version": "1.0.0"
}

When RBAC plugin is enabled, we do not see any issue with jaeger collector, but the issue is with the jaeger query. Does it mean this RBAC plugin support is there for collector only but not for query? Could you please confirm?

[Sidhartha-Biswal]

Hi @yurishkuro,
Could you please share your opinion about this exception?

[Sidhartha-Biswal]

Hi @yurishkuro,

We have added some more debug and error logs. Could you please check the below logs and share your opinion that what could be the reason for the issue?

{"level":"error","ts":1726472345.6342745,"caller":"spanstore/service_operation.go:94","msg":"search services failed.","error":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*ServiceOperationStorage).getServices\n\tgithub.heygears.com/jaegertracing/jaeger/plugin/storage/es/spanstore/service_operation.go:94\ngithub.heygears.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).GetServices\n\tgithub.heygears.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:296\ngithub.heygears.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).GetServices\n\tgithub.heygears.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:102\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.GetServices\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:86\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:164\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:540\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:281\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:225\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.heygears.com/gorilla/mux@v1.8.1/mux.go:212\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.heygears.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.heygears.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3210\nnet/http.(*conn).serve\n\tnet/http/server.go:2092"}

{"level":"error","ts":1726472345.6345367,"caller":"app/http_handler.go:496","msg":"HTTP handler, Internal Server Error","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:496\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:166\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:540\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:281\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:225\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.heygears.com/gorilla/mux@v1.8.1/mux.go:212\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.heygears.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.heygears.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3210\nnet/http.(*conn).serve\n\tnet/http/server.go:2092"}

{"level":"error","ts":1726472345.6348941,"caller":"app/http_handler.go:167","msg":"Error in getting the services","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:167\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:540\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:281\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:225\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.heygears.com/gorilla/mux@v1.8.1/mux.go:212\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4\n\tgithub.heygears.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.heygears.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2220\ngithub.heygears.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.heygears.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3210\nnet/http.(*conn).serve\n\tnet/http/server.go:2092"}