Implement basic support for Otel

[pdabelf5]

Overview

As a site reliability engineer who wants to monitor my traffic traces,
I want to configure NGINX Ingress Controller to send traffic traces to my provider of choice
so that I can have a clear picture of how my system's traffic is behaving

Acceptance Criteria

• Allow tracing to be configured through native NGINX Otel library
• Ensure documentation provides an example of at least one trace provider (e.g. Data Dog or Grafana)

Documentation

NGINX Otel module: https://nginx.org/en/docs/ngx_otel_module.html
Otel vendors: https://opentelemetry.io/ecosystem/vendors/

Example Config Map

Add equivalent support for OpenTelemetry as was deprecated for OpenTracing with a similar UX.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  otel-exporter-endpoint: "[(http|https)://]host:port" # Required
  otel-exporter-trusted-certificate-secret: # Optional nginx.org/ca Secret name 
  otel-exporter-header-name: "name" # Optional
  otel-exporter-header-value: "value" # Optional
  otel-service-name: "unknown_service:nginx" # Optional
  otel-global-trace-enabled: "false" # Optional, enables otel tracing at the http context

Support at the Server/Location level will be via snippet on VS/VSR/Ingress

• Add validation for missing either header name or value
• Add validation for other field present without the required otel endpoint
• update trusted cert on secret or secret name update
• Add unit tests
• Add python tests to verify generated nginx config

[pdabelf5]

the nginx-module-otel package is not available for arm, s390x & ppc64le architectures. These have been removed from the build pipelines to allow testing to complete.

[pdabelf5]

The nginx-module-otel & nginx-plus-module-otel rpm packages depend on the c-ares shared library. The full RHEL distributions include this libarary package, whereas, ubi8 & ubi9 do not include the c-ares package in the base package repository.