Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Configuration guide of structured logging for Kyuubi server #6861

Closed
wants to merge 2 commits into from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Dec 24, 2024

Why are the changes needed?

It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in KafkaAppender and JSON Template Layout, thus the goal could be achieved by just a few configurations.

To simplify the user setup steps, this PR adds log4j-layout-template-json-<version>.jar into Kyuubi binary tarball.

PS: I also plan to support sending engine bootstrap process(e.g. spark-submit) logs into Kafka with specific labels in the follow-up PRs.

How was this patch tested?

Manually test.

Configuration in $KYUUBI_HOME/conf/log4j2.xml

<Configuration status="INFO">
  <Appenders>
    <Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
      <JsonTemplateLayout>
        <EventTemplateAdditionalField key="app" value="kyuubi"/>
        <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
        <EventTemplateAdditionalField key="host" value="${hostName}"/>
      </JsonTemplateLayout>
      <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
      <Property name="compression.type" value="gzip"/>
    </Kafka>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="kafka"/>
    </Root>
  </Loggers>
</Configuration>

Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
Xnip2024-12-25_03-18-52

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added kind:documentation Documentation is a feature! kind:infra license, community building, project builds, asf infra related, etc. kind:build labels Dec 24, 2024
@pan3793
Copy link
Member Author

pan3793 commented Dec 24, 2024

For reviewers who want to verify or play with this feature, awesome-kyuubi/hadoop-testing provides an out-of-box setup via the following rendering command

ansible-playbook build.yaml -e "{grafana_enabled: true, prometheus_enabled: true, loki_enabled: true, kafka_enabled: true, kafka_ui_enabled: true}"

@codecov-commenter
Copy link

codecov-commenter commented Dec 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (d50cf17) to head (9556da2).
Report is 3 commits behind head on master.

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6861   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         687     687           
  Lines       42473   42473           
  Branches     5796    5796           
======================================
  Misses      42473   42473           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pan3793 pan3793 self-assigned this Dec 25, 2024
@pan3793 pan3793 added this to the v1.11.0 milestone Dec 25, 2024
@pan3793 pan3793 closed this in 14e12e9 Dec 25, 2024
@pan3793
Copy link
Member Author

pan3793 commented Dec 25, 2024

Thanks, merged to master

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
kind:build kind:documentation Documentation is a feature! kind:infra license, community building, project builds, asf infra related, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants