Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Bad encoding in Logstash in Filebeat + Logstash + Elasticsearch stack #16835

Open
java-juggled-jazz opened this issue Dec 25, 2024 · 0 comments

Comments

@java-juggled-jazz
Copy link

Hi! I've just deployed Filebeat, Logstash and Elasticsearch
I use Filebeat on Linux machine that uses POSIX locale by default (tried to specify UTF-8 encoding, but nothing has changed)
Logstash within Docker container (locale is en_US.UTF-8) on Linux machine ( locale is C.UTF-8) receives beats but messages have bad decoding
Elasticsearch only receives Logstash entries and stores them as is

Here is Logstash config

input {
  tcp {
    port => 5044
    type => beats
    ssl_enable => false
    codec => plain { charset => "ASCII" }
  }
}

filter {
  if [type] == "beats" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601} %{POSINT:pid} %{WORD:log_level} %{DATA:module} \[(?<request>req-([a-fA-F0-9]{8}(?:-[a-fA-F0-9]{4}){3}-[a-fA-F0-9]{12})\S*)?%{DATA}\] %{GREEDYDATA:message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://XXXXXXXX:9200"]
    index => "XXXXXXXXX-[host][hostname]-%{+YYYY.MM.dd}"
  }
  file {
    path => "logstash-output"
  }
}

Here is Logstash output example

{"message":"\u000FY�*���./����<Udh��3?����3+��\u000F��Rqi�1�\u0005o�\u000F��9,�T�ah��\u001D���\u001E��W\u0003<��\u0000\u0015������\\���fI=g=���6y���w\u0000����2W\u0000\u0000\u0000\u00012C\u0000\u0000\u0003\u0001x^�Tak�6\u0010M�?��O-H>I�e[\u0010��^r��z\u001F�O\ra�����m��v�t����S�n�\u0014\u0002�`���\u001B�{��~�����������\u0018�����\u0011\u0016J(���*�Uj+��&�T�\u001B\u0018>�\u0014]���=bK.��i;Z\u001E\u0019��H������@���\u0001\u0016e*�T����G�#B��\u0014a�R��5[@`�\u0018]�����M\u001F���a\b\u0007���\u000F����v�\u0011\f5\u001DZO������)��\u000E�&X(��","@version":"1","tags":["_grokparsefailure"],"@timestamp":"2024-12-25T14:22:22.436611203Z","event":{"original":"\u000FY�*���./����<Udh��3?����3+��\u000F��Rqi�1�\u0005o�\u000F��9,�T�ah��\u001D���\u001E��W\u0003<��\u0000\u0015������\\���fI=g=���6y���w\u0000����2W\u0000\u0000\u0000\u00012C\u0000\u0000\u0003\u0001x^�Tak�6\u0010M�?��O-H>I�e[\u0010��^r��z\u001F�O\ra�����m��v�t����S�n�\u0014\u0002�`���\u001B�{��~�����������\u0018�����\u0011\u0016J(���*�Uj+��&�T�\u001B\u0018>�\u0014]���=bK.��i;Z\u001E\u0019��H������@���\u0001\u0016e*�T����G�#B��\u0014a�R��5[@`�\u0018]�����M\u001F���a\b\u0007���\u000F����v�\u0011\f5\u001DZO������)��\u000E�&X(��"},"type":"beats"}

I've tried to run Filebeat but write beats into file and charset was decoded properly

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant