postfix:cleanup subject contains a url

[rhysxevans]

Hi

Firstly thanks for the patterns. it is appreciated.

I am having an issue when the "warning" (we add the subject so we can easily find the email) contains strange characters such as below

May 25 11:59:15 mail4 postfix/cleanup[2185]: D8B07E3DB6: warning: header Subject: https://drive.google.com/file/d/0B8wxcvprDYVdlVsdf1kzOVk/view?usp=sharing from o1678917x173.outbound-mail.sendgrid.net[167.89.17.173]; from=<bounces+2320708-7653-frank.test=hsdjasdd.co.uk@sendgrid.net> to=<frank.test@hsdjasdd.co.uk> proto=ESMTP helo=<o1678917x173.outbound-mail.sendgrid.net>

May 25 12:27:10 mail postfix/cleanup[10485]: AF35455A2C: warning: header Subject:  =?UTF-8?Q?ID&A_Awards_2016:_Bathroom_Over_=C2=A3100,000_Award_Coming_Soon?=? =?UTF-8?Q?...?[216.27.86.143]; from=<bounces+724240.47095598.479811@icpbounce.com> to=<frank.fred@fred.com> proto=ESMTP helo=<drone086.ral.icpbounce.com>

Essentially we get

:response=>{"create"=>{"_index"=>"logstash-2016.05.25", "_type"=>"syslog", "_id"=>"AVTnkYeOykeme6L6JtYY", "status"=>400, 
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field name [postfix_https://drive.google.com/file/d/0B8wd8NTtsprDYVdlV29iY1kzOVk/view?usp] cannot contain '.'"}}}, :level=>:warn}

"[tags]"=>[{"message"=>"<22>May 25 12:27:10 mail postfix/cleanup[10485]: AF35455A2C: warning: header Subject:  =?UTF-8?Q?IDOver_=C2=A3100,000_Award_Coming_Soon?=? =?UTF-8?Q?...?= from drone086.ral.icpbounce.com[216.27.86.143]; from=<bounces+724240.47095598.479811@icpbounce.com> to=<frank.sawkins@czechandelo=<drone086.ral.icpbounce.com>", "@version"=>"1", "@timestamp"=>"2016-05-25T11:26:50.330Z", "host"=>"127.0.0.1", "port"=>37311, "type"=>"syslog", "program"=>["postfix/cleanup", "po_queueid"=>"AF35455A2C", "tags"=>["_grok_postfix_success"], "postfix_Subject:"=>"?UTF-8?Q?ID&A_Awards_2016:_Bathroom_Over_=C2=A3100000_Award_Coming_Soon?=?", "postfix_?UTF-8?Q?...?"="bounces+724240.47095598.479811@icpbounce.com", "postfix_to"=>"frank.fred@fred.com", "postfix_proto"=>"ESMTP", "postfix_helo"=>"drone086.ral.icpbounce.com"}, "tags"]}>>]"_index"=>"logstash-2016.05.25", "_type"=>"syslog", "_id"=>"AVTnqspCykeme6L6LenL", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field name [postfix_?UTF-8?'"}}}, :level=>:warn}

This seems to only be happening on the CLEANUP messages.

Is there a way that we can get these formatted correctly ? (some thing like if contains warning: header Subject: the reset is the data for postfix_subject)

Any help is greatly appreciated ?

Thanks

[whyscream]

Looks like a good example for this issue, wasn't able to reproduce this. I'll try to look into it (after the weekend, i'm afraid)

[rhysxevans]

no problems and thanks

[rhysxevans]

Been messing about a bit; I think this is a KV issue with the '=' character

I have also being playing with some regex in host OS and this seems to be along the lines of what I would look for the postfix_subject

cat /var/log/maillog | grep postfix/cleanup | awk -F'Subject:' '{print $2}' | awk -F';' '{ print $1 }' | awk '{ FS = " from (([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-][a-zA-Z0-9]).)([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9-][A-Za-z0-9])[[0-9]." } ; { print $1 }'

I don't profess to know what I am talking about, so the above may be completely and utterly wrong.

Requires "/^Subject:/ WARN" in the postfix headers file and "header_checks = regexp:/etc/postfix/header_checks" in the main.cf

[rhysxevans]

Ok, some more tinkering

I have done the following:

1. Added the following lines to the patterns file
   POSTFIX_SUBJECT_CLEANUP_KEYVALUE %{POSTFIX_QUEUEID:postfix_queueid}: warning: header Subject: %{GREEDYDATA:postfix_subject} from %{HOSTNAME}[%{IP}]; %{GREEDYDATA:postfix_keyvalue_data}
   POSTFIX_SUBJECT_CLEANUP %{POSTFIX_CLEANUP_MILTER}|%{POSTFIX_WARNING}|%{POSTFIX_SUBJECT_CLEANUP_KEYVALUE}

2. in the input filter I have modified the cleanup section to look like

} else if [program] =~ /^postfix.*\/cleanup$/ {
    #Test For Subject in message
    if [message] =~ "warning: header Subject:" {
    grok {
        patterns_dir   => "/etc/logstash/patterns"
        match          => [ "message", "%{POSTFIX_SUBJECT_CLEANUP}" ]
        tag_on_failure => [ "_grok_postfix_cleanup_nomatch" ]
        add_tag        => [ "_grok_postfix_success" ]
    }
    } else {
    grok {
        patterns_dir   => "/etc/logstash/patterns"
        match          => [ "message", "%{POSTFIX_CLEANUP}" ]
        tag_on_failure => [ "_grok_postfix_cleanup_nomatch" ]
        add_tag        => [ "_grok_postfix_success" ]
    }
  }
} else if [program] 

It seems to work for me. I presume there is a much neater solution available

Thanks

[whyscream]

Solved issue in PR #85. Thanks for reporting!