Update to Elastic 7.0.1

[kg-ops]

• Elasticsearch 7.0.1 Release Notes
• Kibana 7.0.1 Release Notes
• Logstash 7.0.1 Release Notes

Blog post

Closes #390

[sdellenb]

Just FYI - I've been running the stack on 7.0.0 since Friday and every night when Logstash wants to roll over the index, it stops working with this message:

[2019-04-14T00:00:01,445][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.04.14", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5b565b59>], :response=>{"index"=>{"_index"=>"logstash-2019.04.14", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [logstash-2019.04.14]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}

I searched my configuration up and down and through the containers but I cannot find a defaults mapping setting. Yesterday, I created the logstash-2019.04.13 index manually and it worked again until the roll over tonight.
The index looks like this:

GET logstash-2019.04.13

{
  "logstash-2019.04.13" : {
    "aliases" : { },
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "path_match" : "message",
            "match_mapping_type" : "string",
            "mapping" : {
              "norms" : false,
              "type" : "text"
            }
          }
        },
        {
          "string_fields" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              },
              "norms" : false,
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "facility" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "geoip" : {
          "dynamic" : "true",
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        },
        "host" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "norms" : false
        },
        "port" : {
          "type" : "long"
        },
        "procid" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "programname" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "severity" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "sysloghost" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "syslogtag" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "provided_name" : "logstash-2019.04.13",
        "creation_date" : "1555149990370",
        "number_of_replicas" : "1",
        "uuid" : "RAvoGCxPSHqu4H8zO5hrlw",
        "version" : {
          "created" : "7000099"
        }
      }
    }
  }
}

I'm quite new to Elastic Search and use it to mirror my rsyslogs to a searchable database (so nothing critical is lost), but I find it quite frustrating to debug such issues.

[antoineco]

@sdellenb did you start a new Stack or upgrade from 6.7? My gut feeling is that the Logstash index template may have changed between the 2 versions and needs to be replaced.

[sdellenb]

I upgraded from 6.7, with the same changes as this PR (except the port). I'll check the index template.

[antoineco]

@sdellenb the release notes mention the field reference parser is now more strict. Are you using a specific plugin to ship your logs to ship your logs to Logstash?

I also found this in the Elasticsearch 7.0.0 documentation: Removal of mapping types.

Sounds like we should wait for the first patch version before we merge this update to the repo :)

[sdellenb]

No plugin, just an rsyslog template generating a JSON for logstash.
I've managed to get it working again by updating the index template with a simplified version of the mappings of the current index as posted above. Thanks for the hint! 👍

On a different note regarding the upgrade to 7.0.0:
discovery.zen.minimum_master_nodes has been deprecated.
I'd have no idea how to address it, but so far it doesn't break anything.

[antoineco]

Thanks for the feedback, much appreciated 🙏 We'll have to address this in the default configuration indeed.

[antoineco]

@kg-ops 7.0.1 is out. Would you mind updating the PR so we can test the upgrade path from 6.7.x?