Dynamic Template Generator for Elasticsearch
One of the problems with setting explicit mappings in Elasticsearch templates is the fields show up in drop-down lists when selecting fields in Kibana, even when the index contains no data for the field. This makes for a poor user experience as operators have to search in Discover to find out whether the index currently contains any data for a field before adding it to a Kibana Visuals, Discover Filter, Machine Learning job or Data Transform.
https://github.com/mbudge/dynamic-template-generator
Beats Agent Templates
Beats sets explicit mappings for hundreds of fields. A good example can be seen in the default template exported from Winlogbeat, which sets the mappings for over 800 different fields. On average a much smaller number of these fields will have data indexed in the Winlogbeat index.
The beats mappings also make it difficult to build generic aliases which point to multiple feeds. This is because when you create the generic index pattern from the alias, Kibana requests the index mapping from all the target indexes in a single request. There is a limit on how many index mappings Kibana can process when creating a new index pattern.
Case In-Sensitive Keyword Search
Beats makes heavy use of keyword fields which are exact match and case-sensitive in Elasticsearch. This means the operator has to remember the case of the string they are searching, otherwise it’s possible some or all documents won’t be returned in the search results. Cyber-security use cases nearly always require case in-sensitive search to make sure all logs are returned during incident response. This is to mitigate the risk of important logs being missed which are linked to malicious activity the operator is investigating. When collecting events from many different systems on the network, it’s nearly always impossible to guarantee the data will arrive pre-normalised.
Dynamic Template Generator
To fix these issues I have written Dynamic Template Generator, a Python script which can convert explicit mappings to dynamic templates. It does this by setting the path to the field in the dynamic templates path_match parameters before setting the explicit mapping in the dynamic templates mapping definition. The script can also apply an uppercase or lowercase normaliser to all keyword fields which exist in the mapping. If you want to add the case normaliser to all keyword fields without converting the mappings to dynamic templates, that is supported too.
The advantage of using dynamic templates is the field is only created in the index mapping when data to populate the field is indexed. This means fields will only show up in the field selection drop-downs in Kibana when searchable data exists. Operators no longer have to search the index to work out what fields to add to the Kibana Visuals, Data Transform etc. The only drawbacks are you will have to wait until existing indexes are deleted or removed from the alias, and using the generator creates manual steps which might prevent you from using Fleet to manage the Beats agent indexes.
Originally published at https://securityengineering.io on November 24, 2020.
