Since Version 6.6, Elasticsearch includes a feature called Index Lifecycle Management to implement detailed index retention policies.
In this post, I show how to use this feature for a very basic (and common) usecase:
How to delete old logging data from an ELK stack
In the following, I assume that the Elasticsearch cluster contains data created by Logstash and Filebeat, and that the Elasticsearch API is available at
http://elasticsearch:9200. In case you have enabled authentication, just add “
-u username:password” to the
1. Create a policy that deletes indices after one month
2. Apply this policy to all existing filebeat and logstash indices
3. Create a template to apply this policy to new filebeat and logstash indices
That’s it! From now on, all data that is older than 30 days will be deleted. Depending on the size of the data, this background operation can take some time.
A final note
If you change the policy (e.g. deleting after 60 instead of 30 days), these changes will not be applied to existing indices. The easiest way to enforce a new policy version on existing indices is to remove and re-add the policy, e.g.comments powered by Disqus