Elastic Stack on Docker, with preconfigured security, tools, self-monitoring, and Prometheus Metrics Exporters
With tools like Curator, Rubban, ElastAlert for Alerting.
Elastic Stack (AKA ELK) Docker Composition, preconfigured with Security, Monitoring, Tools such as ElastAlert for alerting, Rubban and Curator.
Based on Official Elastic Docker Images
Stack Version: 7.7.0.
You can change Elastic Stack version by setting
.envfile and rebuild your images. Any version >= 7.0.0 is compatible with this template.
- Configured as Production Single Node Cluster. (With a multi-node option for experimenting).
- Security Enabled (under basic license).
- SSL Enabled for Transport Layer and Kibana.
- Use Docker-Compose and
.envto configure your entire stack.
- Automated Script that initializes and persist Elasticsearch's Keystore and SSL Certifications.
- Self-Monitoring Metrics Enabled.
- Prometheus Exporters for Stack Metrics.
- ElastAlert preconfigured for Alerting.
- Curator with Crond preconfigured for Automated Scheduled tasks (e.g Snapshots to S3).
- Rubban for Kibana curating tasks.
- Embedded Container Healthchecks for Stack Images.
More points at comparison with deviantony/docker-elk
Clone the Repository
git clone https://github.com/sherifabdlnaby/elastdocker.git
Initialize Elasticsearch Keystore and SSL Certificates
$ make setup
Start Elastic Stack
$ make elk ---- OR ---- $ docker-compose up -d
Visit Kibana at https://localhost:5601
ELASTIC_PASSWORD value in
.envfile for your needs, most importantly
ELASTIC_PASSWORDthat setup your superuser
LOGSTASH_HEAPfor Elasticsearch & Logstash Heap Size and
ELK_VERSIONfor, yk, Stack Version.
Notice that Kibana is configured to use HTTPS, so you'll need to write
localhost:5601in the browser.
To Start Monitoring and Prometheus Exporters
$ make monitoring
To Start Tools (ElastAlert, Rubban, and Curator)
$ make tools
To Start Elastic Stack, Tools and Monitoring
$ make all
To Start 2 Extra Elasticsearch nodes (recommended for experimenting only)
$ make nodes
To Rebuild Images
$ make build
Bring down the stack.
$ make down
- Some Configuration are parameterized in the
elastic's password (default:
ELK_VERSIONElastic Stack Version (default:
ELASTICSEARCH_HEAP, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
LOGSTASH_HEAP, how much Logstash allocate from memory.
- Other configurations which their such as cluster name, and node name, etc.
- Elasticsearch Configuration in
- Logstash Configuration in
- Logstash Pipeline in
- Kibana Configuration in
- ElastAlert Configuration in
- ElastAlert Alert rules in
./tools/elastalert/rules, head to ElastAlert docs to lookup how to create alerts.
- Curator Actions at
- Rubban Configuration using Docker-Compose passed Environment Variables.
Setting Up Keystore
You can extend the Keystore generation script by adding keys to
./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)
To Re-generate Keystore:
Enable SSL on HTTP
By default, only Transport Layer has SSL Enabled, to enable SSL on HTTP layer, add the following lines to
## - http xpack.security.http.ssl.enabled: true xpack.security.http.ssl.key: certs/elasticsearch.key xpack.security.http.ssl.certificate: certs/elasticsearch.crt xpack.security.http.ssl.certificate_authorities: certs/ca.crt xpack.security.http.ssl.client_authentication: optional
⚠️Enabling SSL on HTTP layer will require all clients that connect to Elasticsearch to configure SSL connection for HTTP, this includes all the current configured parts of the stack (e.g Logstash, Kibana, Curator, etc) plus any library/binding that connects to Elasticsearch from your application code.
Adding Two Extra Nodes to the cluster will make the cluster depending on them and won't start without them again.
Makefile is a wrapper around
make helpto know every command.
Elasticsearch will save its data to a volume named
Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the
./secretsdirectory by the setup command.
Make sure to run
make setupif you changed
ELASTIC_PASSWORDand to restart the stack afterwards.
For Linux Users it's recommended to set the following configuration (run as
sysctl -w vm.max_map_count=262144
By default, Virtual Memory is not enough.
Monitoring The Cluster
Via Prometheus Exporters
If you started Prometheus Exporters using
make monitoring command. Prometheus Exporters will expose metrics at the following ports.
|Prometheus Exporter||Port||Recommended Grafana Dashboard|
||Elasticsearch by Kristian Jensen|
||logstash-monitoring by dpavlos|
Head to Stack Monitoring tab in Kibana to see cluster metrics for all stack components.
In Production, cluster metrics should be shipped to another dedicated monitoring cluster.
One of the most popular elk on docker repositories is the awesome deviantony/docker-elk. Elastdocker differs from
deviantony/docker-elk in the following points.
Security enabled by default using Basic license, not Trial.
Persisting data by default in a volume.
Run in Production Mode (by enabling SSL on Transport Layer, and add initial master node settings).
Persisting Generated Keystore, and create an extendable script that makes it easier to recreate it every-time the container is created.
Parameterize credentials in .env instead of hardcoding
elastich:changemein every component config.
Parameterize all other Config like Heap Size.
Add recommended environment configurations as Ulimits and Swap disable to the docker-compose.
Make it ready to be extended into a multinode cluster.
Configuring the Self-Monitoring and the Filebeat agent that ship ELK logs to ELK itself. (as a step to shipping it to a monitoring cluster in the future).
Configured tools and Prometheus Exporters.
The Makefile that simplifies everything into some simple commands.
MIT License Copyright (c) 2020 Sherif Abdel-Naby
PR(s) are Open and Welcomed.