Install test environment on Ubuntu with Elasticsearch, Logstash, Kibana and Filebeat

Get the GPG key:

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Save repository to /etc/apt/sources.list.d/elastic-7.x.list:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update your package lists:

sudo apt update

install Elasticsearch:

sudo apt install elasticsearch

Edit /etc/elasticsearch/elasticsearch.yml:

network.host: localhost
xpack.security.enabled: true

Start ES:

sudo systemctl start elasticsearch

Check status:

sudo systemctl status elasticsearch

Enable Elasticsearch to start up every time your server boots:

sudo systemctl enable elasticsearch

Set passwords for ES:

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

Test ES by sentding a request:

curl -u "elastic:xxxxxxxxxxxxxx" -X GET "localhost:9200"

Install Kibana:

sudo apt install kibana

Edit /etc/kibana/kibana.yml and set IP address where Kibana will be listening, also set the ES user/password pair:

server.host: "aa.bb.cc.dd"
elasticsearch.username: "kibana_system"
elasticsearch.password: "xxxxxxxxxxxxx"

Enable and start Kibana:

sudo systemctl enable kibana
sudo systemctl start kibana

check the Kibana server’s status page - login with elastic user:

http://aa.bb.cc.dd:5601/status

Install Logstash

sudo apt install logstash

Add /etc/logstash/conf.d/02-beats-input.conf:

input {
  beats {
    port => 5044
  }
}

create a configuration file /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  if [@metadata][pipeline] {
	elasticsearch {
  	hosts => ["localhost:9200"]
  	manage_template => false
  	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        pipeline => "%{[@metadata][pipeline]}"
        user => elastic
        password => xxxxxxxxxx
	}
  } else {
	elasticsearch {
  	hosts => ["localhost:9200"]
  	manage_template => false
  	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  	user => elastic
        password => xxxxxxxxxxx
	}
  }
}

Test your Logstash configuration

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

If everything is OK:

sudo systemctl start logstash
sudo systemctl enable logstash

Install Filebeat

sudo apt install filebeat

Edit /etc/filebeat/filebeat.yml:

filebeat.inputs:
...
  enabled: true
...
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
...
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Enable filebeat modules:

sudo filebeat modules enable system

Ingest pipelines:

sudo filebeat setup --pipelines --modules system

load the template:

sudo filebeat setup --index-management -E output.logstash.enabled=false -E output.elasticsearch.hosts=["localhost:9200"] -E output.elasticsearch.username=elastic -E output.elasticsearch.password=xxxxxxxxxxxx

Load dashboards into Kibana:

sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=aa.bb.cc.dd:5601 -E output.elasticsearch.username=elastic -E output.elasticsearch.password=xxxxxxxxxxxx

Now you can start and enable Filebeat

sudo systemctl start filebeat
sudo systemctl enable filebeat

Query ES indesx with data:

curl -u "elastic:xxxxxxxxxxxxxx" -X GET 'http://localhost:9200/filebeat-*/_search?pretty'

Next, go to Kibana - Discover and view your data