PREPARATIONS
#Ref: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
First install Java 8 in Ubuntu 14.04
# Ref: https://www.liquidweb.com/kb/how-to-install-oracle-java-8-on-ubuntu-14-04-lts/
apt-get install python-software-properties software-properties-common
apt-add-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer
java -version
Result:
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
Facilitate updating of all packages via APT repositories
apt-get install apt-transport-https
Save the repository definition to /etc/apt/sources.list.d/elastic-5.x.list:
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
apt-get update
FILEBEAT
Installing filebeat
Filebeat reads lines from defined logs, formats them properly and forwards them to logstash while maintaining a non-clogging pipeline stream
Ref: https://github.com/elastic/beats/tree/master/filebeat
Ref: https://www.elastic.co/guide/en/beats/filebeat/5.5/filebeat-getting-started.html
Ref: https://www.elastic.co/products/beats/filebeat
apt-get install filebeat
mv /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.orig
touch /etc/filebeat/filebeat.yml
mcedit /etc/filebeat/filebeat.yml
(content)
————————
filebeat.prospectors:
- input_type: log
paths:
- /var/log/apache2/access.log
output.logstash:
hosts: ["localhost:5044"]
————————
service filebeat restart
LOGSTASH
Download logstash debian install package and configure it
# Ref: https://www.elastic.co/downloads/logstash
apt-get install logstash
# Result:
.......
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
Preparing Logstash
mcedit /etc/logstash/startup.options
(add the following line at the beginning)
LS_CONFIGS_DIR=/etc/logstash/conf.d/
(Then adjust the following line as follows)
from:
LS_OPTS="--path.settings ${LS_SETTINGS_DIR}"
to:
LS_OPTS="--path.settings ${LS_SETTINGS_DIR} --path.config ${LS_CONFIGS_DIR}"
Start/Stop/Restart logstash
service logstash {start|stop|restart}
Testing logstash
cd /etc/logstash/ ; /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Type: hello world
and and press CTRL-D
(Logstash adds timestamp and IP address information to the message. Exit Logstash by issuing a CTRL-D command in the shell where Logstash is running.)
Results:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
11:22:59.822 [[main]-pipeline-manager] INFO logstash.pipeline – Starting pipeline {“id”=>”main”, “pipeline.workers”=>2, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>5, “pipeline.max_inflight”=>250}
11:22:59.847 [[main]-pipeline-manager] INFO logstash.pipeline – Pipeline main started
The stdin plugin is now waiting for input:
2017-08-23T09:22:59.878Z h270746.stratoserver.net test 1
11:22:59.946 [Api Webserver] INFO logstash.agent – Successfully started Logstash API endpoint {:port=>9601}
11:23:02.861 [LogStash::Runner] WARN logstash.agent – stopping pipeline {:id=>”main”}
The errors and warnings are OK for now. The main result line above that is significant is:
2017-08-23T09:22:59.878Z h270746.stratoserver.net test 1
which adds a timestamp and server name to the input string (test 1)
Configuring logstash
# Note: this test configuration will get input from filebeat and output into a log file which can be watched with tail -f …..
mcedit /etc/logstash/conf.d/apache2.conf
(content)
input {
beats {
port => 5044
type => "apache"
}
}
filter {
if [type] == "apache" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
}
output {
file {
path => "/var/log/logstash_output.log"
}
}
In order to have the proper output sent to elasticsearch then use this output configuration instead:
———————————-
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Securing Filebeat => Logstash with SSL
Ref: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html#configuring-ssl-logstash
Note: Typing by hand below is shown in bold.
Prepare the ertificates directories:
mkdir -p /etc/logstash/certs/Logstash/ /etc/logstash/certs/Beats/
Create client certificates for FileBeat:
/usr/share/elasticsearch/bin/x-pack/certgen
........
Let's get started...
Please enter the desired output file [/etc/elasticsearch/x-pack/certificate-bundle.zip]: /etc/logstash/certs/Beats/certificate-bundle_Beats.zip
Enter instance name: Beats
Enter name for directories and files : Beats
Enter IP Addresses for instance (comma-separated if more than one) []:
Enter DNS names for instance (comma-separated if more than one) []: localhost
Certificates written to /etc/logstash/certs/Beats/certificate-bundle_Beats.zip
Create client certificates for Logstash:
/usr/share/elasticsearch/bin/x-pack/certgen
........
Let's get started...
Please enter the desired output file [/etc/elasticsearch/x-pack/certificate-bundle.zip]: /etc/logstash/certs/Logstash/certificate-bundle_Logstash.zip
Enter instance name: Logstash
Enter name for directories and files : Logstash
Enter IP Addresses for instance (comma-separated if more than one) []:
Enter DNS names for instance (comma-separated if more than one) []: localhost
Certificates written to /etc/logstash/certs/Logstash/certificate-bundle_Logstash.zip
This file should be properly secured as it contains the private keys for all
instances and the certificate authority.
After unzipping the file, there will be a directory for each instance containing
the certificate and private key. Copy the certificate, key, and CA certificate
to the configuration directory of the Elastic product that they will be used for
and follow the SSL configuration instructions in the product guide.
For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
Extract certificates:
unzip /etc/logstash/certs/Beats/certificate-bundle_Beats.zip -d /etc/logstash/certs/Beats/
unzip /etc/logstash/certs/Logstash/certificate-bundle_Logstash.zip -d /etc/logstash/certs/Logstash/
Convert the Logstash key Logstash.key from PKCS#1 to PKCS#8 format:
Reason: the following error message in the logstash.log occured when using the PKCS1 format:
[ERROR][logstash.inputs.beats ] Looks like you either have an invalid key or your private key was not in PKCS8 format. {:exception=>java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/certs/Logstash/Logstash/Logstash.key}
See: https://github.com/spujadas/elk-docker/issues/112
Command:
openssl pkcs8 -in /etc/logstash/certs/Logstash/Logstash/Logstash.key -topk8 -nocrypt -out /etc/logstash/certs/Logstash/Logstash/Logstash.key.PKCS8
Configure Beats for SSL
Content of /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/apache2/access.log
output.logstash:
hosts: ["localhost:5044"]
ssl.certificate_authorities: ["/etc/logstash/certs/Logstash/ca/ca.crt"]
ssl.certificate: "/etc/logstash/certs/Beats/Beats/Beats.crt"
ssl.key: "/etc/logstash/certs/Beats/Beats/Beats.key"
Content of /etc/logstash/conf.d/apache.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/logstash/certs/Logstash/ca/ca.crt"]
ssl_certificate => "/etc/logstash/certs/Logstash/Logstash/Logstash.crt"
ssl_key => "/etc/logstash/certs/Logstash/Logstash/Logstash.key.PKCS8"
ssl_verify_mode => "force_peer"
}
}
Restart both Logstash and Filebeat
service logstash restart
service filebeat restart
NOTE: I’m still having problems with the SSL connection of Filebeat to Logstash where Filebeat throws this error in (/var/log/logstash/logstash-plain.log):
TLS internal error.
The following URL seems to have found some similar problems but because of lack of time I haven’t figured it out yet.
https://discuss.elastic.co/t/mutual-tls-filebeat-to-logstash-fails-with-remote-error-tls-internal-error/85271/3
X-Pack for Logstash
INSTALL X-Pack for logstash
X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, machine learning, and graph capabilities into one easy-to-install package.
X-Pack also provides a monitoring UI for Logstash.
/usr/share/logstash/bin/logstash-plugin install x-pack
Result:
Downloading file: https://artifacts.elastic.co/downloads/logstash-plugins/x-pack/x-pack-5.5.2.zip
Downloading [=============================================================] 100%
Installing file: /tmp/studtmp-bc1c884de6d90f1aaa462364e5895b6b08b050f0b64587b4f5e0a8ec5300/x-pack-5.5.2.zip
Install successful
Configuring X-Pack in Logstash:
The defaults settings created during the installation works best for most cases. For more information see:
https://www.elastic.co/guide/en/logstash/5.5/settings-xpack.html
To Prevent generation of monitoring error messages in logstash.log edit /etc/logstash/logstash.yml and add the following line at the end:
(Ref: https://discuss.elastic.co/t/logstash-breaks-when-disabling-certain-x-pack-features/89511)
xpack.monitoring.enabled: false
ElasticSearch
Installation:
apt-get install elasticsearch
Start/Stop/Restart Elastic search:
/etc/init.d/elasticsearch {start|stop|restart}
To check if elasticsearch has been started:
ps aux | grep $(cat /var/run/elasticsearch/elasticsearch.pid)
Example of result(truncated):
elastic+ 10978 3.2 55.2 4622152 2319168 pts/3 Sl 15:44 0:10 /usr/lib/jvm/java-8-oracle/bin/java ........
The check the Elasticsearch log file:
tail -f /var/log/elasticsearch/elasticsearch.log
NOTE 1:
If you see the line:
[WARN ][o.e.b.BootstrapChecks ] [wJdCtOd] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
and the result of the following command is empty,
grep vm.max_map_count /etc/sysctl.conf
Solution:
Raise the max virtual memory areas vm.max_map_count to 262144 as follows:
Add the following line in the file /etc/sysctl.conf
vm.max_map_count=262144
And run the command:
sysctl -w vm.max_map_count=262144
OR
echo 262144 > /proc/sys/vm/max_map_count
ALSO Make sure the elasticsearch config file (/etc/elasticsearch/jvm.options) has the following entries:
-Xms2g
-Xmx2g
IMPORTANT:
if the following commands are failing it might be because some Virtual Servers are not allowing such changes in the kernel:
eg.
sysctl -w vm.max_map_count=262144
sysctl: permission denied on key ‘vm.max_map_count’
echo 262144 > /proc/sys/vm/max_map_count
-bash: /proc/sys/vm/max_map_count: Permission denied
Elastic search should be able to run anyway but might be limited in performance and may have other issues because of these limitations.
There is no known remedies to this for Strato VM servers.
NOTE 2:
If you see the line:
[WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: ……..
Solution:
No need to worry, the accuracy of the MAC address is not so important in this installation.
NOTE 3:
If you see the line:
[WARN ][o.e.b.BootstrapChecks ] [wJdCtOd] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
If this problem occurs elasticsearch will start but not get initialised properly and most likely not function properly.
Solution:
If elasticsearch is accessed only in a protected environment, disabling this installation of system call filters should be no problem
by editing the file /etc/elasticsearch/elasticsearch.yml and adding the following line:
bootstrap.system_call_filter: false
Restart elasticsearch:
service elasticsearch restart
————————————————————————
X-Pack for elasticsearch
X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, machine learning, and graph capabilities into one easy-to-install package.
Installation:
/usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
Results:
-> Downloading x-pack from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission \\.\pipe\* read,write
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
* java.util.PropertyPermission sun.nio.ch.bugLevel write
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
Continue with installation? [y/N]y
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin forks a native controller @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.
Continue with installation? [y/N]y
-> Installed x-pack
KIBANA
Install kibana package
apt install kibana
Install X-Pack for logstash
X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, machine learning, and graph capabilities into one easy-to-install package.
/usr/share/kibana/bin/kibana-plugin install x-pack
Change built-in users password
Ref: https://www.elastic.co/guide/en/x-pack/5.5/setting-up-authentication.html#reset-built-in-user-passwords
change passwords
curl -XPUT 'localhost:9200/_xpack/security/user/elastic/_password?pretty' -H 'Content-Type: application/json' -d'
{
"password": "elasticpassword"
}
'
curl -XPUT 'localhost:9200/_xpack/security/user/kibana/_password?pretty' -H 'Content-Type: application/json' -d'
{
"password": "kibanapassword"
}
'
curl -XPUT 'localhost:9200/_xpack/security/user/logstash_system/_password?pretty' -H 'Content-Type: application/json' -d'
{
"password": "logstashpassword"
}
'
Update the Kibana server with the new password /etc/kibana/kibana.yml
elasticsearch.password: kibanapassword
Update the Logstash configuration with the new password /etc/logstash/logstash.yml
xpack.monitoring.elasticsearch.password: logstashpassword
Disable Default Password Functionality /etc/elasticsearch/elasticsearch.yml
xpack.security.authc.accept_default_password: false
Start/Stop/Restart kibana
service kibana {start|stop|restart}