PART2-INSTALL ELASTICSEARCH 6 CLUSTER FOR CENTRALIZED SYSLOG

https://www.youtube.com/watch?v=gjhhra19YsA

If you have not read the PART1 document, check below.

https://allmnet.tistory.com/entry/PART-1-INSTALL-ELASTICSEARCH-6-CLUSTER-FOR-CENTRALIZED-SYSLOG

PART 1 INSTALL ELASTICSEARCH 6 CLUSTER FOR CENTRALIZED SYSLOG

Describes how to centralize syslog generated or received logs, especially those known as rsyslog. By centralizing this data, you can more easily track security audits, application behavior monitoring, and other important server information.

Setup rsyslog

Now, let’s configure rsyslog to collect data in syslog format.

sudo nano /etc/rsyslog.conf

Enable imudp

# provides UDP syslog reception

$ModLoad imudp

$UDPServerRun 514

After saving the configuration file and restarting rsyslog again, you are ready to receive remote syslogs You can use netstat | grep 514 command to check network communication status.

If the firewall is enabled, please check the link below and open port udp 514.

https://asecurity.dev/2017/12/ubuntu-firewall-setting-by-ufw/

sudo service rsyslog restart

Make logdata rsyslog json template

Elasticsearch should change the data collected in rsyslog to JSON because all documents must be received in JSON format.

To do this, first create a json template in rsyslog, and create a json template for the syslog format.

sudo nano /etc/rsyslog.d/01-json-template.conf

It is convenient to copy and use the following contents.

template(name="json-template" 

type="list") { 

constant(value="{") 

constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339") 

constant(value="\",\"@version\":\"1") 

constant(value="\",\"message\":\"") property(name="msg" format="json") 

constant(value="\",\"sysloghost\":\"") property(name="hostname") 

constant(value="\",\"severity\":\"") property(name="syslogseverity-text") 

constant(value="\",\"facility\":\"") property(name="syslogfacility-text") 

constant(value="\",\"programname\":\"") property(name="programname") 

constant(value="\",\"procid\":\"") property(name="procid") 

constant(value="\"}\n") }

Forware to log data(json format) on elasticsearch

Now that you have created the template file, configure it to export the rsyslog data.

sudo nano /etc/rsyslog.d/60-output.conf

It is convenient to copy and use the following contents.

# This line sends all lines to defined IP address at port 10514,

# using the "json-template" format template

*.* @private_ip_logstash:10514;json-template

Install Logstash and configuration

Now the task of converting the syslog to json and sending it is complete. Now let’s proceed with a configuration that uses Logstash to send data from rsyslog to Elasticsearch.

First install Logstash.

sudo apt-get install logstash

 

Open and edit the default configuration file to receive Rsyslog messages.

sudo nano /etc/logstash/conf.d/logstash.conf

 

 

It is convenient to copy and use the following contents.

# This input block will listen on port 10514 for logs to come in.

# host should be an IP on the Logstash server.

# codec => "json" indicates that we expect the lines we're receiving to be in JSON format

# type => "rsyslog" is an optional identifier to help identify messaging streams in the pipeline.

input {

     udp {

     host => "logstash_private_ip"

     port => 10514

     codec => "json"

     type => "rsyslog"

     }

}

# This is an empty filter block. You can later add other filters here to further process

# your log lines

filter { }

# This output block will send all events of type "rsyslog" to Elasticsearch at the configured

# host and port into daily indices of the pattern, "rsyslog-YYYY.MM.DD"

output {

     if [type] == "rsyslog" {

          elasticsearch {

               hosts => [ "elasticsearch_private_ip:9200" ]

          }

     }

}

You are ready to create and send inputs and outputs to Elasticserch through the configuration.

You can check the environment configuration by using the configtest command.

sudo service logstash configtest

If you look display “Configuration OK” there are no syntax errors. Try to start logstash and rsyslog

sudo service logstash start

sudo service rsyslog restart

Now you can check with Elasticsearch using the curl command to see if the logs are collected properly.

curl -XGET http://elasticsearch_private_ip:9200/_all/_search