FileBeat and Logstash to insert Data into AWS Elasticsearch

FileBeat to insert Data into Logstash, and Logstash to insert Data into Elasticsearch

*Important point here is latest Elasticsearch version supported on AWS is 7.10, so Logstash and FileBeat must also be on same version.

If not then there will be a possibility of version compatibility.

* If latest version of ES available is x and you are not on cloud then also keep at least (x-1) version on production. It will keep you safe in production and away from product bugs to a lot extent.

Click and Download Filebeat 7.10 and Logstash7.10

Configuration of FileBeat to insert nginx logs into Logstash

Open filebeat.yml in any editor of your choice from location

/etc/filebeat/ on Linux or

C:\Program Files\filebeat-7.10.0 on windows

– paths:
– E:/nginx-1.20.1/logs/.log
input_type: log

enabled: true
path: ${path.config}/modules.d/*.yml

hosts: [“localhost:5044”]

Logstash Configuration

input {
beats {
port => 5044
ssl => false

filter {
grok {
match => [ “message” , “%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}”]
overwrite => [ “message” ]
mutate {
convert => [“response”, “integer”]
convert => [“bytes”, “integer”]
convert => [“responsetime”, “float”]
geoip {
source => “clientip”
target => “geoip”
add_tag => [ “nginx-geoip” ]
date {
match => [ “timestamp” , “dd/MMM/YYYY:HH:mm:ss Z” ]
remove_field => [ “timestamp” ]
useragent {
source => “agent”

output {
elasticsearch {
hosts => [“”]
index => “arun_nginx”
document_type => “%{[@metadata][type]}”
user => “myusername”
password => “mypassword”
manage_template => false
template_overwrite => false
ilm_enabled => false

Commands to Run on run Windows

To run Nginx
cd D:\nginx
start nginx
–to kill nginx process
taskkill /IM “nginx.exe” /F

To run Filebeat

To enable module
.\filebeat.exe modules enable nginx

C:\Program Files\filebeat-7.10.0> .\filebeat.exe -e

To run Logstash

C:\logstash> .\bin\logstash.bat -f .\config\logstash.conf

Author: Arun Singh

Learning is an Habit.