FileBeat to insert Data into Logstash, and Logstash to insert Data into Elasticsearch
*Important point here is latest Elasticsearch version supported on AWS is 7.10, so Logstash and FileBeat must also be on same version.
If not then there will be a possibility of version compatibility.
* If latest version of ES available is x and you are not on cloud then also keep at least (x-1) version on production. It will keep you safe in production and away from product bugs to a lot extent.
Click and Download Filebeat 7.10 and Logstash7.10
Configuration of FileBeat to insert nginx logs into Logstash
Open filebeat.yml in any editor of your choice from location
/etc/filebeat/ on Linux or
C:\Program Files\filebeat-7.10.0 on windows
filebeat:
inputs:
– paths:
– E:/nginx-1.20.1/logs/.log
input_type: log
filebeat.config.modules:
enabled: true
path: ${path.config}/modules.d/*.yml
output:
logstash:
hosts: [“localhost:5044”]
Logstash Configuration
input {
beats {
port => 5044
ssl => false
}
}
filter {
grok {
match => [ “message” , “%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}”]
overwrite => [ “message” ]
}
mutate {
convert => [“response”, “integer”]
convert => [“bytes”, “integer”]
convert => [“responsetime”, “float”]
}
geoip {
source => “clientip”
target => “geoip”
add_tag => [ “nginx-geoip” ]
}
date {
match => [ “timestamp” , “dd/MMM/YYYY:HH:mm:ss Z” ]
remove_field => [ “timestamp” ]
}
useragent {
source => “agent”
}
}
output {
elasticsearch {
hosts => [“https://arun-learningsubway-ybalglooophuhyjmik3zmkmiq4.ap-south-1.es.amazonaws.com:443”]
index => “arun_nginx”
document_type => “%{[@metadata][type]}”
user => “myusername”
password => “mypassword”
manage_template => false
template_overwrite => false
ilm_enabled => false
}
}
Commands to Run on run Windows
To run Nginx
cd D:\nginx
start nginx
–to kill nginx process
taskkill /IM “nginx.exe” /F
To run Filebeat
To enable module
.\filebeat.exe modules enable nginx
C:\Program Files\filebeat-7.10.0> .\filebeat.exe -e
To run Logstash
C:\logstash> .\bin\logstash.bat -f .\config\logstash.conf
