Overview
Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.
Here’s how Filebeat works: When you start Filebeat, it starts one or more inputs that look in the locations you’ve specified for log data. For each log that Filebeat locates, Filebeat starts a harvester. Each harvester reads a single log for new content and sends the new log data to libbeat, which aggregates the events and sends the aggregated data to the output that you’ve configured for Filebeat.
If you are getting below error while importing data into AWS Elasticsearch directly from Filebeat, then this post is for you!
Exiting: 1 error: error loading index pattern: returned 401 to import file: . Response: {“statusCode”:401,”error”:”Unauthorized”,”message”:”Authentication required”}
Exiting: 1 error: error loading index pattern: returned 401 to import file: . Response: {“statusCode”:401,”error”:”Unauthorized”,”message”:”Authentication required”}
This issue comes if you are approaching AWS Elasticsearch with username/password security as
setup.kibana:
host: “https://arun-learningsubway-abxybalglzl3zmkmiq4.ap-south-1.es.amazonaws.com:443/_plugin/kibana/”
output.elasticsearch:
protocol: https
hosts: [“arun-learningsubway-workapps-abxybalglzl3zmkmiq4.ap-south-1.es.amazonaws.com:9200”]
username: “myUsername”
password: “myPassword”
index: “nginx_index_by_arun”
Solution
In Aws, while configuring your Elasticsearch service configure it for whitelisting of IP instead of Master User.
or
Configure FileBeat–> Logstash–>Elasticsearch with master username/password also it will work.