How to send rsyslog to S3 using Fluentd

When you have a server you want to get metric, logs and data to measure the server performance, in this blog post we’ll send rsyslog to S3 and we’ll use fluentd.

Create IAM user

You’ll need a IAM user to send data to S3 so that user will need permissions to do so.

create the new user and use it only for this purpose so you can manage your server, S3, IAM for this project.

Create S3 bucket policy with full permissions ONLY to the specific bucket that is intended for the data/logs.

Install Ruby and awscli

apt-get install ruby-full

install development tools

apt-get install build-essential

install awscli

apt-get install python3-pip

pip3 install awscli

Install fluentd

gem install fluentd

create the configuration file

fluentd --setup /etc/fluent

edit conf file (sample file here)

vi /etc/fluent/fluent.conf

Send log messages to Fluentd

edit

vi /etc/rsyslog.conf

add

*.* @spinningops.com:5140

restart rsyslog

systemctl start rsyslog

check status

systemctl status rsyslog

Start fluentd

fluentd -c fluentd.conf

for the full install please check this GitHub LINK