AWS EC2(Amazon Linux 2023)にfluent-bitをインストールする

AWS fluent-bit

タイトルの通りなのだが、妙なハマり方をしたのでメモ。

Install Commands

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

sudo systemctl start fluent-bit

ここまででインストールはできる。

Fix

しかし妙なエラーが出た。

systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
     Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; preset: disabled)
     Active: active (running) since Mon 2023-08-14 07:38:09 UTC; 42s ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 2732 (fluent-bit)
      Tasks: 7 (limit: 2131)
     Memory: 7.9M
        CPU: 216ms
     CGroup: /system.slice/fluent-bit.service
             └─2732 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [output:file:file.0] worker #0 started
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ warn] [aws_credentials] Failed to initialize profile provider: HOME, AWS_CONFIG_FILE, and AWS_SHARED>
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ warn] [aws_credentials] Failed to initialize profile provider: HOME, AWS_CONFIG_FILE, and AWS_SHARED>
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [output:kinesis_firehose:kinesis_firehose.2] worker #0 started
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [output:s3:s3.3] Using upload size 1000000 bytes
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [output:s3:s3.3] total_file_size is less than 10 MB, will use PutObject API
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ warn] [aws_credentials] Failed to initialize profile provider: HOME, AWS_CONFIG_FILE, and AWS_SHARED>
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [sp] stream processor started
Aug 14 07:38:09 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[2732]: [2023/08/14 07:38:09] [ info] [output:s3:s3.3] worker #0 started

7:38:09] [ warn] [aws_credentials] Failed to initialize profile provider: HOME, AWS_CONFIG_FILE, and AWS_SHAREDの部分である。

結論、こちらを参考にし、/usr/lib/systemd/system/fluent-bit.serviceを編集して環境変数を設定してやれば解決した。

[ec2-user@i-0a307d0088fc56aa3 fluent-bit]$ sudo cat /usr/lib/systemd/system/fluent-bit.service
[Unit]
Description=Fluent Bit
Documentation=https://docs.fluentbit.io/manual/
Requires=network.target
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/fluent-bit
EnvironmentFile=-/etc/default/fluent-bit
ExecStart=/opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf
Restart=always
Environment="ENVIRONMENT=development"
Environment="PROJECT=mybook"
Environment="HOME=/root"
Environment="AWS_CONFIG_FILE=/root/.aws/config"
Environment="AWS_SHARED_CREDENTIALS_FILE=/root/.aws/credentials"



[Install]
WantedBy=multi-user.target
[ec2-user@i-0a307d0088fc56aa3 fluent-bit]$

この通り。

[ec2-user@i-0a307d0088fc56aa3 fluent-bit]$ sudo systemctl status fluent-bit
● fluent-bit.service - Fluent Bit
     Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; preset: disabled)
     Active: active (running) since Mon 2023-08-14 07:54:30 UTC; 1s ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 3375 (fluent-bit)
      Tasks: 7 (limit: 2131)
     Memory: 7.9M
        CPU: 201ms
     CGroup: /system.slice/fluent-bit.service
             └─3375 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ warn] [filter:rewrite_tag:rewrite_tag.0] 'Match' may cause infinite loop.
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [input:emitter:re_emitted] initializing
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [input:emitter:re_emitted] storage_strategy='memory' (memory only)
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [output:file:file.0] worker #0 started
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [output:kinesis_firehose:kinesis_firehose.2] worker #0 started
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [output:s3:s3.3] Using upload size 1000000 bytes
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [output:s3:s3.3] total_file_size is less than 10 MB, will use PutObject API
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [sp] stream processor started
Aug 14 07:54:30 i-0a307d0088fc56aa3.ap-northeast-1.compute.internal fluent-bit[3375]: [2023/08/14 07:54:30] [ info] [output:s3:s3.3] worker #0 started
[ec2-user@i-0a307d0088fc56aa3 fluent-bit]$

ちなみにテストのためCloudwatch、S3、Kinesis Firehoseにログを出力する設定を入れていた。