Log rotate

[NiranjanHD]

Hi team,

I am not able to logrotate logs captured from source application, below are the things i have setup.

currently using fluentd:1.14.x version

Background: I have setup fluentd in kubernetes environment and able to filter out based on attributes which i have configured.

Issue: As my application is not directly generating logs in the application log path, we are pulling out the logs from hostPath of kubernetes-pod-container. Below is my configmap created for fluent.conf to be mounted as a sidecar with the application.

apiVersion: v1
kind: ConfigMap
metadata:
  name: scheduler-fluentd
  labels:
     name: scheduler-fluentd
data:
  fluent.conf: |-
    <source>
        @type tail
        path /var/log/containers/*superdb-airflow-2-scheduler*.log
        path_key dpd-superdb-airflow-2-scheduler_logs
        read_from_head true
        format multiline
        format_firstline /^{"log":/
        format1 /.{81}\s(?<level>\w+)(?<message>.*)/
        tag airflow-scheduler
        pos_file /mnt/scheduler.log.pos
    </source>
    <system>
    <log>
      rotate_size 1024
    </log>
    </system>
    <match airflow-scheduler>
        @type copy
        <store>
          @type relabel
          @label @airflow_warn
        </store>
        <store>
          @type relabel
          @label @airflow_error
        </store>
        <store>
          @type relabel
          @label @airflow_fatal
        </store>
    </match>
    <label @airflow_warn>
       <filter airflow-scheduler>
         @type grep
         regexp1 level WARN
       </filter>
      <filter  airflow-scheduler> 
        @type record_transformer
        <record>
         tag ${tag}
         severity WARNING
        </record>
      </filter>
      <match airflow-scheduler>
        @type google_cloud
        project_id "#{ENV['project_id']}"
        zone asia-northeast1
        vm_name "#{ENV['AIRFLOW_SCHD_HOSTNAME']}"
        vm_id dwt-airflow-scheduler
        use_metadata_service false
        buffer_type memory
        buffer_chunk_limit 1m
        flush_interval 5s
      </match>
    </label>

    <label @airflow_error>
       <filter airflow-scheduler>
         @type grep
         regexp1 level ERROR
       </filter>
      <filter  airflow-scheduler> 
        @type record_transformer
        <record>
         tag ${tag}
         severity ERROR
        </record>
      </filter>
      <match airflow-scheduler>
        @type google_cloud
        project_id "#{ENV['project_id']}"
        zone asia-northeast1
        vm_name "#{ENV['AIRFLOW_SCHD_HOSTNAME']}"
        vm_id dwt-airflow-scheduler
        use_metadata_service false
        buffer_type memory
        buffer_chunk_limit 1m
        flush_interval 5s
      </match>
    </label>

    <label @airflow_fatal>
       <filter airflow-scheduler>
         @type grep
         regexp1 level FATAL
       </filter>
      <filter  airflow-scheduler> 
        @type record_transformer
        <record>
         tag ${tag}
         severity FATAL
        </record>
      </filter>
      <match airflow-scheduler>
        @type google_cloud
        project_id "#{ENV['project_id']}"
        zone asia-northeast1
        vm_name "#{ENV['AIRFLOW_SCHD_HOSTNAME']}"
        vm_id dwt-airflow-scheduler
        use_metadata_service false
        buffer_type memory
        buffer_chunk_limit 1m
        flush_interval 5s
      </match>
    </label>

[fujimotos]

Fluentd does not support rotating other apps' logs.
You need to set up logrotate for that purpose.

[NiranjanHD]

thanks @fujimotos, apologies for the delay in response. following i have one more query. when i exec into the fluentd container the position file which i have set above having 4kb /mnt/scheduler.log.pos and path where the logs need to capture /var/log/containers/ having 4GB, but when i monitor inside container where the volume is mounted the space increases very rapidly maybe in a minute 4GB consumption. can you brief me what causing this ?

[fujimotos]

when i exec into the fluentd container the position file which i have set above having 4kb /mnt/scheduler.log.pos and path where the logs need to capture /var/log/containers/ having 4GB, but when i monitor inside container where the volume is mounted the space increases very rapidly maybe in a minute 4GB consumption. can you brief me what causing this ?

I'm not totally sure what the exact question you're asking is,
but my psychic power suggests two possibilities:

1. Your app is a extremely heavy logger, and it's producing log records at
   a rate of 4GB per minute.
2. You inadvertently set up Fluentd to consume its own logs. So Fluentd
   entered into an infinite loop, because it can produce anothor set of logs
   while processing log records.

[NiranjanHD]

1st point is correct. 2nd one is not. As you see in the above configuration for configmap in kubernetes, source section where path is having sever host path, that hostpath directory will be mount in the kubernetes pod container.

when i logged into the container and went into the path defined above i could only see 4GB consumed out of 500GB in the directory mount.

How did i know ?

• in the container i ran du -sh /var/log/containers which gave 4GB space consumption and no change even after a 3 hours monitoring but when i run du -hT in the container i see directory mounted to the volume consuming each minute 3/4 GB of space out of 500GB, once it reaches it limits pod is evicting in the Kubernetes.

[fujimotos]

The probability is that log files in /var/log/containers are actually symbolic
links to somewhere, and you forgot to mount the source directory.

If that is the case, running du inside the container would report singificantly
small disk usage because it cannot see the target files.

[NiranjanHD]

again sorry for the delay in response. yes /var/log/containers are symbolic and it is pointing to /var/log/pods, I have checked the folders under that I can see all are below 1MB.