Permission denied in cloudlogs agent pod logs with ROKS 4.16 / 4.17 but OK with ROKS 4.15

[anthonylecarrer]

Hello

I'm posting here because you are maintaining source code to be able to deploy cloudlogs agent on IBMCloud platform

I have an issue while deploying CloudLogs Agent on my clusters. Pods are OK and running. Logs are OK for ROKS 4.15 but full off permission denied while running with ROKS 4.16 / 4.17.

Il is the same behaviour with different versions of cloudlogs agent (1.4.2 / 1.6.1 / 1.6.2)

Logs become OK with ROKS 4.16 / ROKS 4.17 if i change the security constraints of the cloudlogs agent daemon set but using helm release i can not change this parameter.

Could you please help me understanding the difference of behaviour while deploying ROKS 4.15 / ROKS 4.17 with the same code / the same parameters and the same version of cloudlogs agent. Have you encountered this behaviour ? Have you checked logs after deploying and not only pods status ?

See attached document for :

• Deployment code (lines 477-612) : main.tf.txt
• cloudlogs agent logs on ROKS 4.15 : logs_roks-415.txt
• cloudlogs agent logs on ROKS 4.17 : logs_roks-417.txt

Regards

[ocofaigh]

Hi @anthonylecarrer I have forwarded the logs onto the Cloud Logs team.

FYI, this module is currently marked as deprecated in favour of these:

• terraform-ibm-monitoring-agent
• terraform-ibm-logs-agent

I don't think it will make a difference here to the error you are seeing, but I would strongly suggest you ensure to use these modules going forward instead as they are the ones we will be maintaining. See the top of the README for migration steps if needed.

[anthonylecarrer]

@ocofaigh i was sure i was posted on the right github repo. thank you for all. I have created the issue on the terraform logs agent repo even if i'm not using it but kind of code really similar to be sure they dont meet the same issue. Actually i dont want to migrate to ibm repo even if my code is quite similar. I just want to understand why i have this issue with ROKS 4.16 / 4.17 and why it works with ROKS 4.15 deploying the same terraform code for the 3 versions .... I also have a support ticket opened on my ibmcloud factory account where i deploy and test my modules and my DAs before offering them to my customers

[anthonylecarrer]

@ocofaigh why this repo is not archived as terraform-ibm-observability-instances ? Isn't it the best way to alert customers about product indisponibility ? My first reflex is not to go and read the README as the product seems to be alive

[ocofaigh]

@anthonylecarrer We are still maintaining both repos currently, hence we have not archived it. I'm just giving you a heads up that eventually it will be archived.

[ocofaigh]

@anthonylecarrer FYI the Cloud Logs team said that this is a known issue with ROKS, and they are awaiting a fix from them. However they said that logs still go through like normal and it should be working regardless of the error in the logs. Can you confirm this is the case?

[anthonylecarrer]

@ocofaigh it seems that logs still go through like normal but it is difficult to find logs in the middle of millions of critical or error messages due to that issue. Millions of critical or error messages cause an increase of cloudlogs cost

[ocofaigh]

@anthonylecarrer Here is an update I got from Cloud Logs team:

We've been advising people to exclude the agent, or if they modify the daemonset to be privilidged or modify the SCC to have the SELinux role as spc_t that will eliminate the messages. Note that modifying the daemonset or SCC might not be doable for everyone based on their security requirements.
As of right now it looks like the RHEL 9.6 release date is set as Oct 6th. So we would want to make them aware that they would want to move to this as soon as it's released. They just need to update the nodes too, since it's a bug fix to the selinux roles so nothing else is needed on their end. The minute the nodes come up with the new version they should be corrected similar to CoreOS

[ocofaigh]

@anthonylecarrer Just to clarify, "exclude the agent" is around filtering logs to make sure they weren't sending the logs agent logs to ICL. Specifically following this doc - https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-configure-include-exclude#exclude-paths.
This can be achieved in the terraform module using the logs_agent_exclude_log_source_paths input.

[anthonylecarrer]

@ocofaigh I have experimented this solution. We have implemented overloading of all parameters as you have done in the repository. This is a temporary solution to advise to our customers to fill nor not the corresponding parameter. We can also patch the helm release with a null_resource in our terraform module to allow usage

Thanks you for all the solutions you try to find

[ocofaigh]

I'm going to close this issue since you have a temp workaround. I understand you have an IBM Cloud support case open so I would use that to track the permanent fix from IKS as no change is needed in the terraform code here.