OpenShift Bootstrap GitOps Repository

This repository contains standardized configurations for bootstrapping new OpenShift clusters with common operational settings.

Structure

├── cluster-configs/          # Cluster-level configurations
│   ├── acm/                 # Advanced Cluster Management
│   ├── developer-hub/       # Red Hat Developer Hub (Backstage)
│   ├── gitops/              # OpenShift GitOps (ArgoCD) (TODO: add second ArgoCD instance for apps)
│   ├── logging/             # Log management and retention
│   ├── security/            # Security policies, RBAC, and authentication
│   ├── storage/             # LVM Storage, StorageClass, Image Registry
│   ├── monitoring/          # Monitoring and alerting setup (TODO)
│   └── networking/          # Network policies and ingress (TODO)
├── applications/            # Application deployments
└── infrastructure/          # Infrastructure components
    └── disk-partitioning/   # SNO disk partitioning (install-time only)

Quick Start

Option A: Bootstrap Everything (New Cluster)

# Deploy all bootstrap configurations
oc apply -k cluster-configs/

Option B: Deploy Components Individually

1. Storage (Deploy First)

# LVM Storage, StorageClass, and Image Registry
oc apply -k cluster-configs/storage/

2. Platform Operators

# OpenShift GitOps (ArgoCD)
oc apply -k cluster-configs/gitops/

# Advanced Cluster Management
oc apply -k cluster-configs/acm/

# Log retention policies
oc apply -k cluster-configs/logging/

3. Monitor Deployment

# Check operator status
oc get subscriptions -A
oc get csv -A

# Check storage
oc get lvmcluster -n openshift-storage
oc get pvc -n openshift-image-registry

# Check GitOps
oc get argocd -n openshift-gitops

# Check ACM
oc get multiclusterhub -n open-cluster-management

Logging Configuration Details

Container Logs (kubelet-log-rotation.yaml)

• Max log file size: 50Mi per container
• Max log files: 5 rotated files kept
• Total per container: ~250Mi maximum
• Effect: Immediate (no reboot required)

System Logs (journald-retention.yaml)

• Max journal usage: 2GB total
• Retention period: 30 days
• Rotation: Daily
• Effect: Requires node reboot via MachineConfig

Current Cluster Analysis

Based on cluster analysis from 2025-08-28:

• Disk usage: 29% (276G/953G)
• Journal logs: 3.5G
• API server logs: 2.1G
• Pod logs: 1.6G
• Status: Manageable but growing

Single Node OpenShift (SNO) Disk Partitioning

⚠️ CRITICAL: Must be done during installation only!

For SNO clusters, separate your root filesystem from container storage:

1. Before installation: Customize infrastructure/disk-partitioning/98-create-a-partition-for-lvmstorage.yaml
2. During installation: Upload the MachineConfig via Assisted Installer
3. After installation: Apply storage configs for LVM-based dynamic provisioning

# Post-installation: Set up LVM storage
oc apply -f cluster-configs/storage/lvmstorage-operator.yaml
oc apply -f cluster-configs/storage/lvmcluster.yaml

See infrastructure/disk-partitioning/README.md for detailed instructions.

Adding to New Clusters

1. For SNO clusters: Use disk partitioning configs during installation
2. For new cluster bootstrap: Apply all configs in cluster-configs/
3. For existing clusters: Apply selectively based on needs
4. With ArgoCD/GitOps: Point to this repo for automated application

Configuration Customization

Adjust Log Retention

Edit the values in the YAML files:

• containerLogMaxSize: Increase for verbose applications
• MaxRetentionSec: Adjust based on compliance requirements
• SystemMaxUse: Scale based on disk size

Add Additional Configs

• Place new configurations in appropriate subdirectories
• Follow the same naming convention: component-purpose.yaml
• Add documentation to this README

Troubleshooting

Check Log Rotation Status

# Container logs
oc logs -n kube-system -l app=node-exporter | grep -i log

# Journal status
oc debug node/NODE_NAME -- chroot /host journalctl --disk-usage

Force Immediate Cleanup

# Clean old container logs (if needed)
oc debug node/NODE_NAME -- chroot /host find /var/log/pods -name "*.log.*" -mtime +7 -delete

# Vacuum journal logs
oc debug node/NODE_NAME -- chroot /host journalctl --vacuum-time=7d