Recreate Unraid auto-update and auto-backup functionalities in Komodo using Borg

[ValentinVie]

Hi all,

Thank you for creating Komodo. Having decided to move towards Komodo to manage my docker containers, I followed the great blog article from foxxmd: https://blog.foxxmd.dev/posts/migrating-to-komodo/. I migrated all my docker containers templates from Unraid to docker-compose. It was a pain but I finally succeeded !

However, there were functionalities I needed to recreate to keep my previous setup working as is:

• I used to auto-update my docker containers with Unraid CA Auto Update Applications plugin. Basically, it provided an update on a schedule: docker compose pull followed by a docker compose up -d --force-recreate command.
• I also used to auto-backup my container persistent configuration files in the appdata share /mnt/user/appdata with AppData Backup plugin. Basically, it created an archive containing a copy of a /mnt/user/appdata while keeping permissions intact.
• We also know that backing up database containers' files should be either done using DB dumps or by stopping a container. If we don't, we risk having a corrupted backup due to pending write calls... So while we are at it, the backup and auto-update process should account for this. Backups should only be performed when all containers are stopped.

Here is the backup process I setup:

• We deploy borgmatic container image.
• On a schedule provided by borgmatic CRON file, the backup launches.
• Borgmatic calls Komodo API to stop all the containers to backup which are tagged as auto-stop.
• Borgmatic backups appdata files to another share /mnt/user/Backups. Ideally on the array rather than on the cache drive... I personally also send all of the backup (encrypted) to AWS to follow proper 3-2-1 backup strategy.
• Borgmatic calls Komodo API to start all the containers tagged as auto-start to restore the services.

Borg calls for Komodo actions named auto-stop-stacks and auto-start-stacks during the backup route. These two actions start or stop the Komodo container stacks depending on their tags. Here is the interesting part of the config.yaml in borgmatic:

commands:
    # Call komodo to stop all containers backup its DB and commit its config.
    # Sleep 5 minutes after the call if successful.
  - before: action
    when:
      - create
    run:
      - >
        curl -sS http://komodo-core:9120/execute 
        --header "Content-Type: application/json" 
        --header "X-Api-Key: ${KOMODO_API_KEY}" 
        --header "X-Api-Secret: ${KOMODO_API_SECRET}" 
        --data '{ "type": "RunAction", "params": { "action": "auto-stop-stacks" } }'  
        && sleep 300 && echo "- Starting Backup"

    # Call komodo to restart the containers after a backup.
  - after: action
    when:
      - create
    run:
      - >
        curl -sS http://komodo-core:9120/execute
        --header "Content-Type: application/json" 
        --header "X-Api-Key: ${KOMODO_API_KEY}" 
        --header "X-Api-Secret: ${KOMODO_API_SECRET}" 
        --data '{ "type": "RunAction", "params": { "action": "auto-start-stacks" } }' 

Here is the code I created for the Komodo actions called by borgmatic. You just have to provide at the very top of the script: your tag name, the command and the maximum wait time for the action to complete. The command keywords can be found here. To start the stack, use StartStack and to stop the stack use StopStack... :

 // Variables
const TAG = 'auto-start';
const MAX_WAIT_TIME = 10 * 60 * 1000; // 10 minutes
const COMMAND = 'StartStack'

// Error tracking
let errors: any[] = [];

// Helper: ensure a tag exists and return its ID
async function getOrCreateTag(tagName: string) {
  const tags = await komodo.read('ListTags', {});
  const existing = tags.find(t => t.name === tagName);

  if (existing) return existing._id.$oid;
  
  try {
    const tagInfo = await komodo.write('CreateTag', { name: tagName });
    return tagInfo._id.$oid;
  } catch(e) {
    console.log(`⚠️ Failed to create tag "${tagName}":`, e);
    errors.push(e);
    return null;
  }
}

// *COMMAND* all stacks with the tag
async function startCommandOnStacks() {
  const autoTagId = await getOrCreateTag(TAG);

  if (!autoTagId) return;
  // Fetch all stacks that have the tag
  const stacks = await komodo.read('ListStacks', { query: { tags: [TAG] } });

  // Step 1: Fire ALL stop commands and track their Update objects
  const updates = await Promise.all(
    stacks.map(async (stack) => {
      try {
        console.log(`🚀 Launching ${COMMAND} on stack "${stack.name}"...`);
        const update = await komodo.execute(COMMAND, { stack: stack.id });
        return { stack, update };
      } catch (e) {
        console.log(`⚠️ Failed to launch ${COMMAND} on stack "${stack.name}":`, e);
        errors.push(e);
        return { stack, update: null }; // Mark as failed
      }
    })
  );

  // Step 2: Wait for ALL updates to reach "Complete" status (with timeout)
  // See doc : https://docs.rs/komodo_client/latest/komodo_client/entities/update/struct.Update.html
  const waitForCompletion = async ({ stack, update }) => {
    if (!update) return; // Skip if the command failed earlier

    const startTime = Date.now();
    while (update.status !== 'Complete') {
      // Check for timeout
      if (Date.now() - startTime > MAX_WAIT_TIME) {
        throw new Error(`Timeout: Stack "${stack.name}" did not succeed to perform ${COMMAND} within ${MAX_WAIT_TIME/1000/60} min`);
      }

      // Wait before polling again
      await new Promise(resolve => setTimeout(resolve, 2000));
      update = await komodo.read('GetUpdate', { id: update._id.$oid }); // Refresh update status
      console.log(`⏳ "${stack.name}" ${COMMAND} status: ${update.status}`);
    }
    // Check for failure
    if (!update.success) {
      throw new Error(`Stack "${stack.name}" failed to to perform ${COMMAND}: ${update.logs.join(', ')}`);
    }
    console.log(`✅ Stack "${stack.name}" has fully performed ${COMMAND}`);
  };

  // Wait for all stacks to complete (or fail)
  await Promise.all(
    updates.map(update =>
      waitForCompletion(update).catch(e => {
        console.log(`❌ Error performing ${COMMAND} on stack "${update.stack.name}":`, e.message);
        errors.push(e);
      })
    )
  );

}
 
// Execute
await startCommandOnStacks();

if (errors.length > 0) {
  throw `⚠️ Encountered ${errors.length} errors during stacks ${COMMAND} commands.`;
}

I explain the whole process in a more detailed blog post here.

Maybe this "execute-action-on-object-(stacks)-with-this-tag-process" could be added to Komodo procedures in the future.

Hope that will help someone. It's not the most elegant solution but it's a solution that works and that scales pretty well with many stacks. If you want to change the backup policy and behavior you just have to change borg config.yaml or the tags associated with your Komodo stacks.

[Scootter23]

@ValentinVie Very nice! I'd like to read more but the link doesn't seem to work at the moment.

[ValentinVie]

Hey ! Sorry, I took my blog down because of the Next.js Server CVE. It should be good now that I have updated. Still, there will be some downtime in the next few days as I am moving out. Let me know if you have any question !