Skip to content

Cobia to DragonFish Storage Migration

Important Changes

iX-Systems no longer maintains or supports any form of PVC-based storage for apps. This includes their own “legacy” storage backend. This means that when using the following SCALE features, you will lose some functionality:

  • SCALE Backup and Restore (unreleased feature available through HeavyScript) - Will no longer back up PVC storage at all.
  • SCALE App Rollback - You will no longer be able to roll back any PVC storage.
  • SCALE App Pool Migration - You will no longer be able to migrate PVC storage using the pool migration feature in SCALE.

The above issues exist both for our new in-house TrueCharts OpenEBS storage solution and for users using already installed apps that use the legacy storage backend from iX-Systems.

For Existing Users and Apps

Pre Cobia-to-DragonFish Upgrade Steps

  1. Ensure apps are up-to-date on Cobia

    Prior to updating to TrueNAS SCALE DragonFish, please ensure that TrueNAS SCALE is updated to the latest version of Cobia (23.10.2), and that all of your TrueCharts apps are fully updated, on the correct trains, and running or otherwise working correctly.

  2. Create a backup using HeavyScript

    For details on installing and creating a backup with HeavyScript, please see the HeavyScript GitHub Repo.

Post Cobia-to-DragonFish Upgrade Steps

  1. Wait at least 30 minutes after Cobia-to-Dragonfish upgrade

    Once you have completed updating your TrueNAS SCALE system from Cobia to DragonFish, the first step is to wait at least 30 minutes after updating and the SCALE WebUI becomes accessible. This allows time for the update to finish and services to be started. Do not restart the system or otherwise interrupt its operation during this time.

  2. Clean up the System

    We now need to clean up some old Cobia objects. Enter the following command in the TrueNAS SCALE shell as root:

    Replace TANK in the command with your pool name that contains the ix-applications dataset

    Remove Old OpenEBS
    rm /mnt/TANK/ix-applications/k3s/server/manifests/zfs-operator.yaml && sudo k3s kubectl delete -f && sudo k3s kubectl delete storageClass openebs-zfspv-default
  3. Create Dataset for PVCs and Install OpenEBS

    Continue with the OpenEBS setup section of our TrueNAS SCALE Getting Started Guide and once complete, proceed to next step.

  4. Migrate Apps to New Storage using TT-Migration Script

    Thanks to great collaboration with HeavyBullets, the TT-Migration tool will automate most of the migration process for existing apps:

    • Existing apps with PVCs or databases should be reinstalled using the TT-Migration script.
    • System train apps (cloudnative-pg, cert-manager, etc.) should NOT be reinstalled. Apps from all other trains should be migrated. You will see a warning in the script if you attempt to migrate a system train app.
    • Apps without PVC storage do still need to be reinstalled to remove every remnant of old storage backends. You may see a notice in the migration script if you attempt to migrate an app with no PVCs, if this is the case, continue with migrating the app (unless it’s on the System train, per above)

    Running the below command in the TrueNAS SCALE shell will (1) clone/download Heavy’s TT-Migration repo to your SCALE system, (2) enter the path for it and (3) execute the migration script as root:

    Install and Run TT-Migration
    git clone && cd TT-Migration && sudo bash

    If you want to see the steps that it performs, or if you’d like to perform the steps yourself manually using the guide, follow the instructions here.

Post Migration Steps

Once you have completed the above, you are finished the migration process. Your apps datasets should now look something like this:

OpenEBS Apps

Where you have a top-level dataset for everything related to apps (in the above image this is the “Apps” one), an “ix-applications” dataset which still retains non-PVC data for your apps and should NOT be deleted, and the dataset you created for OpenEBS use in an earlier part of this guide or when fresh installing SCALE and following our getting started guide. In the above photo, this is the “pvcApps” dataset.

Nested inside “pvcApps” (in the above example) will be the PVC folders for each of your apps that use PVC storage. Taking into account the above example layout, the OpenEBS operator apps’ storage options will look like this (obviously adjust based on your naming and dataset layout):

OpenEBS Storage

As you can see, I have 1 app using PVC storage, managed by OpenEBS, and whose data resides in the “pvc-7f970…” folder nested under pvcApps. You will have one of these nested folders for each app you have installed that uses PVC storage for application or config data.

Feel free to let the system settle for 30 minutes after completing the process and then reboot it to ensure that app apps come back online correctly.

Known Issues with Migrations

Sometimes, things don’t go as smoothly as they should. These are tips to help with any known issues identified as part of the migration process.

CNPG database cannot be restored via the migration script

For instances where an application’s database cannot be restored, the script will prompt you to select a backup (created with HeavyScript) instead. This guide assumes HeavyScript has been installed under the current user’s home directory.

  1. Navigate to HeavyScript’s backup folder for the selected application and list database dumps.

    Check for Backups
    cd ~/heavy_script/database_dumps/APPNAME && ls
  2. Copy the latest database dump to the migration dataset created by the migration script.

    Copy Database
    cp APPNAME_TIMESTAMP.sql.gz /mnt/TANK/migration/APPNAME/backup/
  3. Re-run the migration process and select the backup.

    Rerun Migration
    sudo bash --skip
    Use the backup copied in the previous step
    Select your original app name:
    1) APPNAME
    Enter the number associated with the app: 1
    You have chosen TANK/migration/APPNAME
    Since a CNPG pod was found, the script can attempt to automatically create a database dump. Note that this requires the CNPG pod to have the ability to run successfully. If you have a recent CNPG database dump from another source (e.g., HeavyScript), you can provide that instead.
    The default action is to have the script attempt to create a database dump, and should always be used unless your CNPG pod is problematic.
    Would you like to provide your own database dump file? [y/N]

”Dataset Busy” error when following the migration process/script

Try the following in these EXACT steps, deviation of any kind can result in data loss:

  1. Try to delete the dataset using the command zfs destroy -r "TANK/ix-applications/releases/APPNAME"

    ZFS Destroy
    zfs destroy -r "TANK/ix-applications/releases/APPNAME"
  2. If Step 1 fails, note the name of the failed app(s) and continue migrating ALL remaining apps.

  1. After all apps are migrated, unset the apps pool. TrueNAS SCALE GUI -> Applications -> Settings -> Unset Pool

  2. Go through your list of failed apps and delete them one by one, for example: zfs destroy -r "TANK/ix-applications/releases/APPNAME"

  3. After all failed apps datasets are deleted, re-set the apps pool the apps pool. TrueNAS SCALE GUI -> Applications -> Settings -> Set Pool

  4. Re-run the script with --skip flag for all failed apps.

It is very important to attempt all migrations before unsetting the applications pool. Unsetting the applications pool, or restarting with applications mid-migration, can result in data loss.