Cobia to DragonFish Storage Migration

Execute all steps at once

It’s imperative to execute ALL steps in this guide in one sitting. If you reboot or take any time away from this task, we cannot guarantee your data and apps stay working. Please only continue when you have time available to complete the migration.

It’s also important to note that the migration to DragonFish requires reinstallation of all Apps. For which an automated solution is available in the guide as well. Please read this migration guide carefully before upgrading to DragonFish.

Important Changes

iX-Systems no longer maintains or supports any form of PVC-based storage for apps. This includes their own “legacy” storage backend. This means that when using the following SCALE features, you will lose some functionality:

• SCALE Backup and Restore (unreleased feature available through HeavyScript) - Will no longer back up PVC storage at all.
• SCALE App Rollback - You will no longer be able to roll back any PVC storage.
• SCALE App Pool Migration - You will no longer be able to migrate PVC storage using the pool migration feature in SCALE.

The above issues exist both for our new in-house TrueCharts OpenEBS storage solution and for users using already installed apps that use the legacy storage backend from iX-Systems.

For Existing Users and Apps

Ensure apps are up-to-date on Cobia

Prior to updating to TrueNAS SCALE DragonFish, please ensure that TrueNAS SCALE is updated to the latest version of Cobia (23.10.2), and that all of your TrueCharts apps are fully updated, on the correct trains and running or otherwise working correctly.

Post Cobia-to-DragonFish Upgrade Steps

1. Wait at least 30 minutes after Cobia-to-Dragonfish upgrade

   Once you have completed updating your TrueNAS SCALE system from Cobia to DragonFish, the first step is to wait at least 30 minutes after updating and the SCALE WebUI becomes accessible. This allows time for the update to finish and services to be started. Do not restart the system or otherwise interrupt its operation during this time.

2. Clean up the System

   We now need to clean up some old Cobia objects. Enter the following command in the TrueNAS SCALE shell as root:

   Replace TANK in the command with your pool name that contains the ix-applications dataset

   Remove Old OpenEBS

   rm /mnt/TANK/ix-applications/k3s/server/manifests/zfs-operator.yaml && sudo k3s kubectl delete -f https://truecharts.org/openebsrem.yaml && sudo k3s kubectl delete storageClass openebs-zfspv-default

3. Create Dataset for PVCs and Install OpenEBS

   Continue with the OpenEBS setup section of our TrueNAS SCALE Getting Started Guide and once complete, proceed to next step.

   Dataset Location

   The dataset you create for OpenEBS usage MUST reside on the same storage pool as the existing ix-applications dataset. Refer to the example apps dataset images at the end of the guide.

4. Migrate Apps to New Storage using TT-Migration Script

   Thanks to great collaboration with HeavyBullets, the TT-Migration tool will automate most of the migration process for existing apps: https://github.com/Heavybullets8/TT-Migration

   • Existing apps with PVCs or databases should be reinstalled using the TT-Migration script.
   • System train apps (cloudnative-pg, cert-manager, etc.) should NOT be reinstalled. Apps from all other trains should be migrated. You will see a warning in the script if you attempt to migrate a system train app.
   • Apps without PVC storage do still need to be reinstalled to remove every remnant of old storage backends. You may see a notice in the migration script if you attempt to migrate an app with no PVCs, if this is the case, continue with migrating the app (unless it’s on the System train, per above)

   Systems Train Apps

   NEVER attempt to delete or reinstall system train apps without direction from the support team. Doing so may result in data loss.

   Running the below command in the TrueNAS SCALE shell will (1) clone/download Heavy’s TT-Migration repo to your SCALE system, (2) enter the path for it and (3) execute the migration script as root:

   Install and Run TT-Migration

   git clone https://github.com/Heavybullets8/TT-Migration.git && cd TT-Migration && sudo bash migrate.sh

   If you want to see the steps that it performs, or if you’d like to perform the steps yourself manually using the guide, follow the instructions here.

Tip

After running the above command for the first time, there’s no need to repeatedly clone/download the repo each time after. So after running it the first time, feel free to only run the below each time after:

Run TT-Migration

    cd TT-Migration && sudo bash migrate.sh

Post Migration Steps

Once you have completed the above, you are finished the migration process. Your apps datasets should now look something like this:

Where you have a top-level dataset for everything related to apps (in the above image this is the “Apps” one), an “ix-applications” dataset which still retains non-PVC data for your apps and should NOT be deleted, and the dataset you created for OpenEBS use in an earlier part of this guide. In the above photo, this is the “pvcApps” dataset.

Nested inside “pvcApps” (in the above example) will be the PVC folders for each of your apps that use PVC storage. Taking into account the above example layout, the OpenEBS operator apps’ storage options will look like this (obviously adjust based on your naming and dataset layout):

Feel free to let the system settle for 30 minutes after completing the process and then reboot it to ensure that app apps come back online correctly. Note that the OpenEBS operator now handles all PVC storage functionality for your installed apps, and should not be removed.

Known Issues with Migrations

Sometimes, things don’t go as smoothly as they should. These are tips to help with any known issues identified as part of the migration process.

”Dataset Busy” error when following the migration process/script

Try the following in these EXACT steps, deviation of any kind can result in data loss:

1. Try to delete the dataset using the command zfs destroy -r "TANK/ix-applications/releases/APPNAME"

   ZFS Destroy

       zfs destroy -r  "TANK/ix-applications/releases/APPNAME"

2. If Step 1 fails, note the name of the failed app(s) and continue migrating ALL remaining apps.

Complete Migration First

You MUST complete or start the migration of all apps before continuing. If you do not, the migration of any remaining apps not yet completed will no longer be possible.

YOU HAVE BEEN WARNED

3. After all apps are migrated, unset the apps pool. TrueNAS SCALE GUI -> Applications -> Settings -> Unset Pool

4. Go through your list of failed apps and delete them one by one, for example: zfs destroy -r "TANK/ix-applications/releases/APPNAME"

5. After all failed apps datasets are deleted, re-set the apps pool the apps pool. TrueNAS SCALE GUI -> Applications -> Settings -> Set Pool

6. Re-run the script with --skip flag for all failed apps.

It is very important to attempt all migrations before unsetting the applications pool. Unsetting the applications pool, or restarting with applications mid-migration, can result in data loss.