In the ever-evolving landscape of containerization and application deployment, TrueCharts has been at the forefront of providing users with streamlined and efficient Helm charts. As TrueCharts continues to prioritize user experience and development efficiency, a notable change is on the horizon – the decision to discontinue mirroring containers. We've highlighted below the main reasons for this change.

• Focusing Dev Time on Innovation: TrueCharts has always been committed to delivering high-quality Helm charts that simplify the deployment of applications on Kubernetes. By discontinuing container mirroring, the development team can now redirect their efforts and time towards enhancing existing charts, introducing new features, and ensuring the overall improvement of the TrueCharts ecosystem. This shift in focus aims to bring users an even more polished and feature-rich experience.

• Enhanced Clarity for Chart Users: The decision to cease container mirroring brings increased transparency for TrueCharts users. With a clear distinction between containers provided by TrueCharts and those hosted elsewhere, users can easily identify which containers are in use within their deployments. This added clarity fosters a better understanding of the components powering their applications and promotes a more informed and empowered user community.

• Faster and More Efficient Processing: Our current automated testing, takes a lot of processing time. Part of this is separated testing and releasing of the the container mirror. By removing the container mirror, we can lower the amount of tests run separately by 20%, significantly increasing the speed at which we can follow upstream releases of new versions.

As well, we would like to highlight these two important aspects of this transition below:​

• DockerHub Rate Limits: In a world where some container registries are subject to rate limits, this change does mean users might hit rate-limits when pulling certain containers. Especially those from dockerhub. If this is the case, please contact the container creator and request they use a different container registry.

• Gradual Transition for Current Users: Change can be daunting, especially for existing users accustomed to a certain workflow. Therefore, the discontinuation of container mirroring will not impact current users for at least a year. During this transition period, users can continue to benefit from mirrored images while TrueCharts prepares them for the upcoming change through clear communication and support.

• Seamless and Non-Breaking Change: TrueCharts is committed to making this transition as seamless as possible. Users can expect a non-disruptive experience, with no breaking changes to their current deployments. TrueCharts will provide comprehensive support to guide users through the transition, ensuring that the shift away from mirroring is a smooth and hassle-free process.

Conclusion:​

As TrueCharts takes this strategic step towards discontinuing container mirroring, the focus remains on user experience, transparency, and efficient development. By embracing this change, TrueCharts aims to provide users with an even more robust platform for deploying applications on Kubernetes. As the containerization landscape continues to evolve, TrueCharts remains dedicated to simplifying the deployment process and empowering users with the tools they need for success.

We're very glad to announce a new step in our project: Stability Tiers.

What are Stability Tiers?

Stability Tiers is a tier list of platforms supported by TrueCharts, ranked by how well we think our Charts function on each platform. Of course, all platforms get full access to community support, but we want to give realistic expectations on how many "snags" users can experience on the platform of their choice.

Improved First-Tier Helm Support

With the new tiers, we are also finally ready to announce that we've completed the required work to officially release our Normal Helm Charts as a first-tier supported platform. This also means that our industry-leading community support is now available for Helm users!

We want to make clear that, just as with SCALE, not every setting we offer is going to work well with every Chart. Sadly, we have not documented this very well, if at all. In the future, we want to document the release state (Experimental or GA) clearly for each Helm option in the documentation.

TrueNAS SCALE and Its Tier

While previously we've seen great effort and interest from the developers of TrueNAS SCALE, iXsystems, there's been a shift in priorities towards limiting Kubernetes support and prioritising their own catalogs at the cost of third parties like TrueCharts. We've also noticed a shift away from their previous plans to support multi-node clusters, accompanied by a disappointing disregard for providing any decent backup utility for their platform.

At the same time, we've been working hard on hardening our pipelines by signing both our container builds and Helm Charts. Sadly, TrueNAS SCALE, due to explicit design choices by iXsystems, also does not offer any tooling to ensure Helm Charts have their signatures validated before installation. This leads us to conclude that TrueNAS SCALE Apps are inherently less secure and professional than Helm Charts.

All in all, and after long deliberation, this has led us to decide to move TrueNAS SCALE to a "Second Tier" platform, as we cannot fully guarantee the same stability and reliability that normal Helm offers. This, however, does not mean a decrease in development efforts. We're still planning to fully support the platform where we can and expand both the catalog and our feature set on there in the future.

What it does mean is that some features might be slightly less reliable due to poorly designed "middleware" that is part of TrueNAS SCALE, which we, sadly enough, cannot do much against.

Future Platforms

However, there is more! We're also glad to announce we're working on supporting two more ways of deploying our Helm charts:

• FluxCD
• Rancher

For FluxCD, we hope to create a catalog of pre-made helm-release+kustomize files that can be readily copy-pasted into your GitOps repository! Even better, we're working hard to automate the deployment of GitOps with Flux, Sops-Encryption, and even a dedicated operating system: Talos-OS!

For Rancher, while you can already load our helm charts into Rancher and edit the YAML like any other Helm Chart, we are planning to add custom Rancher GUI elements to each and every published Helm chart. Just like with SCALE, but this time fully Kubernetes aware without complicated middleware!

The Tier List

This leads us to the following Stability Tier List, which will be documented on the website insert link here and adapted where needed:

1. Helm
2. TrueNAS SCALE

We hope this gives users more clarity on which platform to pick and what experience to expect. We're super stoked to expand this list in the future to support more awesome platforms!

We are happy to announce that support for cluster-wide certificates is now available for Truecharts! You can now create a single certificate and use it throughout your cluster. We call these certificates "cluster certificates".

Before you use the new feature, you should sync the Truecharts catalog and update all of your already installed apps to their latest version.

In addition to the cert-manager and clusterissuer apps you need for normal certificates, to use cluster certificates you also need to install our new kubernetes-reflector app from the enterprise train. For most setups installing the app with default settings is sufficent.

Once installed, edit your clusterissuer app and add a new cluster certificate. Note down the name you called it. Edit the app you wish to use the cluster certificate for and go to the Ingress section. If you have previously used a clusterissuer certificate, remove the issuer name. Click on Show Advanced Settings and add a TLS entry. Enter the name of your cluster certificate, and the certificate host(s) which it will be used for.

For a more detailed guide, see our docs.

After building our own MetalLB, CNPG and Prometheus operator charts, we've also now finished the work on building our own Cert-Manager operator chart. As of today this chart will be a requirement for new users if they want to use Cert-Manager and required for all users starting August 1, 2023.

If you have already installed clusterissuer follow the below guidance for installation of the Cert-Manager operator chart.

If you have not already done so add the operator train to TrueCharts as outlined here

1. Run this in the system shell as root:
   k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete4
2. Install cert-manager from the operators train.
3. Update clusterissuer to the latest version of (2.0.1+).

• If you are already on the latest version perform an empty edit of clusterissuer (Edit app and save without making any changes).

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

As part of limiting our promise not to introduce breaking changes to the charts within our Enterprise train, we've ensured both the new and old way of dealing with "operators" were both supported.

Starting August 1, 2023, we will completely drop support for the old (pre-July installs only, internal not user controlled) way of handling operators.

After August 1, 2023, additional checks for operators will be enabled, preventing users from making the mistake of installing charts without the right operator from the operator train present. This means that charts will prevent themselves from being updated when you're still using the old operators at that time.

If you have already installed the metallb, prometheus-operator, and cloudnative-pg operators then no further action is required.

Prerequisites​

Add the operator train to TrueCharts as outlined here

MetalLB​

The MetalLB operator is only required for users of MetalLB, anyone who does not use or plan to use MetalLB can skip this section.

1. Uninstall current metallb from Enterprise train.
2. Run this in the system shell as root: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete
3. Complete MetalLB installation as outlined here

Prometheus​

The Prometheus operator is required for the use of app metrics. Its installation is recommended.

1. Run this in the system shell as root: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete3
2. Install prometheus-operator from the operators train.

CNPG​

The cloudnative-pg operator is required for any applications that utilize postgres. Its installation is recommended.

1. Follow the CNPG Operator Migration Guide to migrate to the new CNPG operator. Ensure you follow the guide carefully as data loss can occur with this migration if proper steps are not followed.

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

After building our own MetalLB operator chart, we've also now finished the work on building our own CloudNative-PG Chart. As of today this chart will be a requirement for new users if they want to run applications featuring a postgresql database.

Updating to the new Cloudnative-PG helm chart for existing users

We want to point out though, that users should update the new CloudNative-PG Helm chart as soon as possible. To update an existing install with applications using postgresql databases to the new system, the following procedure can be used:

We want to explicitly highlight that this procedure will COMPLETELY DESTROY all your databases. It's absolutely crucial to export your databases manually beforehand.

• export all your databases manually, on SCALE using the following guide: https://truecharts.org/manual/SCALE/guides/cnpg-migration-guide (do not rely on heavyscript backups for this!)
• run this in a root shell: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete2
• install the new cloudnative-pg chart from the operators train
• wait a few minutes
• Hit edit and save without changes on all applications using postgresql databases.
• wait a few minutes
• Restore all your databases manually, on SCALE using the following guide: https://truecharts.org/manual/SCALE/guides/cnpg-migration-guide (do not rely on heavyscript backups for this!)

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

Introdocution: Our own Operator Charts

The last few months, we've experimented with injecting so-called "operators" into the cluster directly when using our charts. Manifests for things like: MetalLB, Cert-Manager and CNPG where always loaded. While this system guaranteed users where always running the latest operator versions, we've also encountered some downsides. Primarily:

• Loading manifests from the web is a security issue
• Loading manifests required a pre-install job, with full-cluster permissions. Which is also a security issue.
• Mistakes in the manifests, directly affect all users regardless of version
• It requires creating namespaces outside of the ix-something style, while not an issue that's something somehow iX developers voiced annoyance with.
• It lacks any configurability for users that need a customisation
• It prevents users from using these operators outside of the TrueCharts scope on non-scale systems

To fix all of these issues, we've had quite the challenge. First off we needed to figure out a way of preventing users from installing multiple instances of the same operator. But we also needed to ensure ourselves that users always had the correct operators installed for the charts they want to install.

We've by now designed an industry leading helm logic, that scans your cluster for references of installed operators and compares those to the required operators.

Besides this logic, we also need to write the Helm Charts ourselves. This is a lot of work, as operators are often notoriosly complex to write helm charts for. Luckily we've enough experienced Kubernetes developers that we're certain to pull this off!

First chart: MetalLB

As a first example of our new logic, we're super happy to introduce our first self-build operator helm chart: MetalLB. It will be completely self-contained within it's own namespace, not load dyanamic manifests from the web and doesn't contain risky security practices.

Obviously this chart, in the operators train, has a naming conflict with the old metallb chart in the enterprise train, so the later has been renamed to metallb-config requiring a reinstall. We want to point out that only the new metallb-config chart is compatbile with the new self-build metallb operator.

We are very happy to also announce that the metallb-config chart, is fully compatible with our old and new ways of installing/managing metallb. However, new installs of the old way of handling metallb (without the chart from the operators train), will be actively disabled from now on.

To use MetalLB on new installs, one needs to install both metallb and metallb-config, in that order.

Updating to the new MetalLB helm chart

We want to point out though, that users should update the new MetalLB Helm chart as soon as possible. To update a current install using MetalLB to the new system, the following procedure can be used:

• remove the old metallb chart coming from the enterprise train
• run this in a root shell: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete
• install the new metallb chart from the operators train
• wait a few minutes
• install or update metallb-config to the latest version
• wait a few minutes
• Hit edit on metallb-config and save without changes if you where already on the latest version or it isn't working yet
• wait a few minutes

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

BLUF: Traefik (Stable) is Deprecated. Users need to add the Enterprise channel and install Traefik (Enterprise). https://truecharts.org/manual/SCALE/guides/getting-started#adding-truecharts

The use of TrueNAS Scale Certificates is also deprecated and you must migrate to Clusterissuer (Enterprise). https://truecharts.org/charts/enterprise/clusterissuer/how-to (note: Clusterissuer replaced Cert-Manager)

As some of you might've noticed, Traefik has been a bit outdated the last few weeks. The reason behind this, was a multitude of potentially breaking todo's where left and we don't want to bother users with continues manual intervention on breaking changes. By now we've fixed the remaining issues and will soon release a breaking-change release for traefik and a patch for all the charts.

In short we've ensured that we only use our signature "tc-system" namespace for storing configuration and middlewares for traefik. This ensures consistent behavior for users using ingressClasses and allowed us to, cleanly, fix the known bug where a port got appended to the TrueNAS SCALE "portal" button.

This also means that charts that do not get patches because they are not ported to new common, most notably: Nextcloud Will inherently not work anymore. Though, users would've been ill-adviced using it at all currently... due to the big ongoing nextcloud rework.

Unrelated new issue

We also got the request from iX-systems staff a while ago to limit our use of non-ix-prefixed namespaces on kubernetes. While the other work to do so, requires a lot of work building our own operator helm-charts, these Traefik changes are the initial step to comply to those wishes. The "low hanging fruit".

As we're working hard on building seperate operator helm-charts, instead of handling it in the background.This work leads to a unrelated temporary issue, which has been created on purpose: CNPG will currently only be installed on new systems, if one of our "enterprise" charts is being installed. More news about this will be released later.

For any help, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

Previously we've warned users against using the stop-botton on TrueNAS SCALE. At the same time we also understand, that users expect platform uniformity between Helm and SCALE. That's why we're happy to announce our own solution stop our Charts: TrueCharts Stop-All!

About that stop button​

First off, we would like to go into a bit more depth about the design issues of the TrueNAS SCALE "Stop" Button. We've hinted at it previously, but it's always good to explain why we need to step in ourselves.

The idea with Kubernetes, is that one tool should be managing deployment of objects at a time. Often indicated by a managed-by annotation on said objects. With TrueNAS SCALE, the middleware, triggers a management tool called "Helm" to deploy objects. So far so good, a GUI isn't magically able to trigger other software, after all.

However, with the stop button, iX decided to also start editing some of those objects themselves. Specifically "Deployments" and "StatefullSets", setting them to 0 "replica's" meaning "run nothing". That sounds completely fine, however: In these cases "Helm" is the actuall management tool for those objects, so everything a helm action is triggered, those modifications are instandly removed.

That's where the problems start to become bigger and bigger, because helm actions are triggered more often than people realise. For example: A reboot also triggers helm, requiring the same "hacks" to put things "back to sleep" again.

iX also decided to not even include all default objects that are technically "running", like: Daemonsets, Jobs and Cronjobs. Which leads to issues with breaking jobs or daemonsets/jobs locking access to PVC's. There it becomes more complicated, as kubernetes does not only exists of those "default" objects. There are also "Custom Resources", objects that are defined by other charts and there is also the ability of other management tools, like Operators, to add objects.

When making such changes through Helm, it would be relatively easy for Chart/App developers to mitigate this. However, iX decided not to and does not even expose the "stop" button state to the Chart/App developers, leaving us completely without tools to mitigate these design flaws.

In the end, that leave out how the stop button can get into a near endless state of limbo, if not all running objects are stopped correctly... Putting the cherry on top.

Looking for a better way forward​

With that all concluded, we decided to look into "what needs to be done", to get all our Charts to have "stop" button functionality back again. It's clear that the stop button, even with little fixes, isn't going to be a future proof design. It completely needs to be redesigned, including all it's backend logic. Sadly enough, refactors of said scale (pun not even intended), are currently not the priority of iXsystems, so not something we can rely on for our users.

We concluded that the only way to do so reliably, is through Helm itself. We know which objects we have, how they need to be stopped and can do so reliably through Helm. Which means: Do it ourselves!

The solution: TrueCharts Stop-All​

With the most recent updates, we've introduced a new option: TrueCharts Stop-All This option will cause all your running objects to slowly shut themselves down or, in the case of our postgresql backend (CNPG), "hibernate".

It's designed to feature support for all default kubernetes objects deployed using our common chart: Daemonsets, Deployments, StatefullSets, Jobs and Cronjobs. On top we can easily expand that to cover any operator based objects, like cnpg, that needs to be shut down as-well in the future!

How To Use Stop-All​

On SCALE

On SCALE this is a little checkbox on editing the App. Check it and its done

NOTE: Do not forget to uncheck the "Stop-All" checkbox to start the App again.

Using Helm

On native Helm, the same functionality is also availably: Simply set the following in your values.yaml file:

global
  stopAll: true

We're glad to finally announce the end of our code-freeze. Since a few days we've re-enabled our automatic updates and within a few weeks everything should balance out again automatically!

At the same time, we've not completely finished porting all stable-train charts to the new common, 65 are still missing. But we've clearly label those updates as breaking in the changelog when they come in. Most of those are charts that have more complications than anticipated, so need a little quality time with our maintainers which takes a while.

Known Issues​

Now that we're mostly done, we also need to report a few known issues with the new backend:

1. DO NOT USE THE STOP BUTTON

The Stop button should not be used on any TrueNAS SCALE Apps that uses postgresql. It due to severe design mistakes by iXsystems, it will get into an endless loop and never finish. We're reported the issue to iXsystems and they are not interested in fixing this.

2. Postgresql breaking on reboot

We're seen some edgecases where the new database backend breaks after a reboot. Often after the STOP button was used, though we cannot trace the issue down back to the use of the stop button itself. These issues are reported to the folks over at CNPG and we've also thrown them an email to discuss weither we can fund them to fix these issues

3. hostNetworking changes

After much R&D, our staff have discovered quite a few nasty kubernetes-level bugs with hostNetworking. As a result, we've decided to never enable it by default anymore on any of our charts/apps, as we cannot guarantee its stability. For some charts that, often, require this setting (like tailscale), users would have to manually and explicitly enable it from now on.

The setting has also moved in the GUI.

4. Deprecated certificate system and you

With most Charts ported, we want to highlight the fact that the "TrueNAS SCALE (Deprecated)" certificate option, should not be used anymore. We cannot guarantee it's stability nor can do anything at-all to help out. It will also be removed as an option in the future, though that will be months rather than weeks.

The future​

With the charts slowly all being ported, we can start working on our long-term plans again. One of those plans is a renewed focus on native Helm Charts.

For May and June, we're planning to go all-in on improving documentation for use of our charts as normal Helm charts. At the same time we're going to work on ensuring all our SCALE specific tricks (of which only a few are left, luckily), will have automatic alternatives for normal kubernetes clusters.

To highlight this, we've asked Artifact hub, to highlight our Common-Library chart, as an "official" TrueCharts Helm chart. All users of helm should be able to use the power of this advanced common-library, to build the Helm Charts they please... Without even relying on TrueCharts to host their charts for them!

Check it out here: https://artifacthub.io/packages/helm/truecharts-library-charts/common And also check out the docs as always: https://truecharts.org/manual/helm/common/