Skip to content

2023

28 posts with the tag “2023”

Introducing our AI bot: TrueAI

As 2023 comes to a close, we’re excited to introduce TrueAI, our new support AI for TrueCharts, TrueNAS, and HeavyScript users. This is our way of stepping into 2024 with a focus on enhancing user support through technology.

TrueAI is now available on our Discord server for everyone to use. To interact with the bot, simply mention @TrueAI in your message. The bot is programmed to create a thread in the channel where it’s mentioned, or reply within an existing thread.

Here are some tips for interacting with TrueAI:

  • Indicate whether you’re using TrueNAS or HELM, and mention the version you’re using.
  • Clearly describe what you’re trying to achieve.
  • Mention any specific issues you’re facing.
  • If applicable, include any error messages you’ve encountered.

Please note that TrueAI cannot interact with images or files sent in messages.

If you need more help after using TrueAI, we encourage you to open a support ticket. Our team is ready to assist you.

We hope TrueAI will be a valuable resource for you all. Wishing everyone a happy and successful 2024!

Changes to Storage, Persistence, and PostgreSQL

We’re glad to announce that shortly we will release a big update to all our Apps. This will be a complex update that technically includes a few changes that might break some specific features for some users. While we do not foresee any data loss, it’s imperative to back up your data.

Standardization of PVC Storage

A number of new storage options are being released. These are all optional and should be considered somewhat experimental.

Redesign of Cloudnative-PG

Our Cloudnative-PG backend has been completely rewritten. This should provide more stability, but also carries the risk of introducing new bugs.

Required Migration Steps

If you want to update, here are the instructions:

All Platforms

  • We strongly advise upgrading all charts and not just a select few as the new version of ClusterIssuer might not be fully compatible with charts on old versions.
  • Ingress: In some cases it might be prudent to disable ingress prior to update and enable it again afterwards.
  • Statefulsets: Be sure to remove any statefulsets made by TrueCharts Helm Charts before or after update. See instructions below.

Helm

SCALE

  • Ingress: Most of the cert-manager settings have been moved to “integrations” and they are reset.
  • Ingress: All of the Traefik settings have been moved to “integrations” and they are reset.
  • Ingress: This means you likely will lose TLS and security middlewares after the update, until adapted.
  • CNPG: If present in the old GUI, Instance number and storage size have been moved and reset to defaults.
  • Traefik: After updating, you will need to check a checkbox at the bottom, with a warning confirmation, before you will be able to do any further updates or edits.

Stateful Set Instructions

The charts with statefulsets have a manual step. but it wont require reinstall. make sure to refresh your catalog. We do not have many statefulsets in our catalog, so expect it to be mostly OpenLDAP and/or Redis related.

SCALE

To check which have statefulsets:

Terminal window
k3s kubectl get statefulsets -A | grep "ix-"

Then to delete the statefulset:

Terminal window
k3s kubectl delete statefulset STATEFULSETNAME -n ix-APPNAME

Example:

Terminal window
k3s kubectl delete statefulset blocky-redis -n ix-blocky

Once deleted you can attempt the update (or if you were already updated to latest versions, then edit and save without any changes)

Helm

To check which have statefulsets:

Terminal window
kubectl get statefulsets -A

Then to delete the statefulset:

Terminal window
kubectl delete statefulset STATEFULSETNAME -n APPNAME

Example:

Terminal window
kubectl delete statefulset blocky-redis -n blocky

Once deleted you can attempt the update (or if you were already updated to latest versions, then edit and save without any changes)

🎄 Merry Christmas / Happy Holidays from TrueCharts! 🎄

As the holiday season sparkles around us, we at TrueCharts want to extend our warmest wishes for a Merry Christmas to our incredible community. This year has been a remarkable journey, and we couldn’t have done it without you!

🚀 Major Milestone Achieved! 🚀

We’re thrilled to announce that we’ve successfully transitioned hundreds of our Charts from incubator to stable! This massive endeavor includes dozens of game servers, ensuring more stability and fun for everyone.

🎮 Game On! 🎮

To all the gaming enthusiasts, get ready for enhanced gaming experiences with our latest and more robust game server offerings.

🙏 Support Our Journey 🙏

As we continue to grow and improve, we kindly remind you that donations are always welcome. Your support fuels our passion and helps us maintain and develop even more exciting projects. Consider making a contribution this festive season!

🚨 Transition Notice: Ending support for TrueNAS SCALE Bluefin (22.12) 🚨

Please note, we will be ending support for TrueNAS SCALE Bluefin 22.12.X starting today on 19 December 2023. To continue receiving support and updates in the New Year, we encourage users to upgrade their systems to TrueNAS SCALE Cobia (23.10), which released it’s first update to 23.10.1 on December 19th 2023. We have a large community who can assist you through this important transition on our Discord with channels for #truenas-scale and #scale-apps.

Special Thanks

We want to take a moment to appreciate those who have made our journey so fruitful and new contributors (both directly and indirectly) to bringing TrueCharts to more platforms:

  • Our Staff: For their unwavering dedication and hard work.
  • Code Contributors: Your contributions are the backbone of our progress.
  • Our Community: For your enthusiasm, feedback, and support.
  • Our Donators: Your generosity keeps our mission alive.
  • OpenSource Collective: A big thank you for managing our finances (OpenSource Collective).
  • Ich777: Your game server builds have been invaluable (Ich777 on GitHub).
  • SideroLabs Team: For TalosOS, which we’re excited to support more (SideroLabs).
  • Budimanjojo: For your proactive work on TalHelper, aiding our documentation and support efforts for TalosOS (Budimanjojo on GitHub, TalHelper).

As we look forward to the new year, we’re excited to bring you more updates, enhancements, and surprises. Stay tuned!

Wishing you a joyful and peaceful holiday season.

🌟 Happy Holidays and a Happy New Year from all of us at TrueCharts! 🌟

Exciting Changes on the Horizon in January!

As we bid farewell to 2023, we’re thrilled to share a sneak peek into the future of TrueCharts. January 2024 will mark a significant milestone for us as we unveil a host of groundbreaking changes and new features that will elevate your experience with our charts.

Here’s a glimpse of what’s in store:

Improved CloudNative-PG Support

TrueCharts is committed to enhancing your experience with CloudNative-PG. Our upcoming release will bring improvements to ensure seamless integration and optimal performance.

Prototype for CNPG Backups

Data security is a top priority for us. We are introducing a prototype for CloudNative-PG backups to provide you with robust and reliable data protection solutions.

Completely Rewritten Ingress Backend

Our team has reimagined and rewritten the entire ingress backend to offer enhanced functionality, improved performance, and a more intuitive user experience.

Homepage Integration for All Ingresses

As part of our commitment to accessibility, we’re introducing homepage app integration for all ingresses. This integration will streamline navigation and make TrueCharts even more user-friendly. homepage is a modern, fully static, fast, secure fully proxied, highly customizable application dashboard with integrations for over 100 services and translations into multiple languages.

New Options for iSCSI, SMB, and NFS Storage Backend

Diversify your storage options with new features for iSCSI, SMB, and NFS storage backends. TrueCharts is expanding possibilities to cater to a broader range of user preferences and requirements.

What do these changes mean for our current configurations

Seamless Transition for CloudNative-PG Users

For users who haven’t made modifications to their CloudNative-PG configuration, there’s good news. You won’t need to adapt your configuration in the GUI or values.yaml, ensuring a smooth transition to the latest version of TrueCharts implementation of CNPG.

Ingress Value Restructuring

Certain changes to ingress values will be introduced. Detailed information about these changes will be published around the time of the release. We appreciate your patience and cooperation as we work to streamline and optimize our systems.

Stay Up-to-Date for a Smoother Upgrade

To ensure a seamless transition to the next major step in TrueCharts, we strongly advise all users to ensure they are up-to-date before January 1, 2024. This proactive measure should pave the way for a smooth upgrade and prevent reinstallation of certain charts.

Disclaimer

Important Warning: While we work hard to ensure a smooth transition, we cannot guarantee it. Some users might have to reinstall some charts. However, we are confident that this will be an extremely small minority.

The TrueCharts community has been a driving force behind our growth, and we’re excited to embark on this journey with you. We appreciate your continued support and look forward to ushering in a new era of TrueCharts together.

Stay tuned for more updates and details as we approach the release date!

Best regards and happy holidays,

The TrueCharts Team

The End of Container Mirroring

In the ever-evolving landscape of containerization and application deployment, TrueCharts has been at the forefront of providing users with streamlined and efficient Helm charts. As TrueCharts continues to prioritize user experience and development efficiency, a notable change is on the horizon – the decision to discontinue mirroring containers. We’ve highlighted below the main reasons for this change.

  • Focusing Dev Time on Innovation: TrueCharts has always been committed to delivering high-quality Helm charts that simplify the deployment of applications on Kubernetes. By discontinuing container mirroring, the development team can now redirect their efforts and time towards enhancing existing charts, introducing new features, and ensuring the overall improvement of the TrueCharts ecosystem. This shift in focus aims to bring users an even more polished and feature-rich experience.

  • Enhanced Clarity for Chart Users: The decision to cease container mirroring brings increased transparency for TrueCharts users. With a clear distinction between containers provided by TrueCharts and those hosted elsewhere, users can easily identify which containers are in use within their deployments. This added clarity fosters a better understanding of the components powering their applications and promotes a more informed and empowered user community.

  • Faster and More Efficient Processing: Our current automated testing, takes a lot of processing time. Part of this is separated testing and releasing of the the container mirror. By removing the container mirror, we can lower the amount of tests run separately by 20%, significantly increasing the speed at which we can follow upstream releases of new versions.

As well, we would like to highlight these two important aspects of this transition below:

  • DockerHub Rate Limits: In a world where some container registries are subject to rate limits, this change does mean users might hit rate-limits when pulling certain containers. Especially those from dockerhub. If this is the case, please contact the container creator and request they use a different container registry.

  • Gradual Transition for Current Users: Change can be daunting, especially for existing users accustomed to a certain workflow. Therefore, the discontinuation of container mirroring will not impact current users for at least a year. During this transition period, users can continue to benefit from mirrored images while TrueCharts prepares them for the upcoming change through clear communication and support.

  • Seamless and Non-Breaking Change: TrueCharts is committed to making this transition as seamless as possible. Users can expect a non-disruptive experience, with no breaking changes to their current deployments. TrueCharts will provide comprehensive support to guide users through the transition, ensuring that the shift away from mirroring is a smooth and hassle-free process.

Conclusion:

As TrueCharts takes this strategic step towards discontinuing container mirroring, the focus remains on user experience, transparency, and efficient development. By embracing this change, TrueCharts aims to provide users with an even more robust platform for deploying applications on Kubernetes. As the containerization landscape continues to evolve, TrueCharts remains dedicated to simplifying the deployment process and empowering users with the tools they need for success.

Stability Tiers and Helm Support

We’re very glad to announce a new step in our project: Stability Tiers.

What are Stability Tiers?

Stability Tiers is a tier list of platforms supported by TrueCharts, ranked by how well we think our Charts function on each platform. Of course, all platforms get full access to community support, but we want to give realistic expectations on how many “snags” users can experience on the platform of their choice.

Improved First-Tier Helm Support

With the new tiers, we are also finally ready to announce that we’ve completed the required work to officially release our Normal Helm Charts as a first-tier supported platform. This also means that our industry-leading community support is now available for Helm users!

We want to make clear that, just as with SCALE, not every setting we offer is going to work well with every Chart. Sadly, we have not documented this very well, if at all. In the future, we want to document the release state (Experimental or GA) clearly for each Helm option in the documentation.

TrueNAS SCALE and Its Tier

While previously we’ve seen great effort and interest from the developers of TrueNAS SCALE, iXsystems, there’s been a shift in priorities towards limiting Kubernetes support and prioritising their own catalogs at the cost of third parties like TrueCharts. We’ve also noticed a shift away from their previous plans to support multi-node clusters, accompanied by a disappointing disregard for providing any decent backup utility for their platform.

At the same time, we’ve been working hard on hardening our pipelines by signing both our container builds and Helm Charts. Sadly, TrueNAS SCALE, due to explicit design choices by iXsystems, also does not offer any tooling to ensure Helm Charts have their signatures validated before installation. This leads us to conclude that TrueNAS SCALE Apps are inherently less secure and professional than Helm Charts.

All in all, and after long deliberation, this has led us to decide to move TrueNAS SCALE to a “Second Tier” platform, as we cannot fully guarantee the same stability and reliability that normal Helm offers. This, however, does not mean a decrease in development efforts. We’re still planning to fully support the platform where we can and expand both the catalog and our feature set on there in the future.

What it does mean is that some features might be slightly less reliable due to poorly designed “middleware” that is part of TrueNAS SCALE, which we, sadly enough, cannot do much against.

Future Platforms

However, there is more! We’re also glad to announce we’re working on supporting two more ways of deploying our Helm charts:

  • FluxCD
  • Rancher

For FluxCD, we hope to create a catalog of pre-made helm-release+kustomize files that can be readily copy-pasted into your GitOps repository! Even better, we’re working hard to automate the deployment of GitOps with Flux, Sops-Encryption, and even a dedicated operating system: Talos-OS!

For Rancher, while you can already load our helm charts into Rancher and edit the YAML like any other Helm Chart, we are planning to add custom Rancher GUI elements to each and every published Helm chart. Just like with SCALE, but this time fully Kubernetes aware without complicated middleware!

The Tier List

This leads us to the following Stability Tier List, which will be documented on the website insert link here and adapted where needed:

  1. Helm
  2. TrueNAS SCALE

We hope this gives users more clarity on which platform to pick and what experience to expect. We’re super stoked to expand this list in the future to support more awesome platforms!

Cluster-Wide Certificates

We are happy to announce that support for cluster-wide certificates is now available for Truecharts! You can now create a single certificate and use it throughout your cluster. We call these certificates “cluster certificates”.

Before you use the new feature, you should sync the Truecharts catalog and update all of your already installed apps to their latest version.

In addition to the cert-manager and clusterissuer apps you need for normal certificates, to use cluster certificates you also need to install our new kubernetes-reflector app from the enterprise train. For most setups installing the app with default settings is sufficient.

Once installed, edit your clusterissuer app and add a new cluster certificate. Note down the name you called it. Edit the app you wish to use the cluster certificate for and go to the Ingress section. If you have previously used a clusterissuer certificate, remove the issuer name. Click on Show Advanced Settings and add a TLS entry. Enter the name of your cluster certificate, and the certificate host(s) which it will be used for.

For a more detailed guide, see our docs.

Cert-Manager Operator

After building our own MetalLB, CNPG and Prometheus operator charts, we’ve also now finished the work on building our own Cert-Manager operator chart. As of today this chart will be a requirement for new users if they want to use Cert-Manager and required for all users starting August 1, 2023.

If you have already installed clusterissuer follow the below guidance for installation of the Cert-Manager operator chart.

If you have not already done so add the operator train to TrueCharts as outlined here

  1. Run this in the system shell as root:
    k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete4
  2. Install cert-manager from the operators train.
  3. Update clusterissuer to the latest version of (2.0.1+).
  • If you are already on the latest version perform an empty edit of clusterissuer (Edit app and save without making any changes).

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

Deprecating Old Operators

As part of limiting our promise not to introduce breaking changes to the charts within our Enterprise train, we’ve ensured both the new and old way of dealing with “operators” were both supported.

Starting August 1, 2023, we will completely drop support for the old (pre-July installs only, internal not user controlled) way of handling operators.

After August 1, 2023, additional checks for operators will be enabled, preventing users from making the mistake of installing charts without the right operator from the operator train present. This means that charts will prevent themselves from being updated when you’re still using the old operators at that time.

If you have already installed the metallb, prometheus-operator, and cloudnative-pg operators then no further action is required.

Prerequisites

Add the operator train to TrueCharts as outlined here

MetalLB

The MetalLB operator is only required for users of MetalLB, anyone who does not use or plan to use MetalLB can skip this section.

  1. Uninstall current metallb from Enterprise train.
  2. Run this in the system shell as root: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete
  3. Complete MetalLB installation as outlined here

Prometheus

The Prometheus operator is required for the use of app metrics. Its installation is recommended.

  1. Run this in the system shell as root: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete3
  2. Install prometheus-operator from the operators train.

CNPG

The cloudnative-pg operator is required for any applications that utilize postgres. Its installation is recommended.

  1. Follow the CNPG Operator Migration Guide to migrate to the new CNPG operator. Ensure you follow the guide carefully as data loss can occur with this migration if proper steps are not followed.

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

New CloudNative-PG (CNPG) operator chart.

After building our own MetalLB operator chart, we’ve also now finished the work on building our own CloudNative-PG Chart. As of today this chart will be a requirement for new users if they want to run applications featuring a postgresql database.

Updating to the new Cloudnative-PG helm chart for existing users

We want to point out though, that users should update the new CloudNative-PG Helm chart as soon as possible. To update an existing install with applications using postgresql databases to the new system, the following procedure can be used:

We want to explicitly highlight that this procedure will COMPLETELY DESTROY all your databases. It’s absolutely crucial to export your databases manually beforehand.

  • export all your databases manually, on SCALE using the following guide (do not rely on heavyscript backups for this!)
  • run this in a root shell: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete2
  • install the new cloudnative-pg chart from the operators train
  • wait a few minutes
  • Hit edit and save without changes on all applications using postgresql databases.
  • wait a few minutes
  • Restore all your databases manually, on SCALE using the following guide (do not rely on heavyscript backups for this!)

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

New MetalLB chart and our own operator charts.

Introdocution: Our own Operator Charts

The last few months, we’ve experimented with injecting so-called “operators” into the cluster directly when using our charts. Manifests for things like: MetalLB, Cert-Manager and CNPG where always loaded. While this system guaranteed users where always running the latest operator versions, we’ve also encountered some downsides. Primarily:

  • Loading manifests from the web is a security issue
  • Loading manifests required a pre-install job, with full-cluster permissions. Which is also a security issue.
  • Mistakes in the manifests, directly affect all users regardless of version
  • It requires creating namespaces outside of the ix-something style, while not an issue that’s something somehow iX developers voiced annoyance with.
  • It lacks any configurability for users that need a customization
  • It prevents users from using these operators outside of the TrueCharts scope on non-scale systems

To fix all of these issues, we’ve had quite the challenge. First off we needed to figure out a way of preventing users from installing multiple instances of the same operator. But we also needed to ensure ourselves that users always had the correct operators installed for the charts they want to install.

We’ve by now designed an industry leading helm logic, that scans your cluster for references of installed operators and compares those to the required operators.

Besides this logic, we also need to write the Helm Charts ourselves. This is a lot of work, as operators are often notoriosly complex to write helm charts for. Luckily we’ve enough experienced Kubernetes developers that we’re certain to pull this off!

First chart: MetalLB

As a first example of our new logic, we’re super happy to introduce our first self-build operator helm chart: MetalLB. It will be completely self-contained within it’s own namespace, not load dynamic manifests from the web and doesn’t contain risky security practices.

Obviously this chart, in the operators train, has a naming conflict with the old metallb chart in the enterprise train, so the later has been renamed to metallb-config requiring a reinstall. We want to point out that only the new metallb-config chart is compatible with the new self-build metallb operator.

We are very happy to also announce that the metallb-config chart, is fully compatible with our old and new ways of installing/managing metallb. However, new installs of the old way of handling metallb (without the chart from the operators train), will be actively disabled from now on.

To use MetalLB on new installs, one needs to install both metallb and metallb-config, in that order.

Updating to the new MetalLB helm chart

We want to point out though, that users should update the new MetalLB Helm chart as soon as possible. To update a current install using MetalLB to the new system, the following procedure can be used:

  • remove the old metallb chart coming from the enterprise train
  • run this in a root shell: k3s kubectl delete --grace-period 30 --v=4 -k https://github.com/truecharts/manifests/delete
  • install the new metallb chart from the operators train
  • wait a few minutes
  • install or update metallb-config to the latest version
  • wait a few minutes
  • Hit edit on metallb-config and save without changes if you where already on the latest version or it isn’t working yet
  • wait a few minutes

If you run into additional issues, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

Traefik Changes

BLUF: Traefik (Stable) is Deprecated. Users need to add the Enterprise channel and install Traefik. See how to get started

The use of TrueNAS Scale Certificates is also deprecated and you must migrate to Clusterissuer. (note: Clusterissuer replaced Cert-Manager)

As some of you might’ve noticed, Traefik has been a bit outdated the last few weeks. The reason behind this, was a multitude of potentially breaking todo’s where left and we don’t want to bother users with continues manual intervention on breaking changes. By now we’ve fixed the remaining issues and will soon release a breaking-change release for traefik and a patch for all the charts.

In short we’ve ensured that we only use our signature “tc-system” namespace for storing configuration and middlewares for traefik. This ensures consistent behavior for users using ingressClasses and allowed us to, cleanly, fix the known bug where a port got appended to the TrueNAS SCALE “portal” button.

This also means that charts that do not get patches because they are not ported to new common, most notably: Nextcloud Will inherently not work anymore. Though, users would’ve been ill-advised using it at all currently… due to the big ongoing nextcloud rework.

Unrelated new issue

We also got the request from iX-systems staff a while ago to limit our use of non-ix-prefixed namespaces on kubernetes. While the other work to do so, requires a lot of work building our own operator helm-charts, these Traefik changes are the initial step to comply to those wishes. The “low hanging fruit”.

As we’re working hard on building separate operator helm-charts, instead of handling it in the background.This work leads to a unrelated temporary issue, which has been created on purpose: CNPG will currently only be installed on new systems, if one of our “enterprise” charts is being installed. More news about this will be released later.

For any help, please file a ticket with our dedicated support staff via the #support channel of our discord as normal.

Introducing: TrueCharts Stop-All

Previously we’ve warned users against using the stop-button on TrueNAS SCALE. At the same time we also understand, that users expect platform uniformity between Helm and SCALE. That’s why we’re happy to announce our own solution stop our Charts: TrueCharts Stop-All!

About that stop button

First off, we would like to go into a bit more depth about the design issues of the TrueNAS SCALE “Stop” Button. We’ve hinted at it previously, but it’s always good to explain why we need to step in ourselves.

The idea with Kubernetes, is that one tool should be managing deployment of objects at a time. Often indicated by a managed-by annotation on said objects. With TrueNAS SCALE, the middleware, triggers a management tool called “Helm” to deploy objects. So far so good, a GUI isn’t magically able to trigger other software, after all.

However, with the stop button, iX decided to also start editing some of those objects themselves. Specifically “Deployments” and “StatefulSets”, setting them to 0 “replica’s” meaning “run nothing”. That sounds completely fine, however: In these cases “Helm” is the actual management tool for those objects, so everything a helm action is triggered, those modifications are instantly removed.

That’s where the problems start to become bigger and bigger, because helm actions are triggered more often than people realize. For example: A reboot also triggers helm, requiring the same “hacks” to put things “back to sleep” again.

iX also decided to not even include all default objects that are technically “running”, like: Daemonsets, Jobs and Cronjobs. Which leads to issues with breaking jobs or daemonsets/jobs locking access to PVC’s. There it becomes more complicated, as kubernetes does not only exists of those “default” objects. There are also “Custom Resources”, objects that are defined by other charts and there is also the ability of other management tools, like Operators, to add objects.

When making such changes through Helm, it would be relatively easy for Chart/App developers to mitigate this. However, iX decided not to and does not even expose the “stop” button state to the Chart/App developers, leaving us completely without tools to mitigate these design flaws.

In the end, that leave out how the stop button can get into a near endless state of limbo, if not all running objects are stopped correctly… Putting the cherry on top.

Looking for a better way forward

With that all concluded, we decided to look into “what needs to be done”, to get all our Charts to have “stop” button functionality back again. It’s clear that the stop button, even with little fixes, isn’t going to be a future proof design. It completely needs to be redesigned, including all it’s backend logic. Sadly enough, refactors of said scale (pun not even intended), are currently not the priority of iXsystems, so not something we can rely on for our users.

We concluded that the only way to do so reliably, is through Helm itself. We know which objects we have, how they need to be stopped and can do so reliably through Helm. Which means: Do it ourselves!

The solution: TrueCharts Stop-All

With the most recent updates, we’ve introduced a new option: TrueCharts Stop-All This option will cause all your running objects to slowly shut themselves down or, in the case of our postgresql backend (CNPG), “hibernate”.

It’s designed to feature support for all default kubernetes objects deployed using our common chart: Daemonsets, Deployments, StatefulSets, Jobs and Cronjobs. On top we can easily expand that to cover any operator based objects, like cnpg, that needs to be shut down as-well in the future!

How To Use Stop-All

On SCALE

On SCALE this is a little checkbox on editing the App. Check it and its done

NOTE: Do not forget to uncheck the “Stop-All” checkbox to start the App again.

Using Helm

On native Helm, the same functionality is also available: Simply set the following in your values.yaml file:

global:
stopAll: true

Updates recontinued, common-migration mostly done

We’re glad to finally announce the end of our code-freeze. Since a few days we’ve re-enabled our automatic updates and within a few weeks everything should balance out again automatically!

At the same time, we’ve not completely finished porting all stable-train charts to the new common, 65 are still missing. But we’ve clearly label those updates as breaking in the changelog when they come in. Most of those are charts that have more complications than anticipated, so need a little quality time with our maintainers which takes a while.

Known Issues

Now that we’re mostly done, we also need to report a few known issues with the new backend:

  1. DO NOT USE THE STOP BUTTON

The Stop button should not be used on any TrueNAS SCALE Apps that uses postgreSQL. Due to severe design mistakes by iXsystems, it will get into an endless loop and never finish. We’re reported the issue to iXsystems and they are not interested in fixing this.

  1. PostgreSQL breaking on reboot

We’ve seen some edge cases where the new database backend breaks after a reboot. Often after the STOP button was used, though we cannot trace the issue down back to the use of the stop button itself. These issues are reported to the folks over at CNPG and we’ve also thrown them an email to discuss whether we can fund them to fix these issues.

  1. hostNetworking changes

After much R&D, our staff have discovered quite a few nasty kubernetes-level bugs with hostNetworking. As a result, we’ve decided to never enable it by default anymore on any of our charts/apps, as we cannot guarantee its stability. For some charts that, often, require this setting (like tailscale), users would have to manually and explicitly enable it from now on.

The setting has also moved in the GUI.

  1. Deprecated certificate system and you

With most Charts ported, we want to highlight the fact that the “TrueNAS SCALE (Deprecated)” certificate option, should not be used anymore. We cannot guarantee it’s stability nor can do anything at-all to help out. It will also be removed as an option in the future, though that will be months rather than weeks.

The future

With the charts slowly all being ported, we can start working on our long-term plans again. One of those plans is a renewed focus on native Helm Charts.

For May and June, we’re planning to go all-in on improving documentation for use of our charts as normal Helm charts. At the same time we’re going to work on ensuring all our SCALE specific tricks (of which only a few are left, luckily), will have automatic alternatives for normal kubernetes clusters.

To highlight this, we’ve asked Artifact hub, to highlight our Common-Library chart, as an “official” TrueCharts Helm chart. All users of helm should be able to use the power of this advanced common-library, to build the Helm Charts they please… Without even relying on TrueCharts to host their charts for them!

Check it out here and also check out the docs as always.

*Arr revert

While most of our migration to new common worked out reasonably well, we’ve received many issues with regards to another change. Our change for the “Arr” Apps, like Radarr and Prowlarr, to their new Postgresql backend ended up terribly.

We did not correctly anticipate how hard that migration was going to be for our users and also encountered a number of bugs and design mistakes for those Apps. After long consideration and attempted bug-fixing, we’ve decided to revert the move to Postgreqsql for the “Arr” Apps, back to sqlite.

This also means that after next change (which will be flagged as breaking due to moving back the database change) you will also be able to neatly import your “Arr” App backups from old common again.

We’ve very sorry for this revert and we completely understand that we should’ve done considerably more research before implementing this move to a different database version. The revert should be made available shortly, within 24 hours.

Common Porting Progress

We’re close to releasing releasing the breaking port of another 50+ of our “Stable” train charts to the new common train. With this, we want to look back on a few things we’ve noticed with the initial release:

Breaking Changes

Generally speaking, any change in the first semver digit of our versions, means a potentially breaking changes. How much this affects you, usually is effected by both the updates and your personal setup. In this specific case, we want to make extra clear that 99.9% of our SCALE Apps will require manual reinstall.

For SCALE: This also means anything in databases is going to be completely wiped unless you’ve HeavyScript/TrueTool backups and/or have followed one or more of our community migration guides. We should’ve been more clear that this behavior includes any and all databases and is not limited to MariaDB. Sadly enough this “wipe on App deletion” is a design in TrueNAS SCALE and not something we have influence over.

Our Helm users would, in most cases, with adapting their current values(.yaml) file in accordance with the new structure. though databases will still get wiped when doing the update.

GPU Support

GPU support took two snags:

  • One was an obscure SCALE bug where dicts with one item didn’t get rendered in the GUI (and it’s output) accordingly. We’ve created a temporary patch for this to compensate
  • The other was a minor permission issue, namely an additional group that should’ve been passed that got lost in development-translation from old to new common

Both are by now resolved and (being) rolled out. In the future we plan to prevent at least the first issue more thoroughly by manually checking if the interface behaves correctly when doing big GUI changes.

Addons

We’re still having some issues getting the Addons, primarily the VPN addon, behaving correctly. Mostly this is due to significantly increased hardening of our default kubernetes deployment. We expect this to be fixed within a week or two, in the mean time users depending on our charts being used with VPN might want to wait a little.

Discord

There is some annoyance over the fact we use Discord for support. We’re aware of this and are actually contemplating moving to another platform (as well). Sadly enough we do not have unlimited time available to work on the new common, release a new branding style and expand support to another platform. Users can expect a Discord alternative either end of 2023 or somewhere in 2024.

Verbal Abuse

A much less okay subject is the fact multiple of our staff members have suffered verbal abuse of varying degrees. Some have even led to cases where platform (reddit, discord etc.) needed to step in to take action. While sometimes a staff response might seem a tad blunt or not to your liking, some of the things we’ve seen are completely and utterly unacceptable. We’ve a head moderator, JagrBombs if you’ve any issue with a staff member.

We’ve taken steps to prevent needlessly exposing our staff to this. One of which is limiting our presence within certain communities on an as-needed basis.

Conclusion

In the end we’ve gotten a lot of feedback on the new release. Understandably many users are/where upset a reinstall was required. We want to highlight that we understand the frustration, but with the scope of these changes, a complete rewrite of our Common backend, we didn’t have much choice on SCALE. It’s important to note, that users on SCALE cannot update via the update button in almost all cases, so users do not have to worry about magically losing data by using the update button for this release.

Another topic we’ve seen mentioned was “but they say they are production ready”, we want to be completely clear about this: TrueCharts is not production ready at this time. In the future, after a separate announcement, only our “Enterprise” train will be considered “production ready”. We want to highlight that this does not mean “stable” users can expect these breaking changes more often, as we don’t plan to put another 700+ hours into the common chart any time soon. But it does mean, users should NEVER depend on our stable train for production, unless they do so on their own risk.

We wish all our users the best in going through these migrations and our support staff is available on Discord if you need any help.

Post Easter Stable Update

Hope everyone had an amazing easter, we know we had a busy one to say the least!

We are excited to announce that we have completed porting the first 222 charts in our “stable” train to our new “common” library chart. This chart serves as the basis for all of our apps and charts, and we believe that it will provide a more stable and reliable foundation for all of our future work.

While there are still over 160 charts left to be ported in our stable train, we expect to complete this work before the end of the month. To ensure that we have sufficient time to complete this work, we are extending our code freeze for the stable train until May 1, 2023. After this date, we guarantee that we will resume our normal update schedule.

In addition, we want to make it clear that we have lifted the code freeze for our “Enterprise” and “Dependency” charts, and will continue to provide updates for these charts on a regular basis.

It is important to note that this update is considered “most likely breaking,” and will likely wipe all databases used in charts. We also anticipate that there may be some regressions, which is why we encourage users to file bug reports or contact our support staff if they experience any issues.

We would like to take this opportunity to thank our community for their patience and understanding as we work to improve our platform. We believe that these updates will provide significant benefits in terms of stability, reliability, and functionality, and we look forward to sharing them with our users in the coming weeks and months.

As always, we welcome any feedback or suggestions from our users, and we remain committed to providing the best possible experience for everyone who uses our platform. Thank you again for your support, and we look forward to continuing to work with you in the future.

A new ChartsList

After a lot of work by @xstar and review by our staff, we’ve finally officially released our new fancy charts list. It’s now easier than ever to search and provides all the basics at a single glance! Check it out the charts list

At the same time, we’ve decided to remove/hide the list with default ports and paths. We feel that, in due time, this info should be added to our documentation on a per-application basis, on top of that it often lead to confusion as things like ports are not always as simple as they look in a spreadsheet.

As a side-note, we want to highlight the fact TrueTool is not developed anymore and the repository removed. We want to advice everyone to support @heavyBullets and use HeavyScript

HeavyScript is now also featuring special fixes, to ensure TrueCharts SCALE Apps can be stopped if they cannot be stopped in the GUI. This comes in very handy when you want to mount your PVC volumes for maintenance, check it out!

We hope that these changes makes it easier to use TrueCharts for everyone, even those that have not yet picket-out the Chart they want to use!

A fresh look for TrueCharts

We are excited to announce that the TrueCharts project has undergone a makeover with a fresh new look! Our team has been working hard behind the scenes to bring you a new logo, logo animations, and headers is clean, modern, and easily recognizable, making it a perfect representation of what TrueCharts is all about.

We’ve also created logo animations that bring our new logo to life. These new logo animations will help us stand out and make a lasting impression on our audience.

In addition to the new look, we’re thrilled to announce that we’ve launched our official merch store on Etsy!

By purchasing from our merch store, you’re not only getting a cool piece of TrueCharts swag, but you’re also supporting our project. Every purchase helps us continue to develop and improve our platform, making it the best Helm and App repository out there!

We’re very excited about the new look and the launch of our merch store, and we hope you are too. Stay tuned for more exciting news and updates from the TrueCharts project, as we continue to grow and evolve!

The Future of TrueCharts

Two and a half years ago, we started as nothing but a fork of the k8s-at-home project with added support for TrueNAS SCALE. But since then, we’ve grown quickly and surpassed k8s-at-home in many ways with our fresh custom-made common-chart.

Our focus on providing easy-to-use and reliable Helm charts for various applications has been steadily gaining popularity. As our SCALE support solidifies, we’ve been getting more and more questions about supporting other Kubernetes-based platforms. And we want you to know that we’ve been listening!

Recently, we made the first steps towards giving other platforms the attention they deserve by fixing the release workflow for building a normal Helm repository. Starting Q2 2023, we plan to slowly but steadily begin work on supporting Rancher as well, including full GUI support!

You may slowly see changes in things like our documentation structure, Discord channels, and YouTube video names, including the inevitable broken links. But we believe it will be worth it, lets say “for the greater good”…

In addition to expanding our platform support, we’ve also been secretly working on a complete redesign of our project to match our new multi-platform goals. We’re keeping the redesigns a semi-secret for now, but we’re certain that everyone will be amazed by the incredible artwork our designers have come up with!

But we’re not done yet! With the growth of the project and much feedback (Special shoutout to @HeavyBullets from https://heavysetup.info/), it slowly became clear that we couldn’t both build amazing Charts/Apps and offer specialized tools for the platforms we build for. We want to focus on what we’re good at: building those amazing charts.

Taking everything into account and with some pain in our hearts, but also knowing there is a very good alternative out there called “HeavyScript,” we’ve decided to drop development of TrueTool. We would advise everyone to migrate to HeavyScript, which offers a very similar featureset and workflow as the good ‘ol TrueTool!

Overall, we’re excited for what the future holds for the TrueCharts project. We’re committed to providing easy-to-use and reliable Helm charts for our users, and we’re always looking for ways to improve. We believe that focusing on what we do best will allow us to do even more for our users and the community.

New breaking common release

One of the core components of TrueCharts is our “common” chart, which serves as the basis on which all other charts are built.

The common chart is a massive collection of thousands of lines of code that provide a wide range of features and optimizations for all apps running on the TrueCharts platform. From November of last year until now, the TrueCharts team has been working tirelessly to completely rewrite the common chart from scratch. They’ve also incorporated feedback from the community, including feature requests and bug reports, to ensure that the new common chart meets the needs of all users.

After many months of work, the TrueCharts team is excited to announce that the new common chart is almost here! However, there are some important things that users should be aware of before upgrading. Firstly, deployment of the new common chart will take place in March 2023, and all container updates will be frozen for a month. The new common chart will be deployed in stages for the Enterprise, Dependency (except postgresql), Incubator, and April trains, and then to the stable train and postgresql dependency. This means that users may need to reinstall certain apps, and some settings such as replicas and securityContext will be reset.

Users with PostgreSQL apps will need to be aware that their databases will be nuked, so they will need to take appropriate backups before upgrading. Additionally, users will need to disable VPN before updating, as the new common chart uses a different database backend that allows for the implementation of much-requested backup features and exposes more PostgreSQL settings in the GUI in the future.

Despite these temporary inconveniences, the new common chart promises to deliver many improvements to the TrueCharts platform. For example, there is a new VPN addon based on “Gluetun,” which offers more support for OpenVPN and Wireguard and is a maintained project. This addon will replace the old “OpenVPN” and “WireGuard” options. The new common chart also includes the option to mount a config folder for OpenVPN and the option to mount OpenVPN config directly in values.yaml for native helm users.

There is also an all-new PostgreSQL backend based on “CloudNative-PG,” which supports backup, high-availability, connection pooling, and split RW/RO. Multi-pod support is also now available, which includes potential future GUI compatibility. The new common chart also features automatic detection for env-var conflicts, build-in support for jobs and cronjobs, and a completely new certificate backend based on the industry standard “Cert-Manager.”

In conclusion, the new common chart is a significant update that promises to deliver many improvements and optimizations to the TrueCharts platform. Users should take note of the deployment schedule and be prepared for some temporary inconveniences. However, the long-term benefits of the new common chart should far outweigh any short-term disruptions.

*Summarised:

March will be deployment month, which has the following consequences (until 01-04-2023):

  • In March, we will code freeze all container updates for 1 month
  • During March we will start deploying the new common in stages for the Enterprise, Dependency (except postgresql), Incubator and April trains and in that order.
  • During we will start deploying the new common to the stable train and postgresql dependency.
  • Users might have to reinstall certain apps
  • All apps will have some settings reset such as: replica’s, securityContext etc.
  • Postgresql Apps will automatically nuke their databases
  • Users will have to disable VPN before updating
  • Users will be explained that this is due to moving to a different database backend that allows us to implement the much wanted backup features and expose more postgresql settings in the GUI in the future

A short example of some of the many improvements in the new common chart:

  • A new VPN addon based on “Gluetun”, which offers much more support for OpenVPN and Wireguard and is an actually maintained project (security etc.) It will replace the old “OpenVPN” and “WireGuard” options.
  • The option to (also) mount a config folder instead of just a configfile for OpenVPN.
  • The option to mount OpenVPN config directly in values.yaml, for native helm users
  • An All-New Postgresql backend based on “CloudNative-PG”, supporting things like: Backup, High-Availability, Connection Pooling, Split RW/RO
  • Multi-Pod support, including potential future GUI compatibility.
  • Automatic detection for Env-var conflicts, if user entered custom env-vars conflict with pre-defined ones
  • Build-in support for Jobs and CronJobs, without customizations on a per-app basis.
  • A completely new certificate backend based on the industry standard “Cert-Manager”.
  • Much more hardening of defaults, based on (now automated and required) security scans backed by Datree
  • Much expanded testing suite, counting hundreds of tests for thousands of features

About TrueNAS SCALE 22.12.1

iXsystems just released TrueNAS SCALE 22.12.1 into the wild.

We’ve went ahead and reviewed the release notes, available here: https://www.truenas.com/docs/scale/scale22.12/#22121

There aren’t any changes that should prevent apps from functioning moving from 22.12 to 22.12.1. Hence we’ve decided to expand support for our Apps to both versions as well.

Please be aware, however, that both migration and backups are broken in 22.12, which are both fixed in 22.12.1. So we would still heavily advice anyone to update. Although the TrueTool hotpatch for 22.12, solves the issues for most users as well.

Relaunched: Our Helm Repository

We’re is excited to announce that our native Helm Charts are back in action!

After disabling the release of our Helm Charts to our Helm Repository at the end of last year, we received numerous reports of users being impacted by the decision.

We understand the importance of our Helm Charts and how they help users manage their applications on k8s, so we took the time to rethink our approach and create separate pipelines for our SCALE Apps and native Helm Charts. This means that decisions regarding one of them will no longer impact the other.

While our primary target is still TrueNAS SCALE, we do accept enhancement requests and bug reports that only affect native Helm users. In the future, we hope to have a separate maintainer for native Helm-specific issues, but for now, please understand enhancements and bugs that do not affect TrueNAS SCALE are less of a priority unless very severe.

We are also excited to announce the release of our completely rebuilt common chart next month, which will come with state-of-the-art testing and templating. This will empower users, both within and outside of TrueNAS SCALE, to manage their applications with ease. Be sure to check it out when it’s ready!

In conclusion, we are proud to bring back native Helm Charts to our users, and we hope that this will make managing applications on k8s a seamless and enjoyable experience for all.

Check out the Helm Repository over at: https://charts.truecharts.org

TrueCharts NFS Guide

At TrueCharts, we are always striving to make our platform more accessible and user-friendly. That’s why we are excited to announce the release of our comprehensive guide on using NFS shares to bypass TrueNAS SCALE hostPath validation.

Many of our users have expressed concern about this issue and we are happy to provide a clear and easy-to-follow solution. Our guide covers all the necessary steps to set up NFS shares from scratch and directly mount them with our apps. With just a few minutes of work, you’ll be able to use NFS shares to bypass the hostPath validation issues that have been causing headaches for many of our users.

We hope this guide will provide our users with a clear and straightforward solution to this issue. You can access the full guide here.

At TrueCharts, we are always working to improve our platform and provide the best possible user experience. With this guide, we hope to achieve just that!

February 2023 Staff Changes

TrueCharts is proud to announce some exciting changes to our core team in light of our rapidly growing project. With the growing needs of our community, we have decided to switch positions for two of our team members, @JagrBombs (Steven) and @mintyCrackers.

@mintyCrackers will now take on the role of Community Manager, where they will be responsible for fostering a positive and productive environment for our users. Meanwhile, @JagrBombs (Steven) will now serve as Head Moderator, bringing their technical expertise to the forefront of our moderation efforts.

In addition to these changes, we are thrilled to welcome a new member to our core team, XStar aka “The Noob”. XStar will be joining us as Support Manager and will also be the maintainer of the Incubator train.

With these changes, we are confident that we can take TrueCharts beyond even our wildest dreams. We believe that these additions will not only bring fresh perspectives and new ideas, but also help us better serve and support our community.

Thank you for your continued support of TrueCharts, and stay tuned for more exciting updates in the future!

Meet Jellyfin, OpenSource Media Server

As Linus TechTips recently discovered, Jellyfin is a fantastic solution for watching your media from anywhere and our app makes it incredibly easy to install on TrueNAS SCALE.

With the popularity of Jellyfin on the rise, iX-Systems has put together a great guide for setting it up on TrueNAS SCALE using our TrueCharts App. The guide provides step-by-step instructions for getting started with Jellyfin, making it simple for anyone to enjoy their media collection from anywhere. The guide can be found here

At TrueCharts, we are big supporters of open-source software, and believe it is essential to the continued growth and advancement of technology. Jellyfin is no exception, and we encourage users to contribute to the Jellyfin project by visiting their Github repository or donating to their cause through their Open Collective.

The same goes for TrueCharts. Open-source projects like TrueCharts can only survive with the support of its users. If you would like to show your support for TrueCharts, you can donate to our project through Patreon or through our website

Together, we can continue to advance the world of open-source software and provide users with the tools they need to enjoy their media collections from anywhere. Try out our Jellyfin App for TrueNAS SCALE today and see for yourself how easy it is to get started.

Traefik, Prometheus and Grafana moved to Enterprise Train

We’re started the process of moving some more of the core parts of TrueCharts to our Enterprise train. These Apps are important for most of our users and should get the priority treatment the Enterprise Train is known for. This also means users would be required to reinstall these Apps and they will not be available in the stable train anymore. We’re sorry for the added inconvenience, but we’re quite certain that this choice is going to serve our users best in the long term.

Our plans for the Enterprise train are still going strong:

When we deem TrueNAS SCALE Apps “ready for production use”, the Enterprise train will come with Paid support and much additional documentation for each of its Apps. Until that time, it will still be “just another train” and will contain Apps that are very important to the TrueCharts project as a whole.

Happy 2023, Another year of TrueCharts

Happy 2023 everyone! We hope you had a great holiday season and are ready for a productive and successful 2023. As we start the new year, we wanted to announce that we have dropped support for TrueNAS SCALE 22.02 “Angelfish” completely and will only support TrueNAS SCALE 22.12 “BlueFin” moving forward.

Looking back at the past year, we are thrilled to see the tremendous growth of the TrueCharts project. In 2022, our Discord community grew from barely 1000 members to over 5200, and our project donations increased from just 50 euros a month to more than 250 euros a month. We also saw a significant increase in Github stars and the number of apps we offer.

As we look ahead to 2023, we have big plans for the TrueCharts project. Our goal is to reach 10,000 Discord members and offer 1000 apps by the end of the year. We also hope to increase our project donations to 500 euros per month and use these funds more proactively by offering bounties for much-wanted work on the project. Additionally, we will be putting more time and effort into (video) documentation and have almost completed a full refactor of our Common Chart.

Overall, we are excited for the opportunities and potential that 2023 holds for the TrueCharts project. Thank you for your continued support and we can’t wait to see what the new year brings.