I’ve been absolutely loving my TRMNL, it is such a great way to see at a glance what’s coming up! Just getting into using Liquid which is the templating engine for the widget markup. Super fun so far!
-
-
Keyboard shortcuts for Level1 Techs KVM
One more of those “I have the memory of a goldfish” posts. I keep forgetting all the shortcuts to the Level1Techs KVM (mine specifically is the 4-port hdmi 2.1 KVM).
HotKey for me is Ctrl + Ctrl
Hotkey + 1–4: Computer port selection.
Hotkey + 0: Return to the last computer.
Hotkey + f2: Starts a Manual Scan. This allows users to manually switch back and forth between power-on computers by pressing the up or down arrows to select the previous or next computer.
Hotkey + f3: Adjusts the scan rate between 3, 5, 15 and 30 seconds. The number of beeps will signify the selection.
Hotkey + hold Z + 0: Forbids PCs from sleeping states.
Hotkey + hold Z + 1: Allows PC sleeping states.
Hotkey + hold ALT + F + A + C: Reset factory default settings.
Here is a link to the instructions for the 4port hdmi 2.1 KVM
-
TrueNAS Scale ix-apps default mnt location
Because I have the memory of a goldfish, for the next time I’m trying to remember the location, it’s
/mnt/.ix-apps/
user_config.yaml is the docker configuration
/app_mounts/ contains the actual docker data.
-
Solving performance issues with Plex, like Stuttering in TrueNAS Scale
Switching from Unraid has been a huge learning process for me. There was a lot of positive things about Unraid as well as some negatives. Most of the negatives I had were around raw performance and ZFS configuration options. For positives, it felt like containers worked properly out of the box more often.
Fiddly issues with Kubernetes in TrueNAS has been a bit of an ongoing saga. From my previous post about Plex dying when I swapped GPUs to Minecraft being fiddly to setup or issues around my initial Nginx Reverse Proxy setup.
The latest came up with Plex again. For the most part it was working, but occasionally I’d have weird hiccups with performance. Like the quality being kind of low, the video just stopping or must frustrating was that every video was taking longer to start on my new server then they did with my older Unraid server.
The Specs
Unraid Server
- AMD Ryzen 3600x
- 32GBs DDR4
- nVidia Quadro M2000 4gb
- 1Gigabit NIC
TrueNAS Scale Server
- AMD Epyc 7402
- 128GBs ECC DDR4
- nVidia RTX 3060 12gb
- 10Gigabit NIC
Performance
From the specs you can probably guess that the raw performance numbers were wildly different. A great example is copying a file over to the servers.
On the Unraid server I’d see performance in the single disk write range minus the network adapters max speed so 110-120MB/s, assuming the network wasn’t the limiting factor I could probably get in the 50-250MB/s range due to parity and again, single drive performance. Reads CAN be better which is a great features if you’re wanting a media server with a bunch of random drives but if you’re accessing the same disk for two reads you can still run into performance issues.
With the TrueNAS server you can write at the 10G NICs max speed, I’d see stable 1GB/s writes fairly constantly when transferring ISOs
What was the issue?
So again I kept running into weird performance issues with Plex. I thought maybe it was my wireless network so I tweaked it a little, noticed my Apple TV was connected to the garage AP instead of the living room one so I forced that to switch. I upped the CPU count and ram even though those didn’t seem to be utilized much at all. Nothing seemed to really be helping until I ran across this post https://forums.truenas.com/t/truenas-plex-performance-unable-to-stream-network-to-slow/22223/13
“marcnarc” enabled the “Host Network” option in the TrueNAS configuration for the Plex app. This change allowed the Plex server’s container to use the TrueNAS host’s IP address instead of an internal Kubernetes/Docker subnet address. As a result, Plex performance returned to the levels experienced with CORE.
I think what may have been happening is the following.
- Host Networking ensured that full-speed, direct communication was possible, eliminating performance bottlenecks as well as inefficient relays.
- Running Plex inside a Kubernetes/Docker virtual network introduces extra routing overhead.
- If Plex thought the network was slow or remote, it might have forced lower-quality streams or unnecessary transcoding.
I think the first and second points were the key items. I think there was some relay + some routing issues that were causing the video to stop, take a while to start and suffer from lower quality.
In addition to this I upped the Transcoder default throttle buffer value to 600 to help buffer as much of a transcode as I can ahead of time which I think should help with remote streaming.
In all, these changes together made everything work perfectly and the experience finally seems to be fitting the hardware it’s running on!
-
Swapping GPU’s broke Plex in TrueNAS Scale
Issue Overview
After upgrading from a nVidia Quadro M2000 to an RTX 3060 on my TrueNAS Scale system, I ran into an issue where the Plex app failed to start when selecting the GPU for hardware acceleration. The error was pretty generic but since everything else was working and based on what I was doing I assumed it had to do with my GPU upgrade.
The error itself was pretty generic, at least to me.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 556, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1367, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 268, in nf
rv = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 287, in do_update
app = self.update_internal(job, app, data, trigger_compose=app['state'] != 'STOPPED')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 325, in update_internal
compose_action(app_name, app['version'], 'up', force_recreate=True, remove_orphans=True)
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/compose_utils.py", line 57, in compose_action
raise CallError(
middlewared.service_exception.CallError: [EFAULT] Failed 'up' action for 'plex' app, please check /var/log/app_lifecycle.log for more detailsSteps Taken to Diagnose and Resolve
1. Verified GPU Detection
First I wanted to double check that the card was being detected properly. Prior to this step I had tried going into the Plex App and removing the GPU which allowed Plex to start (further reinforcing my guess that the GPU was at fault) and enabled it again which caused the issue once more. I tried disabling, restarting and enabling but same deal.
• I ran ran
nvidia-smi -Lto confirm that TrueNAS recognized the RTX 3060.
- I ran
nvidia-smito look for errors which showed it was being detected and seemed like it was working right.
• Checked the installed NVIDIA driver version (550.127.05) and CUDA version (12.4).
Checking the OTHER logs
There is another place to look. /var/log/app_lifecycle.log contains logs. I’m not super familiar with this log but from my understanding it is a recent list of logs from K3s updates. This includes pulls, and errors when starting apps.
Using
sudo nano /var/log/app_lifecycle.logI was able to spot the next clue. Comparing the UUID from thenvidia-smi -Lcommand and the GPU that this is having errors with helped me get closer to the issue.
At this point, I think the K3s configuration is somehow holding onto the old GPU configuration. I unfortunately don’t know K3s so let’s figure out where configurations are stored.
Fixing Incorrect GPU Assignments in K3s
Weirdly, finding the mnt point for the Kubernetes configuration was harder then I expected, coming from Unraid. It turns out that it was stored within /mnt/ but hidden with a .. I found this by using
df -h(Disk Filesystem command with -h to make it human readable). I saw that the configs were stored in /mnt/.ix-apps/
At this point I was in the home stretch.
This might expose my lack of Kubernetes knowledge but I needed to figure out the configuration paths and update the GPU. I assumed it would be contained within a plex folder, so I looked at /mnt/.ix-apps/app_configs/Plex/ but the only configuration files were stored in versioned folders which I think might be used for rollback functionality?
In the end it seems like K3s stores all the configuration within one file in /mnt/.ix-apps/user_config.yaml. After a little scrolling there it was.
I was a little worried the IOMMU was the 0000:81:00.0 but I guess it wasn’t. All I had to do was update the uuid value with the new gpu, save and restart the plex app.
This was a bit of a pain, but it was a great learning experience. The best part is that it really helped me understand Kubernetes configuration better. The next thing I need to do is setup regular backup of my K3s configuration to my UNAS Pro.
- I ran
-
-
A moment for Xserve
While installing a new Unifi UNAS Pro it really hit me how good Apples industrial design is. I enjoy a good Thinkpad, I love a little RGB but there is something so beautiful about the design Apple put into something that was meant to sit in a server rack and be seen by only a few people.
At the start of my career in tech I wanted to be a designer because I was inspired by how beautiful everything Apple was.
I bought my first mac, an iBook G4 1Ghz after working on a farm all summer. Shortly after I landed a job designing brochures for a local retirement plan company and slowly realized that I didn’t have the chops for serious design and focused on my IT skills.
There will always be a part of me that watches for these on Ebay and will try to justify building something with them.
Something that would probably be a bit more realistic though would be the iMac G4 with an M4 mod. So cool!
-
How to dismantle a democracy
Really interesting article from The Atlantic titled HOW HITLER DISMANTLED A DEMOCRACY IN 53 DAYS.
More then anytime in my lifetime I feel just how fragile democracy can be. It really reminds me of that line from The Gladiator
There was once a dream that was Rome. You could only whisper it. Anything more than a whisper and it would vanish, it was so fragile.
Democracy is such a special and unique thing. Being able to disagree over and over again. To have differing opinions and ideas, with a promise that as long as those opinions and ideas don’t hurt your neighbor, or your neighbors opinions and ideas didn’t hurt you they were ok.
Today Trump is testing every perception we have of law and order. Finding and exploiting weakness in the law, in our willingness to prosecute the law or our willingness to care.
This article hits on this idea directly.
Hans Frank served as Hitler’s private attorney and chief legal strategist in the early years of the Nazi movement. While later awaiting execution at Nuremberg for his complicity in Nazi atrocities, Frank commented on his client’s uncanny capacity for sensing “the potential weakness inherent in every formal form of law” and then ruthlessly exploiting that weakness.
The American dream isn’t being able to get rich quickly. It’s the idea of freedom, safety and equality. That every person should have the ability to worship, love, dream, speak, live and work as they see fit within the confines of that promise existing for all of us.
Reading this article cements how importance of that promise being for everyone and how important the fight for that promise is.
-
This is a really cool looking project.
I’ve been looking for projects that let me disconnect a little from my phone. One of the major things that gets me is needing to look at meetings or other upcoming content that then spirals into me checking social media.
This seems like a great way to keep things like my cal, server notifications, and reminders in one place. Maybe even using an AI summarization so I could include some other content like a german word to work on each day.
Pros
- No wiring, 3-6 months battery life
- Low distraction
- Looks easy to develop for
- Simple, can be updated from your phone
- Seems well thought out from the product to the backend
- No subscription
- Can run your own webserver for the backend
Cons
- Pretty early on, and a small company so support might end up dying
- One more gadget to support
- Time to develop solutions and work on the integration
As with all other tech, my worry is that the integration time will outweigh the time it saves. But also you’re learning something new so we’ll see.
If you’re interested here is an affiliate link from the videos creator (I forgot to use it when I bought one).
-