Not exactly self hosting but maintaining/backing it up is hard for me. So many “what if”s are coming to my mind. Like what if DB gets corrupted? What if the device breaks? If on cloud provider, what if they decide to remove the server?
I need a local server and a remote one that are synced to confidentially self-host things and setting this up is a hassle I don’t want to take.
So my question is how safe is your setup? Are you still enthusiastic with it?
You must log in or register to comment.
All of your issues can be solved by a backup. My host went out of business. I set up a new server, pulled my backups, and was up and running in less than an hour.
I’d recommend docker compose. Each service gets its own folder inside your docker folder. All volumes are a folder in the services folder. Each night, run a script that stops all of them, starts duplicati, backs up to a remote server or webdav share or whatever, and then starts them back up again. If you want to be extra safe, back up to two locations. It’s not that complicated if it’s just your own services.
Don’t over think it, start small, a home server. Then add stuff, you will see that it’s not that crazy.
I personally have just one home server that locally creates encrypted backups and uploads them to backblaze.
This gives me the privacy I need as everything is on my server that I own while also having the backups on a big reliable company.
It’s not perfect but it fits my threat model
Right now I just play with things at a level that I don’t care if they pop out of existence tomorrow.
If you want to be truly safe (at an individual level, not an institutional level where there’s someone with an interest in fucking your stuff up), you need to make sure things are recoverable unless 3 completely separate things go wrong at the same time (an outage at a remote data centre, your server fails and your local backup fails). Very unlikely for all 3 to happen simultaneously, but 1 is likely to fail and 2 is forseeable, so you can fix it before the 3rd also fails.
Exactly right there with the not worrying. Getting started can be brutal. I always recommend people start without worrying about it, be okay with the idea that you’re going to lose everything.
When you start really understanding how the tech works, then start playing with backups and how to recover. By that time you’ve probably set up enough that you are ready for a solution that doesn’t require setting everything up again. When you’re starting though? Getting it up and running is enough
Gonna just stream of consciousness some stuff here:
Been thinking lately, especially as I have been self-hosting more, how much work is just managing data on disk.
Which disk? Where does it live? How does the data transit from here to there? Why isn’t the data moving properly?
I am not sure what this means, but it makes me feel like we are missing some important ideas around data management at personal scale.
deleted by creator
My incredible hatred and rage for not understanding things powers me on the cycle of trying and failing hundreds of times till I figure it out. Then I screw it all up somehow and the cycle begins again.
TrueNAS scale helps a lot, as it makes many popular apps just a few clicks away. Or for more power-users, stuff like the linux cockpit also really helps.
To directly answer your questions…
- In the event of DB corruption (which hasn’t happened to me yet) I would probably rollback that app to the previous snapshot. I suspect that TrueNAS having ZFS as an underlayment may help in this regard, as it actually detects bitrot and bitflips, which may be the underlying cause of such corruption.
- In the case where a device breaks… if it’s a hard drive that broke, I just pop in a new one and add it to the degraded mirror set. If it’s “something else” that broke, my plan is to pop one of the mirror shards into a spare PoS computer (as truenas scale runs on common x86 hardware) and deal with the ugly-factor until I repair or replace the bigger issue.
- The only way to defend against a cloud provider is replication, so plan accordingly if that is a concern.
- If by “sync’d confidentially” you mean encrypted in transit, I’m pretty sure that TrueNAS has built in replication over SSH. If you meant TNO, then you probably want to build your setup over a cryfs filesystem so no cleartext bits hit the cloud, although on second thought… it’s not really meant for multi-master synchronization… my case just happens to fit it (only one device writes)… so there is probably a better choice for this.
- Setup is a hassle? Yes… just be sure that you invest that hassle into something permanent, if not something like a TrueNAS configuration (where the config gets carried along for the ride with the data) then maybe something like ansible scripts (which is machine-readable documentation). Depending on your organization skills, even hand-written notes or making your own “meta” software packages (with only dependencies & install scripts) might work. What you don’t want to do is manually tweak a linux install, and then forget what is “special” about that server or what is relying on it.
- How safe is my setup? Depends… I still need to start rotating a mirror shard as an offsite backup, so not very robust against a site disaster; Security-wise… I’ve got a lot of private bits, and it works for my needs… as far as I know :)
- Still enthusiastic? I try to see everything as both temporary and a work-in-progress. This can be good in ways because nothing has to be perfect, but can be bad in ways that my setup at any given time is an ugly amalgamation of different experimental ideas that may or may not survive the next “iteration”. For example, I still have centos 7 & python 2 stuff that needs to be migrated or obsoleted.
As an alternative, Unraid. While it’s paid, it strips away a lot of the hassle you mentioned in your post. Has a built in shop where you just click, set up ports/shares and docker containers just spin up for you.
While I’m not a huge fan of their recent subscription model change, I do love their OS (I got I’m still grandfathered into the pre-existing perpetual license.
Absurdly safe.
Proxmox cluster, HA active. Ceph for live data. Truenas for long term/slow data.
About 600 pounds of batteries at the bottom of the rack to weather short power outages (up to 5 hours). 2 dedicated breakers on different phases of power.
Dual/stacked switches with lacp’d connections that must be on both switches (one switch dies? Who cares). Dual firewalls with Carp ACTIVE/ACTIVE connection…
Basically everything is as redundant as it can be aside from one power source into the house… and one internet connection into the house. My “single point of failures” are all outside of my hands… and are all mitigated/risk assessed down.
I do not use cloud anything… to put even 1/10th of my shit onto the cloud it’s thousands a month.
It’s quite robust, but it looks like everything will be destroyed when your server room burns down :)
Fire extinguisher is in the garage… literal feet from the server. But that specific problem is actually being addressed soon. My dad is setting up his cluster and I fronted him about 1/2 the capacity I have. I intend to sync longterm/slow storage to his box (the truenas box is the proxmox backup server target, so also collects the backups and puts a copy offsite).
Slow process… Working on it :) Still have to maintain my normal job after all.
Edit: another possible mitigation I’ve seriously thought about for “fire” are things like these…
https://hsewatch.com/automatic-fire-extinguisher/
Or those types of modules that some 3d printer people use to automatically handle fires…
Yeah I really like the “parent backup” strategy from @[email protected] :) This way it costs much less.
You should edit you post to make this sound simple.
“just a casual self hoster with no single point of failure”
Nah, that’d be mean. It isn’t “simple” by any stretch. It’s an aggregation of a lot of hours put into it. What’s fun is that when it gets that big you start putting tools together to do a lot of the work/diagnosing for you. A good chunk of those tools have made it into production for my companies too.
LibreNMS to tell me what died when… Wazuh to monitor most of the security aspects of it all. I have a gitea instance with my own repos for scripts when it comes maintenance time. Centralized stuff and a cron stub on the containers/vms can mean you update all your stuff in one go
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters DNS Domain Name Service/System Git Popular version control system, primarily for code HA Home Assistant automation software ~ High Availability HTTP Hypertext Transfer Protocol, the Web IP Internet Protocol LVM (Linux) Logical Volume Manager for filesystem mapping LXC Linux Containers NAS Network-Attached Storage PSU Power Supply Unit Plex Brand of media server package RAID Redundant Array of Independent Disks for mass storage RPi Raspberry Pi brand of SBC SBC Single-Board Computer SSH Secure Shell for remote terminal access VPS Virtual Private Server (opposed to shared hosting) ZFS Solaris/Linux filesystem focusing on data integrity nginx Popular HTTP server
15 acronyms in this thread; the most compressed thread commented on today has 3 acronyms.
[Thread #821 for this sub, first seen 21st Jun 2024, 17:05] [FAQ] [Full list] [Contact] [Source code]
¯\_(ツ)_/¯ Yeah. It is kinda hard.
Backups. First and foremost.
Now once that is sorted, what if your DB gets corrupted. You test your backups
Learn how to verify and restore
It is a hassle. That’s why there is a constant back and forth between on prem and cloud in the enterprise
First of all ignore the trends. Fuck docker, fuck nixos, fuck terraform or whatever tech stack gets shilled constantly.
Find a tech stack that is easy FOR YOU and settle on that. I haven’t changed technologies for 4 years now and feel like everything can fit in my head.
Second of all, look at the other people using commercial services and see how stressed they are. Google banned my account, youtube has ads all the time, the app for service X changed and it’s unusable and so on.
Nothing comes for free in terms of time and mental baggage
Yes, you should use something that makes sense to you but ignoring docker is likely going to cause more aggravation than not in the long term.
Yep, I went in this direction…until I gave in during a bare metal install of something…
Docker is not hassle free but usually most setup guides for apps are much much easier with docker
Docker/Podman or any containerized solution is basically the easiest way to get really nice maintenance properties like: updating one app won’t break others, won’t take down the whole system, can be moved from machine to machine.
Containers are a learning curve but I think very worth it for home setups. Compared to something like Kubernetes which I would say is less worth it unless you already know or want to learn Kubernetes.
Docker takes a lot of the management work out of the equation as many of the containers automatically update. Manual updates are as simple as recreating a container with a new image instead of your local one. I would like to add try running Portainer (a graphical management interface for Docker). Breaking out the various options into a GUI helped me learn the ins and outs of Docker better, plus if you end up expanding to multiple docker hosts you can manage them all from one console. I have a desktop, a laptop, and a RPi 4b all running various dockers and having a single pane for management is such a convenience.
Not to mention the advantage of infrastructure as code. All my docker configs are just a dozen or so text files (compose). I can recreate my server apps from a bare VM in just a few minutes then copy the data over to restore a backup, revert to a previous version or migrate to another server. Massive advantages compared to bare metal.
