Out of curiosity, what software is normally being run on your clusters? Based on my reading, it seems like some companies run clusters for business purposes. E.g. an engineering company might use it for structural analysis of their designs, or a pharmaceutical company might simulate the interactions of new drugs. I assume in those cases they’ve bought a license for some kind of high-end software that’s been specifically written to run in a distributed environment. I also found references to some software libraries that are meant to support writing programs in this environment. I assume those are used more by academics who have a very specific question they want to answer (and may not have funding for commercial software) so they write their own code that’s hyper focused on their area of study.

Is that basically how it works, or have I misunderstood?

This actually came up in my research. Folding@Home is considered a “grid computer” According to Wikipedia:

Grid computing is distinguished from … cluster computing in that grid computers have each node set to perform a different task/application. Grid computers also tend to be more heterogeneous and geographically dispersed (thus not physically coupled) than cluster computers.

The primary performance disadvantage is that the various processors and local storage areas do not have high-speed connections. This arrangement is thus well-suited to applications in which multiple parallel computations can take place independently, without the need to communicate intermediate results between processors.

I’ll have to look a little more into the AI stuff. It was actually my first thought, but I wasn’t sure how far I’d get without GPUs. I think they’re pretty much required for Stablediffusion. I’m pretty sure even LLMs are trained on GPUs, but maybe response generation can be done without one.

I’m not sure what you’d want to run in a homelab that would use even 10 machines, but it could be fun to find out.

Oh yeah, this is absolutely a solution in search of a problem. It all started with the discovery that these old (but not ancient, most of them are intel 7th gen) computers were being auctioned off for like $20 a piece. From there I started trying to work backwards towards something I could do with them.

I was looking at HP mini PCs. The ones that were for sale used 7th gen i5s with a 35W TDP. They’re sold with a 65W power brick so presumably the whole system would never draw more than that. I could run a 16 node cluster flat out on a little over a kW, which is within the rating of a single residential circuit breaker. I certainly wouldn’t want to keep it running all the time, but it’s not like I’d have to get my electric system upgraded if I wanted to set one up and run it for a couple of hours as an experiment.

Youtube in has done a remarkably good job carrying the torch of high quality documentaries and educational content beyond the realm of traditional media. Science, art, technology, history. It’s all there, and much of it meets or exceeds the quality of anything the old guard of cable TV channels ever managed to produce.

I’m actually only now realizing that some of the most established channels have been reaching a wide audience with consistent and high quality content for the better part of a decade, and yet I can’t think of any who have successfully broken into more “traditional” media such as television or or even streaming services. That seems exceptionally strange to me. I mean, last month there were headlines about Netflix giving $55 million to an unproven director who proceeded to blow it all on expensive cars instead of filming the show he was hired to make. Who decides to hire that guy over any number of youtube creators who have spent the last ten years cranking out a short video a week along with occasional longer form projects, all with a small crew on a shoestring budget. I can imagine three possible reasons for this. No idea which one(s) could be the real reason, or if there’s something else entirely going on.

1. Hollywood¹ is so insular that they don’t even realize these people exist.
2. Hollywood is so stuck in its ways that they refuse to believe these people could be successful running a larger production.
3. Offers have been made, but those offers have been so restrictive that any number of youtubers have turned them down despite, one would assume, a large amount of money being on the table if they go along with it.

That last one in particular seems unlikely, but I do recall that the popular Primitive Technology channel went quiet for a year or more before abruptly coming back to life. Rumors swirled that he had been hired to turn the concept into a TV show, but the production company kept trying to change things and he eventually gave up and went back to doing it his way on youtube.

¹ used here as shorthand for the more corporate and structured entertainment industry at large.

I remember reading a comment a while ago (on Reddit, ironically) which pointed out that SFW subreddits naming themselves [subject]porn are borrowing the wrong part from the word “pornography”. “Porn” is from greek pornē meaning “prostitute”, but the suffix -graphy means “to write” and is often used to indicate “the study of” the thing it’s attached to (e.g. geography, cryptography, demography, etc.)

It would be more accurate, and perhaps less controversial, if these communities named themselves earthography, spaceography, unixograpy, etc. As an added bonus, the -grapy suffix is also prominent in “photography” which is appropriate considering that many of these communities are places where people share photos of the subject matter.

Slightly off topic, but as long as we’re ranting about DNS…

Proxmox handles DNS for each container as a setting in the hypervisor. It’s not a bad way of simplifying things, but if, hypothetically, you didn’t know about that, then you could find yourself in a situation where you spend an entire afternoon trying every single one of the million different ways to edit DNS in Linux and getting increasingly frustrated because the IP gets overwritten every time you restart the container no matter what you do, until eventually you figure out that the solution is just like three clicks and a text entry box in the Proxmox GUI!

…Hypothetically, of course.