Natanox

To my knowledge it isn’t them constantly running that wears them out most, but spinning up and down very often. Weren’t NAS drives designed to never spin down for that very reason?

Well, they arguably can also be used as one big long-term storage. Not sure who’d need to save so much data for a long time, but there surely will be at least some people who do and buy the “modern solution” over old HDDs thinking they’re better in general. As the “family backup” for example, or as cold storage solution in faculties that can be quickly accessed if needed.

Read somewhere about a professor who used SSDs to “permanently” store important data on SSDs (perhaps in the comments of the article above) for a few years. Well, wasn’t that permanent…

More reliable

Heavily depends. If you want to use it as long-term cold storage you absolutely should not use SSDs, they’re losing data when left unpowered for too long. While HDDs are also not perfect in retaining data forever, they won’t fail as quickly when left on a shelf.

I try to like your project really hard given it’s open source, the only proper one in the social media manager category that’s self-hostable at that… but my god, this whole generative AI stuff combined with social media and marketing sounds like the epiphany of sloppy shit.

Depends on which GPU you compare it with, what model you use, what kind of RAM it has to work with, ecetera. NPU’s are purpose-built chips after all. Unfortunately the whole tech is still very young, so we’ll have to wait for stuff like ollama to introduce native support for an apples-to-apples comparison. The raw numbers to however do look promising.

May take a look at systems with the newer AMD SoC’s first. They utilize the systems’ RAM and come with a proper NPU, once ollama or mistral.rs are supporting those they might give you sufficient performance for your needs for way lower costs (incl. power consumption). Depending on how NPU support gets implemented it might even become possible to use NPU and GPU in tandem, that would probably enable pretty powerful models to be run on consumer-grade hardware at reasonable speed.

They would run with 8x speed each. Should not be too much of a bottleneck though, I don’t expect the performance to suffer noticeably more than 5% from this. Annoying, but getting a CPU+Board with 32 lanes or more would throw off the price/performance ratio.

I’m currently looking for this as well. As far as my investigation went right now I’ll probably go for 2x AMD Instinct MI50. Each of them has equivalent to slightly higher performance than a P40, however usually only 16gb VRAM (If you’re super lucky you might get one with 32gb, those are usually not labeled as such though; probably binned MI60). With two of them you got 32gb VRAM and quite the performance for, right now, 200€ / card. Alternatively you should be able to run quantized models on a single card as well.

If you don’t mind running ROCm instead of CUDA this seems like a good bang for the buck. Alternatively you might look into AMDs new line of “AI” SoCs (for example Frameworks Desktop computer). They seem to be really good as well, and depending on your usecase might be more useful than an equally priced 4090.