RE: Explorations in failure - trying to get a LLM to detect LLM generated content on HIVE

I ran this on my RTX 4090, so stuff that fits within 24GB VRAM, but that does severely limit the types of models that you can run, and the token size you can feed.

You can run a fair few. They range from "yeah, that's neat" to "please, stop giving me your word salad". Commercial options and large models probably remain inaccessible unless you're using a Mac, as you suggest, with tons of fast, integrated memory.

LM Studio (its available on Mac, Linux and Windows!) was pretty performant under Linux and Windows where I tested it. My Macbook only has 24GB of ram (and its a bog standard M2 air, otherwise) is incredibly slow compared to the 4090 for any AI workloads - but I think the newer chips and more RAM are a lot more performant.

LM studio also tells you within its interface what can be offloaded entirely into memory, to avoid huge amounts of swap, and uselessly small token / context.