Prelude

A few years ago, I saw a Craft Computing video on the AsRock Rack 4U 12G BC-250, a twelve node server mining rack, originally designed to make use of binned PS5 APUs. At the time, I found them a weird and interesting piece of computer hardware, but I had zero use for them, so I forgot about them. When crypto mining was at its peak, it made sense for these to exist, but after the cryptocurrency crash, they basically became paperweights and sold for very cheap on Ebay. They had very minimal driver support, and had no way to use them for anything but mining. A small community started work on getting drivers to work for the APU, and after not too long, the drivers were mainlined into the Linux kernel. Now, they have full support for Vulkan, which made them much more appealing and slightly pricier. After the launch of Valve's Steam Machine, a small resurgence of videos appeared on my youtube feed where people converted them to DIY gaming consoles.

I had a different use case in mind.

After a bit of research, I found that these BC250s have 16GB of GDDR6 and a Vulkan capable APU. This means I can run small LLM inference, using llama.cpp.

The Cyan Skillfish*

After deliberating a little, I decided to get myself a BC250 off of Aliexpress. For the low price of 150 euro, you can get 16GB of moderately fast memory, in this economy, that's a steal. With a powersupply and 3D printed case, I now have my own BC250. I installed Fedora Server 43 and all the power and performance tweaks that the community recommends. Linux can dynamically allocate the memory between CPU and GPU but I want as much memory to be available to the GPU for inference

*Cyan Skillfish is the codename given to the BC250's APU