u/acconboy
-
20points
-
0points
How I chose to build my local AI for under a grand (self.LocalLLM) [Discussion]
-
1points
How I built my local AI for under a grand all in (self.LocalAIServers)
-
1points
How I built my local AI for under a grand all in (self.SelfHostedAI)
-
1points
How I built my local AI for under a grand all in (self.LocalLLM) [Discussion]
acconboy
acconboy
per my reply below - The reason for this build - I just finished up a Ryzen9 9950x build on a gigabyte B850 AI Top mobo with a pair of Radeon R9700 AI Pro GPUs. Runs lightning quick, but the cost of entry there is out of reach for a bunch of folks (over 5 grand). I wanted to find an easy and really inexpensive way to bolt it all together without having to A. build anything and B. have it fast enough to be useful so that folks can get a place to start
acconboy
The reason for this build - I just finished up a Ryzen9 9950x build on a gigabyte B850 AI Top mobo with a pair of Radeon R9700 AI Pro GPUs. Runs lightning quick, but the cost of entry there is out of reach for a bunch of folks (over 5 grand). I wanted to find an easy and really inexpensive way to bolt it all together without having to A. build anything and B. have it fast enough to be useful so that folks can get a place to start
acconboy
The reason for this build - I just finished up a Ryzen9 9950x build on a gigabyte B850 AI Top mobo with a pair of Radeon R9700 AI Pro GPUs. Runs lightning quick, but the cost of entry there is out of reach for a bunch of folks (over 5 grand). I wanted to find an easy and really inexpensive way to bolt it all together without having to A. build anything and B. have it fast enough to be useful so that folks can get a place to start
acconboy
Yeah, that is why I wanted to do this build - I just finished up a Ryzen9 9950x build on a gigabyte B850 AI Top mobo with a pair of Radeon R9700 AI Pro GPUs. Runs lightning quick, but the cost of entry there is out of reach for a bunch of folks (a bit over 5 grand). I wanted to find an easy and really inexpensive way to bolt it all together without having to A. build anything and B. have it fast enough to be useful so that folks can get a place to start
acconboy
not yet, it is in the middle of embedding all of the Project NOMAD stuff - contains a full copy of Wikipedia (no images), full copy of the Khan Academy k-12 and university coursework, full copy of the Sikana practical learning coursework, full copy of global maps down to the street level, full copy of the CyberChef tool set, Full copy of the Project Gutenberg everything library. Will likely take a while to finish ingesting, but will benchmark when it is done if you like. Am also running chat prompts against it while it is doing all of the above and getting around 25t/s
acconboy
Around 25 t/s or so. very useable
acconboy
I have a couple of them lying around, and no one else has built a similar cost fully embedded portable system with 16gb vram that I can take anywhere ( kind of the point of Project NOMAD)
acconboy
Also, for what the latest 5090 costs ($12000), I can build and run a pair of the Ryzen/Radeon machines and cluster them
acconboy
haven't run it against the Ryzen/Radeon or the serpent canyon with the A770M. The Ryzen build is running NodeWeaver for the virtualization stack and pci passthrough of the pair of cards to the the VM running docker with llamacpp and orinth 1.5-35B along with switchyard and hermes. It goes from cold to booted ready to run in a wallclock 20-ish seconds. Holds right around 125 t/s on pretty much all queries/prompts. Did do a bit of benchmarking on the Strix Halo though with the same stack (other than running qwen 3.8-flash-next 27B running Vulkan alongside having an MOE and judge model loaded in RAM at the same time. This is what I get on it with a handful of other X86 workloads/vm's running on the same machine
acconboy
I actually really liked his country album. Felt more real and less about anger.
acconboy
Am running a ryzen 9 9950x witha pair of the 9700s. They scream under Vulkan. Intel killed off most of their open-source efforts so was a nonstarter for me. Shame, I have an A770 and liked it quite alot, but intel dropped the ball
acconboy
High availability and hardware failure tolerance. In this case, I can run my other X86 workloads on the same box at the same time
acconboy
Doing the same on ms-s1 max, but virtualized under NodeWeaver with the same model under Vulcan
acconboy
Well, I have a 1992 Chevrolet K2500 pickup that I repair myself when needed and take care of the maintenance myself. Have well over 300,000 miles on it and it runs fantastic. For what a new truck payment is monthly, I could drop a rebuilt engine in 2x a year and still spend less..
acconboy
Amusingly enough, I used the new style Rhino ramps to level out the front of my RV and yes, they indeed started bending and cracking. I switched over to the old style ones from Harbor Freight a year ago and those are way more solid - no flexing or bending at all.
acconboy
all true, my serpent canyons have 32 gb each & I don't do too much video creation, but the Serpents (Nuc12 Enthusiast) have the ARC A770M built in.
acconboy
not bad, a touch slower, and dependent on which model it is running. Totally useable
now that the indexing is done, here is a quick screengrab of output in T/S. 40 and change
https://preview.redd.it/xrt4te8g4wsh1.jpeg?width=1280&format=pjpg&auto=webp&s=10fbf055ed200eba7e4bc46faca277e402211eab