u/ru552

Put it on The Ocho

I love Claude, but I cancelled after seeing this last week. They have the best model for my use right now, but they can't have my money anymore unless they change their stance.

two mac books

then you turn the computer off

use outlines or instructor and in-context learning with 5 examples

I run it on my mac will ollama and pull the model from ollama here:https://ollama.com/library/llama3:70b

Your first 3 points are debatable regarding OAI having an "edge". I give you the last 2. 4o actually seems to go backwards in some real world areas (coding specifically) compared to April versions of 4t so better long answers and reasoning is mostly a vibe. Faster responses is true compared to previous OAI models, but not when compared to models running on Groq.

got a lot of Phi-3 humpers in here...

some of them have 160k context length

I don't care anything about these models, but you sir, deserve an award for your naming convention.

yes, but 120b is a normal size llm to run on apple silicon. There are much larger.

Maybe https://www.gradio.app/

You forget the best thing about the Mac. There isn't a model available today that won't run on apple silicon with 192GB of ram.

https://github.com/run-llama/llama_index

There's a ton of examples on their github.

This one's local, uses ollama for LLava, uses lamaindex for RAG.

https://docs.llamaindex.ai/en/latest/examples/multi_modal/ollama_cookbook.html#

Lammaindex is my go to for RAG

https://github.com/run-llama/llama_index

https://www.llamaindex.ai/

That's the easiest way I've found to use RAG.

Apple GPU can only access up to 96GB on the 128GB RAM version for example.

sudo sysctl iogpu.wired_limit_mb=12345

That takes care of that for you. You probably want to leave atleast 6GB of RAM for the OS though.

with that much unified RAM, you don't need GGUF. You can pretty much run any model if you allocate 122g of your ram to run on the GPU.

edit before it's asked, here's the command to change how much ram is allocated to the GPU:

sudo sysctl iogpu.wired_limit_mb=12345

restarting the mac will set it back to system default

https://huggingface.co/mlx-community

It's only been out for < week.

Well, Japan has declared that training LLMs on copywritten works falls under fair use. That's a big gray area that's now black / white for people in Japan.

an m1 macbook

The things in RAM are processed by the CPU. The GGUF versions of models is what allows this to happen since you can run models that don't fit entirely in VRAM. The upside is you can run large models, the downside is that they are slow since you're using CPU.