u/optimized-adam

The problem with current tokenizers isn't really that they are not "optimized" enough, which to me seems to be the main argument for joint learning of the tokenization function during training.

In fact, moving the learning of a tokenization function into the neural space is likely to just hide all the weird stuff that will be learned when training on large-scale data. With current tokenizers, at least we have some pretty decent ways to detect "SolidGoldMagikarp"-tokens and adding/removing tokens is possible (when applying proper methods).

hmm doesn't your point about Wq and Wk only hold for a token attending to its own key? How would we collapse Wq and Wk into Wqk when attending to different tokens?

You don’t understand how tax brackets work

Is that really correct though? RoPE only modifies the key and query states via rotation; and the angle between a token at position 128 and 256 will be exactly the same as between position 0 and 128. the angle is never used for anything else but the key-query dot product in the attention mechanism, so I don’t think we can say that RoPE encodes absolute positions in any meaningful sense for the model.

Yes it should be possible, have a look at this approach: LLM2Vec https://arxiv.org/pdf/2404.05961

They go further to turn the Causal LM into a sentence embedder but the first stage of continued pretraining for next masked token prediction should work for your case.

You are indeed correct and my interpretation was wrong.

LayerNorm does not completely remove the norm information whereas the proposed approach completely removes vector norm No, LayerNorm scales each vector to sqrt(d) norm, removing this information.

Yeah, with mixed-precision you might even end up using more memory in some cases but you get to take advantage of Tensor Cores!

This is a really, really good reply. Very few people can stay composed and thoughtful in online debates.

I went for the ML PhD and am very happy. Lots of things have happened for ML in the meantime though!

Falsch, Sam Altman will „$7 trillion“ für ein neues Unternehmen auftreiben. Vielleicht größenwahnsinnig, aber nicht so wie hier dargestellt.

Falsch, Sam Altman will „$7 trillion“ für ein neues Unternehmen auftreiben. Vielleicht größenwahnsinnig, aber nicht so wie hier dargestellt.

The image you linked matches the code, no? Notice how there is always an ADD and then a norm.

This should not be here.

Great work! I found the idea of using Capcode very intriguing and well-motivated. You write Capcode takes longer to learn but does not affect results positively or negatively. Did you observe any positive effects of using Capcode?

As an academic, I use Weights & Biases' Free Tier for Academics and it works well for me.

Neither are right, training is done in parallel using a technique called „teacher forcing“ but for inference, you sample autoregressively (talking about GPT-style models)

The 50304 was about the vocab size, not batch size (though having the batch size be a multiple of 64 should also be done probably)!

On comparing (cross-entropy) loss between different vocabularies: https://sjmielke.com/comparing-perplexities.html

TL;DR: maybe you need to do some normalization or use negative log-likelihood instead.

Monetized or not, if they are there, then there should be some proof-of-concept out there, no?

Not saying there are none, but I am skeptical indeed.

Okay let’s get concrete: In a western democracy like the U.S., will the average person have increased wellbeing?

That was a nice read :)

Would you say it’s fair to summarize all those (except maybe for the medical / protein discovery stuff) as „increased productivity“? I’m not questioning use cases of LLMs but more what they imply for society at large.

Is there a product / service already offering this?

I definitely see the potential but are we there yet? Regarding i.e. factuality and hallucinations.