u/maxtility

We provide a novel view on scaling laws, showing that the dataset size provides a hard limit on model size in terms of compression performance and that scaling is not a silver bullet.

...
Surprisingly, Chinchilla models, while trained primarily on text, also appear to be general-purpose compressors, as they outperform all other compressors, even on image and audio data (see Table 1).

We show that long-lived systems with punctuated chaos can magnify Planck length perturbations to astronomical scales within their lifetime, rendering them fundamentally indeterministic.

Also relevant: r/AIPersonhood

Follow along: https://wandb.ai/lance777/lightning\_logs/reports/metric-train\_loss-23-09-02-15-26-17---Vmlldzo1MjkzNzMw?accessToken=9843chbl7rfi1w03hxttpcnbo9z8t6088pw3ddn4h8teunaq0cy7j8hw9c5i02ve

Achieve grokking in an LLM?

I wonder if high enough training-token-to-parameter ratio might trigger grokking.

Live training loss: https://wandb.ai/lance777/lightning\_logs/reports/metric-train\_loss-23-09-02-15-26-17---Vmlldzo1MjkzNzMw?accessToken=9843chbl7rfi1w03hxttpcnbo9z8t6088pw3ddn4h8teunaq0cy7j8hw9c5i02ve

Dupe: https://www.reddit.com/r/mlscaling/comments/161txxh/andrej\_karpathy\_deep\_neural\_nets\_33\_years\_ago\_and/

• Expected availability via GCP in the fall
• Google "may start using it in some products before then"
• "it could also integrate video and audio [trained from YouTube] into the Gemini models"
• "Two longtime DeepMind executives, Oriol Vinyals and Koray Kavukcuoglu, are in charge of Gemini alongside Jeff Dean ... They oversee hundreds of employees involved in Gemini’s development."
• "Google’s lawyers have been closely evaluating the training. In one instance, they made researchers remove training data that had come from textbooks—which could help the model answer questions about subjects like astronomy or biology—over concerns about pushback from copyright holders."

"[H]undreds of people [are] scrambling to release a group of [Gemini] large machine-learning models—one of the highest-stakes products the company has ever built—this fall."

Paper link: https://arxiv.org/abs/2308.05713

At present growth rate, complete disassembly of the Earth in about 94 years (the year 2117).

https://archive.is/CH4U1

The origins of the Indo-European language family are hotly disputed. Bayesian phylogenetic analyses of core vocabulary have produced conflicting results, with some supporting a farming expansion out of Anatolia ~9000 years before present (yr B.P.), while others support a spread with horse-based pastoralism out of the Pontic-Caspian Steppe ~6000 yr B.P. Here we present an extensive database of Indo-European core vocabulary that eliminates past inconsistencies in cognate coding. Ancestry-enabled phylogenetic analysis of this dataset indicates that few ancient languages are direct ancestors of modern clades and produces a root age of ~8120 yr B.P. for the family. Although this date is not consistent with the Steppe hypothesis, it does not rule out an initial homeland south of the Caucasus, with a subsequent branch northward onto the steppe and then across Europe. We reconcile this hybrid hypothesis with recently published ancient DNA evidence from the steppe and the northern Fertile Crescent.

Paper in Science: https://www.science.org/doi/10.1126/science.abg0818

Paper in Nature Aging: https://www.nature.com/articles/s43587-023-00451-9

Maximum likelihood = Mar 2025

Mean = Jan 2027

worth noting that all of this is quite generic to just transformer language models in general. if/when openai was to release models as weights (which I can neither confirm nor deny!) then most of the code here would be very relevant.

https://twitter.com/karpathy/status/1683704060925591554

r/AIPersonhood

Preprint: https://arxiv.org/abs/2301.11772

Announcement: https://ai.meta.com/blog/generative-ai-text-images-cm3leon/

Paper: https://npg.copernicus.org/articles/30/117/2023/

On arXiv: https://arxiv.org/abs/2303.02021

See r/QuantumArchaeology

Nanohertz and picohertz regimes seem more relevant than high-frequency regimes.