More Parameters, Fewer Errors: Double Descent, Grokking, and the Riddle of Generalization

The Hook: A Network That Memorizes Noise

Picture an experiment simple enough to set up in an afternoon, whose result nevertheless shook half a century of statistical textbook wisdom.

You take CIFAR-10, a standard image dataset: 50,000 small color photographs across ten classes — airplane, car, bird, cat, and so on. You take an ordinary convolutional network, the kind that defined the state of the art at the time. Then you do something absurd: you throw away the labels and replace them with random ones. The picture of a cat is now labeled "truck"; the airplane is labeled "frog." There is no structure left, no learnable rule, nothing. The mapping is pure noise.

Then you train.

The network learns it. Not approximately — completely: training error zero. It has memorized 50,000 arbitrary assignments. And if you replace the images themselves with Gaussian noise, stripping the inputs of structure as well, it learns that too.

This result is due to Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals, and it appeared at ICLR 2017 under a title that was itself a program: Understanding deep learning requires rethinking generalization. Their phrasing: modern convolutional networks "easily fit a random labeling of the training data," and the phenomenon is "qualitatively unaffected by explicit regularization" — dropout, weight decay, and data augmentation change nothing fundamental about it.

Now comes the part that hurts. According to classical learning theory, a model that can memorize 50,000 random labels is worthless. Its capacity is so large that its generalization guarantees collapse: anything that can fit everything has learned nothing. The theory says this model will fail on new data.

That same model, trained with the correct labels, achieves world-class accuracy on images it has never seen.

Both statements are true. And between those two truths lies the most interesting open question in machine learning: why does deep learning work when, according to the theory we have, it should not?

This article tells the story of what we have learned in the roughly ten years since that finding. It is not a story with a triumphant ending. It is the story of a curve we had drawn wrong, of a prehistory we had failed to read, of a phenomenon with a silly name that exposed the temporal dimension of the problem — and of an objection that claims we were labeling the wrong axis all along.

Part 1: What Classical Statistics Promised

To understand what broke, you have to know what was promised. And the promise was elegant, intuitive, and reliable for decades.

A model's expected error can be decomposed into components. For squared loss the decomposition is exact:

Expected test error = bias² + variance + irreducible noise

Bias measures systematic error: how far is the best possible function my model class can even represent from the truth? A straight line asked to approximate a parabola has high bias, no matter how much data you give it.

Variance measures instability: how much does my learned function change if I swap the training set for a different one of equal size drawn from the same distribution? A degree-49 polynomial through 50 points has enormous variance — nudge one point and the curve swings wildly somewhere else.

From this follows the famous U-curve. Increasing model complexity lowers bias and raises variance. Early on, the bias gain dominates and test error falls. Eventually the variance loss dominates and test error rises. Between them lies an optimum — the "sweet spot" — and the entire discipline of model selection consisted of finding it: cross-validation, AIC, BIC, regularization paths, pruning decision trees.

This curve is in every textbook. It is painted on the wall of every statistics seminar room. And it carries a clear normative consequence: a model that interpolates the training data exactly — zero training error — is by definition past the optimum. It has memorized the noise. It is overfitted. It will generalize poorly.

That consequence was so self-evident that it stopped being perceived as a hypothesis at all and became a property of the world. And it is precisely the part that is wrong — or at least so incomplete that it misleads.

Part 2: The Double Descent

In 2019, Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal published a paper in the Proceedings of the National Academy of Sciences (volume 116, pp. 15849–15854) with a programmatic title: Reconciling modern machine-learning practice and the classical bias–variance trade-off. Their proposal: the U-curve is not wrong. It is merely truncated. For decades we had looked only at its left half, because no one was looking to the right.

The central quantity is the interpolation threshold: the point at which the model's capacity is just sufficient to hit the training data exactly (roughly: number of parameters \(\approx\) number of training examples). Up to that point, everything behaves as the textbook says: U-curve, minimum, then rise. And right at the threshold, test error spikes — there the model is at its worst, because there is exactly one function passing through all the points, and it has to accommodate every ripple of noise.

The surprising part begins to the right. Push capacity further, far past the interpolation threshold, and test error falls again — often below the level of the classical sweet spot on the left. Belkin et al. put it this way: "increasing model capacity beyond the point of interpolation results in improved performance." The complete curve therefore has two valleys: the classical one on the left and a second, often deeper one on the right. Hence the name double descent.

Why Things Improve Beyond the Threshold

The explanation is counterintuitive at first and then, once you have seen it, nearly obvious.

To the left of the threshold there is no interpolating solution — the model cannot hit the data exactly, so it looks for the best available approximation. At the threshold there is exactly one — and you have no choice, you get it, including every contortion the noise forces upon it.

To the right of the threshold there are infinitely many interpolating solutions. All have zero training error. And with that, the decisive question is no longer "does my model interpolate?" but "which of the many interpolating solutions does my training procedure pick?"

That is where the leverage lies. Gradient descent does not pick at random. It has a preference — for solutions of small norm, meaning "smooth," low-energy functions. This is called implicit regularization: the optimizer regularizes without being asked to. The more parameters you have, the larger the space of interpolating solutions, and the smoother the smoothest among them can be. More capacity therefore means not more contortion but more choice — and the optimizer uses that choice well.

Belkin et al. demonstrated this not only for neural networks but for a range of simpler model classes where one can compute and follow along: Random Fourier Features (random trigonometric basis functions whose count can be varied cleanly), decision trees, and random forests. That was the rhetorically most important part of their work: double descent is not a peculiarity of deep networks. It is a phenomenon of overparameterized interpolation as such.

Part 3: The Prehistory Nobody Had Read

Intellectual honesty on this topic requires turning a footnote into a chapter. Because double descent was not new in 2019. It was rediscovered.

A year after Belkin's article, in 2020, Marco Loog, Tom Viering, Alexander Mey, Jesse Krijthe, and David Tax published a two-page letter in the same Proceedings (volume 117, pp. 10625–10626) under the dry title A brief prehistory of double descent. Their finding: the curve had been known for decades under a different name, namely the "peaking phenomenon" — the observation that learning curves exhibit a peak exactly where model complexity reaches sample size, only to fall again afterward.

The evidence Loog et al. marshal is concrete:

• Vallet et al. (1989) experimentally demonstrated double-descent curves for classifiers based on minimum-norm linear regression.
• Opper et al. (1990) supplied the first theoretical result: the pseudo-inverse solution improves as one moves away from the peak at \(\alpha = 1\) (sample size relative to complexity).
• Skurichina and Duin (1998) showed experimentally that adding random features improves performance precisely at the critical point — that is double descent in pure form, 21 years before the name.
• Raudys and Duin (1998) analyzed the effect for the pseudo-Fisher linear discriminant, Duin (2000) was first to show it on real-world data, and Krijthe and Loog (2016) extended it to the semi-supervised setting.

Their statement of the basic pattern — "With increasing N, the risk initially decreases, attains a minimum, and then increases until N equals n" — is exactly the left half of the Belkin curve, written in the language of 1990s pattern recognition.

I am of the opinion that this letter contains the more important lesson, beyond the bibliographic correction. Phenomena do not become part of knowledge through observation alone, but by being observed within the right conceptual frame. Vallet and Opper saw a curiosity in a corner case. Belkin et al. saw the same curve but stood in a world where overparameterized models were the industrial default — and could therefore read it as the refutation of a textbook rule. The same data point, two different insights. Anyone working in a fast-moving field should take that as a warning: the chance that a surprising finding has been made before is high. The chance that it was understood back then is considerably lower.

Part 4: Deep Double Descent — and the Second Axis

Belkin's curve describes capacity. But deep learning has more than one dial: you can make the model bigger, you can train longer, you can use more data. What happens along those axes?

The answer came from Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever in Deep Double Descent: Where Bigger Models and More Data Hurt (arXiv December 2019, ICLR 2020, later J. Stat. Mech. 2021, article 124003). They found three things, and the third is the most uncomfortable.

First: model-wise double descent. Across ResNet18 with varying width, five-layer CNNs, and Transformer encoder-decoders on CIFAR-10, CIFAR-100, IWSLT'14 (German→English), and WMT'14 (English→French), the double valley shows up in modern architectures. The peak is not located arbitrarily but systematically at the interpolation threshold: "The peak in test error systematically occurs at the interpolation threshold."

Second: epoch-wise double descent. This is the genuinely new finding. Hold model size fixed and vary only training duration, and the same shape appears: test error falls, rises again, and then falls once more. In Nakkiran's words: "double descent occurs not just as a function of model size, but also as a function of the number of training epochs."

That two different axes produce the same curve suggests the real quantity is neither of them. Nakkiran et al. therefore propose effective model complexity (EMC): not the parameter count, but the largest number n of training examples that the training procedure can still fit (up to \(\varepsilon\)) without error. Formally:

\(\mathrm{EMC}^{D}_{\varepsilon}(T) := \max{\, n \mid \mathbb{E}_{S \sim D^{n}}\left[\mathrm{Error}_S(T(S))\right] \le \varepsilon \,}\)

The crucial symbol is T: complexity is not a property of the model but of the training procedure — architecture, optimizer, learning rate, epochs, regularization, all together. EMC grows with width, but it also grows with training duration. That is why both axes yield the same curve: they are two routes to increasing the same quantity.

From this they formulate the generalized double descent hypothesis: well below EMC \(\approx n\), more complexity lowers test error. Well above, more complexity lowers it too. Only in the critically parameterized regime, where EMC \(\approx n\), is the behavior indeterminate — there "a perturbation of T that increases its effective complexity might decrease or increase the test error."

That is a remarkably honest sentence for a paper. It says: in a narrow regime — precisely the one many practitioners inhabit for budget reasons — the direction of an intervention's effect is not predictable.

The Uncomfortable Finding: More Data Can Hurt

Now the third result, the one that contradicts intuition most sharply. The EMC perspective implies a prediction nobody would volunteer: if more data shifts the interpolation threshold rightward, then a model that was comfortably overparameterized before can be pushed into the critical regime by the data increase — and get worse there.

Nakkiran et al. observed exactly that. For Transformers on IWSLT'14 German→English, enlarging the dataset by a factor of 4.5 (from roughly 4,000 to roughly 18,000 examples) produced worse test performance. Their phrasing: "Increasing the number of samples by a factor of 4.5 results in worse test performance."

This is neither a measurement error nor a pathological special case. It is the logical consequence of the geometry: more data is good when you are far from the threshold; it is dangerous when it slides the threshold directly underneath your model. The practical takeaway worth memorizing: "more data always helps" is not a truth but a heuristic with a known exception zone — and the exception is exactly the situation where model size and data size are similarly scaled.

That label noise acts as the amplifier here fits the picture. Nakkiran et al. tested 10%, 15%, and 20% label noise; with more noise, the peak at the interpolation threshold becomes markedly more pronounced. With clean labels it is often so flat that it goes unnoticed. That incidentally explains why the phenomenon went undetected for so long: in well-curated benchmark datasets it is weak. In real enterprise data — where 10% wrong labels is on the optimistic side — it is strong.

Part 5: Benign Overfitting — the Theory Behind It

Phenomenology is useful, but it explains nothing. Why is an interpolating model allowed to be good? Here statistical learning theory supplies an answer, though in a setting deliberately kept simple: linear regression.

Peter Bartlett, Philip Long, Gábor Lugosi, and Alexander Tsigler published Benign Overfitting in Linear Regression in PNAS in 2020 (DOI 10.1073/pnas.1907378117). Their opening names the riddle precisely: "The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data."

Their approach: study the minimum-norm solution that interpolates the data exactly, and ask under what conditions on the data distribution its risk stays small. The answer depends not on parameter count but on the structure of the covariance matrix of the inputs — characterized through notions of effective rank.

The intuition worth carrying away is this: the noise in the training labels has to go somewhere. In a model with few directions, it must go into the directions that also carry the signal prediction — and there it spoils the prediction. In a model with very many directions, especially many weakly weighted ones, the noise can spread across a large number of unimportant dimensions where it does little harm. Overfitting becomes benign when there are enough harmless hiding places for the noise.

Their result is notably unromantic: overparameterization is "essential for benign overfitting in this setting" — but it is not sufficient. The distribution has to cooperate, and the range of admissible distributions is considerably narrower in finite-dimensional spaces whose dimension grows faster than the sample than in the infinite-dimensional case.

For practice this means: benign overfitting is not a law of nature you can lean on, but a property of particular data-model constellations. That it works well for images and language is an empirical fact about images and language, not about interpolation as such.

Part 6: Grokking — Insight Arriving Late

In 2022, Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra published a finding that pushed the temporal dimension to its extreme: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (arXiv, January 2022). The name is a verb from Robert Heinlein's novel Stranger in a Strange Land, where it means roughly "to understand something all the way through."

The setup is deliberately tiny: a small Transformer trained on algorithmically generated tasks — modular addition, for instance, a + b mod p. You give the network part of the addition table for training and hold back the rest as the test set.

What happens unfolds in phases:

Phase 1 — memorization. After relatively few steps the network has memorized the training examples: 100% training accuracy. Test accuracy sits at chance level. This is the classical picture of overfitting. Every textbook and every sensible early-stopping criterion says, at this point: stop, it will not get better.

Phase 2 — nothing. You keep training. And keep training. Tens of thousands of steps during which training accuracy stays pinned at 100% and test accuracy does not budge. From any observer's perspective, training is finished and has failed.

Then, suddenly, phase 3. Test accuracy leaps from chance level to near-perfect. The network has found the rule. Power et al. describe the improvement as going "from random chance level to perfect generalization," and doing so "well past the point of overfitting." They also found that the smaller the training set, the longer the wait — "smaller datasets require increasing amounts of optimization for generalization."

If double descent refutes the claim "interpolation is the end," then grokking refutes the claim "when the validation curve goes flat, there is nothing left to gain." Both tenets were load-bearing walls of everyday practice.

What Happens Inside

A sudden jump in a metric is an invitation to mystify. The only cure is to look inside the model — which is exactly what Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt did in Progress measures for grokking via mechanistic interpretability (ICLR 2023).

They took the small modular-addition Transformers and fully reverse-engineered the learned algorithm. The result is beautiful: the network has translated addition into rotation. It maps the inputs to angles, uses discrete Fourier transforms and trigonometric identities to represent the sum as a rotation about a circle, and reads the answer off the angle. Modular addition is rotation — the network found the problem's natural representation by itself. This was confirmed not by interpreting diagrams but by ablations in Fourier space: switch off the identified frequency components and performance collapses.

With that understanding they defined progress measures that continuously track what accuracy only reveals as a jump. And with those, training decomposes into three phases: memorization, circuit formation (the Fourier mechanism gradually grows in the weights while the metrics look flat), and cleanup (the memorizing components are dismantled, typically driven by weight decay).

Their own summary dispels the mysticism: grokking "rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components."

That is the real insight: the jump was never in the model. It was in our metric. Test-set accuracy is a threshold measurement — it shows nothing while the generalizing solution is weaker than the memorizing one, and everything as soon as it prevails. Underneath runs a continuous process.

For anyone who monitors systems, this is a transferable lesson. A metric that stays flat for a long time and then tips over abruptly is strong evidence that you are observing a threshold rather than the process. The question is then not "why did it happen suddenly?" but "which continuous quantity should I have been measuring instead?"

Omnigrok: Grokking as Geometry

Ziming Liu, Eric Michaud, and Max Tegmark supplied the complementary view in Omnigrok: Grokking Beyond Algorithmic Data (ICLR 2023) — not "which algorithm" but "which landscape."

Their observation: plot training and test loss against the weight norm and the two curves have different shapes. Training loss falls and then stays low across a wide range of norms — the shape of an "L." Test loss has a genuine minimum at an intermediate norm and rises on both sides — the shape of a "U." They call this the LU mechanism.

This turns grokking into an itinerary. Initialization drops the network at large weight norm. There it quickly reaches low training loss (the L is flat) but high test loss (the U is high there) — memorization. Weight decay then slowly pulls the norm down, toward the U's minimum. Training loss stays low the whole time, nothing visibly happens — until the network wanders into the region of good generalization, which in the metric looks instantaneous.

This effortlessly explains what is seen empirically: the dependence on dataset size, the dependence on weight decay, the emergence of representations. And it gave Liu et al. a test stronger than any explanation: they could induce grokking on demand — on image, language, and molecule tasks, far outside algorithmic toy data — and conversely eliminate it for algorithmic data by adjusting the initialization norm. A mechanism that lets you switch a phenomenon on and off is considerably more than a narrative.

Part 7: The Objection — Is Double Descent an Artifact?

A story consisting solely of confirming findings should make you suspicious. So, to the objection — and it is a good one.

Alicia Curth, Alan Jeffares, and Mihaela van der Schaar argue in A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning (NeurIPS 2023) that double descent in the classical, non-deep examples — trees, boosting, linear regression — is an artifact of parameter counting.

The core of their argument: when you increase "the complexity" from left to right in such examples, you silently switch complexity axes along the way. With tree ensembles, for instance, you first grow the trees (depth, leaves) — producing the classical U-curve. Once that is exhausted, you increase the number of trees in the ensemble — a completely different axis, along which averaging over models reduces variance. The famous curve is then not one curve but two curves glued together. Their phrasing: "the second descent appears exactly (and only) when and where the transition between these underlying axes occurs."

Their constructive counterproposal is to count not raw parameters but the effective number of parameters a model actually uses when issuing predictions for new inputs — a quantity from smoother theory that measures how much smoothing occurs across training labels. That quantity notably does not keep growing in the interpolation regime, even though the raw parameter count does. And when you plot test error against it, here is what happens: "once we control for the effective parameters of each model in this way, we consistently observe that each example indeed folds back into shapes that are best characterized as standard convex (U-shaped) curves."

The curve folds back into a U. Their punchline in five words: "not all model parameters are created equal."

How should this be weighed? I am of the opinion that this objection convincingly demystifies the classical examples without dissolving the deep-learning riddle. It shifts the question, and does so productively: no longer "why does the U-curve break?" but "what is the right complexity axis for a neural network?" Nakkiran's effective model complexity and Curth's effective parameter count are two answers to the same question from different traditions. That both were needed shows one thing above all: parameter count, the number with which we name models and write press releases, is largely useless as a measure of complexity.

Part 8: A Decision Framework for Practice

Four phenomena, one question: what do you do with this when you have to train a model on Monday? The table below is my attempt to translate the findings into actions.

Three things I regard as robust takeaways for everyday work:

First: the validation set has not shrunk in importance, it has grown. When theory supplies no reliable prediction about direction — and in the critical regime it explicitly does not — then measuring is not the cheap alternative to understanding but the only available procedure. A clean test set never used for tuning is, under these circumstances, worth more than any intuition about parameter counts.

Second: "more parameters than data points" is no longer an alarm signal. It is the default condition of every modern model. Rejecting an architecture in a code review on the grounds that "it has more parameters than we have examples" means invoking a rule whose domain of validity has been known to be limited since 2019.

Third: monotonicity assumptions deserve testing, not belief. "Bigger is better," "longer is better," "more data is better" — all three are empirically right most of the time and demonstrably not always. The regime where they flip is the same in each case: the critical zone around the interpolation threshold.

The Central Takeaway

The bias-variance U-curve has not been refuted. It has been exposed as a special case — valid to the left of the interpolation threshold, blind to everything happening on the right, and drawn on an axis (parameter count) that turned out to be the wrong measure of complexity. The counter-movement from Curth et al. confirms this from the other side: with the right axis, the curve folds back into a U in many classical cases. Both findings point the same way — our problem was never the curve, it was its labeling.

The shared pattern behind Zhang, Belkin, Nakkiran, Bartlett, Power, Nanda, and Liu is this: what a model can do is not what it does. Capacity says which functions are representable. What actually gets learned is determined by the interplay of optimizer, initialization, regularization, and data structure — and that interplay has no established measure. This is why the same network could memorize CIFAR-10 with random labels and generalize with real ones: capacity was identical in both cases, while the learning procedure's implicit preference met a different data structure each time.

Concretely, if you want to do one thing this week: take the last model where you stopped training because "the validation curve wasn't going down anymore," and instrument three continuous quantities: loss instead of accuracy, the norm of the weights, and the training error on the noisiest 10% of your labels. If any of them still shows movement during the flat phase, you probably stopped too early — and you have gained a diagnostic tool that threshold metrics will never give you.

Cross-References in the Vault

Double descent and the scaling laws are two answers to the same question about the relationship between model size, data volume, and error — Chinchilla gives the recipe, this article the pathology at the margin: How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI. The mechanistic reconstruction of the Fourier algorithm in grokking comes straight from the toolbox described here: The Ghost in the Machine: How to Read a Neural Network From the Inside. The implicit regularization underlying all of it is a property of gradient descent itself: The Error That Learns Backward: Backpropagation, the Chain Rule, and the Algorithm Behind Deep Learning. That the grokking experiments ran on small Transformers is no accident — the architecture is what makes such micro-experiments tractable in the first place: Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI. Philosophically, the problem at stake here is Hume's: why should a function that fits seen data fit unseen data? Double descent shows the answer hinges on selection preference, not on data volume: Why the Sun Will Rise Tomorrow: Hume's Problem of Induction, Grue, and the Limits of Learning. And the idea that a long phase of apparent standstill can be the actual learning process has a surprisingly precise parallel in human learning research: Desirable Difficulties: Why Effortless Learning Deceives and Memory Lives on Effort.

Something to Think About

Grokking is not a phenomenon of the model but of our measurement: the jump arises because we are watching a threshold while a continuous process runs beneath it. Test accuracy shows nothing until the generalizing procedure outweighs the memorizing one — before that, nothing at all.

Which progress measures in your own work are threshold metrics in this sense? A build that is red or green; a project that is "on track" until it isn't; a skill one either "has" or "doesn't have." And if you had to find a continuous quantity for one of them that shows progress before the flip — which one would it be, and why aren't we measuring it?

Sources

• C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, "Understanding deep learning requires rethinking generalization," International Conference on Learning Representations (ICLR) 2017, arXiv:1611.03530 (quotes: "easily fit a random labeling of the training data," "qualitatively unaffected by explicit regularization"): https://arxiv.org/abs/1611.03530
• M. Belkin, D. Hsu, S. Ma, S. Mandal, "Reconciling modern machine-learning practice and the classical bias–variance trade-off," Proceedings of the National Academy of Sciences U.S.A. 116, 2019, pp. 15849–15854, DOI 10.1073/pnas.1903070116 (quote: "increasing model capacity beyond the point of interpolation results in improved performance"): https://www.pnas.org/doi/10.1073/pnas.1903070116
• M. Loog, T. Viering, A. Mey, J. H. Krijthe, D. M. J. Tax, "A brief prehistory of double descent," Proceedings of the National Academy of Sciences U.S.A. 117(20), 2020, pp. 10625–10626, DOI 10.1073/pnas.2001875117 (cites Vallet et al. 1989, Opper et al. 1990, Skurichina & Duin 1998, Raudys & Duin 1998, Duin 2000, Krijthe & Loog 2016; quotes: "With increasing N, the risk initially decreases, attains a minimum, and then increases until N equals n," "increasing PFLD's complexity simply by adding random features can improve performance when N = n"): https://www.pnas.org/doi/10.1073/pnas.2001875117
• P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, I. Sutskever, "Deep Double Descent: Where Bigger Models and More Data Hurt," ICLR 2020; also Journal of Statistical Mechanics: Theory and Experiment 2021, article 124003, DOI 10.1088/1742-5468/ac3a74 (EMC definition; "double descent occurs not just as a function of model size, but also as a function of the number of training epochs," "The peak in test error systematically occurs at the interpolation threshold," "Increasing the number of samples by a factor of 4.5 results in worse test performance"): https://arxiv.org/abs/1912.02292
• P. L. Bartlett, P. M. Long, G. Lugosi, A. Tsigler, "Benign overfitting in linear regression," Proceedings of the National Academy of Sciences U.S.A., 2020, DOI 10.1073/pnas.1907378117 (quotes: "The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology," "overparameterization is essential for benign overfitting in this setting"): https://arxiv.org/abs/1906.11300
• A. Power, Y. Burda, H. Edwards, I. Babuschkin, V. Misra, "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets," arXiv:2201.02177, January 2022 (quotes: "from random chance level to perfect generalization," "well past the point of overfitting," "smaller datasets require increasing amounts of optimization for generalization"): https://arxiv.org/abs/2201.02177
• N. Nanda, L. Chan, T. Lieberum, J. Smith, J. Steinhardt, "Progress measures for grokking via mechanistic interpretability," ICLR 2023, arXiv:2301.05217 (discrete Fourier transforms and trigonometric identities, ablations in Fourier space, three phases of memorization/circuit formation/cleanup; quote: "arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components"): https://arxiv.org/abs/2301.05217
• Z. Liu, E. J. Michaud, M. Tegmark, "Omnigrok: Grokking Beyond Algorithmic Data," ICLR 2023, arXiv:2210.01117 (LU mechanism, inducing grokking on image, language and molecule tasks, eliminating it for algorithmic data): https://arxiv.org/abs/2210.01117
• A. Curth, A. Jeffares, M. van der Schaar, "A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning," Advances in Neural Information Processing Systems (NeurIPS) 2023 (quotes: "the second descent appears exactly (and only) when and where the transition between these underlying axes occurs," "once we control for the effective parameters of each model in this way, we consistently observe that each example indeed folds back into shapes that are best characterized as standard convex (U-shaped) curves," "not all model parameters are created equal"): https://proceedings.neurips.cc/paper_files/paper/2023/file/aec5e2847c5ae90f939ab786774856cc-Paper-Conference.pdf

← All articles