The Context Horizon

Why Ten Billion Genuinely Usable Tokens May Be a Late-2040s Achievement — and Why That Still Would Not Be AGI

Keywords: artificial general intelligence; AGI; large language models; long context; context windows; attention; KV cache; HBM; memory bandwidth; scientific research; automated research; machine reasoning; AI scaling; Llama 4; hardware forecasting; research agents

Abstract

The current discussion of artificial intelligence routinely confuses three different quantities: the number of tokens a model is permitted to accept, the number of tokens it can retrieve or approximately represent, and the number of tokens it can use reliably as a coherent evidential workspace. Those quantities are not equivalent. A model may advertise a million-token or ten-million-token context while still exhibiting substantial degradation when the task requires associative retrieval, multi-hop reasoning, aggregation, contradiction tracking, or the preservation of weak but decisive evidence across a long input. This distinction matters particularly for research. A competent synthesis paper is not merely a long summary; it requires the maintenance of provenance, contradictory findings, methodological limitations, dependencies among claims, and a stable account of what follows from what. Novel scientific work raises the bar further because the system must not only retrieve and synthesise but also generate hypotheses, preserve constraints, test implications, detect failures, and distinguish a new result from a plausible recombination of familiar material.

This article develops a deliberately conservative framework for thinking about those requirements. It distinguishes nominal, effective, and high-fidelity context; examines the quadratic computational burden of conventional dense self-attention; analyses KV-cache, memory-bandwidth, and interconnect constraints; and uses current NVIDIA memory trajectories as one empirical anchor for a local four-accelerator forecast. The central engineering estimate is not a claim of inevitability: under aggressive continuation of recent high-bandwidth-memory growth, and assuming substantial architectural improvements, a four-accelerator local system with something approaching a genuinely usable ten-billion-token working set becomes plausible in the late 2040s, with approximately 2048 as a useful central scenario. The analysis also argues that this capability would not itself constitute artificial general intelligence. Context is a necessary resource for some forms of expert work, but intelligence additionally requires generalisation, robust long-horizon action, reliable world modelling, error correction, causal and counterfactual reasoning, and the acquisition of new skills under distribution shift. The evidence therefore supports neither the claim that AGI is imminent nor the stronger claim that a particular context size is sufficient for AGI. It supports a narrower conclusion: present systems are extraordinary tools, but advertised long-context capacity remains far ahead of demonstrated high-fidelity reasoning over long contexts, and the hardware and architectural requirements for research-grade machine cognition at corpus scale remain formidable.

Thesis

The most important scaling problem in advanced AI is no longer simply whether a model can be presented with more tokens. It is whether information remains usable: faithfully accessible, mutually constraining, provenance-preserving, and available to reasoning when its relevance becomes apparent only much later. Current long-context models have widened the input aperture dramatically, yet benchmark evidence repeatedly shows that nominal context length can exceed effective reasoning length by orders of magnitude on demanding tasks. For serious automated research, the problem is therefore simultaneously computational, architectural, epistemic, and physical.

A ten-billion-token active working corpus is a useful thought experiment because it exposes the difference between marketing scale and scientific scale. Under a conventional KV-cache assumption of roughly 50 KB per token — an illustrative figure rather than a universal constant — ten billion tokens imply roughly 500 TB of high-speed state. Four accelerators would require on the order of 125 TB each before model weights, temporary activations, redundancy, and other state are counted. Contemporary accelerators are measured in hundreds of gigabytes, not hundreds of terabytes. Meanwhile, exact dense self-attention retains a quadratic computational burden in sequence length even when clever kernels eliminate the need to materialise the full attention matrix in HBM. Hardware scaling alone is therefore unlikely to deliver brute-force trillion-token cognition. The more plausible route is a hybrid architecture in which raw evidence remains losslessly addressable while formal, symbolic, hierarchical, or state-space mechanisms maintain a compact but verifiable active structure.

That conclusion cuts both ways. It makes naive extrapolations from context-window marketing less credible, but it also means that a monolithic dense context is not the only imaginable path to machine-generated novelty. The relevant question is not whether every token literally attends to every other token on every operation. The relevant question is whether every material dependency can be recovered, checked, and allowed to invalidate downstream conclusions when required. Until systems can do that across large bodies of evidence — and do it while planning, experimenting, correcting themselves, and generalising beyond familiar task structures — claims that we are already near AGI remain premature.

---

1. The Great Numerical Seduction

Artificial intelligence has acquired a peculiar affection for large integers. Parameter counts gave way to training-token counts; training-token counts gave way to context windows; context windows are now giving way to claims of effectively limitless memory. The numbers are impressive because they are real in one narrow engineering sense. Meta’s Llama 4 Scout model card, for example, states a context length of 10 million tokens, while Google’s Gemini documentation describes models with context windows of one million tokens or more (Google, n.d.; Meta, 2025). The mistake begins only when an interface limit is treated as an intellectual capacity. The fact that a model can accept a sequence of a given length does not establish that it can reason with every relevant relation contained inside that sequence. That distinction is not pedantry; it is the difference between a storage specification and an epistemic capability.

The evidence for this distinction is now substantial. Liu et al. (2023) showed that models could perform much worse when relevant information appeared in the middle of a long context rather than near the beginning or end. LongBench found that models struggled as contexts became longer even on tasks explicitly designed to measure long-text understanding (Bai et al., 2024). RULER then demonstrated that near-perfect performance on simple needle-in-a-haystack retrieval could coexist with severe degradation once tasks required more needles, multi-hop tracing, aggregation, or other forms of structured use (Hsieh et al., 2024). NoLiMa made the problem harder by removing easy lexical overlap between the query and the relevant evidence; at 32K tokens, 11 of 13 evaluated models fell below half of their strong short-context baseline, and even a leading model suffered a large decline (Modarressi et al., 2025). The common pattern is that context admission is easier than context utilisation.

That empirical point should change the language used in discussions of AGI. A ten-million-token window may be a genuine engineering achievement and still be a poor approximation of ten million tokens of reliable working memory. The difference can be expressed with three terms. Nominal context is the maximum input length accepted by the system. Effective context is the subset of that input over which the model maintains useful task performance. High-fidelity context is the stronger condition required for rigorous research: material facts remain available even when weakly signalled, relevant relations can be discovered after the fact, contradictions persist until resolved, source provenance is not lost, and changing one assumption can propagate correctly through downstream conclusions. Contemporary systems increasingly possess the first property and sometimes impressive portions of the second. The third remains much more demanding.

The distinction matters because ordinary language tasks forgive omission in ways research does not. If a model forgets one descriptive detail in a 400-page novel, the summary may remain perfectly serviceable. If it misses one exclusion criterion in a meta-analysis, one boundary condition in a derivation, one statutory exception in a legal analysis, or one sign reversal in a proof, the entire result may become invalid. Research is not simply long text generation. It is constraint maintenance under uncertainty. The machine must preserve not only propositions but their status: observed, assumed, derived, disputed, superseded, replicated, contradicted, or unresolved. A context window that behaves like an approximate associative memory may be adequate for conversation yet inadequate for scholarship.

This is the first reason current systems should not be treated as near-AGI merely because context windows are expanding quickly. Intelligence is not measured by the amount of text that can be poured into a prompt. Chollet (2019) argues that skill on familiar tasks can be purchased through priors and data and therefore should not be confused with general intelligence; the deeper issue is efficiency of adaptation and generalisation to situations not already encoded by the system or its developers. Long context can improve access to prior information, but it does not by itself solve generalisation. Indeed, stuffing more of the world into the prompt can sometimes conceal the absence of generalisation by turning a novel problem into a retrieval problem.

The point is not that long context is unimportant. It is that long context is one necessary resource among several. The closer the target task comes to serious professional reasoning, the more damaging it becomes to confuse capacity with reliable use. A legal researcher, mathematician, economist, engineer, or scientist does not merely need access to more pages. They need a stable intellectual workspace in which claims can constrain other claims. The practical frontier should therefore be defined not by the largest number printed in a model card, but by the largest context over which demanding, adversarially designed tasks can be performed without material degradation.

2. What “True Context” Should Mean for Research

The phrase true context can easily become another slogan unless it is defined operationally. It should not mean that every token must literally interact with every other token through dense attention at every layer and every decoding step. That requirement confuses one implementation with the underlying capability. A future architecture could be exact in the relevant scientific sense while using hierarchical indexing, symbolic structures, persistent stores, state-space models, graphs, or other mechanisms. The decisive criterion is functional: can a material fact anywhere in the corpus reliably become active when required, and can it modify or invalidate conclusions that depend upon it?

For research, this implies at least six properties. First, the system needs addressability: the original evidence must remain recoverable rather than disappearing into an irreversible summary. Second, it needs provenance: claims must retain links to the sources, data, transformations, and assumptions from which they arose. Third, it needs dependency tracking: if premise A supports result B, and A is later weakened, B must be reconsidered. Fourth, it needs contradiction persistence: incompatible findings cannot simply be averaged into fluent prose. Fifth, it needs precision control: equations, code, measurements, and categorical constraints must not be degraded by lossy semantic paraphrase. Sixth, it needs revisability: information judged irrelevant at one stage must remain recoverable if later discoveries make it important.

These requirements explain why retrieval-augmented generation is useful but insufficient as a general solution. Retrieval reduces the amount of material presented to the model and can therefore improve efficiency dramatically. Yet retrieval has an unavoidable decision point: something decides what to retrieve. If relevance is obvious from lexical or semantic similarity, this can work very well. If relevance emerges only after a chain of reasoning, the retrieval mechanism may never surface the decisive source. NoLiMa is especially revealing because it reduces lexical overlap and requires latent association; performance falls as the model loses the easy cue that tells it where to look (Modarressi et al., 2025). Scientific reasoning routinely contains precisely this structure. The important paper may not use the terminology of the new theory. The important anomaly may look irrelevant until a model has already formed a hypothesis.

Compression faces the same epistemic difficulty. Summarisation is a mapping from a richer representation into a smaller one. Unless that mapping is lossless, some distinctions disappear. The question is not whether compression is bad — human researchers compress constantly — but whether the original information remains accessible and whether the system knows when its summary is insufficient. A human scholar writes notes but can reopen the paper. A competent machine system should do the same, only more systematically. The wrong architecture is not one that compresses; it is one that compresses and then behaves as though nothing was lost.

This is also why the phrase “all-to-all cross-attention” should be used with care. Dense self-attention is one way of permitting every token to influence every other token. It is computationally expensive, and Keles et al. (2022) establish conditional lower bounds showing that standard self-attention cannot generally be made truly subquadratic without changing the computational problem or violating widely used complexity assumptions. Yet scientific fidelity does not necessarily require recomputing every pairwise relation continuously. What it requires is that potentially important relations remain discoverable. A dependency graph can make one distant connection explicit without evaluating quadrillions of irrelevant token pairs. A theorem prover can invalidate downstream steps when an axiom changes without “remembering” the proof as a sequence of natural-language tokens. A database can preserve exact values without representing them in a neural hidden state.

The future research system is therefore more likely to resemble a cognitive operating system than an infinitely stretched chat window. The language model may propose hypotheses and interpret ambiguous text. A structured memory may store entities, claims, equations, and provenance. Formal tools may verify algebra, code, or logical implications. Search systems may locate candidate evidence. A scheduler may decide which uncertainties deserve additional computation. Crucially, each component would preserve routes back to source evidence. This arrangement is not an evasion of context; it is a better definition of context. Information is part of the system’s effective context if it can be summoned reliably, precisely, and at the right time without having been silently destroyed.

That architecture also fits the empirical direction of research. PagedAttention improves KV-cache management by borrowing ideas from virtual memory rather than pretending physical memory is infinite (Kwon et al., 2023). KVQuant compresses the cache and demonstrates that very long inference can be made more memory efficient, but it does so by explicit quantisation with measurable quality trade-offs rather than by magic (Hooper et al., 2025). Mamba explores selective state-space models with linear sequence scaling and recurrent inference, showing that attention is not the only plausible sequence architecture (Gu & Dao, 2024). None of these results solves research-grade long-context reasoning. Together, however, they show that the engineering path is already moving away from the simple story that more tokens merely require a wider Transformer window.

The correct target is therefore fidelity under scale. A system should be credited with a large context only to the extent that information in that context remains behaviourally available under tasks designed to expose failure. The more consequential the domain, the stricter the test should be. A model that can retrieve a memorable phrase from ten million tokens has demonstrated retrieval. A model that can reconcile hundreds of weakly linked sources, discover a hidden contradiction, update a model, re-run downstream implications, and explain which sources changed the conclusion has demonstrated something much closer to research cognition.

3. The Quadratic Wall Is Real — but It Is Not the Whole Wall

Conventional dense self-attention has an uncomfortable property: the number of pairwise interactions grows with the square of sequence length. If sequence length rises by a factor of 10, the pairwise attention term rises by roughly 100. If sequence length rises from 100,000 tokens to 10 billion tokens, sequence length increases by a factor of 100,000 and the pairwise term increases by a factor of 10 billion. That arithmetic is not a forecast of total system cost because kernels, model dimensions, sparsity, batching, and architecture matter, but it captures why brute-force extrapolation becomes absurd so quickly.

FlashAttention demonstrates an important qualification. Dao et al. (2022) showed that exact attention can be computed without storing the full attention matrix in high-bandwidth memory, using tiling and IO-aware algorithms to reduce costly memory traffic. This is a major engineering improvement because memory movement is often as important as raw arithmetic. It does not, however, make exact dense attention linear in sequence length. It changes memory behaviour and constants; it does not repeal the pairwise nature of the calculation. Keles et al. (2022) go further by establishing conditional lower bounds for the computation of self-attention. The implication is not that long contexts are impossible. It is that an architecture insisting upon exact conventional all-pairs attention at enormous scale eventually encounters a mathematical barrier rather than merely an implementation inconvenience.

The phrase “quadratic wall” can nevertheless mislead if it is treated as proof that scientific reasoning must wait for hardware capable of brute-force dense attention over trillions of tokens. The human brain plainly does not compare every memory representation with every other representation at every moment. Human cognition uses selective attention, schemas, external notes, symbolic notation, libraries, citations, and repeated re-reading. A machine can use analogous computational separation without becoming intellectually dishonest. The challenge is to make selection auditable and reversible. Sparse attention becomes dangerous when omitted evidence becomes permanently invisible; it becomes powerful when the system can detect uncertainty, revisit neglected regions, and validate that its selection did not change the result.

This observation reframes the engineering problem. Dense attention gives one strong guarantee: every token is mathematically available to every query within the permitted mask. Alternative architectures need to reproduce the useful part of that guarantee without paying the full cost. They may do so through structured state, multi-scale memory, learned routing, graphs, caches, symbolic objects, or repeated passes. But the burden of proof shifts to them. They must show not merely that average benchmark performance is good, but that critical dependencies are preserved under adversarial conditions.

For research, exactness is task dependent. A literature review does not require floating-point identity between every textual representation. It does require exactness for citations, quantities, and logical dependencies. A mathematical derivation requires symbolic fidelity to variables, signs, assumptions, domains, and transformations. A program analysis requires exact code and control flow. A clinical evidence synthesis requires exact eligibility criteria, endpoints, effect sizes, and study distinctions. The architecture can therefore economise by applying different representations to different information types. Natural-language prose can be represented semantically; equations can remain symbolic; tables can remain structured; source text can remain immutable; derived claims can carry explicit provenance.

This is why I do not accept the proposition that a single giant monolithic context window is the only true route to machine novelty. It is one route, and perhaps the easiest to conceptualise, but it may be the least attractive physically. If ten billion or one trillion tokens must all remain resident in a dense attention mechanism, the hardware burden becomes grotesque. If the same corpus can instead be made losslessly addressable while a verified working structure is kept active, then the effective scientific context may be enormous without requiring every token pair to be recomputed continuously. The distinction is the same one computer science has used for decades: address space, working set, cache, and persistent storage are not the same thing.

The central requirement is therefore not literal simultaneity but logical simultaneity. A fact should be able to constrain the answer even if it is not physically resident in the fastest memory at the exact instant a token is generated. If a later step depends on that fact, the system must know enough to retrieve and check it. If two distant claims conflict, the conflict must become visible. If a foundational assumption changes, dependent conclusions must be marked stale. The path beyond the quadratic wall is likely to be found not by pretending the wall does not exist, but by changing what has to cross it.

4. The KV-Cache Problem: Memory Is Not a Footnote

During autoregressive generation, a Transformer ordinarily stores keys and values for previous tokens so that they do not need to be recomputed from scratch at each decoding step. This KV cache grows linearly with sequence length. Hooper et al. (2025) give the general relationship: cache size depends on layer count, number of KV heads, head dimension, numeric precision, batch size, and sequence length. That immediately exposes a weakness in popular back-of-the-envelope claims. There is no universal “bytes per token” constant. A figure such as 50 KB per token may be a useful scenario for a compressed or efficient model, but it is not a law of nature. Different architectures can be far larger or smaller.

The 50 KB figure is nevertheless illuminating because even optimistic arithmetic becomes enormous at ten billion tokens. At 50 KB per token, ten billion tokens require about 500 TB of KV state. Divided across four accelerators, that is approximately 125 TB per accelerator, before accounting for model weights, runtime buffers, redundant state, routing metadata, temporary activations, or safety margins. Even reducing the figure by an order of magnitude would still imply tens of terabytes per accelerator. The order-of-magnitude conclusion is therefore robust: a brute-force, high-fidelity ten-billion-token active cache is far outside the memory capacity of present local GPU systems.

Current hardware provides a useful anchor. NVIDIA’s H100 has 80 GB of HBM and up to 3 TB/s of memory bandwidth in the SXM configuration (NVIDIA, n.d.-a). H200 raises capacity to 141 GB and bandwidth to 4.8 TB/s (NVIDIA, n.d.-b). A DGX B200 system provides 1.44 TB across eight Blackwell GPUs and 64 TB/s of aggregate HBM bandwidth, equivalent to about 180 GB per GPU (NVIDIA, n.d.-c). Rubin moves to as much as 288 GB of HBM4 and up to 22 TB/s of bandwidth per GPU, while an eight-GPU DGX Rubin NVL8 provides 2.3 TB total GPU memory (NVIDIA, 2026). These are spectacular advances, but the comparison with 125 TB per accelerator is sobering.

If one takes the progression from approximately 80 GB in H100 to 288 GB in Rubin as a rough four-year trajectory, memory capacity has grown by about 3.6 times. That corresponds to an annualised growth rate in the high thirties percentage range. Extrapolating such a rate indefinitely is dangerous because semiconductor, packaging, yield, power, cost, and memory-stack constraints do not promise exponential growth forever. Yet as a scenario calculation it is useful. Growing from 288 GB to approximately 125 TB requires a further increase of more than 400 times. Sustaining a rate near 38% per year would take roughly two decades, pushing the raw memory-capacity crossover into the mid-2040s.

That calculation is the origin of the late-2040s estimate, not a mystical belief in the year 2048. Memory capacity alone might cross the relevant scale around 2045 under an aggressively favourable extrapolation. A usable system would arrive later because capacity is only one constraint. Bandwidth must allow the system to read and update the state. Interconnect must allow four cards to operate coherently. Compute throughput must support the attention or alternative memory mechanism. Power and cooling must remain feasible in an individual workstation rather than a hyperscale facility. Software must avoid spending most of its time moving bytes rather than reasoning. Long-context quality must remain stable rather than collapsing as the window expands.

Bandwidth is especially easy to underestimate. Rubin’s 22 TB/s per GPU is extraordinary by present standards. Yet a hypothetical 125 TB of active state per GPU cannot simply sit there. If an operation required streaming that entire state once, the lower-bound transfer time would be measured in seconds even at tens of terabytes per second, before any computation occurred. Real inference performs many operations across layers and tokens, so a naive design could spend most of its life waiting for memory movement. The future therefore requires not merely more memory but more intelligent locality: the right state must be near the computation that needs it.

Interconnect compounds the problem. A local four-GPU system is attractive because it bounds the prediction: no rack-scale clusters, no remote memory fabric, and no hidden hyperscaler. But once the working state is sharded, a reasoning operation that crosses shards incurs communication. Contemporary NVLink and NVSwitch are extraordinarily fast relative to PCIe, yet they remain orders of magnitude slower than on-chip arithmetic and cannot make physically separate HBM stacks behave as though distance were free. As contexts grow, architecture increasingly becomes a data-placement problem. The fastest multiply is useless if the operands spend their time travelling.

This makes the “four GPUs by 2048” forecast conditional. It assumes that memory density continues improving strongly, that bandwidth grows at least comparably, that model architectures use KV state far more economically than conventional dense Transformers, and that interconnect plus software can keep a substantial fraction of theoretical bandwidth useful. It also assumes that “local computer” in the late 2040s can tolerate power and cooling requirements that would be considered exotic today. The estimate is therefore best read as a technological horizon: the point at which the numbers cease to be obviously absurd. It is not a guarantee that the exact configuration will exist on a retail shelf.

5. Why a Ten-Million-Token Model Is Not Yet a Ten-Million-Token Researcher

Meta’s Llama 4 Scout illustrates the distinction particularly well because its official model card gives a 10-million-token context length (Meta, 2025). That number is not fictitious. It describes a supported context length. Yet the model card’s long-context evaluation does not establish that Scout performs research-grade all-corpus synthesis across ten million tokens with invariant quality. This is a general problem in model evaluation: maximum context length is easier to state than effective context under every meaningful task.

Long-context benchmarks demonstrate why the gap matters. RULER deliberately extends beyond single-needle retrieval by introducing multi-hop tracing and aggregation. Its authors found that models with large advertised windows often failed to maintain performance as length and task complexity increased (Hsieh et al., 2024). NoLiMa removes literal matching cues and produces large performance declines at context lengths far below the nominal maxima of the tested models (Modarressi et al., 2025). InfinityBench uses realistic and synthetic tasks with average context length above 100K and likewise concludes that current long-context models require substantial advances to process such inputs effectively (Zhang et al., 2024). These are not attacks on long-context engineering. They are measurements of the difference between having a window and using it.

The gap widens further when the target is a synthesis paper. A competent synthesis is not equivalent to answering a set of isolated questions about a corpus. The writer must construct a representation of the literature itself: schools of thought, shared assumptions, methodological differences, temporal developments, replication patterns, conflicting estimates, definitional drift, and gaps. Sources are not independent. Five papers may repeat one dataset. Ten reviews may ultimately depend on two primary experiments. A highly cited claim may be a misquotation propagated through secondary literature. The evidence graph is therefore more important than the document count.

A system can write a fluent synthesis while failing this test. Language models are exceptionally good at producing local coherence. Paragraphs can sound balanced and scholarly even when the global evidential structure is incomplete. The danger is greatest when compression has erased provenance. Once an intermediate summary says “the literature generally supports X,” downstream generation may treat that sentence as a fact even if the original corpus contained three incompatible definitions of X. The problem is not hallucination in the narrow sense of inventing a citation. It is epistemic flattening: turning a structured and contested body of evidence into a smooth narrative.

A research-grade context system must resist that flattening. It should maintain claim-level records and know which sources support each claim. It should distinguish direct evidence from inference and inference from speculation. It should retain negative and null results. It should identify whether apparent replication is genuinely independent. It should mark uncertainty rather than smoothing it away. And it should be able to trace any substantive sentence in the final synthesis back to the evidence that warrants it. Those functions impose memory and reasoning requirements that a simple context-length number does not measure.

This is also why a large corpus may require substantially more processing tokens than its raw size. A corpus of ten million input tokens might require hundreds of millions or billions of internal processing tokens if the system repeatedly revisits sources, tests alternative classifications, compares claims, checks citations, runs calculations, and red-teams its own synthesis. Research is iterative. A human scholar may reread the same paper five times for different purposes. A machine researcher should be expected to do something similar. The important resource is therefore not only context length but reasoning budget over a persistent corpus.

The implication for timelines is straightforward. We may obtain nominal ten-million-token local contexts long before we obtain ten million tokens of robust, adversarially tested, high-fidelity research context. Indeed, in some architectures we already possess the former. The question for AGI and automated science is the latter. When a benchmark tells us that performance falls sharply at 32K once lexical shortcuts are removed, it is difficult to argue that a ten-million-token model is already behaving like a human expert with ten million tokens of reliable active evidence.

6. Minimum Context for Human-Level Performance: The Only Defensible Way to Estimate It

There is no scientifically established token count for “human-level intelligence.” Human cognition does not operate as a Transformer context window, and human experts routinely use external memory: books, notes, search systems, whiteboards, source code, laboratory notebooks, citation managers, and colleagues. Any table claiming that humans “have” a particular token context would therefore be pseudo-precision. The useful question is different: what minimum machine working set would make a task feasible without forcing lossy omission of information that a competent human can keep externally accessible? The estimates below should be read in that operational sense.

A second qualification is equally important. Bigger is not always better. An expert conversation may require only a few thousand relevant tokens if the system has stable long-term knowledge. A legal matter may require millions because the record, authorities, pleadings, and evidence all matter. A scientific synthesis may require tens or hundreds of millions if the field is large. The context requirement is therefore determined by the dependency structure of the task, not by intelligence in the abstract. Human-level performance in one domain cannot be converted mechanically into a universal AGI number.

Operational context estimates

Selected areaRepresentative taskPlausible minimum high-fidelity working setWhy this scale may be requiredEveryday expert conversationMaintain a complex discussion with commitments, documents, and prior decisions50K–250K tokensEnough for a long interaction plus selected supporting material; stable long-term memory could reduce active need.Advanced software engineeringModify a mature codebase with architecture, tests, issues, and documentation1M–20M tokensNon-local dependencies, configuration, interfaces, historical decisions, tests, and generated artifacts can span a repository.Complex litigation or regulatory matterAnalyse record, authorities, pleadings, expert reports, and chronology5M–100M tokensCorrect conclusions may depend on a small exception buried anywhere in a large evidential record.Single-field systematic synthesisProduce a defensible review with provenance and contradiction tracking10M–200M tokensHundreds to thousands of papers plus extracted evidence, methods, tables, and citation graphs.Cross-disciplinary research programmeIntegrate several fields with different terminology and methods100M–2B tokensMultiple literatures, datasets, methodological traditions, and dependency chains must remain mutually accessible.Frontier theoretical researchDevelop and verify a new theory against a mature field500M–10B tokensRequires literature, proofs, failed approaches, data, models, simulations, counterexamples, and repeated verification.Broad machine scientific workspaceMaintain many fields, experiments, and long-lived hypotheses concurrently10B–1T+ tokensThis is closer to an institutional memory than a single paper and would require structured persistent memory, not necessarily dense attention.

These ranges are intentionally broad because there is no empirical law linking task quality to token count. Their purpose is to show orders of magnitude and to expose a hidden assumption in many AGI forecasts: professional human performance depends on access to large external information environments. A human lawyer does not memorise the entire disclosure bundle. A scientist does not hold every paper in biological working memory. They create systems of access. A machine that is denied equivalent external structure and judged only by a single prompt window is being asked to solve the wrong engineering problem.

The lowest range, expert conversation, illustrates why raw context is not synonymous with intelligence. A highly competent system might perform excellently with 100K active tokens if it possesses reliable long-term semantic memory and knows how to retrieve exact prior commitments. Conversely, a system with ten million tokens can still fail if it cannot distinguish important commitments from irrelevant chatter. The requirement is not maximal accumulation but selective retention without irreversible loss.

Software engineering raises the scale because codebases are explicit dependency networks. A change in one interface can break distant modules, tests, deployment scripts, or documentation. The model needs more than textual recall: it needs symbol resolution, call graphs, build state, version history, test results, and issue context. This is exactly the kind of task where structured representations can beat a monolithic token stream. A million-token window may hold a useful portion of a repository, but a human-level engineering agent needs the ability to move through a much larger space while preserving exact dependencies.

Legal analysis is even more unforgiving because seemingly peripheral evidence can become dispositive. A system may need to cross-reference a witness statement with an exhibit, a contractual clause, a procedural rule, and a later appellate authority. The relevant fact may not be lexically similar to the legal question. High-fidelity access to 5M–100M tokens is therefore plausible for large matters, but again the key is not simultaneous attention. It is the capacity to treat the case file as one coherent evidential object.

Systematic synthesis expands the requirement because a field is not a document set but a network of claims. A review of 500 full papers can easily exceed several million raw tokens before notes, extracted tables, supplementary materials, datasets, and citation networks are counted. A serious synthesis also reprocesses material repeatedly. A 10M–200M-token high-fidelity workspace is therefore not extravagant if the objective is to beat a good human scholar rather than to produce an attractive literature-summary draft. The machine must be able to explain not only what the literature says but why it believes the literature says it.

Frontier scientific work is where context becomes a poor single metric. A new theory may require access to an enormous corpus but only a small subset of facts at any given inference step. It may also require simulation, formal proof, experimental design, and tools that cannot be represented faithfully as prose tokens. The plausible 500M–10B range is therefore better interpreted as an addressable research environment than as a dense prompt. If a system can maintain lossless source access, exact formal state, and dependency-aware retrieval, it may perform human-level research with a much smaller neural working set. If it cannot, even ten billion nominal tokens may be insufficient.

These estimates also show why a trillion-token context should not be treated as an AGI threshold. A trillion-token scientific memory could support extraordinary breadth while the underlying system remained brittle outside trained workflows. Conversely, a genuinely general learner might adapt efficiently with far less explicit context by acquiring abstractions and using tools. Context is a resource. General intelligence is the capacity to deploy resources across novel problems.

7. A Good Synthesis Paper Is an Earlier and Cleaner Test Than AGI

A useful forecast target is not “when does AGI arrive?” but “when can an AI system reliably produce a better synthesis paper than a strong human researcher?” The latter can be specified and tested. Give humans and machines the same bounded corpus, the same time or compute budget, and the same requirements for evidence extraction, contradiction analysis, methodological critique, provenance, and written synthesis. Then blind the outputs and evaluate factual accuracy, coverage, source use, structure, calibration, originality of synthesis, and the rate of material omission. This would be far more informative than asking whether a chatbot sounds intelligent.

The task is demanding but not fundamentally novel in the scientific sense. A synthesis paper reorganises and evaluates existing knowledge. It may produce an original framework, but it does not necessarily require a new law of nature, theorem, or empirical discovery. That makes it a lower bar than paradigm-changing research. If current systems cannot reliably outperform strong humans on bounded, source-complete synthesis, claims of autonomous scientific genius are premature.

There are reasons to expect AI eventually to dominate this task. Machines can read faster, maintain more exact provenance, compare more sources, and rerun analyses without fatigue. They can search citation networks exhaustively. They can preserve competing interpretations instead of unconsciously favouring the one encountered most recently. They can maintain detailed evidence matrices far larger than a human can comfortably manipulate. Once long-context fidelity and research tooling mature, synthesis is exactly the sort of intellectual activity where machine advantages should become overwhelming.

But those advantages appear only if the system stops losing the corpus. A research agent that reads 2,000 papers sequentially and compresses each into a few sentences may be fast but not superior. It has merely automated note-taking. The breakthrough comes when the entire evidence structure remains queryable and revisable. If a late source exposes a flaw in an early classification, the system should automatically identify every affected claim. If two meta-analyses share primary studies, it should recognise the dependence. If a celebrated result disappears under a different inclusion rule, it should test and report the sensitivity. That is a machine synthesis worth preferring to a human one.

LongBench v2 points toward the challenge by including contexts ranging from 8,000 to 2 million words across tasks such as multi-document QA, repository understanding, and structured-data reasoning (Bai et al., 2025). The benchmark is intentionally difficult, and results show that performance depends not simply on window size but on reasoning. Yet even LongBench v2 is not equivalent to writing a defensible 20,000-word systematic synthesis with an audit trail. The future benchmark must evaluate the whole research process rather than only answers to questions drawn from long inputs.

A realistic prediction is therefore two-stage. Machine-generated synthesis papers that look better than ordinary human work are already common because models excel at prose, organisation, and local comparison. Machine-generated syntheses that are epistemically better — more complete, more accurate, more traceable, more sensitive to contradiction, and more robust under adversarial checking — require much stronger memory architecture and validation. The second threshold is the one that matters. It is also the threshold likely to arrive before genuine machine-generated scientific novelty, because synthesis does not require the additional leap from organising known possibilities to generating and validating genuinely new ones.

If the late-2040s scenario for ten-billion-token local high-fidelity research context is approximately right, superior machine synthesis could arrive earlier than that because most bounded reviews need far less than ten billion active tokens. A plausible window is the late 2030s to 2040s for systems that reliably outperform strong humans on very large synthesis tasks under strict evidential scoring, although architecture could move the date substantially forward. The claim is not that ten billion tokens are necessary for every review. It is that the underlying technologies needed for high-fidelity, dependency-aware corpus reasoning probably mature progressively on the way to that scale.

The distinction also provides a better way to discuss “human level.” A machine need not imitate human cognitive limitations to beat a human researcher. It can use databases, exact arithmetic, theorem provers, and large external memory. Human-level should mean comparable or superior performance under a fair task definition, not biological similarity. The danger lies only in declaring victory on the basis of prose quality while ignoring the invisible evidential work that makes scholarship reliable.

8. Why We Are Not Yet Close to AGI

The phrase artificial general intelligence has no universally accepted technical definition, which makes categorical claims difficult. Still, serious definitions share a family resemblance: breadth across domains, adaptation to new tasks, transfer of knowledge, robust reasoning under distribution shift, autonomous goal-directed behaviour, and the ability to acquire skills rather than merely replay them. Chollet’s (2019) formulation is particularly useful because it separates skill from intelligence. A system can display enormous skill if its training distribution, toolchain, and prompt supply the relevant structure. General intelligence is revealed more clearly when the problem is genuinely new.

Current frontier systems are impressive precisely because they possess broad learned priors. They can code, translate, summarise, solve many mathematical problems, interpret images, and produce sophisticated prose. But broad task coverage is not the same as robust generality. Long-context failures expose one limitation: the model’s access to supplied evidence degrades under scale and task complexity. Agentic failures expose another: long-horizon execution remains brittle. Research-agent benchmarks expose a third: systems can automate pieces of science while still struggling with open-ended strategy, resource management, algorithmic novelty, and sustained experimental judgement.

InnovatorBench is instructive because it evaluates agents on realistic end-to-end LLM research tasks requiring runnable artifacts rather than only written answers. Wu et al. (2025) report that frontier-model agents show promise but struggle with fragile algorithmic tasks and long-horizon decision making, including resource management and template-like reasoning. This is a much more relevant AGI signal than a chatbot benchmark. General intelligence must survive the accumulation of small decisions. A system that is brilliant for five minutes but unreliable over an eleven-hour research process is not a general autonomous researcher.

Evidence from ordinary expert work also counsels against equating striking demonstrations with dependable professional replacement. In a randomised controlled trial involving experienced open-source developers working on mature repositories they already knew well, Becker et al. (2025) found that access to early-2025 AI tools increased completion time by 19% even though participants expected the tools to make them faster. This result does not imply that AI coding tools are generally harmful, and it is tied to a specific population, tool generation, and task design. Its importance is methodological: measured performance in real expert workflows can diverge sharply from both subjective impressions and benchmark narratives. An AGI claim must survive that kind of field evaluation across many domains, not merely demonstrations in which the model chooses its strongest terrain.

Human professional performance also includes error recovery. Experts make mistakes constantly, but they possess procedures for detecting them: tests, peer review, replication, unit checks, adversarial argument, formal proof, and institutional memory. Current LLMs often generate answers in a way that makes internal error states hard to inspect. Scaling context does not automatically create an error-correcting architecture. A billion-token chain of reasoning with a one-percent chance of unnoticed corruption at each critical stage is worse than a thousand-token chain with external verification. AGI therefore requires not simply more reasoning but controlled reasoning.

Grounding is another gap. Language models learn statistical structure from representations of the world. Multimodal training and tool use improve this, but general intelligence in open environments requires stable causal models, persistent identity, time, action consequences, and the distinction between observation and inference. A research system that proposes an experiment must understand not merely the textual pattern of an experiment but what can actually be measured, what confounds the result, and how an intervention changes the system. These capabilities may be learned, engineered, or hybrid, but they are not supplied by context size alone.

Planning presents a related difficulty. Long-horizon goals require decomposition, resource allocation, revision after failure, and resistance to locally attractive but globally destructive actions. A model that optimises each conversational turn can still fail a project. Real researchers abandon hypotheses, redesign experiments, wait for evidence, and distinguish a promising anomaly from noise. This involves value of information and metareasoning: deciding what to think about next. A large context provides more material for those decisions but does not solve the decision problem.

Generalisation under distribution shift remains the deepest issue. If a model is trained on nearly every familiar task format and given tools that encapsulate known procedures, high benchmark performance may say little about how it behaves when both the system and its developers confront something genuinely new. Chollet’s critique remains relevant: skill can be purchased with priors. The stronger test is the efficiency with which a system turns limited new evidence into competent behaviour. A model with enormous context but weak abstraction may drown in information; a more general learner may infer the governing principle from a handful of examples.

The strongest argument against imminent-AGI claims is therefore cumulative rather than singular. Present systems have outstanding local competence but incomplete long-context fidelity; powerful reasoning but unreliable long-horizon execution; broad knowledge but uncertain adaptation to truly novel structures; fluent explanation but weak guarantees of provenance; and growing tool use without robust autonomous verification. None of these defects alone proves that AGI is decades away. Together, they show that there is no empirical basis for treating current context-window growth as a simple countdown to general intelligence.

9. The Error-Propagation Problem Is More Subtle Than “Dropped Negative Signs”

It is tempting to describe long-horizon mathematical reasoning as a problem in which one dropped minus sign at token one billion poisons a derivation at token nine billion. The intuition is sound — small errors can have distant consequences — but the mechanism should be stated more carefully. Neural inference is not a single floating-point calculation whose rounding error mechanically accumulates through billions of tokens. Errors arise from representation, attention, generation, tool use, approximation, retrieval, and decision making. The important issue is logical error propagation, not merely numerical round-off.

A robust research architecture should therefore treat important intermediate results as objects with validation status. An equation can be checked symbolically. A numerical result can carry units, tolerances, and provenance. A theorem can be passed to a proof assistant. A dataset transformation can be logged and reproduced. A claim extracted from a paper can retain the exact quoted or structured source span. Once knowledge is represented this way, a later inconsistency can trigger recomputation of dependent results. This is more reliable than asking a neural model to keep a perfect latent memory of every prior reasoning step.

This is another reason trillion-token monolithic context is probably the wrong target. The larger the reasoning horizon, the more the system needs checkpoints of verified state. Human mathematics works this way. A published theorem can be reused without mentally replaying the entire proof, because the community treats it as a verified object — while retaining the proof for audit. Software engineering works this way through tests and version control. Science works this way through data, protocols, and replication. A machine scientist should inherit the same architecture at much higher speed.

The distinction also reduces hardware pressure. If verified intermediate results can be represented compactly, the model does not need every low-level token active at every step. Yet this compression is epistemically safe only because it is reversible. The theorem object points to the proof. The extracted coefficient points to the table. The summary claim points to the included studies. A later challenge can reopen the underlying evidence. This is fundamentally different from lossy summarisation that discards the path back to the source.

The real engineering challenge is therefore not perfect uninterrupted thought. It is perfectly recoverable thought. A future AGI does not need to maintain a trillion-token stream without any local mistake. It needs to detect, localise, and repair mistakes before they contaminate trusted conclusions. That requires memory architecture, verification systems, and metacognitive control. Hardware supplies capacity; architecture supplies integrity.

10. Forecasting the Ten-Billion-Token Local Machine

Any hardware forecast beyond a decade should be treated as a scenario, not a prophecy. Technology growth is lumpy. Packaging changes, memory standards, optical interconnects, new numerical formats, chiplets, cooling, and model architecture can shift effective capacity faster than a simple exponential. Conversely, physical limits, economics, supply chains, or declining returns can slow progress abruptly. The value of a forecast lies in exposing assumptions, not in pretending the date is certain.

Start with the local constraint: four high-end accelerators. H100 provides 80 GB per GPU; H200 provides 141 GB; B200-class DGX systems provide roughly 180 GB per GPU; Rubin provides up to 288 GB (NVIDIA, n.d.-a, n.d.-b, n.d.-c, 2026). A four-Rubin system would therefore provide about 1.15 TB of aggregate HBM. Under the illustrative 50 KB-per-token assumption, a 10-billion-token KV state would require about 500 TB. The gap is roughly 400-fold in memory capacity even before other model state is counted.

If high-end GPU memory capacity continued growing at an annualised rate around the high thirties, closing a 400-fold gap would take roughly nineteen to twenty years. Starting from the 2026 Rubin era yields a raw crossover in the mid-2040s. That is already an optimistic assumption because compound growth rarely continues smoothly at the frontier. But the scenario is not absurd: three-dimensional memory stacking, packaging, and specialised accelerators could produce dramatic gains over two decades.

The second constraint is bandwidth. H100’s 3 TB/s rose to H200’s 4.8 TB/s and Rubin’s 22 TB/s. Bandwidth has therefore grown faster than capacity across these generations, which is encouraging. Yet a 500-TB active state changes the scale of the problem. If future architectures must stream large fractions of that state frequently, bandwidth becomes the dominant limit. The only plausible design is one in which most operations touch a carefully selected subset while the remainder stays rapidly addressable.

The third constraint is compute. Exact dense attention over ten billion tokens is not simply a memory problem. If a current high-fidelity task scale is taken as 100K tokens, ten billion is 100,000 times longer. A quadratic term scales by ten billion. No plausible continuation of conventional GPU throughput alone closes that gap within a few product generations. Architectural change is therefore not optional. The late-2040s forecast implicitly assumes sparse, hierarchical, state-space, symbolic, or otherwise structured mechanisms that avoid applying dense attention indiscriminately.

The fourth constraint is software efficiency. Hardware specifications describe peaks. Real applications achieve lower utilisation because data movement, synchronisation, branching, kernel launches, and irregular workloads introduce overhead. Research agents are especially irregular: they search, parse documents, run code, invoke tools, wait for simulations, and branch based on results. A machine capable of theoretical peak throughput may still be slow at science. Mature scheduling and memory systems will therefore matter as much as raw FLOPS.

The fifth constraint is cost and power. The phrase “people can run it” implies something beyond a national laboratory. A four-card workstation can be expensive and power hungry, but it remains individually deployable. Today’s H200-class or future Rubin-class systems already challenge ordinary electrical and thermal assumptions. By the late 2040s, a high-end research workstation may use advanced liquid cooling and power budgets that seem extreme by desktop standards. The prediction therefore concerns high-end personal or laboratory ownership, not mass-market laptops.

Putting the constraints together yields a reasonable scenario ladder:-

2032–2035: genuinely useful multimillion-token local contexts become routine for demanding professional work, with much better effective-context fidelity than present systems.

-

2035–2040: tens to hundreds of millions of high-fidelity addressable tokens become feasible on expensive local systems, especially with structured memory and aggressive cache compression.

-

2040–2045: billion-token-class research workspaces become technically credible, though probably not through naive dense attention.

-

2045–2050: a ten-billion-token local research environment becomes plausible on a four-accelerator system if memory, bandwidth, and architecture all continue improving strongly.

-

Central scenario: approximately 2048. This is the point at which the hardware arithmetic and the architectural requirements could plausibly intersect, not the date on which AGI is scheduled to arrive.

The final sentence is the important one. A ten-billion-token system in 2048 could still be narrow. It might be the best literature synthesiser ever built and remain poor at robotics, social inference, novel physical environments, or autonomous goal formation. Hardware capacity sets a ceiling on what can be represented; it does not determine the algorithm that uses it.

11. What About One Trillion Tokens and Machine-Generated Novelty?

If ten billion genuinely usable local tokens are a late-2040s problem, it is tempting to multiply the timeline and place one trillion tokens in 2070–2090. That may be a reasonable brute-force scenario, but it should not be treated as a direct extrapolation. One trillion is 100 times ten billion. If memory capacity alone continued compounding at 38% annually, a further 100-fold increase would take roughly fourteen years. That would move a 2048-style capacity threshold into the early 2060s. Yet compute, bandwidth, interconnect, economics, and architecture could make the practical date much later. Conversely, better representations could make literal trillion-token residency unnecessary.

The scientific question is not whether novelty requires a trillion tokens. Human scientists create novel work without anything resembling a trillion-token biological context. They exploit institutions and external memory. A laboratory, field, or discipline collectively stores vastly more information than any individual actively remembers. Novelty emerges from selective access, abstraction, and experiments that create new evidence. A machine system can do the same. The relevant future threshold may therefore be a trillion-token addressable scientific environment combined with a much smaller active verified state.

This distinction undermines both extreme optimism and extreme pessimism. The optimist cannot simply point to a 10M model card and infer that trillion-token scientific cognition is a few scaling steps away. The pessimist cannot simply square one trillion and conclude that machine novelty is physically impossible. Both arguments assume that the future system must be today’s Transformer enlarged without conceptual change. History suggests otherwise. When scaling becomes expensive, computer architecture introduces hierarchy.

The likely machine scientist will have layers of memory. Immutable source storage will preserve raw papers, data, code, images, and logs. Indexes will support fast candidate retrieval. Structured knowledge will represent entities, equations, variables, methods, and claims. A working memory will hold the currently active problem. Formal systems will verify exact relationships. Learned models will interpret ambiguous material and generate candidate hypotheses. A global controller will allocate attention and computation. Such a system may behave as though it possesses a trillion-token intellectual workspace without ever placing a trillion token vectors in one attention matrix.

Machine-generated novelty then becomes a question of search and evaluation. The system must generate hypotheses that are not trivial restatements of its training data, identify discriminating tests, allocate resources, interpret unexpected results, and update the model of the problem. It must know when an apparent novelty is already known under different terminology. It must test whether the new result survives alternative specifications. It must distinguish a surprising artefact from a new phenomenon. These operations are far beyond next-token prediction considered in isolation, but they can be built around a language model.

Research-agent benchmarks are beginning to expose these difficulties. InnovatorBench requires agents to carry out concrete machine-learning research tasks and reports weaknesses in fragile algorithms, long-horizon decisions, resource management, and template dependence (Wu et al., 2025). That is exactly what one would expect before genuine autonomous science. The system can execute many local moves but still lacks the global judgement that tells it which research programme is worth pursuing and when the evidence has overturned its initial plan.

A late-century estimate for fully autonomous, broadly capable machine science is therefore defensible as a conservative scenario, but not because a trillion-token dense window is mathematically mandatory. It is defensible because scientific novelty combines memory with generalisation, experimentation, verification, planning, and institutional competence. Any one of those components may become the bottleneck. The future may surprise us with a breakthrough architecture that accelerates several at once. The honest forecast should therefore give ranges and conditions rather than pretending a single scaling law controls intelligence.

12. Red-Teaming the Argument

A serious forecast should be attacked before it is published. The first objection is that the hardware projection is too pessimistic because it assumes a large KV-cache footprint. That criticism is correct in part. KV memory per token is architecture dependent, grouped-query or multi-query attention can reduce KV heads, quantisation can reduce precision, and state-space architectures can remove conventional KV caches altogether. Hooper et al. (2025) demonstrated 10-million-token inference for a quantised LLaMA-7B configuration across eight A100 GPUs, illustrating how much compression changes the memory equation. The fix is to treat 50 KB per token as a scenario, not as an intrinsic constant. The late-2040s date therefore applies to a high-fidelity local working set under conservative memory assumptions, not to the first technical demonstration of a 10B-length sequence.

The second objection is that the argument overstates the need for dense attention. That criticism is also correct if “true context” is defined as all-pairs attention. Scientific reasoning does not logically require every token pair to interact on every layer. The corrected definition is functional: the system must preserve the ability to recover and enforce any material dependency. Sparse and hierarchical approaches are acceptable if they are auditable, reversible, and robust to hidden relevance. This correction makes the architecture more plausible while preserving the central critique of nominal context windows.

The third objection is that human experts do not possess giant context windows, so assigning million- or billion-token minima to human-level performance is incoherent. The answer is that the table does not model biological working memory. It models the machine-accessible working environment required to reproduce human professional performance without arbitrary information loss. Humans externalise memory continuously. A fair machine architecture should be permitted to do the same. The relevant metric is therefore addressable high-fidelity information, not neural simultaneity.

The fourth objection is that AGI might emerge from small-context agents with excellent tools long before billion-token context becomes feasible. This is possible. If general intelligence is primarily efficient abstraction and adaptive tool use, context size may be a weak predictor. The article therefore does not claim that ten billion tokens are necessary for AGI. It claims that very large high-fidelity information environments are likely necessary for some forms of superhuman research synthesis and broad scientific institutional memory. AGI could, in principle, arrive earlier through better learning and memory architecture.

The fifth objection is that the article may understate current progress by focusing on long-context failure benchmarks. Models have improved rapidly, and benchmark weaknesses discovered in one generation can disappear in the next. That is true. RULER, NoLiMa, LongBench, and related evaluations are snapshots, not permanent laws. The correct inference is not that models will always fail at 32K or 100K. It is that advertised context length has repeatedly failed to predict robust effective context. Future claims should therefore be validated on demanding tasks rather than accepted from specification sheets.

The sixth objection is that forecasting to 2048 from four years of GPU memory growth is statistically fragile. This is the strongest forecasting objection. An annualised rate derived from H100-to-Rubin evolution cannot legitimately be treated as a stable natural constant for two decades. The fix is to present 2048 as a scenario centre, not a regression result. It says: if high-end memory growth remains strong, if bandwidth and interconnect scale comparably, if architecture reduces unnecessary all-pairs computation, and if high-fidelity context is the target, then the late 2040s are a plausible local-hardware horizon. Remove those assumptions and the date moves.

The seventh objection is that “we are nowhere close to AGI” is stronger than the evidence permits. No benchmark can prove temporal distance. A conceptual breakthrough can collapse a forecast. The defensible wording is therefore that current evidence does not justify claims that AGI is imminent, and that several demonstrated deficiencies remain between current systems and robust general intelligence. “Not close” should be understood as an engineering assessment under continuity assumptions, not a theorem about the future.

The eighth objection is that context fidelity may not scale monotonically with memory. Larger models, better training, inference-time reasoning, or specialised retrieval may improve effective context without proportionate hardware. This is likely. The article’s focus on physical resources must therefore be paired with an architecture argument. The prediction is not “wait for more HBM.” It is “HBM, bandwidth, compute, and architecture must jointly improve.” That is a harder requirement, but it is also more realistic.

After these corrections, the argument becomes narrower and stronger. There is no universal minimum token count for intelligence. There is no universal KV size per token. Dense attention is not the only acceptable form of context. Hardware extrapolation cannot determine an exact AGI date. What is well supported is that nominal window size exceeds effective high-fidelity reasoning size on current benchmarks; research-grade synthesis imposes stricter provenance and dependency requirements than retrieval; brute-force dense attention scales badly; KV memory and bandwidth remain major constraints; and general intelligence requires capacities not measured by context length alone.

13. A Better Research Programme for Measuring Progress

If the industry wants meaningful evidence of progress toward machine research intelligence, it should stop treating context length as a headline metric and begin reporting a context fidelity curve. For each sequence length, the model should be tested on retrieval, associative retrieval, multi-hop reasoning, aggregation, contradiction detection, provenance tracing, exact numerical constraints, and dependency revision. The resulting curve would show not merely where the model accepts input but where different capabilities begin to fail. A “10M context” claim could then be decomposed into, for example, 10M admission, 2M retrieval, 500K multi-hop, and 100K high-fidelity contradiction tracking. That would be far more informative.

Second, research benchmarks should include late-relevance reversal. Insert information early in a corpus that appears unimportant, then introduce a later hypothesis that makes it decisive. The model should revisit the early material without being told where it is. This directly tests the weakness of retrieval systems that can find only what the current query already knows how to describe. Scientific discovery depends heavily on such reversals.

Third, evaluations should test dependency invalidation. Give the system a chain of derived conclusions based on a source assumption, then reveal that the assumption is wrong or qualified. The system should identify every downstream statement that needs revision. This tests whether the model has built an evidential graph or merely generated a plausible narrative.

Fourth, synthesis benchmarks should measure source independence. A machine should recognise when many papers trace back to one dataset or one unreplicated claim. Counting citations is not evidence synthesis. A research system should construct genealogies of evidence and discount duplicated support.

Fifth, benchmarks should require auditable final products. Every major substantive claim should have machine-readable provenance. Tables should link to source rows. Calculations should be reproducible. Contradictory sources should be exposed rather than silently harmonised. Reviewers should be able to challenge any sentence and obtain the evidence path that produced it.

Sixth, evaluation should include resource rationality. An autonomous researcher must decide where to spend compute, which paper deserves full reading, which experiment is informative, and when more searching has diminishing returns. This is a metareasoning problem. A system that achieves a result only by spending 10,000 times more computation than a competent human may still be useful, but it has not demonstrated comparable intelligence efficiency.

Seventh, the community should separate research assistance, research synthesis, and research novelty. Assistance means helping with bounded subtasks. Synthesis means constructing a defensible account of existing evidence. Novelty means producing and validating a result not contained in the corpus. These stages require overlapping but increasingly demanding capabilities. Conflating them allows progress on assistance to be marketed as progress on autonomous discovery.

Finally, AGI evaluations should include unknown task structures, not merely harder questions drawn from familiar distributions. Chollet’s argument remains central: a system’s generality is best revealed when it must acquire a new skill efficiently. Long context should support that adaptation by giving the model evidence and tools. It should not substitute for adaptation by placing the answer somewhere in the prompt.

14. The Likely Architecture of a Real Machine Researcher

A mature machine researcher is unlikely to be one model with one prompt. It will be a layered system. At the base will be immutable raw evidence: papers, datasets, code, experimental records, correspondence, and instrument outputs. Above that will be content-addressed indexes and structured extraction. The system will represent claims, entities, methods, variables, equations, causal relationships, and citations as explicit objects. A neural model will interpret ambiguity and propose connections. Formal tools will verify what can be verified. A planning layer will maintain goals, open questions, and uncertainty.

This architecture resembles a well-run research institution more than a solitary mind. Different memory systems serve different functions. Raw archives preserve detail. Databases preserve structure. Working memory supports the immediate problem. Verification systems police exactness. Search discovers relevant evidence. The generative model provides flexible abstraction. The controller decides when to move information between levels. What matters is that the levels are connected by provenance and that no irreversible summary is mistaken for the original evidence.

Such a system also offers a route to safer scaling. Rather than trusting a trillion-token latent state, it can enforce permissions and verification at object boundaries. A proposed citation can be checked against the source. A mathematical transformation can be validated before becoming trusted state. A dataset can be versioned. A hypothesis can remain explicitly speculative until supported. This turns epistemic status into system state rather than prose style.

The architecture would also make machine creativity easier to evaluate. A hypothesis can be compared against the stored literature to determine whether it is genuinely new. The system can search for counterexamples. It can record failed attempts rather than forgetting them. It can vary assumptions systematically and detect which conclusions are robust. Novelty becomes a property of a research graph, not a rhetorical impression generated by fluent text.

Most importantly, this architecture removes the false binary between “all tokens in context” and “lossy RAG.” There is a large design space in between. A theorem prover does not need the prose of every textbook active in a neural cache. A legal system does not need every page of disclosure in GPU memory if it can guarantee exact retrieval and dependency tracking. A scientific agent does not need every data point represented as a token if the data remain queryable in their native structure. The future of context is therefore likely to be heterogeneous.

The consequence for hardware forecasts is that raw token counts become increasingly metaphorical. A research system might have a ten-billion-token addressable corpus, a hundred-million-token semantic working memory, a million explicit structured claims, and a hundred-thousand-token neural reasoning window. Which number is its “context”? The only meaningful answer is behavioural: what information can reliably constrain its decisions? That is why context fidelity, not context length, should become the central metric.

15. Conclusion: The Future Is Large, but Not Infinite

The AI industry is right about one thing: context will become dramatically larger. It is already doing so. Models that once handled a few thousand tokens now accept hundreds of thousands, millions, and in some cases advertised windows of ten million. Hardware memory is growing. Bandwidth is growing. Quantisation is improving. New sequence architectures are reducing dependence on conventional attention. None of this should be dismissed.

But scale invites a familiar intellectual error: mistaking a quantity we can count for the capability we actually care about. A context window is a number. Research competence is a relationship among memory, reasoning, evidence, tools, verification, and adaptation. The first can rise rapidly while the second lags.

The benchmark literature already demonstrates this gap. Models that look excellent on simple retrieval can degrade sharply when context becomes longer, when literal matching is removed, when multi-hop relations are required, or when information must be aggregated across a large input (Bai et al., 2024; Hsieh et al., 2024; Modarressi et al., 2025; Zhang et al., 2024). The conclusion is not that long-context models are fraudulent. It is that “supports N tokens” and “reasons reliably over N tokens” are different claims.

For scientific research, the stronger claim is the one that matters. A good synthesis paper requires a persistent evidential model, not merely fluent summarisation. A machine that genuinely surpasses strong human researchers will have to preserve provenance, contradictions, methodological distinctions, and dependency structures over very large corpora. It will have to revisit earlier evidence when later discoveries change its relevance. It will have to know when compression has become dangerous. It will have to validate its own intermediate results.

The physical constraints are substantial. Conventional exact self-attention has an essentially quadratic computational structure. KV cache grows with sequence length and can dominate memory. HBM capacity and bandwidth are advancing quickly but remain far below the requirements of a naive ten-billion-token active state. Under a deliberately aggressive continuation of recent memory growth, the mid-2040s may be when raw four-GPU capacity begins to approach the relevant order of magnitude; allowing for bandwidth, compute, interconnect, architecture, and software pushes a practical central scenario toward approximately 2048.

That date should not be fetishised. An architectural breakthrough could bring high-fidelity research context forward. A slowdown in memory scaling could push it back. The number is useful only because it forces the assumptions into the open. Ten billion genuinely usable tokens on a local four-accelerator machine are not “next year with a better prompt.” They are a formidable systems problem.

Nor would solving that problem automatically produce AGI. A gigantic context can make a system knowledgeable without making it general. General intelligence requires efficient adaptation, robust transfer, autonomous planning, reliable error correction, grounded models of action and consequence, and competence under task structures that neither the model nor its developers have already anticipated. Current systems display pieces of this picture, sometimes spectacularly. They do not yet demonstrate the whole.

The more plausible route to machine scientific novelty therefore abandons the fantasy of one monolithic mind staring at every token simultaneously. It builds a research architecture in which all relevant evidence remains addressable, important relationships are explicit, formal claims can be verified, errors can invalidate downstream conclusions, and the model can decide what deserves attention without permanently discarding what it does not yet understand.

That architecture may eventually make a trillion-token scientific environment ordinary. It may allow machines to write synthesis papers that are not merely faster than human work but epistemically better. It may allow them to generate and test new theories. It may ultimately support something deserving the name general intelligence.

But there is a considerable distance between accepting ten million tokens and understanding ten million tokens; between understanding a corpus and maintaining it as a coherent research object; between synthesising what humanity already knows and discovering something humanity does not; and between being broadly useful and being generally intelligent.

Infinity is not required.

Reliability is.

And reliability at scale remains the harder invention.

---

References

Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., & Li, J. (2024). LongBench: A bilingual, multitask benchmark for long context understanding. arXiv. https://arxiv.org/abs/2308.14508

Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., & Li, J. (2025). LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv. https://arxiv.org/abs/2412.15204

Becker, J., Rush, N., Barnes, B., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. Model Evaluation & Threat Research. https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study-paper.pdf

Chollet, F. (2019). On the measure of intelligence. arXiv. https://arxiv.org/abs/1911.01547

Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. arXiv. https://arxiv.org/abs/2205.14135

Google. (n.d.). Long context: Gemini API. Google AI for Developers. https://ai.google.dev/gemini-api/docs/long-context

Gu, A., & Dao, T. (2024). Mamba: Linear-time sequence modeling with selective state spaces. arXiv. https://arxiv.org/abs/2312.00752

Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., & Gholami, A. (2025). KVQuant: Towards 10 million context length LLM inference with KV cache quantization. arXiv. https://arxiv.org/abs/2401.18079

Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What’s the real context size of your long-context language models? arXiv. https://arxiv.org/abs/2404.06654

Keles, F. D., Wijewardena, P. M., & Hegde, C. (2022). On the computational complexity of self-attention. arXiv. https://arxiv.org/abs/2209.04881

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. arXiv. https://arxiv.org/abs/2309.06180

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv. https://arxiv.org/abs/2307.03172

Meta. (2025). Llama 4 model card. GitHub. https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md

Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025). NoLiMa: Long-context evaluation beyond literal matching. arXiv. https://arxiv.org/abs/2502.05167

NVIDIA. (n.d.-a). NVIDIA H100 Tensor Core GPU datasheet. https://resources.nvidia.com/en-us-hopper-architecture/nvidia-tensor-core-gpu-datasheet

NVIDIA. (n.d.-b). NVIDIA H200 GPU. https://www.nvidia.com/en-us/data-center/h200/

NVIDIA. (n.d.-c). NVIDIA DGX B200 datasheet. https://resources.nvidia.com/en-us-dgx-software/dgx-b200-datasheet

NVIDIA. (2026). Inside NVIDIA Rubin GPU architecture: Powering the era of agentic AI. NVIDIA Technical Blog. https://developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/

Wu, Y., Fu, D., Si, W., Huang, Z., Jiang, M., Li, K., Xia, S., Sun, J., Xu, T., Hu, X., Lu, P., Cai, X., Ye, L., Zhu, W., Xiao, Y., & Liu, P. (2025). InnovatorBench: Evaluating agents’ ability to conduct innovative LLM research. arXiv. https://arxiv.org/abs/2510.27598

Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., & Sun, M. (2024). InfinityBench: Extending long context evaluation beyond 100K tokens. arXiv. https://arxiv.org/abs/2402.13718

---

← Back to Substack Archive