---
title: "Shannon Didn't Discover Information. He Measured It."
description: "The 1948 contribution was not defining information: it was giving it a unit, and thereby ending seventy years of qualitative argument. What the AI field quotes is the formula. What it needed to inherit is the move."
author: "Anderson Henrique"
date: "2026-08-30T00:00:00Z"
updated: "2026-09-14T22:27:27.662319Z"
category: "ai"
tags: ["Information Theory","Shannon","Entropy","Metrics","Benchmarks","Perplexity","Engineering","NTLabs"]
canonical: "https://www.ntlabs.dev/en/blog/shannon-nao-descobriu-a-informacao"
locale: "en"
---

# Shannon Didn't Discover Information. He Measured It.

*The 1948 contribution was not saying what information is. It was giving it a unit, and thereby ending seventy years of qualitative argument.*

## The meeting that would not end

"Is the system slow?" Every software team has spent weeks stuck on that question. Support says yes, because tickets are up. The backend team says no, because the server dashboard looks calm. Someone opened the app on their phone and found it fast. Someone opened it late in the afternoon and found it awful. Nobody is lying, and nobody convinces anybody, because nothing is actually in dispute beyond impressions of the same system.

What ends the conversation is someone defining what is being measured: the time between the click and the response, seen from the side of the person clicking, at the 95th percentile, in milliseconds. With that quantity in place, the server's comfortable average and the support complaints stop contradicting each other. Both were right about different things, and only one of them was the question.

The argument is not won by a better argument. It is ended by a unit.

That move has a name, a date, and an author. And the field I work in has not performed it yet.

## Seventy years of communication without a unit of information

When Claude Shannon published "A Mathematical Theory of Communication" in the Bell System Technical Journal in 1948, electrical communication was already a mature industry. The telegraph was a century old. The telephone, seventy years. Commercial radio, three decades. There were submarine cables, switching exchanges, continental networks, and an entire engineering discipline devoted to improving them.

What did not exist was agreement about what was being transmitted.

Engineers argued about the capacity of a channel, about which medium was more efficient, about whether a system made good use of available bandwidth, and they argued all of it without a common quantity underneath: each one measured what his equipment allowed him to measure, in units that did not convert into one another. Words per minute on the telegraph. Bandwidth in hertz on the telephone. Signal-to-noise ratio in decibels. None of those answers "how much information got through," because none of them measures information: they all measure the apparatus.

Two papers came close. Harry Nyquist, in 1924, showed that telegraph speed had a ceiling tied to bandwidth. Ralph Hartley, in 1928, went further and proposed measuring information by the logarithm of the number of possible messages. And, a detail that matters a great deal, Hartley argued explicitly that psychological factors had to be excluded for the thing to be measurable at all.

Hartley had the right instinct and an incomplete tool. His measure treats all possible messages as equally likely, which is almost never true of language, of images, or of any real source. The missing piece is the one Shannon brought: probability. Information stops being how many messages exist and becomes how surprising the one that arrived is.

## The move, in four steps

What Shannon did can be read as a formula. It is worth more read as a procedure, because a procedure is what you can copy.

**First, isolate the quantity.** Information is not the signal, not the channel, not the apparatus, and not the message. It is the reduction of uncertainty at the destination. That sounds like a philosophical definition and is not: it is a choice of boundary that makes everything else possible. Before measuring anything, you have to say precisely which thing.

**Second, strip out what does not belong to it.** The sentence in the paper is blunt to the point of sounding brutal: the semantic aspects of communication are irrelevant to the engineering problem. Shannon did not say meaning does not matter. He said it does not enter this measure, and that insisting on keeping it inside was the reason for seventy years without a theory.

**Third, define the unit.** The bit, a name he credits to John Tukey in a footnote. One bit is the uncertainty of a yes-or-no question with equally likely answers. Everything else is conversion. From there on, telegraph, telephone, and radio become comparable, because they finally share a quantity underneath.

**Fourth, prove the limits.** This is the step that separates a metric from a theory, and it is the one most often forgotten. Shannon did not stop at proposing entropy: he proved what it implies. The source coding theorem says entropy is the floor of lossless compression, and that no scheme, present or future, goes below it. The noisy-channel coding theorem says every channel has a capacity, and that below it you can communicate with error as small as you like, even though the channel is wrong all the time.

That fourth step is what gives a measure the power to refuse. A metric describes what happened. A theory says what cannot happen. When someone shows up promising lossless compression below the entropy of the source, nobody needs to audit the product: the claim is dead at the door.

And the formula, the one everyone quotes, is the residue of that process. It is what remains after the four steps, not what produces them. Quoting it without them is keeping the shell.

## Why stripping out meaning was the decisive step

The second step is the counterintuitive one, and that is exactly why it is the most transferable.

To the engineers of the day, meaning was the entire point. Communication exists to transmit meaning, and a theory of communication that declares meaning irrelevant looks like a theory that gave up on the problem. Shannon did the opposite of giving up: he saw that as long as meaning stayed inside, there would be no quantity at all, because meaning is not additive, is not comparable across people, and does not survive a scale.

To measure is, first of all, to decide what not to count. A thermometer that also told you whether the day is pleasant would not be a better thermometer: it would be an instrument without a scale, and therefore not a thermometer. The rigor of a measure comes as much from what it captures as from what it refuses to capture.

There is a price, and Shannon paid it with his eyes open. The measure starts saying uncomfortable things. A page of random noise has maximum entropy. A page of poetry does not. The theory does not tell them apart by worth, only by predictability. That sounds like a defect and is precisely the opposite: it is the proof that the measure is measuring what it said it would, and not the opinion of whoever reads it.

Worth noting that he knew. In 1951 he published "Prediction and Entropy of Printed English," where he estimated the entropy of written English by asking people to guess the next letter of a text. He landed around one bit per character. The same man who expelled meaning from the theory came back later to measure how much meaning makes a text predictable. That is not a contradiction: it is the difference between excluding a variable from the instrument and denying that it exists.

## The AI field is pre-1948

This is where the history stops being history.

We argue today about whether a model reasons. Whether it understands. Whether a system is agentic. Whether we are close to something. Those arguments have the same shape as the ones that occupied communication engineering in 1930: competent people, real evidence on both sides, and no quantity actually in dispute. Each side measures what its equipment allows, and the conclusions do not convert into one another.

Benchmarks do not fix this, and it is worth being precise about why. A benchmark is an instrument, not a scale. It suffers from contamination, because the test leaks into training and the measure loses its meaning without losing its appearance. It suffers from saturation, because once everyone gets almost everything right it has stopped measuring the difference that matters and keeps producing numbers anyway. And it suffers from incomparability: two benchmarks claiming to measure the same capability rank models differently, and there is no conversion between them, because there is no shared quantity underneath. A thermometer and a barometer disagree too.

Evals do not fix it either, and confusing the two is an expensive mistake. An eval measures task success. It is useful, it is necessary, and I build them. But task success is a property of the model-task pair, not of the quantity that was in dispute. Saying a model reasons because it passed a set of tasks is the same move as saying a channel has high capacity because the call sounded fine yesterday.

And now the part that convinces me this is a diagnosis and not a complaint.

The field has exactly one Shannonian unit in everyday use, and it is the single region where progress is comparable, cumulative, and uncontroversial: cross-entropy, and its presentational sibling, perplexity. Measured in bits per token. It says, literally, how much uncertainty remains about the next symbol. It is the same quantity from 1948 applied to the same question Shannon asked in 1951, except that the one guessing the next letter is now the machine.

Notice what happens where that unit holds. Scaling laws were discovered there, and they let you predict the performance of a large model before training it. Compression and prediction turned out to be the same operation seen from two angles. Comparisons across architectures hold up over years, not just until the next release. Nobody argues about whether one model has lower perplexity than another on the same corpus: you measure, and the conversation is over.

Now notice where the unit does not hold. Reasoning, understanding, agency, alignment, situational awareness. None of those has a unit, and all of them have years of debate without convergence, with the same arguments returning on a cycle with each generation of models.

The conclusion is not that the field is incompetent. It is that the field is post-1948 in the layer where it does engineering and pre-1948 in the layer where it does discourse, at the same time. And both layers share a vocabulary, which makes the rigor of one look like it covers the other.

## Measure the instrument before trusting the measurement

There is a practical corollary to Shannon's move that costs little and prevents a lot, and it is not in the 1948 paper: before believing a measurement, check that the instrument was reading.

An example of my own, recent and embarrassing. The repository's secret scanner ran on every commit and came back clean. Three hundred and thirty-seven green runs, over months. The configuration extended a rule set without turning on the key that actually loads that rule set, so the scanner ran with zero rules. It never said "there are no secrets." It was saying "I did not look," and I read the second sentence as though it were the first. When the rules came in, four hundred and thirty findings appeared.

The same pattern elsewhere in the same repository: a change-detection step failed, the tests sat in a skipped state because nothing had been declared for them to run, and the CI gate passed. Green. Always green.

What those two cases share with the slow-system meeting is the structure, not the subject. A number existed, was consulted, drove decisions, and was measuring nothing. Worse than having no measure is having one that cannot come out wrong, because it manufactures confidence without producing information.

In Shannon's terms this is literal, not a metaphor. A source that emits only one symbol has zero entropy. A green that cannot turn red carries no bit. If the outcome was certain before the run, the run informed you of nothing.

The rule I now follow is mundane, and I should have had it earlier: run the positive control. Before accepting a zero, deliberately plant the thing the instrument is supposed to catch, and check that it catches it. If it does not, the zero is not a result. It is a silence.

## What to inherit

Shannon is almost always quoted for the formula, and the formula is the part that travels worst outside its own problem. What transfers to any field is the four-step procedure, and it is cheap:

1. What exactly is the quantity in dispute?
2. What has to come out of it for it to become measurable?
3. What is the unit, and what does one of it mean?
4. What are the limits, and what would be wrong if someone violated them?

Applying that to "is the system slow" takes an afternoon and ends a meeting. Applying it to "the scanner is clean" takes fifteen minutes and surfaces four hundred and thirty findings. Applying it to "the model reasons" is a generation of work, and that is exactly why it is worth starting now instead of publishing more adjectives.

I will close with the criterion that stayed with me, the shortest test I know for whether a measure is a measure:

**A number that cannot come out wrong is a number that is not measuring anything.**