SPOKEN
OpenSubtitles 2018
Film and television subtitles. Excellent for conversational English, and the reason a list built on it alone over-ranks interjections.
CC-BY-SA
Source
Method
How the English word frequency list is built
The whole site hangs off one claim: that this is the real order of English by frequency. That claim is only worth anything if the working is visible, so here it is.
The corpora
NEWS
Leipzig Corpora — news 2023
English-language journalism. Brings in the vocabulary of public life — government, economy, law — that dialogue barely touches.
CC-BY
Source
ENCYCLOPEDIC
Leipzig Corpora — Wikipedia 2016
Formal, descriptive prose, and the source that keeps abstract and technical vocabulary in proportion.
CC-BY
Source
One corpus is a genre, not a language. Subtitles alone would teach you to follow a film and leave you lost in a newspaper; Wikipedia alone would do the reverse. Each word is ranked on how it behaves across all three.
How they are combined
01
Rates, not counts
The corpora are different sizes, so raw counts cannot be compared. Each is converted to occurrences per million words first.
02
Geometric mean
The three rates are averaged in log space. This rewards a word that is moderately common everywhere over one that spikes in a single register.
03
What is removed
Proper nouns, detected by how often the word appears capitalised in the case-preserving sources. Plus abbreviations and web debris. A word must appear in at least two corpora to rank at all.
04
What is added
Pronunciation, from CMUdict. Parts of speech, levels, example sentences and translations are written by hand.
What English made us change
- Pronunciation is stored, not derived. Spanish spelling predicts sound well enough to compute it; English does not, so every transcription comes from CMUdict. It is General American and broad, so a British learner’s vowel in “bath” is not what this list shows.
- The sources disagree about apostrophes. Leipzig keeps them; the subtitle list splits on them, so “don’t” arrives as “don” plus “t”. Only plain alphabetic words are ranked, and the subtitle corpus abstains on those stems rather than counting them — otherwise “don” ranks 535th on four million occurrences that are all “don’t”.
- “I” is always capitalised, so the proper-noun filter discarded it until it was named as an exception. Loosening the filter instead would have let real names back in.
- British and American spellings rank as separate entries — centre and center, defence and defense — because words are ranked as they are written. This is honest about the data and unhelpful to a learner, and it may change.
What is not finished
- The lexicon is written by hand: 5,000 of the 5,000 words have a part of speech, a level, an example sentence and a meaning in every language here. Levels are a judgement rather than an official mapping, and the translations give the commonest equivalents, not every sense.
- The coverage figures are measured against these corpora, with proper nouns and numerals left in the denominator. That is why they are lower than the numbers usually quoted: the first 1,000 words cover 72.6% of subtitle dialogue but only 63.8% averaged across dialogue, news and encyclopedic prose. Quoting the first figure alone is how “1,000 words is 75% of English” became folklore. The full measurement, and where the 75% comes from.
- Three corpora are still three corpora. Spoken English that never becomes a subtitle, regional vocabulary and specialist registers are all under-represented.
The same method, for Spanish
cincomil ranks the 5,000 commonest Spanish words from the same three kinds of corpus, blended the same way, with the same proper-noun filter and the same refusal to quote a coverage figure it has not measured.