Probing Conceptual Universals in a 200-Language Neural Translator

1. Introduction

What does it mean for a neural network to "understand" a concept across languages? When Meta's NLLB-200 translates "water" from English into Yoruba, Hindi, and Japanese, does its encoder construct a unified, language-neutral representation of the concept — or does it merely learn language-labeled clusters that look universal from the outside?

This question sits at the intersection of two literatures that have largely developed in parallel. In NLP interpretability, Chang, Tu, and Bergen (2022) demonstrated that multilingual model embeddings can be decomposed into language-sensitive axes (encoding vocabulary, script, and language identity) and language-neutral axes (encoding part-of-speech, token position, and universal syntactic structure). In cognitive science, Correia et al. (2014) used fMRI multivariate decoding to show that the human anterior temporal lobe supports a language-independent conceptual hub — a classifier trained to recognize Dutch words could decode which English word a bilingual had heard.

InterpretCognates bridges these literatures by probing NLLB-200's internal geometry with tools drawn from both traditions. The project occupies a genuinely novel scientific niche: prior mechanistic interpretability work on multilingual models covers at most 88 languages (Chang et al., 2022); we operate on 200, with translation supervision rather than masked LM pretraining, using an encoder-decoder architecture that the geometry literature has almost entirely overlooked. The external validation resources we bring to bear — ASJP v20 phonetic distances, CLICS³ colexification data, Berlin & Kay color universals — have never been tested against NLLB's representation space.

The foundational architecture in the psycholinguistics literature is Dijkstra and van Heuven's BIA+ model (2002), which proposes that bilinguals maintain a single integrated lexicon accessed in a language-nonselective, parallel fashion. NLLB's shared-encoder architecture is the direct computational implementation of this finding: the same weights process all 200 languages simultaneously, with language selection deferred to the decoder's forced BOS token. This architectural mapping provides the interpretive framework for every encoder-level analysis we run.

The core questions we address:

1. Does NLLB's encoder contain a geometrically isolable, language-neutral conceptual store?
2. Does NLLB's representation space implicitly encode the phylogenetic tree of human languages?
3. Does the embedding space reflect universal conceptual associations independent of language?
4. Are semantic relationships between concepts (man→woman, big→small) translationally invariant across 135 languages?

2. Model & Method

We use facebook/nllb-200-distilled-600M, the 600M-parameter distilled dense variant of NLLB-200 (NLLB Team et al., 2022). The distillation from a sparse mixture-of-experts to a dense model is significant: it means every attention head and every FFN neuron participates in processing every language, making the model particularly amenable to mechanistic analysis. The model is a shared-encoder, encoder-decoder transformer with 12 encoder layers and 16 attention heads per layer, covering 200 languages across dozens of scripts.

Swadesh Stimulus Corpus

Our primary stimulus set is the Swadesh 100-item list — 101 core vocabulary concepts (body parts, basic natural phenomena, low numerals, kinship terms) with translations in 135 NLLB-supported languages spanning 19 language families. Swadesh (1952) predicted that these items are culturally stable and resistant to borrowing, making them markers of genealogical relationship across millennia. Jäger (2018) formalized this into the ASJP database covering ~7,000 languages.

Embedding Extraction

For each word in each language, we use a word-in-context extraction strategy following standard transformer probing methodology (Hewitt & Manning, 2019; Tenney et al., 2019). Each target word is placed inside a fixed carrier sentence — "I saw a {word} near the river." — which is translated into the target language and run through the NLLB encoder with the appropriate source-language BCP-47 code set via the tokenizer (tokenizer.src_lang = lang_code). From the encoder's last hidden states, we extract only the activations corresponding to the target word's subword tokens and mean-pool them, yielding a contextually disambiguated 1024-dimensional embedding. This avoids the degenerate representations that bare-word inputs can produce in encoder-decoder models.

Isotropy Correction (ABTT)

Rajaee and Pilehvar (2022) showed that multilingual transformer embedding spaces are severely anisotropic — representations cluster in a narrow cone, biasing cosine similarity toward frequent token patterns rather than genuine semantic proximity. We apply All-But-The-Top (ABTT) post-processing: subtract the global mean, then remove the top-k principal components (with k=3) before computing cosine similarity. Every quantitative similarity claim in this report uses isotropy-corrected vectors unless otherwise noted.

Language Mean-Centering

Chang et al. (2022) demonstrated that raw embeddings are dominated by language-sensitive variance. The language-neutral axes — which carry actual semantic content — only emerge after per-language mean subtraction. We subtract each language's centroid vector from all of that language's embeddings before PCA projection, transforming the visualization from language-family clusters to concept-meaning clusters. The ratio of between-concept to within-concept distance before and after centering quantifies the strength of the conceptual store.

3. The Conceptual Manifold

To build intuition, we begin with a single concept: "water". The carrier sentence "I drink {water}." places the target word in sentence-final position with high cloze probability, then the word is replaced by its translation in each of 30 typologically diverse languages. For each translation, the full sentence is encoded, but only the hidden states corresponding to the target word's subword tokens are extracted and mean-pooled. These word-in-context embeddings are then projected into 3D via PCA. The result is a conceptual manifold — the geometric surface in NLLB's 1024-dimensional space along which the concept "water" is represented across languages.

Translations

Embedding Geometry

Each target word is embedded inside the carrier sentence shown below, with {word} replaced by the translation in each language. The 3D PCA scatter plot shows each language's embedding of "water" as a point. Points cluster by language family — Romance languages huddle together, as do Slavic, Turkic, and Niger-Congo languages — confirming the language-sensitive variance that Chang et al. (2022) identified.

Embedding Geometry Statistics

The embedding cloud for a single concept reveals key geometric properties of the manifold. We quantify the distribution of pairwise cosine similarities across all 29 language embeddings to characterize how tightly the concept is represented. Higher mean similarity and lower variance indicate stronger conceptual convergence.

Cross-Language Similarity

The cosine similarity matrix below quantifies how similar each pair of language embeddings is for the concept "water." High similarity between typologically distant languages (e.g., Finnish and Japanese) is evidence that the model represents the concept at a level deeper than surface form.

4. Swadesh Convergence Ranking

With single-concept intuition established, we scale up to the full Swadesh 100-item list. For each of the 101 core vocabulary concepts, we embed every translation across all 135 languages (producing over 13,000 embeddings total), then compute the mean pairwise cosine similarity across all language pairs. Concepts with high mean similarity converge strongly across languages — their representations are close in NLLB's embedding space regardless of which language expresses them.

Swadesh's cultural stability hypothesis (1952) predicts that these core vocabulary items should converge more than culturally specific vocabulary. The scatter plot below tests this prediction by placing every concept along two axes: how phonetically similar its translations are across Latin-script languages (x-axis) and how strongly it converges in NLLB's embedding space (y-axis). Concepts that converge strongly despite low phonetic similarity are the strongest candidates for genuine conceptual universals.

Category Summary

Grouping Swadesh concepts by semantic category reveals which types of concepts are most universally represented. For each category the chart below shows NLLB embedding convergence alongside orthographic and phonetic similarity among Latin-script translations. Categories where embedding convergence clearly exceeds surface-form similarity provide the strongest evidence that the model's universality reflects semantic structure rather than shared word form.

The Polysemy Confound

Inspection of the bottom-ranked items reveals a pattern: concepts with the lowest cross-lingual convergence are disproportionately polysemous. "Bark" (tree bark vs. dog bark vs. to bark), "lie" (to recline vs. to tell a falsehood), "see" (visual perception vs. to understand) — low convergence for these items may reflect translation ambiguity rather than cultural instability. The word-in-context extraction strategy partially mitigates this: the fixed carrier sentence provides a mild noun-sense bias, and the encoder contextualizes each word against its sentential frame. However, a single carrier sentence cannot fully disambiguate all senses, so polysemy remains a confound. Controlling for it (e.g., via WordNet sense counts as a covariate) is essential before interpreting the bottom of the ranking as evidence against conceptual universality.

Variance Decomposition: Surface Form vs. Semantic Universals

A key methodological risk: high cross-lingual convergence could reflect cognate overlap (shared surface forms from common etymological origins) rather than genuine conceptual universality. To disentangle these, we regress embedding convergence on orthographic similarity (mean pairwise normalized Levenshtein distance across 21 Latin-script languages) and approximate phonetic similarity (same metric on phonetically normalized forms).

The residual for each concept — its semantic surplus — measures the degree to which it converges beyond what shared surface form would predict. Concepts above the regression line are the strongest candidates for genuine conceptual universals. The complement of R² gives the variance attributable to deeper semantic structure, providing a principled answer to: is this real semantics, or just shared spelling?

Methodology Notes

5. Phylogenetic Structure

If NLLB's encoder genuinely captures something about the relationships between languages, its embedding distance matrix should correlate with independently established measures of linguistic relatedness. We test this by computing a language pairwise cosine distance matrix averaged over all Swadesh concepts, then examining whether the resulting structure recovers known phylogenetic relationships.

The phylogenetic correlation experiment bridges three literatures simultaneously. If NLLB's cosine distance matrix over Swadesh vocabulary correlates significantly with ASJP phonetic distances, the result speaks to NLP researchers (geometry of multilingual models), cognitive scientists (universal language network), and historical linguists (computational phylogenetics). No single prior paper has attempted this. We use a Mantel test — correlating the upper triangles of two distance matrices via Spearman rank correlation with a permutation test for significance.

3D PCA Projection

Principal Component Analysis projects the language centroid vectors into three dimensions, preserving the dominant axes of variation. If the model has learned phylogenetic structure, languages from the same family should cluster together — and the relative positions of families should roughly mirror their known genealogical relationships. Toggle between raw embeddings (showing language-family clustering) and mean-centered embeddings (revealing deeper structural relationships).

Hierarchical Clustering

The dendrogram below shows hierarchical clustering of languages by their NLLB embedding distances. The tree structure can be compared against the known phylogenetic tree of human languages to assess how well the model's learned geometry recapitulates established genealogical relationships.

Mantel Test

The Mantel test asks a simple question: do languages that sound alike also get encoded alike? For every pair of the 88 languages with both NLLB and ASJP data, we have two numbers:

1. X-axis — ASJP Phonetic Distance: how different the two languages' core vocabulary sounds, based on the Automated Similarity Judgement Programme taxonomy. Higher = more phonetically distant.
2. Y-axis — NLLB Embedding Distance: how far apart the two languages' Swadesh-word embeddings are inside NLLB's encoder (cosine distance, averaged over all concepts). Higher = the model represents them more differently.

If the model has implicitly learned genealogical relationships, we expect a positive correlation: closely related languages (low ASJP distance) should also be close in embedding space (low embedding distance).

Reading the scatter plot

ASJP distances are computed from real ASJP v20 word lists (Wichmann et al., 2022): for each pair of languages, we calculate the mean Normalized Levenshtein Distance across all shared 40-item Swadesh words in ASJP phonetic transcription. This produces a continuous distribution reflecting genuine phonetic similarity.

• Closely related pairs (low ASJP distance): language pairs within the same branch, e.g., French–Spanish (both Romance) or Russian–Polish (both Slavic). These share recent common ancestors and retain many cognate word forms.
• Moderately related pairs (mid ASJP distance): pairs from different branches of the same macro-family, e.g., English–Russian (Germanic vs. Slavic) or Spanish–Hindi (Romance vs. Indo-Iranian).
• Unrelated pairs (high ASJP distance): pairs from entirely different language families, e.g., English–Chinese or Swahili–Finnish.

A positive correlation across the full range confirms that NLLB's encoder has implicitly learned phonetic/genealogical relationships from translation co-occurrence statistics alone.

Conceptual Maps by Language Family

Each point in the concept map represents a Swadesh concept positioned by its average embedding across all languages in a selected family. PCA is fitted on overall centroids so that per-family maps share the same coordinate space, enabling direct visual comparison of how different language families organize the same conceptual inventory.

If all families produce similar concept maps, the conceptual structure is truly language-universal. Systematic differences between families may reveal culture-specific or typology-specific semantic biases encoded by the model.

6. Validation Tests

The experiments above establish that NLLB's encoder contains structure that correlates with known linguistic and cognitive universals. The following validation tests probe this structure from multiple independent angles, each testing a specific hypothesis against external benchmarks from cognitive science and historical linguistics.

6.1 Isotropy Correction Test

Does the Swadesh convergence ranking hold under isotropy (ABTT) correction? Rajaee and Pilehvar (2022) showed that multilingual transformer embeddings are severely anisotropic, meaning raw cosine similarity is dominated by frequency artifacts rather than genuine semantic proximity. If our convergence ranking changes dramatically after isotropy correction, the signal is an artifact; if it remains stable (high Spearman ρ), the structure is genuinely geometric.

6.2 Swadesh vs. Non-Swadesh Comparison

Swadesh's cultural stability hypothesis predicts that core vocabulary converges more strongly across languages than culturally specific vocabulary. We test this by comparing 101 Swadesh concepts against 60 non-Swadesh concepts (e.g., "government," "university," "airport," "democracy") — modern, abstract, and culturally contingent terms. Both sets are embedded across the same languages, and a one-sided Mann-Whitney U test evaluates whether Swadesh concepts show significantly higher mean cross-lingual similarity.

6.3 CLICS³ Colexification Test

Rzymski, Tresoldi et al.'s CLICS³ (2019) database identifies concept pairs that are lexified by a single word across many unrelated languages — revealing universal conceptual adjacencies. We extract colexification pairs directly from the CLICS³ SQLite database, filtering to pairs attested in ≥3 language families. If NLLB assigns higher cosine similarity to frequently colexified pairs (e.g., "foot"–"leg" at 52 families, "hand"–"arm" at 48 families) than to never-colexified pairs, this bridges typological linguistics and neural representation analysis in a way no prior paper has attempted.

6.4 Conceptual Store Metric

The conceptual store hypothesis (Correia et al., 2014) predicts that after removing language-identity information, the ratio of between-concept distance to within-concept distance should increase substantially. We compute this ratio on raw embeddings and after per-language mean-centering. The prediction from the executive summary: at least a 2× improvement factor after centering.

6.5 Semantic Offset Invariance

This experiment tests a question that goes beyond first-order convergence (do translations of the same concept cluster together?) to second-order structure: are relationships between concepts preserved across languages? The hypothesis is that semantically fundamental relationships — man→woman, big→small, I→we, eat→drink — produce offset vectors that are translationally invariant: the direction from "man" to "woman" in English embedding space should be parallel to the direction from "homme" to "femme" in French embedding space.

Mean consistency close to 1.0 indicates high cross-lingual invariance — the relationship is a universal property of the conceptual space. The per-family breakdown is analytically important: invariance that holds across unrelated families (Indo-European, Sino-Tibetan, Niger-Congo) is strong evidence for language-universal conceptual structure, while invariance restricted to related families may reflect shared etymology. No prior paper has tested semantic offset invariance across 135 languages in a translation model. The closest precedent is Mikolov et al.'s (2013) word2vec analogy work, which operated within a single language.

Per-Family Invariance Heatmap

The heatmap below decomposes offset invariance by language family. Each cell shows the mean consistency of a semantic relationship within a specific language family. Uniform warmth across families indicates the relationship is truly universal; cold spots in specific families reveal where the model's encoding diverges from the cross-lingual norm.

6.6 Berlin & Kay Color Circle

Berlin and Kay (1969) established that human color categorization follows universal constraints: languages add color terms in a predictable order, and perceptually adjacent colors (red/orange, blue/green) are categorized similarly across cultures. If NLLB's embedding space captures these universals, the arrangement of color term centroids (averaged across 30+ languages) should recover the perceptual color circle — red adjacent to orange and purple, green adjacent to yellow and blue — regardless of which languages contributed the terms.

6.6b Layer-wise Color Circle Evolution

The color circle in Figure 13 uses final-layer (layer 11) representations. But how does this structure emerge across the encoder stack? The animation below shows the same color term embeddings extracted from each of the 12 encoder layers, all projected into the same PCA coordinate space (fit on the final-layer centroids). Use the slider to scrub through layers manually, or press Animate to watch the representational space self-organize in real time — scattered noise in early layers coalescing into the tight, perceptually ordered color circle by the final layer.

6.7 Carrier Sentence Robustness

All preceding analyses embed target words inside a fixed carrier sentence. Does this carrier drive our convergence findings, or would bare words produce the same ranking? We repeat the full embedding extraction and convergence analysis using decontextualized embeddings — target words embedded in isolation, without any surrounding context.

If the carrier sentence is merely providing a consistent frame (as intended) rather than imposing structure, the contextualized and decontextualized convergence rankings should be highly correlated. A low correlation would suggest that the carrier sentence, not semantics, drives the observed patterns.

6.8 Layer-wise Emergence of Semantic Structure

All preceding analyses use representations from the final (12th) encoder layer. But where in the encoder stack does cross-lingual semantic structure emerge? We repeat the convergence analysis at each of the 12 encoder layers on a diverse 39-language subset.

If semantic convergence builds gradually across layers — with surface-level features dominating early layers and abstract semantics dominating later ones — this would parallel the "NLP pipeline" effect documented by Tenney et al. (2019) and provide a developmental lens on the conceptual store hypothesis.

6.9 Controlled Non-Swadesh Comparison

The original Swadesh vs. non-Swadesh comparison (Section 6.2) is confounded by loanword bias: modern terms like telephone, democracy, and hotel share surface forms across dozens of languages due to cultural borrowing, inflating their apparent convergence. To obtain a properly controlled comparison, we constructed a second non-Swadesh set of frequency-matched, non-loanword concrete nouns with cross-linguistic orthographic diversity comparable to the Swadesh set.

Under this controlled baseline, Swadesh concepts exhibit convergence commensurate with or exceeding that of the matched controls — consistent with the hypothesis that culturally stable, universally attested concepts develop robust cross-lingual representations despite maximal surface-form diversity.

7. Discussion

Taken together, these results constitute a coherent empirical story: NLLB-200's encoder contains a language-neutral conceptual store, its geometry reflects universal conceptual structure, and its representation of core vocabulary correlates with cross-cultural cognitive universals. The BIA+ two-system architecture maps directly onto NLLB's encoder-decoder split — the encoder functions as the language-nonselective "word identification system," while the forced BOS token mechanism functions as the "task-decision system" that imposes language constraints.

The mean-centering result is perhaps the most revelatory: subtracting per-language mean vectors reorganizes the PCA from language-family clusters to concept-meaning clusters, providing the first direct geometric test of whether a neural translation model develops something functionally analogous to the anterior temporal lobe's conceptual hub. The improvement in between/within concept distance ratio quantifies how much of the embedding space is dedicated to language-identity information versus genuine conceptual content.

The phylogenetic correlation, if confirmed at statistically significant levels, would be a landmark finding: a neural machine translation model has implicitly learned the genealogical tree of human language from translation co-occurrence statistics alone, without any training signal about linguistic phylogeny. This result speaks simultaneously to the NLP community (the geometry encodes more than task-relevant features), to cognitive scientists (computational models can develop language-network structure paralleling the human brain), and to historical linguists (neural embedding distances may complement traditional phonetic distance measures).

Limitations & Cautions

• Tokenization artifacts. Cross-script comparisons are confounded by SentencePiece tokenization: English-French cognate pairs share many subword tokens; English-Japanese pairs share none. All attention entropy experiments should be restricted to Latin-script language pairs until edit-distance is included as a covariate.
• Cosine similarity without correction is unreliable. Every quantitative similarity claim must use isotropy-corrected vectors. Raw cosine is appropriate only for qualitative visualization, clearly labeled as such.
• Non-Swadesh translations are AI-generated. The 60-concept non-Swadesh corpus was compiled programmatically and should be spot-checked by native speakers before publication claims depend on it.
• ASJP coverage varies. The phylogenetic analysis uses real ASJP v20 word-level phonetic distances (Wichmann et al., 2022), but not all NLLB languages have ASJP word lists. Languages without ASJP coverage are excluded from the Mantel test.
• Fixed carrier sentence. All word-in-context embeddings use a single translated carrier sentence. Our decontextualized baseline (Section 6.7) shows high rank correlation (ρ = 0.867), confirming that the carrier does not drive the main patterns. Nevertheless, a multi-template approach with averaged embeddings would further reduce frame-specific bias.

Layer-wise Developmental Parallels

The layer-wise trajectory analysis adds a developmental dimension to these structural parallels. Semantic convergence increases monotonically across the 12 encoder layers, with a sharp rise around layer 1 and the Conceptual Store Metric exhibiting a phase transition at layer 6. This mirrors hierarchical processing in the human language network, where primary auditory cortex encodes acoustic features, posterior temporal regions encode phonological and lexical information, and the anterior temporal lobe hub integrates meaning across modalities and languages. Concrete, perceptually grounded concepts achieve high cross-lingual convergence earlier in the encoder stack than abstract or polysemous concepts, suggesting a hierarchy of representational abstraction.

Future Directions

Several promising directions emerge from this work: per-head cross-attention decomposition (finding universal semantic alignment heads), resource-level asymmetry testing (the RHM prediction that L1→L2 translation proceeds via conceptual mediation while L2→L1 can bypass it), and scaling analyses across model sizes (from 600M to the full 3.3B NLLB-200). The core empirical story — Swadesh convergence, variance decomposition, phylogenetic correlation, colexification, offset invariance, layer-wise emergence, and carrier-sentence robustness — provides a comprehensive probe of the encoder's conceptual geometry.