Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
MCML Authors
Ali Modarressi
Yihong Liu
Dr.
Hinrich Schütze
Prof. Dr.
Core PI
Abstract
Ali Modarressi
Yihong Liu
Dr.
Hinrich Schütze
Prof. Dr.
Core PI
Abstract
Activation patching can identify a mixture-of-experts (MoE) block whose clean output restores a corrupted factual prediction. However, because the block output combines contributions from multiple routed experts, block-level rescue does not establish whether the recovery localizes to an individual expert or depends on the routed expert set. We study this question on single-token COUNTERFACT contrasts by corrupting subject-token embeddings, restoring clean block outputs, and then restoring clean-minus-noised expert updates under fixed routing. In Qwen3-30B-A3B-Base, a discovery sweep selects layer 44, and held-out analysis identifies L44E069 as a recurrent routed contributor with positive specificity over same-layer active experts. Its effect is fact-matched and improves true-token probability and rank, which explains part of the layer rescue. In Mixtral-8x7B-v0.1, the selected recurrent singleton is not specific; matched-size controls instead show specificity of the clean-routed top-2 set. These findings show that successful MoE-block restoration does not necessarily imply localization to a single expert.
inproceedings LML+26a
EMNLP 2026
Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. To be published. Preprint available.
Authors
Y. Lu • A. Modarressi • Y. Liu • H. SchützeLinks
URLResearch Area
BibTeXKey: LML+26a