Steering MoE LLMs via Expert (De)Activation

MCML Authors

Ali Modarressi

→ Group Hinrich Schütze
 Computational Linguistics

Hinrich Schütze

Prof. Dr.

Principal Investigator

Computational Linguistics

Abstract

Mixture-of-Experts (MoE) in Large Language Models (LLMs) routes each token through a subset of specialized Feed-Forward Networks (FFN), known as experts. We present SteerMoE, a framework for steering MoE models by detecting and controlling behavior-linked experts. Our detection method identifies experts with distinct activation patterns across paired inputs exhibiting contrasting behaviors. By selectively (de)activating such experts during inference, we control behaviors like faithfulness and safety without retraining or modifying weights. Across 11 benchmarks and 6 LLMs, our steering raises safety by up to +20% and faithfulness by +27%. In adversarial attack mode, it drops safety by -41% alone, and -100% when combined with existing jailbreak methods, bypassing all safety guardrails and exposing a new dimension of alignment faking hidden within experts.

misc

---

Preprint

Sep. 2025

---

Authors

M. Fayyaz • A. Modarressi • H. Deilamsalehy • F. Dernoncourt • R. Rossi • T. Bui • H. Schütze • N. Peng

---

Links

---

Research Area

 B2 | Natural Language Processing