Document Utilities

Architecture of ChatGPT

Modern ChatGPT-class systems are widely understood to use Mixture of Experts (MoE) ideas rather than running one single dense network for every token. In an MoE design, a router decides which specialized expert subnetworks should process each segment of input. This keeps capability high while controlling compute cost per token.

Dense vs MoE Intuition

  • Dense model: Every parameter path is active for every token.
  • MoE model: A router activates only a subset of experts per token.
  • Benefit: Higher effective capacity without paying full dense-compute cost each step.

Routing Diagram (Conceptual)

User Prompt
↓
Router Network
↓
Expert 1 (Code)
Expert 2 (Math)
Expert 3 (Grammar)
↓
Combined Output

Why Routing Helps

Different tasks stress different competencies. Code generation benefits from syntax priors and long dependency handling, math needs reasoning patterns, and grammar/style needs linguistic calibration. MoE routing lets specialized sub-networks contribute where they are strongest, then aggregate into one coherent response.

Educational Caveat

Proprietary architectures are not fully public in all details. The model above is an educational abstraction that captures the key principle: conditional computation with expert routing.

Takeaway: MoE is about routing intelligence efficiently, not just making one giant always-on network.