Compression studies of LLMs have shown large amounts of capacity can be trimmed with little loss on many tasks. In addition, for a given token, only a fraction of the LLM contributes meaningfully to the output:
In OPT-30B, 66B and 175B, approximately 95% of MLP neurons and 80% of attention heads can be omitted for a given token in their contextual-sparsity experiment. Different inputs use different subsets.
https://proceedings.mlr.press/v202/liu23am/liu23am.pdf
Of course, the above paper is quite dated, but I believe modern architectures to be sparse (or heavily redundant) wrt tasks as well.
How do we take advantage of this? I believe in the training phase we should experiment with:
Only training a fraction of the LLM that is active for a given token. This means identifying some percentage of the LLM that meaningfully contributes to the output, and ignoring the updates for the rest. Presumably, this could speed up calculations.
And/or repeatedly training that percentage on the target until it successfully fits the output.
In local experiments, I had improved on the Modded-NanoGPT world record by 20-30% using the above methodology (I also initialized NanoGPT to be more orthogonal), but I had yet to verify the results myself, or replicate it on an 8xH100 cluster.
It is clear from various papers that LLMs have a large amount of redundancy (circuits that all implement the same task). I am curious if there are ways we could reduce this with some custom loss function, or at least get more bang for our buck.

