An experiment I would like to try is as follows:
Take a transformer. We’ll call it the golden transformer.
Initialize it using a scheme that includes randomness.
Create a random set of inputs and feed it to the above transformer to create a set of outputs. Consider this the input/output set.
From this, we can take another transformer and calculate:
How close can it get to the golden transformer using standard gradient descent methods and how does this vary as it’s given more or less samples.
How close does gradient descent push the candidate transformer towards the golden transformer at each step.
How different is a gradient descent step from an oracle optimizer step that moves directly toward the golden transformer's weights? The oracle trajectory follows a straight line through parameter space from the candidate's initial weights to the golden weights, reaching them in a single step with a learning rate of 1. By comparing the direction, magnitude, and resulting distance reduction of gradient descent updates against this oracle trajectory, we can measure how efficiently gradient descent approaches a known solution.
If the candidate transformer is made much larger than the golden transformer does it converge faster or help with convergence? Does it generalize or overfit?
If the candidate transformer and the golden transformer are the same size and architecture, how close does the candidate transformer’s internals get to the golden transformer’s before it stops improving? Does it completely match the input-output of the golden transformer but have majorly different internal weights?
Can such a strategy be used to train a transformer to better guide gradient descent or some other kind of learned optimizer?
Of course initialization scheme matters a lot for all of the above, and I would expect there need to be some varied analysis.