Use Mergekit to shrink model size ?

[claeyz]

I have a question that may sound silly. Has anyone used mergekit to shrink model size instead of combining several models ?

For example we could take as sources the same model, consider it's 2 models, and use merge kit to merge layers so that we would go for example from 24 layers to 12 layers. That could be a good base then to distill the created student on the teacher.

[spydaz]

you can use the method wih slices ... as it gives you the chance t select the layer range hence your merge may create a different B ... ie if you choose slices 0-10 then for the next model 10-20 then you model wil be 20 layers only ... hence if they were both 7b models then the model will be smaller ! .. for 7b 32 layers ...