In this talk I will describe an experimental framework we designed to quickly run controlled experiments and gain insights into the inner workings of language models. This framework currently serves as a basis for running many experiments in our team, some of which will be explained during this talk.