If the hole in the middle of your records is too small, you may notice that you have to force it onto the spindle. Removing it is equally just as difficult. All your records should be able to almost be dropped right onto it.
The main problem however is when you start to mix and especially when you start to scratch. As it is stuck to the spindle (and therefore the platter) the spindle acts as a brake, so you can't spin it back very far at all.
To fix this, just put a pen (or similar item) in the middle to stretch it.
Made it too wide now? Read this guide about reducing the hole size