Server will make players "Timed out", and they can't rejoin the server until it is restarted.

So my friend and I have started a modded SMP which we have been playing on, and just recently sometimes we would both get "Timed out", and the server console still thinks that we are on, but we aren't (I've tried kicking ourselves on the console, but it doesn't work). If we were to join the server, it would stay on Encrypting, then after a minute it would just be Timed Out. It has already happened 3 times today in the span of 2 hours, and its so annoying because we have to restart the server which takes 4 minutes each time.

Mod list

The server ran fine with 3 people and 19 mods a few days ago, but now it can't run 2 people with 14 mods... I don't get it..

Is there a way I can stop this from happening? Thanks

I also forgot to mention that when this does happen, the server will never stop. It stays open until someone restarts the server. I've waited 2 hours and the server still hasn't closed (nobody is even online even though it says otherwise)

the server constantly sends messages of "server running ** ticks behind"