Can't connect to server when my friend is online

Whenever my friend comes on the modded server we made (forge skyfactory 4) I always lag, I can't chat, I can't interact with anything. It wasn't like this until a few days ago, me and my friend were just in the twilight forest and defeated 2 naga bosses and got back to the overworld, I got called by my parents and went afk, thought it was fast enough to were I can go back to my laptop and not get kicked, but I got kicked, and I tried to log in again, and it says "Connection Lost Internal Exception: java.io.IOException: An existing connection was forcibly closed by remote host" and my friend said I left the server without even joining (check attachment for the photo). I also tried to trace route and it says that it timed out. This only happens when my friend is in the server, when I'm alone it's fine. Can someone help me?