Xbox One Cast to Annoying Issue

I use the cast to function a lot from my pc to Xbox one and 9 times out of 10 it will work brilliantly but I've noticed if I don't use my Xbox for a period of time at all, cast to will not work. My Xbox is still connected to the internet and works as I watched a YouTube video earlier but cast to wouldn't work at all. The Xbox showed on my pc in the cast to open but nothing popped up at all. I tried renaming my Xbox, finding my device again, disconnecting from the Xbox internet but nothing would work. In the end I had to reset my Xbox from the console menu. Are other people having this issue?

Hoping somebody can help with this as it's so frustrating when it happens.

Please anyone?