I was able to solve this problem by browsing to the following folder location:

%APPDATA%\Microsoft\Windows\Recent\AutomaticDestinations

and deleting all the files in there. To get into the...

This will show the "Close Window" option but is there any way to have the option show up by default, as it used to, when I right-click a file explorer window?

I'm not sure what caused this problem but it seemed to appear somewhat suddenly a few weeks back or so.

Whenever I right-click a file explorer window (i.e. This PC, Downloads etc.) on the taskbar,...