Can't connect to home server until I ping myself from it (IPv6 only)

I have an old home server running ubuntu server 20.04 which is reachable at the IPv6 link local address fe80::38%25. For some reason, about 3 days ago, I stopped being able to communicate with the server @ fe80::38%25 until I SSH into it using its IPv4 address and ping my own computers IPv6 address. The first few ICMP sequences will always fail, but around the 4-6th one my computer will respond and everything will be fine until I restart my computer. My windows firewall is set to allow all outgoing traffic and only allow incoming traffic that matches exceptions. Disabling the Windows firewall completely does solve the issue, but when I enable it and add an exception allowing all traffic from fe80::/64, it still doesn't work? This doesn't happen with any other devices. My computer can ping my routers link local IPv6 address without issue with no firewall modifications needed, and the global IPv6 address of the home server works flawlessly. In addition, other devices can reach fe80::38%25 no problem.

Maybe someone else with more extensive knowledge of the windows firewall could tell me what's going on and what I'm doing wrong?
Thanks.

Windows Version 10.0.18363.900