Extremely High RAM utilization, high Paged Pool

The issue started in mid-February when I received multiple warnings and critical alerts from my RMM software for several PCs I administer - the RMMs settings for RAM alerts are a warning at 85% and critical at 90%.
All of these systems have been deployed for over 2 years and I've never had any of them give high memory alerts before. Now, of the 12 PCs in the office, 10 PCs are going to critical alert with the RAM use slowly creeping up over a 4+ day period. The users are complaining of the PCs becoming slow, lagging, or unresponsive. A reboot tends to make the PC usable again until the RAM gets critical again.


Image 1 is RMM metrics from a users PC showing RAM (purple line) before the issues started, and after February 12^th, the RAM creeping to +90% after the issues started
Image 2 is a current image of the RMM metrics for the last month





To make the issue more obvious, here is an image from our conference room Intel NUC, that is idle most of the time. I know that the spikes are the right side are while the PC was mostly idle and I was rebooting the PC to get the RAM to drop:


To help you understand some of the things I have tried, here are samples of the alerts I got from my RMM software:





As I mentioned above, I have 10 PCs, some are idle, having this issue. I have 2 actively used PCs that are NOT having the issue. In trying to troubleshoot, I have basically compared the idle systems with the issue to the 2 active user PCs that are not having the issue.
Here are images from Process Explorer, Task Manager and RAMMAP from an idle system that is having the issue:




Initially seeing epsecurityservice at the top of Process Explorer and in the RMM alert, I thought BitDefender Endpoint Security (GravityZone) was the issue. I applied a “disable” policy to BitDefender on an idle system and the issue did seem to stop after a reboot.

HOWEVER, seeing svchost in the list, I also tried a Clean Boot, but disabled everything EXCEPT BitDefender and my RMM software so I could continue to connect to the system.

Again, the issue seemed to stop with a Clean Boot and BitDefender still enabled.

So I compared Process Explorer metrics for a system with the issue and one that doesn’t have the issue and found that epsecurityservice.exe had roughly the same RAM use between the two systems… and the numbers were not increasing steadily over time.
This made me think BitDefender isn’t the issue but perhaps something it uses in the OS.

So I then looked at RAMMAP and Task Manager, and that is when I saw the extremely high Paged Pool on the idle system and a relatively small Paged Pool on the system in use. So I checked all the systems with the issue and they all have extremely high paged pools, even on systems that have been rebooted and left idle for days.

Here are screenshots of for one of my systems that is not having the RAM issue:




And just for a comparison reference, here is what my RMM metrics for RAM look like on one of my PCs that doesn't have the issue:



Here is a quick rundown of these systems configurations which is making it hard to determine if it is a driver issue since 4 different configurations are having the issue:



I'm not sure whether the huge Page Pool is okay, but it doesn't seem normal when compared to how my RAM utilization is for the 2 systems that are not having any issues.

If anyone has suggestions on what else I should try to resolve this, I’d greatly appreciate it.

If you want more information, please give me step-by-step simply to save time – I don’t want to waste any time assuming I know what you want and giving you the wrong info.

Thanks in advance for the help!!

Hello bumgarb42,
The problem is being caused by Endpoint. Try to update it or uninstall it - whichever fixes the issue.

If uninstalling fixes it, try reinstalling it and see if the problem remains fixed or comes back.

Thanks for the response steve108. I have a support ticket into BitDefender pointing this out and asking for any tool they might have that doesn't a full uninstall to allow for a CLEAN install of BitDefender.

Just curious, why you think the CLEAN BOOT where I left BitDefender enabled also seemed to clear up the issue?

It is weird that this is not impacting all the systems at that location.

I'll check back with you if doing the uninstall/reinstall doesn't work.

Thanks!

bumgarb42 said:

Thanks for the response steve108. I have a support ticket into BitDefender pointing this out and asking for any tool they might have that doesn't a full uninstall to allow for a CLEAN install of BitDefender.

Just curious, why you think the CLEAN BOOT where I left BitDefender enabled also seemed to clear up the issue?

It is weird that this is not impacting all the systems at that location.

I'll check back with you if doing the uninstall/reinstall doesn't work.

Thanks!

Maybe something the clean boot didn't start is linked with this problem - I have no idea what that might be.

Thank you again, steve108, for the response.

I used a BitDefender supplied removal tool to fully uninstall Endpoint, reboot, and reinstall Endpoint.
Prior to this I had the test system using a "Disable All" policy for Endpoint. (between orange lines in the image)
After install, the issue came back (after 2nd orange line)

HOWEVER, between 0100 and 0300 hrs on March 29th (this morning), all PCs with the issue updated from Product version 6.6.24.337 (engine 7.88017) to version 6.6.25.362 (engine 7.88018), and this seems to have correlated to RAM being released on all of these systems (green line in image).


Other examples of correction:



The previous BitDefender version was released around mid-February when I started having the issue so hopefully, that is the fix. I'll continue to monitor.

Thanks for the help!

it's probably solved until the next buggy driver or app

I just found it in the Bitdefender Endpoint Security Tools Version 6.6.25.362 release notes (windows)

"The Firewall driver generated a memory leak in EPSecurityService on Windows machines."
Bitdefender Endpoint Security Tools Version 6.6.25.362 Release Notes (Windows)

Thanks again!

- - - Updated - - -

I am very disappointed to report that the memory leak was reintroduced about 7 hours after it was resolved during a BitDefender security engine update. I have alerted BitDefender to the issue which will hopefully see it resolved sooner than the previous leak.