You are here
BLUF - Mediaserver VM usage not dropping as expected according to PVE GUI during idle periods.
Alright, first off, these images are awesome. The MediaServer got me off the ground and running way faster than the last one I tried building from instructions.
The data:
Server is Supermicro X10DRL-i with 2x E5-2670 and 256GB RAM, running Proxmox 5.4.124.
Hypervisor on a two-drive SSD ZFS mirror configuration.
The four spinning disk drives in hotswap bays are setup in a raidz2 configuration. Test VM's are then allocated storage from this pool.
No ZFS cache disk setup at this time.
Currently, two VMs setup. MediaServer v16.1 and Linux Mint 20.2.
Now the issue I'm observing.
Linux Mint (and other VMs in the past) seem to report accurate RAM usage. It goes up and down with demand.
MediaServer does not. I've allocated 48GB to the VM and during a file upload (using WinSCP), the usage will climb to 97% (if I'm moving more than ~50GB) and stay there, even well after the transfer is done.
I recall reading that ZFS will hold data to write to the pool in RAM, if there is no cache disk. But I would have figured once all data was uploaded that as the data is written, the usage would drop back down. It does not. 4hrs after transfer complete, the VM still shows 97% of allocated RAM in use, despite sitting at idle the entire time.
Possibly unrelated (but I'll mention anyway in case it is helpful) is the VM does not respect Proxmox command to shutdown/restart from the PVE gui. Shutdown/reboot at command line inside of VM works.
I've only been working in Proxmox for a few months (as I have time) and I've spun up a number of VMs. This is the first and only time I've observed this issue. I have kept Proxmox up-to-date but I can't point to any specific updates that might have triggered this behavior. And the fact it seems unique only to the Turnkey VM makes me think there is some issue between the two in their configuration.
Hey Timmy.
To be honest, I'm not sure.
I have a Media Server instance running on my Proxmox server too. Mine is an LXC container and only has 4GB assigned to it and idles at 1GB usage. Mine isn't backed by ZFS. I have the OS filesystem running from a thin LVM volume on a SSD, with the media storage provided by a number of WD red HDDs in a singe logical volume, bind mounted into my Media Server instance at /srv/storage.
Neither my TurnKey Media Server nor Proxmox are the latest release, which may or may not have any relevance.
I'd be inclined to try to pin down what is actually chewing up the RAM. From within the VM/container run:
For example, here's my Media Server:
total used free shared buff/cache available Mem: 4.0Gi 978Mi 56Mi 16Mi 3.0Gi 3.0Gi Swap: 6.0Gi 349Mi 5.7GiNote that you need to read the 'available' column. As you can see buffer/cache is "using" most of my RAM leaving only 56MB 'free', but that would be cleared immediately if it's required, so there is actually 3GB available.
If yours is showing lots of free RAM internally, but not within Proxmox, then I'm not sure what's going on there. Personally, I'd be inclined to ask on the Proxmox forums. I suspect that they'd have more insight into what might cause that. If you do post over there, please feel free to post a link to that here.
As to not responding to the Promxox interface for shutdown/restart, it depends. If yours is also an LXC container, then I'm not clear what could cause that. If you have installed to a "proper" VM (i.e. KVM) then there is more that can go wrong there, but generally it should "just work". I vaguely recall someone else having an issue with that ages ago and it was some setting within Proxmox that they changed that fixed it. If yours is the same case, then IIRC that was a setting about using ACPI calls?! If Proxmox is sending ACPI calls, then TurnKey should respond to those. However, there is also a way it can be handled via the QEMU guest agent. And TurnKey doesn't include the agent by default, so installing that is a possible alternate route.
If you want to install the qemu-guest-agent package, do this (includes a reboot):
Please let me know if that helps.
What I've found so far
Ran the free -h command prior to starting a transfer.
Then several times during a large transfer.
The internal count jives with proxmox on the way up. I waited and once the transfer completed, I watched the lights continue go (indicating the write is carrying on). Once the lights were off, I checked it again.
Then I looked into it and found a Debian command to clear the cache/buff.
Proxmox jives with the internal measurements after the cache clear. So Proxmox appears to be reading accurately.
root@mediaserver ~# free -h
total used free shared buff/cache available
Mem: 47Gi 365Mi 46Gi 10Mi 255Mi 46Gi
Swap: 5.9Gi 0B 5.9Gi
root@mediaserver ~# free -h
total used free shared buff/cache available
Mem: 47Gi 378Mi 42Gi 10Mi 4.6Gi 46Gi
Swap: 5.9Gi 0B 5.9Gi
root@mediaserver ~# free -h
total used free shared buff/cache available
Mem: 47Gi 379Mi 35Gi 10Mi 11Gi 46Gi
Swap: 5.9Gi 0B 5.9Gi
root@mediaserver ~# free -h
total used free shared buff/cache available
Mem: 47Gi 377Mi 32Gi 10Mi 14Gi 46Gi
Swap: 5.9Gi 0B 5.9Gi
root@mediaserver ~# free -h
total used free shared buff/cache available
Mem: 47Gi 429Mi 30Gi 10Mi 16Gi 46Gi
Swap: 5.9Gi 0B 5.9Gi
root@mediaserver ~# sync; echo 3 > /proc/sys/vm/drop_caches
root@mediaserver ~# free -h
total used free shared buff/cache available
Mem: 47Gi 425Mi 46Gi 10Mi 150Mi 46Gi
Swap: 5.9Gi 0B 5.9Gi
In so far as the power off/restart command issue: I'm not sure how to see what command Proxmox sends to the VM. I've got qemu agent disabled. I think I recall reading that thread yesterday. The VM is KVM. I have not dived into LXC at all and a quick look at the how-to, I've definitely not done that.
I've got ACPI enabled and verified packages installed on turnkey. Went ahead and installed QEMU on the mediaserver and enabled it on proxmox. QEMU agent works.
That shows there is no issue.
Whilst the RAM is still in use by buffer/cache and is not "free", if you look at the 'available' column, it is still available for use by other apps. So there isn't actually any issue there. If something needed to use the RAM, it would be instantly freed. Why the data remains cached in RAM after writing to disk I'm not sure, but what you've shown here shows that there isn't actually anything wrong or any issue here.
And it sounds like installing the qemu-guest-agent solved the PVE UI shutdown/restart issue?
Follow up
I get it isn't bad; the memory is there and available.
But it does give me a misconception that the VM is under provisioned on memory when it isn't and makes it hard to judge performance when the main dashboard is, effectively, a lie. But reading around further last night, knowing what I know now, I see this is a long-standing complaint.
Guess I'll test and see how long it tries to hold that data. Maybe after an hour or two, it'll drop it. Otherwise, after doing uploads, I'll just clear it manually of clean data.
Apologies. My final sentence wasn't clear. Yes. QEMU agent resolved the issue.
You could probably write a script to do that...
If you want to make it so Promxox sees (more closely) what the actual available RAM is, then you could write a script for that!? Something along the lines of checking the 'free' vs 'available' and if the difference is too high, runs the commands you posted before (i.e. 'sync; echo 3 > /proc/sys/vm/drop_caches'). Then run that script as a cron job.
It's a bit hacky, and as already discussed the issue is only cosmetic, but it should be fairly easy to achieve your desired outcome. If you're unfamiliar with bash, now is probably a great time to learn a bit! :)
As a hint, you can get the free and available RAM (in kb) like this:
And if your script is /usr/local/bin/clear-free-ram, then you can make a cron job that will run every 5 minutes like this:
Put it on the list
Yeah.. it is 'cosmetic' for the most part. Need to read up on Proxmox for scripting that sort of thing.
Put it on the list of things to do. After figuring out this VNC issue.
Thanks for the assist.
No worries.
Good luck with it all and please don't hesitate to post back with any other questions and/or feedback you have.
Only Suggestion
Donation via paypal!
Add new comment