-
Notifications
You must be signed in to change notification settings - Fork 205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] mfs.cgi "RAM used" - SWAP not included #515
Comments
We show resident RAM usage in CGI INFO tab, this is quite typical. RSS + VIRT usage is shown on the charts and there it should not increase after your swapoff operation (just change proportion between light and dark blue) - did you check your charts? We could show both in INFO tab too, we just never felt the need, because typically mfsmaster should not be allowed to swap. |
Typically mfsmaster is not swapping, but we had a lot of metadata changes in cluster and metadata download operation (big file operation, file caches etc) "moved" some mfsmaster memory to swap which had impact on overall performance during metadata save (same system drive holds swap and metadata and during metadata download/save they fight for IO). I see it this way. I think indication on mfs.cgi dashboard that mfsmaster is in SWAP would be nice to have overall as a "red light" to be checked. If I understand this can be made if this data is avaible on charts (you are correct, charts change colors but not overall sum of memory). Mayby "SWAP used" next to "RAM used" or make RAM usage in red if it includes SWAP space? We have 196GB of RAM for masters (leader and followers), mfsmaster process takes ~105GB so there should be no swapping but if all other memory is used by caches... strange things can happen. There is an option to turn off SWAP, but it's not recommended overall. This is what I've experienced (Leader master server). You can see that even SNMP was not responding sometimes during metadata download/save. Cluster also was slow or unresponsive. Metadata file is ~38GB. |
No, issue #303 is the problem of RAM growth during metadata save, when there is another operation that changes the data drastically during saving time (like chunkserver and chunks registering). This is a specific case that we put safeguards against since then - in certain cases (registering, leadership change etc.) metadata save is just not done - it's done at the next scheduled time. |
@chogata np, make it as "nice-to-have", changed title to feature ;) |
System information
Your moosefs version and its origin (moosefs.com, packaged by distro, built from source, ...).
3.0.116 PRO
Operating system (distribution) and kernel version.
Ubuntu 18.04 LTS
Describe the problem you observed.
Master server used some SWAP memory few days ago. After starting "swapoff -a" to force it to RAM I've noticed that "RAM used" on mfs.cgi panel started to increase by amount of SWAP that was pushed to RAM.
Can you reproduce it? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Make master/follower server use SWAP, check the value of "RAM used" and then "swapoff" it back to RAM.
Include any warning/errors/backtraces from the system logs.
No warnings etc., just misinformation.
The text was updated successfully, but these errors were encountered: