Something is afoot in the above graph, what I can tell you is at around 10:15pm is when I disabled transparent hugepages on this box. So where why am I suddenly buffering and caching more?

What are transparent hugepages?

From what I understand hugepages were introduced to maximize the use of the TLB (translation lookaside buffer) which is a fixed size CPU cache used to speed up virtual memory address to physical memory address lookups. Hugepages combines multiple smaller pages into one huge page such that the TLB can potentially contain more useful pages as a whole and we can skip deep lookups more often. Now this really only matters for systems that are allocating large amounts of memory and in fairly sizeable chunks, or so I assume.

Where did my vfs cache go?

The reason this attracted my attention was because we were having IO problems on a few of our RHEL6 boxes that normally had no issues when run on identical hardware and running FreeBSD instead. The reason was pretty clear, we’re hardly caching anything and we keep going to disk for frequently accessed blocks, but why?

I bet it has something to do with this statement from https://lwn.net/Articles/423584/:

“The current patch only works with anonymous pages; the work to integrate huge pages with the page cache has not yet been done.”

These boxes are primarily running apache with mmap enabled, I’m guessing that when we mmap a file (which is 99% of the time going to be far bigger than one 4k page) we’re allocating a hugepage, and as such skip any sort of page cache for future accesses of that same data. I could be entirely wrong on how those internals work as I havent looked too deeply but it would certainly explain why we’re seeing this behaviour.

So this begs the question of why this is enabled by default on RHEL6, normally they’re pretty conservative on potentially performance breaking features. I can hazard a guess that it’s because the majority of RedHats customers are running large JVM installations where hugepages would actually help and not small apache setups like we are.

Oh well, for now: echo never > /sys/kernel/mm/redhat_transparent_hugepage