技术博客 数据库、运维与工程实践笔记

HugePages on Oracle Linux 64-bit

2024-07-12Linux/AIX / Oracle / RAC
Purpose
Scope
Details
Introduction
Why Do You Need HugePages?
How to Configure
Check and Validate the Configuration
Troubleshooting
Known Problems and Limitations
Further Reading
References

Applies to:

Linux OS – Version Enterprise Linux 4.0 to Oracle Linux 7.2 with Unbreakable Enterprise Kerne [3.8.13] [Release RHEL4 to OL7U2]

Oracle Database – Enterprise Edition

Linux x86-64

Oracle Linux

Red Hat Enterprise Linux (RHEL)

SUSE Linux Enterprise Server (SLES)

Purpose

This document aims to provide.

  • Basic configuration of HugePages on 64-bit Linux
  • Fundemental reasons to use HugePages on Linux
  • References to known problems
  • References to technical background on HugePages

Scope

Information in this document is useful for Linux system administrators and Oracle database administrators working with system administrators.

This document covers information about Linux HugePages for 64-bit architectures. For more generic and uses on 32-bit and for references please see Document 361323.1

The configuration steps provided here is primarily for Oracle Linux. Still the same concepts and configurations should apply to other Linux distributions.

Details

Introduction

HugePages is a feature of the Linux kernel which allows larger pages to manage memory as the alternative to the small 4KB pagesize. For a detailed introduction, see Document 361323.1

Why Do You Need HugePages?

HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If your combined database SGAs is large (like more than 8GB, can even be important for smaller), you will need HugePages configured. Note that the size of the SGA matters. Advantages of HugePages are:

  • Larger Page Size and Less # of Pages: Default page size is 4K whereas the HugeTLB size is 2048K. That means the system would need to handle 512 times less pages.
  • Reduced Page Table Walking: Since a HugePage covers greater contiguous virtual address range than a regular sized page, a probability of getting a TLB hit per TLB entry with HugePages are higher than with regular pages. This reduces the number of times page tables are walked to obtain physical address from a virtual address.
  • Less Overhead for Memory Operations: On virtual memory systems (any modern OS) each memory operation is actually two abstract memory operations. With HugePages, since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided.
  • Less Memory Usage: From the Oracle Database perspective, with HugePages, the Linux kernel will use less memory to create pagetables to maintain virtual to physical mappings for SGA address range, in comparison to regular size pages. This makes more memory to be available for process-private computations or PGA usage.
  • No Swapping: We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead. HugePages are universally regarded as pinned.
  • No 'kswapd' Operations: kswapd will get very busy if there is a very large area to be paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are used, kswapd is not involved in managing them. See also Document 361670.1

Document 361323.1

How to Configure

The configuration steps below will guide you to do a persistent system configuration where you would need to do a complete reboot of the system. Please plan your operations accordingly:

Step 1: Have the _memlock_ user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than installed RAM. e.g. If you have 64GB RAM installed, you may set:

[60397977]()[

]()

There is no harm in setting this value larger than your SGA requirements.

The parameters will be set by default on:

  • Oracle Linux with oracle-validated package (See Document 437743.1) installed.
  • Oracle Exadata DB compute nodes

Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the _memlock_ limit

>

[60397977]()

Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic Memory Management (AMM) feature which is incompatible with HugePages. Disable AMM before proceeding. To disable, set the initialization parameters MEMORY_TARGET and MEMORY_MAX_TARGET to 0 (zero). Please see Document 749851.1 for further information in case you encounter the error below:

Document 1453227.1

Step 4: Make sure that all your database instances are up (including ASM instances) as they would run on production. Use the script _hugepages_settings.sh_ in Document 401749.1 to calculate the recommended value for the _vm.nr_hugepages_kernel parameter. e.g.:

[$ ./hugepages_settings.sh

…

Recommended setting: vm.nr_hugepages = 1496]()

Note: You can also calculate a proper value for the parameter yourself but that is not advised if you do not have extensive experience with HugePages and concepts.

Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:

>

[vm.nr_hugepages = 1496]()

This will make the parameter to be set properly with each reboot.

Step 6: Stop all the database instances and reboot the server

(Although settings could have been done dynamically they would not be effective to the extent we require before a reboot. The best practice is to do a persistent system configuration and perform a reboot to complete the configuration as presented through the steps above)

What If the Database / SGA Configurations Change?

The performed configuration is basically based on the RAM installed and combined size of SGA of database instances you are running. Based on that when:

  • Amount of RAM installed for the Linux OS changed
  • New database instance(s) introduced
  • SGA size / configuration changed for one or more database instances
  • Poor database performance
  • System running out of memory or excessive swapping
  • Database instances cannot be started
  • Crucial system services failing

Check and Validate the Configuration

After the system is rebooted, make sure that your database instances (including the ASM instances) are started. Automatic startup via OS configuration or CRS, or manual startup (whichever method you use) should have been performed. Check the HugePages state from_ /proc/meminfo_. e.g.:

The values in the output will vary. To make sure that the configuration is valid, the _HugePages_Free_ value should be smaller than _HugePages_Total._

Also there should be some _HugePages_Rsvd _if PRE_PAGE_SGA is 'false' for all the Oracle database instances.. _HugePages_Rsvd_ counts free pages that are reserved for use (requested for an SGA, but not touched/mapped yet). PRE_PAGE_SGA determines if the all SGA pages are read-in when the instance starts up. If parameter is set to 'true' then the OS page table entries are prebuilt for each page of the SGA, leading to HugePages reservation of those pages. For Oracle database versions before 12.1 the default value for PRE_PAGE_SGA is 'false'. So the _HugePages_Rsvd_ would be higher than 0. But since 12c the PRE_PAGE_SGA defaults to 'true' which would cause the _HugePages_Rsvd_ to be 0.

The sum of _Hugepages_Free _and _HugePages_Rsvd_ may be smaller than your total combined SGA as instances allocate pages dynamically and proactively as needed.

Troubleshooting

Some of the common problems and how to troubleshoot them are listed in the following table:

Symptom Possible Cause Troubleshooting Action

System is running out of memory or swapping | Not enough HugePages to cover the SGA(s) and therefore the area reserved for HugePages are wasted where SGAs are allocated through regular pages. | Review your HugePages configuration to make sure that all SGA(s) are covered.

Databases fail to start | memlock limits are not set properly | Make sure the settings in limits.conf apply to database owner account.

One of the database fail to start while another is up | The SGA of the specific database could not find available HugePages and remaining RAM is not enough. | Make sure that the RAM and HugePages are enough to cover all your database SGAs

Cluster Ready Services (CRS) fail to start | HugePages configured too large (maybe larger than installed RAM) | Make sure the total SGA is less than the installed RAM and re-calculate HugePages.

HugePages_Total = HugePages_Free | HugePages are not used at all. No database instances are up or using AMM. | Disable AMM and make sure that the database instances are up. See Doc ID 1373255.1

Database started successfully and the performance is slow | The SGA of the specific database could not find available HugePages and therefore the SGA is handled by regular pages, which leads to slow performance | Make sure that the HugePages are many enough to cover all your database SGAs

Known Problems and Limitations

Below are some of the known and related problems and limitations related to the HugePages feature:

Further Reading

To be able to do advanced / manual configurations with HugePages you need to understand the implementation and theory behind the concept. You may read the following for further information:

References

NOTE:361323.1 – HugePages on Linux: What It Is… and What It Is Not…

NOTE:749851.1 – HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux

NOTE:1373255.1 – 11.2.0.1/11.2.0.2 to 11.2.0.3 Grid Infrastructure and Database Upgrade on Exadata Database Machine

NOTE:401749.1 – Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration

NOTE:1392497.1 – USE_LARGE_PAGES To Enable HugePages

NOTE:361670.1 – Slow Performance with High CPU Usage on 64-bit Linux with Large SGA

NOTE:1557478.1 – ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, and UEK2 and above