1. Introduction to Cluster-Level Deployment
2. Cluster-Wide Configuration Principles
3. Cluster-Level Configuration via vSphere Client GUI
   1. Pre-Validation
   2. Manage configuration at cluster level
   3. Draft Desired Configuration
   4. Post Validation
4. Integration with vSphere Cluster Services
   1. Distributed Resource Scheduler (DRS) Awareness
   2. High Availability (HA)
5. Operational Monitoring and Capacity Planning
   1. Performance Metrics & vCenter Charts
6. Capacity Planning Considerations
   1. NVMe Device Health and Lifecycle
7. Conclusion

Introduction to Cluster-Level Deployment

Part 1 established host-level vSphere Memory Tiering activation. This segment extends deployment to a multi-host vSphere cluster, addressing consistency and integration with distributed services. It details vSphere Client GUI configuration, interaction with vMotion, DRS, and HA, and covers operational monitoring and capacity planning. vCenter Server functions as the central control plane for cluster-wide management.
This blog will be structured in Three distinct sections:

Part 1: Deploying vSphere 9.0 Memory Tiering Over NVMe: Host-Level Activation.
Part 2: Deploying vSphere 9.0 Memory Tiering: Cluster Integration and Management.
Part 3: Deploying vSphere 9.0 Memory Tiering: Considerations with vSAN Environments

Cluster-Wide Configuration Principles

Effective cluster deployment of Memory Tiering mandates configuration homogeneity.

• Uniformity: All ESXi hosts must consistently enable Memory Tiering with comparable NVMe device characteristics (type, performance, provisioned capacity).
• Impact of Inconsistency: Non-uniform configurations impair DRS decisions, induce vMotion failures, and introduce VM performance variability.
• Centralized Management: vCenter Server provides comprehensive management and orchestration.
• Architectural Segregation: For heterogeneous hardware environments, isolating Memory Tiering-enabled hosts into dedicated clusters or host groups is recommended for controlled resource pools.

Cluster-Level Configuration via vSphere Client GUI

The vSphere Client offers a direct, interactive method for configuring Memory Tiering on individual hosts. For cluster-wide deployment, this process is repeated for each host to ensure uniformity.

Pre-Validation

Capture 1: –

1. Click on the Cluster.
2. Click on Summary.
   Note:- We can see the memory of the cluster level is configured is 288 GB. each host i have given the 96GB.(each host memory 96 GB * total nos of hosts 03= 288 GB).

Manage configuration at cluster level

Capture 1:-

1. Click on Cluster.
2. Click on Configure.
3. Click on Configuration under Desired State.
4. Click on Create Configuration.

Capture 2: –

5. Click on Import from Reference host.

Capture 3: –

6. Click on radio Button.
7. Click on Import.

Capture 4: –

8. Click on Close.
Note:- Here we are importing the image & configuration of host.

Capture 5: –

9. Click on Next.

Capture 6:-

10. Click on Next.

Capture 7 :-

11. Click on Finish and Apply.

Capture 8:-

12. Click on Configure.
13. Click on Configuration under Desired State.
14. Click on Compliance.

Draft Desired Configuration

Capture 1: –

1. Click on the cluster.
2. Click on Configure.
3. Click on Configuration Under Desired State.
4. Click on Draft.
5. Click on Create Draft.

Capture 2: –

6. Click on vmkernel under ESX.
7. Click on Options.

Capture 3: –

8. Click on Common Settings.
9. Click on Edit.

Capture 4: –

10. Find the memory Tiering and click on TRUE
Note: – By default the memory tiering value is FALSE.
11. Click on Save.

Capture 5: –

12. Click on RUN PRE-CHECK.

Capture 6: –

Note: – As you can see pre-checks are successfully completed.
13. Click on APPLY CHANGES.

Capture 7: –

Note: – when you click on apply changes that page redirect to remediate page.
14. Click on Next.

Capture 8: –

15. Click on Next.

Capture 9: –

16. Click on Remediate.

Capture 10: –

17. As we can see the ESX-01d is in the maintenance mode and ready to reboot. In this stage configuration are applied and execute once the ESX-01d come from exit maintenance mode. The same process will apply for other hosts in the cluster.

Capture 11: –

18. As you can see the remediation completed successfully.

Post Validation

Host level Memory Tiering: –

Capture 1: –
ESX-01d: –

1. Click on Configure.
2. Click on Overview under hardware.
   Note : – As we can see the Total memory is increased from 96 GB to 192GB and Memory tiering is software based.
   
   ESX-02d: –
   

3. Click on Configure.
4. Click on Overview under hardware.
Note : – As we can see the Total memory is increased from 96 GB to 192GB and Memory tiering is software based.

ESX-03d: –

5. Click on Configure.
6. Click on Overview under hardware.
Note : – As we can see the Total memory is increased from 96 GB to 192GB and Memory tiering is software based.

Cluster level Memory Tiering

Capture 2: –

7. Click on Cluster.
8. Click on summary the total memory is increased from 288GB to 575.99GB.

Capture 3: –

9. Click on Hosts from the taskbar.
10. Click on Hosts.
Note: – As we can in the capture 3 Memory tiering is software based and NVMe device status is available.

Integration with vSphere Cluster Services

The true value of cluster-level Memory Tiering lies in its seamless integration with vSphere’s core distributed services.

vMotion Behavior

Consistency: Inconsistent Memory Tiering configurations or capacity between source and destination hosts can lead to vMotion failures.
Pre-Requisites: For successful vMotion of a VM leveraging tiered memory, the destination host must also have Memory Tiering enabled and possess sufficient available Tier 1 (NVMe) and Tier 0 (DRAM) capacity to accommodate the migrating VM.
Migration Process: During a live vMotion, the hypervisor intelligently handles the transfer of memory pages. Active (hot) pages residing in DRAM are migrated first. Inactive (warm) pages residing in the NVMe tier are transferred as required, potentially impacting vMotion stun time and network bandwidth if a substantial amount of Tier 1 data needs to be moved. The vMotion process is designed to minimize performance impact to the running VM.

Distributed Resource Scheduler (DRS) Awareness

Performance vs. Consolidation: DRS algorithms balance the desire for higher consolidation enabled by tiered memory against potential performance degradation if VMs are excessively reliant on slower Tier 1 memory.
Resource Metrics: DRS is a resource management mechanism that balances compute workloads across a cluster. With Memory Tiering, DRS incorporates both Tier 0 (DRAM) and Tier 1 (NVMe) capacities and utilization into its resource metrics.
Initial Placement: When powering on a new virtual machine or during a manual placement, DRS considers the total effective memory capacity (DRAM + NVMe) available on each host, aiming to place the VM on the host that best meets its memory demands while maintaining cluster balance
Load Balancing & Migration: DRS proactively recommends and executes VM migrations (or generates recommendations in manual mode) to optimize resource utilization across the cluster. These recommendations consider the performance characteristics of both memory tiers, striving to keep active memory in DRAM and to balance the usage of Tier 1 NVMe resources.

High Availability (HA)

VM Restart: In the event of an ESXi host failure, vSphere HA automates the restart of affected virtual machines on other available hosts within the cluster.
Resource Allocation: When selecting a restart host, HA considers the combined available memory resources (DRAM + NVMe) of potential hosts to ensure that the restarted VM can be accommodated. This maintains service continuity by quickly bringing VMs back online on a host with adequate resources.

Operational Monitoring and Capacity Planning

Effective management of Memory Tiering in a clustered environment requires proactive monitoring and strategic capacity planning.

Performance Metrics & vCenter Charts

• Memory Tier Utilization: Monitor the utilization of both Tier 0 (DRAM) and Tier 1 (NVMe) at the VM, host, and cluster levels. High NVMe utilization, especially when DRAM is not fully consumed, might indicate inefficient memory access patterns or suboptimal VM memory allocation.
• Page Migration Rates: Track metrics indicating the frequency of memory page movements between Tier 0 and Tier 1.
  • Tier1_Page_Ins/sec: Pages being moved from NVMe to DRAM (read operations).
  • Tier1_Page_Outs/sec: Pages being moved from DRAM to NVMe (write operations).
  • High Page_Ins coupled with high latency could signify that too many active pages are being pushed to NVMe, potentially impacting VM performance.
• NVMe Device Performance: Monitor standard NVMe device metrics (IOPS, throughput, latency) for the dedicated Memory Tiering devices. Elevated latency on the NVMe device could directly impact VMs relying on Tier 1 memory.

Capacity Planning Considerations

• DRAM-Dependent NVMe Sizing: Tier 0 (DRAM) serves as the primary resource; Tier 1 (NVMe) augments.The optimal Tier 1 (NVMe) capacity is correlated with the host’s physical DRAM size. VMware provides specific guidelines to ensure a balanced performance profile, as detailed below:|

DRAM:NVMe in Ratio	NVMe Device Partition Size	DRAM Size in GB	NVMe Used in GB
1:1 (default)	1TB	512	512
1:1	256GB	512	256
4:1	1TB	512	128
4:1	32GB	256	32

Workload Analysis:

• VDI: VDI environments often exhibit good tiering candidates due to user inactivity periods, leading to many “warm” pages.
• Databases/In-Memory Caches: These typically require maximum DRAM and should only utilize Tier 1 for truly inactive segments or if explicitly designed for such tiered storage.
• Memory Oversubscription: Memory Tiering allows for higher memory oversubscription ratios than with DRAM-only hosts, but this must be carefully monitored. Excessive oversubscription, even with Tier 1, can lead to performance bottlenecks if the NVMe tier itself becomes a bottleneck.

NVMe Device Health and Lifecycle

• Write Endurance (TBW/DWPD): Monitor the write endurance (Terabytes Written / Drive Writes Per Day) of the dedicated NVMe devices. While memory tiering is designed to be efficient, constant page-outs to NVMe contribute to wear.
• Failure Domain: Understand the impact of a dedicated NVMe device failure. While not directly storing VM data, its failure will disable Memory Tiering on that host, potentially impacting VM performance or causing memory exhaustion. Redundancy (e.g., via host hardware RAID 1 if supported and not interfering with dedicated use) should be considered where feasible.

Conclusion

In this segment, we explored the procedures for deploying vSphere Memory Tiering across a cluster, focusing on interactive setup through the vCenter GUI. We examined the critical interactions between Memory Tiering and fundamental vSphere cluster services—vMotion, DRS, and HA—demonstrating how these services adapt to and leverage the tiered memory architecture for enhanced VM mobility and resource management. Best practices for monitoring and operational management in a clustered environment were also discussed. Building upon these cluster-wide strategies, Part 3: Deploying vSphere Memory Tiering: Considerations with vSAN Environments will address the specific architectural and deployment considerations when integrating Memory Tiering over NVMe within a vSAN-enabled cluster, focusing on the unique challenges and planning required when sharing NVMe resources in such an environment.