Introduction

In VMware Cloud Foundation (VCF) 9.1, certificate management underwent a major architectural evolution. Control shifted away from standalone SDDC Manager user interface workflows into centralized fleet management orchestrated by VCF Operations.
Understanding component-to-component TLS handshakes, identity trust models, and certificate lifecycle flows across management and workload domains is critical for designing resilient private clouds.

The Core Infrastructure: Dual-Scope CA Architecture & Prerequisites

VCF 9.1 splits Certificate Authority (CA) orchestration into two distinct operational scopes. Understanding this boundary prevents truststore mismatches during deployment.

VCF Management Scope

• Covered Services: Fleet Management (VCF Operations UI), Identity Broker, VCF Automation, and VCF Operations for Logs.
• PKI Requirement: Strictly mandates an external Enterprise CA (such as Microsoft Active Directory Certificate Services) integrated via Microsoft CA Web Enrollment REST APIs.
• Function: Anchors the root of trust for management plane operations and Single Sign-On (SSO) token signing (OAuth2/OIDC).
  
  Note :- Prerequisite IIS Configuration Warning: On the Microsoft CA server, the Certification Authority Web Enrollment role must be enabled, and Basic Authentication must be explicitly turned on in IIS for the /certsrv virtual directory. The IIS site must also be bound to a valid, trusted HTTPS server certificate; un-trusted or self-signed certificates on the web enrollment endpoint will cause VCF Operations TLS handshakes to fail.

VCF Instance Scope

• Covered Services: SDDC Manager, vCenter Servers, NSX Managers, and ESXi hosts across all domains.
• PKI Requirement: Flexible support for Microsoft Enterprise CA or OpenSSL/embedded CA fallback.
• Function: Secures internal API paths, management daemons, and control plane communication channels.
  
  Note :–CA Template Configuration Rules:
  1. Subject Name Settings: The custom Microsoft CA template must be explicitly configured with “Supply in the request” enabled under the Subject Name tab so VCF Operations can inject mandatory FQDN and IP SAN values.
  2. Case Sensitivity: When specifying the template name in the VCF Operations CA configuration UI or API, the template name string is strictly case-sensitive.

End-to-End Certificate Flow Architecture

The following diagram maps the trust boundaries, inter-component communication channels, protocols, and port numbers across the VCF 9.1 stack.

Component Intercommunication & Protocol Matrix

Each component pair in the VCF stack relies on specific authentication mechanisms, protocols, and ports:

• VCF Operations <–> SDDC Manager & vCenter: Uses REST APIs over Port 443. VCF Operations validates the endpoint’s Machine SSL certificate against its trusted CA store before establishing the management session.
• vCenter Server <–> ESXi Hosts: Uses Ports 443 and 902 via the vpxa management agent. vCenter acts as a local Sub-CA through VMware Certificate Authority (VMCA), pushing custom or VMCA-signed leaf certificates down to ESXi host daemons.
• SDDC Manager <–> NSX Manager: Uses REST APIs over Port 443 utilizing Principal Identity (PI) certificates. These PI certificates authenticate SDDC Manager commands without requiring administrative password exchanges.
• NSX Manager <–> NSX Edges & vDefend: Uses Port 1234 over TLS for control plane communication, enforcing distributed firewall (DFW) policies and packet inspection state synchronization.
• ESXi / vCenter <–> VCF Operations for Logs: Uses Port 6514 for encrypted log transport over TLS Syslog.

Multi-Domain Trust Boundaries

Certificate orchestration behaves differently across domain structures:

• Management Domain: Houses core control appliances (VCF Operations, SDDC Manager, Identity Broker). Certs here anchor the primary administrative plane.
• VI Workload Domains (WLD): Compute and storage pools running application workloads maintain dedicated vCenter Servers and NSX Managers. VCF Operations centrally issues and rotates Machine SSL and PI certificates to these components over Port 443. The WLD vCenter subsequently handles VMCA host certificate distribution inside its cluster boundaries.
• Isolated Workload Domains: Designed for strict compliance environments (e.g., PCI-DSS). They maintain isolated intermediate CA sub-chains, ensuring that a compromised certificate in a workload domain cannot compromise management plane telemetry.

The 60-Day Automated Renewal Workflow & SAN Enforcement

VCF 9.1 features native certificate auto-renewal. When an environment reaches 60 days prior to certificate expiration, VCF Operations automatically triggers the lifecycle workflow.

Strict SAN Mandate

(Broadcom KB 424807)

During CSR generation, Broadcom rules require that the Subject Alternative Name (SAN) field explicitly contains both:

1. The full FQDN of the appliance (e.g., vcenter01.corp.xxx).
2. The primary management IP Address (e.g., 192.168.10.15).

If a Microsoft CA template is configured to drop IP addresses or build SAN fields automatically from Active Directory, Fleet Management lifecycle validation will fail.

Operational Execution Constraints

• Private Key Locality (Steps 1–2): Private keys are generated strictly on the target appliance (stored in local /etc/ssl/ or JKS keyrings) during CSR creation. Only the public CSR payload transits over the network; private keys never leave the endpoint.
• Serial Thread Locking (Steps 3–5): VCF Operations processes fleet renewals sequentially per workload domain. Triggering parallel renewal requests across multiple instances simultaneously locks the underlying API thread pool and causes task timeouts.
• Self-Healing Rollback (Steps 6–7): If the target appliance fails its internal post-rotation TLS handshake (Step 6) after restarting management services (like vpxd or proton), it automatically reverts to its cached truststore before returning an error status to VCF Operations.

Programmatic Certificate Management (REST API)

Platform engineers automating infrastructure deployments can interact with Fleet Management using native REST API endpoints in VCF Operations:

1. Generate CSR: POST https://<vcf-ops-fqdn>/suite-api/api/fleet-management/certificate-management/csrs
2. Submit Certificate & Install: POST https://<vcf-ops-fqdn>/suite-api/api/fleet-management/certificate-management/certificates/install
3. Track Workflow Progress: GET https://<vcf-ops-fqdn>/suite-api/api/workflows/requests/{requestId}

Operational Troubleshooting & Break-Glass Diagnostics

When certificate handshakes fail, isolated components can cause SDDC Manager or VCF Operations workflows to stall.

Key Log File Locations

1. VCF Operations Fleet Management: /var/log/vmware/vcf/fleet/fleet-management.log
2. SDDC Manager Commons Services: /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
3. vCenter Server VPXD Log: /var/log/vmware/vpxd/vpxd.log
4. NSX Manager API Logs: /var/log/proton/nsxapi.log

Useful Diagnostic CLI Commands

Inspect an endpoint’s active TLS certificate and full trust chain directly from the CLI:

Bash

openssl s_client -connect vcenter01.corp.local:443 -showcerts

Verify that the truststore on an SDDC Manager appliance contains the required Root and Intermediate CAs:

Bash

keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certs.jks -storepass changeit

Check an ESXi host’s local SSL certificate details:

Bash

openssl x509 -in /etc/vmware/ssl/rmi.crt -text -noout

Conclusion

Centralizing certificate lifecycle management within VCF Operations in version 9.1 resolves the fragmented administrative workflows of legacy releases. By enforcing strict SAN formatting, aligning dual-scope CA structures, and mapping exact port handshakes across management and workload domains, enterprise architects can build robust, security-compliant VCF cloud environments.

Reference

Broadcom TechDocs: VCF 9.1 Fleet Management & Certificate Administration — Official product documentation for VCF Operations certificate configuration, dual-scope boundaries, and central CA template binding.

Broadcom Support Knowledge Base (KB 424807) — Detailed requirements for FQDN and IP address Subject Alternative Name (SAN) enforcement during CSR generation.

The 60-Day Automated Renewal Workflow (KB 427937) – VCF Operations Fleet Management documentation) specifically for the 60-day cron trigger mechanism.

VMware Ports and Protocols Database — Official port matrix verifying TCP/UDP transport channels (Ports 443, 902, 1234, and 6514) across VCF management and workload domain components.

Broadcom TechDocs: VMware Cloud Foundation 9.1 Platform Architecture & Design Guide — Official design guidelines for VCF 9.1 platform topology, fleet management scaling, and control plane runtime boundaries..