Which protocol runs between the vSmart controllers and WAN Edge routers when the vSmart controller acts like a route reflector?

IPsec is used for data plane encryption between WAN Edge peers and does not run between vSmart and WAN Edge routers for control plane route exchange.

BGP does not run between vSmart controllers and WAN Edge routers; OMP is the protocol used in this role, even though vSmart behaves like a route reflector conceptually.

OMP does not run outside the DTLS/TLS connection; all SD-WAN control plane traffic between vSmart and WAN Edge is encapsulated within the secure DTLS or TLS session.

OMP is the proprietary Cisco SD-WAN routing protocol that operates between vSmart and WAN Edge devices. It runs encapsulated inside the DTLS or TLS control plane tunnel that is established during device onboarding. The vSmart controller functions analogously to a BGP route reflector by receiving OMP routes from all WAN Edge clients and reflecting them to all other WAN Edge routers.