Which IPsec mode must be used when encrypting data over a public network between two servers with RFC1918 IP addresses?

Main mode is an IKE (Internet Key Exchange) negotiation mode used to establish the security association (SA) for IPsec, not an IPsec *encryption* mode itself.

Aggressive mode is an IKE negotiation mode, a faster alternative to main mode, but it is also not an IPsec *encryption* mode.

IPsec transport mode encrypts only the payload of the IP packet and maintains the original IP header. It is typically used for host-to-host or end-to-end encryption where the hosts are directly communicating or reachable via public IP addresses, and it does not hide the original source/destination IP addresses, making it unsuitable for traversing a public network with RFC1918 internal addresses.

IPsec tunnel mode encapsulates the entire original IP packet (including its original IP header, which contains the RFC1918 source and destination addresses) within a new IP packet with new public IP headers. This is essential when traversing a public network like the internet, especially when the internal servers use private RFC1918 addresses that are not routable on the public internet.