L2TP/IPSec or L2TP over IPSec

L2TP encapsulates original packets inside a PPP frame (performing compression when possible) and inside a User Datagram Protocol (UDP)-type packet assigned to port 1701. Because the UDP packet format is an IP packet, L2TP automatically uses IPSec to secure the tunnel, in accord with the security settings in the user configuration of the L2TP tunnel. The IPSec Internet Key Exchange (IKE) protocol negotiates security for the L2TP tunnel; certificate-based authentication is the default. This authentication process uses computer certificates, not user certificates, to verify that the source and destination computers both trust each other. If IPSec transport security is successfully established, L2TP negotiates the tunnel (including compression and user authentication options) and performs access control that is based on the user identity.

The L2TP/IPSec packet structure looks like the following example. The PPP Payload contains the original IP datagram, and the italicized text represents what is encrypted with IPSec.
IP headerIPSec ESP headerUDP headerL2TP headerPPP headerPPP PayloadIPSec ESP trailerIPSec Auth trailer

Microsoft Point-to-Point Encryption Protocol (MPPE), which can be used to secure the PPP payload when the Extensible Authentication Protocol Transport Layer Security (EAP-TLS) or Microsoft Challenge-Handshake Authentication Protocol (MS-CHAP) is used, is negotiated by Windows when the L2TP peer (client or server) requests it.

MPPE uses the Rivest-Shamir-Adleman (RSA) RC4 stream encryption and either 40-bit, 56-bit, or 128-bit secret keys. MPPE keys are generated from the MS-CHAP and EAP-TLS user-authentication process. The remote access server can be configured to require data encryption. If the remote access client cannot perform the required encryption, the connection attempt is rejected and the following error message (#742) appears:
The remote computer does not support the required data encryption type.

IPSEC is negotiated before PPP starts; MPPE is negotiated after PPP starts. PPP runs over L2TP using IPSec. During the PPP authentication phase, a user name is sent to the Remote Access Server (RAS) component of the virtual private network (VPN) server by using the configured authentication protocol (MS-CHAP, for example). The RAS server then matches the user name and other call properties to a Remote Access Policy. Each policy has a profile, and RAS compares the conditions of the incoming call with the profile to determine whether to accept the connection request.

(Note: Basic L2TP/IPSec troubleshooting in windows XP )