Hello,
There is a report of connection errors on DB server when we see the following logs.
This DB communication is over a site to site VPN between two sets of clustered SRX 340s in different DCs.
(IPs are redacted with Xs and Ys.)
// SRX 1
May 28 05:09:09 SRX1 kmd[1900]: IPSec negotiation failed with error: No proposal chosen. IKE Version: 2, VPN: Not-Available Gateway: Not-Available, Local: X.X.X.X/500, Remote: Y.Y.Y.Y/500, Local IKE-ID: X.X.X.X, Remote IKE-ID: Y.Y.Y.Y, VR-ID: 0
May 28 05:09:17 SRX1 kmd[1900]: KMD_VPN_DOWN_ALARM_USER: VPN VPN-Y.Y.Y.Y from Y.Y.Y.Y is down. Local-ip: X.X.X.X, gateway name: GW-VPN-59_167_65_9, vpn name: VPN-Y.Y.Y.Y, tunnel-id: 131087, local tunnel-if: st0.17, remote tunnel-ip: Not-Available, Local IKE-ID: X.X.X.X, Remote IKE-ID: Y.Y.Y.Y, AAA username: Not-Applicable, VR id: 0, Traffic-selector: , Traffic-selector local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Traffic-selector remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), SA Type: Static, Reason: IPSec SAs cleared as corresponding IKE SA deleted
May 28 05:09:17 SRX1 kmd[1900]: IPSec negotiation failed with error: Timed out. IKE Version: 2, VPN: VPN-Y.Y.Y.Y Gateway: GW-VPN-59_167_65_9, Local: X.X.X.X/500, Remote: Y.Y.Y.Y/500, Local IKE-ID: X.X.X.X, Remote IKE-ID: Y.Y.Y.Y, VR-ID: 0
May 28 05:09:27 SRX1 kmd[1900]: KMD_PM_SA_ESTABLISHED: Local gateway: X.X.X.X, Remote gateway: Y.Y.Y.Y, Local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Direction: inbound, SPI: 0x4fc3f15d, AUX-SPI: 0, Mode: Tunnel, Type: dynamic, Traffic-selector:
May 28 05:09:27 SRX1 kmd[1900]: KMD_PM_SA_ESTABLISHED: Local gateway: X.X.X.X, Remote gateway: Y.Y.Y.Y, Local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Direction: outbound, SPI: 0xeaea89e0, AUX-SPI: 0, Mode: Tunnel, Type: dynamic, Traffic-selector:
May 28 05:09:27 SRX1 kmd[1900]: KMD_VPN_UP_ALARM_USER: VPN VPN-Y.Y.Y.Y from Y.Y.Y.Y is up. Local-ip: X.X.X.X, gateway name: GW-VPN-59_167_65_9, vpn name: VPN-Y.Y.Y.Y, tunnel-id: 131087, local tunnel-if: st0.17, remote tunnel-ip: Not-Available, Local IKE-ID: X.X.X.X, Remote IKE-ID: Y.Y.Y.Y, AAA username: Not-Applicable, VR id: 0, Traffic-selector: , Traffic-selector local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Traffic-selector remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), SA Type: Static
May 28 05:09:27 SRX1 kmd[1900]: IKE negotiation successfully completed. IKE Version: 2, VPN: VPN-Y.Y.Y.Y Gateway: GW-VPN-59_167_65_9, Local: X.X.X.X/500, Remote: Y.Y.Y.Y/500, Local IKE-ID: X.X.X.X, Remote IKE-ID: Y.Y.Y.Y, VR-ID: 0, Role: Initiator
%
// SRX 2
May 28 05:09:09 SRX2 kmd[4430]: IPSec negotiation failed with error: No proposal chosen. IKE Version: 2, VPN: VPN-X.X.X.X Gateway: GW-VPN-59_167_18_97, Local: Y.Y.Y.Y/500, Remote: X.X.X.X/500, Local IKE-ID: Y.Y.Y.Y, Remote IKE-ID: X.X.X.X, VR-ID: 0
May 28 05:09:27 SRX2 kmd[4430]: KMD_PM_SA_ESTABLISHED: Local gateway: Y.Y.Y.Y, Remote gateway: X.X.X.X, Local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Direction: inbound, SPI: 0xeaea89e0, AUX-SPI: 0, Mode: Tunnel, Type: dynamic, Traffic-selector:
May 28 05:09:27 SRX2 kmd[4430]: KMD_PM_SA_ESTABLISHED: Local gateway: Y.Y.Y.Y, Remote gateway: X.X.X.X, Local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Direction: outbound, SPI: 0x4fc3f15d, AUX-SPI: 0, Mode: Tunnel, Type: dynamic, Traffic-selector:
May 28 05:09:27 SRX2 kmd[4430]: IKE negotiation successfully completed. IKE Version: 2, VPN: VPN-X.X.X.X Gateway: GW-VPN-59_167_18_97, Local: Y.Y.Y.Y/500, Remote: X.X.X.X/500, Local IKE-ID: Y.Y.Y.Y, Remote IKE-ID: X.X.X.X, VR-ID: 0, Role: Responder
%
Tunnel is not 'failing' per se, it always stays up this is just a random, short-lived issue (e.g. 10 seconds in the example above). However, this is enough to cause the DB to log connection errors.
I see that SRX1 marks the VPN as down at May 28 05:09:17 and back up at May 28 05:09:27
SRX2 isn't aware of SRX1 deleting the IKE SA, but at May 28 05:09:27 when it logs 'IKE negotiation successfully completed'.
I've run request security ike debug-enable for the peers at the same time the issue happens, but nothing is seen in the kmd file.
ike traceoptions capture information for all tunnels and as this issue happens at random times aren't very suitable.
Please can anyone help?
Many thanks