We have two sides of an environment where we statically NAT ranges of private to public IPs and/or vice versa. On one side of this, we leverage a vSRX (on 15.1X49-D110.4), in which this traffic only lives in the global routing instance.
On the backup link, we have IPSec terminating into a RI (MNO) on an SRX240 (on 12.3X48-D65.1), which then passes traffic into the global RI.
With either path, if I generate traffic from the public to the private, NAT functions as expected. I see an approproate session and translation created within either SRX and away we go.
If I generate traffic from the private to public, I see the vSRX create a reverse NAT as expected, however, the SRX does not.
We route between these RIs on the SRX240 via lt interfaces, and the (truncated) NAT policy is as follows:
set security nat static rule-set MNO_NAT from zone INSIDE
set security nat static rule-set MNO_NAT rule 3_TEST2 match destination-address x.x.79.249/32
set security nat static rule-set MNO_NAT rule 3_TEST2 then static-nat prefix 10.59.15.254/32
set security nat static rule-set MNO_NAT rule 3_TEST2 then static-nat prefix routing-instance MNO
The rule set on the vSRX is identical save the RI statement
My assumption is that it's a configuration issue, however, we did seem to have this working properly when we leveraged rib-groups instead of lt interfaces. The how and why we changed is another conversation for another day. And I'm near the point to revert back to using rib-groups.
While trying to troubleshoot this, I'd created a source NAT rule as per this link to no avail; I'd seen the same behavior.
Taking some traces, I see the following for the failed translation:
10:59:28.978800:CID-0:RT: <10.59.15.254/1->x.x.50.173/24285;1> matched filter TO_LIBRE:
10:59:28.978800:CID-0:RT: packet [84] ipid = 0, @0x43d532d0
10:59:28.978800:CID-0:RT: ---- flow_process_pkt: (thd 2): flow_ctxt type 1, common flag 0x0, mbuf 0x43d53080, rtbl_idx = 4
10:59:28.978800:CID-0:RT: flow process pak, mbuf 0x43d53080, ifl 71, ctxt_type 1 inq type 6
10:59:28.978800:CID-0:RT: in_ifp <MNO:st0.0>
10:59:28.978800:CID-0:RT: flow_process_pkt_exception: setting rtt in lpak to 0x64226648
10:59:28.978800:CID-0:RT: host inq check inq_type 0x6
10:59:28.978800:CID-0:RT: tifp st0.0
10:59:28.978800:CID-0:RT: pkt out of tunnel.Proceed normally
10:59:28.978800:CID-0:RT: st0.0:10.59.15.254->x.x.50.173, icmp, (8/0)
10:59:28.978800:CID-0:RT: find flow: table 0x526d79e0, hash 63187(0xffff), sa 10.59.15.254, da x.x.50.173, sp 1, dp 24285, proto 1, tok 16393
10:59:28.979003:CID-0:RT: no session found, start first path. in_tunnel - 0x56a76708, from_cp_flag - 0
10:59:28.979045:CID-0:RT: Not a traffic-selector enabled tunnel. returing EOK
10:59:28.979045:CID-0:RT: search gate for MNO:10.59.15.254/1->x.x.50.173/24285,1
10:59:28.979045:CID-0:RT: gate_search_specific_bucket: no gate found
10:59:28.979121:CID-0:RT: search gate for MNO:10.59.15.254/1->x.x.50.173/24285,1
10:59:28.979121:CID-0:RT: gate_search_specific_bucket: no gate found
10:59:28.979121:CID-0:RT: search widecast gate for MNO:10.59.15.254/1->x.x.50.173/24285,1
10:59:28.979121:CID-0:RT: gate_search_widecast_bucket: no gate found
10:59:28.979121:CID-0:RT: flow_first_create_session
10:59:28.979121:CID-0:RT: First path alloc and instl pending session, natp=0x5b695c78, id=187634
10:59:28.979121:CID-0:RT: flow_first_in_dst_nat: in <st0.0>, out <N/A> dst_adr x.x.50.173, sp 1, dp 24285
10:59:28.979121:CID-0:RT: chose interface st0.0 as incoming nat if.
10:59:28.979121:CID-0:RT: flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to x.x.50.173(24285)
10:59:28.979121:CID-0:RT: flow_first_routing: vr_id 4, call flow_route_lookup(): src_ip 10.59.15.254, x_dst_ip x.x.50.173, in ifp st0.0, out ifp N/A sp 1, dp 24285, ip_proto 1, tos 68
10:59:28.979121:CID-0:RT: Doing DESTINATION addr route-lookup
10:59:28.979121:CID-0:RT: flow_ipv4_rt_lkup success x.x.50.173, iifl 0x47, oifl 0x4e
10:59:28.979356:CID-0:RT: routed (x_dst_ip x.x.50.173) from MNO (st0.0 in 0) to lt-0/0/0.345, Next-hop: x.x.50.52
10:59:28.979356:CID-0:RT: flow_first_policy_search: policy search from zone MNO-> zone MNO (0x0,0x15edd,0x5edd)
10:59:28.979400:CID-0:RT: Policy lkup: vsys 0 zone(9:MNO) -> zone(9:MNO) scope:0
10:59:28.979400:CID-0:RT: 10.59.15.254/2048 -> x.x.50.173/5478 proto 1
10:59:28.979400:CID-0:RT: app 0, timeout 60s, curr ageout 60s
10:59:28.979400:CID-0:RT: permitted by policy ALLOW_ALL_MNO(10)
10:59:28.979400:CID-0:RT: packet passed, Permitted by policy.
10:59:28.979400:CID-0:RT: flow_first_src_xlate: nat_src_xlated: False, nat_src_xlate_failed: False
10:59:28.979400:CID-0:RT: flow_first_src_xlate: incoming src port is : 1.
10:59:28.979400:CID-0:RT: flow_first_src_xlate: src nat returns status: 0, rule/pool id: 0/0, pst_nat: False.
10:59:28.979400:CID-0:RT: dip id = 0/0, 10.59.15.254/1->10.59.15.254/1 protocol 0
10:59:28.979400:CID-0:RT: choose interface lt-0/0/0.345(P2P) as outgoing phy if
10:59:28.979400:CID-0:RT: is_loop_pak: No loop: on ifp: lt-0/0/0.345, addr: x.x.50.173, rtt_idx:4
10:59:28.979614:CID-0:RT: -jsf : Alloc sess plugin info for session 704374824178
10:59:28.979614:CID-0:RT: [JSF]Normal interest check. regd plugins 28, enabled impl mask 0x0
10:59:28.979702:CID-0:RT: +++++++++++jsf_test_plugin_data_evh: 3
10:59:28.979702:CID-0:RT: [JSF]Plugins(0x0, count 0) enabled for session = 704374824178, impli mask(0x0), post_nat cnt 0 svc req(0x0)
10:59:28.979755:CID-0:RT: -jsf : no plugin interested for session 704374824178, free sess plugin info
10:59:28.979755:CID-0:RT: flow_first_service_lookup(): natp(0x5b695c78): app_id, 0(0).
10:59:28.979755:CID-0:RT: service lookup identified service 0.
10:59:28.979755:CID-0:RT: flow_first_final_check: in <st0.0>, out <lt-0/0/0.345>
10:59:28.979755:CID-0:RT: In flow_first_complete_session
10:59:28.979755:CID-0:RT: flow_first_complete_session, pak_ptr: 0x52028c50, nsp: 0x5b695c78, in_tunnel: 0x56a76708
10:59:28.979755:CID-0:RT: construct v4 vector for nsp2
10:59:28.979755:CID-0:RT: existing vector list 0x204-0x4b521b50.
10:59:28.979755:CID-0:RT: Session (id:187634) created for first pak 204
10:59:28.979755:CID-0:RT: first pak processing successful
10:59:28.979755:CID-0:RT: flow_first_install_session======> 0x5b695c78
10:59:28.979755:CID-0:RT: nsp 0x5b695c78, nsp2 0x5b695d08
10:59:28.979755:CID-0:RT: make_nsp_ready_no_resolve()
10:59:28.979755:CID-0:RT: flow_ipv4_rt_lkup success 10.59.15.254, iifl 0x47, oifl 0x47
10:59:28.979755:CID-0:RT: route lookup: dest-ip 10.59.15.254 orig ifp st0.0 output_ifp st0.0 orig-zone 9 out-zone 9 vsd 0
10:59:28.979755:CID-0:RT: route to 10.59.15.254
10:59:28.979755:CID-0:RT: no need update ha
10:59:28.979755:CID-0:RT: Installing s2c NP session wing
10:59:28.980062:CID-0:RT: first path session installation succeeded
10:59:28.980062:CID-0:RT: flow got session.
10:59:28.980062:CID-0:RT: flow session id 187634
10:59:28.980062:CID-0:RT: vector bits 0x204 vector 0x4b521b50
10:59:28.980106:CID-0:RT: skip pre-frag: is_tunnel_if- 0, is_if_mtu_configured- 0
10:59:28.980106:CID-0:RT: encap vector
10:59:28.980106:CID-0:RT: no more encapping needed
10:59:28.980106:CID-0:RT: mbuf 0x43d53080, exit nh 0x120010
10:59:28.980106:CID-0:RT: flow_process_pkt_exception: Freeing lpak 0x52028c50 associated with mbuf 0x43d53080
10:59:28.980106:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)And the good side form the vSRX
13:49:59.572864:CID-0:THREAD_ID-01:RT: <10.255.49.73/1->x.x.49.170/31499;1,0x0> matched filter TO_LIBRE:
13:49:59.572868:CID-0:THREAD_ID-01:RT: packet [84] ipid = 0, @0x19e25ed2
13:49:59.572875:CID-0:THREAD_ID-01:RT: ---- flow_process_pkt: (thd 1): flow_ctxt type 15, common flag 0x0, mbuf 0x68e9d400, rtbl_idx = 0
13:49:59.572876:CID-0:THREAD_ID-01:RT: flow process pak fast ifl 76 in_ifp ge-0/0/0.345
13:49:59.572877:CID-0:THREAD_ID-01:RT: ge-0/0/0.345:10.255.49.73->x.x.49.170, icmp, (8/0)
13:49:59.572879:CID-0:THREAD_ID-01:RT: find flow: table 0x28812f40, hash 29145(0xffff), sa 10.255.49.73, da x.x.49.170, sp 1, dp 31499, proto 1, tok 9, conn-tag 0x00000000
13:49:59.572884:CID-0:THREAD_ID-01:RT: no session found, start first path. in_tunnel - 0x0, from_cp_flag - 0
13:49:59.572885:CID-0:THREAD_ID-01:RT: search gate for MNO-MVNO:10.255.49.73/1->x.x.49.170/31499,1
13:49:59.572887:CID-0:THREAD_ID-01:RT: gate_search_specific_bucket: no gate found
13:49:59.572888:CID-0:THREAD_ID-01:RT: search gate for MNO-MVNO:10.255.49.73/1->x.x.49.170/31499,1
13:49:59.572889:CID-0:THREAD_ID-01:RT: gate_search_specific_bucket: no gate found
13:49:59.572889:CID-0:THREAD_ID-01:RT: search widecast gate for MNO-MVNO:10.255.49.73/1->x.x.49.170/31499,1
13:49:59.572890:CID-0:THREAD_ID-01:RT: gate_search_widecast_bucket: no gate found
13:49:59.572891:CID-0:THREAD_ID-01:RT: flow_first_create_session
13:49:59.572894:CID-0:THREAD_ID-01:RT: Save init hash spu id 0 to nsp and nsp2!
13:49:59.572895:CID-0:THREAD_ID-01:RT: First path alloc and instl pending session, natp=0x2f7262c0, id=133393
13:49:59.572896:CID-0:THREAD_ID-01:RT: flow_first_in_dst_nat: in <ge-0/0/0.345>, out <N/A> dst_adr x.x.49.170, sp 1, dp 31499
13:49:59.572898:CID-0:THREAD_ID-01:RT: chose interface ge-0/0/0.345 as incoming nat if.
13:49:59.572900:CID-0:THREAD_ID-01:RT: flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to x.x.49.170(31499)
13:49:59.572901:CID-0:THREAD_ID-01:RT: [JSF] Do ingress interest check. regd ingress plugins(1)
13:49:59.572903:CID-0:THREAD_ID-01:RT: [JSF][0]plugins(0x0) enabled for session = 261993138449 implicit mask(0x0), service request(0x0)
13:49:59.572904:CID-0:THREAD_ID-01:RT: -jsf : no plugin ingress interested for session 261993138449
13:49:59.572905:CID-0:THREAD_ID-01:RT: flow_first_routing: vr_id 0, call flow_route_lookup(): src_ip 10.255.49.73, x_dst_ip x.x.49.170, in ifp ge-0/0/0.345, out ifp N/A sp 1, dp 31499, ip_proto 1, tos 38
13:49:59.572907:CID-0:THREAD_ID-01:RT: Doing DESTINATION addr route-lookup
13:49:59.572910:CID-0:THREAD_ID-01:RT: flow_ipv4_rt_lkup success x.x.49.170, iifl 0x4c, oifl 0x48
13:49:59.572911:CID-0:THREAD_ID-01:RT: routed (x_dst_ip x.x.49.170) from MNO-MVNO (ge-0/0/0.345 in 0) to ge-0/0/0.0, Next-hop: 185.18.51.250
13:49:59.572916:CID-0:THREAD_ID-01:RT: flow_first_policy_search: policy search from zone MNO-MVNO-> zone untrust (0x0,0x17b0b,0x7b0b)
13:49:59.572919:CID-0:THREAD_ID-01:RT: Policy lkup: vsys 0 zone(9:MNO-MVNO) -> zone(7:untrust) scope:0
13:49:59.572922:CID-0:THREAD_ID-01:RT: 10.255.49.73/2048 -> x.x.49.170/44421 proto 1
13:49:59.572926:CID-0:THREAD_ID-01:RT: app 0, timeout 60s, curr ageout 60s
13:49:59.572927:CID-0:THREAD_ID-01:RT: permitted by policy permit-all(6)
13:49:59.572930:CID-0:THREAD_ID-01:RT: packet passed, Permitted by policy.
13:49:59.572932:CID-0:THREAD_ID-01:RT: reverse mip xlate 10.255.49.73/1 -> x.x.161.73/1 (on ge-0/0/0.0)
13:49:59.572933:CID-0:THREAD_ID-01:RT: flow_first_src_xlate: nat_src_xlated: True, nat_src_xlate_failed: False
13:49:59.572936:CID-0:THREAD_ID-01:RT: flow_first_src_xlate: hip xlate: 10.255.49.73->x.x.161.73 at ge-0/0/0.0 (vs. ge-0/0/0.0)
13:49:59.572937:CID-0:THREAD_ID-01:RT: dip id = 0/0, 10.255.49.73/1->x.x.161.73/1 protocol 0
13:49:59.572939:CID-0:THREAD_ID-01:RT: choose interface ge-0/0/0.0(P2P) as outgoing phy if
13:49:59.572942:CID-0:THREAD_ID-01:RT: is_loop_pak: No loop: on ifp: ge-0/0/0.0, addr: x.x.49.170, rtt_idx:0
13:49:59.572945:CID-0:THREAD_ID-01:RT: [JSF]Normal interest check. regd plugins 35, enabled impl mask 0x0
13:49:59.572947:CID-0:THREAD_ID-01:RT: get NULL sess plugin info 0x2f7262c0
13:49:59.572951:CID-0:THREAD_ID-01:RT: get NULL sess plugin info 0x2f7262c0
13:49:59.572953:CID-0:THREAD_ID-01:RT: get NULL sess plugin info 0x2f7262c0
13:49:59.572954:CID-0:THREAD_ID-01:RT: get NULL sess plugin info 0x2f7262c0
13:49:59.572955:CID-0:THREAD_ID-01:RT: +++++++++++jsf_test_plugin_data_evh: 3
13:49:59.572958:CID-0:THREAD_ID-01:RT: get NULL sess plugin info 0x2f7262c0
13:49:59.572958:CID-0:THREAD_ID-01:RT: get NULL sess plugin info 0x2f7262c0
13:49:59.572961:CID-0:THREAD_ID-01:RT: [JSF]Plugins(0x0, count 0) enabled for session = 261993138449, impli mask(0x0), post_nat cnt 0 svc req(0x0)
13:49:59.572963:CID-0:THREAD_ID-01:RT: -jsf : no plugin interested for session 261993138449, free sess plugin info
13:49:59.572964:CID-0:THREAD_ID-01:RT: flow_first_service_lookup(): natp(0x2f7262c0): app_id, 0(0).
13:49:59.572965:CID-0:THREAD_ID-01:RT: service lookup identified service 0.
13:49:59.572965:CID-0:THREAD_ID-01:RT: flow_first_final_check: in <ge-0/0/0.345>, out <ge-0/0/0.0>
13:49:59.572967:CID-0:THREAD_ID-01:RT: flow_first_final_check: flow_set_xlate_vector.
13:49:59.572968:CID-0:THREAD_ID-01:RT: In flow_first_complete_session
13:49:59.572968:CID-0:THREAD_ID-01:RT: flow_first_complete_session: pak_ptr is xlated packet
13:49:59.572969:CID-0:THREAD_ID-01:RT: flow_first_complete_session, pak_ptr: 0x5cdfcd50, nsp: 0x2f7262c0, in_tunnel: 0x0
13:49:59.572970:CID-0:THREAD_ID-01:RT: construct v4 vector for nsp2 and nsp
13:49:59.572971:CID-0:THREAD_ID-01:RT: existing vector list 0x1200-0x759e2190.
13:49:59.572971:CID-0:THREAD_ID-01:RT: existing vector list 0x1200-0x759e2190.
13:49:59.572972:CID-0:THREAD_ID-01:RT: Session (id:133393) created for first pak 1200
13:49:59.572972:CID-0:THREAD_ID-01:RT: first pak processing successful
13:49:59.572973:CID-0:THREAD_ID-01:RT: flow_first_install_session======> 0x2f7262c0
13:49:59.572973:CID-0:THREAD_ID-01:RT: nsp 0x2f7262c0, nsp2 0x2f726380
13:49:59.572974:CID-0:THREAD_ID-01:RT: make_nsp_ready_no_resolve()
13:49:59.572978:CID-0:THREAD_ID-01:RT: flow_ipv4_rt_lkup success 10.255.49.73, iifl 0x4c, oifl 0x4c
13:49:59.572980:CID-0:THREAD_ID-01:RT: route lookup: dest-ip 10.255.49.73 orig ifp ge-0/0/0.345 output_ifp ge-0/0/0.345 orig-zone 9 out-zone 9 vsd 0
13:49:59.572981:CID-0:THREAD_ID-01:RT: route to 10.249.1.1
13:49:59.572983:CID-0:THREAD_ID-01:RT: no need update ha
13:49:59.572983:CID-0:THREAD_ID-01:RT: Installing c2s NP session wing
13:49:59.572984:CID-0:THREAD_ID-01:RT: Installing s2c NP session wing
13:49:59.572985:CID-0:THREAD_ID-01:RT: first path session installation succeeded
13:49:59.572986:CID-0:THREAD_ID-01:RT: flow got session.
13:49:59.572986:CID-0:THREAD_ID-01:RT: flow session id 133393
13:49:59.572987:CID-0:THREAD_ID-01:RT: vector bits 0x1200 vector 0x759e2190
13:49:59.572988:CID-0:THREAD_ID-01:RT: flow_xlate_pak
13:49:59.572989:CID-0:THREAD_ID-01:RT: flow_handle_icmp_xlate
13:49:59.572989:CID-0:THREAD_ID-01:RT: xlate_icmp_pak
13:49:59.572994:CID-0:THREAD_ID-01:RT: post addr xlation: x.x.161.73->x.x.49.170.
13:49:59.572995:CID-0:THREAD_ID-01:RT: post addr xlation: x.x.161.73->x.x.49.170.
13:49:59.572996:CID-0:THREAD_ID-01:RT: skip pre-frag: is_tunnel_if- 0, is_if_mtu_configured- 0
13:49:59.572999:CID-0:THREAD_ID-01:RT: mbuf 0x68e9d400, exit nh 0xd0010
13:49:59.572999:CID-0:THREAD_ID-01:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)So I'm at a loss as to why this is occuring. I've attached a sanitized config that's relevant for this setup.
I'm certainly open to suggestions. I do have a JTAC case open, but it's not going to be followed up on until Monday.