In my last post, I went over the configuration of Proxmox, Vyos, and SROS. Here I want to show over how the setup looks and works with real VMs attached to this.
Above I have tried to show the topology. The VM in the bottom left is running on compute2 (Proxmox hypervisor), and attached to the VNET “Vlan31” which has the IP of 172.16.0.69. This VM’s gateway exists on the pair of Vyos instances, which tie the VXLAN tunnel into a VRF. The other VM, 172.16.1.69, is attached to a normal Proxmox bridge/vlan that is routed on the 7210 within the VPRN service. So these VMs are just to show connectivity between these systems.
With the Vnet configured in Proxmox, it is possible to assign that to a VM’s interface. Which simply looks like this below:
The other VM, I’m simply using a traditional linux bridge, and a vlan.
The VM above is the 172.16.0.69 VM, and showing connectivity to the other VM on 1.69. Let’s dive in how it works.
First I will lay out the IPs and macs for reference.
| 172.16.0.69 (testclone15) | BC:24:11:F9:4D:AE |
| 172.16.1.69 (testclone16) | BC:24:11:9E:9A:41 |
First we can look at the EVPN type 2 route (mac address) for the first host.
A:KPTPE01# show router bgp routes evpn mac mac-address BC:24:11:F9:4D:AE
===============================================================================
BGP Router ID:10.11.0.2 AS:65000 Local AS:65000
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Ip Address
NextHop
Label1
-------------------------------------------------------------------------------
*>i 192.168.254.18:2 bc:24:11:f9:4d:ae ESI-0
0 Seq:0 N/A
192.168.254.18
VNI 31And we see it as we expect on the SROS router. We have the mac address, the VNI it’s attached to and the next hop which will be the interface on compute2 (Proxmox hypervisor).
vyos@devkptvyos02:~$ show evpn mac vni all
VNI 31 #MACs (local and remote) 4
Flags: N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
d0:99:d5:5a:c8:ec remote 192.168.254.137 0/0
ea:89:3c:f0:86:82 local br1 1 0/0
86:a8:39:b4:ec:43 remote 192.168.254.137 0/0
bc:24:11:f9:4d:ae remote 192.168.254.18 0/0Then on Vyos, we can see the mac address of the VM is learned. This is because SROS is sending the EVPN route to the Vyos instances, and it installs it in it’s table. The gateway which is 172.16.0.1 that exists within our vrf called “test-vprn” has a mac of ea:89:3c:f0:86:82
Now the packet will route to the gateway, and Vyos will look in it’s route table on how to get to 172.16.1.0/24.
vyos@devkptvyos02:~$ show ip route vrf test-vprn
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF test-vprn:
C>* 172.16.0.0/24 is directly connected, br1, 6d12h41m
B> 172.16.1.0/24 [200/0] via 10.11.0.2 (vrf default) (recursive), label 131056, weight 1, 01:48:05
* via 10.0.0.8, eth1.34 (vrf default), label 131071/131056, weight 1, 01:48:05Now Vyos has a route using a MPLS transport and service label to the SROS router.
Lastly on the Nokia router, it will simply deliver it over vlan 31.
A:KPTPE01# show router 231 route-table
===============================================================================
Route Table (Service: 231)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
172.16.0.0/24 Remote BGP VPN 01h54m38s 170
10.11.0.140 (tunneled) 0
172.16.1.0/24 Local Local 11d08h37m 0
vlan31 0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:KPTPE01# At this point, we can see some of what the traffic looks like on the network. I am capturing from “compute2” for anything with the port 4789 for VXLAN.
tcpdump -i any port 4789
13:43:29.824872 enp4s0 Out IP 192.168.254.18.53966 > 192.168.254.140.4789: VXLAN, flags I, vni 31
IP 172.16.0.69 > 172.16.1.69: ICMP echo request, id 3, seq 9, length 64 <– Inner
13:43:30.827019 enp4s0 In IP 192.168.254.140.52047 > 192.168.254.18.4789: VXLAN, flags I, vni 31
IP 172.16.1.69 > 172.16.0.69: ICMP echo reply, id 3, seq 10, length 64 <– Inner
So we can see the VM 172.16.0.69 sending a ping request to 172.16.1.69. That packet is encapsulated in VXLAN, and send from the Proxmox IP to Vyos. We also see the return, also encapsulated in VXLAN.
So of course, doing this all with two hosts in a single network isn’t all that useful per say. But imagine having many hosts, and even those hosts not being in the same area, that can support being on the same L2 network and mobility. Although stretching L2 across WANs is not always the best idea, but I’ll leave that up to your discretion 🙂