In my last post, I went over the configuration of Proxmox, Vyos, and SROS. Here I want to show over how the setup looks and works with real VMs attached to this.
Above I have tried to show the topology. The VM in the bottom left is running on compute2 (Proxmox hypervisor), and attached to the VNET “Vlan31” which has the IP of 172.16.0.69. This VM’s gateway exists on the pair of Vyos instances, which tie the VXLAN tunnel into a VRF. The other VM, 172.16.1.69, is attached to a normal Proxmox bridge/vlan that is routed on the 7210 within the VPRN service. So these VMs are just to show connectivity between these systems.
With the Vnet configured in Proxmox, it is possible to assign that to a VM’s interface. Which simply looks like this below:
The other VM, I’m simply using a traditional linux bridge, and a vlan.
The VM above is the 172.16.0.69 VM, and showing connectivity to the other VM on 1.69. Let’s dive in how it works.
First I will lay out the IPs and macs for reference.
172.16.0.69 (testclone15)
BC:24:11:F9:4D:AE
172.16.1.69 (testclone16)
BC:24:11:9E:9A:41
First we can look at the EVPN type 2 route (mac address) for the first host.
A:KPTPE01# show router bgp routes evpn mac mac-address BC:24:11:F9:4D:AE
===============================================================================
BGP Router ID:10.11.0.2 AS:65000 Local AS:65000
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Ip Address
NextHop
Label1
-------------------------------------------------------------------------------
*>i 192.168.254.18:2 bc:24:11:f9:4d:ae ESI-0
0 Seq:0 N/A
192.168.254.18
VNI 31
And we see it as we expect on the SROS router. We have the mac address, the VNI it’s attached to and the next hop which will be the interface on compute2 (Proxmox hypervisor).
vyos@devkptvyos02:~$ show evpn mac vni all
VNI 31 #MACs (local and remote) 4
Flags: N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
d0:99:d5:5a:c8:ec remote 192.168.254.137 0/0
ea:89:3c:f0:86:82 local br1 1 0/0
86:a8:39:b4:ec:43 remote 192.168.254.137 0/0
bc:24:11:f9:4d:ae remote 192.168.254.18 0/0
Then on Vyos, we can see the mac address of the VM is learned. This is because SROS is sending the EVPN route to the Vyos instances, and it installs it in it’s table. The gateway which is 172.16.0.1 that exists within our vrf called “test-vprn” has a mac of ea:89:3c:f0:86:82
Now the packet will route to the gateway, and Vyos will look in it’s route table on how to get to 172.16.1.0/24.
vyos@devkptvyos02:~$ show ip route vrf test-vprn
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF test-vprn:
C>* 172.16.0.0/24 is directly connected, br1, 6d12h41m
B> 172.16.1.0/24 [200/0] via 10.11.0.2 (vrf default) (recursive), label 131056, weight 1, 01:48:05
* via 10.0.0.8, eth1.34 (vrf default), label 131071/131056, weight 1, 01:48:05
Now Vyos has a route using a MPLS transport and service label to the SROS router.
Lastly on the Nokia router, it will simply deliver it over vlan 31.
A:KPTPE01# show router 231 route-table
===============================================================================
Route Table (Service: 231)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
172.16.0.0/24 Remote BGP VPN 01h54m38s 170
10.11.0.140 (tunneled) 0
172.16.1.0/24 Local Local 11d08h37m 0
vlan31 0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:KPTPE01#
At this point, we can see some of what the traffic looks like on the network. I am capturing from “compute2” for anything with the port 4789 for VXLAN.
tcpdump -i any port 4789
13:43:29.824872 enp4s0 Out IP 192.168.254.18.53966 > 192.168.254.140.4789: VXLAN, flags I, vni 31 IP 172.16.0.69 > 172.16.1.69: ICMP echo request, id 3, seq 9, length 64 <– Inner
13:43:30.827019 enp4s0 In IP 192.168.254.140.52047 > 192.168.254.18.4789: VXLAN, flags I, vni 31 IP 172.16.1.69 > 172.16.0.69: ICMP echo reply, id 3, seq 10, length 64 <– Inner
So we can see the VM 172.16.0.69 sending a ping request to 172.16.1.69. That packet is encapsulated in VXLAN, and send from the Proxmox IP to Vyos. We also see the return, also encapsulated in VXLAN.
So of course, doing this all with two hosts in a single network isn’t all that useful per say. But imagine having many hosts, and even those hosts not being in the same area, that can support being on the same L2 network and mobility. Although stretching L2 across WANs is not always the best idea, but I’ll leave that up to your discretion 🙂
With Proxmox 8, and production support of software defined networking, I started to take a harder look at what is possible.
From my understanding, this feature is built on FRR. In this blog, I am looking to take an unconventional approach at using this. From the Proxmox documentation, most of what this feature seems to be used for is networking nodes together which could be over a WAN etc. There are limited options that are available in the GUI, which of course is all you need for this feature to work. I wanted to see what was happening under the hood, and expand on this by having networking features outside of just the hosts too.
Ideally, my setup would consist of VXLAN tunnels terminating on a router, where I have a layer 3 gateway within a VRF. Problem is, while the 7210 supports EVPN, it only supports transport over EVPN-MPLS. Most every opensource project that I have found only supports VXLAN. So my idea was then to use Vyos in the middle, which can terminate the VXLAN tunnels from the Proxmox nodes, and then route from a VRF, to another VRF on the 7210 via MPLS.
Topology
The first picture I’m depicting the control plane of this setup. I am using a Nokia 7210 as a route reflector with several different address families. EVPN for the Proxmox nodes, and EVPN/VPN-IPv4/v6 towards Vyos. More on VPN-IPv6 though, as sadly ran into some unresolved issues.
The second pictures shows the transport tunnels. Simply, VXLAN from Proxmox to Vyos ( It would be full mesh, I neglected to draw it here) and LDP tunnels to the 7210.
Proxmox Configuration
There are some prerequisites that are all covered in the Proxmox documentation in chapter 12. All of these settings are under Datacenter >SDN. First let’s setup a “controller” which is really just defining an external router which will run the EVPN protocol. Under options, add a EVPN controller.
Here simply give it a name, ASN, and peer IP. Then on my router config, I defined a peer group and a cluster id, which means this peer group will act as a route reflector:
A:KPTPE01>config>router>bgp# info
----------------------------------------------
group "iBGP-RR"
family ipv4 vpn-ipv4 evpn
cluster 10.11.0.2
peer-as 65000
advertise-inactive
bfd-enable
neighbor 192.168.254.13
description "compute3"
exit
neighbor 192.168.254.18
description "compute2"
exit
exit
Next, need to define a zone. Proxmox gives this as a definition of “A zone defines a virtually separated network. Zones are restricted to specific nodes and assigned permissions, in order to restrict users to a certain zone and its contained VNets.” In my case, I’m using this to define the vxlan tunnel, and it’s endpoints, which end up being my two vyos instances.
Last thing is to create a vnet. The vnet will be what VMs are actually attached to, and to create a broadcast domain. This config is simple, basically tying together a zone and give it a name a tag, which the tag ends up being the VNI.
Exploring the details
Looking on the Nokia, we can look at what the include multicast routes. As a quick overview, inclusive multicast routes are what are advertised between routers to announce the service. If two peers have RTs and a VNI match, they can build a VXLAN tunnel and exchange data.
A:KPTPE01# show router bgp routes evpn inclusive-mcast rd 192.168.254.18:2 hunt
===============================================================================
BGP Router ID:10.11.0.2 AS:65000 Local AS:65000
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN Inclusive-Mcast Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network : N/A
Nexthop : 192.168.254.18
From : 192.168.254.18
Res. Nexthop : 192.168.254.18
Local Pref. : 100 Interface Name : vlan1
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None
Connector : None
Community : bgp-tunnel-encap:VXLAN target:65000:31
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 192.168.254.18
Flags : Valid Best IGP
Route Source : Internal
AS-Path : No As-Path
EVPN type : INCL-MCAST
ESI : N/A
Tag : 0
Originator IP : 192.168.254.18 Route Dist. : 192.168.254.18:2
Route Tag : 0
Neighbor-AS : N/A
Orig Validation: N/A
Add Paths Send : Default
Last Modified : 04d19h19m
-------------------------------------------------------------------------------
PMSI Tunnel Attribute :
Tunnel-type : Ingress Replication Flags : Leaf not required
MPLS Label : VNI 31
Tunnel-Endpoint: 192.168.254.18
First we can see from the communities, the encapsulation is VXLAN, and the route target. Also then the VNI, which is the tag we picked as in “31” in this case. If this router supported vxlan, we would be good to go configuring a VPLS service with VXLAN encapsulation. But some more here, and where would the fun be then?
Vyos Configuration
I will keep expanding on this, but for now the relevant configuration for the VXLAN side of this communication.
#This is the VXLAN termination interface in the same L2 as the hypervisors
ethernet eth1 {
address 192.168.254.137/24
hw-id bc:24:11:51:9c:48
mtu 9000
#Routed interface towards 7210
vif 32 {
address 10.0.0.7/31
mtu 9000
}
}
loopback lo {
address 10.11.0.137/32
}
vxlan vxlan31 {
ip {
enable-directed-broadcast
}
mtu 9000
parameters {
nolearning
}
port 4789
source-address 192.168.8.167
vni 31
}
}
protocols {
bgp {
address-family {
l2vpn-evpn {
advertise-all-vni
advertise-svi-ip
vni 31 {
route-target {
both 65000:31
}
}
}
}
#7210 RR client
neighbor 10.11.0.2 {
address-family {
ipv4-vpn {
}
l2vpn-evpn {
}
}
peer-group iBGP-RR-PE
}
#7210 RR Client
neighbor fc00::2 {
address-family {
ipv6-vpn {
}
}
remote-as 65000
}
parameters {
router-id 10.11.0.137
}
peer-group iBGP-RR-PE {
remote-as 65000
}
system-as 65000
}
#IGP towards 7210 and enable LDP
isis {
interface eth1.32 {
network {
point-to-point
}
}
interface lo {
passive
}
level level-1-2
metric-style wide
net 49.6901.1921.6800.0167.00
}
mpls {
interface eth1.32
ldp {
discovery {
transport-ipv4-address 10.11.0.137
}
interface eth1.32
router-id 10.11.0.137
}
}
}
I tried to comment the import parts of the config, and to provide it all with examples. At a high level ethernet eth1 is the interface in the same vlan as the hypervisors to send vxlan traffic between the two. Ethernet1.32 is a point to point interface (just being transported over a vlan in my network) to the 7210 to run LDP over.
VPN-IPv4 Configuration
Now turning focus to the other side of Vyos, the mpls/vpn-ipv4 configuration. To recap the traffic flow will look as such:
Proxmox hypervisor <—VXLAN-> vyos (with L3 gateway for VXLAN service) <—MPLS/LDP —> 7210 with a VPRN service
Using VRRP to provide gateway redundancy between the two vyos instances for a gateway. This way if a instance goes down, the other vyos instance will still provide a valid gateway.
Bridge interface is created with the members of the vxlan31 interface, which is what defines the VNI parameters
Then the vrf configuration
Most import is defining the route target, redistribute the local interfaces, route distinguisher and to import/export using vpn routes.
Now the 7210:
A:KPTPE01# configure service vprn 231
A:KPTPE01>config>service>vprn# info
----------------------------------------------
route-distinguisher 65000:231
auto-bind-tunnel
resolution any
exit
vrf-target target:65000:231
interface "vlan31" create
address 172.16.1.1/24
sap 1/1/25:31 create
ingress
exit
egress
exit
exit
exit
no shutdown
On the Nokia side, a little more simple as we are only focused with the VPN-IPv4 side of things here
RD and RT is defined
I created a L3 interface here just to have something to route to outside the interface in vyos
Review and Checks
To close out this part, let’s run some show commands and make sure the environment is ready to support some clients.
*A:KPTPE01# show router bgp summary
===============================================================================
BGP Router ID:10.11.0.2 AS:65000 Local AS:65000
===============================================================================
BGP Summary
===============================================================================
Neighbor
Description
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
10.11.0.137
prdkptvyos01
65000 21466 0 06d09h28m 1/1/312 (VpnIPv4)
21947 0 4/0/40 (Evpn)
10.11.0.140
prdkptvyos02
65000 10 0 00h02m24s 1/1/313 (VpnIPv4)
213 0 5/0/45 (Evpn)
192.168.254.13
compute3
65000 323110 0 11d05h15m 1/0/40 (Evpn)
326755 0
192.168.254.18
compute2
65000 322278 0 11d04h34m 1/0/40 (Evpn)
325924 0
-------------------------------------------------------------------------------
*A:KPTPE01# show router ldp discovery
===============================================================================
LDP IPv4 Hello Adjacencies
===============================================================================
Interface Name Local Addr State
AdjType Peer Addr
-------------------------------------------------------------------------------
to-devkptvyos01 10.11.0.2:0 Estab
link 10.11.0.137:0
to-devkptvyos02 10.11.0.2:0 Estab
link 10.11.0.140:0
-------------------------------------------------------------------------------
*A:KPTPE01# show router tunnel-table
===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
-------------------------------------------------------------------------------
10.11.0.137/32 ldp MPLS 65542 9 10.0.0.7 20
10.11.0.140/32 ldp MPLS 65541 9 10.0.0.9 20
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================
A:KPTPE01>config>service>vprn# show router 231 route-table protocol bgp-vpn
===============================================================================
Route Table (Service: 231)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
172.16.0.0/24 Remote BGP VPN 00h26m19s 170
10.11.0.140 (tunneled) 0
-------------------------------------------------------------------------------
vyos@devkptvyos01:~$ show mpls ldp binding
AF Destination Nexthop Local Label Remote Label In Use
ipv4 10.11.0.2/32 10.11.0.2 16 131071 yes
ipv4 10.11.0.137/32 0.0.0.0 imp-null - no
ipv4 10.11.0.140/32 10.11.0.2 18 131045 yes
vyos@devkptvyos01:~$ show mpls ldp neighbor
AF ID State Remote Address Uptime
ipv4 10.11.0.2 OPERATIONAL 10.11.0.2 6d10h56m
#View EVPN Type 2 Inclusive multicast routes
vyos@devkptvyos01:~$ show bgp vni all
VNI: 31
BGP table version is 1041, local router ID is 10.11.0.137
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
#192.168.254.13 and .18 are proxmox hosts
*>i[3]:[0]:[32]:[192.168.254.13]
192.168.254.13 100 0 i
RT:65000:31 ET:8
*>i[3]:[0]:[32]:[192.168.254.18]
192.168.254.18 100 0 i
RT:65000:31 ET:8
#The two vyos routers
*> [3]:[0]:[32]:[192.168.254.137]
192.168.254.137 32768 i
ET:8 RT:65000:31
*>i[3]:[0]:[32]:[192.168.254.140]
192.168.254.140 100 0 i
RT:65000:31 ET:8
vyos@devkptvyos01:~$ show ip route vrf test-vprn
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF test-vprn:
C>* 172.16.0.0/24 is directly connected, br1, 6d11h23m
B> 172.16.1.0/24 [200/0] via 10.11.0.2 (vrf default) (recursive), label 131056, weight 1, 6d11h22m
* via 10.0.0.6, eth1.32 (vrf default), label 131071/131056, weight 1, 6d11h22m
I know this is a ton of data above, but what I’m trying to show is that we have the expected inclusive multicast EVPN routes from Proxmox and the Vyos instances. Then there is MPLS/LDP connectivity between vyos and the 7210. Finally in the route tables, we have a route bidirectionally between the two so in theory if hosts were to communicate between the two it should work.
This is all I have for this post. The next post I will actually enable VMs on these networks, and show the connectivity.
A few years ago, after installing Ubuntu server for the 100th time, I thought how cool it would be to automate the build. I also had some some inspiration from my workplace and the tooling that was used.
I personally build mostly Ubuntu servers, so that is what this will be focused on. My other systems, like Windows or other distros are infrequent but I hope to add them to add them to my catalog.
The code
I created this script which is on GitHub here. It is somewhat customized to my situation, but I hope to grow it into something more modular.
First I built this around the idea of having IPAM data in Phpipam, and a supplemental mysql database. Also once the VM is built, I have the option to add the VM to Librenms.
To run the code, first need to fill in the secret file here. This is imported by the various python files, to use as credentials.
Then to run the script there are a few command line inputs:
--template_id TEMPLATE_ID proxmox template ID to clone
--new_vmid NEW_VMID VM ID of new cloned vm
--new_vm_name NEW_VM_NAME name of new vm
--ipam_section IPAM_SECTION ipam section, enter ipv4 or ipv6
--disk_size DISK_SIZE disk size of disk after clone
--add_librenms ADD_LIBRENMS add device to librenms after vm is booted. new_vm_name to be set
--vlan_id VLAN_ID vlan ID assignment for VM and DB lookup
The vlan id field is used to query the mysql database, to get the correct IPv4 and IPv6 address assignment. The vlan then is updated on the cloned VM with this information as well.
The database table looks like this below as an example:
First to start, Ubuntu offers specific images for cloud init here. Find the release, and architecture you need. In my example I used wget to download directly to Proxmox:
Next create the VM that we will use as a template. The memory and network are just place holders, my script updates these parameters when the clone happens:
Now import the downloaded img file as a disk to the VM. Note this will be now an unused disk in the GUI:
#Syntax: qm importdisk "vmID" "image-file-name" "storage-lvm-name"
#My example:
qm importdisk 302 ubuntu-22.04-server-cloudimg-amd64.img compute2-zvol-vm1
<abbreviated output>
transferred 2.2 GiB of 2.2 GiB (100.00%)
Successfully imported disk as 'unused0:compute2-zvol-vm1:vm-302-disk-0'
With it attached to the VM, now convert it to a real disk
#Syntax:
qm set 302 --scsihw virtio-scsi-pci --scsi0 "lvm-disk-name:vm-disk-name" (which was created in the last command)
#My example:
qm set 302 --scsihw virtio-scsi-pci --scsi0 compute2-zvol-vm1:vm-302-disk-0
Now adding a cloud init drive:
qm set 302 --ide2 compute2-zvol-vm1:cloudinit
update VM 302: -ide2 compute2-zvol-vm1:cloudinit
And some final settings:
#Make scsi drive bootable:
qm set 302 --boot c --bootdisk scsi0
#Attach a display:
qm set 302 --serial0 socket --vga serial0
Now it time to utilize the custom cloud init configuration. Cloud init supports 3 different types of configuration files:
user
network
meta
These allow you expand configuration that is not available within the GUI of Proxmox. A lot of good information is available here of all the possibilities.
My main goal is to install packages which I apply to all VMs, some iptables statements for QoS, and finally copying over some configuration files for several of the services. This is my example file.
Any of the custom cloud init files, need to exist on a data store which is enabled for snippets. By default the local storage is enabled for this, and the location would be /var/lib/vz/snippets
The last step of this is to associate the custom file with cicustom utility. For example:
qm set <VM_ID> --cicustom "user=local:snippets/user.yaml"
Example:
qm set 302 --cicustom "user=local:snippets/user.yaml"
Running the code
python3 build_vm_.py --template_id 302 --new_vmid 223 --new_vm_name testclone16 --disk_size 20G --vlan_id 66
Connecting to database prdkptdb01.internal.keepingpacetech.com
Running Query on database for vlan 66
IPv4 network found: {'network': '192.168.66.128', 'gateway': '192.168.66.129', 'subnet_mask': '25'}
IPv6 network found: {'network': 'fd00:0:0:66b::', 'gateway': 'fc00:0:0:66b::1', 'subnet_mask': '64'}
Running clone vm
checking vm status 223
clone
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
VM is still locked
checking vm status 223
vm must be cloned
was able to write IP to IPAM
A few details on what the script does. First, there seems to be documented cases where when a custom file is used, Proxmox does not update the host name automatically. From a great post on the Proxmox forums, I found that you can set the host name via the smbios settings available the options in Proxmox.
The script will read the UUID of the cloned vm, and set it again with a rest call. Also with the serial, here we can set the host name. Then of course, it sets the cloud init config with the network settings that I choose in the development.
Now the fun part! Time to boot the VM, and if you watch the console it will eventually stop scrolling and have a line “cloud-init reached target” if all was successful. Enjoy your freshly built and customized vm!
If you missed part one, here is a link to read that first. It will give a good overview of how I divided up my network. This part I wanted to dive into is how I setup BGP peering between two systems. Even those this is specific to my needs, this would be a great overview of how to accomplish any BGP peering, so I wanted to bring everyone along!
I am using eBGP between pfSense and each VPRN service on the Nokia router. I will assume, in this post, that you know how to already configure additional vlan interfaces on pfSense, but if it helps to see that config I can always add it; just leave a comment!
pfSense FRR Package
To install the FRR package, it is as simple as any other package in pfSense. Navigate to system > package manager, and install “frr”
Now, under services, you should notice a number of new FRR options. These are broken up by protocol, so you will see some options for OSPF, RIP and BGP. Technically, in this setup, you can use anything you prefer but I like BGP in this instance to make nice import/export policies and even have options to still have reachability between VPRNs when pfSense is down.
Now under FRR > BGP we will start with some global parameters
In my setup, I use pfSense as the first AS I consider to use which is 64500. Then I give it a router ID which is the LAN interface.
Then in FRR’s newest version that pfSense uses, it is updated to respect RFC 8212 where bgp routes are not imported/exported unless a policy is applied. For now, to keep it simple, I have a policy which simply accepts all. This is under services > FRR Global > Route Map. Here, I simply created one that is named “Allow-All” and set action to “accept.”
Building First BGP Peer
Under FRR BGP > Neighbor > Add we can now define the first neighbor that we want to establish a IPv4 BGP session with.
If you take a look at my topology diagram from part 1, or the close up picture of the topology below, I use 10.1.100.0/24 for my IPv4 point to point links. Each vlan, which represents a routed link from pfSense to the Nokia router, is a /29. I went to this because I am using two pfSense VMs, and that allows for the 3 CARP addresses needed on pfSense. CARP is a way to have a virtual floating IP – more in another post to come. But this allows a fail over between the pfSense VMs and the Nokia will not need to update any BGP peer, it will simply re-establish if there is a failure.
In any case, for a simple IPv4 eBGP peer, you need 3 things:
IP address of the neighbor
ASN (autonomous system number)
Route Map
The rest should be able to stay default. There are of course a lot of options in BGP. Not that I want to skip over these, but personally I like to keep pfSense simple. I’m not an expert at FRR and these options in the GUI. I like to create the complexity on the Nokia as I am just more used to it.
Lastly make sure to go to services > FRR Global > and check the “Enable” box to turn the package up.
Building peer on the Nokia router
To start out, I simply have printed the interfaces. I have standardized on naming my interfaces “to-pfsense-01-vlxxx” which the vlan corresponds to the VPRN number I use. Notice here, the IPv4 IP is 10.1.100.12 which we defined in pfSense as the neighbor earlier.
For some of the basic parameters, like ASN, router ID, etc. This is defined at the root level of the VPRN service, which is “/configure service vprn 101”
Now I have created a bgp group in the VPRN service. Here I simply defined the neighbor, its ASN, a family, and an export policy.
For completeness, here is the policy:
I wish I could talk about all of this, but I will be skipping a few things to cover later in another blog post. If you already know the other families and extensions of BGP, you probably understand some of the above. To keep this from being a book, I will go over the basics.
Each prefix is in this service, VPRN 101, will be evaluated top down in this policy. So I have several entries that perform drops, accepts, and then finally if a route does not match any of the other entries, the default action is to drop it.
Prefix lists, are just groups of networks. For example the “Routed-Links”
In an effort to try to keep the routing tables clean as possible I, for example, block any of the /29 networks in the 10.1.100.0/24 blockwhich I have set up between pfSense and this Nokia router. The longer just means that anything that falls within the 10.1.100.0/24 range – that has a longer subnet mask, included here. My /29s are longer. I could define every /29 here, but that’s a lot more work, and then every time I would turn up a new VPRN I would have to update policies (which is not ideal).
The main points of interest for this post are the “direct” networks. Direct means any of the layer 3 interfaces that are configured on this router. In my first picture where I looked at the L3 interface, you can see VL66 and VL666. These are direct interfaces, and since I want to be able to have other hosts on my network reach these, and conversely, those networks be able to access other hosts, I am exporting these to pfSense.
Validation
If everything goes right, the peer should be established.
There is a ton of information to digest in this image:
We can see the BGP information that is local to this VPRN service
The AS is 64601 (we defined this in pfSense for the neighbor, and local in the Nokia VPRN service)
BGP is operationally up
The router ID for this service is 10.11.0.2
The first neighbor, we can see is the IP of pfSense, 10.1.100.9.
It has been up for 6+ days (when it is in the up state a addr family will be shown)
Also we can see that we have received 36 routes, 31 are active in the route table, and 5 are advertised (this is from the heading Rcv/Act/Sent)
Finally, the address family is IPv4
Now let’s looked at received routes, and advertised. This is called the RIB (Routing Information Base) – I cut this down as we do not need to see everything:
Routes that exist in the received RIB, you can think of as a “dumping ground.” Routes are populated here from whatever the peer has sent, but that does not mean that it will be the active route in the route table. An import policy may reject these, and/or other protocols which are advertising the same prefix to this router that may be preferred. If the logic and router determines the route should be installed in the route table because it is best, it will have a status code. Here looking at the static codes u*>i, means that it is used, valid, best, and IGP (mostly just a old carry over when EGP was a protocol before BGP). If we look in the route table we can see the default route exists via the protocol BGP and a next hop of pfSense.
Not forgetting the advertised routes, we can see the two 192.168.66.0 (and 128)/25 networks are there, with a few others which match other accepts in the policies which are being sent to pfSense.
Similarly, we can check pfSense for the information it has. First under Status > FRR. BGP, this will show you the simiar RIB information. Here we can see the two /25s (among others of course) that pfSense learns about from the various eBGP peers.
Under status > routes, shows the actual FIB (routing table) in pfSense. Again, we can see the routes are installed, and the RIB gave us that clue they would be as well because the status code is *>
This closes out this part of the discussion. At this point we have a eBGP peer up, and routes are being advertised between the routers. From this point, firewall rules can be assigned to allow/block traffic, and nat rules need to be created to allow the hosts to access the internet since they are private RFC 1918 space. In the next part, I will cover the firewall rules and NAT rules I have applied for this. Eventually, we will get to IPv6, which will be very similar but with a few slight changes.
Homelab’s logical network was a lot more work than I thought to put together, but it has been a great exercise to finally put this down on paper.
In part one, I am trying to give some background to the services I run, and how I setup my network. I will have a follow-up with some more hardware based details, and more specific vm/containers.
To start, I have some EOL Nokia network gear for my core routing and use pfSense for edge firewall/VPN/HA. I have logically broken up these services in IP VRFs (Virtual Routing and Forwarding) or a Nokia specific service – a VPRN (Virtual Private Routed Network)
This allows me to control access between the VPRN services, through a firewall (pfSense) instead of relying on ACLs on the Nokia router, or moving all my routing to pfSense, which would slow performance. This, in my opinion, is the best of both worlds. Routers route, and firewalls inspect. Now when I have services on multiple networks, within the same VPRN, I get the benefit of wireline bandwidth as it will route like a normal router. Otherwise, if traffic needs to cross VPRNs, it will need to go through pfSense where firewall rules are evaluated.
Without further ado here is the drawing!
Here is a quick run down of the services from top down:
Offsite:
I run Truenas scale on an old server over a Wireguard site-to-site VPN running on a Raspberry Pi. This is not powered on all the time, and I use it to replicate snapshots from my main Truenas core’s SMB shares.
The other offsite location I have is a Synology NAS, to which I also replicate Truenas core’s SMB shares using rsync.
I have Comcast business service and several static IPv4 IPs and IPv6 block.
On my Edge, I have two pfSense VMs running on two hypervisors which provide HA failover for themselves and my internal network.
Below is a list of the VPRN services:
VPRN
VPRN Name
Purpose
Notes
100
Inband/AD
Hypervisor gui mgmt, and AD Services
Critical AD services (DNS/DHCP/NPS)
101
Infrastructure services
Network hardware mgmt, and VMs running critical software
Syslog/snmp monitoring/stats
102
Unrestricted clients
Clients that have no rules blocking to other zones
TLS/PEAP WLAN clients or domain joined machines
103
Guest/IOT
Guest SSID and IOT devices
104
DMZ
Mix of internal DMZ and inbound services allowed
Web/mail/NTP pool members
105
Restricted Devices
Devices that have access blocked to AD/infrastructure services
clients that do not need full access to the internal network
106
File Services
Mostly for SMB access
Truenas scale SMB shares
108
Cameras
NVR and cameras
Blueiris NVR and IP cameras
109
Storj
Storj node operator VM
Hopefully that layout makes sense so you can understand how I went about separating these. Feel free to ask me questions! Now, let’s spend a little bit of time on the routing and how it looks:
I am using VPRN service 100 as an example. I send the default route via the eBGP neighbor (pfSense). Then if I look for another prefix that would be in another VPRN (this case 102) it does not route directly to that network; instead it goes to the firewall first even though both of these exist on the same router. A cleanup I want to do is to only send the default route from pfSense to these VPRNs, not on specific routes, but that will be a future blog post!
We can view the local configured subnets in the VPRN. These would be exported to pfSense so other VPRNs, and pfSense would know how to reach these hosts.
Finally, in pfSense, here is a quick shot of how the route table looks. First there is 192.168.8.0/25 that is in VPRN 100, and the 192.168.9.0/24 prefix that is in VPRN 102. Notice how they have two different eBGP next hop IPs which correspond to the interface on the Nokia router.
Part 2 I’ll dive into the FRR package in pfSense and the BGP setup on the Nokia router.
“Homelab” is a colloquial term to describe a home server (or entire datacenter, if you have the space) that hobbyists can play around with. I’d like to take this post to talk about the evolution of my Homelab. If you have spent any time on reddit.com/r/homelab, you will probably see people asking “what do you use this hardware for?” For everyone it is different.
I have played with technology since I was a kid, but it started to become more serious when I was in high school and working at a small IT support company. Around that time, I decided to start my own consumer-focused computer repair company to “compete” with Best Buy, etc.
Because I learned a lot through my job, I wanted to set up a file server just like we had at the IT support company so I could host all my common repair tools centrally. A customer had a old machine they were not using, and I was able to get a copy of windows server 2008 from high school for free with a promotion they were running to provide students with copies.
From there, Homelab continued to grow. I built a new machine where I could back up customers’ computers when I needed to format them, and also store family documents, games, movies etc.
Choice of hardware for the first server. An AMD Athlon X2, I believe 4gig of ram and a few hard drives for storage
From what I remember, at this point I had set up several machines (pictured below). One running Windows Server 2008 for mostly a file server, and an old Dell machine to be a firewall running Untangle. I believe the middle machine was running Windows 7 to be a media server that I would copy Blu-ray movies to, and also had a cable card in it to distribute live TV to other TV’s in the house with Windows Media Center.
When it came time for college, Homelab moved with me to an apartment. I found out that my local Goodwill store sold computer hardware. And, at times, you could find old enterprise gear there. I started to accumulate hardware and this is where (from taking the CCNA and learning more in college, and my first internship) I could actually start to work on network equipment at a basic level. I ran OSX on a Mac mini server model, and pfsense on a Watch Guard firewall. I believe I ran the switches just for studying for CCNA at the time.
College Homelab 1.0
My second apartment in college was a little bigger, so I continued to grow this more:
College Homelab 2.0
Fast forwarding to today – and what an upgrade! – this is my setup:
Nokia 7210 MXP. This serves as my main router. Many posts to come on this. The SAS-X is not used, just still in the rack 🙂
Some PDUs. The one is APC which is able to be controlled via a browser.
Next, a Comcast Modem, Mikrotik CRS310-1G-5S-4S+, which serves 10G connectivity to my two hypervisors and my Truenas disk storage for iSCSI traffic. Lastly, it has connectivity for my desktop PC at 2.5G.
Last of the networking equipment is my main switch, a Huawei S5300 that all my devices plug into. It has a funny origin story: this came from my local KMart when it closed. It has 4x10G ports, which independently serve as an uplink to the 7210 router, 10G connectivity for VMs hosted on the hypervisors, and 10G to the Mirkotik switch. It is POE and all my other devices within my home connect to it. In the next picture, when I purchased this home, one of my first projects was wiring cat6 everywhere (and now even to my shed).
The front of the rack of the compute/storage of the lab starts with the Rosewell RSV-L4500U that runs Truenas core. It has a mix of disks, which are 2x8TB and 4x10TB WD Red drives, and 6x1TB SSDs.
The HDDs are set up in a 3x mirrored VDEVs. The SSDs are also set up in their own mirrors. I use this as a very rudimentary tiered storage. My VMs that need performance, and are important, have their disks on one of the SSD pools, while the other VMs are running on the HDDs. Finally, any of my SMB shares also come from the HDD pool. It has a Intel Celron 64920 and 32GB of ECC ram.
Next is a Dell r330. It runs Proxmox, and just a few VMs that are critical + a GPU for encoding videos from my NVR. This has just a 8 core Xeon E3-1230 and 32gig of ram.
On to the next, is a Dell R630. This is my main server, and it has a pair of 12 core Xeon E5-2643v3 CPUs, and 128gig of RAM.
Lastly, is my main battery backup the APC UPS 1500, with a mgmt card. My VMs all use apcupsd, to learn the status of the battery, and I have auto shutdowns in place to theoretically shutdown all the hosts before the battery runs out.
I think everything has to start what it physically looks like! In my next post I plan to talk in depth about mostly routing, and the logical setup I use to segment my network.
