Month: November 2023

If you missed part one, here is a link to read that first. It will give a good overview of how I divided up my network. This part I wanted to dive into is how I setup BGP peering between two systems. Even those this is specific to my needs, this would be a great overview of how to accomplish any BGP peering, so I wanted to bring everyone along!

I am using eBGP between pfSense and each VPRN service on the Nokia router. I will assume, in this post, that you know how to already configure additional vlan interfaces on pfSense, but if it helps to see that config I can always add it; just leave a comment!

pfSense FRR Package

To install the FRR package, it is as simple as any other package in pfSense. Navigate to system > package manager, and install “frr”

Now, under services, you should notice a number of new FRR options. These are broken up by protocol, so you will see some options for OSPF, RIP and BGP. Technically, in this setup, you can use anything you prefer but I like BGP in this instance to make nice import/export policies and even have options to still have reachability between VPRNs when pfSense is down.

Now under FRR > BGP we will start with some global parameters

In my setup, I use pfSense as the first AS I consider to use which is 64500. Then I give it a router ID which is the LAN interface.

Then in FRR’s newest version that pfSense uses, it is updated to respect RFC 8212 where bgp routes are not imported/exported unless a policy is applied. For now, to keep it simple, I have a policy which simply accepts all. This is under services > FRR Global > Route Map. Here, I simply created one that is named “Allow-All” and set action to “accept.”

Building First BGP Peer

Under FRR BGP > Neighbor > Add we can now define the first neighbor that we want to establish a IPv4 BGP session with.

If you take a look at my topology diagram from part 1, or the close up picture of the topology below, I use 10.1.100.0/24 for my IPv4 point to point links. Each vlan, which represents a routed link from pfSense to the Nokia router, is a /29. I went to this because I am using two pfSense VMs, and that allows for the 3 CARP addresses needed on pfSense. CARP is a way to have a virtual floating IP – more in another post to come. But this allows a fail over between the pfSense VMs and the Nokia will not need to update any BGP peer, it will simply re-establish if there is a failure.

In any case, for a simple IPv4 eBGP peer, you need 3 things:

• IP address of the neighbor
• ASN (autonomous system number)
• Route Map

The rest should be able to stay default. There are of course a lot of options in BGP. Not that I want to skip over these, but personally I like to keep pfSense simple. I’m not an expert at FRR and these options in the GUI. I like to create the complexity on the Nokia as I am just more used to it.

Lastly make sure to go to services > FRR Global > and check the “Enable” box to turn the package up.

Building peer on the Nokia router

To start out, I simply have printed the interfaces. I have standardized on naming my interfaces “to-pfsense-01-vlxxx” which the vlan corresponds to the VPRN number I use. Notice here, the IPv4 IP is 10.1.100.12 which we defined in pfSense as the neighbor earlier.

For some of the basic parameters, like ASN, router ID, etc. This is defined at the root level of the VPRN service, which is “/configure service vprn 101”

Now I have created a bgp group in the VPRN service. Here I simply defined the neighbor, its ASN, a family, and an export policy.

For completeness, here is the policy:

I wish I could talk about all of this, but I will be skipping a few things to cover later in another blog post. If you already know the other families and extensions of BGP, you probably understand some of the above. To keep this from being a book, I will go over the basics.

Each prefix is in this service, VPRN 101, will be evaluated top down in this policy. So I have several entries that perform drops, accepts, and then finally if a route does not match any of the other entries, the default action is to drop it.

Prefix lists, are just groups of networks. For example the “Routed-Links”

In an effort to try to keep the routing tables clean as possible I, for example, block any of the /29 networks in the 10.1.100.0/24 blockwhich I have set up between pfSense and this Nokia router. The longer just means that anything that falls within the 10.1.100.0/24 range – that has a longer subnet mask, included here. My /29s are longer. I could define every /29 here, but that’s a lot more work, and then every time I would turn up a new VPRN I would have to update policies (which is not ideal).

The main points of interest for this post are the “direct” networks. Direct means any of the layer 3 interfaces that are configured on this router. In my first picture where I looked at the L3 interface, you can see VL66 and VL666. These are direct interfaces, and since I want to be able to have other hosts on my network reach these, and conversely, those networks be able to access other hosts, I am exporting these to pfSense.

Validation

If everything goes right, the peer should be established.

There is a ton of information to digest in this image:

1. We can see the BGP information that is local to this VPRN service
   • The AS is 64601 (we defined this in pfSense for the neighbor, and local in the Nokia VPRN service)
   • BGP is operationally up
   • The router ID for this service is 10.11.0.2
2. The first neighbor, we can see is the IP of pfSense, 10.1.100.9.
   • It has been up for 6+ days (when it is in the up state a addr family will be shown)
   • Also we can see that we have received 36 routes, 31 are active in the route table, and 5 are advertised (this is from the heading Rcv/Act/Sent)
   • Finally, the address family is IPv4

Now let’s looked at received routes, and advertised. This is called the RIB (Routing Information Base) – I cut this down as we do not need to see everything:

Routes that exist in the received RIB, you can think of as a “dumping ground.” Routes are populated here from whatever the peer has sent, but that does not mean that it will be the active route in the route table. An import policy may reject these, and/or other protocols which are advertising the same prefix to this router that may be preferred. If the logic and router determines the route should be installed in the route table because it is best, it will have a status code. Here looking at the static codes u*>i, means that it is used, valid, best, and IGP (mostly just a old carry over when EGP was a protocol before BGP). If we look in the route table we can see the default route exists via the protocol BGP and a next hop of pfSense.

Not forgetting the advertised routes, we can see the two 192.168.66.0 (and 128)/25 networks are there, with a few others which match other accepts in the policies which are being sent to pfSense.

Similarly, we can check pfSense for the information it has. First under Status > FRR. BGP, this will show you the simiar RIB information. Here we can see the two /25s (among others of course) that pfSense learns about from the various eBGP peers.

Under status > routes, shows the actual FIB (routing table) in pfSense. Again, we can see the routes are installed, and the RIB gave us that clue they would be as well because the status code is *>

This closes out this part of the discussion. At this point we have a eBGP peer up, and routes are being advertised between the routers. From this point, firewall rules can be assigned to allow/block traffic, and nat rules need to be created to allow the hosts to access the internet since they are private RFC 1918 space. In the next part, I will cover the firewall rules and NAT rules I have applied for this. Eventually, we will get to IPv6, which will be very similar but with a few slight changes.

Homelab’s logical network was a lot more work than I thought to put together, but it has been a great exercise to finally put this down on paper.

In part one, I am trying to give some background to the services I run, and how I setup my network. I will have a follow-up with some more hardware based details, and more specific vm/containers.

To start, I have some EOL Nokia network gear for my core routing and use pfSense for edge firewall/VPN/HA. I have logically broken up these services in IP VRFs (Virtual Routing and Forwarding) or a Nokia specific service – a VPRN (Virtual Private Routed Network)

This allows me to control access between the VPRN services, through a firewall (pfSense) instead of relying on ACLs on the Nokia router, or moving all my routing to pfSense, which would slow performance. This, in my opinion, is the best of both worlds. Routers route, and firewalls inspect. Now when I have services on multiple networks, within the same VPRN, I get the benefit of wireline bandwidth as it will route like a normal router. Otherwise, if traffic needs to cross VPRNs, it will need to go through pfSense where firewall rules are evaluated.

Without further ado here is the drawing!

Here is a quick run down of the services from top down:

• Offsite:
  • I run Truenas scale on an old server over a Wireguard site-to-site VPN running on a Raspberry Pi. This is not powered on all the time, and I use it to replicate snapshots from my main Truenas core’s SMB shares.
  • The other offsite location I have is a Synology NAS, to which I also replicate Truenas core’s SMB shares using rsync.
• I have Comcast business service and several static IPv4 IPs and IPv6 block.
• On my Edge, I have two pfSense VMs running on two hypervisors which provide HA failover for themselves and my internal network.
• Below is a list of the VPRN services:

VPRN	VPRN Name	Purpose	Notes
100	Inband/AD	Hypervisor gui mgmt, and AD Services	Critical AD services (DNS/DHCP/NPS)
101	Infrastructure services	Network hardware mgmt, and VMs running critical software	Syslog/snmp monitoring/stats
102	Unrestricted clients	Clients that have no rules blocking to other zones	TLS/PEAP WLAN clients or domain joined machines
103	Guest/IOT	Guest SSID and IOT devices
104	DMZ	Mix of internal DMZ and inbound services allowed	Web/mail/NTP pool members
105	Restricted Devices	Devices that have access blocked to AD/infrastructure services	clients that do not need full access to the internal network
106	File Services	Mostly for SMB access	Truenas scale SMB shares
108	Cameras	NVR and cameras	Blueiris NVR and IP cameras
109	Storj	Storj node operator VM

Hopefully that layout makes sense so you can understand how I went about separating these. Feel free to ask me questions! Now, let’s spend a little bit of time on the routing and how it looks:

I am using VPRN service 100 as an example. I send the default route via the eBGP neighbor (pfSense). Then if I look for another prefix that would be in another VPRN (this case 102) it does not route directly to that network; instead it goes to the firewall first even though both of these exist on the same router. A cleanup I want to do is to only send the default route from pfSense to these VPRNs, not on specific routes, but that will be a future blog post!

We can view the local configured subnets in the VPRN. These would be exported to pfSense so other VPRNs, and pfSense would know how to reach these hosts.

Finally, in pfSense, here is a quick shot of how the route table looks. First there is 192.168.8.0/25 that is in VPRN 100, and the 192.168.9.0/24 prefix that is in VPRN 102. Notice how they have two different eBGP next hop IPs which correspond to the interface on the Nokia router.

Part 2 I’ll dive into the FRR package in pfSense and the BGP setup on the Nokia router.

“Homelab” is a colloquial term to describe a home server (or entire datacenter, if you have the space) that hobbyists can play around with. I’d like to take this post to talk about the evolution of my Homelab. If you have spent any time on reddit.com/r/homelab, you will probably see people asking “what do you use this hardware for?” For everyone it is different.

I have played with technology since I was a kid, but it started to become more serious when I was in high school and working at a small IT support company. Around that time, I decided to start my own consumer-focused computer repair company to “compete” with Best Buy, etc.

Because I learned a lot through my job, I wanted to set up a file server just like we had at the IT support company so I could host all my common repair tools centrally. A customer had a old machine they were not using, and I was able to get a copy of windows server 2008 from high school for free with a promotion they were running to provide students with copies.

From there, Homelab continued to grow. I built a new machine where I could back up customers’ computers when I needed to format them, and also store family documents, games, movies etc.

From what I remember, at this point I had set up several machines (pictured below). One running Windows Server 2008 for mostly a file server, and an old Dell machine to be a firewall running Untangle. I believe the middle machine was running Windows 7 to be a media server that I would copy Blu-ray movies to, and also had a cable card in it to distribute live TV to other TV’s in the house with Windows Media Center.

When it came time for college, Homelab moved with me to an apartment. I found out that my local Goodwill store sold computer hardware. And, at times, you could find old enterprise gear there. I started to accumulate hardware and this is where (from taking the CCNA and learning more in college, and my first internship) I could actually start to work on network equipment at a basic level. I ran OSX on a Mac mini server model, and pfsense on a Watch Guard firewall. I believe I ran the switches just for studying for CCNA at the time.

My second apartment in college was a little bigger, so I continued to grow this more:

Fast forwarding to today – and what an upgrade! – this is my setup:

Nokia 7210 MXP. This serves as my main router. Many posts to come on this. The SAS-X is not used, just still in the rack 🙂

Some PDUs. The one is APC which is able to be controlled via a browser.

Next, a Comcast Modem, Mikrotik CRS310-1G-5S-4S+, which serves 10G connectivity to my two hypervisors and my Truenas disk storage for iSCSI traffic. Lastly, it has connectivity for my desktop PC at 2.5G.

Last of the networking equipment is my main switch, a Huawei S5300 that all my devices plug into. It has a funny origin story: this came from my local KMart when it closed. It has 4x10G ports, which independently serve as an uplink to the 7210 router, 10G connectivity for VMs hosted on the hypervisors, and 10G to the Mirkotik switch. It is POE and all my other devices within my home connect to it. In the next picture, when I purchased this home, one of my first projects was wiring cat6 everywhere (and now even to my shed).

The front of the rack of the compute/storage of the lab starts with the Rosewell RSV-L4500U that runs Truenas core. It has a mix of disks, which are 2x8TB and 4x10TB WD Red drives, and 6x1TB SSDs.

The HDDs are set up in a 3x mirrored VDEVs. The SSDs are also set up in their own mirrors. I use this as a very rudimentary tiered storage. My VMs that need performance, and are important, have their disks on one of the SSD pools, while the other VMs are running on the HDDs. Finally, any of my SMB shares also come from the HDD pool. It has a Intel Celron 64920 and 32GB of ECC ram.

Next is a Dell r330. It runs Proxmox, and just a few VMs that are critical + a GPU for encoding videos from my NVR. This has just a 8 core Xeon E3-1230 and 32gig of ram.

On to the next, is a Dell R630. This is my main server, and it has a pair of 12 core Xeon E5-2643v3 CPUs, and 128gig of RAM.

Lastly, is my main battery backup the APC UPS 1500, with a mgmt card. My VMs all use apcupsd, to learn the status of the battery, and I have auto shutdowns in place to theoretically shutdown all the hosts before the battery runs out.

I think everything has to start what it physically looks like! In my next post I plan to talk in depth about mostly routing, and the logical setup I use to segment my network.