Blog

Homelab’s logical network was a lot more work than I thought to put together, but it has been a great exercise to finally put this down on paper.

In part one, I am trying to give some background to the services I run, and how I setup my network. I will have a follow-up with some more hardware based details, and more specific vm/containers.

To start, I have some EOL Nokia network gear for my core routing and use pfSense for edge firewall/VPN/HA. I have logically broken up these services in IP VRFs (Virtual Routing and Forwarding) or a Nokia specific service – a VPRN (Virtual Private Routed Network)

This allows me to control access between the VPRN services, through a firewall (pfSense) instead of relying on ACLs on the Nokia router, or moving all my routing to pfSense, which would slow performance. This, in my opinion, is the best of both worlds. Routers route, and firewalls inspect. Now when I have services on multiple networks, within the same VPRN, I get the benefit of wireline bandwidth as it will route like a normal router. Otherwise, if traffic needs to cross VPRNs, it will need to go through pfSense where firewall rules are evaluated.

Without further ado here is the drawing!

Here is a quick run down of the services from top down:

• Offsite:
  • I run Truenas scale on an old server over a Wireguard site-to-site VPN running on a Raspberry Pi. This is not powered on all the time, and I use it to replicate snapshots from my main Truenas core’s SMB shares.
  • The other offsite location I have is a Synology NAS, to which I also replicate Truenas core’s SMB shares using rsync.
• I have Comcast business service and several static IPv4 IPs and IPv6 block.
• On my Edge, I have two pfSense VMs running on two hypervisors which provide HA failover for themselves and my internal network.
• Below is a list of the VPRN services:

VPRN	VPRN Name	Purpose	Notes
100	Inband/AD	Hypervisor gui mgmt, and AD Services	Critical AD services (DNS/DHCP/NPS)
101	Infrastructure services	Network hardware mgmt, and VMs running critical software	Syslog/snmp monitoring/stats
102	Unrestricted clients	Clients that have no rules blocking to other zones	TLS/PEAP WLAN clients or domain joined machines
103	Guest/IOT	Guest SSID and IOT devices
104	DMZ	Mix of internal DMZ and inbound services allowed	Web/mail/NTP pool members
105	Restricted Devices	Devices that have access blocked to AD/infrastructure services	clients that do not need full access to the internal network
106	File Services	Mostly for SMB access	Truenas scale SMB shares
108	Cameras	NVR and cameras	Blueiris NVR and IP cameras
109	Storj	Storj node operator VM

Hopefully that layout makes sense so you can understand how I went about separating these. Feel free to ask me questions! Now, let’s spend a little bit of time on the routing and how it looks:

I am using VPRN service 100 as an example. I send the default route via the eBGP neighbor (pfSense). Then if I look for another prefix that would be in another VPRN (this case 102) it does not route directly to that network; instead it goes to the firewall first even though both of these exist on the same router. A cleanup I want to do is to only send the default route from pfSense to these VPRNs, not on specific routes, but that will be a future blog post!

We can view the local configured subnets in the VPRN. These would be exported to pfSense so other VPRNs, and pfSense would know how to reach these hosts.

Finally, in pfSense, here is a quick shot of how the route table looks. First there is 192.168.8.0/25 that is in VPRN 100, and the 192.168.9.0/24 prefix that is in VPRN 102. Notice how they have two different eBGP next hop IPs which correspond to the interface on the Nokia router.

Part 2 I’ll dive into the FRR package in pfSense and the BGP setup on the Nokia router.

“Homelab” is a colloquial term to describe a home server (or entire datacenter, if you have the space) that hobbyists can play around with. I’d like to take this post to talk about the evolution of my Homelab. If you have spent any time on reddit.com/r/homelab, you will probably see people asking “what do you use this hardware for?” For everyone it is different.

I have played with technology since I was a kid, but it started to become more serious when I was in high school and working at a small IT support company. Around that time, I decided to start my own consumer-focused computer repair company to “compete” with Best Buy, etc.

Because I learned a lot through my job, I wanted to set up a file server just like we had at the IT support company so I could host all my common repair tools centrally. A customer had a old machine they were not using, and I was able to get a copy of windows server 2008 from high school for free with a promotion they were running to provide students with copies.

From there, Homelab continued to grow. I built a new machine where I could back up customers’ computers when I needed to format them, and also store family documents, games, movies etc.

From what I remember, at this point I had set up several machines (pictured below). One running Windows Server 2008 for mostly a file server, and an old Dell machine to be a firewall running Untangle. I believe the middle machine was running Windows 7 to be a media server that I would copy Blu-ray movies to, and also had a cable card in it to distribute live TV to other TV’s in the house with Windows Media Center.

When it came time for college, Homelab moved with me to an apartment. I found out that my local Goodwill store sold computer hardware. And, at times, you could find old enterprise gear there. I started to accumulate hardware and this is where (from taking the CCNA and learning more in college, and my first internship) I could actually start to work on network equipment at a basic level. I ran OSX on a Mac mini server model, and pfsense on a Watch Guard firewall. I believe I ran the switches just for studying for CCNA at the time.

My second apartment in college was a little bigger, so I continued to grow this more:

Fast forwarding to today – and what an upgrade! – this is my setup:

Nokia 7210 MXP. This serves as my main router. Many posts to come on this. The SAS-X is not used, just still in the rack 🙂

Some PDUs. The one is APC which is able to be controlled via a browser.

Next, a Comcast Modem, Mikrotik CRS310-1G-5S-4S+, which serves 10G connectivity to my two hypervisors and my Truenas disk storage for iSCSI traffic. Lastly, it has connectivity for my desktop PC at 2.5G.

Last of the networking equipment is my main switch, a Huawei S5300 that all my devices plug into. It has a funny origin story: this came from my local KMart when it closed. It has 4x10G ports, which independently serve as an uplink to the 7210 router, 10G connectivity for VMs hosted on the hypervisors, and 10G to the Mirkotik switch. It is POE and all my other devices within my home connect to it. In the next picture, when I purchased this home, one of my first projects was wiring cat6 everywhere (and now even to my shed).

The front of the rack of the compute/storage of the lab starts with the Rosewell RSV-L4500U that runs Truenas core. It has a mix of disks, which are 2x8TB and 4x10TB WD Red drives, and 6x1TB SSDs.

The HDDs are set up in a 3x mirrored VDEVs. The SSDs are also set up in their own mirrors. I use this as a very rudimentary tiered storage. My VMs that need performance, and are important, have their disks on one of the SSD pools, while the other VMs are running on the HDDs. Finally, any of my SMB shares also come from the HDD pool. It has a Intel Celron 64920 and 32GB of ECC ram.

Next is a Dell r330. It runs Proxmox, and just a few VMs that are critical + a GPU for encoding videos from my NVR. This has just a 8 core Xeon E3-1230 and 32gig of ram.

On to the next, is a Dell R630. This is my main server, and it has a pair of 12 core Xeon E5-2643v3 CPUs, and 128gig of RAM.

Lastly, is my main battery backup the APC UPS 1500, with a mgmt card. My VMs all use apcupsd, to learn the status of the battery, and I have auto shutdowns in place to theoretically shutdown all the hosts before the battery runs out.

I think everything has to start what it physically looks like! In my next post I plan to talk in depth about mostly routing, and the logical setup I use to segment my network.