FreeBSD 13.2 [RC1]

Santiago Martinez1 Comment

NIC Tested with SRIOV and Bhyve PCI-PASS

VendorModelPortsDriverSRIOV TestedNum of PF Bhyve PCI-PASSVF Attached to OS
IntelXL7102×10/40Gixl from portsYES64WORKSWORKS
IntelX7222x10G BaseTixl from portsYES64WORKSWORKS
IntelXXV7102x25G(@10G)ixl from portsYES64WORKSWORKS
Inteli3504x10GTBCTBCTBCTBCTBC
BCMBCM574162x10G -CRAPin kernel––––

Issues:

VendorModelPortsDriversProblem
IntelXXV7102×10/25GKernel/BaseInterfaces only work at 10G, ( well better to put the right transceiver 🙂 )
IntelXXV7102×10/25GIntel pkgInterfaces only work at 10G, ( well better to put the right transceiver 🙂 )
BCMBCM574162x10GKernel/BaseDriver is broken, VLAN cannot be used. Patch is required to make them work.

Test environment for XXV710: ( will do others when time permits).

The Server is a super micro, the machine is running FreeBSD 13.2 RC1 with FRR 7.X release. The Spirent TG is just sending bulk traffic ( 100 src, 100 dst, 1:1, 512b, 9.5Gbps) not to benchmark the FreeBSD but to make sure the cards don’t play funny games. The interfaces on both routers and servers are untagged. No drops or errors were found.

This is the output of the tester:

Test environment for XL710:

The Server is a supermicro, the machine is running FreeBSD 13.2 RC1. The 40G interfaces are connected to an MX1004. The MX has been configured with a bridge domain which contains interfaces et-0/1/11 and et-0/1/6.

HYPER01-B2B {
    interface et-0/1/11.0;
    interface et-0/1/5.0;
}
et-0/1/5 {
    encapsulation ethernet-bridge;
    unit 0;
}
et-0/1/11 {
    encapsulation ethernet-bridge;
    unit 0;
}

The following diagram depicts the physical network topology:

Interface IXL0 has been configured with 64 VF, without passthrough, hence the host OS can see 64 iavf interfaces. Interface IXL1 has been configured with 64 VF, with passthrough, One of the VF has been passed into the VM.

pkt-gen is running on both the host OS and virtual machine. Interfaces are reaching 39.2@512b.

Test environment for iavf (passed by Bhyve),

The VM is running FreeBSD 13.2 RC1 with FRR 7.X release, bhyve is passing one PF from each NIC (ixl0 and ixl1), those are present as iavf0 and iavf1. The Spirent TG is just sending bulk traffic ( 100 src, 100 dst, 1:1, 512b, 9.5Gbps). No drops or errors were found.

Juniper vMX on Bhyve

Santiago MartinezLeave a comment

The post shows how to quickly setup a Virtual MX under bhyve, using netmap to interconnect the virtual routers. Control and data-plane VM are connected using in-kernel FreeBSD bridge. In this case the release is : 20.4R3-S1.3

For the control plane VM:

loader="uefi-csm"
cpu=2
memory=4G
wired_memory="yes"
virt_random="1"
graphics="yes"

network0_type="virtio-net"
network0_switch="BR_DCN_VF"

network1_type="virtio-net"
network1_switch="VMX01_INT"

disk0_type="virtio-blk"
disk0_name="vmx-re-image.img"

disk1_type="virtio-blk"
disk1_name="vmx-re-hdd.img"

disk2_type="virtio-blk"
disk2_name="vmx-re-usb.img"

uuid="75b768f5-7398-11ec-b978-3cecef32c22c"
network0_mac="58:9c:fc:05:c3:54"
network1_mac="58:9c:fc:0f:bd:71"

For the Virtual forwarding plane image, the config looks like:

HYPER01-B2B {
    interface et-0/1/11.0;
    interface et-0/1/5.0;
}
et-0/1/5 {
    encapsulation ethernet-bridge;
    unit 0;
}
et-0/1/11 {
    encapsulation ethernet-bridge;
    unit 0;
}

Juniper vSRX3.0 under Bhyve

Santiago MartinezLeave a comment

Before we start, I want to explain why I wanted to run vSRX on top of FreeBSD + bhyve, the reason behind is that I already received multiple questions and weird faces about it.

Well, the truth is that it is just personal. I have spent 20+ years working with Linux every day, from PC based routers, wireless CPE based on linux and many other horrible things. It was great and fun, but most important everything was simple. This simplicity and coherence is something that I cannot find on Linux distribution any more.

On the other hand, I have been using FreeBSD for a few years now, mostly for lab environments and home usage, until this year that I have a perfect fit on one of my employer’s customer network. We wanted a stable, performant OS that allow us to jails some network services ( DNS, Radius, Syslog, etc) without all the mess from Linux and friends.

During the network deployment phase, we faced delays on hardware delivery (yeah, thanks covid-19, you really suck) and we couldn’t get the Firewall appliances to protect our servers/services in time. To avoid delaying thing we decided to deploy a Firewall VNF (Virtual Network Function). To make the story short, I ended deploying the Juniper Virtual SRX (vSRX3.0) using FreeBSD and Bhyve.

As it is publicly know, Juniper Junos (Juniper network operating system that runs the control plane of most Juniper devices) is based on FreeBSD. While there are some version of vSRX that run a nested virtual machine mode (Main VM runs Linux with a process for data-plane and a nested VM with Junos for control-plane), the one utilized here is called vSRX3.0, and there is no nested VMs. It’s just runs Junos (based on BSD) with the data plane implemented using DPDK.

As we didn’t need high capacity for this deployment, it was just good enough to use virtio-net based ethernet devices instead of SR-IOV or PCI-Passthrough. Hence, as bhyve support virtio-net and virtio-block we shouldn’t face too many issues running vSRX.

Our assumptions were correct, we only needed, the vSRX3.0 image (yeah obviously) and as the image comes as qcow2, is required to convert it to raw format. After doing this, the image can be launched as any other machine, with the minor caveat that the loader should be set to uefi-csm.

This is the configuration required to boot a vSRX3.0 on FBSD 12.2 + Bhyve + vm-bhyve as virtualization management).

loader="uefi-csm"
cpu=4
memory=8G

# First Ethernet will be allocated to management (fxp0).
network0_type="virtio-net"
network0_switch="vm-BR-PRIVATE"

# second Ethernet will be allocated to ge-0/0/0.
network1_type="virtio-net"
network1_switch="bridge1"

# second Ethernet will be allocated to ge-0/0/1.
network2_type="virtio-net"
network2_switch="bridge2"

# here is where we indicate to use the image that we converted to raw.
disk0_type="virtio-blk"
disk0_name="disk0.img"

So, far we have not faced any issues and its rock solid.

Hope you enjoy it, even though it was not technical at all.

Santi