NVMe and PLX bridges (Chapter 1)
One of my current projects involves rebuilding my home storage solution. As-is, my NAS setup uses a TrueNAS Core virtual machine with an LSI 9200-8e passed through to the VM which is multi-pathed to an MD1200 Powervault. The pool topology is pretty simple with a 12 disk array in 6 single-mirror VDEVs. Performance and resiliency has been totally acceptable, but concerns around power draw and lack of meaningful upgrade path (short of playing musical chairs with higher capacity disks) have been simmering for a year or so now. For the next iteration, I’ve decided to move from a virtualized solution with external storage to using TrueNAS Scale on bare metal with onboard disks to cut down on as many moving parts as possible (figuratively in terms of hypervisor shenanegains and literally in terms of cabling and rack space)
The crux of this writeup though comes from a combination of a few things. First, given that I’m planning on occupying a few slots with GPUs and networking cards, space will be at a premium for the optane m.2 NVMe drives I’m planning to use in a metadata VDEV. These P1600X drives use a PCIe 3.0 x4 connection which would be wasted on the r720’s four x8 or two x16 slots (this is the xd config, so we’re shy an additional x8 slot).
Compounding this issue, the 12th generation Dell poweredge servers that this r720 is a part of, were the final without PCIe bifurcation; meaning we cannot “split” the slot for direct connectivity.
How then would we go about efficiently making use of both the physical slots and their connectivity? a PLX bridge!
Before we begin, I do want to explain that these drives are not intended or expected to be the fastest thing on the block. They’ll be pulling long term duty as metadata storage devices for a new storage pool composed of spinning disks. All these need to do is accelerate pool operations that HDDs are not optimized for:
The ZFS Special Device can store meta data (where files are, allocation tables, etc) AND, optionally, small files up to a user-defined size on this special vdev. This would be a great use-case for solid-state storage.
This will dramatically speed up random I/O and cut, by half, the average number of spinning-rust I/Os needed to find and access a file. Operations like
ls -lrwill be insanely way faster.
This ServeTheHome Thread inventories a few switching card options focusing on those based on PEX and ASM controllers
Bifurcation vs. Switching?
PCIe bifurcation effectively lets multiple devices share the same physical address, but this is dependent on support both from the motherboard and BIOS. A PCIe Switch or bridge will occupy the same physical space, but present itself as a single device to the root controller for addressing during initialization. If the motherboard and bios do not both support bifurcation (giving a heads up that more than one device can be present in that location), the most common result is only the “first” PCIe device will be recognized.
Bifurcation preserves all performance of the connected device as they’re configured to directly use the connections as though they were isolated devices. Switching requires routing traffic first through the switch and then to the target devices. This does add processing overhead, but has much better compatibility with older (or consumer) boards that lack support for bifurcation.
One huge caveat with these bridges however is looking at the upstream and downstream interfaces and their inherent limitations compared to native connections.
Performance impacts
For our testing, we’ll be using the system we upgraded in another post (Dell Poweredge R720 with a pair of Xeon E5-2697v2 processors)
These results will not be representative for much outside of this unit, but the combination of lower clock speed on the CPU, PCIe gen3 slots, and slower memory may mean we bottleneck somewhere else in standard-use before the storage.
To be clear, there will be performance left on the table (especially given that we’ll be using optane drives for this experiment), but the goal is more to see if sacrificing an extra PCIe x8 slot for a single NVMe drive is a worthwhile tradeoff for the difference in speed or latency
To prep for this, I purchased two PCIe switches, one based on the ASM1812 controller, and one using the successor ASM2812. (un)fortunately the seller I purchased the lower spec card from sent along a second ASM2812 card, so we’ll only be testing on that platform.
Looking at the specs, the 2812 cards (despite being physically x8) have a “1, 2, or 4 lane” connection upstream at PCIe gen 3.0 speeds. Downstream it’s capable of exposing 8 lanes at PCIe 3.1 though which should accomodate the optane drives, but this will naturally be limited by the connection to the root device.
Testing!
This methodology is based on a likeminded setup outlined in a LevelOneTechs Thread and all credits go to the author for inspiring this quick set of tests.
we’ll be making use of fio, which is bundled with the TrueNAS Scale version we’re using (v22.12.2) using the following parameters:
1 | fio --bs=128k --direct=1 --directory=/mnt/foo --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based |
For pools with multiple drives, I’ll be arranging them in single-mirror single VDEV arrangements (or single drive ‘control’ pools) with stock settings.
one interesting point here too is that while we have 4 optane drives attached, two are direct, and two are switched. All four drives though are visible “normally”
on a lspci --vv | grep "Intel Corporation Device 2525" all devices reported transparently
(truncated version)
1 | 04:00.0 Non-Volatile memory controller: Intel Corporation Device 2525 (prog-if 02 [NVM Express]) |
for a switched drive: lspci -vv -s 47:00.0 we even see the expected link state (minus the switch bottleneck)
1 | LnkSta: Speed 8GT/s (ok), Width x4 (ok) |
Directly attached Pool
The drives in this pool are not making use of the PLX bridge card. They are on individual carrier boards and are directly connected at full PCIe 3x4 links
1 | Jobs: 6 (f=6): [m(1),E(1),m(1),_(1),m(1),_(1),m(1),_(1),m(1),E(2),m(1)][100.0%][r=1204MiB/s,w=1176MiB/s][r=9635,w=9406 IOPS][eta 00m:00s] |
Not a terribly great start, but lets look at a single drive:
1 | Jobs: 12 (f=12): [m(12)][100.0%][r=1207MiB/s,w=1180MiB/s][r=9655,w=9439 IOPS][eta 00m:00s] |
Switch Connected Pool
1 | Jobs: 12 (f=12): [m(12)][100.0%][r=951MiB/s,w=948MiB/s][r=7606,w=7580 IOPS][eta 00m:00s] |
around 30% worse, but something else seems off…
a single drive for good measure:
1 | Jobs: 12 (f=12): [m(12)][100.0%][r=329MiB/s,w=335MiB/s][r=2629,w=2677 IOPS][eta 00m:00s] |
Conclusion?
Not looking great in either case… While these drives are technically only rated for 1760MB/s read and 1050MB/s write, their 4k blocksize IOPS should be closer to 400k and 250k respectively.
This may be an issue with the initial pool configuration or testing at a blocksize of 128k
More to come in chapter two!