Storage Spaces Direct (S2D) - Poor write performance with 5 nodes with 24 Intel P3520 NVME SSDs each over 40Gb IB network

Need a little help with my S2D cluster which is not performing as I had expected.

Details:

5 x Supermicro SSG-2028R-NR48N servers with 2 x Xeon E5-2643v4 CPUs and 96GB RAM

Each node has 24 x Intel P3520 1.2TB NVME SSDs

The servers are connected over an Infiniband 40Gb network, RDMA is enabled and working.

All 120 SSDs are added to S2D storage pool as data disks (no cache disks). There are two 30TB CSVs configured with hybrid tiering (3TB 3-way mirror, 27TB Parity)

I know these are read intensive SSDs and that parity write performance is generally pretty bad but I was expecting slightly better numbers then I'm getting:

Tested using CrystalDiskMark and diskspd.exe

Multithreaded Read speeds: < 4GBps (seq) / 150k IOPs (4k rand)

Singlethreaded Read speeds: < 600MBps  (seq) 

Multithreaded Write speeds: < 400MBps  (seq) 

Singlethreaded Write speeds: < 200MBps (seq) / 5k IOPS (4k rand)

I did manage to up these numbers by configuring a 4GB CSV cache on the CSVs and forcing write through on the CSVs:

Max Reads: 23GBps/500K IOPs 4K IOPS, Max Writes:2GBps/150K 4KIOPS

That high read performance is due to the CSV cache which uses memory. Write performance is still pretty bad though. In fact it's only slight better than the performance I would get for a single one of these NVME drives. I was expecting much better performance from 120 of them!

I suspect that the issue here is that Storage Spaces is not recognising that these disks have PLP protection which you can see here:

Get-storagepool "*S2D*" | Get-physicaldisk |Get-StorageAdvancedProperty

FriendlyName          SerialNumber       IsPowerProtected IsDeviceCacheEnabled
------------          ------------       ---------------- --------------------                   
NVMe INTEL SSDPE2MX01 CVPF7165003Y1P2NGN            False                     
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NVMe INTEL SSDPE2MX01 CVPF717000JR1P2NGN            False                     
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
NVMe INTEL SSDPE2MX01 CVPF7254009B1P2NGN            False                     
WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.

Any help with this issue would be appreciated.

Thanks.