The New Event Builder of the LHCb Experiment: Network Optimisations for the Online Cluster

The LHCb experiment at CERN LHC collider investigates the c- and b- quarks flavour sector of the standard model of particle physics. For the Run III of LHC (2022-2025), LHCb has been upgraded to withstand proton-proton collisions of 14 TeV centre-of-mass energy up to an istantaneous luminosity of $2\times10^{33}$ cm$^{-2}$s$^{-1}$. The entire online data acquisition chain has been redesigned to fully exploit the higher luminosity. The upgraded system is designed to perform full readout and online reconstruction of the full 40 MHz rate without any low-level hardware trigger, with an expected aggregated throughput of $\sim$32 Tb/s. This requires a state-of-the-art design in the networking and computing sides. The online computer cluster is built using ~170 servers, interconnected via a 200 Gb/s InfiniBand-based network. The data from the LHCb sub-detectors are fragmented throughout all the servers and have to be assembled before the event reconstruction can take place; this process is known as Event Builder. The first design of the Event Builder software used MPI for the parallel network communication, which presents an underperforming warmup phase and a suboptimal handling of dead processes, leading to potential dead time and consequent data loss. The solution was the design of a custom network library based on the InfiniBand Verbs API called InfiniBuilder. InfiniBuilder has been designed as a drop-in replacement for MPI calls, which can be further customised by leveraging the hardware capabilities of InfiniBand. InfiniBuilder resolves the issues reported with MPI and is capable of achieving better performance in throughput tests. The results of the benchmarks have shown that InfiniBuilder have achieved the design goals, leading to its integration in the current version of the LHCb Event Builder software, which will enable efficient data taking during Run III.