The ATLAS Data Acquisition and High Level Trigger system

2016 Journal of Instrumentation  
A : This paper describes the data acquisition and high level trigger system of the ATLAS experiment at the Large Hadron Collider at CERN, as deployed during Run 1. Data flow as well as control, configuration and monitoring aspects are addressed. An overview of the functionality of the system and of its performance is presented and design choices are discussed. K : Control and monitor systems online; Data acquisition concepts; Online farms and online filtering; Trigger concepts and systems
more » ... are and software) Contents A Tables 128 B Definitions 129 C Acronyms 130 References 133 The ATLAS TDAQ Collaboration 144 1Except for a few test runs the bunch spacing was 50 ns for Run 1, at the highest luminosity this resulted in an average of 35 interactions per bunch-crossing. -3 - JINST 11 P06008 Calorimeter triggers miss EM Jet E T E T µ Muon trigger Detector front-ends L2 trigger Central trigger processor Timing, trigger and control distribution Calorimeters Muon detectors DAQ L1 trigger Regionsof-Interest Figure 3. Block scheme of the first level trigger. possible, to minimize the lengths of the cables used for forwarding the analog sums to the trigger and to minimize the time needed for sending the trigger accepts to the on-detector readout electronics. By choosing appropriate thresholds the L1 trigger has been operated during Run 1 with a maximum accept rate of 60-65 kHz, somewhat lower than the maximum design rate of 75 kHz, to prevent excessive dead time. The readout of the detector has been upgraded during the long shutdown of 2013 and 2014 to allow for 100 kHz accept rate. The L1 trigger can handle an input rate equal to the maximum bunch-crossing rate of 40 MHz. Its maximum latency is about 2.5 µs, i.e. smaller than the maximum of about 3 µs imposed by the depth of the on-detector buffer memories. This latency includes the transit times of signals between detectors and trigger system and the time required for sending the trigger accepts to the on-detector readout electronics. Data corresponding to events accepted by L1 are further analyzed by software running in computer farms to provide two further levels of triggering. The second level (L2) makes use of a fraction of the full precision detector data and reduces the rate further. The original design aimed for 3.5 kHz, although during Run 1 a maximum rate of about 5-6 kHz was allowed. The design value of the output rate of the last trigger level, which has been given the name "Event Filter" (EF), is about 200-300 Hz, during Run 1 the maximum output rate was about twice as high. The two levels of the software trigger are collectively known as the High Level Trigger (HLT). L1 accept decisions are distributed via the TTC (Timing, Trigger and Control) system [5] [6] [7] to the readout electronics, on-detector as well as off-detector, see figure 4. The Central Trigger Processor (CTP) of the first level trigger receives from the RF2TTC interface [4, 8] three clock signals with a frequency equal to 3564 times the revolution frequency of a bunch of 11.2 kHz, i.e. 40.078 MHz (one clock signal for each beam and one clock signal equal to the maximum collision rate), and two clock signals with a frequency equal to the revolution frequency. The CTP uses the LHC clock signal as clock for sending information via the Local Trigger Processor (LTP) modules [9], TTC-VME (TTCvi) modules [10] and TTCex laser transmitters [11]. -4 -2016 JINST 11 P06008 DAQ ROL ROL Figure 4. Overview of generation and distribution of timing and trigger signals by the Timing Trigger and Control (TTC) system and of the readout of the detector. Readout As illustrated in figure 4 the TTC information is received either by the front-end electronics directly via TTC receiver ASICs (TTCRx [12]), for examples see refs. [13] (LAr calorimeters) or [14] (MDTs), or indirectly via the ReadOut Drivers (RODs), for examples see refs. [15] (Pixels) or [16] (TRT). The RODs of the sub-detectors of which the front-end electronics connect directly to the TTC system also receive the TTC information. The RODs connect via the ReadOut Links (ROLs) to the DAQ (Data AcQuisition) system. Data are pushed from the front-end electronics upon the arrival of L1 accepts into the RODs and then forwarded via the ROLs. The L1 accept signals are accurately timed with respect to the associated bunch-crossings to facilitate reading out of data corresponding to the correct bunch-crossing. As indicated in the figure the TTC system is subdivided into "TTC partitions". For test and calibration purposes these partitions can be operated in parallel, with the LTP modules generating triggers instead of the CTP. The buffers of the DAQ system are grouped in the same way as the RODs from which they receive data. The rest of the DAQ system can be logically subdivided ("partitioned"), so that independent and simultaneous data acquisition for different TTC partitions is possible. The RODs are sub-detector specific and custom built, in most cases in the form of 9U VME cards. The buffers of the DAQ system, the ReadOut Buffers (ROBs), are also custom built but do not have sub-detector specific functionality. The RODs are considered to be part of the sub-detector electronics, while the ROBs are part of the DAQ system. The links (ROLs) connecting RODs to ROBs make use of the S-link protocol [17, 18] and consist of optical fibers. Each ROB connects to a single ROL. For most sub-detectors the maximum throughput per link is 160 MB/s, as originally specified, but the ROBs can handle up to 200 MB/s. Data sent across the links are checked for transmission errors. By means of an XON-XOFF flow protocol data transmission is halted when -5 - JINST 11 P06008 a ROB cannot receive additional data. Table 1 provides an overview of the TTC partitions, the number of ROLs per partition, as well as the amount of data produced per partition. For each L1 accept, information is output to the L2 system on "Regions of Interest" (RoIs) found in L1, i.e. geographical areas in the detector defined by the pseudorapidity η and the azimuthal angle φ of the objects which triggered L1. The L2 trigger subsequently requests the corresponding full precision data from the ROBs in which the data are stored. After analysis of the data received the trigger can also request additional data. Upon an accept of the L2 trigger, which also can be forced, e.g. for a calibration trigger, the Event Builder requests all event data from the ROBs and forwards these to the Event Filter. After acceptance the event is stored for further offline analysis. A so-called luminosity block is a set of events collected during a short time interval (1-2 minutes) for which the conditions for data taking were stable (approximately constant luminosity, no change in detector operating conditions). Together with the RoIs the luminosity block number, assigned by the L1 trigger, is also communicated to the L2 system, as trigger conditions may depend on it. The luminosity block number is stored in the event data forwarded to the Event Filter by the Event Builder. An overview of the ATLAS electronics can be found in ref. [19]. Identification of events and data format The front-end electronics send event data, via sub-detector specific links, to the RODs. The format and organization of these data are sub-detector specific as well. Event data are associated with an L1 identifier (L1Id) and a bunch-crossing identifier (BCId). At the start of a run all bits of the L1Id are set to 1. The L1Id is incremented upon reception of the L1 accept signal sent via the TTC system, so the L1Id of the first event in a run is 0. L1 accept (L1A) signals and messages are encoded by the TTC system using one of the LHC clock signals (by means of Biphase Mark encoding [5, 6, 12] ). This clock is recovered by the TTC receiver ASICs and used for incrementing the BCId. The latter is reset to 0 after a Bunch Counter Reset (BCR) command is received, which is sent once per orbit period via the TTC system. The L1Id is reset to its start value (all bits 1) upon receipt of an Event Counter Reset (ECR) command. ECR commands are sent every few seconds to minimize the probability that incorrect L1Ids occur owing to missed L1A signals or, if this happens, to minimize the number of incorrect L1Ids. For each event the BCIds of all fragments should be identical, which allows a check of the correctness of the L1Ids. The RODs assemble event fragments, each with a header and trailer as defined in ref. [20], from the data received from the front-end electronics. The header consists of the following nine 32-bit words: start of header marker, the size of the header (always 9), a number indicating the version of the data format of the fragment, an identifier of the ROD, the number of the run, the extended L1Id, the BCId, the L1 trigger type and finally a word reserved for sub-detector specific information. The extended L1Id stores in its least significant 24 bits the L1Id and in its 8 most significant bits the ECR identifier (ECRId), which is obtained by counting the ECR commands (starting from 0). The latter counting is done by the RODs. The counting of L1As and LHC clock cycles (for forming the BCIds) is done in the TTC receiver ASICs, but may also be done independently in the RODs and in the front-end electronics, which permits an additional check for incorrect L1Ids and BCIds in the RODs. Each event fragment may also contain status information, the last word of the three word trailer of each fragment indicates whether the status information precedes or follows the event data -6 - Inter-process communication In view of the size and the distributed nature of the ATLAS TDAQ system support for inter-process communication by highly scalable distributed middleware with excellent performance is required. Because of the long lifetime of the ATLAS experiment the middleware has to be easily extensible and maintainable. The requirements have been met by adopting the Common Object Request Broker Architecture (CORBA) standard of the Object Management Group (OMG) [32] and making use of the omniORB [33] (for C++) and JacORB [34] (for Java) implementations of the Object Request Broker (ORB). CORBA has one essential weak point: the complexity of the communication model and of the communication API. This complexity is due to the flexibility offered by CORBA to developers of distributed applications. To overcome this issue, a light-weight software wrapper called IPC has -13 -2016 JINST 11 P06008 -2016 JINST 11 P06008 Internal connections: RAM CPU FLASH CPLD FPGA PHY BRIDGE GE PCI J T A G RAM 2
doi:10.1088/1748-0221/11/06/p06008 fatcat:wmldwzaqqnhrdgkjn3rdm4mvey