Beruflich Dokumente
Kultur Dokumente
AbstractThe upgrades of the Belle experiment and the KEKB and four layers silicon strip detector. The silicon vertex de-
accelerator aim to increase the data set of the experiment by a tector is surrounded by the central drift chamber which recon-
factor 50. This will be achieved by increasing the luminosity of the structs the tracks of the charged particles and measures its mo-
accelerator which requires a signicant upgrade of the detector. A
new pixel detector based on DEPFET technology will be installed mentum. The particle identication system is represented by
to handle the increased reaction rate and provide better vertex res- the focusing aerogel ring imaging Cherenkov detector and the
olution. One of the features of the DEPFET detector is a long in- time-of-propagation counters. The energy of the electrons and
tegration time of s, which increases detector occupancy up to photons is measured in the high resolution CsI(Tl) electromag-
3%. The detector will generate about 22 GB/s of data. An FPGA- netic calorimeter. The outermost detector of the experiment de-
based two-level read-out system, the Data Handling Hub, was de-
veloped for the Belle II pixel detector. The system consists of 40 tects long lived kaons and muons with the resistive plate cham-
read-out and eight controller modules. All modules are built in bers and scitillator detectors.
TCA form factor using Xilinx Virtex-6 FPGA and can utilize up
to 4 GB DDR3 RAM. The system was successfully tested in the II. PIXEL DETECTOR
beam test at DESY in January 2014. The functionality and the ar- The silicon pixel detector will be installed as the innermost
chitecture of the Belle II Data Handling Hybrid system as well as
the performance of the system during the beam test are presented detector layer in Belle II. The detector is an active-pixel de-
in the paper. tector built using DEpleted P-channel Field Effect Transistor
(DEPFET) [3] technology. This technology allows us to build
Index TermsData acquisition, eld programmable gate arrays,
a detector with a very low material budget ( per layer
high energy physics instrumentation computing.
[4]), which reduces multiple scattering and provides spatial res-
olution below m, improving vertex resolution.
I. INTRODUCTION The matrix of a pixel detector module, called a half ladder,
consists of DEPFET pixels. The integration time of a
half ladder is s. Two half ladders are glued on the far edge
of the silicon frame together and form a mechanical module, a
T HE Belle II experiment [1] is a successor of the Belle ex-
periment, the B-Factory located in the High Energy Ac-
celerator Research Organization, KEK, in Tsukuba, Japan. The
ladder. The pixel detector composed of 20 mechanical modules
is arranged in two cylindrical layers around the interaction point
of the accelerator. The inner layer consists of eight modules with
experiment will focus on precise measurements of the avor
an average radius of 14 mm and sensitive length of 90 mm. The
physics reactions at low energies (10 GeV) to observe signa-
outer layer consists of 12 modules with an average radius of
tures of new particles and obtain deviations from the Standard
22 mm and sensitive length of 123 mm [4].
Model predictions. The upgrade of the Belle experiment aims
The half ladder forms the read-out unit. The DEPFET matrix
to increase the recorded data set of the B-Factory by a factor 50.
is operated in a rolling shutter mode by three kinds of appli-
Besides the upgrade of the accelerator SuperKEKB [2] that will
cation-specic integrated circuits (ASICs) bump bonded on the
bring an increase in the luminosity by adopting a nano-beam
silicon frame of the half ladder: six SwitcherB, four drain cur-
collision scheme at the interaction point, the signicant detector
rent digitizers (DCD) and four data handling processors (DHP).
upgrade is required to cope with the increased reaction rate.
The pixels in the matrix are controlled by two lines: gate line
The Belle II detector consists of the following subdetectors.
activates transistors and the clear line erases the charge accu-
The innermost part of the Belle II is the silicon vertex detector.
mulated in the internal gate of the DEPFET. The gate and clear
The detector consists of the two layers silicon pixel detector
lines of the DEPFET matrix are steered by the SwitcherB [5]
ASIC. These chips are connected in the daisy chain that propa-
Manuscript received June 15, 2014; revised December 22, 2014; accepted
gates the externally generated detector read-out sequence. The
April 10, 2015. Date of publication May 21, 2015; date of current version June
12, 2015. This work was supported in part by the European Commission under gate and clear lines of each four rows in the matrix are con-
the FP7 Research Infrastructures project AIDA under Grant 262025, by the trolled by the same SwitcherB channel in a so-called 4-fold read
German Ministry of Education and Research, Excellence Cluster Universe, and
out: four detector rows are digitized in the same read-out cycle.
by the Maier-Leibnitz-Laboratory.
The authors are with the Physikdepartment E18, Technische Univer- The read-out cycle consists of activating four detector rows and
sitt Mnchen, 85748 Garching, Germany (e-mail: dmytro.levit@tum.de; clearing rows at the end of the cycle. The period of the read-out
igor.konorov@cern.ch; daniel.greenwald@tum.de; stephan.paul@tum.de).
cycle is 100 ns.
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org. The digitization of the drain current is performed in the DCD
Digital Object Identier 10.1109/TNS.2015.2424713 ASIC [6]. A DCD has 256 8-bit analog to digital converter
0018-9499 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1034 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 62, NO. 3, JUNE 2015
that the hits are already ordered in incremental row number se-
quence. The core of the algorithm is implemented as a chain of
nite state machines, representing one detector row. Each state
machine processes hits of two neighboring columns. Therefore
the algorithm has 32 state machines for 64 detector columns
corresponding to the number of columns handled by one DHP
chip. The clustering process receives one hit per clock. The
state machine checks the status of its direct neighbours and if
one of the neighbours is active, then the state machine takes its
cluster number. Otherwise the next free cluster number is used.
When there are two active neighbours, the clusters are merged
and the state machine assigns the smallest cluster number to the
hit. To remap preliminary cluster numbers to nal ones there is
a table that stores the new cluster number at the address of the
old cluster number. After the preliminary cluster numbers have
been assigned to all hits, a remapping of the cluster numbers is
performed in the order of pixels stored in the hit FIFO.
Since only the smallest cluster numbers are propagated, the Fig. 6. Sub-event builder algorithm.
cluster number remapping requires at most two look-ups to
the table to nd the true cluster number. Once the true cluster
number is found, the value is written into the look-up table to DHEDHC link. In the case if a DHEDHC link is not estab-
speed up the following remapping. Since the remapping algo- lished, the framing state machine ignores the corresponding
rithm is deterministic and the hits are received in the correct intermediate FIFO, generates empty frame for this link to
order, the data stream can be processed in a pipeline. Finally, preserve subevent data structure and switches to reading the
four data streams are merged together by remapping clusters next FIFO. The state machines enclose data frames with the
that are located on the frame border. same event number by a header and a trailer frame, thereby
The verication of the algorithm was performed in rmware building a subevent that contains information from up to ve
and in software in parallel. The data in the pixel detector data detector modules. Finally, the complete subevents are written
format were generated by the software running on a dedicated into a large FIFO that is implemented in the external memory.
PC. Then the data were loaded into FPGA over Ethernet and The data from the external FIFOs are directly sent to outgoing
sent to the software for verication. The clustered data were links.
downloaded from FPGA and compared with the results of the Since the data frames already contain event number infor-
cluster recovery in hardware. This test proved the ability of the mation, the consistency of the subevent is directly checked in
cluster recovery algorithm in FPGA to recover the clusters in the framing state machines. The subevent-builder core also pro-
the data stream correctly. vides the possibility to mask incoming and outgoing channels.
The data ready to be sent downstream are formated into In this case, the round-robin algorithm is altered to bypass inac-
DHE frames that provide service information on the data type tive channels.
and event number. The DHE frames are then sent to the DHC
module for subevent building. E. Slow Control
The slow control of the system is implemented as an abstrac-
D. Subevent Builder tion layer between high-level system-control algorithms in Ex-
The DHC performs online subevent building in rmware. perimental Physics and Industrial Control System (EPICS) and
The data ow in the subevent building algorithm is shown in low-level hardware registers. The DHE and DHC are controlled
Fig. 6: the received frames are buffered in a Xilinx Block RAM directly over Ethernet using a UDP-based protocol IPBus [15].
FIFO [14] and then written into one of the four intermediate The front-end ASICs are connected into a JTAG chain and are
FIFOs, implemented as a Xilinx Block RAM FIFO as well, that controlled by a hardware master in the DHE. A middleware li-
correspond to the four outgoing links. The active intermediate brary was developed that translates IPBus and JTAG protocols
FIFO is determined by a round-robin algorithm. Overow of the for the high-level EPICS slow-control network.
buffering FIFO is prevented by using the Native Flow Control The ATCA carrier board provides only a single Ethernet con-
feature of the Aurora protocol. This feature allows us to sus- nection for the whole read-out system. All modules on the car-
pend data transmission over the Aurora link until the ll level rier board are accessible by a unique IP address. This is done
of the FIFO falls under a predened threshold. During this time by implementing a simple Ethernet hub in the FPGA logic of
period the data are buffered in the external memory of the DHE the DHC to share this connection with the DHE modules. The
modules. hub broadcasts all received Ethernet frames to the DHE mod-
The frames stored in the intermediate FIFOs are then read ules and to the IPBus client on the DHC. The Ethernet frames
out by framing state machines in event framer cores. The order are transmitted over the same Aurora link that is used to read
in which the intermediate FIFOs are read by framing state out data from the DHE. The replies are multiplexed with the
machines is static. Each intermediate FIFO corresponds to a data stream in the DHE. The multiplexer assigns high priority
LEVIT et al.: FPGA BASED DATA READ-OUT SYSTEM OF THE BELLE II PIXEL DETECTOR 1037
to the data and low priority to the Ethernet stream. The received
replies on the DHC are then transmitted into the slow-control
Ethernet network.
The JTAG master is implemented as a two-level system. The
hardware master in the DHE consists of a state machine that exe-
cutes external commands. The control registers of the hardware
master are mapped to the IPBus registers. The software master
is built as an EPICS driver using the asynPortDriver class. The
main task of this master is generation of the JTAG commands
and transmission of the commands to the hardware master on the
DHE over IPBus. The software maintains not only the knowl-
edge of the JTAG registers available in the ASICs, but also of
the bit elds inside of the registers. Every bit eld that controls
a specic function in an ASIC is connected to an EPICS register
in the software. If a JTAG register access is scheduled, the soft-
ware constructs the bitstream using the cached values of the bit Fig. 7. DEPFET module bonded on the Hybrid6 board. Courtesy of M. Schnell.
elds and writes the bistream commands into the corresponding
DHE module over IPBus.
Fig. 10. Data and trigger rates test at the beam test setup.
B2TT system passes along this information and triggers the
read-out of the pixel detector and silicon vertex detector.
Once the trigger is received by the DHC module, the trigger During the beam test, approximately 60 million good events
frame is transmitted to the DHE module, where the trigger signal were recorded by the system.
for the front-end electronics is generated. In response to the
trigger signal, the DHP chips send up to two data frames in V. CONCLUSIONS
the zero-suppressed format. Then the frames are formated into
The read-out system for the Belle II pixel detector was de-
DHE format and sent to the DHC. The DHC formats data in
veloped and integrated into the EPICS slow control, and the
the DHC format and sends them to the Online Selection Node
pixel detector read-out chain. The full read-out chain was eval-
(ONSEN) [17]. A copy of the DHE data is also sent over Eth-
uated in the beam test at DESY. During the test we built a data
ernet to the standalone DAQ PC. On the standalone DAQ PC the
acquisition chain which resembles the future Belle II pixel de-
data are monitored by the data quality monitor and forwarded to
tector read-out chain: an external trigger, data-read out modules,
the EUDAQ PC to be merged with the telescope data.
subevent builder, online data reduction using data from outer
The ONSEN system is designed to perform online data reduc-
detectors, data quality monitoring, event builder, and the com-
tion by using the information from the outer detectors of Belle
patibility with the Belle II analysis software framework [18].
II. The data from outer detectors are used to generate Regions
With this setup we successfully operated the DEPFET sensor
of Interest (ROI), the intersection points of the particle tracks
and gained better understanding of the system behaviour. The
with the planes of the pixel detector, by extrapolating the tracks
performance of the system was tested at a trigger rate of 5 kHz
down to the interaction point. The ROIs are received by ONSEN
which is lower than the designed trigger rate of 30 kHz. The
from two sources: the FPGA-based silicon-strip-detector-only
test of the subevent building algorithm and the system perfor-
track nder Data Concentrator (DATCON) and software-based
mance measurements at the designed trigger rate remain the
high-level trigger. Both ROI sources were used in the beam test
open issues for the next system test with the nal version of
although they used data from the same detector. The pixel data
the DEPFET sensor in the autumn 2015.
were ltered in the ONSEN by checking the data against ROIs.
The ltered data were then sent to the event builder farm, which
merged data streams from the pixel detector and silicon strip REFERENCES
detector. [1] T. Abe et al., Belle II Technical Design Report, Tsukuba, Japan, KEK
Report 2010-1, 2010, arXiv:1011.0352.
D. System Performance at the Beam Test [2] Y. Ohnishi et al., Accelerator design at superKEKB, Progr. Theor.
Exper. Phys., vol. 2013, no. 3, 2013.
A dry run was performed on the DHEDHC setup to test the [3] J. Kemmer and G. Lutz, New detector concepts, Nucl. Instrum.
performance of the system before declaring the system opera- Methods Phys. Res. Sec. A: Accel., Spectrometers, Detectors Assoc.
Equip., vol. 253, no. 3, pp. 365377, 1987.
tional. An articial trigger with a dened frequency was gen- [4] S. Furletov, The Belle II pixel vertex detector, J. Instrum., vol. 7, no.
erated by the B2TT system, and the data were read out from 01, p. C01014, 2012.
the DEPFET module and sent to the ONSEN module. No high- [5] I. Peric, P. Fischer, J. Knopf, and T. H. H. Nguyen, DCDB and
SWITCHERB, the readout ASICS for Belle II DEPFET pixel
level trigger was generated during the dry run and therefore the detector, in Proc. IEEE Nucl. Sci. Symp. Med. Imaging Conf.
run duration was limited by the capacity of the memory in the (NSS/MIC), Oct. 2011, pp. 15361539.
ONSEN system. The system ran at a trigger rate of 5 kHz with a [6] J. Knopf, P. Fischer, C. Kreidl, and I. Peric, A 256 channel 8-bit cur-
rent digitizer ASIC for the Belle-II PXD, J. Instrum., vol. 6, no. 01,
continuous data stream of 17 MB/s for 6 minutes without errors p. C01085, 2011.
(Fig. 10). [7] M. Lemarenko et al., Test results of the data handling processor for the
The read-out system was in continuous operation for seven DEPFET pixel vertex detector, J. Instrum., vol. 8, no. 01, p. C01032,
2013.
days with the average trigger rate of 300 Hz. The rate of the [8] Aurora 8B/10B Protocol Specification, SP 002 Xilinx. San Jose, CA,
data stream generated by the detector averaged around 200 kB/s. USA, 2014.
LEVIT et al.: FPGA BASED DATA READ-OUT SYSTEM OF THE BELLE II PIXEL DETECTOR 1039
[9] J. Cachemiche, Module Management Controller mezzanine [14] Virtex-6 FPGA Memory Resources, User Guide 363, Xilinx, San
boardSpecication, no. 3.0, 2011 [Online]. Available: Jose, CA, USA, 2014, pp. 4366.
https://espace.cern.ch/ph-dep-ESE-BE-uTCAEvaluationProject/ [15] R. Frazier, G. Iles, D. Newbold, and A. Rose, Software and rmware
MMC_project/Shared Documents/MMC mezzanine/Specica- for controlling CMS trigger and readout hardware via gigabit ethernet,
tion/MMC_ specication_v3.0.pdf Phys. Proc., vol. 37, no. 0, pp. 18921899, 2012.
[10] N. D. Dayot, Development of an ATCA IPMI controller mezzanine [16] M. Friedl et al., The Belle II silicon vertex detector, Nucl. Instrum.
board to be used in the ATCA developments for the ATLAS Liquid Methods Phys. Res. Sec. A: Accel., Spectrometers, Detectors Assoc.
Argon upgrade, J. Instrum., vol. 7, no. 01, p. C01020, 2012 [Online]. Equip., vol. 732, no. 0, pp. 8386, 2013.
Available: http://stacks.iop.org/1748-0221/7/i=01/a=C01020=0pt [17] B. Spruck et al., The Belle II pixel detector data acquisition and re-
[11] M. Nakao, C. Lim, M. Friedl, and T. Uchida, Minimizing dead time of duction system, IEEE Trans. Nucl. Sci., vol. 60, no. 5, pp. 37093713,
the Belle II data acquisition system with pipelined trigger ow control, Oct. 2013.
IEEE Trans. Nucl. Sci., vol. 60, no. 5, pp. 37293734, Oct. 2013. [18] D. Kim, The software library of the coming Belle II experiment and
[12] M. Nakao, Timing distribution for the Belle II data acquisition its simulation package, in Proc. IEEE Nucl. Sci. Symp. Med. Imaging
system, J. Instrum., vol. 7, no. 01, p. C01028, 2012. Conf. (NSS/MIC), Oct. 2013, pp. 14.
[13] Virtex-6 FPGA Memory Interface Solutions, User Guide 406,
Xilinx, San Jose, CA, USA, 2011, pp. 17152.