Trigger distribution problems

In some occasions one of the two Trigger Distribution Boards (TDB) stops sending triggers either to all the connected boards or to a few of them. In the past this mostly happened to the left (wall) side TDB, connected to boards 14 to 28 controlling left half of ECal, EVeto, SAC, and Target.

This syndrome appears with errors messages on the Merger log file like those shown below, followed by the crash of the DAQ system due to desynchronization.

*** Board 28 - Internal time mismatch 0000000000000000000000001b81abae 0000000000000000000000001b81aba9 - Skipping event    16920
*** Board 28 - Board time 1336082315102 greater than Trigger time 1335972668538: setting board as MISSING in this event
ADCBoard::NextEvent - WARNING - Current event was not used by previous trigger: see if it is usable now
*** Board 24 - Internal time mismatch 0000000000000000000000001c91d88e 0000000000000000000000001c91d88c - Skipping event    16922
*** Board 16 - Board time 1336125603917 greater than Trigger time 1336125603900: setting board as MISSING in this event
*** Board 24 - Board time 1336151521118 greater than Trigger time 1336125603900: setting board as MISSING in this event
ADCBoard::NextEvent - WARNING - Current event was not used by previous trigger: see if it is usable now
ADCBoard::NextEvent - WARNING - Current event was not used by previous trigger: see if it is usable now
*** Board 16 - Board time 1336125603917 less than Trigger time 1336151521112: skip event and try to recover
*** Board 14 - Internal time mismatch 0000000000000000000000001d34591c 0000000000000000000000001d34591e - Skipping event    16925
*** Board 14 - Board time 1336295852817 greater than Trigger time 1336215915262: setting board as MISSING in this event
*** Board 22 - Board time 1336215915281 greater than Trigger time 1336215915262: setting board as MISSING in this event
ADCBoard::NextEvent - WARNING - Current event was not used by previous trigger: see if it is usable now
ADCBoard::NextEvent - WARNING - Current event was not used by previous trigger: see if it is usable now

To verify the correct functioning of the TDBs, the most expedite solution is to stop the run, execute the clean up procedure, and check the Trigger Distribution System in standalone mode, i.e. without a full DAQ run.

Please ask the Run Coordinator before using this procedure

Procedure to start trigger generation in standalone mode:

• On one terminal login as daq@l0padme1
• cd DAQ/trigger
• ./PadmeTrig.exe -c standalone_dualtimer.cfg
• Wait for the Trigger Server initialization to complete: - Trigger board initialized: waiting for start_file 'start_trig'
• On a second terminal login as daq@l0padme1
• cd DAQ/trigger
• ./start_trig.sh

At this point on the first terminal you should start seeing a steady flow of trigger generation messages:

- Start file 'start_trig' found: starting Triggers
- Sending start_run command to trigger board
trig_set_register cmd = 0x010000000001
2020/02/19 07:39:00 - Starting trigger generation
- Enabling requested triggers.
trig_set_register cmd = 0x010201f41401
Current trigger mask: 0x01
- Trigger         0 0x023c011b7da9c5ca  118072395210 0x01   60 0 1
- Trigger        10 0x0246011b7f4d35e9  118099883497 0x01   70 0 1     343.604ms     331.287ms
- Trigger        20 0x0250011b8262bd63  118151626083 0x01   80 0 1     646.782ms     662.571ms
- Trigger        30 0x025a011b84e43d6f  118193667439 0x01   90 0 1     525.517ms     522.986ms
- Trigger        40 0x0264011b874d1619  118234093081 0x01  100 0 1     505.321ms     505.506ms
...

The trigger generation can be stopped at any time by issuing the ./stop_trig.sh command on the second terminal.

While the standalone triggers generation is active, ask for access to BTF and use the oscilloscope there to verify if both TDBs are distributing the triggers. To this end you can use a LEMO cable to connect the oscilloscope to any of the 32+32 channels of the two TDBs (2 channels of the left TDB and 4 channels of the right TDB are free) and check that a NIM-level (0V/-0.8V) trigger signal is being generated. If you suspect that only a few channels are not generated, feel free to disconnect any of the connected LEMO cables, check the channel with the oscilloscope, and put the original LEMO cable back in (make sure that the LEMO connector is well inserted!).

If you see that a TDB has problems, i.e. the board is not sending triggers to one or more channels, you can try the following:

• Switch the NIM crate hosting the board off
• Try and push the TDB back in its slot: sometimes the electric contacts can be a bit shaky, especially in the left crate.
• Switch the NIM crate back on
• Check if the triggers are back

If you are certain that the problem is with the TDB, you can replace it with a spare one:

• Switch the NIM crate off
• Remove all LEMO cables from the faulty board (keep them together in a single bunch)
• Replace the faulty board with the spare
• Reconnect all LEMO cables to the new board. N.B. the order with which the cables are connected to the board is not relevant. Make sure that the LEMO connectors are correctly inserted!
• Switch the NIM crate back on
• Check if the triggers are back

If triggers are still missing, this could be a problem either with the connection cables or with the Trigger Server Board: please wait for an expert to intervene.

If you manage to get the TDBs back to work, you can stop the standalone trigger system (see above), make sure that all Veto boards are back in operation after switching off and on the NIM crates, and then start a new run.