STT Troubleshooting

1. STT Missing or FEB in UMon
2. CTT Input Problems
3. PCI Hangs
4. SCL Interrupt Disconnect

1. Click on Check Status button in the sttResetGui
2. If any of the status registers show 0xffffffff or probable hang is reported then go to the PCI Hang Section
3. If not then:
   • Click on Crate in the STT Status Display to produce a debug dump for the affected crate
   • Ask the DAQ shifter to issue an SCL Init
4. If the SCL Init does not help, this may be a:
   • CTT Problem or
   • PCI Hang
5. If you determine the problem is neither a CTT Problem or a PCI Hang, try the following.
   • Ask DAQ shifter to Pause Run
   • Click on Hard Reset in sttResetGui
   • Resume Run after you see the following lines on the screen
     Enabling Data Taking
     *****************Crate 0x7n is Reset ************************
6. If the SBC is missing in umon but there are no FEBs then: Hang, try the following.
   • Ask DAQ shifter to Pause Run
   • Click on Hard Reset in sttResetGui
   • Reset the SBC
   • Resume Run after you see the following lines on the screen
     Enabling Data Taking
     *****************Crate 0x7n is Reset ************************
7. If the above doesn't help, try the steps to recover from a PCI Hang

• Symptoms indicating CTT Problems
  • CTT Crate x13 is missing/FEB
  • CTT Error appears in the FRC monitoring GUI.
• Symptoms indicating STT Problem
  • No registers show 0xffffffff in Check Status button in sttResetGui. If yes, this is a PCI Hang
• Solutions for CTT Problems
  1. SCL Init
  2. Check the FRC's LTB
     • Click on the button Check Errors/Busy in the STT reset GUI. Look at the LTB Status display at the bottom of the readout.
     • If it's empty (all 0's in the Blocks column) this is a CTT problem. Go to next step
     • If it's not empty this is an STT problem. Check for a PUT_DONE_TIMEOUT in the BM-0 Status Register and follow instructions for PCI Hangs.
  3. If the SCL Init doesn't work then ask the CFT/CTT shifter to issue a resetSTT and/or a FixCTT

• Symptoms indicating a PCI Hang
  • Registers show 0xffffffff in Check Status button in sttResetGui.
• Warning Do not click on the Hard Reset Button if any of the PCI busses (1,2,3) are hung.
• Solutions
  1. Ask DAQ shifter to Pause Run
  2. Click on Set FRC Only in STT Download GUI
  3. Click on PCI Reset in STT Reset GUI
  4. Reboot CPU by typing reboot on IOC terminal corresponding to crate.
  5. Wait for Reboot to Finish - you should see the following lines (for e.g. crate 0x70)
     Done executing startup script /online/ioc/ppc/mv2300/d0olstt00/startup ->
  6. Click on FRC On in STT Reset GUI
  7. Ask for an SCL Init
  8. Resume Run

How to Recognize the Problem

1. We will remain 100% busy in th daq_dialog.
   (Typically, when there is no such problem you see the busy % decreasing when a sclinit goes through)
2. On clicking Check Busy/Errors you will see 'SCL INTERRUPT' yes.
3. On the IOC (cpu) window you will see an error message akin to 'SCL Interrupt disconnected'
4. In the monitoring gui - on the cpu page - you will see a SCL INIT ERROR.

How to Fix the Problem

1. Ask for a run pause
2. Click on Set Frc Only in the Download Gui
3. Reboot the cpu
4. Click on FRC On in the sttResetGui
5. Resume Run
6. If there is a PCI crash on any card, you will need to click on PCI Reset before rebooting the cpu.