Frequent Failure on bias_dets after recent power cycle due to snow break (turned off ATCAs during a storm)

[samdayweiss]

SATP3 has been getting frequent failures (epics failures) on the bias_dets function, on both crates. The frequency seems to be correlated with our power cycle on January 6th around 1:20 am UTC. The pysmurf controller and smurf streamer logs during the failure are attached below. The issue was resolved by restarting the RSSI connection for the slot with the failure (https://github.com/simonsobs/daq-discussions/discussions/113#discussioncomment-10845936)

Pysmurf controller log:

ERROR:pyrogue.Variable.RemoteVariable.AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[6].CryoChannels.etaI[22]:int too big to convert
Traceback (most recent call last):
  File "/usr/local/src/rogue/python/pyrogue/_Variable.py", line 324, in set
    self._block.set(self, value)
  File "/usr/local/src/rogue/python/pyrogue/_Block.py", line 362, in set
    ba = var._base.toBytes(value)
  File "/usr/local/src/rogue/python/pyrogue/_Model.py", line 128, in toBytes
    ba = value.to_bytes(byteCount(self.bitSize), self.endianness, signed=True)
OverflowError: int too big to convert
ERROR:pyrogue.Variable.RemoteVariable.AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[6].CryoChannels.etaI[22]:Error setting value '44729' to variable 'AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[6].CryoChannels.etaI[22]' with type Int16. Exception=int too big to convert
1736874256.985186:pyrogue.epicsV3.Value: Error setting value from epics: smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:SysgenCryo:Base[6]:CryoChannels:etaMagArray

ERROR:pyrogue.Variable.RemoteVariable.AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[7].CryoChannels.etaI[0]:"Memory Error for AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[7].CryoChannels.etaI[0] at address 0x81b10000 Invalid value '-91420' for base type <class 'int'> with bit size 16"
ERROR:pyrogue.Variable.RemoteVariable.AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[7].CryoChannels.etaI[0]:Error setting value '-91420' to variable 'AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[7].CryoChannels.etaI[0]' with type Int16. Exception="Memory Error for AMCc.FpgaTopLevel.AppTop.AppCore.SysgenCryo.Base[7].CryoChannels.etaI[0] at address 0x81b10000 Invalid value '-91420' for base type <class 'int'> with bit size 16"
1736874260.670343:pyrogue.epicsV3.Value: Error setting value from epics: smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:SysgenCryo:Base[7]:CryoChannels:etaMagArray

[samdayweiss]

@jlashner I reset the RSSI connection and ran bias dets again and this worked, so we'll see if the solution remains effective (and doesn't require a hammer).