Issue with amp setup

[eehealy]

We are running into an issue when we try to set up amps. We are getting an error about a power supply being off. I am wondering if we have an issue with a config file setting or something, because I think we are using the right script (setup_amps).

Any debugging help would be much appreciated!

[ 2024-07-15 17:55:17 ]  Starting setup_amps
[ 2024-07-15 17:55:17 ]  get_amp_drain_voltage: The power supply for amp 50k1 is off, therefore returning 0.0.
[ 2024-07-15 17:55:17 ]  get_amp_drain_voltage: The power supply for amp 50k2 is off, therefore returning 0.0.
[ 2024-07-15 17:55:17 ]  get_amp_drain_voltage: The power supply for amp hemt1 is off, therefore returning 0.0.
[ 2024-07-15 17:55:17 ]  get_amp_drain_voltage: The power supply for amp hemt2 is off, therefore returning 0.0.
[ 2024-07-15 17:55:17 ]  set_amp_drain_voltage: hemt2: this drain is currently disabled ; setting control DAC29 to 10V, enabling LDO, and then setting control DAC29 to voltage required to achieve desired drain voltage.  In the brief time for which the control DAC is set to 10V and the LDO is disabled, the cryocard will put out a small, load dependent voltage.
[ 2024-07-15 17:55:17 ]  set_amp_drain_voltage: 50k2: this drain is currently disabled ; setting control DAC28 to 10V, enabling LDO, and then setting control DAC28 to voltage required to achieve desired drain voltage.  In the brief time for which the control DAC is set to 10V and the LDO is disabled, the cryocard will put out a small, load dependent voltage.
[ 2024-07-15 17:55:17 ]  set_amp_drain_voltage: 50k1: this drain is currently disabled ; setting control DAC32 to 10V, enabling LDO, and then setting control DAC32 to voltage required to achieve desired drain voltage.  In the brief time for which the control DAC is set to 10V and the LDO is disabled, the cryocard will put out a small, load dependent voltage.
[ 2024-07-15 17:55:17 ]  set_amp_drain_voltage: hemt1: this drain is currently disabled ; setting control DAC31 to 10V, enabling LDO, and then setting control DAC31 to voltage required to achieve desired drain voltage.  In the brief time for which the control DAC is set to 10V and the LDO is disabled, the cryocard will put out a small, load dependent voltage.
[ 2024-07-15 17:55:17 ]  hemt2 current not within tolerance, scanning for correct gate voltage
[ 2024-07-15 17:55:17 ]  Setting hemt2 gate to -258440.6723592532 via DAC 27, given -1.0 Volts
[ 2024-07-15 17:55:18 ]  -22.787134525038724
[ 2024-07-15 17:55:18 ]  Setting hemt2 gate to -284284.06723592535 via DAC 27, given -1.0999973984000002 Volts
[ 2024-07-15 17:55:18 ]  -22.787134525038724
[ 2024-07-15 17:55:18 ]  Setting hemt2 gate to -310128.0672359253 via DAC 27, given -1.19999713824 Volts
[ 2024-07-15 17:55:18 ]  -22.787134525038724
[ 2024-07-15 17:55:18 ]  Setting hemt2 gate to -335972.06723592535 via DAC 27, given -1.2999968780800002 Volts
[ 2024-07-15 17:55:18 ]  -22.787134525038724
[ 2024-07-15 17:55:18 ]  Setting hemt2 gate to -361816.06723592535 via DAC 27, given -1.3999966179200003 Volts
[ 2024-07-15 17:55:19 ]  -22.787134525038724
[ 2024-07-15 17:55:19 ]  Setting hemt2 gate to -387660.06723592535 via DAC 27, given -1.4999963577600002 Volts
[ 2024-07-15 17:55:19 ]  -22.787134525038724
[ 2024-07-15 17:55:19 ]  Setting hemt2 gate to -413504.06723592535 via DAC 27, given -1.5999960976000003 Volts
[ 2024-07-15 17:55:19 ]  -22.787134525038724
[ 2024-07-15 17:55:19 ]  Setting hemt2 gate to -439348.06723592535 via DAC 27, given -1.6999958374400002 Volts
[ 2024-07-15 17:55:20 ]  -22.787134525038724
[ 2024-07-15 17:55:20 ]  Setting hemt2 gate to -465192.06723592535 via DAC 27, given -1.7999955772800003 Volts
[ 2024-07-15 17:55:20 ]  -22.787134525038724
[ 2024-07-15 17:55:20 ]  Setting hemt2 gate to -491036.0672359253 via DAC 27, given -1.8999953171200001 Volts
[ 2024-07-15 17:55:20 ]  -22.787134525038724
[ 2024-07-15 17:55:20 ]  Setting hemt2 gate to -516880.06723592535 via DAC 27, given -1.9999950569600002 Volts
[ 2024-07-15 17:55:21 ]  -22.787134525038724
[ 2024-07-15 17:55:21 ]  Gate voltage adjustment would go out of range (-2.0, 0). Unable to change hemt2_drain_current to desired value
[ 2024-07-15 17:55:21 ]  Failed determining hemt2 gate voltage

[jlashner]

Hi Erin! I'm looking at the setup amps function, and it seems to enable amps in this else branch:

sodetlib/sodetlib/operations/uxm_setup.py

Lines 125 to 131 in 6de534a

	else:
	cc_rev = 'c02'
	amp_list = S.C.list_of_c02_amps
	S.C.write_ps_en(0b11)
	time.sleep(exp['amp_enable_wait_time'])
	if S.C.read_ps_en() != 3:
	raise ValueError("Could not enable amps.")

I can't remember off the top of my head why we do it there, since it looks like it doesn't happen for major version == 4... but I know this is regularly used at site and works relatively well.

Can you:

• post the version number that is returned from S.C.get_fw_version()
• Double check that the SCSI cable is all the way plugged in and try again? Sometimes we have found that if the SCSI is partially unplugged some cryocard commands like this will fail, so this is a good thing to double check.

[eehealy]

Ok thanks! I'll double check the SCSI cables now. The version we are on is: (4,0,7)

[eehealy]

Ok so we remated the SCSI connectors and didn't see a difference.

We are now wondering if it has something to do with the S.C.write_ps_en(). We turned the amps off with this function, and when we attempted alter this function, we got a different result, so it seems like we might have disabled our power supplies by accident.

@jlashner do you know the correct way to enable the psus, do you think that this could be the source of the issue?

[eehealy]

When we run S.get_amplifier_biases() we get

[ 2024-07-15 18:35:21 ]  get_amp_drain_voltage: The power supply for amp 50k1 is off, therefore returning 0.0.
[ 2024-07-15 18:35:21 ]  get_amp_drain_voltage: The power supply for amp 50k2 is off, therefore returning 0.0.
[ 2024-07-15 18:35:21 ]  get_amp_drain_voltage: The power supply for amp hemt1 is off, therefore returning 0.0.
[ 2024-07-15 18:35:21 ]  get_amp_drain_voltage: The power supply for amp hemt2 is off, therefore returning 0.0.
{'50k1_gate_volt': 0.0,
 '50k1_drain_volt': 0.0,
 '50k1_drain_current': 0.29211306526336983,
 '50k1_enable': False,
 '50k2_gate_volt': 0.0,
 '50k2_drain_volt': 0.0,
 '50k2_drain_current': 0.29211306526336983,
 '50k2_enable': False,
 'hemt1_gate_volt': -2.0048469742400004,
 'hemt1_drain_volt': 0.0,
 'hemt1_drain_current': -0.05331653329891589,
 'hemt1_enable': False,
 'hemt2_gate_volt': -1.9999947968000003,
 'hemt2_drain_volt': 0.0,
 'hemt2_drain_current': -0.07321653329891589,
 'hemt2_enable': False}

It looks like all of the amplifiers are not enabled. How would we enable them?

[eehealy]

Ran S.C.write_ps_en(0b1111) and didn't rerun setup_amps(), and things look better...

I think it might be an issue with our config file, because the amp currents are way too high.

{'50k1_gate_volt': 0.0,
 '50k1_drain_volt': 4.000003911132813,
 '50k1_drain_current': 55.33919736378285,
 '50k1_enable': True,
 '50k2_gate_volt': 0.0,
 '50k2_drain_volt': 4.000003911132813,
 '50k2_drain_current': 55.33919736378285,
 '50k2_enable': True,
 'hemt1_gate_volt': -2.0048469742400004,
 'hemt1_drain_volt': 0.8000042998313902,
 'hemt1_drain_current': 33.80703452503872,
 'hemt1_enable': True,
 'hemt2_gate_volt': -1.9999947968000003,
 'hemt2_drain_volt': 0.8000042998313902,
 'hemt2_drain_current': 33.787134525038724,
 'hemt2_enable': True}

[eehealy]

Replying to request help again!

We were able to get our amps to draw the right current with non-smurf power supplies. So this does not seem to be an issue with the amplifiers themselves.

[jlashner]

So I believe the 33 and 55 A readout is not physical, and is a sign that there is a problem with the current readout. We've seen it before in a previous other discussions:

• Amplifier Issues (fixed by jackhammer) #419
• Current issue with new LATR cryocard #420

We should check if @tpsatt remembers how we solved these, but it seems in at least one of them, hammering the slot fixed the issue so it might be worth just trying that first.

Around this time the LAT there was also an issue with the seating of the DSUBs on the cryocard side. When you checked the DC cable, did you check both the SCSI and dsubs?

[eehealy]

Thanks, Jack! Helpful to see this issue was resolved by hammering.

Unfortunately, during debugging we have encountered a new issue in that our cryocard does not seem to be turning on. We had disconnected from smurf to check our amplifiers and after powering everything back on, we don't have power to this cryocard anymore. I've tried replacing the SCSI cable.

[eehealy]

magically turned back on after hammering + 15ish minutes

[eehealy]

Ok so, we've swapped out the SCSI cable, checked and remated the DC cable, and power-cycled the crate and hammered.

When we ran setup_amps we initially got the same 55 and 33 mA values, so we hammered again. Now we are getting a new issue:

[ 2024-07-16 16:28:59 ]  Starting setup_amps
[ 2024-07-16 16:28:59 ]  get_amp_drain_voltage: The power supply for amp 50k1 is off, therefore returning 0.0.
cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]
[ 2024-07-16 16:29:04 ]  Command failed: smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]
[ 2024-07-16 16:29:04 ]  Retry attempt 1 of 5
cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]
[ 2024-07-16 16:29:09 ]  Retry attempt 2 of 5
cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]
[ 2024-07-16 16:29:14 ]  Retry attempt 3 of 5
cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]
[ 2024-07-16 16:29:19 ]  Retry attempt 4 of 5
cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]
[ 2024-07-16 16:29:24 ]  Retry attempt 5 of 5
cannot connect to smurf_server_s3:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[26]

And the error was that epics couldn't connect. We tried hammering again but got this same error.

[eehealy]

Restarted jupyter docker and jupyter notebook. Seemed to solve all of these problems. Thanks for the help @jlashner! Was able to use some old github discussions to continue debugging.

[jlashner]

huh, as always I wish just hammering weren't the solution, but glad things are working!!