Sayma JESD intermittent initialization failure

[sbourdeauducq]

On Sayma, the AD9154 JESD initialization sometimes fails with "bad SYNC" or "JESD ready timeout".
@enjoy-digital suspects that this may be because the clocks generated by the HMC830 are not clean (#860 (comment)).
@gkasprow can you check that?

[gkasprow]

@sbourdeauducq can you describe any additional conditions when it happens? During normal operation? While initialisation? The board is cold or hot? Any additional info will help to recreate the problem.
Does it happen on all boards? What is Tx signal level?

[sbourdeauducq]

This is at initialization, though we don't know at the moment if the link is reliable afterwards when the initialization succeeds. The boards are hot and I don't know if it happens with a cold board. What @enjoy-digital was suggesting you check is whether the HMC830 outputs clean clocks or not.

[gkasprow]

@sbourdeauducq OK, I will check it with spectrum analyser. This could be a problem with PLL loop stability. Is the PLL locked all the time? I didn't investigate it because at the time there was no configuration available and I used some typical values to check if from HW point of view only.

[sbourdeauducq]

My understanding of the HMC830 is that if the PLL unlocks, it will not automatically try to relock or it will do so only once (depending on a bit in a configuration register).

[sbourdeauducq]

Please compile ARTIQ and test the HMC830 with the exact same settings that we have.

[sbourdeauducq]

@enjoy-digital Maybe it is just m-labs/jesd204b#12? I thought that issue was fixed already.

[enjoy-digital]

@sbourdeauducq: i'm pretty sure all was fine with kcu105 + ad9154 fmc and also with my sayma_test design. I need to do more tests but a quick check of the generated clocks would be interesting.

[sbourdeauducq]

@gkasprow Have you installed ARTIQ?

[sbourdeauducq]

Now we are running with the HMC830 bypassed, and 1.2GHz sent directly to the DAC.
On some board power-cycles the initialization never completes...

[    29.562550s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[    29.836006s]  WARN(board_artiq::ad9154): AD9154-0 config attempt #94 failed (JESD ready timeout), retrying
[    29.854638s]  INFO(board_artiq::ad9154): AD9154-0 found
[    29.858550s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[    30.132006s]  WARN(board_artiq::ad9154): AD9154-0 config attempt #95 failed (JESD ready timeout), retrying
[    30.150637s]  INFO(board_artiq::ad9154): AD9154-0 found
[    30.154549s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[    30.428008s]  WARN(board_artiq::ad9154): AD9154-0 config attempt #96 failed (JESD ready timeout), retrying
[    30.446640s]  INFO(board_artiq::ad9154): AD9154-0 found
[    30.450552s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[    30.724008s]  WARN(board_artiq::ad9154): AD9154-0 config attempt #97 failed (JESD ready timeout), retrying
[    30.742639s]  INFO(board_artiq::ad9154): AD9154-0 found
[    30.746551s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[    31.020007s]  WARN(board_artiq::ad9154): AD9154-0 config attempt #98 failed (JESD ready timeout), retrying
[    31.038642s]  INFO(board_artiq::ad9154): AD9154-0 found
[    31.042557s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
panic at src/libcore/result.rs:906: cannot initialize AD9154: "JESD ready timeout"

[sbourdeauducq]

And the errors are varied...

[     3.023563s]  WARN(board_artiq::ad9154): AD9154-1 config attempt #0 failed (bad SYNC), retrying
[     3.041238s]  INFO(board_artiq::ad9154): AD9154-1 found
[     3.045153s]  INFO(board_artiq::ad9154): AD9154-1 configuration...
[     3.131565s]  WARN(board_artiq::ad9154): AD9154-1 config attempt #1 failed (bad CODEGRPSYNCFLG), retrying

(I'm doing some tests of DAC synchronization, so I have a script that power-cycles and reloads the boards many times, and this hits a lot of bugs...)

[gkasprow]

@sbourdeauducq do you read back the PLL AND DAC chip registers after configuration?

[jbqubit]

1.2GHz sent directly to the DAC.

Are you driving single-ended or differential (using 180-deg RF splitter)?

[hartytp]

I drove it single-ended (soldered a 50R resistor across one SMP). At >=1GHz anything between +0dBm and +10dBm is fine (IIRC, those chips can take up to 2Vpp single ended).

[sbourdeauducq]

Are you driving single-ended or differential (using 180-deg RF splitter)?

Single-ended, see the email I sent to the list.

[jbqubit]

For same .bit loaded into RTM and AMC FPGAs sometimes I see varied behavior for repeatedly issuing artiq_flash -t sayma start.

• In some cases configuration happens quickly.
• In some cases configuration attempts exceed 100 before other errors.
• These two cases can be observed in rapid succession with no other changes to setup.

[sbourdeauducq]

@enjoy-digital I still see this problem regularly #861 (comment)

[enjoy-digital]

@sbourdeauducq: yes i also saw it. I think i also see it on the KCU105 + AD9154 so i'll investigate with this setup.

[enjoy-digital]

Fixed via m-labs/jesd204b@03718be.

[sbourdeauducq]

Just went through a dozen Sayma restarts without seeing any bugs. Good!

[hartytp]

Wow! Nice.