Shuttler DAC #253
Comments
|
We are able to drive 32 channels x 10MS/s from 4 EEMs. The question is if
Kasli is able to refresh such amount of channels simultaneously. And to
build 320 channel system we would need 5 Kaslis.
15.08.2017 11:24 PM "hartytp" <notifications@github.com> napisał(a):
… I've started sketching out requirements for a fast DAC to be used for ion
shuttling/splitting. It's on the Wiki
<https://github.com/m-labs/sinara/wiki/Fast_DAC>.
All comments welcome...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#253>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEH-vqvfLzFd5M4rTLLnqwHQOHib0maFks5sYn0FgaJpZM4O4b0u>
.
|
|
On AMC we can fit 64 or even 128 DAC channels with stacked SCSI
connectors.
|
|
@gkasprow is that 32 channels, EACH at 10 MS/sec? I think that probably serial control is going to be challenging to meet the kind of update rates we want. The DAC used on the PDQ boards is the Analog Devices AD9726, It has 0.5 LSB typical DNL, 10 ppm/C gain and offset drift, and takes parallel LVDS data at up to 400 MSPS. Noise is at the Johnson noise limit for -6 dBFS output, runs up about 5 dB at 0dBFS. Dissipation is about 0.6 W/channel (!!!) In the PDQ, it is run at 50 or 100 MSPS. AFAICT most of the other DACs available with sufficient speed to drive > MHz signals have at least 2 LSB typical DNL, and 100 ppm/C gain coefficients, both of which are problematic. The downside of the AD9726 is that it is a "not recommended for new designs" product, i.e. they are phasing out the lifecycle of the chip. The LTC1668 is a potential alternative part, with specs somewhat worse than the AD9726 but better than a lot of the competition. The noise is a factor of a few above the Johnson noise limit (50 pA/rtHz, or 5 nV/rtHz into 100 ohms), but it goes to 50 MSPS and the power dissipation is much lower (180 mW/channel). Another chip worth looking at is the LTC-2000-16. This is more expensive, about $90/channel, and has export restrictions which could be problematic, although I think Poland is OK (China/Hong Kong is definitely not OK). Power dissipation at 100 MSPS would be ~300 mW, DNL is 0.5 LSB, gain drift is 5 ppm/C. It's in a BGA package, which is a pain. |
|
If we start with selecting the DAC, then we can match N channels to an FPGA which should probably be on the same board as the DACs for wiring and throughput reasons. The interface between that board and the next upstream device (Kasli or Metlino) is likely to be higher level (DRTIO) and not EEM for throughput and simplicity reasons. And then we arrive at AMCs as the form factor. |
|
I agree with @jordens here in general, although if price point is highly sensitive on this one we could basically implement "Kasli PDQ" -- effectively N DACs (AD9726 perhaps) linked to a medium-sized Artix on an EEM. If much of the PDQ gateware could be ported over, including the SPI communications over EEM, this would be a simple board to implement that would also be inexpensive, and require only the inexpensive chassis rather than the more pricey uTCA. The price you pay is obviously that you would have less bandwidth to update the output waveform of the DACs, feedback becomes slower, etc. NIST would be very interested in an EEM-form-factor PDQ replacement card with similar spec/properties (although with much deeper memory, and possibly higher sample rates). |
|
@dhslichter ACK. It is just that restricting the link to SPI (or anything we can run over EEM) would likely lead to the same split-brain effect we see with PDQ (or any other non-DRTIO device with significant data): data here, control there and data needs to be uploaded ahead of time, synchronized, maintained and can not be swapped or parametrized easily. |
|
@dhslichter Thanks for the DAC recommendations.
Do we really need this much bandwidth? IIRC, if we go for 10MSPS, we get more choices of DAC (and a less crazy power budget!).
If you're serious about going to 200+ channels, then I really think AMCs are the only way to go. Not just because of the wiring complexity/space required for large numbers of Kasli/EEMs, but also because you need a chassis with really good thermal management (otherwise, thermal drifts will be a killer due to the amount of power dissipation on these boards). At that point, you need something like a proper uTCA rack. For a cheaper approach, you an always buy a single unit AMC chassis and run the AMC over DRTIO fibre. |
|
I've been looking at lower resolution DACs. Did we discuss that in such
application 12bits is sufficient?. Then we could make very low cost EEM
with such 32 DACs with constant update rate of 10MS/s. And MTCA with high
performance DACs.
|
|
I meant something like dac7822.
|
12 bit would not be sufficient for us.
Given the reasons discussed above, I don't think we'd use the EEM DAC board. @gkasprow Will the EEM + Kasli alternative really be cheaper than the AMC? My guess is that in the end, the AMC solution will be cheaper and more convenient. |
|
Big questions for me about this design:
|
|
If we pack 64x 90$ DACs then platform cost does not matter:)
|
|
Exactly. |
|
Hey @camacazio could you explain to us why you went for 50 MHz with PDQ? Would 10 MHz work as well? |
|
|
I would argue we want to be able to synthesize signals with bandwidths up to typical secular frequencies, if we are looking to do fast (diabatic) transport. I would say that 10 MSPS is the very low end of what one would want, I'm fine with AMC in the end but I think we should try to pack as many DACs in as possible in this case. |
|
Detachable AFE is definitely a good thing, and will allow people to customize a great deal. I would keep it inside the chassis. I think given that customizability, we want the underlying DAC board not to be underpowered sample-rate-wise, especially if we are going to be in an expensive chassis. |
I'm happy shooting for 64 channels per AMC. The only downside of packing the channels in is that the silicon cost is quite high, so users who only want, say, 10 channels are wasting a lot of money on unused DACs.
Agreed, we shouldn't go below this. And, if we can find a good DAC that's much faster, and doesn't have much worse noise/etc then it'd be great to go faster. But, let's take this as a lower limit on the BW for now.
Let's copy Zotino and supply +-12V + GND on the HD68 cable. That way, the front-end can be powered via the SCSI cable assuming it doesn't need too much current. |
If the DAC update rate might be resonant with any ion secular frequencies you have to think a lot harder about filtering and/or frequency planning. In Be+ and Mg+ surface traps you are often right around 10MHz so I think at least 20MSPS would be preferable so that this isn't an issue. Also to echo Daniel's comment, there is interest in pushing transport further into the diabatic regime at NIST. We'd have to have to ask the experts (eg. Didi) to see what their aspirations are.
The Sandia HOA 2.0 can take +/- 20V so I think this would need to be an option. I guess AFE spec is a somewhat separate discussion though. |
|
We could follow rtm approach and make mezzanines:) i.e. fmc ones or smaller
ones with i.e. 8 dac channels...
|
|
We can also make just 32channel fmc boards and plug them into std amc
carrier i.e. AFC(K) OH design which has HPC FMC
|
That could be a nice solution if you can fit that many DACs on an FMC, as well as the required regulators and FPGAs (probably not enough IO on an FMC to run these DACs directly from the connector)! Assuming you can get 2 FMCs on an AMC, you could then have up to 64 channels per AMC. Another thought: we should make sure that this board has front-panel clock, and trigger input SMAs. That would extend the range of uses (e.g. for non-ARTIQ users) |
|
I did such 16 channel OH boards nased on dac1401d125
16.08.2017 15:04 "Grzegorz Kasprowicz" <kasprowg@gmail.com> napisał(a):
… We can also make just 32channel fmc boards and plug them into std amc
carrier i.e. AFC(K) OH design which has HPC FMC
|
Good idea!
Okay, let's see if we can get the AFE to +-20V.
Were there FPGAs on the mezzanine? How did you do the IO for the DACs? |
|
HPC FMC has really lot of pins. I used all pins of biggest ARTIX chip
16.08.2017 15:16 "hartytp" <notifications@github.com> napisał(a):
… Also to echo Daniel's comment, there is interest in pushing transport
further into the diabatic regime at NIST. We'd have to have to ask the
experts (eg. Didi) to see what their aspirations are.
Good idea!
The Sandia HOA 2.0 can take +/- 20V so I think this would need to be an
option. I guess AFE spec is a somewhat separate discussion though.
Okay, let's see if we can get the AFE to +-20V.
I did such 16 channel OH boards nased on dac1401d125
Were there FPGAs on the mezzanine? How did you do the IO for the DACs?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#253 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AEH-vg2_dMV4F8g4QoGisJoctDlAsgaUks5sY1wpgaJpZM4O4b0u>
.
|
|
This shouldn't be called "fast DAC", it's slower than Sayma. Maybe "medium speed DAC" or something along those lines. |
|
If we put that on FMCs (and 16 channels per FMC with two FMCs per AMC would already be nice IMHO) then we can also do an EEM with a (little) FPGA being an FMC carrier, i.e. have the choice of having the FMC on AMC or on EEM. And let's just use a code name instead of "something DAC". |
|
@jordens Agreed: if we can get this on an FMC, that's great. As you say, if anyone really wants to use it with Kasli, we can then make an adapter board. This could also be a generic (passive) FMC to many IDC board. One option would be to give this female header, so that it can mate directly with Kasli, mounting via spacers and the mount holes. Having said that, maybe the best approach is to put the discussion about form-factor on hold for now. Let's first figure out how much bandwidth people need, and what order of filter the AFE needs. Once we've done that and chosen DACs/OpAmps, we'll be in a better position to figure out whether it can fit on an FMC/etc. |
|
To fit 32 channels on FMC we would need to go for quad dacs with 16 bit
lvds port and max frequency twice higher than we need. In this way we
would connect 2 chips in parallel to each port and use 4 x 16 channels
ports of HPC FMC with control lines.Example is DAC3484. It would be hard to
physically fit and route 16 dual channel dacs on FMC
17.08.2017 07:53 "hartytp" <notifications@github.com> napisał(a):
… @jordens <https://github.com/jordens> Agreed: if we can get this on an
FMC, that's great. As you say, if anyone really wants to use it with Kasli,
we can then make an adapter board. This could also be a generic (passive)
FMC to many IDC board. One option would be to give this female header, so
that it can mate directly with Kasli, mounting via spacers and the mount
holes.
------------------------------
Having said that, maybe the best approach is to put the discussion about
form-factor on hold for now.
Let's first figure out how much bandwidth people need, and what order of
filter the AFE needs. Once we've done that and chosen DACs/OpAmps, we'll be
in a better position to figure out whether it can fit on an FMC/etc.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#253 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AEH-vnpRgSPIQVAXdF2ieZgV4jz2JV76ks5sZEXZgaJpZM4O4b0u>
.
|
@dhslichter right, cooling is about the same on AMCs + RTMs. Generally the idea of RTMs is to have more space to spread things out to allow better cooling, and to remove things like SFPs that block airflow. As @gkasprow said, I wouldn't suggest designing an RTM with an ultrascale FPGA. If we go down the RTM route, we would just have 1-2 cheap Artix-7 FPGAs to act as deserialisers. The power consumption for that is way lower (although still not negligible). |
|
@dhslichter Taking a step back, let's make sure that we have a specification in mind for what we're aiming to achieve here. For Zotino, our target was 10ppm pk-pk. i.e. 30Hz on a 3MHz axial mode. Much more than that and it gets to be a pain for high-fidelity gates (drifts are likely to be slow, so one can do complex gate sequences, but I'd rather have higher performance HW and simpler gate sequences if at all possible). Are you happy with that specification, or do you want to make a different suggestion? So long as we agree on a specification, and produce HW that meets it, I'm not fussed about the AMC/RTM thing -- I'm only suggesting it as, i think, the easiest way of meeting our specification, but I'll be happy with anything that works and isn't too hacked.
I definitely agree that we should hook the temperature sensors up. If nothing else, they're a very useful diagnostic to have access to. My point is just that if DAC temperature compensation/lookup tables are going to be essential to meeting our drift specification, then I'd like to see a coherent plan for how that will work (implemented in HW? Done via kernels? What BW? What noise will this induce on the DAC's output for DC levels?). |
The same result is achievable, with less pain, by reducing the number of channels per card.
As I said earlier, SFPs are not essential. |
|
@hartytp I have eval boards for the relevant parts (LTC2000-16 and ADA4927), and will do performance testing on the AFE output circuits. Timescale is probably not going to be immediate, but hopefully in the next ~month. Will want to get draft design in place before we go too far. Regarding drift spec, something like 10 ppm pk-pk seems reasonable to me as well, on slow timescales (such that things like Walsh sequences can give immunity). I am not sure how one would likely implement the DAC temperature compensation, but my guess is that the most likely thing would be that we just use the sensors open-loop as a debugging tool, and we use the ion(s) as the servo on the digital waveforms that are output from the DAC. @sbourdeauducq your point about the number of channels per card is well taken. However, the use case for these DACs generally involves traps with ~100 electrodes, with near/medium term plans to go to 200+ electrodes, so we need to achieve some reasonable channel density with the hardware in racks. Going to 8 channels per card, for example, would limit us to <100 channels in a full 19" uTCA crate, which is less than ideal. If we can run 16 channels, or even up to 20 channels, on a single card with reasonable thermal performance, that would be much preferable. However, we want to allow some safety margin on the thermal performance for sure. I think we can reduce the gain on the output amplifier stages, and reduce the supply voltages as a result, to win something on the power dissipation there -- should go down to 125-250 mW per channel, depending on output. This would put the DAC + AFE power budget for a 16-channel board at about 10 W per board worst case (all max-scale output voltages), including the LDO dissipation for the AFE. |
I think we're on board with this -- for this application, the thermal and real estate penalty from SFP outweighs its benefits, and as long as we have risk-mitigating solutions to enable SFP communications in the absence of Metlino (SFP_RTM or SATA-SFP adapters), I am satisfied. |
|
Cool, I think we're converging on something.
Ideally, I'd like to achieve this stability over a period of hours. Longer than that, and we can servo to the ion as you said.
👍
👍 |
|
A simple thing that could be done would be to use an averaged DAC temperature in the crate, collected on slow timescales, to servo the crate fan speed. This would get rid of the sorts of common-mode fluctuations due to input air temperature fluctuations from the lab (for example), which is sort of a lowest-order effect. We could further contemplate putting local heaters under the DACs (or on a common heat sink, if we use a shared heat sink) for trimming within a ~few K range, although after our experience with Zotino I am wary of this. |
|
Realistically the dissipation from the cards is going to be dependent on the output voltages due to the AFE amplifiers, so there is going to be a certain level of unavoidable thermal gradient both within and between cards. When we do a thermal simulation it would be good to see how much of a DAC temperature difference is seen when the AFE amplifier dissipation varies. |
IIRC, when we were discussing Sayma a while back @gkasprow posted some nice data of servoing the fan speed to control an individual AMC to about 0.1C. Not sure what software would have to be written to make this work with Shuttler.
Yeah, i'd like to avoid this kind of thing if we possibly can. My guess (but, needs simulation) is that with good design, this is probably not necessary, since 10W is not that high a heat load for am AMC, and the bulk of that is static (i.e. does not depend on DAC voltage).
Yes. Let's make sure that the air hits the DACs first, then the AFE amplifiers, then the FPGA. That way, we minimise the effect that the amplifiers have on the DAC temperatures. If it's a problem, we can also set the output common-mode voltage so that the power consumption is independent of the differential output voltage. |
But RTMs achieve (if anything) marginally higher densities than more AMCs, for a lot of pain. |
👍
I have revised my AFE thoughts so that we are running with a gain of 3 and +/- 2.5V supplies for the amps, to give +/- 1.5Vdiff swings at the outputs. If you go lower on the gain, you start to degrade the SNR too much. However, with this, one has a dissipation of 100 mW per channel from the quiescent current, which goes up to 150 mW at full scale in either direction. So with 16 channels that would be a variation of 800 mW dissipation worst case for the entire board depending on output voltages. Thus all assumes a 200 ohm load. If we go to driving a 100 ohm load (for example, driving a load-terminated 100-ohm differential pair on a SCSI cable or some such, need to check the nonlinearity of the amplifier with these small loads), dissipation goes to 190 mW at full scale. |
|
It seems we are converging to sth that has chance to work. Initial specification may look like below
|
We need ODELAY for SDRAM. |
Does it, or are there still packet losses depending on the media converter model and the phase of the moon? |
|
@sbourdeauducq how do you make SDRAM running on Kasli which has only HR banks? |
There's only one chip, so we shift the clock instead. |
|
I used 4 chips (8 bit) on AFC board, with Artix. How the Xilinx core manages to make it working? It works at 200MHz (400Mbit)/s without issues. |
I think we can go lower than +/- 5V on the supply if needed for power budget, but I think it's important to do bipolar supplies, we won't be able to achieve the desired performance with single-supply AFE amps.
I prefer a wider SDRAM because I think it gives us more flexibility. I would suggest that we might implement two SDRAMs as in Sayma (except both 32 bit if this is all we can fit pin-wise), so that we can have separate memory buses for the processor and for the waveform generation.
The LTC2000 has an internal clock synchronizer circuit and control registers that allow one to adjust the phase of the data clock with respect to the sample clock. Thus as long as the data and DCLKI lines are well length-matched for any given DAC, we should be OK and able to synchronize the DAC outputs. Since our target data rate is 150 MSPS max, the constraints on the length matching are not extremely stringent. @sbourdeauducq do you think that the Ultrascale issues with BUFGCE_DIV, IOSERDES, and IODELAY would raise major issues for this design? Data out is DDR at 1/2 the RTIO clock frequency. |
|
@gkasprow I'm not sure how it's done on A7, but on some devices the Xilinx MIG uses undocumented FPGA features. The technique with ODELAY or clock shifting is simple and solves the problem for the boards we have (KC705, Kasli), so I have not looked deep into the Xilinx core. If required, the undocumented features are certainly possible to reverse-engineer though, especially as things like 'SecureIP' are not secure at all. I think you could even run at much higher rates than 400Mbps. |
|
@dhslichter do you have such devkit? @sbourdeauducq what exactly do you want to test with it? The FPGA code ADI delivers? |
|
I don't have this devkit. But we should prototype the ARTIQ support using it. |
|
Artiq support is big think, the most important is to make sure that it will work without much pain being driven using SSTL IO instead of LVDS. Another aspect to test is clock. I believe we can test it using devkit with some HW modification. Negative LVDS line can be connected to VTT by FPGA IO pin with internal Thevenin termination enabled. |
|
@sbourdeauducq @gkasprow I have the devkit for the LTC2000-16 (qty 1). It's an FMC card, could be tested out on a KCU105 pretty readily it seems. The eval suite from Analog Devices is designed for the KC705, so easy to test basic functionalities that way. We should be able to buy an additional devkit and loan it to WUT as needed for testing. Last time I looked there were export controls on the LTC2000, so we need to check if additional paperwork would be required to have one go to Hong Kong -- Poland and Germany are both OK IIRC. |
|
I'd avoid another Ultrascale board until Sayma insane behavior that might be attributable to it is sorted out. |
|
@sbourdeauducq agreed. IIUC the idea here is to reuse as much of Sayma as we can, rather than designing a new board from scratch and beginning the bug finding process all over again. So, it doesn't make any sense to begin this (or Metlino for that matter) until we are happy that Sayma works reliably with ARTIQ. |
|
My goal is as @hartytp has stated, which is to make sure we have learned all the lessons we need to learn from Sayma AMC before embarking on a new board, and to be able to reuse parts of Sayma AMC to the extent possible. Based on funding considerations as well, Shuttler is a board that we would plan to develop in earnest starting in 6-12 months, not right now. |
I've started sketching out requirements for a fast DAC to be used for ion shuttling/splitting. It's on the Wiki.
All comments welcome...
The text was updated successfully, but these errors were encountered: