How to connect a PCIe device to a chipyard design - scala

Question

I'm trying to connect a PCIe device to a chipyard design using the existing edge overlay for the VCU118 (slightly modified because I'm using a different board but this should not matter).
#michael-etzkorn already posted an issue on Github about this in which they explain how they only got this working using two different clocks.
I'd appreciate it if I could get some pointers as to how this is done (the issue leaves out some implementation details of the configs) and also if it would be possible to do this without adding an extra clock (#michael-etzkorn points out that this could cause some issues).

Answer 1

Based on the work in your gist, it looks like you've answered most of your original question, but since I already typed this out I'll include it as an answer here.
To hook up any port, you'll essentially need to do three things.
Create an IOBinder
Create a HarnessBinder
Hook up the diplomatic nodes in the TestHarness
The IOBinder takes the bundles from within the system and punches them through to chiptop. The HarnessBinder connects the IO in ChipTop to the harness. Diplomacy negotiates the parameters for the diplomatic nodes. This step may be optional, but many modules, like the XDMA wrapper in the PCIe overlay, are diplomatic so this is usually a required step.
IOBinder
The IOBinder can take your CanHAveMasterTLMMIOPort and punch out pins for it
class WithXDMASlaveIOPassthrough extends OverrideIOBinder({
(system: CanHaveMasterTLMMIOPort) => {
val io_xdma_slave_pins_temp = IO(DataMirror.internal.chiselTypeClone[HeterogeneousBag[TLBundle]](system.mmio_tl)).suggestName("tl_slave_mmio")
io_xdma_slave_pins_temp <> system.mmio_tl
(Seq(io_xdma_slave_pins_temp), Nil)
}
})
This can look relatively the same for each port. However, I've found experimentally, I had to flip the temp pin connection <> for CanHaveSlaveTLPort.
HarnessBinder
The harness binder retrieves that port and connects it to the outer bundle. The pcieClient bundle is retrieved from the harness and connected to ports.head which is returned from IOBinders. This is essentially a fancy functional programming way to clone the IO and connect it to the bundle in ChipTop.
class WithPCIeClient extends OverrideHarnessBinder({
(system: CanHaveMasterTLMMIOPort, th: BaseModule with HasHarnessSignalReferences, ports: Seq[HeterogeneousBag[TLBundle]]) => {
require(ports.size == 1)
th match { case vcu118th: XDMAVCU118FPGATestHarnessImp => {
val bundles = vcu118th.xdmavcu118Outer.pcieClient.out.map(_._1)
val pcieClientBundle = Wire(new HeterogeneousBag(bundles.map(_.cloneType)))
// pcieClientBundle <> DontCare
bundles.zip(pcieClientBundle).foreach{case (bundle, io) => bundle <> io}
pcieClientBundle <> ports.head
} }
}
})
Also, I should note: this isn't the ideal way to connect to the harness as its possible BundleMap user fields are generated and they won't be driven unless you have that pcieClientBundle <> DontCare there. I found I had to expose AXI ports instead and modified the overlay to output axi nodes to get Diplomacy to work between the testharness and ChipTop regions.
A write up of that problem along with some more info at:
What are these `a.bits.user.amba_prot` signals and why are they only uninitialized conditionally in my HarnessBinder?
All this code is at that question.
TestHarness Diplomatic Connections
val overlayOutput = dp(PCIeOverlayKey).last.place(PCIeDesignInput(wrangler=pcieWrangler.node, corePLL=harnessSysPLL)).overlayOutput
val (pcieNode: TLNode, pcieIntNode: IntOutwardNode) = (overlayOutput.pcieNode, overlayOutput.intNode)
val (pcieSlaveTLNode: TLIdentityNode, pcieMasterTLNode: TLAsyncSinkNode) = (pcieNode.inward, pcieNode.outward)
val inParamsMMIOPeriph = topDesign match { case td: ChipTop =>
td.lazySystem match { case lsys: CanHaveMasterTLMMIOPort =>
lsys.mmioTLNode.edges.in(0)
}
}
val inParamsControl = topDesign match {case td: ChipTop =>
td.lazySystem match { case lsys: CanHaveMasterTLCtrlPort =>
lsys.ctrlTLNode.edges.in(0)
}
}
val pcieClient = TLClientNode(Seq(inParamsMMIOPeriph.master))
val pcieCtrlClient = TLClientNode(Seq(inParamsControl.master))
val connectorNode = TLIdentityNode()
// pcieSlaveTLNode should be driven for both the control slave and the axi bridge slave
connectorNode := pcieClient
connectorNode := pcieCtrlClient
pcieSlaveTLNode :=* connectorNode
Clock Groups... (unsolved)
pcieWrangler was my attempt at hooking up the axi_aclk. It's not correct. This just creates a second clock with the same 250 MHz frequency as the axi_aclk and so it mostly works but using a second clock isn't correct.
val sysClk2Node = dp(ClockInputOverlayKey).last.place(ClockInputDesignInput()).overlayOutput.node
val pciePLL = dp(PLLFactoryKey)()
pciePLL := sysClk2Node
val pcieClock = ClockSinkNode(freqMHz = 250) // Is this the reference clock?
val pcieWrangler = LazyModule(new ResetWrangler)
val pcieGroup = ClockGroup()
pcieClock := pcieWrangler.node := pcieGroup := pciePLL
Perhaps you can experiment and find out how we can hook up the axi_aclk as the driver for the axi async logic :)
I'd be happy to open a question about that since I don't know the answer yet myself.
To answer some follow up questions
How do I know how big of an address range I should reserve for PCIe (master and control)?
For control, you can match the size of in the overlay 0x4000000. For the master port, you'd ideally just hook a DMA engine up to that that has access to the full address range on the host. Otherwise, you have to do AXI2PCIE BAR translation logic to access different regions of host memory which isn't fun.
How do I connect the interrupt node to the system?
I believe this is only if you're connecting a PCIe Root Complex. If you're connecting a device, you shouldn't need to worry about the interrupt node. If you do wish to add it, you'll have to add the NExtInterrupts(3)++ to your config. I only got so far as that and my uncommented code in the TestHarness before I realized I didn't need it. If you feel you do, we can open a new question and try answering this more fully.