Copyright 2012 Xilinx

Xilinx Answer 50234

Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide




Important Note: This downloadable PDF of an answer record is provided to enhance its usability and
readability. It is important to note that answer records are Web-based content that are frequently updated as new
information becomes available. You are reminded to visit the Xilinx Technical Support Website and review Xilinx
Answer 50234 for the latest version of this solution.





Introduction
This document describes techniques for debugging issues related to Integrated PCIe

Block Wrapper in Virtex

-6
FPGA by using ChipScope

analyzer. You should have a clear understanding on the flow of packets through different
interfaces of your design when debugging PCIe issues. You should be able to identify what types of packets are seen
on those interfaces by analyzing the packet header. This document provides a detailed description on tracking
packets through different interfaces in the core. Some helpful techniques on how to select signals for triggering in
ChipScope analyzer and how to set the correct trigger to capture packets on different interfaces are discussed.

The main objective of this document is to help you debug your design by going into the low level details of the core.
This document will help you to debug issues related to both link training and PCIe packet traffic. ChipScope analyzer
screenshots are provided to illustrate how to analyze packets and signals.
Signal and Packet Analysis Interfaces
Figure 1 shows a block diagram of different components in Integrated PCIe Block in Virtex-6 FPGA. It shows different
interfaces between those components. The interfaces are numbered to make it easier to reference in rest of the
document.


Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 1


Copyright 2012 Xilinx

Figure 1 Top-level Functional Blocks and Interfaces

The main interfaces are as follows:

1. PCIe Link Interface
2. Transceiver Interface
3. MIM Interface
4. TRN TX Interface
5. TRN RX Interface
6. PIO TX Interface
7. PIO RX Interface
8. CFG Interface
9. Physical Layer Interface

In this document, it is illustrated on how to use ChipScope analyzer to capture and track a particular packet along
each of these interfaces. It does not offer much flexibility in triggering for the correct capture with only one trigger port.
If you need to trigger only when a completion packet is presented to the core for a particular incoming read, you would
need to trigger only after two conditions are met; memory read packet on the receive interface and the corresponding
completion packet on the transmit interface. In the following sections, techniques to capture packets in similar
scenarios are discussed.
Inserting ChipScope Core
This section describes the process of using ChipScope

Pro Core Inserter to capture PCIe signals. For more flexibility

in triggering and hence being able to trigger in a specific condition, 8 trigger ports have been created. These groups
can be categorized as follows:

LTSSM signal
Global signals
TX/RX PIPE interface control/data signals
TX/RX MIM interface control/data signals
TX/RX TRN interface control/data signals

In the example below, all of the data signals have also been included as trigger signals. The main reason for doing
this is to be able to trigger ChipScope analyzer based on the data being received or transmitted (e.g. if it is required to
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 2


Copyright 2012 Xilinx
trigger on the incoming TLP at the GT receive interface, the trigger could be set by assigning FB to the GT receive
interface data signals).

The ChipScope Pro Core Inserter flow is used to capture signals in the design. The input file is the design NGC file. In
the case that the signals are not visible in the ChipScope Pro Core Inserter GUI, the reason could be because these
signals are optimized away during synthesis. You can avoid this by using KEEP attribute in the source file to stop XST
from optimizing a particular signal. This is illustrated in the example code below:

In VHDL, declare the KEEP attribute in the file architecture before the begi n keyword:

at t r i but e keep: st r i ng

After KEEP and the signal have been declared, specify the VHDL constraint as follows:

at t r i but e keep of signal_name: si gnal s i s t r ue;

In Verilog, add the following:

( * KEEP = " {TRUE}" *) wi r e signal_name;

Following are the steps for using ChipScope Pro Core Inserter.


Figure 2 Select NGC File

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 3



Figure 3 Select 8 Trigger Ports
Figure 3 shows the selection of 8 trigger ports. Each port can be separately configured depending upon the number of
signals to be included in each port. Figure 4 shows the Data Width and Data Depth selections, and the width selected
for each port.


Figure 4 - Set Capture Parameters for Each Port
Figure 5 shows an overview of all three ports: Clock Port, Trigger Ports and Data Port.
Copyright 2012 Xilinx
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 4


Copyright 2012 Xilinx


Figure 5 Final Port Structure after Selecting Signals for Each Port
Figure 6 shows the module of hierarchy of the example design. The list of signals in each module can be viewed by
selecting the modules in the Structure/Nets window.
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 5


Copyright 2012 Xilinx

Figure 6 - Example Design Project Structure in ChipScope Analyzer
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 6


Copyright 2012 Xilinx
Trigger Port 0
Figure 7shows a list of signals selected for Trigger Port 0. All of these signals are related to GTs. Also, trn_lnk_up_n
signal is listed in this trigger port. When trn_lnk_up_n signal is asserted, it indicates the core has finished link training
and is now in L0 state. While you are debugging issues related to link training, it would be useful to check the status of
these signals in ChipScope analyzer. These signals should be checked as the first step in debugging link training
issues. All of these signals should toggle correctly.

GTXRXRESET: This port is asserted High, and then de-asserted to start the full GTX transceiver RX reset sequence.
The sequence takes about 120 s to complete, and systematically resets all subcomponents of the GTX transceiver
RX.

RXRESETDONE: This port goes High when GTX transceiver RX has finished reset and is ready for use.

TXDETECTRX: This input activates the receive detection sequence. The sequence ends when PHYSTATUS is
asserted to indicate that the results of the test are ready on RXSTATUS.

TXRESETDONE: This port goes High when the GTX transceiver TX has finished reset and is ready for use.

PLLLKDET: This active-high PLL frequency lock signal indicates that the PLL has achieved lock. The GTX
transceiver and its clock outputs are not reliable until this condition is met.


Figure 7 Signals in Trigger Port 0
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 7


Copyright 2012 Xilinx
Trigger Port 1
This port includes the ltssm signal which is the main signal when debugging link training issues. Whenever there is a
problem with the link, it is always advised to check this signal in ChipScope analyzer to see what state the core is in.
Under normal working conditions, this signal will show the value indicating L0 state. If the link is bad due to signal
integrity issues on the board, the ltssm frequently goes into recovery state which is an indication of a bad link.


Figure 8 - Add Signals in Trigger Port 1
Trigger Port 2
This port includes signals coming out of the GT interface. Probing the signals on this interface helps determine
whether there are any issues at the physical level. If the host is sending packets and there is no toggling on the rxdata
or txdata, it would indicate that the GTs are not properly constrained. During link training, the physical ordered sets
TS1s and TS2s are exchanged between the link partners. If the link is not coming up, you should always check this
interface to make sure TS1s and TS2s are correctly exchanged.

The data on txdata and rxdata are scrambled, however, the control characters such as FB (Start of TLP), FD (End of
TLP) and so on, are not scrambled. If you think that a packet is not coming into the core, or is not going out, probe
rxdata and txdata respectively, and look for the FB character followed by the FD character. The FB character is
qualified by pipe_rxn_char_is_k<n>signal indicating that the corresponding character on rxdata is a control character.
The presence of FB on rxdata or txdata indicates the presence of a TLP. In the case when you are generating a MSI
interrupt and the MSI packet is not making it to the root complex, the best way to check is to make sure the packet has
gone out of the core or not. There is no signal in the core that indicates a packet has successfully transmitted to the
PCIe link but if you can see FB on the txdata, that would be a very strong indication that the packet has gone out of
the core. MSI packet has a definite size. Check for FB on the GT interface on txdata, and then FD after that. Count the
number of bytes between these two characters, if it is equal to the number of a valid MSI message, this would indicate
that the MSI packet generated by the core has gone out.

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 8


Copyright 2012 Xilinx

Figure 9 - Add Signals in Trigger Port 2 (Part 1)
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 9


Copyright 2012 Xilinx

Figure 10 - Add Signals in Trigger Port 2 (Part 2)













Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 10


Copyright 2012 Xilinx


Figure 11 - Add Signals in Trigger Port 2 (Part 3)

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 11


Copyright 2012 Xilinx
Trigger Port 3

Figure 12 - Add Signals in Trigger Port 3 (Part 1)







Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 12


Copyright 2012 Xilinx


Figure 13 - Add Signals in Trigger Port 3 (Part 2)
Trigger Port 4
This port includes all signals at the receive side of the RX buffer interface (MIM RX Interface). All incoming packets
are first stored in this buffer before sending out to the user interface. If a packet is not making it to the user interface,
first check this interface by triggering on mim_rx_wen and verify if you can see the packet on this interface or not. If
you do see the packet written into the buffer, check whether the packet is read back from the buffer or not. If you see
the packet on this interface and not on the user interface, this would be an indication that the packet has been
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 13


Copyright 2012 Xilinx
dropped inside the core. This is very unlikely to happen but if you run into any such scenarios, please create a
WebCase with Xilinx Technical Support.


Figure 14 - Add Signals in Trigger Port 4
Trigger Port 5
Similar to the incoming packets, the outgoing packets are also buffered in the TX Buffer before they are transmitted.
As discussed for the receive buffer, perform a similar test for the receive buffer in the case where you could see a
packet being transmitted by the user application on the TRN TX interface (Figure 1, Interface 4), but it is not showing
up at the host.





















Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 14


Copyright 2012 Xilinx


Figure 15 TX Memory Interface Signals in Trigger Port 5
Trigger Port 6
This port includes all receive side signals between the core and the user application. If the host is sending packets
downstream and you think your backend application is not receiving those packets, then the first thing to do is to
check this interface. One probable cause for the packet not making it to the user interface might be that the incoming
packet is not hitting the BAR range. If this is the case, the corresponding signals: trn_rbar_hit_n<n>will not be
asserted. In that case, you should make sure the application at the host is targeting the correct memory address. If
you want to check whether or not any packets are making it to the user application, set your trigger on trn_rsof_n. In
the case where the host is performing memory read from the endpoint user application and you are not getting the
corresponding completion back from the endpoint, you should first verify whether or not you can see Memory Read
TLP on the trn receive interface.

















Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 15


Copyright 2012 Xilinx


Figure 16 TRN Receive Interface Signals in Trigger Port 6
Trigger Port 7
This port includes the TX side of the signals between the user application and the core. If the host is not receiving
completions for the memory reads, probe this interface and verify whether or not the completion packet has been
generated by your user application and presented to the core.

In the case when there are no sufficient credits at the host, the core de-asserts trn_tdst_rdy_n. This signal is also de-
asserted if the core is not ready to accept packets from the user application. If you do not see any packets coming in
at the host from the endpoint, trigger ChipScope analyzer on this signal and make sure it is not de-asserted
indefinitely.
















Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 16


Copyright 2012 Xilinx


Figure 17 TRN Transmit Interface Signals in Trigger Port 7

Clock Signal

Figure 18 - Clock Signal
Select Data the same as Trigger in Figure 4. This will add all trigger signals in the data waveform as well.











Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 17


Copyright 2012 Xilinx


Figure 19 - Add Data Signals

After inserting the ChipScope IP core, run the implementation again (without running the synthesis) to generate the
bitstream. After programming the device on ML605, open ChipScope analyzer, the ChipScope IP core should be
detected by the analyzer.
LTSSM Signal Analysis
After the FPGA configuration, the two connected devices go through the link training process. The main states to
consider while debugging link training issues are DETECT, POLLING, CONFIGURATION, and L0. Please refer to
Chapter 4 of the PCI Express Base Specification for detailed descriptions of these LTSSM states.

The section provides a list of signals to capture for debugging a link training issue in the ChipScope tool along with a
detailed analysis of the signals. A number of ChipScope tool waveform screenshots are provided for each LTSSM
state to illustrate the toggling of signals. If you are capturing signals in the ChipScope tool, compare your captures
with the screenshots provided below to make sure the signals in your design are toggling correctly.

The captures are taken from the PIO Example Design of Virtex-6 FPGA Integrated Block for PCI Express

v1.7
generated from the CORE Generator

graphical user interface for ML605 board.

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 18


Copyright 2012 Xilinx
LTSSM States
The signal below indicates which LTSSM state the core is currently at. The core goes to recovery state to achieve bit
lock and symbol lock. If the ChipScope tool capture shows frequent transition to the recovery state, it normally
indicates a noisy link.

pl_ltssm_state<5:0>

Different states of the link training state machine (indicated by pl_ltssm_state) are shown in Table 1:

000000b:Det Quiet
000001b:Det Quiet Gen2
000010b:Det Active
000011b:Det Active Second
000100b:Pol Active
000101b:Pol Config
000110b:Pol Comp Pre Send Eios
000111b:Pol Comp Pre Tim-out
001000b:Pol Comp Send Pattern
001001b:Pol Comp Post Send Eios
001010b:Pol Comp Post Time-out
001011b:Cfg Lwidth St0
001100b:Cfg Lwidth St1
001101b:Cfg Lwidth Ac0
001110b:Cfg Lwidth Ac1
001111b:Cfg Lnum Wait
010000b:Cfg Lnum Acpt
010001b:Cfg Complete 1
010010b:Cfg Complete 2
010011b:Cfg Complete 4
010100b:Cfg Complete 8
010101b:Cfg Idle
010110b:L0
010111b:L1 Entry0
011000b:L1 Entry1
011001b:L1 Entry2
011010b:L1 Idle
011011b:L1 Exit
011100b:Rec Rcvrlock
011101b:Rec Rcvrcfg
011110b:Rec Speed 0
011111b:Rec Speed 1
100000b:Rec Idle
100001b:Hot Rst
100010b:Disabled Entry0
100011b:Disabled Entry1
100100b:Disabled Entry2
100101b:Disabled Idle
100110b:Dp Cfg Lwidth St0
100111b:Dp Cfg Lwidth St1
101000b:Dp Cfg Lwidth St2
101001b:Dp Cfg Lwidth Ac0
101010b:Dp Cfg Lwidth Ac1
101011b:Dp Cfg Lnum Wait
101100b:Dp Cfg Lnum Acpt
101101b:To 2 Detect
101110b:Lpbk Entry0
101111b:Lpbk Entry1
110000b:Lpbk Active0
110001b:Lpbk Exit0
110010b:Lpbk Exit1
110011b:Lpbkm Entry0
110100b 111111b: Reserved
Table 1 : LTSSM States
pipe_rx_status indicates the receiver status and error codes as shown in Table 2.

000: Data Received OK
001: One Skip Symbol (SKP) added
010: One SKP removed
011: Receiver detected
100: 8B/10B decode error
101: Elastic Buffer overflow
110: Elastic Buffer underflow
111: Receive disparity error
Table 2 : pipe_rx_status Signal
Figure 20 shows the LTSSM state at Pol Config on Trigger Port 1


Figure 20 - Set Different LTSSM States to Trigger
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 19


Copyright 2012 Xilinx
DETECT State
Figure 22 shows a capture of the GT interface signals when the ChipScope tool is triggered on Detect Active state
(pl_ltssm_state<5:0>=000010).

Figure 21 - Trigger LTSSM State - Detect Active (Part 1)




















Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 20


Copyright 2012 Xilinx


Figure 22 - Trigger LTSSM State - Detect Active (Part 2)
During the DETECT state, the receiver detection takes place on each lane. If the detection process is done correctly,
the following sequence should be observed in the ChipScope tool capture; this is also described in (Xilinx Answer
39525).

PCIe Hard Block asserts TxDet ect Rx.

GTP performs receiver DETECT.
After the receiver is detected, GTP asserts pi pe_r x_phy_st at us and puts 011 on pi pe_r x_st at us to
indicate the receiver is present.

PCIe Hard Block then de-asserts TxDet ect Rx and pi pe_t x_el ec_i dl e.

Receiver detect is used by the transmitter to detect if a receiver is present at the other end of the link. The Virtex-6
FPGA Integrated Block for PCI Express and GTX Transceivers support this functionality. However, due to various
reasons such as board signal integrity or issues with the link partner receiver, sometimes it is helpful to bypass the
receiver detect function and allow the core to go straight to the LTSSM polling state. For more details on how to
bypass receiver detect, refer to (Xilinx Answer 45859).

For more details on debugging receiver detect issue, see (Xilinx Answer 39380).

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 21


Copyright 2012 Xilinx
The ChipScope Pro tool capture of the above sequence is shown in Figure 23.


Figure 23 - Detect State Details
Another signal that you should pay attention to is the plllkdet signal. This port indicates that the VCO rate is within
acceptable tolerances of the desired rate; in other words, the assertion of this signal indicates that the internal PLL is
successfully locking on to the incoming reference clock. The clock_locked signal, as shown in Figure 24, indicates
whether or not the fabric PLL has successfully locked. For successful link training, both the plllkdet and clock_locked
signals should be asserted. Therefore, before capturing other signals to debug link training issues, it is advised to
check these signals first to make sure both of them are asserted.


Figure 24 - plllkdet and clock_locked Signals During DETECT
When each link partner enters into POLLING state, it begins transmitting TS1 ordered sets. However, each link
partner might not enter POLLING at the same time, so it is possible that the Xilinx endpoint might be transmitting TS1s
on pi pe_t x_dat a while still receiving 00h on the pi pe_r x_dat a pins. Hence, in the ChipScope Pro tool, when
TS1 appears at pi pe_t x_dat a, pi pe_r x_dat a might still be 00.
POLLING State
To check whether or not TS1 transmission has started, trigger the ChipScope tool when l t ssm_st at e enters
POLLING. The screenshots below show the ChipScope tool capture of the signals when the endpoint device enters
POLLING.ACTIVE state.









Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 22


Copyright 2012 Xilinx


Figure 25 - Trigger LTSSM State Polling.Config (Part 1)
























Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 23


Copyright 2012 Xilinx


Figure 26 - Trigger LTSSM State Polling.Config (Part 2)
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 24


Copyright 2012 Xilinx
CONFIGURATION State

Figure 27 - Trigger LTSSM State Configuration
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 25


Copyright 2012 Xilinx


Figure 28 - Trigger LTSSM State - Configuration Idle (Part 1)























Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 26


Copyright 2012 Xilinx


Figure 29 - Trigger LTSSM State - Configuration Idle (Part 2)
After CONFIGURATION, the next state is L0 which is the normal working state. As shown in Figure 28, the initial
phase of link training completes after trn_lnk_up_n is asserted. For more information on this link trained signal, see
(Xilinx Answer 39522). At times, the link might not come up if configured for higher lane width due to signal integrity
issues on the board. In such cases, the best way to check is to force the link to train down by taping off the upper
lanes as described in (Xilinx Answer 38988). For more information on debugging link training issues, see (Xilinx
Answer 34873).

Common errors that can occur are 8b/10b errors which can lead to the link being lost, the link not coming up, and the
link training down to a lower lane width. For more information on 8b/10b errors, refer to (Xilinx Answer 39590). When
ASPM is enabled at the host, it might cause an issue during link training. It is recommended to disable ASPM. For
more details on how to do disable ASPM, see (Xilinx Answer 36325).

Although trn_link_up_n is asserted, sometimes the device might not be detected by the system. This can be caused
by the FPGA configuration timing. For more information and on how to debug this issue, see (Xilinx Answer 34800).

During normal running of the system, it could freeze. Although this might be due to various reasons, one probable
cause could be completion timeouts which is described in (Xilinx Answer 35034). Other causes could be the
generation of malformed TLPs due to incorrect use of trn_trem_n as described in (Xilinx Answer 35748).


Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 27


Copyright 2012 Xilinx
Tracking Received TLPs
This section describes how to track an incoming TLP packet all the way from the GT RX interface to the user
application RX interface.

As shown in Figure 30, an incoming TLP packet (highlighted with red arrows) will arrive on the RX GTX interface (2),
and goes into and comes out of the MIM interface (3), and the core then presents it on the TRN RX interface (4).


Figure 30 Incoming TLP Packets on Different Interfaces
The packet that was tracked in the example below was performed by writing to the backend memory of the example
design user application and reading back from the same memory location in the PciTree. The incoming TLP packet on
the GT RX interface is identified by checking the FB (start of TLP packet) character on rxdata. The example design
was generated for four lanes, the FB (1byte) will start on pipe_rx0_data, and FD (which indicates the end of TLP
packet) will be on pipe_rx3_data.

The ChipScope analyzer setup, described earlier in this document, has 7 trigger ports in total. In M2 and M3, there are
PIPE RX and TX interface signals respectively. To trigger on an incoming FB, trigger on either the lower 8-bit or the
upper 8-bit of pipe_rx0_data by setting it to FB as shown below:

pipe_rx0_data =16b FBXX & pipe_rx0_char_is_k =2b10

OR

pipe_rx0_data =16b XXFB & pipe_rx0_char_is_k =2b01

The input pipe_rx0_char_is_k signal determines the control bit(s) for the received data; the lower bit corresponds to
the lower byte of pipe_rx0_data, while the upper bit corresponds to the upper byte.

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 28


Copyright 2012 Xilinx
Tracking Vendor Defined Messages
Figure 31 and Figure 32 show how to set the trigger in Trigger Port M2 to track the start of the TLP packet (FB) which
could be on either the higher or lower byte of pipe_rx0_data[15:0].

Figure 31 - Trigger FB on pipe_rx_data with Upper Data Byte Set

If the above setting does not trigger, try setting the FB trigger on the lower byte of the data as shown below:


Figure 32 - Trigger FB on pipe_rx_data with Lower Data Byte Set
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 29


Copyright 2012 Xilinx
Meanwhile, you must enable M2 to be in the Trigger Condition Equation as shown in Figure 33.

Figure 33 - Enable M2 as Trigger
Press the trigger button in the ChipScope tool. The next step is to perform Memory Write in PciTree. The
expectation is that the ChipScope tool will get triggered with the incoming Memory Write packet. However, in this
particular test, even without performing Memory Write in the PciTree, ChipScope gets triggered due to Vendor
Defined messages that are sent by the host to the Endpoint.

Figure 34 shows a TLP coming from the host to the Endpoint. FB is on pipe_rx0_data and the corresponding FD is
on pipe_rx3_data on the GTX interface.


Figure 34 - Vendor Defined TLP Packet on GTX Interface
Figure 35 shows a close-up view of the packet on the MIM interface. The xx73. on the mim_rx_wdata when
mim_rx_wen is asserted indicates a Vendor Defined Message.


Figure 35 - Vendor Defined TLP Packet on MIM Interface
The same packet travels further to the TRN interface as shown in Figure 36. The signal trn_rsof_n is asserted with
0x7300_0001_0000_007F on trn_rd. The data starting with 0x73 means a Vendor Defined Message TLP.


Figure 36 Vendor Defined TLP Packet on TRN Interface
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 30


Copyright 2012 Xilinx
Tracking Memory Write TLP
Since the ChipScope tool is getting triggered anyway due to the host sending these Vendor Defined Messages
continuously to the Endpoint core, the main question here is how to trigger only when Memory Write TLP is received
in the core.

0x40 at the start of a TLP indicates a Memory Write TLP packet. Set the trigger condition when trn_rsof_n is asserted
and trn_rd =0x40xx_xxxx_xxxx_xxxx as shown in Figure 37.


Figure 37 - Trigger on trn_rsof_n and Memory Write TLP
Then, set the trigger condition to M6 as shown in Figure 38.


Figure 38 - Enable M6 Port to Trigger

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 31


Copyright 2012 Xilinx
After pressing the trigger button in the ChipScope tool, perform a Memory Write 0xdeadbeef to 0x00000008 in
the PciTree as shown in Figure 40. After the ChipScope tool triggers, the next step is to track back the packet in these
interfaces: TRN ->MIM ->PIPE, and verify whether or not the 0xdeadbeef packet appears on all these interfaces.


Figure 39 PciTree Xilinx PCI Express Block Plus Configuration Space
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 32


Copyright 2012 Xilinx

Figure 40 - Memory Write using PciTree
Figure 41 shows deadbeef on the TRN interface.


Figure 41 - Memory Write TLP packet on TRN Interface

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 33


Copyright 2012 Xilinx
If the incoming Memory Write TLP is correct and is hitting the correct BAR address, trn_rbar_hit_n<0>should be
asserted as shown in Figure 42.


Figure 42 - trn_rbar_hit_n<0> is Asserted
Figure 43 shows the same packet on the MIM interface.


Figure 43 - Memory Write TLP Packet on MIM Interface
Figure 44 shows the same packet on the PIPE interface. As shown in the figure, FB on pipe_rx0_data with
pipe_rx0_char_is_k asserted indicates the start of the Memory Write TLP. The corresponding FD on pipe_rx3_data
with assertion of pipe_rx3_char_is_k indicates the end of the Memory Write TLP. The packet in Figure 44 is definitely
a Memory Write TLP since you are tracking the packet back from TRN and MIM interface. It is not possible to tell
whether the packet is Memory Write TLP, Memory Read TLP, or some other packet types by just checking the packet
data on the PIPE interface because the packets are scrambled.


Figure 44 - Memory Write TLP Packet on PIPE Interface

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 34


Copyright 2012 Xilinx
Tracking Memory Read and Completion TLP
When performing a Memory Read request, as it is a Non-Posted request, there will be a corresponding completion
TLP coming back from the Endpoint. As shown in Figure 45, the memory read TLP (highlighted in red) will go through
interfaces (2) ->(3) ->(4) ->(6); and the completion TLP (highlighted in green) will come back through interfaces (7) -
>(5) ->(3) ->(2).


Figure 45 Tracking Memory Read and Completion Packet
To track both Memory Read TLP and the corresponding completion packet, use triggers on both the TRN RX and
TRN TX side. When you perform a memory read, trn_rsof_n is asserted with a TLP header trn_rd
0x00xx_xxxx_xxxx_xxxx on the TRN RX interface. The 0x00 means a 32-bit Memory Read request. On the TX
side of the TRN interface, there will be the corresponding Completion. To trigger on this packet, set trigger with
trn_tsof_n asserted and trn_td =0x4Axx_xxxx_xxxx_xxxx. Figure 46 and Figure 47 show the correct trigger settings
as described.


Figure 46 - Triggers for Memory Read Packet
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 35


Copyright 2012 Xilinx


Figure 47 - Triggers for Completion Packet from Endpoint
As the Memory Read comes first, then is followed by the completion packet, perform the trigger sequence as shown in
Figure 48.


Figure 48 - Enable Sequence Trigger

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 36


After setting the triggers as described above, press the trigger button, and then click on
button in the PciTree as shown in Figure 49.


Figure 49 - Auto Read Memory in PciTree
Figure 50 shows a capture of TRN interface with Memory Reads and the corresponding completion packets.


Figure 50 - Memory Read and Completion Packet
Copyright 2012 Xilinx
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 37


Copyright 2012 Xilinx
Figure 51 shows a close-up view of Figure 50. On the TRN RX interface, trn_rsof_n is asserted first with
0x00xx_xxxx_xxxx_xxxx, which means a 32-bit Memory Read request. On the TRN TX interface, there is the
corresponding completion packet with the assertion of trn_tsof_n as indicated by the cursor.


Figure 51 - Completion Packet
In Figure 51, there are two completion packets on the TRN TX interface. The main question here is which completion
packet belongs to the Memory Read on the TRN RX interface. This can be known by checking the TAG field of the
packet header.

Figure 52 shows the Memory Read request header format.

Figure 52 - Request Header Format for 32-bit Addressing of the Memory
In Figure 51, trn_rd =0x0000_0001_0000_070F.

0x0000_0001_0000_070F=
0000_0000_0000_0000_0000_0000_0000_0001_0000_0000_0000_0000_0000_0111_0000_1111

Figure 53 shows the analysis of the Memory Read packet header which gives the TAG as 0x07.


Figure 53 - Memory Read Packet Header

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 38


Copyright 2012 Xilinx
The two completion packets in Figure 51 are as follows.

trn_td =0x4a00_0001_0200_0004_0000_0704_0000_0000
trn_td =0x4a00_0001_0200_0004_0000_0808_EFBEADDE

The Completion header format is shown in Figure 54.

Figure 54 - Completion Header Format
The breakdown of the first completion packet header will be as follows:

0x4a00_0001_0200_0004_0000_0704_0000_0000=
0100_1010_0000_0000_0000_0000_0000_0001_0000_0010_0000_0000_0000_0000_0000_0100_
0000_0000_0000_0000_0000_0111_0000_0100_0000_0000_0000_0000_0000_0000_0000_0000



Figure 55 - Completion Packet Header
Figure 55 shows the TAG value in this completion packet as 0x07 which is the same as the TAG value in the Memory
Read TLP. The second completion packet has TAG field of 0x08.

Since multiple trigger was done in the ChipScope tool, auto memory read in PciTree reads all memory locations in
the address range, they are read back as all 0x0000_0000 which is shown in Figure 56.


Figure 56 - Memory Read Request with Completion Packets with data = 0x0000_0000

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 39


Copyright 2012 Xilinx
Figure 57 and Figure 58 show the Memory Read packet in Figure 53 on MIM interface.

Figure 57 - Memory Read on MIM Interface

Figure 58 - Memory Read (0xdeadbeef) on MIM Interface
Figure 59 shows the Memory Read data 0x0000_0000 on MIM interface shown in Figure 56.


Figure 59 - Memory Read (0x000000000) on MIM Interface
If you do not see completion packets coming to the host, make sure the completion packet is sent by the user
application to the Endpoint core by probing the TRN interface, then check the MIM interface, and then the PIPE
interface. If you see the completion packet on the TRN interface, you can check whether or not that packet made it to
the PIPE interface by tracking the corresponding FB character on the PIPE TX interface.


Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 40


Copyright 2012 Xilinx
Set M3 pipe_tx0_data [0:7] =FB or pipe_tx0_data [8:15] =FB as shown in Figure 60 and Figure 61.


Figure 60 - Trigger on FB on pipe_tx0_data Higher Byte

Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 41



Figure 61 - Trigger on FB on pipe_tx0_data Lowe Byte
Set the trigger as follows:

M6 (Memory Read on TRN RX Interface) ->M7 (completion packet on TRN TX Interface) ->M3 (FB on PIPE
Interface).


Figure 62 Sequence Trigger to Track Outgoing FB on pipe_tx_data for the Corresponding Completion
Packet on TRN TX Interface

Copyright 2012 Xilinx
Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 42


Copyright 2012 Xilinx
Figure 63 and Figure 64 shows FB and FD for completion packets on TX PIPE interface.


Figure 63 - Memory Read on PIPE Interface

Figure 64 - FB and FD Shown on PIPE Interface
Conclusion
This document presented different aspects of debugging issues in the Virtex-6 FPGA Endpoint Block Wrapper by
going through the packet and signal analysis. If the user of the core is experiencing issues with link training or the
packets are not transmitted or received correctly, it is recommended to go through this document and check the
provided suggestions. It is expected that with this document, the user will be able to capture the relevant signals
specific to the issue in the ChipScope tool, and perform analysis of the captured waveform to figure out where the
problem might be.

If this document does not help to resolve the problem, please create a WebCase with Xilinx Technical Support. Attach
all of the captured ChipScope Pro tool waveforms, and the details of your investigation and analysis.
References
1. Virtex-6 FPGA Integrated Block for PCI Express User Guide
2. Xilinx PCI Express Documents
3. Virtex-5 Integrated PCI Express Block Plus - Debugging Guide for Link Training Issues
4. Virtex-5 Endpoint Block Plus for PCI Express - Debugging and Packet Analysis Guide with Downstream Port
Model and PIO Example Design
5. IP Release Notes
6. Xilinx Solution Center for PCI Express
7. ChipScope Pro Tutorials
8. ChipScope Pro User Guide
Revision History
08/16/2012 - Initial Release



Xilinx Answer 50234 - Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 43