Home » Juniper » Juniper MX packet capture at PFE level (transit traffic)

Juniper MX packet capture at PFE level (transit traffic)

┬áHere is an example of packet capture at PFE level. The purpose of this is to capture a packet leaving your Juniper router, just before it heads to ‘the wire’ In our example, we have a LAG, ae10, with two physical members xe-0/0/0 and xe-1/0/1. Because we don’t know which physical link the packet of interest will be sent to after being hashed, we need to check both.

In the example below I’ve already checked and ruled out xe-0/0/0. My packet is being hashed to xe-1/0/1, so let’s follow this example. On xe-0/0/0 I did the very same thing, however no packets were dumped and at that point I moved to xe-1/0/1.

First logon to the PFE of interest

@mx2.lis.pt.re0> start shell pfe network fpc0
NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash)
NPC1(mx2.lis.pt.re1 vty)#

First, turn off console syslogging, as it will drive you crazy with firewall messages, if you don’t.

NPC1(mx2.lis.pt.re1 vty)# set syslog tty disable
Logging has been disabled on this tty
NPC1(mx2.lis.pt.re1 vty)#

Now identify the PFE ID i.e. MQ chip ID for the ingress/egress interface of interest. eg: Here we try mqchip 0 to see if our IFD (interface xe-1/0/1) is there. I see it does indeed use MQ chip 0, as the IFD xe-1/0/1 is listed:

NPC1(mx2.lis.pt.re1 vty)# show mqchip 0 ifd
 Input    IFD     IFD       LU
 Stream  Index    Name      Sid  TClass
 ------ ------ ---------- ------ ------
<snip>
   1025    201   xe-1/0/1      0     hi
   1026    201   xe-1/0/1      0    med
   1027    201   xe-1/0/1      0     lo
<snip>

Ensure that packet capture is disabled by turning it OFF. If it is already off you should get confirmation.

NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem disable
Packet via DMEM functionality is not enabled

Ready capture

NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem enable

Now we need to build the command string, in the form:

test jnh <MQCHIP ID> packet-via-dmem capture 0x3 <SRCIP><DSTIP>

In our example we know the MQ chip ID is ‘0’. 0x3 is a special value meaning capture the “m2l pkt” and “pkt_head”.

In our example we want to use the following SRC/DST IP pair as filter:

SRC IP = 185.21.217.55
DST IP = 193.136.2.25

But we must convert them to hex:

SRC IP = 185.21.217.55 = b915d937
DST IP = 193.136.2.25 = c1880219

The complete commmand:

NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem capture 0x3 b915d937c1880219
DO NOT OMIT THIS NEXT STEP!

Set packet-via-dmem filter to 0x0, i.e. turn off capture

If you don’t do this, packet-via-dmem will consume resources on the PFE and unpredictable results may arise!

NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem capture 0x0

Dump the capture to console, as hex

NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem dump
PFE 0 Parcel Dump:
Wallclock: 0x0cb6b39a
Received 112 byte parcel:
Dispatch cookie: 0x0070000000000000
0x00 0x0c 0x80 0x00 0x20 0x00 0x01 0x74
0x03 0x01 0x00 0x81 0x03 0x00 0x00 0x00
0x00 0x00 0x0c 0x2a 0x45 0x20 0x00 0x54
0x8a 0x93 0x40 0x00 0x3a 0x01 0x60 0x07
0xb9 0x15 0xd9 0x37 0xc1 0x88 0x02 0x19
0x08 0x00 0x2a 0xc9 0x9e 0x42 0x00 0x01
0x95 0x01 0xb2 0x56 0x00 0x00 0x00 0x00
0x27 0xc8 0x01 0x00 0x00 0x00 0x00 0x00
0x10 0x11 0x12 0x13 0x14 0x15 0x16 0x17
0x18 0x19 0x1a 0x1b 0x1c 0x1d 0x1e 0x1f
0x20 0x21 0x22 0x23 0x24 0x25 0x26 0x27
0x28 0x29 0x2a 0x2b 0x2c 0x2d 0x2e 0x2f
0x30 0x31 0x32 0x33 0x34 0x35 0x36 0x37
Wallclock: 0x0cb6bd50
Sent 115 byte parcel:
0x08 0xbf 0xe0 0x19 0x70 0x00 0x00 0x08
0x20 0x12 0x80 0x08 0x60 0xe0 0xac 0xf1
0x68 0x4e 0x73 0xac 0x4b 0xc8 0x43 0x5f
0xca 0x81 0x00 0x01 0x4d 0x08 0x00 0x45
0x80 0x00 0x54 0x8a 0x93 0x40 0x00 0x3a
0x01 0x5f 0xa7 0xb9 0x15 0xd9 0x37 0xc1
0x88 0x02 0x19 0x08 0x00 0x2a 0xc9 0x9e
0x42 0x00 0x01 0x95 0x01 0xb2 0x56 0x00
0x00 0x00 0x00 0x27 0xc8 0x01 0x00 0x00
0x00 0x00 0x00 0x10 0x11 0x12 0x13 0x14
0x15 0x16 0x17 0x18 0x19 0x1a 0x1b 0x1c
0x1d 0x1e 0x1f 0x20 0x21 0x22 0x23 0x24
0x25 0x26 0x27 0x28 0x29 0x2a 0x2b 0x2c
0x2d 0x2e 0x2f 0x30 0x31 0x32 0x33 0x34
0x35 0x36 0x37
<snip>

IMPORTANT: Disable packet-via-dmem (no confirmation is given, so do it twice)

NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem disable
NPC1(mx2.lis.pt.re1 vty)# test jnh 0 packet-via-dmem disable
Packet via DMEM functionality is not enabled
NPC1(mx2.lis.pt.re1 vty)#

We select a packet; in our example we only sent 3 packets, so we know which is which. In our example we want to see the egress packet headers, so we take the ‘Sent’ parcel. eg:

0x08 0xbf 0xe0 0x19 0x70 0x00 0x00 0x08
0x20 0x12 0x80 0x08 0x60 0xe0 0xac 0xf1
0x68 0x4e 0x73 0xac 0x4b 0xc8 0x43 0x5f
0xca 0x81 0x00 0x01 0x4d 0x08 0x00 0x45
0x80 0x00 0x54 0x8a 0x93 0x40 0x00 0x3a
0x01 0x5f 0xa7 0xb9 0x15 0xd9 0x37 0xc1
0x88 0x02 0x19 0x08 0x00 0x2a 0xc9 0x9e
0x42 0x00 0x01 0x95 0x01 0xb2 0x56 0x00
0x00 0x00 0x00 0x27 0xc8 0x01 0x00 0x00
0x00 0x00 0x00 0x10 0x11 0x12 0x13 0x14
0x15 0x16 0x17 0x18 0x19 0x1a 0x1b 0x1c
0x1d 0x1e 0x1f 0x20 0x21 0x22 0x23 0x24
0x25 0x26 0x27 0x28 0x29 0x2a 0x2b 0x2c
0x2d 0x2e 0x2f 0x30 0x31 0x32 0x33 0x34
0x35 0x36 0x37

To decode this in tshark, we need to clean it up a little first. Remove the 0x which is just denoting that the value is hex.

If you save the data into a file, you can use sed:

cat dumpfile | sed 's/0x//g'

OR

sed -i 's/0x//g' dumpfile

If if you use Notepad/Notepad++:

* Paste into new document * CTRL+H to open Find/Replace * Find: ‘0x’ * Replace # null * CTRL+A to replace all * Done!

What we have is not the ‘on-the-wire’ frame; there are some bytes we need to discard. The best way to do this is find the Ethertype byte and work backwards from there.

Search for:

IPv4 = 0x0800
IPv6 = 0x86dd
MPLS = 0x8847

In this case, we have IP traffic, so we keep every byte from 12 bytes before the Ethertype, until the end. e.g.: In the example given above, discard the first 17 bytes. Note that this is NOT a fixed value – may be 16/17/18 bytes depending on VLAN tagging, and other factors.

Back to our example frame, we throw away 17 bytes and are left with:

4e 73 ac 4b c8 43 5f
ca 81 00 01 4d 08 00 45
80 00 54 8a 93 40 00 3a
01 5f a7 b9 15 d9 37 c1
88 02 19 08 00 2a c9 9e
42 00 01 95 01 b2 56 00
00 00 00 27 c8 01 00 00
00 00 00 10 11 12 13 14
15 16 17 18 19 1a 1b 1c
1d 1e 1f 20 21 22 23 24
25 26 27 28 29 2a 2b 2c
2d 2e 2f 30 31 32 33 34
35 36 37

Now we are now going to decode this in tshark. You can do this via CLI yourself …

OR

Alternatively you can just paste into http://sadjad.me/phd/.

Via CLI:

Paste the data above into a file called ‘dumpfile’, eg:

$ nano dumpfile, then paste, then F3 to save, then CTRL+Q to quit.

* Create a new file, starting with “00000000 ” (-n switch suppresses newline) … * Then append your dumpfile to it, but change newlines to spaces using tr … NOTE: We have to do this because tshark can’t read from stdin. * tshark requires a byte string prepended with the hex offset in number of bytes. * In this case, we start at the 0th byte, hence 00000000.

$ echo -n "00000000 " > dumpfile1 && cat dumpfile | tr '\n' ' ' >> dumpfile1

Now write the PCAP:

$ text2pcap dumpfile1 dumpfile.pcap
Input from: dumpfile1
Output to: dumpfile.pcap
Output format: PCAP
Wrote packet of 98 bytes.
Read 1 potential packet, wrote 1 packet (138 bytes).

Now you can use tcpdump to read the PCAP. emoticon
At this point you’ve used the webpage or tshark directly, and you should see your headers.

You may be wondering if you can do this for multiple packets – yes you can. You will need to perform the same steps for each packet. The webpage ‘Packet Hexdump Decoder’ only decodes one packet at a time. If using the tshark method, you will need to create a textfile with <offset> <dump>, one packet per line. The first line will have offset ‘00000000’, and subsequent lines need to have their offset properly calculated. eg: If the first line has 40 bytes of dump, the second line needs offset ‘00000028’, decimal 40 = hex 28 The text2pcap manpage explains this better than I can.

Enjoy!


Credits:

James Burnett @ GEANT

Kamalesh Rajendran @ Juniper JTAC

David Roy @ Orange

Sadjad Fouladi @ sadjad.me


References:

https://www.wireshark.org/docs/man-pages/text2pcap.html

https://www.wireshark.org/docs/man-pages/tshark.html

http://www.juniper.net/us/en/training/jnbooks/day-one/networking-technologies-series/packet-walkthrough-mx-series/

http://junosandme.over-blog.com/article-trio-card-packet-capture-pfe-commands-115251441.html

http://sadjad.me/phd/