Previous Section  < Free Open Study >  Next Section

Bandwidth and Delay Measurements

As you know by now, MPLS TE is mostly about moving traffic around on your network, away from congested points and onto less-congested paths.

But what traffic do you move? From where? To where?

In order to move traffic away from spots that would otherwise be congested, you first need to know how much traffic you want to send across the network. You also need to know something about the properties of the paths onto which you'd move traffic.

To make these determinations, you need accounting and measurement tools, some of which you can find in Cisco IOS Software. Here are some of the many different measurement tools provided by Cisco IOS Software that you can use to figure out where your congestion lies:

  • Physical interface statistics

  • IP accounting

  • CEF accounting

  • BGP policy accounting

  • NetFlow accounting

  • Traffix Matrix Statistics (TMS)

  • Using TE tunnels for traffic measurement

  • Service Assurance Agent (SAA)

All these tools do different things, however, so they might not give you all the information you need to figure out how much traffic is going where in your network.

The most important thing you need to know about your network traffic is how much of it there is. This chapter covers NetFlow, TMS, and a method of using TE tunnels to measure traffic demand.

Another thing that's useful to measure is the delay and jitter from one point in the network to another. You can measure delay and jitter using SAA, which is also covered in this chapter.

The first four measurement tools (interface statistics, IP accounting, CEF accounting, and BGP policy accounting) are not covered in this chapter or this book. Check www.cisco.com for more information about those specific tools.

In order to successfully place TE tunnels in your network, the first thing you need to do is decide what the border of your TE tunnel network will be—whether you have TE tunnels edge-to-edge, or only within the core.

As soon as you've decided where TE tunnels will be placed, start thinking of this area of your network as the TE cloud. If you're going to apply a strategic full mesh of some kind, whether edge-to-edge or only in the WAN, you need to get an idea of what traffic is going into this cloud, where it's entering, and where it's leaving. This is commonly known as a traffic matrix. Figure 10-1 shows the TE cloud within a small sample network used in the TMS discussion.

Figure 10-1. TE Cloud Within a Network

graphics/10fig01.gif

A traffic matrix is a measurement of how much traffic is going from an entry point in the TE cloud to an exit point from that TE cloud. Table 10-1 shows a possible traffic matrix for the network shown in Figure 10-1.

Table 10-1. Traffic Matrix for Figure 10-1
From Seattle San Jose Dallas New York Miami
To          
Seattle N/A 500 Mbps 100 Mbps 300 Mbps 200 Mbps
San Jose 525 Mbps N/A 175 Mbps 400 Mbps 225 Mbps
Dallas 75 Mbps 350 Mbps N/A 200 Mbps 50 Mbps
New York 250 Mbps 600 Mbps 190 Mbps N/A 160 Mbps
Miami 50 Mbps 200 Mbps 100 Mbps 230 Mbps N/A

Note that traffic can be asymmetric—for example, 50 Mbps from Seattle to Miami, but 200 Mbps from Miami to Seattle. This is perfectly valid, and MPLS TE is quite capable of dealing with asymmetric bandwidth demands.

Generating a traffic matrix is the first step in building a backbone that can carry your traffic efficiently.

But how do you build that traffic matrix? Many ways exist, but the main three are covered here:

  • NetFlow

  • TMS

  • TE tunnels

NetFlow

NetFlow is deployed pretty extensively in networks. You can use NetFlow for two things:

  • Determining what your network traffic patterns look like before you add MPLS Traffic Engineering to the mix

  • Determining what your network traffic patterns look like after you add MPLS Traffic Engineering to the mix

NetFlow gathers information about traffic flows—different protocol types, packet sizes, flow lengths, and so on. NetFlow monitors IP packets going through a router to gather this information.

NetFlow does not currently gather any information about packets that already have a label on them, but NetFlow can be used to measure IP packets that enter a TE tunnel headend and are sent down TE tunnels. Because TE tunnels are just interfaces, any traffic going out TE tunnel interfaces is captured for NetFlow analysis if NetFlow is enabled on the interface the traffic came in on.

You can then export this data to a collector, such as Cisco's NetFlow Flow Collector, and feed this data to an analyzer such as Cisco's NetFlow FlowAnalyzer. When all is said and done, you have a pretty accurate picture of what kind of traffic exists on your network and where it's going.

Rolling out NetFlow in its full glory is a nontrivial task. NetFlow also has lots of uses besides traffic planning, such as billing and Denial of Service (DoS) tracking. All of those uses are pretty nifty but are beyond the scope of this book.

The bottom line is that because you can use NetFlow for planning purposes, you can use it to help plan TE layout. And because NetFlow tracks traffic sent out a TE tunnel interface just like it tracks traffic going out any other type of interface, you can use NetFlow to get an idea of what sort of traffic is flowing across your network even after you deploy TE tunnels. For more information on NetFlow, see Appendix B,"CCO and Other References."

Traffic Matrix Statistics

Not surprisingly, TMS is pretty good at building a traffic matrix. TMS is a simple, low-overhead method of gathering statistics about packets entering and traversing your network. It is an extension to CEF accounting that lets you collect information about the number of packets and bytes destined for a particular nonrecursive prefix. Both TMS and NetFlow count traffic, but TMS sacrifices some of NetFlow's granularity in exchange for a lower-overhead method of gathering the same kind of information. TMS works with both IP packets and MPLS-labeled packets.

TMS allows you to gather information about the following:

  • The number of packets and bytes switched toward a given nonrecursive prefix

  • Information about "internal" and "external" traffic (discussed later)

  • BGP next-hop and neighbor AS information for a given destination network

You can gather this information for both IP packets and labeled packets.

TMS is based on Cisco Express Forwarding (CEF). CEF uses the concept of a nonrecursive prefix. A nonrecursive prefix is essentially a route that was not learned via BGP. IGP-learned routes, static routes, and directly connected routes are all examples of nonrecursive prefixes. A few nonrecursive prefixes are not in the routing table (CEF has the concept of a host adjacency, which is a /32 prefix installed for a host on a broadcast network, such as Ethernet), but generally speaking, nonrecursive prefixes are non-BGP routes.

You can see a router's nonrecursive prefixes using show ip cef non-recursive, as shown in Example 10-1.

Example 10-1 Output of show ip cef non-recursive

gsr1#show ip cef non-recursive

Prefix              Next Hop             Interface

2.3.4.0/24          attached             POS0/1

3.3.3.3/32          attached             Null0

4.4.4.4/32          attached             Null0

172.27.232.0/21     attached        Ethernet0

172.27.232.6/32     172.27.232.6  Ehernet0

172.27.235.66/32    172.27.235.66 Ethernet0

172.27.235.85/32    172.27.235.85        Ethernet0

192.168.1.12/32     2.3.4.12             POS0/1

Example 10-2 shows the routing table that matches up to these nonrecursive prefixes.

Example 10-2 Routing Table for Example 10-1

gsr1#show ip route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

       * - candidate default, U - per-user static route, o - ODR



Gateway of last resort is 172.27.232.6 to network 0.0.0.0



     2.0.0.0/24 is subnetted, 1 subnets

C       2.3.4.0 is directly connected, POS0/1

     3.0.0.0/32 is subnetted, 1 subnets

S       3.3.3.3 is directly connected, Null0

     4.0.0.0/32 is subnetted, 1 subnets

S       4.4.4.4 is directly connected, Null0

     172.27.0.0/21 is subnetted, 1 subnets

C       172.27.232.0 is directly connected, Ethernet0

     192.168.1.0/32 is subnetted, 2 subnets

i L2    192.168.1.12 [115/10] via 2.3.4.12, POS0/1

C       192.168.1.1 is directly connected, Loopback0

S*   0.0.0.0/0 [1/0] via 172.27.232.6

CEF's nonrecursive accounting lets you track traffic destined for these nonrecursive prefixes, even if your network traffic is destined for a BGP-learned route that is using that nonrecursive prefix as a next hop.

TMS adds some more accounting granularity to CEF's nonrecursive accounting, as well as adding the concept of internal and external interfaces. The internal/external distinction is an administrative one. You can flag interfaces coming from outside your network (or, in this case, from outside your MPLS TE cloud) as external and account for those packets separately from traffic flowing on internal interfaces.

Figure 10-2 flags the interfaces from inside the cloud as internal (I) interfaces and the customer-facing interfaces as external (E).

Figure 10-2. Network with Internal and External Interfaces

graphics/10fig02.gif

TMS, unlike NetFlow, does not export its data to a server. Rather than using NetFlow's "push" model, where data is pushed from a router to a NetFlow collector, TMS stores its data on the router and allows network administrators to collect data when they want to.

Configuring TMS

TMS is simple to configure. You do two things to enable TMS:

  • Enable ip cef accounting non-recursive globally

  • Optionally, configure an interface with ip cef accounting non-recursive {internal | external}

That's all you need to do. Interfaces are set to ip cef accounting non-recursive internal by default, so you only need to change the configuration for external interfaces. However, TMS data collection isn't enabled unless you configure ip cef accounting non-recursive globally.

Displaying TMS Data

You have two ways to display TMS data. One is via show ip cef, as demonstrated in Example 10-3.

Example 10-3 Output of show ip cef Showing TMS Data

Seattle#show ip cef 192.168.1.15

192.168.1.15/32, version 15, epoch 0, per-destination sharing

0 packets, 0 bytes

  tag information set

    local tag: 18

  via 192.168.22.15, Serial1/0, 0 dependencies

    next hop 192.168.22.15, Serial1/0

    valid adjacency

    tag rewrite with Se1/0, point2point, tags imposed: {0}

  5 packets, 500 bytes switched through the prefix

  tmstats: external 0 packets, 0 bytes

           internal 5 packets, 500 bytes

  30 second output rate 0 Kbits/sec

The four highlighted lines at the end of Example 10-3 show the TMS information recorded for a given nonrecursive prefix (in this case, 192.168.1.15). They're pretty straightforward. They show the total number of bytes and packets destined for a specific prefix, as received on both external and internal interfaces.

You can also access TMS information by retrieving data files directly from the router. TMS stores its data in a few different files:

  • tmstats_ascii— Collects information about destination nonrecursive (IGP-learned) prefixes and the traffic forwarded toward them. This file also collects information about MPLS TE tunnel traffic.

  • tmstats_binary— Collects the same information as tmstats_ascii, but in a more compact binary format. tmstats_binary is not covered in this chapter, because it contains the same information that's in tmstats_ascii.

  • tmasinfo— Collects information about a particular nonrecursive IGP prefix/mask, the recursive destination IP addresses behind that IGP prefix (as learned via BGP), and the neighbor AS traversed to get to that destination IP address.

The TMS data files are in system:/vfiles/, which is a little-known ephemeral file system within Cisco IOS Software DRAM. Later examples show how you can access these files.

By now, you might be getting an idea of how and where TMS is useful. By tracking the amount of traffic destined for a particular nonrecursive prefix (say, a BGP next hop that's the exit point from your MPLS TE cloud), you can build a matrix of input and output traffic and use it to size your TE tunnels.

The following example takes a detailed look at exactly what data TMS saves and how it can be used to build a traffic matrix.

Consider the network shown in Figure 10-3.

Figure 10-3. Simple TMS Network

graphics/10fig03.gif

All three GSRs are in the same AS—41.

AS42 is advertising a full BGP table to gsr7. 3.0.0.0/8 and 4.0.0.0/8 are two such routes learned from AS 42. gsr7 has gsr2 and gsr6 as IBGP neighbors. It is setting next-hop-self on the BGP routes it advertises to gsr2 and gsr6. gsr7's RID is 192.168.1.7.

gsr2 is sending traffic to 3.3.3.3 and 4.4.4.4, which are located in AS42.

gsr6 is doing TMS on the interface facing gsr2. Example 10-4 shows the configuration for this interface on gsr6.

Example 10-4 Configuration for the gsr6 Interface Between gsr6 and gsr2

!

ip cef accounting non-recursive

!

interface POS3/3

 ip address 192.168.11.6 255.255.255.0

end

!

As mentioned earlier, TMS stores the data it gathers in some files in system:/vfiles/. Example 10-5 shows these files.

Example 10-5 TMS Data in system:/vfiles/

gsr6#dir system:/vfiles/

Directory of system:/vfiles/



   15  -r--           0              <no date>  tmasinfo

   13  -r--           0              <no date>  tmstats_ascii

   14  -r--           0              <no date>  tmstats_binary

One thing to note is that the file size (the third column in Example 10-5) always shows 0. Data is not actually stored in a file; it is dynamically generated as you access the file itself.

As Example 10-6 demonstrates, gsr6's routing table has four connected routes (two network routes, two subnet routes), one static route, and three IS-IS routes (one network, two subnet). gsr6 also has 80,627 BGP routes.

Example 10-6 Routing Table for gsr6

gsr6#show ip route summary

Route Source    Networks    Subnets     Overhead    Memory (bytes)

connected       2           2           560         640

static          1           0           64          160

isis            1           2           464         480

  Level 1: 0 Level 2: 3

bgp 3402        70910       9717        5160128     12255304

  External: 0 Internal: 80627 Local: 0

internal        824                                 965728

Total           71738       9721        5161228     13223008

As you can see in Example 10-7, the contents of tmstats_ascii contain the prefixes (directly connected and learned via IGP) shown in Example 10-6.

Example 10-7 tmstats_ascii File Information About All IGP Routes

gsr6#more system:/vfiles/tmstats_ascii

VERSION 1 | ADDR 172.27.235.75 | AGGREGATION TrafficMatrix.ascii |  SYSUPTIME 541175 |

  RouterUTC 3212749402 | NTP unsynchronized | DURATION 0 |

p| 192.168.1.7/32 | 444745 | 698| 58464 | 0 | 0

p| 192.168.1.2/32 | 444745 | 0 | 0 | 0 | 0

p| 192.168.1.6/32 | 444745 | 0 | 0 | 0 | 0

p| 192.168.12.0/24 | 444745 | 0 | 0 | 0 | 0

p| 192.168.13.0/24 | 444745 | 0 | 0 | 0 | 0

p| 192.168.11.0/24 | 530312 | 713 | 47040 | 0 | 0

The highlighted line in Example 10-7 shows the TMS entry for gsr7's router ID. All traffic destined for the IP address 192.168.1.7, as well as traffic destined for all recursive prefixes (BGP prefixes) that resolve to 192.168.1.7, are counted in this line.

All the fields in the highlighted line of output are documented in the next section, but the important ones are as follows:

  • The IP address (192.168.1.7/32)

  • The number of packets switched toward that prefix (698)

  • The total number of bytes in those 698 packets (58464, an average of about 84 bytes per packet)

The byte and packet counters go up whenever a packet is switched toward 192.168.1.7, even if a packet is destined for a BGP route whose next hop is 192.168.1.7.

Example 10-8 shows that the tmasinfo file provides a mapping of all BGP routes to their next hop. The tmasinfo file contains information about all of its 80,627 BGP routes. The output shown here has been truncated for brevity. It makes for pretty boring reading after the first 1200 pages anyway.

Example 10-8 tmasinfo File Mapping Information About BGP/Next-Hop Routes

gsr6#more system:/vfiles/tmasinfo

VERSION 1 | ADDR 172.27.235.75 | AGGREGATION ASList.ascii | SYSUPTIME 541237 |

  routerUTC 3212749464 | DURATION 0

192.168.1.7/32 | 42 | 3.0.0.0/8

192.168.1.7/32 | 42 | 4.0.0.0/8

192.168.1.7/32 | 42 | 4.0.37.184/30

192.168.1.7/32 | 42 | 4.2.47.0/25

192.168.1.7/32 | 42 | 4.2.55.0/24

192.168.1.7/32 | 42 | 6.0.0.0/8

...

...

Note that tmasinfo has no traffic counters; its job is to present the BGP portion of the routing table should you care to examine it. The highlighted lines in Example 10-8 show the tmasinfo entries for 3.0.0.0/8 and 4.0.0.0/8, which are the prefixes that gsr2 is sending traffic toward. The three columns in each of the highlighted lines are the BGP next hop, the next hop AS for that route, and the route itself. These columns are more fully explained in the next section.

Understanding TMS Data Files

tmstats_ascii and tmasinfo share a common header format, even though the data these files carry is different. Table 10-2 breaks down the common header format.

Table 10-2. TMS Common Header Format
Field Description
VERSION 1 The TMS data version—currently 1.
ADDR address The router ID of the router generating the TMS data file.
AGGREGATION name The type of aggregation being performed. tmstats_ascii always says TrafficMatrix.ascii, and tmasinfo always says ASList.ascii.
SYSUPTIME time System uptime in milliseconds.
NTP [synchronized | unsynchronized] Seen only in tmstats_ascii. Tells you whether the router is synchronized with an NTP server.
routerUTC time The time of data collection in seconds since 1900/01/01. Because data is gathered by examining this file, the routerUTC timestamp is what time the router thought it was when it started displaying the file in question.
DURATION time The time needed to capture the data, in seconds. This is almost always 0. This time is provided in case it takes a significant amount of time to build the various data files so that you can extrapolate how much traffic might not have been recorded by TMS.

Then there's the data itself. tmasinfo and tmstats_ascii have different formats for the data they contain. Table 10-3 describes each line of tmstats_ascii from the highlighted line of Example 10-7.

Table 10-3. tmstats_ascii Data Information
Line Description
p Denotes that information in this line applies to an IP prefix rather than a TE tunnel. "p" shows up in the first column if a packet has been received as IP or with a label that was distributed with TDP or LDP. Packets that are received as part of an MPLS TE tunnel are marked with a "t" instead (see Table 10-5).
192.168.1.7/32 IGP prefix being accounted for.
444745 Amount of time (in milliseconds) since the record for this prefix was first created. If the prefix leaves the routing table, all accounting data is lost; when the prefix comes back, the counter starts again.
698 Number of packets received on all internal TMS interfaces for this prefix.
58464 Number of bytes received on all internal TMS interfaces for this prefix.
0 Number of packets received on all external TMS interfaces for this prefix.
0 Number of bytes received on all external TMS interfaces for this prefix.

Table 10-4 describes each line of tmasinfo from Example 10-8.

Table 10-4. tmasinfo Data Information
Line Description
192.168.1.7/32 BGP next hop.
42 For the prefix specified in the next column, the next-hop AS in this route's AS path.
3.0.0.0/8 Prefix learned from the next-hop AS.

Given this information, it is easy to determine how much traffic entered the router on all external interfaces, how much traffic entered the router on all internal interfaces, and where traffic exited.

tmstats_ascii has enough information to do per-exit-point accounting. This gives you your traffic matrix—you know the amount of traffic destined for a particular nonrecursive prefix and the amount of time in which that traffic has arrived, and you can easily compute the traffic rate. In the preceding example, with 698 packets heading toward 192.168.1.7 in the last 444745 milliseconds, this means that the average traffic rate is about 1.5 pps. This isn't the busiest network in the world, but it's good enough to serve as an example here.

Because TMS is pull-based, you can let statistics accumulate on the router for as long as you like. The packet and byte counters are 64-bit counters. So even on an OC-48, which runs at approximately 2.5 Gbps, you only need to pull TMS data once every (264 / (2.5 * 109)) seconds, or 234 years, before the counters wrap.

Now for the bad news. TMS stats are stored in a CEF loadinfo structure. CEF is a switching path based on the routing table. This means that when the routing table changes, the CEF table changes. So any data accumulated by CEF counters disappears if the corresponding route disappears. And when the route comes back, the TMS data structure counters are all zeroed out. This means that TMS is good for long-term data collection about general trends, but it's not the device you want to use if you need to make sure that you're measuring every single packet that has ever crossed the interface.

TMS can also track packets sent through TE tunnels. At TE midpoints, tmstats_ascii displays a line like the following:


t | 192.168.1.22 21 | 547059 | 44 | 3696 | 0 | 0

This information is similar to the information recorded if tmstats_ascii is dealing with packets not destined down a TE tunnel.

Table 10-5 describes the information from the tmstats_ascii output.

Table 10-5. TE Tunnel Tracking Information from tmstats_ascii
Field Description
t TE tunnel
192.168.1.22 RID of the tunnel tail
21 Tunnel ID
547059 Router uptime at which TE LSP was created
44 Number of packets received on all internal TMS interfaces for this prefix
3696 Number of bytes received on all internal TMS interfaces for this prefix
0 Number of packets received on all external TMS interfaces for this prefix
0 Number of bytes received on all external TMS interfaces for this prefix

Getting TMS Information off the Router

There are three ways to get tmstats_ascii, tmstats_binary, and tmasinfo off the router:

  • Use more file system:/vfiles/{filename} (this is not recommended for retrieving the TMS binary file).

  • Use copy system:/vfiles/tmstats_ascii {rcp | ftp | tftp} from the CLI to copy the TMS file to a server for analysis.

  • Use the FTP client MIB to achieve the same thing as the CLI.

As soon as you've got the TMS info off the router, build a traffic matrix like the one shown in Table 10-1. As soon as you've got that matrix, go forth and build TE tunnels! Search CCO for "traffic matrix statistics," or see Appendix B for more information on TMS.

Using TE Tunnels for Traffic Measurement

So far, you've learned how to use a few Cisco IOS Software accounting tools to measure traffic demands. There is another way that doesn't require rolling out a specific traffic-measurement tool, or any management devices other than what you're already using to monitor router interfaces.

If you want to find out how much traffic is going from one router to another, just build an MPLS TE tunnel between the two routers. If you do that, and you enable autoroute on the tunnel interface, all traffic destined for the tunnel tail or anything behind it traverses the tunnel. Because the tunnel is just an interface, you simply have to check the traffic rate on the interface to see the amount of traffic going from the tunnel headend to the tunnel tail.

The recipe is pretty simple:

Step 1. Deploy 0-bandwidth tunnels with a dynamic path option from ingress node to egress (where ingress and egress are the ingress and egress of your TE cloud).

Step 2. Monitor these interfaces as you would any other interface (using IF-MIB or show interfaces).

Step 3. Optionally, resize the tunnels to reflect the amount of bandwidth they're carrying. If you do this, you'll find you've deployed TE tunnels using the strategic online methodology.

It's that easy!

In Figure 10-4, tracking interface utilization on Tunnel0 gives you the amount of traffic destined for Router D and Networks 1, 2, and 3 (because they are behind Router D).

Figure 10-4. Sample Network for Tracking Tunnel Utilization

graphics/10fig04.gif

Some existing networks use MPLS TE just to do measurement, without steering traffic down paths other than what the IGP would take. The advantages of doing your measurements using MPLS TE are that you have

  • No external data collection box, other than what you're already using to monitor interfaces

  • No performance impact, because the per-destination accounting is a by-product of the TE implementation

One thing to keep in mind is that your load-balancing capabilities change. With IP, you inherently load-share traffic across equal-cost paths; with TE, you have to explicitly configure more than one LSP if you want to share traffic across different paths. If you configure a single TE tunnel between two routers and enable autoroute on that tunnel, all traffic between those two routers takes that single tunnel, which of course follows a single physical path. If the traffic between those two routers would normally take multiple equal-cost paths, to achieve the same effect with MPLS TE, you need to configure multiple TE tunnels, across all paths from headend to tail, which that IP traffic would take.

Service Assurance Agent

Bandwidth is only half the story. You've also got to care about delay. Referring back to Figure 10-1, if you want to get from Seattle to San Jose, and you have enough bandwidth only by going via Miami, you probably don't want to take that path. You might prefer to take the path with less available bandwidth, because the cost of obtaining the necessary bandwidth is too much delay.

Cisco IOS Software has a powerful facility for measuring delay called Service Assurance Agent (SAA). You can use SAA to get the measurements for one-way or round-trip delay on a circuit, as well as other performance measurements (HTTP response, DNS response, jitter, and so on), all on a per-DSCP basis.

SAA does a lot more than what is covered in this chapter, but you have enough here to get you started:

  • Basic SAA theory and operation

  • How to measure two-way delay using ICMP

  • How to measure one-way delay using specialized SAA probes

What Is SAA?

SAA grew out of a project known as the Response Time Reporter (RTR). Although officially now named SAA, the RTR configuration syntax is still used.

The basic idea behind SAA is that it can measure the latency of several types of network traffic between any two points. SAA can do this by sending ICMP echo requests, HTTP GET requests, FTP file requests, and several other types of requests.

SAA is made up of two things—probes and responders. They are fairly self-explanatory. A probe is something that sends out packets that measure properties of a network, and a responder is something that responds to a probe. The device that sends the probe is sometimes called the client, and the responder is sometimes called the server.

There are several different types of probes. SAA can send probes that measure things as basic as ICMP and as complex as one-way jitter, FTP, HTTP, DLSW, DNS, and DHCP. Complete SAA operations aren't covered in this book, but this section covers ICMP and jitter probes, which is enough to get you started with SAA.

Not all devices can respond to all SAA probes. For example, an ICMP probe is just a ping (an ICMP echo request). An HTTP probe is much the same. It fetches a specific Web page and reports back on how long it took to retrieve the page. You can't send an HTTP probe to a device that's not running a Web server (obviously).

Some probes are meant to be responded to by a specific SAA responder. Specifically, there is a one-way jitter probe that relies on two things:

  • The device that is the other end of the probe can recognize and respond to specialized SAA jitter probes.

  • The device sending the probe and the device responding to the probe have their clocks synchronized via NTP.

One-way and two-way probes are covered more in the next two sections.

Two basic uses for SAA are

  • Measuring delay and jitter between two points in the network (from one POP to another across the WAN, or from a dialup server to a Web farm, for example)

  • Measuring delay and jitter between two directly connected routers

The second use is a degenerate case of the first, but in the context of MPLS TE, these two cases have two different applications, so they're listed separately.

It's useful to be able to measure end-to-end delay to get an idea of whether you're violating your Service-Level Agreements (SLAs) to your users. Most networks offer some guarantee on the maximum latency encountered in a network, and SAA is one way to measure that latency.

Those who are serious about SAA measurements often deploy a dedicated SAA router in each POP. A small but powerful router such as the Cisco 3620 can deal with dozens or hundreds of SAA probes, so it's easy to set up a mesh of probes between each POP.

Another way to use SAA is to determine the one-way delay on a link between two routers, as opposed to determining the one-way delay between two arbitrary points in the network. Simply run one-way delay probes between two directly connected neighbors on a physical link rather than across the network.

This gives you a number you can use with the physical-interface command mpls traffic-eng administrative-weight. If you want to build delay-sensitive tunnels (see Chapter 3, "Information Distribution"), take the one-way delay, multiply by 10, and set that to the interface administrative weight.

So if you know that a circuit from Dallas to Miami has an unloaded one-way delay of 22 ms, set the administrative weight on the interface to 220.

Multiplying by 10 gives you some room to play with. If you have two links with a delay of 22 ms, but you'd rather prefer one to the other, give one a cost of 220 and the other a cost of 221, or anything up to 229.

SAA can be monitored, maintained, and controlled using Cisco's Internetwork Performance Monitor (IPM) software or via MIBs, but it can also be dealt with purely from the CLI. SAA is geared toward being managed by a MIB. (The CLI output is pretty much an exact duplicate of what's in the MIB, even when it would be nice if the CLI prettied up the output a bit.) But it's certainly possible to use SAA purely from the CLI, especially for something as simple as determining the delay between two points.

If you're going to use the CLI, it's a good idea to read the Cisco Response Time Monitoring (RTTMON) (CISCO-RTTMON-MIB), because it explains a lot about what information the MIB can get for you.

Measuring Two-Way Delay

Measuring two-way delay is just about as simple as it gets. You configure SAA to send an ICMP echo request to a particular destination, and SAA reports back the round-trip time. If you want to find the one-way delay, simply assume that the network is symmetric and divide the reported delay number by 2.

The basic SAA configuration is simple. The following two global configuration lines define an SAA probe:


rtr probe-number

 type echo protocol ipIcmpEcho target-addr [source-ipaddr ipaddr]

The first line, rtr 1, defines the probe. The probe number is simply a number you assign. You can build dozens or hundreds of probes on a router; you number each one separately. You can give RTR probes a number up to 2,147,483,647. If you're going to build an RTR probe for each TE tunnel, just make the RTR probe number the same as the tunnel interface number to keep things straight.

Configuring rtr probe-number puts you into a submode called config-rtr. In that submode, you create an echo probe of type ipIcmpEcho, giving it a target IP address and an optional source IP address. This defines a probe that is simply a ping.

After defining the probe, you need to tell it when to run. You have lots of options here (see Appendix B for more information), but the most basic configuration is


rtr schedule probe-number start-time now

This is done in global configuration mode. It tells the router to periodically initiate the probe, starting now. By default, each probe is sent out every 60 seconds. You can change the probe frequency with the frequency seconds command:


rtr 3

 type echo protocol ipIcmpEcho 192.168.1.14

 frequency 10

There's no need to configure an SAA Responder for an ICMP echo probe, because the probe sent out is just a regular ICMP ping.

You can examine the results of an SAA probe with the command show rtr distributions-statistics probe-number, as shown in Example 10-9.

Example 10-9 show rtr distributions-statistics 1 Command Output

vxr15#show rtr distributions-statistics 1

        Captured Statistics

Entry    = Entry Number

StartT   = Start Time of Entry (hundredths of seconds)

Pth      = Path Index

Hop      = Hop in Path Index

Dst      = Time Distribution Index

Comps    = Operations Completed

OvrTh    = Operations Completed Over Thresholds

SumCmp   = Sum of Completion Times (milliseconds)

SumCmp2L = Sum of Completion Times Squared Low 32 Bits (milliseconds)

SumCmp2H = Sum of Completion Times Squared High 32 Bits (milliseconds)

TMax     = Completion Time Maximum (milliseconds)

TMin     = Completion Time Minimum (milliseconds)



Entry StartT  Pth Hop Dst Comps   OvrTh   SumCmp     SumCmp2L   SumCmp2H   TMax      TMin

1     1568    1   1   1   105     0       64541      59035203   0          1662      127

Another way to display this information is with the command show rtr distributions-statistics probe-number full, as shown in Example 10-10.

Example 10-10 show rtr distributions-statistics 1 full Command Output

vxr15#show rtr distributions-statistics 1 full

        Captured Statistics

Entry Number: 1

Start Time Index: 20:20:03.000 EST Wed Apr 3 2002

Path Index: 1

Hop in Path Index: 1

Time Distribution Index: 1

Operations Completed: 105

Operations Completed Over Thresholds: 0

Sum of Completion Times (milliseconds): 64541

Sum of Completion Times Squared Low 32 Bits (milliseconds): 59035203

Sum of Completion Times Squared High 32 Bits (milliseconds): 0

Completion Time Maximum (milliseconds): 1662

Completion Time Minimum (milliseconds): 127

Examples 10-9 and 10-10 contain the same information; it's just formatted differently. In either case, the interesting information is highlighted. You see

  • The number of operations completed (105)

  • The sum of completion times (64,541 ms)

  • The sum of squared completion times (59,035,203 ms)

  • The maximum and minimum completion times (1662 ms and 127 ms, respectively)

The sum of completion times squared is a 64-bit value, stored as two 32-bit values.

Because this is output from an ICMP echo probe, the completion time here is just how long it took the probed device to respond. Not only can you calculate the mean response time (1.66 ms), but you also have the sum of squares, so you can do more advanced statistics on the response time.

Measuring One-Way Delay

Measuring two-way delay isn't always the best way to figure out your network delay. What you really want is one-way delay. You can always configure a two-way delay and divide by 2, but this assumes that the delay is the same in both directions. Networks can have different delays in different directions, because of buffering issues, L2 circuit routing, and so on.

Fortunately, SAA lets you configure a probe that measures one-way delay. The probe configuration here is much like the ICMP probe you've already seen:


rtr probe-numer

 type jitter dest-ipaddr dest-addr dest-port dest-port

The main difference here is that the probe is of type jitter.

You also need to tell the router to actually send the probe. This doesn't change from the ICMP probe:


rtr schedule probe-number start-time now

There's one more thing you need to do, and that's to configure the SAA Responder. An SAA one-way probe works by putting a timestamp in the packet it sends, sending a probe, and getting a response to its probe that has the timestamp at which the probe was received.

To configure an SAA Responder, you simply use the global command rtr responder on the device that is the probe's dest-addr. As soon as the responder is enabled, it responds to all probes sent its way.

Because the one-way jitter probe relies on a timestamp, both the probing device and the responding device must have their clocks synchronized; this is generally done using Network Time Protocol (NTP).

As soon as a jitter probe is running, you can see the results of that probe with the command show rtr collection-statistics probe-number, as shown in Example 10-11.

Example 10-11 show rtr collection-statistics 2 Command Output

vxr15#show rtr collection-statistics 2

        Collected Statistics



Entry Number: 2

Target Address: 192.168.1.14, Port Number: 2000

Start Time: 20:22:03.000 EST Wed Apr 3 2002

Latest Oper Sense: Unknown

RTT Values:

NumOfRTT: 244   RTTSum: 70434   RTTSum2: 23198478

Packet Loss Values:

PacketLossSD: 0 PacketLossDS: 0

PacketOutOfSequence: 0  PacketMIA: 36   PacketLateArrival: 0

InternalError: 0        Busies: 11

Jitter Values:

NumOfJitterSamples: 216

MinOfPositivesSD: 35    MaxOfPositivesSD: 40

NumOfPositivesSD: 218   SumOfPositivesSD: 7942  Sum2PositivesSD: 289520

MinOfNegativesSD: 0     MaxOfNegativesSD: 0

NumOfNegativesSD: 0     SumOfNegativesSD: 0     Sum2NegativesSD: 0

MinOfPositivesDS: 1     MaxOfPositivesDS: 47

NumOfPositivesDS: 29    SumOfPositivesDS: 266   Sum2PositivesDS: 9806

MinOfNegativesDS: 1     MaxOfNegativesDS: 1

NumOfNegativesDS: 29    SumOfNegativesDS: 29    Sum2NegativesDS: 29

Interarrival jitterout: 0       Interarrival jitterin: 0

One Way Values:

NumOfOW: 244

OWMinSD: 60     OWMaxSD: 443    OWSumSD: 54182  OWSum2SD: 14670634

OWMinDS: 59     OWMaxDS: 108    OWSumDS: 16252  OWSum2DS: 1135116

DS means "from destination to source," and SD means "from source to destination." When you configure a one-way probe, SAA automatically measures in both directions—from the client to the responder and from the responder to the client.

Of particular interest is the set of data highlighted in the output of Example 10-11. Both Round-Trip Time (RTT, in the RTT Values: section) and one-way delay values (in the One Way Values: section) are measured; jitter is also measured. Although jitter is important for understanding the impact on VoIP and other real-time services in your network, for MPLS TE the most useful data is the one-way delay. Table 10-6 shows the breakdown of the data in the One Way Values: section.

Table 10-6. One Way Values: Section Definition
Data Value
NumOfOW Number of successful one-way probes since the current probe was configured
OWMinSD Minimum one-way delay from the source (the client) to the destination (the server, or SAA Responder)
OWMaxSD Maximum one-way delay from the source to the destination
OWSumSD Sum of one-way delay times from the source to the destination
OWSum2SD Sum of squares of the one-way delay times from the source to the destination

The data that ends in DS (OWMinDS, OWMaxDS, and so on) is the same information, but from the server to the client.

To compute the mean one-way delay from client to server, simply divide OWSumSD by NumOfOW. In Table 10-6, the mean one-way delay from client to server is 54182 ms / 244 probes, or 222.06 ms.

Configuring DSCP in an SAA Probe

You can control the DSCP byte in any SAA probes. This is useful if you've made different latency guarantees for different classes of service, because you should then be measuring latency in each class. You configure the DSCP using the tos keyword, which controls the entire DSCP byte.

Table 10-7 shows each ToS setting you'd use to configure a particular DSCP setting. See the "DiffServ and IP Packets" section in Chapter 6, "Quality of Service with MPLS and MPLS TE," for more information on DSCP. The ToS byte can be configured in either hex or decimal; Table 10-7 shows the hexadecimal configuration.

Table 10-7. Mapping the ToS Byte to DSCP Values
DSCP Class Name ToS Value
Default 0x00
CS1 0x20
CS2 0x40
CS3 0x60
CS4 0x80
CS5 0xA0
CS6 0xC0
CS7 0xE0
AF21 0x48
AF22 0x50
AF23 0x58
AF31 0x68
AF32 0x70
AF33 0x78
AF41 0x88
AF42 0x90
AF43 0x98
EF 0xE8

For example, to create an ICMP echo probe in the EF class, you'd use the configuration shown in Example 10-12.

Example 10-12 Sample Configuration for an ICMP Echo Probe with EF DSCP

rtr 4

 type echo protocol ipIcmpEcho 192.168.1.14

 tos 0xE8

This section has only scratched the surface of what SAA can do. See Appendix B for more information on SAA and its capabilities.

    Previous Section  < Free Open Study >  Next Section