Internet Peering (Spring, 1999)
-------------------------------
By The Prophet

As anyone who has a dialup Internet account knows, there are plenty of
providers. Everyone wants to sell you a dialup account. Providers use many
different backbones - sometimes multiple ones. And yet, if you dial into any of
them and go to http://www.2600.com, you're likely to see the 2600 web page
load.

How that page loads is really a remarkable event. Many people don't realize
that the Internet is not all one network. It is a network of networks, operated
by a myriad of providers. Each of these operate a backbone, which consists of
high-speed links (usually T-3 and above) between "Points of Presence" (POPs)
located in major cities. By far the largest backbone is the legacy MCI.NET,
which is now operated by Cable and Wireless and was renamed CW.NET. Cable and
Wireless also owns cwix.net, which they are slowly integrating into CW.NET. As
of this writing, MCI Worldcom is the second largest backbone operator (though
catching up quickly), operating uu.net (formerly alter.net), wcom.net (formerly
compuserve.net), and ans.net (previously owned by AOL, and before that ANS
CO+RE Systems). And in a distant third place is Sprint. There are a number of
smaller backbone providers as well AGIS, Digex, GlobalCenter, Exodus, CRL,
netaxs, and others. Many of these, paradoxically, lease fiber trunk capacity
from MCI Worldcom (this has obviously led to friction, as the bandwidth
provider of many backbones is also a major competitor).

Of course, not every network extends to every point on the Internet. For
instance, ANS handles a great deal of traffic into and out of Albuquerque,
since they are one of only a few backbones with POPs there. Some great places
to see network maps and POPs for the various ISPs are their web pages, or the
Boardwatch Directory of Internet Service Providers. In order to solve the
problem of moving packets from one point to another, backbones peer with one
another.

Peering is, at its essence, the passing of traffic between networks. Let's
start with a traceroute, which shows the routers between an origin and a
destination:

 traceroute to www.fbi.gov (32.97.253.60), 30 hops max, 40 byte packets 1
 hil-qbu-ptt-vty254.as.wcom.net (206.175.110.254) 245 ms 218 ms 253 ms 2
 hil-ppp2-fas2-1.wan.wcom.net (209.154.35.35) 216 ms 209 ms 210 ms 3
 hil-core1-fas4-1-0.wan.wcom.net (205.156.214.161) 210 ms 227 ms 226 ms 4
 chi-core1-atm5-0-1.wan.wcom.net (209.154.150.5) 434 ms 223 ms 215 ms 5
 chi-peer1-fdd0-0.wan.wcom.net (205.156.223.164) 222 ms 1882 ms 1815 ms 6
 ameritech-nap.ibm.net (198.32.130.48) 369 ms 222 ms 222 ms 7 165.87.34.199
 (165.87.34.199) 231 ms 303 ms 228 ms 8 www.fbi.gov (32.97.253.60) 233 ms 241 ms
 242 ms

This may look like a bunch of gobbledygook at first glance. However, it is very
revealing about how peering works.

You can see that the first stop is a terminal server in wcom.net (formerly
compuserve. net), probably located in Columbus, Ohio. The connection bounces
from there to an ethernet port, to an ATM router, and over a high-speed link to
another ATM router in Chicago. Once in Chicago, it proceeds to the peering
point (at Ameritech NAP), is handed off to IBM.NET, hits a router that isn't
identified (probably somewhere in the Washington, DC area), and finally ends up
at www.fbi.gov. Bear in mind that when www.fbi.gov sends data back, it does not
necessarily follow the same path. The path that is followed is based on route
advertisements and other factors, which a good set of TCP/IP texts, like the
TCP/IP Illustrated series, reviews in detail.

You will notice that Ameritech NAP is the peering point that was used. There
are actually four "official" NAPs, set up under the review of the NSF. They are
the Ameritech NAP in Chicago, the New York NAP (which is actually in
Pennsauken, New Jersey across the river from Philadelphia), the Sprint NAP
(which is in West Orange, New Jersey, near Newark), and the PacBell NAP in the
San Francisco area.

This system of NAPs is supplemented by two "unofficial" NAPs known as the MAE
s. These are Metropolitan Area Ethernets (hence the acronym) that are operated
in the Washington, DC and Silicon Valley areas by MFS (now owned by MCI
Worldcom). Additionally, the Federal Government operates two Federal Internet
eXchanges (FIX s), one at Moffett Field in California and one in the
Washington, DC area. The FIX's handle Internet traffic bound to and originating
from .MIL sites and some .GOV sites. Finally, CIX operates a peering point in a
Palo Alto, CA WilTel POP. This is mostly a salutatory point and is rarely used
nowadays. At one point, all commercial Internet traffic was transited through
CIX, but the NAPs were set up in part because of infighting between the
competing backbones who could not agree on who was allowed to peer at CIX.
Finally, many larger backbones have set up private peering points among
themselves. For instance, since Cable and Wireless' acquisition of MCI.NET,
they have set up a number of private peering points to exchange traffic with
their own CWIX.NET.

Peering is a very controversial area. For one, end-to-end performance of a
backbone is positively coordinated with the number and speed of peering points.
Therefore, a smaller Internet backbone that cannot afford a number of private
peers, or to peer at every MAE and NAP, is likely to have poorer performance.
Additionally, backbones often cannot agree with whom they will peer. For
instance, bbnplanet.net (now owned by GTE) decided that exodus.net was no
longer worthy of peering, even though exodus.net offered to peer with BBN at
any place in the country it liked. BBN claimed that exodus. net was leeching
their bandwidth - though one must wonder who's really better off in the value
equation, since BBN hosts many dialup and corporate users, and Exodus hosts
primarily very popular web sites (like Yahoo! and ESPN Sportszone). How useful
are the dialup accounts to customers without good performance to popular web
sites? This is a question other backbones considering similar actions would be
wise to consider.

The controversy is somewhat justified. Peering requires sharing BGP route
advertisements, which if used improperly can blackhole large parts of the
network (imagine large amounts of CW.NET traffic being routed via a 56K link to
Iran - this is conceivably possible with bad BGP). Clearly, larger networks 
don't want clueless admins from smaller networks creating such episodes.
Additionally, larger networks wonder why they should pay to transit traffic
cross-country to a MAE for a smaller network that may only haul the traffic
across town from the peering point. This is the case with many very small peers
at MAE WEST in the San Francisco area. Many backbones at first demanded "hot
potato routing," so as to shift traffic away from their networks onto the
network to which packets were bound as soon as possible. However, the opposite
demand is often the case with smaller backbones (such as Exodus): They re told
to do "cold potato" routing, meaning that Exodus is expected to deliver traffic
bound for UUNet at the nearest UUNet peering point to the IP for which the
traffic is bound. Meanwhile, UUNet does "hot potato" routing, shifting Exodus
traffic to their network as quickly as possible!

Meanwhile, while all of this is going on, people are buying and expecting
access to the Internet. This is an important point. My mother is, for her
$19.95 per month, not buying access to CW.NET's network. She wants to use the
Internet to visit knitting, cooking, and travel web sites. She knows how to
send me e-mail, but wouldn't know what a NAP was if one bit her on the leg.
Customers are justifiably angry if they are unable to reach certain points on
the Internet, or if the performance is awful. This puts backbones between a
rock and a hard place. Those providers who are clued seem the most likely to
actively seek multiple peering points with multiple providers, and PSI is a
market leader in this regard - they'll peer with anyone operating a backbone,
free of charge. Others, such as UUNet, are demanding that smaller providers
purchase circuits from them at regular customer rates until they meet certain
criteria (which seems to change frequently). And finally, the MAEs and NAPs are
collapsing under their own weight. They handle so much traffic that the
majority of "net lag" is introduced at these peering points. Many larger
networks are eschewing these peering points altogether in favor of private
peering points. The problem with this, of course, is that it makes certain
parts of the Internet faster than other parts, which drives traffic away from
the smaller backbones, which makes the bigger networks even larger, so they can
create more private peers...you get the idea. One backbone threw up their hands
and gave up on the idea of public peering. SAVVIS buys transit from most other
backbones, routes traffic exclusively through their own data centers, and by
keeping more than 80 percent of their traffic away from the NAPs, has
consistently performed very well in Keynote Systems network performance tests.

I don't know where all of this will end. Nobody does. But I'll pull out my
crystal ball anyway. Historically, backbones have been great at creating murky
peering arrangements, using convoluted reasoning. This is likely to continue.
Chances are that we'll see the existing small backbones either solidify their
positions, become acquired by larger players, or run out of venture capital and
disappear. However, it's pretty unlikely that the Internet will cease to exist.
It's dependent on peering, the backbone operators know this, and while there
may be power struggles and political games as exist in any large organization,
there are also too many competitors for anyone to try to "steal" the Internet
(by cutting off peering). Jack Rickard, editor of Boardwatch Magazine, put it
best: "Trying to control the Internet is like trying to choke a Jell-O snake in
a swimming pool full of Wesson oil." Wise words, which astute backbones will
heed.