Ethernet The Occasional Outsider

coondoggie writes to mention an article at NetworkWorld about the outsider status of Ethernet in some high-speed data centers. From the article: "The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range. But in data centers, where CPUs may be sharing data in memory across different connected machines, the smallest hiccups can fail a process or botch data results. 'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,' Garrison says. This forced many data center network designers to look beyond Ethernet for connectivity options."

Long Live!

Long Live the Token Ring!

One Ring to rule them all

Didn't RTFA? -Infiniband, FC and Myrinet beat Eth0

Actually, even with Gigabit ethernet availability HPTC and other network intensive data center operations have moved to Fibre Channel and things like:

Infiniband http://en.wikipedia.org/wiki/Infiniband [wikipedia.org]

and Myrinet http://en.wikipedia.org/wiki/Myrinet [wikipedia.org]

http://h20311.www2.hp.com/HPC/cache/276360-0-0-0-1 21.html [hp.com]
HP HPTC site

-What's the speed of dark?

My idea: a vat of salt water & CAT5

In our Data Center, we have a great big vat of steaming salt water and we drop one end of the cat5 cables from each server into the vat....those packets that can't figure out where they're going just drop to the bottom and die ...we have to drain this packet-goo out once a month. (but we do recycle it...we press it into CDs and sell them on Ebay)

(Seriously, haven't people heard cut-through switches which just look at the first part of the header and switch based on that... store-and-forward switches are so "1990s")

TDz.

Store & Forward ONLY for 10 to 100 to 1,000.

There are only TWO reasons to use Store & Forward.

#1. You're running different speeds on the same switch (why?).

#2. You really want to cut down on broadcast storms (just fix the real problem, okay?)

Other than that, go for the speed! Full duplex!

For performance, run the same speed.

People run different speeds on the same switch all the time, and for not necessarily poor reasons: If you have a SMB (in this case, that's small or medium business) with maybe one big fileserver, you don't need to run gigabit to everyone...

What's with the "need to"?

I'm talking performance. Store & Forward hammers your performance. In my experience, you get better performance when you run the server at 100Mb full duplex (along with all the workstations) and use Cut Through than if you have the server on a Gb port, but run Store & Forward to your 100Mb workstations.

30 GB? Take that NSA and your outdated 622MB!

The NSA's network sniffer, recently discovered at an AT&T broadband center, can only sniff up to 622MB [slashdot.org]. Sounds to me like if you use an InfiniBand switch, that would effectively make the output of the NSA's network sniffers complete gibberish.

100ms ethernet latency?

I don't think I need to read anymore, well, I did verify that the number really appears in the article.
This author does not understand the subject material.

(I suppose you could deliberatly overload a switch enough to get this number, maybe, but that would be silly, and your switch would need 1.25Mbytes of packet cache.)

Re:100ms ethernet latency?

Looks like the author fucked up the definition of millisecond too:

"By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second"

http://www.google.com/search?hl=en&q=define%3Amill isecond&btnG=Google+Search [google.com]
"One thousandth of a second"

Seriously. How the fuck does this idiot get published?

Low-cost options?

Ultra-low latency networking is a minor interest of mine, but one I've never had the chance to really pursue. Can anyone familiar with the landscape recommend some low-cost options for experimenting with this stuff? Or maybe just let me down gently. "No, Sammy, there are no low-cost options. And there's no Santa Claus."

Not an Auspicious Start

From the article, three paragraphs in:
"(By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second)"

That would be one-thousandth, not millionth (aka micro second). Not a good start...

When you get to many hops

I just blame it on the ether-bunny.

Software design

The origional post makes some comments that
sharing memory ... the smallest hiccups can fail a process or botch data results.
Sounds like bad design, or a known design trade off.
Quite reasonable, when on a slow link, until I know better assume the data I have is correct, if it isn't throw it out and start over. Not wildly different than branch prediction or other approaches to this type of information.

'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,'
Faster is faster, not really a shocking concept.

Did you mean "microseconds"?

The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range.

I don't know what sort of switches you use, but my home LAN, with two hops (including one over a wireless bridge) through only slightly-above-lowest-end DLink hardware, I consistantly get under 1ms.



When you get into application-layer clustering, milliseconds of latency can have an impact on performance

Again, I get less than 1ms, singular.



Now, I can appreciate that any latency slows down clustering, but the ranges given just don't make sense. Change that to "microseconds", and it would make more sense. But Ethernet can handle single-digit-ms latencies without breaking a sweat.

Milliseconds?

On my planet, a millisecond is a full thousandth of a second, not just one millionth.

Oh, well. People tell me I'm just slow.

sharing memory over ethernet?

That just sounds daft. Given the bottle neck harddrives are for cpu's, it doesn't sound like a great shock that when you gotta wait for your data over ethernet you're going to see problems.

Maybe I should RTFA...

Lesson: Use appropriate Tech.

Ethernet's strength is it's flexiblity, not it's speed per se. It can handle changing network environments where hardware or software is added and removed continually, and you never know quite where the bandwith is most needed. You just plug it all in, and ethernet does a decent job of neotiating who gets to use the bandwidth.

But it's never been a really high speed protocol. It's easy to beat, speed-wise, as long as you know what the network use looks like ahead of time.

Which of course is a killer for most general use, but for specialty use that's not so much of a problem.

Store & Forward Unnecessary?

The article's a bit lacking on details, but... Isn't store and forward unnecessary? It's definitely possible to get it down to a much lower latency than is stated in the article if you don't use it.

Channel Bonding

I have a cluster of 45 dual Xeon processing nodes. Latencies average about 210 usec the same as could be expected in any 100Mbs connection, but using channel bonding my bandwidth is double that of a single ethernet connection. I don't have the need for faster, all our processes are wholly independent and don't need to do message passing.

Re:Channel Bonding

So you have an environment with requirements totally unlike the ones described in the article and needing none of the solutions illustrated in the article. Hey...thanks for letting us know. Maybe the other million Slashdot users with environments irrelevant to the post can let us know what they have as well.

No kidding

Er, yeah. No kidding.

When I was writing applications at the San Diego Supercomputer Center, latency between nodes was the single greatest obstacle to getting your CPUs to running at their full capacity. A CPU waiting to get its data is a useless CPU.

Generally speaking, clusters who want high performance used something like Myrnet instead of ethernet. It's like the difference between consumer, prosumer, and professional products you see in, oh, every industry across the board.

As a side note, how many parallel apps solve the latency issue is by overlapping their communication and computation phases, instead of having them in discrete phases, this can greatly reduce the time a CPU is idle.

The KeLP kernel does overlapping automatically for you if you want: http://www-cse.ucsd.edu/groups/hpcl/scg/kelp.html [ucsd.edu]

OK article, bad title

The article's worth reading, if you're not already familiar with currently popular cluster interconnects, but the title of "Data center networks often exclude Ethernet" is totally bogus.

I guess "Some Tiny Percentage of Data Centers use Something Faster than Ethernet in addition to Ethernet" didn't fit on the page.

Milliseconds? 100's of them?

--- malin.vidarlo.net ping statistics --- 15 packets transmitted, 15 received, 0% packet loss, time 14003ms rtt min/avg/max/mdev = 0.310/0.347/0.375/0.019 ms 2 hops, over 100Mb ethernet with a cheapass switch (8 port unmanaged hp). Seems like he got no grip on numbers...

Someone needs to look at their network...

Just had a quick ping to the beeb... via a wireless hop onto my ethernet network, two hops to my adsl router, then 6 hops around Nildram's network (ATM into their network then god knows, probably some form of gigabit ethernet) and a couple more hops to the bbc.

Average latency is around 20ms.

Now I know this isn't as plain as straight ethernet but I'd have guessed the latency if anything on ATM + the change from 802.11g to ethernet to atm to ethernet to whatever would have been worse.

So either someone is using cheep hardware or has misconfigured their network.

Apart from that if I was running a cluster each machine would probably have two NIC's depending on their use - one using gigabit ethernet to provide the internal network between nodes on the cluster and the other for external use. The external network would be as normal, the internal network I'd ensure had minimal routers/switches between the nodes and any switches/routers where a) good quality and b) correctly configured.

Real-World Experience

All this time I've been playing Quake over LAN and I thought my ping was about 5ms. Silly me, it's clearly in the range of 100ms, even worse than when I take it online!

Whoops...

The worst post!

I wonder what's happening to slashdot. That's as bad as technical news can get. Ethernet latency -- 100ms?? Typical Ethernet latencies are around a few hundred microseconds. Even the ping round-trip time from my machine to google.com is about 20ms.

$ ping google.com
PING google.com (64.233.167.99) 56(84) bytes of data.
64 bytes from 64.233.167.99: icmp_seq=1 ttl=241 time=20.1 ms
64 bytes from 64.233.167.99: icmp_seq=2 ttl=241 time=19.6 ms
64 bytes from 64.233.167.99: icmp_seq=3 ttl=241 time=19.5 ms

What a shame that such a post is on the front page of slashdot! Someone please s/milli/micro.

Slashdot summary wrong, actual article is better

The slashdot summary is wrong. If you read the actual article the author has it mostly correct except for one comment near the end.

Ethernet latency is about 100uS through a gigE switch, round-trip. A full-sized packet takes about 200uS (micro seconds), round-trip. Single-ended latency is about half of that.

There are proprietary technologies that have much faster interconnects, such as the infiniband technology described in the article. But the article also mentions the roadblock that a proprietary technology respresents over a widely-vendored standard. The plain fact of the matter is that ethernet is so ridiculously cheap these days it makes more sense to solve the latency issue in software, for example by designing a better cache coherency management model and by designing better clustered applications, then it does with expensive proprietary hardware.

-Matt

Ethernet Problems, IB problems, etc

Note: I do have a dog in this fight.
One thing that isn't mentioned in the article is the amount of CPU power required to send out ethernet packets. The typical rule is 1 GHz of processing power is required to send 1 Gb of data on the wire. So, if you want to send 10 Gbs of data, you'd need 10 GHz of processor - pretty steep price. Some companies have managed to get this down to 1 GHz/3 Gbs of processing, and one startup(NetEffect) is now claiming roughly ~0.1 Ghz for ~8 Gbs on the wire, using iWarp. With this, your system can be processing information rather than creating packets.
The problem with Infiniband, Myranet, etc is that they require another card in your system (and associated heat problems, size issues, etc), special switches and equipment, and new training for your staff on how to get it up and going. However, IWarp, which is based on TCP/IP can use your standard DHCP, ping, tracert, ipconfig, etc and can allow a single card to be used for networking to the outside world (TCP/IP), clustering in the datacenter(IWarp), and storage (IScsi). 1 card, no special new software widgets, 10 Gb speeds.
However, you cant go and buy a iWarp card from Fry's today. Although, you cant buy an infiniband or myranet card there either

Tolkien ring

I had recently considered using this Tolkien ring until I found out that deinstallation is very difficult. Something about having to take it to a smelter.

Well yeah...

Well, except the oblig. s/ms/us, but pretty much yeah. With Pathscale (now QLogic) Infinipath HTX cards, you can get 1.5 us latency between nodes, Myrinet 10 G PCI-E can get about 2.5 us. Note that there is now 10Gb ethernet making inroads to compete on terms of throughput (which Infinband SDR, Myrinet are roughly 10 Gbps), but latency is of course still problematic. One chief advantage of non-ethernet is those networks are source routed and every node has a full topology map of how to get to their destinations. This has the benefit of distributing the task of routing to more processors, as well as making intelligent routing decisions. With ethernet, switches have a very heavy routing burden in a busy network. Compare this to a Myrinet or Infiniband switch which merely needs to look at the next port tag and send it on. By and large when trying to do benchmarks on these technologies, we generally don't worry too much about which switch is used. Contrast with Ethernet where we have to be mindful of the packets per second capability of the switch...

Of course, on a large scale network, it is much simpler and easier to do switch-routing frames, but for tightly controlled networks, source-routed can be very advantageous.

I will say switch routed frames have the *potential* for much better utilization of multi-port aggregations, but largely the member of a multi-port aggregation used to send a packet is not based on port congestion, but rather on a hash of the MAC address referenced in the packet, which is nothing a source routed network couldn't do.

Ugh, they must have read an old paper

All new switches (that are decent) employ cut-through forwarding.

For those who don't understand...

Most (all?) Ethernet hardware reads in an entire packet, looks at it, then sends it on to a destination. This makes building routers and switching hardware fairly easy but extremely slow.

If you go to a high-speed network, what you get is a packet being forwarded as it's being read. By the time the first few bits are through the switch, it should be able to figure out the next hop and have the packet moving in that direction. Phone companies have huge problems with the delays in Ethernet. This is why the ATM protocol was invented, it's hard to use, awkward and not too graceful, but it can fly through a switching network like nobody's business.

Ethernet is also extremely sloppy--Any switch along the way is allowed to throw a packet away and wait for the originator to resend causing a HUGE hiccupp in the communication stream (Most if not all routers do this whenever an address is not in it's forwarding table yet).

IIRC the faster protocols see a "Routing" packet in the stream and set up forwarding hardware before getting the actual packet/stream, then wait until the end of the packet (or entire stream) to tear the route down again.

Ethernet, however, due to it's simplicity is bridging the gaps. It's a pretty crappy protocol in general, but we keep throwing better, smarter hardware at it in an effort to brute-force it into the parameters we require. (I work for a company that makes Ethernet over fiber hardware, and have worked for companies based around ATM, SONET and other interesting solutions).

I guess the point of the article was to remind a world that is coming to believe that ethernet is the end-all be-all of networking that it was always just the simplest hack available and therefore the easiest to deal with.

Just like SNMP.

Not so sure

But on our network we vlan'd everything out. All servers on one vlan, I.T. on another vlan, and then major groups on their own vlans. Keeps traffic nice and segregated which is why the I.T. shop has iTunes sharing turned on full blast.

But here's where I notice some performance. We've got all the servers on a gigabit vlan. I can shift a 300MB file between servers in under 20 seconds. Transitioning a 5MB link takes five minutes.

So we did what we could to eliminate latency and we see it in the performance of our network.

Could somebody clue me in?

"But in data centers, where CPUs may be sharing data in memory across different connected machines..."

I have re-read this bit like twenty times and still have no idea what it means. The terms used clashes badly, which leads me to believe that the guy has no idea what he was talking about

The didn't do enough research on the topic

The funny thing is that there is already a solution to their problem out there. Raptor Networks Technologies [raptor-networks.com] already has their ethernet switches in a bunch of places and have (so far) proven that their distributed network technology runs circles around Cisco's (and others) centralized architecture and costs even less. They could probably keep up with the needs of these data centers. I've spoken to guys who use their hardware and they all say 'wow.' This sounds like a perfect network for Raptor's hardware. Anyone else ran into it?

Blue Gene

I went over to the BlueGene/L's [llnl.gov] page to see how they manage to shuffle data around. They seem to use 1GB Ethernet links for IO nodes.

Any ideas why Ethernet is not an outsider here.

* lon3st4r *