Software and hardware annotations 2007 February

This document contains only my personal opinions and calls of judgement, and where any comment is made as to the quality of anybody's work, the comment is an opinion, in my judgement.

[file this blog page at: digg del.icio.us Technorati]

070225d Ethernet speed and power draw
An article reports typical Ethernet card power consumptions which can be 1-2W for 100mb/s Fast Ethernet, 4-5W for 1gb/s, and 20-25W for 10gb/s Ethernet, and mentions that using 1gb/s Ethernet where 100gb/s would be sufficient can waste quite a bit of power. Even more so (twice as much) than the article states, because for every NIC in a PC or servers there is now an equivalent switch port, as modern Ethernet is nearly always PtP and no longer based on the idea of a shared passive medium.
070225c DMZ using static ARP entries
Chatting with a friend I mentioned an amusing an somewhat paranoid technique a smart guy I worked with used to set-up DMZ LAN: that all nodes in it have not only static UP addresses, but static ARP translations only, and disabling all ARP traffic. The purpose of that is to prevent someone who has compromised one node to be able to access and compromise other nodes, which requires seeing at least their Ethernet addresseses, which would not be observable in a switched LAN.
It may seem over the top, and I tend to favour resource-based security over node-based security, but it does not have much of a downside and might be quite helpful, especially as DMZ nodes are in a DMZ precisely because they might be compromised, and static translations reduce the risk of ARP spoofing, as summarized by in the classic book on firewalls:
2.1.2 ARP
[ ... ] There is considerable risk here if untrusted nodes have write access to the local net. Such a machine could emit phony ARP queries or replies and divert all traffic to itself; it could then either impersonate some machines or simply modify the data streams en passant. This is called ARP spoofing and a number of Hacker Off-the-Shelf (HOTS) packages implement this attack.
The ARP mechanism is usually automatic. On special security networks, the ARP mappings may be statically hardwired, and the automatic protocol suppressed to prevent interference. If we absolutely never want two hosts to talk to each other, we can ensure that they don't have ARP translations (or have wrong ARP translations) for each other for an extra level of assurance. It can be hard to ensure that they never acquire the mappings, however.
070225b Ring network topologies, bridging and routing
As I am inclined to optimism and to count my blessings, so I am quite happy with the physical topology of a large network I have to deal with, as it is 2 nested rings (hexagons actually) with 3 links between the two rings. Quite nice because it can be a resilient topology with a low max-hops, sort of like a cheap torus. Too bad that it is physically patched as a tree (argh) which is also fully bridged (and still VLANed and IIRC until recently without spanning tree and unmonitored too).
Usually I think that Ethernet bridging is usually a lazy substitute for IP routing, but even when there is a case for it, it can be done in several ways, and I was quite amused to see that single or dual ring Ethernet topologies are common enough that there is even a special spanning tree algorithm especially designed for them.
070225 Not so good USB network adapters
Out of desperate urgency I have recently bought some no-name USB Fast Ethernet network interfaces from a local shop and I have not been happy. For one thing of four I bought one was DOA, and another was defective, being sort of working but with a dead activity LED. Even worse I later read the fine print and discovered that the ADMtek Pegasus chipset it uses is remarkably Fast Ethernet and Full Speed USB, that is 100mb/s on the Ethernet side but only 12mb/s on the USB side, as the 480mb/s USB2 is called Hi-Speed USB, and Full Speed USB is 12mb/s USB1. Not an awesome combination...
070223 Multiple interfaces on the same LAN
I just mentioned that I prefer simple setups, like one subnet and interface/node per LAN, and as something not quite nice multiple interfaces on the same for the same LAN.
This actually happened to me in the case of my simple and effective video-on-demand server: initially it had two interfaces attached to two separate LANs (in the specific case hubs), on each of which half of the clients were attached, each half in a separate subnet. Therefore the bandwidth to be shared between all the clients was twice that of a single LAN. However some less than admirable people at some point changed that behind my back to a single LAN, with all clients randomly in a single subnet, and both interfaces on that LAN (a switch).
Hardware wise it was not that bad: a decent switch has more aggregate (backplane) bandwidth than two hubs (but not two switches...), and one can use two interfaces connected to two ports on the same switch as a way of giving a server twice the bandwidth (if the switch backplane allows for that). The big issues are: As to the first issue, balancing outgoing traffic across both interfaces can be done in several different ways, depending on whether one wants to balance by connection or by packet most cases balancing by connection under GNU/Linux can be done fairly simply using the nexthop routing extension, otherwise one needs to use the teql virtual interface (important teql detail: for many releases of Linux it is easiest to make it work if the teql virtual interface has the same IP address as the first interface bound to it). Even if adding equalize to nexthop should balance by packet like teql, that seems to me less reliable than teql.
As to the second issue, it arises because Linux quite helpfully but somewhat incorrectly assumes that IP addresses belong to nodes and not interfaces, and thus will ARP reply any of the node's IP addresses on any of its interfaces. To stop this one can use the hidden interface property in early versions of Linux (or for later ones thanks to a forward port) or the arp_announce and arp_ignore ones in recent versions of Linux.
070218 Multiple subnets (and VLANs) on the same wire
Well, somewhat odd logical network architectures are unfortunately a topic that resurfaces, and I was talking recently with people at a site where for whatever reasons there is a fully bridged large one site LAN (involving a couple dozen network segments) with multiple subnets on it (each subnet tied to a VLAN), and a single (or dual, but changes the story little) routers among the subnets.
Given that it is a fully bridged LAN, why a router? Well, because multiple subnets on the same LAN require it, the same as multiple subnets on different LANs: only dual-homed (or dual-address) nodes can address directly two different IP subnets. So if the same LAN has nodes in the 192.168.1.0/24 and 10.1.0.0/16 subnets, at least one node with an address in both is needed to route among the two.
With multiple-homed nodes routing between subnets on different LANs all is clear and easy: the node reads a packet from one interface on one LAN, and copies it to another interface on another LAN. But if multiple subnets are used on the same LAN, then packets are copied back to the same LAN it came from, effectively halving the bandwidth of the LAN in the case of inter-subnet traffic. Odds are that effective bandwidth will be more than halved, because the copying back will prevent back-to-back packets (but full-duplex links may suffer less from this), and thus full channel optimization.
If the LAN is a bridged one it may happen that the inter-subnet router mode is also a bridge and thus that in favourable conditions no copy back to the same segment is needed if the sending node and the target node are on different segments of the bridged LAN and both their Ethernet addresses are in the spanning tree tables.
In general my preference is to keep things straighforward: don't bridge, just one routed subnet per LAN, a single interface per LAN per node. Unfortunately sometimes one has to contend with exceptions, like multiple subnets per LAN, or even multiple interfaces on the same node to the same LAN and subnet. But very very rarely there is a good reason for any of these complications, which also impact resilience, as routing among subnets adds to the number of bits of active equipment that can fail.