veritas llt… at last… :-)

I’ve been using Veritas Cluster Server for a few months now, and I can say that it performs really fine, with many more features than I could expect. One thing that bothered me from the beginning, though, is that 9/10 of my servers being part of a cluster were cluttered of strange/weird messages in the dmesg about the llt protocol/subsystem.

llt (low latency transport) is the protocol used in Veritas Cluster Server for heartbeat communication between cluster members. As the (poor, in this case) documentation states, it should perform really much bettere than tcp-udp/ip, which is a must in case of clusters, and even more a must since afair, llt transports also locks/state information about Veritas Clustered Filesystem. This in particular is a piece of jewel imho, since I have been chasing a fast/reliable yet simple approach to a shared filesystems for ages ;-).

Well, about the llt issue, it appeared that apart from being so smart and quick, it complained in my poor dmesgs about “lost packets” or maybe “packets out of sync”. To put it simply, it turned out (after emailing a Veritas support representative) that in case you use more than one nic per cluster for heartbeat purposes (that’s my case, indeed 😉 ), they should by default not share the same networking hardware (hub, switch). In my case all clusters share the same couple of Gigabit Ethernet (copper) switches, which in turn are tied together by a h.a. setup.

In conclusion, there was a solution directly on the servers’ side. Just change one entry in /etc/llttab and you’re set. Actually, this caused the different nics to “pair up” to their right partner in the cluster (e.g. eth2 to eth2, eth3 to eth3). This is the so called “SAP”, which defaults to the hex value “0xcafe”, and which I have changed , as suggested, to “0xcaf0” instead. This had to be done on all nodes and according to the named interfaces (only on eth3, in my case).

Well… this probably may sound heavy voodoo (and mostly useless 🙂 ) to all the fellas that didn’t get the chance to dig their nose in the VRTS stuff… but once you’re at it… I can guarantee this cluster software is a really good piece of code. Expensive… but it’s really well worth the price. Actually, it’s one of the few non-opensource solutions that I would suggest (while I’m using succesfully MANY opensource clustering solutions… from openssi to linux-ha, from ultramonkey to heartbeat, lvs and friends). Ah, and btw it cleaned up my poor dmesgs at last… and afaik even the clustered filesystem and the cluster overall should perform even better.