ARTICLE UPDATED 05/22/2009 — Update at bottom of the page.
Likely old news to many of you, but dladm is pretty slick.
A company that I work with has some Sun X4500 boxes that they use for various tasks. The X4500, also known as Thumper, shipped with four e1000g interfaces and 24TB or 48TB of raw storage, all in a 4U 160 pound box. Though discontinued by Sun and replaced with the nicer x4540 box these old Thumpers are fine machines.
Recently I found the need to use three of the four interfaces for a larger pipe to a Cisco switch that allows port aggregation. The reasons for doing this or for using 3 ports are not really important, in your environment you might do things differently. After the box was racked and cabled the switch was configured for the ports used.
Moving on to the server, I was amazed at how simple it was to bond the three interfaces together in Solaris 10 using the dladm command.
On this hardware the devices were e1000g0 — > e1000g3:
The man page for dladm contained all I needed to get started.
First, a look at the devices. (switch ports were disabled at this time.)
To create an aggregate with the three devices and give it an integer as the aggregate instance the command was simply:
You can see from the output above that the devices must not be in use, e1000g0 was so the dladm command failed. That should be expected. Though e1000g0 had already been configured with an IP address, I was connected to the thumper via our Lantronix serial console. It would be unwise to unplumb the interface you came in on.
At this point ifconfig -a should not show anything (unless you have other devices or you’ve configured e1000g3, etc.).
Initially the switch ports were disabled at the Cisco, they were up for the following look at the aggregate created above.
Your MAC addresses will not look like the modified example.
We can now see what appears to be a new device, the list shows all of the link devices. The one we are interested in is aggr1.
All that is needed is to plumb up the interface (aggr1) and we are good to go.
Make sure to rename /etc/hostname.e1000g0 to /etc/hostname.aggr1 if you want the machine to be available on the network after a reboot.
Nice, it just works!
Now to do some testing….film at 11.
Update 030909:
I forgot to mention that I’ll be trying this out with JumboFrames enabled (and disabled). To get there I’ve found better results with editing /kernel/drv/e1000g.conf then by using ndd. Same holds true for when you are forced to disable auto-negotiation and hard code the media settings. Have a look at /kernel/drv/e1000g.conf, comments in the file are sufficient.
Example entry for JumboFrames follows:
If you want to hardcode media/duplex e1000g you can edit /kernel/drv/e1000g.conf and reboot.
Update 052209:
After testing some zfs send / zfs rcv transfers from another thumper (using mbuffer) I decided to set LACP mode to active (both on the switch and the server).
Here is the info on how that was done.
The switch involved runs CatOS and the following command is what was used there (for the ports 5/5 -> 5/7). This impacts the whole module so consider this carefully before pressing enter :)
When time permits I’ll post some transfer times using mbuffer,rsync, etc.
Likely old news to many of you, but dladm is pretty slick.
A company that I work with has some Sun X4500 boxes that they use for various tasks. The X4500, also known as Thumper, shipped with four e1000g interfaces and 24TB or 48TB of raw storage, all in a 4U 160 pound box. Though discontinued by Sun and replaced with the nicer x4540 box these old Thumpers are fine machines.
Recently I found the need to use three of the four interfaces for a larger pipe to a Cisco switch that allows port aggregation. The reasons for doing this or for using 3 ports are not really important, in your environment you might do things differently. After the box was racked and cabled the switch was configured for the ports used.
Moving on to the server, I was amazed at how simple it was to bond the three interfaces together in Solaris 10 using the dladm command.
On this hardware the devices were e1000g0 — > e1000g3:
The man page for dladm contained all I needed to get started.
First, a look at the devices. (switch ports were disabled at this time.)
root@thumper2 # dladm show-dev
e1000g0 link: down speed: 0 Mbps duplex: half
e1000g1 link: unknown speed: 0 Mbps duplex: half
e1000g2 link: unknown speed: 0 Mbps duplex: half
e1000g3 link: unknown speed: 0 Mbps duplex: halfTo create an aggregate with the three devices and give it an integer as the aggregate instance the command was simply:
root@thumper2 # dladm create-aggr -d e1000g0 -d e1000g1 -d e1000g2 1
dladm: create operation failed: Device busy (invalid interface name)
root@thumper2 # ifconfig e1000g0 unplumb
root@thumper2 # dladm create-aggr -d e1000g0 -d e1000g1 -d e1000g2 1
root@thumper2 #You can see from the output above that the devices must not be in use, e1000g0 was so the dladm command failed. That should be expected. Though e1000g0 had already been configured with an IP address, I was connected to the thumper via our Lantronix serial console. It would be unwise to unplumb the interface you came in on.
At this point ifconfig -a should not show anything (unless you have other devices or you’ve configured e1000g3, etc.).
Initially the switch ports were disabled at the Cisco, they were up for the following look at the aggregate created above.
root@thumper2 # dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:14:4f:ff:ff:ff (auto)
device address speed duplex link state
e1000g0 0:14:4f:ff:ff:ff 1000 Mbps full up attached
e1000g1 0:14:4f:ff:ff:ff 1000 Mbps full up attached
e1000g2 0:14:4f:ff:ff:ff 1000 Mbps full up attachedYour MAC addresses will not look like the modified example.
We can now see what appears to be a new device, the list shows all of the link devices. The one we are interested in is aggr1.
root@thumper2 # dladm show-link
e1000g0 type: non-vlan mtu: 1500 device: e1000g0
e1000g1 type: non-vlan mtu: 1500 device: e1000g1
e1000g2 type: non-vlan mtu: 1500 device: e1000g2
e1000g3 type: non-vlan mtu: 1500 device: e1000g3
aggr1 type: non-vlan mtu: 1500 aggregation: key 1
root@thumper2 #All that is needed is to plumb up the interface (aggr1) and we are good to go.
root@thumper2 # ifconfig aggr1 plumb 192.168.1.2 netmask 255.255.255.0 up
root@thumper2 # ifconfig aggr1
aggr1: flags=1000843 mtu 1500 index 3
inet 192.168.1.2 netmask ffffff00 broadcast 192.168.1.255
ether 0:14:4f:fff:ff:ff
root@thumper2 #
root@thumper2 # ping thumper4
thumper4 is alive
root@thumper2 # Make sure to rename /etc/hostname.e1000g0 to /etc/hostname.aggr1 if you want the machine to be available on the network after a reboot.
Nice, it just works!
Now to do some testing….film at 11.
Update 030909:
I forgot to mention that I’ll be trying this out with JumboFrames enabled (and disabled). To get there I’ve found better results with editing /kernel/drv/e1000g.conf then by using ndd. Same holds true for when you are forced to disable auto-negotiation and hard code the media settings. Have a look at /kernel/drv/e1000g.conf, comments in the file are sufficient.
Example entry for JumboFrames follows:
MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;If you want to hardcode media/duplex e1000g you can edit /kernel/drv/e1000g.conf and reboot.
Update 052209:
After testing some zfs send / zfs rcv transfers from another thumper (using mbuffer) I decided to set LACP mode to active (both on the switch and the server).
Here is the info on how that was done.
[root@thumper2:~]# dladm modify-aggr -PL2 -l active 1
[root@thumper2:~]#
[root@thumper2:~]# dladm show-aggr -L
key: 1 (0x0001) policy: L2 address: 0:14:4f:99:99:99 (auto)
LACP mode: active LACP timer: short
device activity timeout aggregatable sync coll dist defaulted expired
e1000g0 active short yes yes yes yes no no
e1000g1 active short yes yes yes yes no no
e1000g2 active short yes yes yes yes no no
[root@thumper2:~]#The switch involved runs CatOS and the following command is what was used there (for the ports 5/5 -> 5/7). This impacts the whole module so consider this carefully before pressing enter :)
set port lacp-channel 5/5,5/6,5/7 mode activeWhen time permits I’ll post some transfer times using mbuffer,rsync, etc.
No comments:
Post a Comment