Hi Patrick,
Thanks so much for your reply. I hope you enjoyed your vacation time and I'm really sorry you have to deal with me upon your return!
So, I wouldn't normally assign MAC addresses to the VFs, I only went that route because of the random behavior of the VFs after starting the VM that the hostdev interfaces were assigned to.
For this reply, I changed my guest from FreeBSD to CentOS to perhaps simplify things but what I was originally seeing and the reason I even set them was because I was arbitrarily getting an all zeros MAC randomly on the VMs interfaces.
First, in response to your recommended steps.
1) ixgbevf is being blacklisted from within /etc/modprobe.d/blacklist.conf
blacklist ixgbevf
2) I am loading ixgbe and VFs from within /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=16
options loop max_loop=128
3) Now begging with my new CentOS VM, I have not assigned MAC addresses to the VFs.
root@ubuntu:~# ip link show dev eth4
5: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto
vf 4 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto
vf 5 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto
vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
4) VLANs are assigned (see output from 3)
5) Here is the applicable snippet from my guest's XML configuration
<interface type='hostdev' managed='yes'> | |
<mac address='52:54:00:58:5f:0c'/> | |
<source> | |
<address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x0'/> | |
</source> | |
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> | |
</interface> | |
<interface type='hostdev' managed='yes'> | |
<mac address='52:54:00:d4:63:a7'/> | |
<source> | |
<address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x1'/> | |
</source> | |
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> | |
</interface> | |
<interface type='hostdev' managed='yes'> | |
<mac address='52:54:00:51:5c:0a'/> | |
<source> | |
<address type='pci' domain='0x0000' bus='0x81' slot='0x10' function='0x2'/> | |
</source> | |
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> | |
</interface> |
6) And the result from starting the guest which still reflects bizarre behavior. Notice from the log output below that a MAC was set to VF0 two times, a MAC was set for VF1 but VF2 was never set.
root@ubuntu:~# ip link show dev eth4
5: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:58:5f:0c, spoof checking on, link-state auto
vf 1 MAC 52:54:00:51:5c:0a, spoof checking on, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto
vf 4 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto
vf 5 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto
vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
Jul 22 12:53:40 ubuntu kernel: [ 3138.012846] pci-stub 0000:81:10.0: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.012936] pci-stub 0000:81:10.2: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.012951] pci-stub 0000:81:10.4: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.012972] pci-stub 0000:81:10.6: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.012988] pci-stub 0000:81:11.0: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013002] pci-stub 0000:81:11.2: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013017] pci-stub 0000:81:11.4: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013031] pci-stub 0000:81:11.6: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013045] pci-stub 0000:81:12.0: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013057] pci-stub 0000:81:12.2: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013068] pci-stub 0000:81:12.4: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013080] pci-stub 0000:81:12.6: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013091] pci-stub 0000:81:13.0: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013102] pci-stub 0000:81:13.2: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013113] pci-stub 0000:81:13.4: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013125] pci-stub 0000:81:13.6: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013136] pci-stub 0000:81:10.1: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013149] pci-stub 0000:81:10.3: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013166] pci-stub 0000:81:10.5: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013178] pci-stub 0000:81:10.7: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013190] pci-stub 0000:81:11.1: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013201] pci-stub 0000:81:11.3: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013213] pci-stub 0000:81:11.5: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013224] pci-stub 0000:81:11.7: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013235] pci-stub 0000:81:12.1: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013246] pci-stub 0000:81:12.3: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013258] pci-stub 0000:81:12.5: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013269] pci-stub 0000:81:12.7: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013280] pci-stub 0000:81:13.1: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013291] pci-stub 0000:81:13.3: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013302] pci-stub 0000:81:13.5: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.013314] pci-stub 0000:81:13.7: claimed by stub
Jul 22 12:53:40 ubuntu kernel: [ 3138.033103] ixgbe 0000:81:00.0: setting MAC 52:54:00:58:5f:0c on VF 0
Jul 22 12:53:40 ubuntu kernel: [ 3138.033109] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.
Jul 22 12:53:40 ubuntu kernel: [ 3138.033112] ixgbe 0000:81:00.0: The VF MAC address has been set, but the PF device is not up.
Jul 22 12:53:40 ubuntu kernel: [ 3138.033115] ixgbe 0000:81:00.0: Bring the PF device up before attempting to use the VF device.
Jul 22 12:53:40 ubuntu kernel: [ 3138.045072] ixgbe 0000:81:00.1: setting MAC 52:54:00:d4:63:a7 on VF 0
Jul 22 12:53:40 ubuntu kernel: [ 3138.045078] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.
Jul 22 12:53:40 ubuntu kernel: [ 3138.045081] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.
Jul 22 12:53:40 ubuntu kernel: [ 3138.045084] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.
Jul 22 12:53:40 ubuntu kernel: [ 3138.053024] ixgbe 0000:81:00.0: setting MAC 52:54:00:51:5c:0a on VF 1
Jul 22 12:53:40 ubuntu kernel: [ 3138.053030] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.
Jul 22 12:53:40 ubuntu kernel: [ 3138.053033] ixgbe 0000:81:00.0: The VF MAC address has been set, but the PF device is not up.
Jul 22 12:53:40 ubuntu kernel: [ 3138.053036] ixgbe 0000:81:00.0: Bring the PF device up before attempting to use the VF device.
Jul 22 12:53:40 ubuntu kernel: [ 3138.305853] type=1400 audit(1406058820.760:42): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-d3724bdc-fc97-37d0-437c-33f5d54fb811" pid=2795 comm="apparmor_parser"
Jul 22 12:53:40 ubuntu kernel: [ 3138.419792] pci-stub 0000:81:10.0: enabling device (0000 -> 0002)
Jul 22 12:53:42 ubuntu kernel: [ 3139.873334] pci-stub 0000:81:10.0: kvm assign device
Jul 22 12:53:42 ubuntu kernel: [ 3139.874950] pci-stub 0000:81:10.1: enabling device (0000 -> 0002)
Jul 22 12:53:42 ubuntu kernel: [ 3139.978660] pci-stub 0000:81:10.1: kvm assign device
Jul 22 12:53:42 ubuntu kernel: [ 3139.981161] pci-stub 0000:81:10.2: enabling device (0000 -> 0002)
Jul 22 12:53:42 ubuntu kernel: [ 3140.082788] pci-stub 0000:81:10.2: kvm assign device
Jul 22 12:53:49 ubuntu kernel: [ 3146.897520] kvm [2797]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd
Jul 22 12:53:49 ubuntu kernel: [ 3146.967250] kvm: zapping shadow pages for mmio generation wraparound
Jul 22 12:53:51 ubuntu kernel: [ 3149.537229] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X
Jul 22 12:53:51 ubuntu kernel: [ 3149.537245] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.557331] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.557372] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.577287] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.577329] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.605431] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.605471] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.621296] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.621327] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.645417] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X
Jul 22 12:53:52 ubuntu kernel: [ 3149.645458] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X
I'm not ruling out yet that I could be doing something wrong.
Thanks for your assistance.
JC