82599ES SR-IOV on Ubuntu 14.04 - VF behavior

Hi,

This is my first visit/post here to this community so I hope I'm asking the right questions in the right place.

I am using the 82599ES adapter with SR-IOV enabled under Ubuntu 14.04 under kernel 3.13.0-30 but this same behavior was witness under 3.13.0-24.  The corresponding VFs are created and visible in 'lspci' and 'ip link show'.  I am intending to pass VFs to guest OSs in KVM running libvirt 1.2.2 as 'hostdev' devices in the guest's XML configuration file.

Once the VFs are created, I proceed to set VF parameters such as a MAC and VLAN ID.  I believed MACs were to be automatically generated by the driver but perhaps I'm wrong as it doesn't seem to be the case.  So in this particular case, I set the VF MAC to the auto generated MAC that libvirt assigns to the interface by editing the guest through virsh edit.

root@ubuntu:~# ip link set eth4 up

root@ubuntu:~# ip link set eth4 vf 0 mac 52:54:00:4d:8b:bc

root@ubuntu:~# ip link set eth4 vf 0 vlan 100

root@ubuntu:~# ip link set eth4 vf 1 mac 52:54:00:5e:3b:e2

root@ubuntu:~# ip link set eth4 vf 1 vlan 200

root@ubuntu:~# ip link set eth4 vf 2 mac 52:54:00:19:e3:f8

root@ubuntu:~# ip link set eth4 vf 2 vlan 300

root@ubuntu:~# ip link set eth4 vf 3 mac 52:54:00:1c:7e:79

root@ubuntu:~# ip link set eth4 vf 3 vlan 100

root@ubuntu:~# ip link set eth4 vf 4 mac 52:54:00:45:c9:7d

root@ubuntu:~# ip link set eth4 vf 4 vlan 200

root@ubuntu:~# ip link set eth4 vf 5 mac 52:54:00:32:13:0e

root@ubuntu:~# ip link set eth4 vf 5 vlan 300

This results in something like the following (dmesg output below also).

root@ubuntu:~# ip link show dev eth4

11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

    vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

    vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

    vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

    vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

    vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

Jul 16 10:29:21 ubuntu kernel: [ 8518.503869] ixgbe 0000:81:00.0: registered PHC device on eth4

Jul 16 10:29:21 ubuntu kernel: [ 8518.608585] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready

Jul 16 10:29:21 ubuntu kernel: [ 8518.672821] ixgbe 0000:81:00.0 eth4: detected SFP+: 3

Jul 16 10:29:21 ubuntu kernel: [ 8518.912379] ixgbe 0000:81:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Jul 16 10:29:21 ubuntu kernel: [ 8518.912536] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

Jul 16 10:29:28 ubuntu kernel: [ 8526.350238] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

Jul 16 10:29:28 ubuntu kernel: [ 8526.350244] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.363123] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 0

Jul 16 10:29:28 ubuntu kernel: [ 8526.387115] ixgbe 0000:81:00.0: setting MAC 52:54:00:5e:3b:e2 on VF 1

Jul 16 10:29:28 ubuntu kernel: [ 8526.387120] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.399143] ixgbe 0000:81:00.0: Setting VLAN 200, QOS 0x0 on VF 1

Jul 16 10:29:28 ubuntu kernel: [ 8526.432961] ixgbe 0000:81:00.0: setting MAC 52:54:00:19:e3:f8 on VF 2

Jul 16 10:29:28 ubuntu kernel: [ 8526.432967] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.435878] ixgbe 0000:81:00.0: Setting VLAN 300, QOS 0x0 on VF 2

Jul 16 10:29:29 ubuntu kernel: [ 8526.468947] ixgbe 0000:81:00.0: setting MAC 52:54:00:1c:7e:79 on VF 3

Jul 16 10:29:29 ubuntu kernel: [ 8526.468953] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:29 ubuntu kernel: [ 8526.471825] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 3

Jul 16 10:29:29 ubuntu kernel: [ 8526.495211] ixgbe 0000:81:00.0: setting MAC 52:54:00:45:c9:7d on VF 4

Jul 16 10:29:29 ubuntu kernel: [ 8526.495217] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:29 ubuntu kernel: [ 8526.561157] ixgbe 0000:81:00.0: Setting VLAN 200, QOS 0x0 on VF 4

Jul 16 10:29:29 ubuntu kernel: [ 8526.564616] ixgbe 0000:81:00.0: setting MAC 52:54:00:32:13:0e on VF 5

Jul 16 10:29:29 ubuntu kernel: [ 8526.564621] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:31 ubuntu kernel: [ 8528.628424] ixgbe 0000:81:00.0: Setting VLAN 300, QOS 0x0 on VF 5

I am passing VF 0-2 to one guest OS and VF 3-5 to a second OS. 

Guest1

Guest2

And then I start the guests.  They boot normal and detect VFs I've passed through but this is where things get hokey.  The guest may sporadically have an all zeros MAC and sometimes not.  But most notably is this difference in the output of 'ip link show dev eth4' compared to before the guest was started.

Before Guest Start

root@ubuntu:~# ip link show dev eth4

11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

    vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

    vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

    vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

    vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

    vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

After Guest Start

root@ubuntu:~# ip link show dev eth4

11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC 52:54:00:4d:8b:bc, spoof checking on, link-state auto

    vf 1 MAC 52:54:00:19:e3:f8, spoof checking on, link-state auto

    vf 2 MAC 52:54:00:45:c9:7d, spoof checking on, link-state auto

    vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

    vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

    vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

The VLAN ID is missing for VF0-2.  VF1 has VF2's original MAC and VF2's MAC has become VF4's MAC.

Here is the resulting dmesg output from starting my guests.

Jul 16 10:37:56 ubuntu kernel: [ 9034.298337] pci-stub 0000:81:10.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298359] pci-stub 0000:81:10.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298369] pci-stub 0000:81:10.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298378] pci-stub 0000:81:10.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298385] pci-stub 0000:81:11.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298411] pci-stub 0000:81:11.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298428] pci-stub 0000:81:11.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298439] pci-stub 0000:81:11.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298453] pci-stub 0000:81:12.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298466] pci-stub 0000:81:12.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298478] pci-stub 0000:81:12.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298489] pci-stub 0000:81:12.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298499] pci-stub 0000:81:13.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298509] pci-stub 0000:81:13.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298519] pci-stub 0000:81:13.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298528] pci-stub 0000:81:13.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298539] pci-stub 0000:81:10.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298550] pci-stub 0000:81:10.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298561] pci-stub 0000:81:10.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298571] pci-stub 0000:81:10.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298580] pci-stub 0000:81:11.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298591] pci-stub 0000:81:11.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298601] pci-stub 0000:81:11.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298611] pci-stub 0000:81:11.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298621] pci-stub 0000:81:12.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298633] pci-stub 0000:81:12.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298643] pci-stub 0000:81:12.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298653] pci-stub 0000:81:12.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298662] pci-stub 0000:81:13.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298672] pci-stub 0000:81:13.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298683] pci-stub 0000:81:13.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298693] pci-stub 0000:81:13.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.324752] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

Jul 16 10:37:56 ubuntu kernel: [ 9034.324759] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:37:56 ubuntu kernel: [ 9034.344723] ixgbe 0000:81:00.1: setting MAC 52:54:00:5e:3b:e2 on VF 0

Jul 16 10:37:56 ubuntu kernel: [ 9034.344730] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

Jul 16 10:37:56 ubuntu kernel: [ 9034.344733] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

Jul 16 10:37:56 ubuntu kernel: [ 9034.344736] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

Jul 16 10:37:56 ubuntu kernel: [ 9034.360702] ixgbe 0000:81:00.0: setting MAC 52:54:00:19:e3:f8 on VF 1

Jul 16 10:37:56 ubuntu kernel: [ 9034.360708] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:37:56 ubuntu kernel: [ 9034.665913] audit_printk_skb: 69 callbacks suppressed

Jul 16 10:37:56 ubuntu kernel: [ 9034.665915] type=1400 audit(1405532276.635:35): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-b1ce4fac-83d7-3f00-8127-07ab6f1e15e3" pid=2779 comm="apparmor_parser"

Jul 16 10:37:56 ubuntu kernel: [ 9034.880720] pci-stub 0000:81:10.0: enabling device (0000 -> 0002)

Jul 16 10:37:58 ubuntu kernel: [ 9036.454955] pci-stub 0000:81:10.0: kvm assign device

Jul 16 10:37:58 ubuntu kernel: [ 9036.456580] pci-stub 0000:81:10.1: enabling device (0000 -> 0002)

Jul 16 10:37:58 ubuntu kernel: [ 9036.558625] pci-stub 0000:81:10.1: kvm assign device

Jul 16 10:37:58 ubuntu kernel: [ 9036.559693] pci-stub 0000:81:10.2: enabling device (0000 -> 0002)

Jul 16 10:37:58 ubuntu kernel: [ 9036.662765] pci-stub 0000:81:10.2: kvm assign device

Jul 16 10:38:22 ubuntu kernel: [ 9060.288671] kvm: zapping shadow pages for mmio generation wraparound

Jul 16 10:38:22 ubuntu kernel: [ 9060.882172] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 0

Jul 16 10:38:22 ubuntu kernel: [ 9060.893544] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.921582] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.921601] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.945515] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.945528] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.053636] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.073718] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.073732] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.101789] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.101803] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.187991] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 1

Jul 16 10:38:23 ubuntu kernel: [ 9061.201800] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.225840] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.225863] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.261958] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.261972] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

Jul 16 10:38:31 ubuntu kernel: [ 9069.898230] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 0

Jul 16 10:38:42 ubuntu kernel: [ 9080.567505] ixgbe 0000:81:00.1: setting MAC 52:54:00:1c:7e:79 on VF 1

Jul 16 10:38:42 ubuntu kernel: [ 9080.567510] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

Jul 16 10:38:42 ubuntu kernel: [ 9080.567512] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

Jul 16 10:38:42 ubuntu kernel: [ 9080.567513] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

Jul 16 10:38:42 ubuntu kernel: [ 9080.583695] ixgbe 0000:81:00.0: setting MAC 52:54:00:45:c9:7d on VF 2

Jul 16 10:38:42 ubuntu kernel: [ 9080.583702] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:38:42 ubuntu kernel: [ 9080.595660] ixgbe 0000:81:00.1: setting MAC 52:54:00:32:13:0e on VF 2

Jul 16 10:38:42 ubuntu kernel: [ 9080.595666] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

Jul 16 10:38:42 ubuntu kernel: [ 9080.595669] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

Jul 16 10:38:42 ubuntu kernel: [ 9080.595672] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

Jul 16 10:38:42 ubuntu kernel: [ 9080.866600] type=1400 audit(1405532322.783:36): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-ed80fbe8-a135-6d45-1457-5d9d00f1d5ce" pid=2805 comm="apparmor_parser"

Jul 16 10:38:42 ubuntu kernel: [ 9081.052820] pci-stub 0000:81:10.3: enabling device (0000 -> 0002)

Jul 16 10:38:44 ubuntu kernel: [ 9082.503332] pci-stub 0000:81:10.3: kvm assign device

Jul 16 10:38:44 ubuntu kernel: [ 9082.504917] pci-stub 0000:81:10.4: enabling device (0000 -> 0002)

Jul 16 10:38:44 ubuntu kernel: [ 9082.605369] pci-stub 0000:81:10.4: kvm assign device

Jul 16 10:38:44 ubuntu kernel: [ 9082.606482] pci-stub 0000:81:10.5: enabling device (0000 -> 0002)

Jul 16 10:38:44 ubuntu kernel: [ 9082.709469] pci-stub 0000:81:10.5: kvm assign device

Jul 16 10:39:08 ubuntu kernel: [ 9106.462615] kvm: zapping shadow pages for mmio generation wraparound

Jul 16 10:39:08 ubuntu kernel: [ 9107.064370] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

Jul 16 10:39:08 ubuntu kernel: [ 9107.088349] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

Jul 16 10:39:08 ubuntu kernel: [ 9107.088364] pci-stub 0000:81:10.3: irq 200 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.108372] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.108387] pci-stub 0000:81:10.3: irq 200 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.209645] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 2

Jul 16 10:39:09 ubuntu kernel: [ 9107.224472] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.252548] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.252563] pci-stub 0000:81:10.4: irq 202 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.272562] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.272578] pci-stub 0000:81:10.4: irq 202 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.364623] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.384680] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.384695] pci-stub 0000:81:10.5: irq 204 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.404698] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.404713] pci-stub 0000:81:10.5: irq 204 for MSI/MSI-X

Jul 16 10:39:28 ubuntu kernel: [ 9126.991794] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 2

root@ubuntu:~# cat /etc/modprobe.d/ixgbe.conf

options ixgbe max_vfs=16

options loop max_loop=128

blacklist ixgbevf

I removed the module and re-added to provide the dmesg output here.

root@ubuntu:~# modprobe ixgbe

Jul 16 10:20:31 ubuntu kernel: [ 7988.720851] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k

Jul 16 10:20:31 ubuntu kernel: [ 7988.720856] ixgbe: Copyright (c) 1999-2013 Intel Corporation.

Jul 16 10:20:31 ubuntu kernel: [ 7988.721376] ixgbe 0000:81:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

Jul 16 10:20:32 ubuntu kernel: [ 7988.952543] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.952824] pci 0000:81:10.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953029] pci 0000:81:10.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953200] pci 0000:81:10.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953367] pci 0000:81:11.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953533] pci 0000:81:11.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953698] pci 0000:81:11.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953870] pci 0000:81:11.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954039] pci 0000:81:12.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954214] pci 0000:81:12.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954400] pci 0000:81:12.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954629] pci 0000:81:12.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954805] pci 0000:81:13.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954976] pci 0000:81:13.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.955148] pci 0000:81:13.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.955400] pci 0000:81:13.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.955842] ixgbe 0000:81:00.0 (unregistered net_device): SR-IOV enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7988.979951] ixgbe 0000:81:00.0: irq 173 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7988.979989] ixgbe 0000:81:00.0: irq 174 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7988.980011] ixgbe 0000:81:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1

Jul 16 10:20:32 ubuntu kernel: [ 7988.980144] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available

Jul 16 10:20:32 ubuntu kernel: [ 7988.980148] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

Jul 16 10:20:32 ubuntu kernel: [ 7988.980234] ixgbe 0000:81:00.0: MAC: 2, PHY: 9, SFP+: 3, PBA No: E66560-006

Jul 16 10:20:32 ubuntu kernel: [ 7988.980237] ixgbe 0000:81:00.0: 90:e2:ba:47:2c:30

Jul 16 10:20:32 ubuntu kernel: [ 7988.982095] ixgbe 0000:81:00.0 eth4: IOV is enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7988.982253] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection

Jul 16 10:20:32 ubuntu kernel: [ 7988.982599] ixgbe 0000:81:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

Jul 16 10:20:32 ubuntu kernel: [ 7989.212846] pci 0000:81:10.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213062] pci 0000:81:10.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213161] pci 0000:81:10.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213257] pci 0000:81:10.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213354] pci 0000:81:11.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213451] pci 0000:81:11.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213547] pci 0000:81:11.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213713] pci 0000:81:11.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214277] pci 0000:81:12.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214454] pci 0000:81:12.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214607] pci 0000:81:12.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214740] pci 0000:81:12.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214924] pci 0000:81:13.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215088] pci 0000:81:13.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215222] pci 0000:81:13.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215398] pci 0000:81:13.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215567] ixgbe 0000:81:00.1 (unregistered net_device): SR-IOV enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7989.239728] ixgbe 0000:81:00.1: irq 175 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7989.239767] ixgbe 0000:81:00.1: irq 176 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7989.239793] ixgbe 0000:81:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1

Jul 16 10:20:32 ubuntu kernel: [ 7989.239930] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available

Jul 16 10:20:32 ubuntu kernel: [ 7989.239934] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

Jul 16 10:20:32 ubuntu kernel: [ 7989.240020] ixgbe 0000:81:00.1: MAC: 2, PHY: 9, SFP+: 4, PBA No: E66560-006

Jul 16 10:20:32 ubuntu kernel: [ 7989.240023] ixgbe 0000:81:00.1: 90:e2:ba:47:2c:31

Jul 16 10:20:32 ubuntu kernel: [ 7989.242601] ixgbe 0000:81:00.1 eth5: IOV is enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7989.242852] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection

root@ubuntu:~# lspci | grep 82599

81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

A few additional notes on what I've done.  I've taken the kernel as mentioned before from 3.13.0-24 to 3.13.0-30.

I am using the packaged driver 3.15.1-k in Ubuntu 14.04 as it appears there is issues with compiling the latest driver on 14.04 according to bug #415 and #421, so I have been unsucesfull there.

I've upgrade iproute2 from 3.12.0-2 to 3.15.0

I've performed SR-IOV using another vendor's card under Ubuntu 14.04 and same versions of libvirt and iproute2 and didn't experience this behavior so for that reason I've started here.

Any help or ideas on why things appear to get scrabled is appreciated. 

Thanks!

JC