If I'm reading this correctly, I think the problem is due to the fact that you need to bring the interface up after the driver is loaded.
So make sure you have a cable plugged in, then issue:
ip link set eth4 up
this should bring the link up on the PF, then you can assign the VLAN IDs tothe VF, and then assign the VM to the VM and then bring up the VM.
Give that a try.
- Patrick