IVS seems to fail with multiple switches in Mininet

[lantz]

I accepted Rich Lane's pull request to integrate IVS support into Mininet, but I am having trouble getting IVS to work correctly with reasonable numbers of switches.

For example, with this configuration,

sudo mn --switch ivs --controller ref --topo linear,20

This is a topology consisting of a string of 20 switches, each of which has a single host connected to it.

I wait for all of the hosts to connect to the controller, but pingall still fails. It's puzzling since the control traffic looks OK - I'm seeing a PACKET_IN and a PACKET_OUT for the ARP request, but it doesn't seem to be reaching the destination hosts.

If I use static ARP, then I see the PACKET_IN and PACKET_OUT (which is flooded because of the learning switch algorithm) for the ICMP request, but I think I am not seeing the reply, so it may not be reaching the destination host.

I am testing this on Ubuntu 12.04 with the Linux ubuntu1 3.2.0-31-generic #50-Ubuntu SMP Fri Sep 7 16:16:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux kernel

With smaller numbers of switches, it works fine:

$ sudo mn --switch ivs --topo linear,10 --controller ref --test pingall
*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2 h3 h4 h5 h6 h7 h8 h9 h10 
*** Adding switches:
s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 
*** Adding links:
(h1, s1) (h2, s2) (h3, s3) (h4, s4) (h5, s5) (h6, s6) (h7, s7) (h8, s8) (h9, s9) (h10, s10) (s1, s2) (s2, s3) (s3, s4) (s4, s5) (s5, s6) (s6, s7) (s7, s8) (s8, s9) (s9, s10) 
*** Configuring hosts
h1 h2 h3 h4 h5 h6 h7 h8 h9 h10 
*** Starting controller
*** Starting 10 switches
s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 
*** Ping: testing ping reachability
h1 -> h2 h3 h4 h5 h6 h7 h8 h9 h10 
h2 -> h1 h3 h4 h5 h6 h7 h8 h9 h10 
h3 -> h1 h2 h4 h5 h6 h7 h8 h9 h10 
h4 -> h1 h2 h3 h5 h6 h7 h8 h9 h10 
h5 -> h1 h2 h3 h4 h6 h7 h8 h9 h10 
h6 -> h1 h2 h3 h4 h5 h7 h8 h9 h10 
h7 -> h1 h2 h3 h4 h5 h6 h8 h9 h10 
h8 -> h1 h2 h3 h4 h5 h6 h7 h9 h10 
h9 -> h1 h2 h3 h4 h5 h6 h7 h8 h10 
h10 -> h1 h2 h3 h4 h5 h6 h7 h8 h9 
*** Results: 0% dropped (90/90 received)
*** Stopping 10 switches
s1 ..s2 ...s3 ...s4 ...s5 ...s6 ...s7 ...s8 ...s9 ...s10 ..
*** Stopping 10 hosts
h1 h2 h3 h4 h5 h6 h7 h8 h9 h10 
*** Stopping 1 controllers
c0 
*** Done
completed in 7.619 seconds

Open vSwitch and the OpenFlow reference switch also work fine. (Note there is currently a race condition with --pingall since it can fail if all the switches don't connect - that's why I also ran the test manually and made sure that all of the switches had connected.)

[rlane]

This seems to be a race condition during switch startup. It works reliably up to 16 switches but starts failing at 17. If I put a time.sleep(0.05) in the IVSSwitch start method I've gotten up to 32 switches with no failures.

I noticed that the netlink messages the switch uses to find about new ports aren't arriving on the broken IVS instances. Raising the rx buffer size didn't help. I'll keep investigating.

[rlane]

Using --innamespace is another workaround.