The IOSDriver's BGP neighbor details cannot be pickled, which causes issues with peering-manager

[paketb0te]

Cheers,

we are running peering-manager, which uses the wonderful NAPALM library to fetch data from routers.

For some reason we are still running IOS devices from the stone age in some locations, which causes us a little issue - NAPALM is of course perfectly fine talking to these devices, as the built-in IOSDriver "just works" (thanks again to all the wonderful people contributing to this great piece of software!).

BUT peering-manager then wants to process that data, and stores that data in redis for caching - and this is where the trouble starts, because apparently, the data that is returned by IOSDriver.get_bgp_neighbors_detail() does not like to be pickled. It is my understanding that this is because it uses a defaultdict in combination with a lambda function to generate a nested defaultdict, and only named functions can be pickled, not lambdas.

As far as I can see, there are two simple solutions to this:

1. replace the lambda function with a named function:

# OLD
bgp_detail = defaultdict(lambda: defaultdict(lambda: []))

# NEW
def inner_defaultdict(): return defaultdict(list)
bgp_detail = defaultdict(inner_defaultdict)

or
2) don't use defaultdicts at all, and create the missing dicts / list manually (as is done e.g. in the IOSXRDriver)

bgp_detail = {}
...
for neigh in bgp_sum:
    ...
    vrf_name = details["routing_table"]
    if vrf_name not in bgp_detail.keys():
        bgp_detail[vrf_name] = {}
    remote_as = details["remote_as"]
    if remote_as not in bgp_detail[vrf_name].keys():
        bgp_detail[vrf_name][remote_as] = []

happy to implement this, just let me know which one you prefer.

[paketb0te]

@mirceaulinic, @ktbyers, is there something I can do to make evaluating this issue and the associated PRs easier for you?

I understand you probably have important things to do so I would like to make it as easy for you as possible to review this minor request.

Maybe a TL,DR:
My proposed changes do not change the logical behaviour of the IOSDriver at all, it's just replacing a lambda function with a named function so the object can be pickled.

Thank you very much for the great work you have put into NAPALM!