[VPP-1924] dhcp: dhcp client causes vpp crash when in multi threads/workers #3386

vvalderrv · 2025-02-02T09:41:13Z

Description

Version: v20.05-14~gd4b5fdde4

Reproduce: use multi threads vpp, set dhcp client intfc GigabitEthernet0/7/0

When vpp is used in multi threads/workers, the dhcp client causes the vpp crash.

the backtrace is

#0 0x00007ffff53c7387 in raise () from /lib64/libc.so.6

#1 0x00007ffff53c8a78 in abort () from /lib64/libc.so.6

#2 0x000000000040858e in os_panic () at /data/workspace/vpp/src/vpp/vnet/main.c:366

#3 0x00007ffff61688eb in debugger () at /data/workspace/vpp/src/vppinfra/error.c:84

#4 0x00007ffff6168cd6 in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7fffb3d2b1d0 "%s:%d (%s) assertion `%s' fails") at /data/workspace/vpp/src/vppinfra/error.c:143 #5 0x00007fffb3cba1e5 in vlib_time_now (vm=0x7ffff7f81780 <vlib_global_main>) at /data/workspace/vpp/src/vlib/main.h:299

#6 0x00007fffb3cbd0db in dhcp_client_for_us (bi=636440, b=0x10026d8600, ip=0x10026d870e, udp=0x10026d8722, dhcp=0x10026d872a) at /data/workspace/vpp/src/plugins/dhcp/client.c:282

#7 0x00007fffb3ce0688 in dhcp_proxy_to_client_input (vm=0x7fffb474dcc0, node=0x7fffb4cbf900, from_frame=0x7fffb4c78000) at /data/workspace/vpp/src/plugins/dhcp/dhcp4_proxy_node.c:568

#8 0x00007ffff7ed7b44 in dispatch_node (vm=0x7fffb474dcc0, node=0x7fffb4cbf900, type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7fffb4c78000, last_time_stamp=4188694816810460)

at /data/workspace/vpp/src/vlib/main.c:1235

#9 0x00007ffff7ed82ff in dispatch_pending_node (vm=0x7fffb474dcc0, pending_frame_index=4, last_time_stamp=4188694816810460) at /data/workspace/vpp/src/vlib/main.c:1403

#10 0x00007ffff7ed9fc2 in vlib_main_or_worker_loop (vm=0x7fffb474dcc0, is_main=0) at /data/workspace/vpp/src/vlib/main.c:1862

#11 0x00007ffff7edaa29 in vlib_worker_loop (vm=0x7fffb474dcc0) at /data/workspace/vpp/src/vlib/main.c:1996

#12 0x00007ffff7f1bda5 in vlib_worker_thread_fn (arg=0x7fffbc2bb180) at /data/workspace/vpp/src/vlib/threads.c:1842

#13 0x00007ffff61876b8 in clib_calljmp () at /data/workspace/vpp/src/vppinfra/longjmp.S:123

#14 0x00007fef4f5fec10 in ?? ()

#15 0x00007ffff7f15bc0 in vlib_worker_thread_bootstrap_fn (arg=0x7fffbc2bb180) at /data/workspace/vpp/src/vlib/threads.c:584

I checked the newest code, this commit 77d9838282 introduced this bug.

in vlib_time_now under this case,

vm->thread_index is main thread, __os_thread_index is the worker thread.

Assignee

Jan Cavojsky

Reporter

Faicker Mo

Comments

jan_cavojsky (Mon, 21 Sep 2020 12:02:57 +0000): This bug is solved in master branch. Reason, why this patchset don't will be merged into branch 20.05 is in conversation on this link: https://gerrit.fd.io/r/c/vpp/+/28707
jan_cavojsky (Tue, 8 Sep 2020 11:35:01 +0000):
Hi,

I know, why vpp on my PC worked correctly.

I tried reproduce this bug on branch, where was this bug fixed, and therefore didn't possible reproduced it.

I successful reproduce this bug on branch 2005, where was not fixed.

I tested behavior of vpp after applying of commit for solving this bug.

After applying this commit vpp works correct without crashing.

I made "cherry-pick" of commit, that solved this problem into branch 2005, that you was using: https://gerrit.fd.io/r/c/vpp/+/28707

jan_cavojsky (Mon, 7 Sep 2020 07:44:46 +0000):
Hi,

I tried reproduced this bug, but not successfully.

My conditions was:

#56

My config file is little different:

unix {
 nodaemon
 log /var/log/vpp.log
 full-coredump
 cli-listen /run/vpp/cli.sock
}
cpu {
 main-core 1
 corelist-workers 2
}
dpdk {
 dev default {
 num-rx-queues 1
 num-tx-queues 1
num-rx-desc 256
 num-tx-desc 256
 }
dev 0000:00:08.0
dev 0000:00:09.0
uio-driver igb_uio
 no-tx-checksum-offload
}
plugins {
 path /opt/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
plugin default { disable }
 plugin dpdk_plugin.so { enable }
 plugin nat_plugin.so { enable }
 plugin dhcp_plugin.so { enable }
 plugin ping_plugin.so { enable }
}

used command for assign of ip address by command in vpp:

set interface state GigabitEthernet0/8/0 up
set dhcp client intfc GigabitEthernet0/8/0

Can you please give me more info about how reproduce this bug? How are your conditions?

Thanks.

JIRAUSER13812 (Thu, 27 Aug 2020 01:46:14 +0000): /etc/vpp/startup.conf

unix {

nodaemon

log /var/log/vpp.log

full-coredump

cli-listen /run/vpp/cli.sock

gid vpp

}
api-segment {

gid vpp

}
socksvr {

default

}
cpu {

main-core 1

corelist-workers 2

}
dpdk {

dev default {

num-rx-queues 1

num-tx-queues 1
num-rx-desc 256

num-tx-desc 256

}
dev 0000:00:02.0

dev 0000:00:03.0
uio-driver igb_uio

no-tx-checksum-offload

}
plugins {

path /home/xxx/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
 
plugin default { disable }

plugin dpdk_plugin.so { enable }

plugin nat_plugin.so { enable }

plugin dhcp_plugin.so { enable }

plugin ping_plugin.so { enable }
}

run vpp and vppctl to execute commands,

set interface state GigabitEthernet0/2/0 up

set dhcp client intfc GigabitEthernet0/2/0

make sure GigabitEthernet0/2/0 dhcp client is OK(so you may need a dhcp server)

jan_cavojsky (Wed, 26 Aug 2020 12:06:08 +0000):
Hi

please how I can reproduce this bug? Can you give me config file for vpp or also commands for linux if are necessary?

Thanks.

Original issue: https://jira.fd.io/browse/VPP-1924

vvalderrv · 2025-02-02T09:41:14Z

This bug is solved in master branch. Reason, why this patchset don't will be merged into branch 20.05 is in conversation on this link: https://gerrit.fd.io/r/c/vpp/+/28707

vvalderrv · 2025-02-02T09:41:16Z

Hi,
I know, why vpp on my PC worked correctly.
I tried reproduce this bug on branch, where was this bug fixed, and therefore didn't possible reproduced it.
I successful reproduce this bug on branch 2005, where was not fixed.
I tested behavior of vpp after applying of commit for solving this bug.
After applying this commit vpp works correct without crashing.
I made "cherry-pick" of commit, that solved this problem into branch 2005, that you was using: https://gerrit.fd.io/r/c/vpp/+/28707

vvalderrv · 2025-02-02T09:41:17Z

Hi,
I tried reproduced this bug, but not successfully.
My conditions was:

simulating on virtualbox
os version: Linux johny-VirtualBox 5.3.0-62-generic nat: nat44-ed cli bug fix #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
running of vpp on docker container
vpp version: v20.09-rc0~370-g67fe77854
dhcp server setting was directly in virtualbox by setting network card into NAT
after binding of network interface was used vfio-pci driver, uio driver was loaded in linux, but not used, I don't understand why

My config file is little different:

unix {
 nodaemon
 log /var/log/vpp.log
 full-coredump
 cli-listen /run/vpp/cli.sock
}
cpu {
 main-core 1
 corelist-workers 2
}
dpdk {
 dev default {
 num-rx-queues 1
 num-tx-queues 1
num-rx-desc 256
 num-tx-desc 256
 }
dev 0000:00:08.0
dev 0000:00:09.0
uio-driver igb_uio
 no-tx-checksum-offload
}
plugins {
 path /opt/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
plugin default { disable }
 plugin dpdk_plugin.so { enable }
 plugin nat_plugin.so { enable }
 plugin dhcp_plugin.so { enable }
 plugin ping_plugin.so { enable }
}

used command for assign of ip address by command in vpp:

set interface state GigabitEthernet0/8/0 up
set dhcp client intfc GigabitEthernet0/8/0

Can you please give me more info about how reproduce this bug? How are your conditions?
Thanks.

vvalderrv · 2025-02-02T09:41:19Z

/etc/vpp/startup.conf

unix {

nodaemon

log /var/log/vpp.log

full-coredump

cli-listen /run/vpp/cli.sock

gid vpp

}
api-segment {

gid vpp

}
socksvr {

default

}
cpu {

main-core 1

corelist-workers 2

}
dpdk {

dev default {

num-rx-queues 1

num-tx-queues 1
num-rx-desc 256

num-tx-desc 256

}
dev 0000:00:02.0

dev 0000:00:03.0
uio-driver igb_uio

no-tx-checksum-offload

}
plugins {

path /home/xxx/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
 
plugin default { disable }

plugin dpdk_plugin.so { enable }

plugin nat_plugin.so { enable }

plugin dhcp_plugin.so { enable }

plugin ping_plugin.so { enable }
}

run vpp and vppctl to execute commands,

set interface state GigabitEthernet0/2/0 up

set dhcp client intfc GigabitEthernet0/2/0

make sure GigabitEthernet0/2/0 dhcp client is OK(so you may need a dhcp server)

vvalderrv · 2025-02-02T09:41:20Z

Hi
please how I can reproduce this bug? Can you give me config file for vpp or also commands for linux if are necessary?
Thanks.

vvalderrv closed this as completed Feb 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VPP-1924] dhcp: dhcp client causes vpp crash when in multi threads/workers #3386

[VPP-1924] dhcp: dhcp client causes vpp crash when in multi threads/workers #3386

vvalderrv commented Feb 2, 2025

vvalderrv commented Feb 2, 2025

vvalderrv commented Feb 2, 2025

vvalderrv commented Feb 2, 2025

vvalderrv commented Feb 2, 2025

vvalderrv commented Feb 2, 2025