After doing edit-config on candidate target, the commit operation failed with segmentation fault

[hungta]

After doing update on candidate target, performed commit to apply all changes to running target, but the segmentation fault seen:

GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin/safplus_mgt...done.
[New LWP 21830]
[New LWP 21818]
[New LWP 21819]
[New LWP 21821]
[New LWP 21824]
[New LWP 21825]
[New LWP 21832]
[New LWP 21828]
[New LWP 21822]
[New LWP 21831]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/clovis/s7/target/x86_64-linux-gnu/bin/safplus_mgt --modpath=/home/clovis/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 dlq_insertAfter (newP=0x7fa1f0002ce0, nodeP=0x7fa1f00101f0) at dlq.c:496
496 ((dlq_hdrT *) newP)->next->prev = newP;
(gdb) f 0
#0 dlq_insertAfter (newP=0x7fa1f0002ce0, nodeP=0x7fa1f00101f0) at dlq.c:496
496 ((dlq_hdrT *) newP)->next->prev = newP;
(gdb) p ((dlq_hdrT *) newP)->next
$1 = (struct TAGdlq_hdrT *) 0x0
(gdb) p ((dlq_hdrT *) newP)
$2 = (dlq_hdrT ) 0x7fa1f0002ce0
(gdb) p ((dlq_hdrT)nodeP)->next
$3 = (struct TAGdlq_hdrT *) 0x0
(gdb) bt
#0 dlq_insertAfter (newP=0x7fa1f0002ce0, nodeP=0x7fa1f00101f0) at dlq.c:496
#1 0x00007fa2058ec7f5 in apply_write_val (editop=OP_EDITOP_COMMIT, scb=scb@entry=0x7fa1f0000970,
msg=msg@entry=0x7fa1f0003410, target=target@entry=0xb9dae0, parent=parent@entry=0xb372d0,
newval=newval@entry=0x7fa1f0002ce0, curval=curval@entry=0x7fa1f00101f0,
done=done@entry=0x7fa1f48ee7a7) at agt_val.c:1754
#2 0x00007fa2058ece6d in invoke_cpxval_cb (done=true, curparent=0xb372d0, curval=0x7fa1f00101f0,
newval=0x7fa1f0002ce0, target=0xb9dae0, msg=0x7fa1f0003410, scb=0x7fa1f0000970,
editop=OP_EDITOP_COMMIT, cbtyp=AGT_CB_APPLY) at agt_val.c:2361
#3 invoke_btype_cb (cbtyp=cbtyp@entry=AGT_CB_APPLY, editop=,
scb=scb@entry=0x7fa1f0000970, msg=msg@entry=0x7fa1f0003410, target=target@entry=0xb9dae0,
newval=newval@entry=0x7fa1f0002ce0, curval=0x7fa1e4002190, curparent=curparent@entry=0xb372d0)
at agt_val.c:2540
#4 0x00007fa2058ecf0a in invoke_cpxval_cb (done=false, curparent=0xbbb660, curval=0xb372d0,
newval=0x6, target=0xb9dae0, msg=0x7fa1f0003410, scb=0x7fa1f0000970, editop=,
cbtyp=12302464) at agt_val.c:2405
#5 invoke_btype_cb (cbtyp=cbtyp@entry=AGT_CB_APPLY, editop=editop@entry=OP_EDITOP_COMMIT,
scb=scb@entry=0x7fa1f0000970, msg=msg@entry=0x7fa1f0003410, target=target@entry=0xb9dae0,
newval=newval@entry=0xbbb660, curval=curval@entry=0xb372d0, curparent=0xbbb660,
curparent@entry=0xb372d0) at agt_val.c:2540
#6 0x00007fa2058edc46 in handle_callback (cbtyp=cbtyp@entry=AGT_CB_APPLY,
editop=editop@entry=OP_EDITOP_COMMIT, scb=scb@entry=0x7fa1f0000970, msg=msg@entry=0x7fa1f0003410,
target=target@entry=0xb9dae0, newval=0xbbb660, curval=0xb372d0, curparent=0xb372d0)
at agt_val.c:3192
#7 0x00007fa2058efaf8 in agt_val_apply_commit (scb=scb@entry=0x7fa1f0000970,
msg=msg@entry=0x7fa1f0003410, source=source@entry=0xb68f90, target=target@entry=0xb9dae0,
save_nvstore=save_nvstore@entry=true) at agt_val.c:5870
#8 0x00007fa2058d6ee9 in commit_invoke (scb=0x7fa1f0000970, msg=0x7fa1f0003410,
methnode=0x7fa1f48eeb50) at agt_ncx.c:2459
#9 0x00007fa2058dd786 in agt_rpc_dispatch (scb=0x7fa1f0000970, top=) at agt_rpc.c:1627
#10 0x00007fa2058e47b4 in agt_top_dispatch_msg (ppscb=ppscb@entry=0x7fa1f48eed18) at agt_top.c:154
#11 0x00007fa2058e08d6 in agt_ses_process_first_ready () at agt_ses.c:884

Vladimir Vassilev and everyone, could you take a look at the issue? Thanks.

[vlvassilev]

Hello,

I can see right away that you are using custom version of YUMA where you have "((dlq_hdrT *) newP)->next->prev = newP;" at line 496. None of the versions known to me has that assignment at that line.

I can only help if I have exact version of the source code (2) and detailed description of how the problem can be produced and if possible an automated test case like the test cases part of yuma123/netconf/test/netconfd (3).

http://yuma123.org/wiki/index.php/Reporting_issues_and_debugging#Reporting_issues

Vladimir

[hungta]

Hello Vladimir,

Your quick response is very appreciated.

Yes, we use the custom of yuma123 to do our specific customization but we don't change a lot, especially the code causing the above issue. The codes below is excerpted from yours at master branch, they're not different from ours (dlq.c)

void dlq_insertAfter (void *newP, void *nodeP)
{
#ifdef CPP_ICHK
if (nodeP==NULL || newP==NULL)
{
err_msg(ERR_INTERNAL_PTR);
return;
}
if (nodeP==newP)
{
err_msg(ERR_INTERNAL_VAL);
return;
}
#endif

ENTER_CRIT;    
((dlq_hdrT *) newP)->hdr_typ            = DLQ_DATA_NODE;
((dlq_hdrT *) newP)->prev       = nodeP;
((dlq_hdrT *) newP)->next       = ((dlq_hdrT *) nodeP)->next;
((dlq_hdrT *) newP)->next->prev  = newP;
((dlq_hdrT *) newP)->prev->next  = newP;
EXIT_CRIT;

} /* END function dlq_insertAfter */

The issue here is actually not at ((dlq_hdrT *) newP)->next->prev = newP; But the actual issue is that the nodeP->next is NULL at the previous line: ((dlq_hdrT *) newP)->next = ((dlq_hdrT *) nodeP)->next;
Because ((dlq_hdrT *) nodeP)->next is NULL which causes segfault at line 496: ((dlq_hdrT *) newP)->next->prev = newP;

The exact version of the source code (2) is yuma version 2.2.5

How to reproduce:

1. Edit the netconfd.conf to set the initial target is candidate and set log level to debug4
2. Start netconfd
3. We created an app including a yang file and it was generated to C++ source code and its structure was added so that netconf server can be communicate to. Start the app
4. We can retrieve information from our app by get-config
5. Send edit-config which updates the data defined by the yang file on candidate. This was successful (an ok xml returned)
6. We'd like these changes should be also applied to running, so we send commit and the issue seen.

We don't know if this test is a part of your automated test

Thank you and regards,
Hung



[vlvassilev]

The reported backtrace does not reveal details about the commit transaction causing the problem. In addition the sourcefiles are not only outdated but modified with custom changes. You might be hitting a problem that was fixed long ago. I just released the 2.10-1 version of yuma123 (packaged and available in Debian unstable) and If you can reproduce the problem with that version it would make it possible for me to help. As for a test case reproducing the problem I can recommend looking into https://github.com/vlvassilev/yuma123/blob/master/netconf/test/netconfd/edit-config/session.replace-leaf.yangcli.py for an example of how scripted session can be reproduced (in the same directory you can find ncclient and litenc based session scripts). Then any problem you detect can be submitted with a simple automated test case that will be added to the test suite.

[hungta]

Thanks a lot for your help.
I'll run the test with your suggested yuma123 version (2.10-1) and let you know the result.

Btw, is the version 2.10-1 of yuma123 available on sourcefore?

Best regards,
Hung

[vlvassilev]

Yes both https://github.com/vlvassilev/yuma123 and https://sourceforge.net/projects/yuma123 have identical master branch git repository copies. You can download also 2.10 release tar either form sourceforge or yuma123_2.10.orig.tar.gz from a debian repository.