chorderl:fix_fingers_all() global deadlock

[brucify]

Deadlock when everyone is fixing fingers and the same time. Due to use of gen_server:call at find_successor and find_predecessor

8> chorderl:node_status().
<0.158.0>: (query_successor) Successor is: {5,<0.159.0>}
 ===================== {4,<0.158.0>} ======================
|  name: 'chorderl_\203a\004'
|  id: 4
|                                                          |
|  Pred: {3,<0.157.0>} --->
|                      {4,<0.158.0>}
|                                 ---> Succ: {5,<0.159.0>}
|  Last 8 Fingers:                                         |
| {0,{0,<0.123.0>}}
| {6,{0,<0.123.0>}}
| {5,{5,<0.159.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.159.0>: (query_successor) Successor is: {6,<0.160.0>}
 ===================== {5,<0.159.0>} ======================
|  name: 'chorderl_\203a\005'
|  id: 5
|                                                          |
|  Pred: {4,<0.158.0>} --->
|                      {5,<0.159.0>}
|                                 ---> Succ: {6,<0.160.0>}
|  Last 8 Fingers:                                         |
| {1,{0,<0.123.0>}}
| {7,{0,<0.123.0>}}
| {6,{6,<0.160.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.160.0>: (query_successor) Successor is: {7,<0.161.0>}
 ===================== {6,<0.160.0>} ======================
|  name: 'chorderl_\203a\006'
|  id: 6
|                                                          |
|  Pred: {5,<0.159.0>} --->
|                      {6,<0.160.0>}
|                                 ---> Succ: {7,<0.161.0>}
|  Last 8 Fingers:                                         |
| {2,{0,<0.123.0>}}
| {0,{0,<0.123.0>}}
| {7,{7,<0.161.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.161.0>: (query_successor) Successor is: {0,<0.123.0>}
 ===================== {7,<0.161.0>} ======================
|  name: 'chorderl_\203a\007'
|  id: 7
|                                                          |
|  Pred: {6,<0.160.0>} --->
|                      {7,<0.161.0>}
|                                 ---> Succ: {0,<0.123.0>}
|  Last 8 Fingers:                                         |
| {3,{0,<0.123.0>}}
| {1,{0,<0.123.0>}}
| {0,{0,<0.123.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.123.0>: (query_successor) Successor is: {1,<0.155.0>}
 ===================== {0,<0.123.0>} ======================
|  name: 'chorderl_\203a\000'
|  id: 0
|                                                          |
|  Pred: {7,<0.161.0>} --->
|                      {0,<0.123.0>}
|                                 ---> Succ: {1,<0.155.0>}
|  Last 8 Fingers:                                         |
| {4,{0,<0.123.0>}}
| {2,{0,<0.123.0>}}
| {1,{1,<0.155.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.155.0>: (query_successor) Successor is: {2,<0.156.0>}
 ===================== {1,<0.155.0>} ======================
|  name: 'chorderl_\203a\001'
|  id: 1
|                                                          |
|  Pred: {0,<0.123.0>} --->
|                      {1,<0.155.0>}
|                                 ---> Succ: {2,<0.156.0>}
|  Last 8 Fingers:                                         |
| {5,{0,<0.123.0>}}
| {3,{0,<0.123.0>}}
| {2,{2,<0.156.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.156.0>: (query_successor) Successor is: {3,<0.157.0>}
 ===================== {2,<0.156.0>} ======================
|  name: 'chorderl_\203a\002'
|  id: 2
|                                                          |
|  Pred: {1,<0.155.0>} --->
|                      {2,<0.156.0>}
|                                 ---> Succ: {3,<0.157.0>}
|  Last 8 Fingers:                                         |
| {6,{0,<0.123.0>}}
| {4,{0,<0.123.0>}}
| {3,{3,<0.157.0>}}
|  Total: 3
|                                                          |
 ==========================================================


<0.157.0>: (query_successor) Successor is: {4,<0.158.0>}
 ===================== {3,<0.157.0>} ======================
|  name: 'chorderl_\203a\003'
|  id: 3
|                                                          |
|  Pred: {2,<0.156.0>} --->
|                      {3,<0.157.0>}
|                                 ---> Succ: {4,<0.158.0>}
|  Last 8 Fingers:                                         |
| {7,{0,<0.123.0>}}
| {5,{0,<0.123.0>}}
| {4,{4,<0.158.0>}}
|  Total: 3
|                                                          |
 ==========================================================


[ok,ok,ok,ok,ok,ok,ok,ok]
9> chorderl:fix_fingers_all().
[ok,ok,ok,ok,ok,ok,ok,ok]
<0.158.0>: (fix_fingers) Fixing index 2: {6,{0,<0.123.0>}}
<0.159.0>: (fix_fingers) Fixing index 2: {7,{0,<0.123.0>}}
<0.160.0>: (fix_fingers) Fixing index 2: {0,{0,<0.123.0>}}
<0.161.0>: (fix_fingers) Fixing index 2: {1,{0,<0.123.0>}}
<0.123.0>: (fix_fingers) Fixing index 2: {2,{0,<0.123.0>}}
<0.155.0>: (fix_fingers) Fixing index 2: {3,{0,<0.123.0>}}
<0.156.0>: (fix_fingers) Fixing index 2: {4,{0,<0.123.0>}}
<0.157.0>: (fix_fingers) Fixing index 2: {5,{0,<0.123.0>}}
<0.158.0>: (find_predecessor) Finding predecessor of 6 locally
<0.159.0>: (find_predecessor) Finding predecessor of 7 locally
<0.160.0>: (find_predecessor) Finding predecessor of 0 locally
<0.161.0>: (find_predecessor) Finding predecessor of 1 locally
<0.123.0>: (find_predecessor) Finding predecessor of 2 locally
<0.155.0>: (find_predecessor) Finding predecessor of 3 locally
<0.156.0>: (find_predecessor) Finding predecessor of 4 locally
<0.157.0>: (find_predecessor) Finding predecessor of 5 locally
<0.158.0>: (find_predecessor_remote) Finding predecessor of 6 remotely at {5,
                                                                           <0.159.0>}
<0.159.0>: (find_predecessor_remote) Finding predecessor of 7 remotely at {6,
                                                                           <0.160.0>}
<0.160.0>: (find_predecessor_remote) Finding predecessor of 0 remotely at {7,
                                                                           <0.161.0>}
<0.161.0>: (find_predecessor_remote) Finding predecessor of 1 remotely at {0,
                                                                           <0.123.0>}
<0.123.0>: (find_predecessor_remote) Finding predecessor of 2 remotely at {1,
                                                                           <0.155.0>}
<0.155.0>: (find_predecessor_remote) Finding predecessor of 3 remotely at {2,
                                                                           <0.156.0>}
<0.156.0>: (find_predecessor_remote) Finding predecessor of 4 remotely at {3,
                                                                           <0.157.0>}
<0.157.0>: (find_predecessor_remote) Finding predecessor of 5 remotely at {4,
                                                                           <0.158.0>}
10> 22:43:17.990 [error] gen_server 'chorderl_�a^E' terminated with reason: {timeout,{gen_server,call,[<0.160.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.991 [error] gen_server 'chorderl_�a^D' terminated with reason: {timeout,{gen_server,call,[<0.159.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.991 [error] gen_server 'chorderl_�a^F' terminated with reason: {timeout,{gen_server,call,[<0.161.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.991 [error] gen_server 'chorderl_�a^G' terminated with reason: {timeout,{gen_server,call,[<0.123.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.991 [error] gen_server 'chorderl_�a^@' terminated with reason: {timeout,{gen_server,call,[<0.155.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.991 [error] gen_server 'chorderl_�a^A' terminated with reason: {timeout,{gen_server,call,[<0.156.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.992 [error] gen_server 'chorderl_�a^B' terminated with reason: {timeout,{gen_server,call,[<0.157.0>,{query_successor}]}} in gen_server:call/2 line 204
22:43:17.992 [error] gen_server 'chorderl_�a^C' terminated with reason: {timeout,{gen_server,call,[<0.158.0>,{query_successor}]}} in gen_server:call/2 line 204
10> 22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\005' with 8 neighbours exited with reason: {timeout,{gen_server,call,[<0.160.0>,{query_successor}]}} in gen_server:terminate/7 line 826
22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\007' with 0 neighbours exited with reason: {timeout,{gen_server,call,[<0.123.0>,{query_successor}]}} in gen_server:terminate/7 line 826
22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\000' with 0 neighbours exited with reason: {timeout,{gen_server,call,[<0.155.0>,{query_successor}]}} in gen_server:terminate/7 line 826
22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\006' with 5 neighbours exited with reason: {timeout,{gen_server,call,[<0.161.0>,{query_successor}]}} in gen_server:terminate/7 line 826
22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\001' with 0 neighbours exited with reason: {timeout,{gen_server,call,[<0.156.0>,{query_successor}]}} in gen_server:terminate/7 line 826
22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\002' with 0 neighbours exited with reason: {timeout,{gen_server,call,[<0.157.0>,{query_successor}]}} in gen_server:terminate/7 line 826
22:43:17.998 [error] CRASH REPORT Process 'chorderl_\203a\003' with 0 neighbours exited with reason: {timeout,{gen_server,call,[<0.158.0>,{query_successor}]}} in gen_server:terminate/7 line 826
** exception error: {timeout,{gen_server,call,[<0.160.0>,{query_successor}]}}
10>

[brucify]

Solution:

Make find_successor async for finger updates:

handle_call({find_successor, NewNode}, _From, LoopData) ->