Ring returned in riak_core_ring:all_owners(Ring) contains references to removed nodes

[asyncmind0]

Partitions = [P || {P, '[email protected]'} <- riak_core_ring:all_owners(Ring)].

following instructions for repair in here https://www.tiot.jp/riak-docs/riak/kv/2.2.6/using/repair-recovery/repairs/#repairing-partitions

I found that in my cluster there is a lot of references to old nodes
My cluster only has the following valid nodes

dev-riak1.example.com
dev-riak2.example.com
dev-riak3.example.com
dev-riak4.example.com
dev-riak5.example.com

But the Ring contains

[{'[email protected]',{793,63760267758}},
             {'[email protected]',{2657,63759791951}},
             {'[email protected]',{2492,63759791951}},
             {'[email protected]',{2257,63759785044}},
             {'[email protected]',{1652,63759445321}},
             {'[email protected]',{2001,63760226110}},
             {'[email protected]',{4652,63673217155}},
             {'[email protected]',{7409,63685197199}},
             {'[email protected]',{3152,63721849744}},
             {'[email protected]',{711,63683938533}},
             {'[email protected]',{1503,63721056942}},
             {'[email protected]',{4712,63721084448}},
             {'[email protected]',{5190,63721701233}},
             {'[email protected]',{1700,63660968738}},
             {'[email protected]',{6339,63673205425}},
             {'[email protected]',{2139,63673186576}},
             {'[email protected]',{2108,63676347325}},
             {'[email protected]',{1482,63721608275}},
             {'[email protected]',{745,63683941378}}],
            {64,
             [{0,'[email protected]'},
              {22835963083295358096932575511191922182123945984,
               '[email protected]'},
              {45671926166590716193865151022383844364247891968,
               '[email protected]'},
              {68507889249886074290797726533575766546371837952,
               '[email protected]'},
              {91343852333181432387730302044767688728495783936,
               '[email protected]'},
              {114179815416476790484662877555959610910619729920,
               '[email protected]'},
              {137015778499772148581595453067151533092743675904,
               '[email protected]'},
              {159851741583067506678528028578343455274867621888,
               '[email protected]'},
              {182687704666362864775460604089535377456991567872,
               '[email protected]'},
              {205523667749658222872393179600727299639115513856,
               '[email protected]'},
              {228359630832953580969325755111919221821239459840,
               '[email protected]'},
              {251195593916248939066258330623111144003363405824,
               '[email protected]'},
              {274031556999544297163190906134303066185487351808,
               '[email protected]'},
              {296867520082839655260123481645494988367611297792,
               '[email protected]'},
              {319703483166135013357056057156686910549735243776,
               '[email protected]'},
              {342539446249430371453988632667878832731859189760,
               '[email protected]'},
              {365375409332725729550921208179070754913983135744,
               '[email protected]'},
              {388211372416021087647853783690262677096107081728,
               '[email protected]'},
              {411047335499316445744786359201454599278231027712,
               '[email protected]'},
              {433883298582611803841718934712646521460354973696,
               '[email protected]'},
              {456719261665907161938651510223838443642478919680,
               '[email protected]'},
              {479555224749202520035584085735030365824602865664,...},
              {...}|...]},
            {dict,6,16,16,8,80,48,
                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                  {{[],[],[],[],
                    [[riak_core_ring_tainted|{meta_entry,true,63760267758}]],
                    [[symbolic_clustername|
                      {meta_entry,"dev-riak",63739630764}]],
                    [],
                    [[riak_repl_ring|{meta_entry,{dict,...},63759488652}]],
                    [[{bucket,<<"_yz_"...>>}|{meta_entry,[...],...}]],
                    [],[],
                    [[{...}|...]],
                    [],[],...}}},
            {'[email protected]',{1479,946122,353764}},
            [],
            [{'[email protected]',{invalid,[{'[email protected]',{2209,
                                                                                              63759791951}},
                                                          {'[email protected]',{1,63759785044}},
                                                          {'[email protected]',{1,63720652846}}],
                                                         []}},
             {'[email protected]',{valid,[{'[email protected]',{2206,
                                                                                            63759444688}},
                                                        {'[email protected]',{1,63720692674}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true,false]},
                                                                               {{riak_core,security},[true,false]},
                                                                               {{riak_core,staged_joins},[true,false]},
                                                                               {{riak_core,vnode_routing},[proxy|...]},
                                                                               {{riak_kv,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'[email protected]',{valid,[{'[email protected]',{2069,
                                                                                            63759444717}},
                                                        {'[email protected]',{1,63720709605}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true,false]},
                                                                               {{riak_core,security},[true,false]},
                                                                               {{riak_core,staged_joins},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'[email protected]',{valid,[{'[email protected]',{1422,
                                                                                            63759445321}},
                                                        {'[email protected]',{1,63721340231}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true,false]},
                                                                               {{riak_core,security},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'[email protected]',{valid,[{'[email protected]',{1802,
                                                                                            63760226110}},
                                                        {'[email protected]',{1,63721340231}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'[email protected]',{valid,[{'[email protected]',{681,
                                                                                            63760267758}},
                                                        {'[email protected]',{1,63759791951}},
                                                        {'[email protected]',{3,63754251063}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'[email protected]',{invalid,[{'[email protected]',{3854,
                                                                                  63673217155}},
                                                    {'[email protected]',{5,63673217156}},
                                                    {'[email protected]',{1,63660953533}}],
                                                   []}},
             {'[email protected]',{invalid,[{'[email protected]',{1,
                                                                                  63660961602}},
                                                    {'[email protected]',{6209,63685197199}},
                                                    {'[email protected]',{6,63685197207}},
                                                    {'[email protected]',{1,63683442945}},
                                                    {'[email protected]',{6,63660932914}}],
                                                   []}},
             {'[email protected]',{invalid,[{'[email protected]',{2,
                                                                                         63721849745}},
                                                     {'[email protected]',{4,63683893655}},
                                                     {'[email protected]',{2059,63721849744}}],
                                                    []}},
             {'[email protected]',{invalid,[{'[email protected]',{3,
                                                                                   63683938543}},
                                                     {'[email protected]',{525,63683938533}}],
                                                    []}},
             {'[email protected]',{invalid,[{'[email protected]',{1,
                                                                                   63673234572}},
                                                     {'[email protected]',{3,63721056943}},
                                                     {'[email protected]',{1263,...}}],
                                                    []}},
             {'[email protected]',{invalid,[{'[email protected]',{2,
                                                                                  63660967356}},
                                                    {'[email protected]',{2,...}},
                                                    {'[email protected]',{...}},
                                                    {'[email protected]',...},
                                                    {...}],
                                                   []}},
             {'[email protected]',{invalid,[{'[email protected]',{3,...}},
                                                    {'[email protected]',{...}},
                                                    {'[email protected]',...},
                                                    {...}],
                                                   []}},
             {'[email protected]',{invalid,[{'[email protected]',{...}},
                                                    {'[email protected]',...},
                                                    {...}],
                                                   []}},
             {'[email protected]',{invalid,[{'[email protected]',...},
                                                    {...}|...],
                                                   []}},
             {'[email protected]',{invalid,[{...}|...],[]}},
             {'[email protected]',{invalid,[...],...}},
             {'[email protected]',{invalid,...}},
             {'[email protected]',{...}}],
            '[email protected]',
            [{'[email protected]',[{'[email protected]',{2656,
                                                                                     63759585618}},
                                                 {'[email protected]',{2477,63759444688}},
                                                 {'[email protected]',{2256,63759444717}},
                                                 {'[email protected]',{1652,63759445321}},
                                                 {'[email protected]',{1960,63759445973}},
                                                 {'[email protected]',{656,63754251063}},
                                                 {'[email protected]',{4652,63673217155}},
                                                 {'[email protected]',{7409,63685197199}},
                                                 {'[email protected]',{3152,63721849744}},
                                                 {'[email protected]',{711,63683938533}},
                                                 {'[email protected]',{1503,63721056942}},
                                                 {'[email protected]',{4712,63721084448}},
                                                 {'[email protected]',{5190,...}},
                                                 {'[email protected]',{...}},
                                                 {'[email protected]',...},
                                                 {...}|...]},
             {'[email protected]',[{'[email protected]',{2657,
                                                                                     63759791951}},
                                                 {'[email protected]',{2492,63759791951}},
                                                 {'[email protected]',{2257,63759785044}},
                                                 {'[email protected]',{1652,63759445321}},
                                                 {'[email protected]',{2001,63760226110}},
                                                 {'[email protected]',{789,63760267758}},
                                                 {'[email protected]',{4652,63673217155}},
                                                 {'[email protected]',{7409,63685197199}},
                                                 {'[email protected]',{3152,63721849744}},
                                                 {'[email protected]',{711,63683938533}},
                                                 {'[email protected]',{1503,63721056942}},
                                                 {'[email protected]',{4712,...}},
                                                 {'[email protected]',{...}},
                                                 {'[email protected]',...},
                                                 {...}|...]},
             {'[email protected]',[{'[email protected]',{2657,
                                                                                     63759791951}},
                                                 {'[email protected]',{2492,63759791951}},
                                                 {'[email protected]',{2257,63759785044}},
                                                 {'[email protected]',{1652,63759445321}},
                                                 {'[email protected]',{2001,63760226110}},
                                                 {'[email protected]',{789,63760267758}},
                                                 {'[email protected]',{4652,63673217155}},
                                                 {'[email protected]',{7409,63685197199}},
                                                 {'[email protected]',{3152,63721849744}},
                                                 {'[email protected]',{711,63683938533}},
                                                 {'[email protected]',{1503,...}},
                                                 {'[email protected]',{...}},
                                                 {'[email protected]',...},
                                                 {...}|...]},
             {'[email protected]',[{'[email protected]',{2657,
                                                                                     63759791951}},
                                                 {'[email protected]',{2492,63759791951}},
                                                 {'[email protected]',{2257,63759785044}},
                                                 {'[email protected]',{1652,63759445321}},
                                                 {'[email protected]',{2001,63760226110}},
                                                 {'[email protected]',{789,63760267758}},
                                                 {'[email protected]',{4652,63673217155}},
                                                 {'[email protected]',{7409,63685197199}},
                                                 {'[email protected]',{3152,63721849744}},
                                                 {'[email protected]',{711,...}},
                                                 {'[email protected]',{...}},
                                                 {'[email protected]',...},
                                                 {...}|...]},
             {'[email protected]',[{'[email protected]',{2657,
                                                                                     63759791951}},
                                                 {'[email protected]',{2492,63759791951}},
                                                 {'[email protected]',{2257,63759785044}},
                                                 {'[email protected]',{1652,63759445321}},
                                                 {'[email protected]',{2001,63760226110}},
                                                 {'[email protected]',{789,63760267758}},
                                                 {'[email protected]',{4652,63673217155}},
                                                 {'[email protected]',{7409,63685197199}},
                                                 {'[email protected]',{3152,...}},
                                                 {'[email protected]',{...}},
                                                 {'[email protected]',...},
                                                 {...}|...]},
             {'[email protected]',[{'[email protected]',{2657,
                                                                                     63759791951}},
                                                 {'[email protected]',{2492,63759791951}},
                                                 {'[email protected]',{2257,63759785044}},
                                                 {'[email protected]',{1652,63759445321}},
                                                 {'[email protected]',{2001,63760226110}},
                                                 {'[email protected]',{789,63760267758}},
                                                 {'[email protected]',{4652,63673217155}},
                                                 {'[email protected]',{7409,...}},
                                                 {'[email protected]',{...}},
                                                 {'[email protected]',...},
                                                 {...}|...]},
             {'[email protected]',[{'[email protected]',{4652,
                                                                         63673217155}},
                                           {'[email protected]',{5501,63673217065}},
                                           {'[email protected]',{56,63673217065}},
                                           {'[email protected]',{3325,63673217065}},
                                           {'[email protected]',{3763,63673217065}},
                                           {'[email protected]',{1700,63660968738}},
                                           {'[email protected]',{6339,...}},
                                           {'[email protected]',{...}},
                                           {'[email protected]',...},
                                           {...}|...]},
             {'[email protected]',[{'[email protected]',{4652,
                                                                         63673217155}},
                                           {'[email protected]',{7409,63685197199}},
                                           {'[email protected]',{1742,63685177941}},
                                           {'[email protected]',{711,63683938533}},
                                           {'[email protected]',{940,63685175606}},
                                           {'[email protected]',{4137,...}},
                                           {'[email protected]',{...}},
                                           {'[email protected]',...},
                                           {...}|...]},
             {'[email protected]',[{'[email protected]',{1072,
                                                                                63721811710}},
                                            {'[email protected]',{585,63721849665}},
                                            {'[email protected]',{498,63721806643}},
                                            {'[email protected]',{329,63721823494}},
                                            {'[email protected]',{422,...}},
                                            {'[email protected]',{...}},
                                            {'[email protected]',...},
                                            {...}|...]},
             {'[email protected]',[{'[email protected]',{4652,
                                                                          63673217155}},
                                            {'[email protected]',{6756,63683936507}},
                                            {'[email protected]',{1259,63683894630}},
                                            {'[email protected]',{711,...}},
                                            {'[email protected]',{...}},
                                            {'[email protected]',...},
                                            {...}|...]},
             {'[email protected]',[{'[email protected]',{784,
                                                                                63720908410}},
                                            {'[email protected]',{187,63720952694}},
                                            {'[email protected]',{103,...}},
                                            {'[email protected]',{...}},
                                            {'[email protected]',...},
                                            {...}|...]},
             {'[email protected]',[{'[email protected]',{784,
                                                                               63720908410}},
                                           {'[email protected]',{187,...}},
                                           {'[email protected]',{...}},
                                           {'[email protected]',...},
                                           {...}|...]},
             {'[email protected]',[{'[email protected]',{910,...}},
                                           {'[email protected]',{...}},
                                           {'[email protected]',...},
                                           {...}|...]},
             {'[email protected]',[{'[email protected]',{...}},
                                           {'[email protected]',...},
                                           {...}|...]},
             {'[email protected]',[{'[email protected]',...},
                                           {...}|...]},
             {'[email protected]',[{...}|...]},
             {'[email protected]',[...]},
             {'[email protected]',...},
             {...}],
            [{'[email protected]',{43,63759785047}},
             {'[email protected]',{528,63721849745}},
             {'[email protected]',{336,63685197207}},
             {'[email protected]',{259,63665215960}},
             {'[email protected]',{120,63660958341}}]}

[martinsumner]

Apologies that this wan't answered before.

As far as I can see all the extra nodes are marked as invalid and so will play no actual role in the ring. I've just double-checked in a test cluster, and it looks like nodes that have left will remain referenced in the ring as invalid.

The precise reasons why this tombstone state is never garbage collected, I don't know. I can't find this documented anywhere in the code, or in the riak_core wiki.

I don't believe there currently exists a function to clean out this history. It may be relatively simple to extend riak_core to allow this to happen, but unless the history is causing an active problem, I'm not sure this would ever be prioritised.