Regent: memory leak

[mariodirenzo]

My solver that is written in Regent tends to leak a lot of memory during its execution.
For instance, I've seen runs that leak 3.5GB of memory per node every 5 minutes.
Following an initial inspection of the issue done with @lightsighter and @magnatelee, the problem should be related to an accumulation of futures that are not deleted during the run.
@magnatelee and maybe @lightsighter should still have some logs related to this issue.

[lightsighter]

I'll be monitoring it, but assigning to @magnatelee for now.

[aheirich]

this is preventing soleil-x from running to completion. After 4-5 hours of running the hit_to_openchannel test case on 5 nodes of Piz Daint the program falls over with out of memory error.

I instrumented legion_c.cc to trace the creation and deletion of Futures. Here is the output for a typical Future

~/PSAAP/soleil-x-master/src> grep 0x2ab828693f40 run.log
nid04339 legion_index_launcher_execute_reduction new future 0x2ab828693f40
nid04339 legion_task_get_future new future 0x2ab828693f40
nid04339 legion_future_destroy deletes future 0x2ab828693f40
nid04339 legion_future_destroy deletes future 0x2ab828693f40
nid04339 legion_index_launcher_execute_reduction new future 0x2ab828693f40
nid04339 legion_task_get_future new future 0x2ab828693f40
nid04339 legion_future_destroy deletes future 0x2ab828693f40
nid04339 legion_future_destroy deletes future 0x2ab828693f40
nid04339 legion_future_from_untyped_pointer new future 0x2ab828693f40
nid04339 legion_task_get_future new future 0x2ab828693f40
nid04339 legion_future_destroy deletes future 0x2ab828693f40

Notice there is an extra new that is not matched by a delete.

After running 500 steps of soleil-x (master branch) there were 19103 Futures that followed this pattern. That seems like an excessive number.

[elliottslaughter]

Current best guess is that the culprit is an expression like f() + 1 where the future from f() is not being destroyed because it's getting used as an input to the binary expression. This should be fixable within the normalizer, which will turn this into:

var x = f()
x + 1

And then we'll follow the usual rules for vars which should already be sufficient to catch this.

Having said that, it would really help to have a more information about where this happens in the original source code, this is a complete guess at the moment in terms of where the problem may be.

Leaving @magnatelee assigned to fix the normalizer issue.

[elliottslaughter]

Attaching a reproducer:

test_future_leak.rg.txt

CC_FLAGS=-DLEGION_GC ./install.py --debug
./regent.py test_future_leak.rg -level legion_gc=2 -logfile gc_%.log
../tools/legion_gc.py -l gc_0.log

Reports:

LEAK SUMMARY
  LEAKED FUTURES: 100

[magnatelee]

All future leaks that I was able to identify have been fixed as of 456853e. No future leaks were found after running 50 iterations of Soleil-X. The fix has been pushed to both master and control_replication. I'll let @aheirich confirm the fix and close this issue.

[lightsighter]

I think it would be good if @mariodirenzo can also confirm the fix in the HTR.

[mariodirenzo]

Is 456853e compatible with the parallelizer in parallelizer-dcr-new?
I'll be happy to check that but it will take me about one week.
I could try it on sapling earlier but I'd like to avoid modifying that version of the runtime and of the code until #712 is fixed

[magnatelee]

The fix is now available in parallelizer-dcr-new as well, so you can pull and try again. I'll also try to test HTR on sapling.

[aheirich]

I'm afraid soleil-x is still failing after less than five hours of running. Out of memory.

[magnatelee]

What's the error mode? Can you run it with Legion GC enabled and send me a log file? At least the Soleil-X in master was not leaking any futures. Which branch of Soleil-X are you using?

To enable Legion GC, you need to re-install Regent with CC_FLAGS=-DLEGION_GC and run it with -level legion_gc=2. After the program completes, you will be able to find ?.log files. To get complete logs, the program shouldn't crash.

[aheirich]

-rw-r--r-- 1 aheirich d108 194391853 Feb 10 21:19 gc_0.log
-rw-r--r-- 1 aheirich d108 46924862 Feb 10 21:19 gc_4.log
-rw-r--r-- 1 aheirich d108 48753543 Feb 10 21:19 gc_1.log
-rw-r--r-- 1 aheirich d108 47509165 Feb 10 21:19 gc_2.log
-rw-r--r-- 1 aheirich d108 47351561 Feb 10 21:19 gc_3.log
GC.tar.gz

[magnatelee]

Legion GC founds a cycle in Alan's case, so I suspect this is a runtime bug. Assigning @lightsighter to this issue as well to have him take a look at the issue.

[lightsighter]

While I agree that there is a cycle in the reference counting graph, it is for an index space and a partition, not a future and is therefore highly unlikely to be the source of the persistent memory leak. I don't see any cyclic references on future objects.

[lightsighter]

Did we confirm that the futures are not being leaked any longer?

[magnatelee]

Here is the output from Legion GC:

LEAK SUMMARY
  LEAKED FUTURES: 256
  Leaked Future Maps: 0
  LEAKED CONSTRAINTS: 1759
  LEAKED MANAGERS: 16
  Pinned Managers: 0
  LEAKED VIEWS: 29
  LEAKED EQUIVALENCE SETS: 1068
  LEAKED INDEX SPACES: 3127
  LEAKED INDEX PARTITIONS: 741
  LEAKED FIELD SPACES: 445
  LEAKED REGIONS: 800
  LEAKED PARTITIONS: 491

Apparently, there are still some futures that are leaked, but leaks on other handles, such as equivalence sets, seem more severe.

[lightsighter]

If the run crashed then those numbers are meaningless. Legion GC only soundly reports leaks if the runtime shuts down correctly, otherwise you're just recording how many live objects there were when the runtime died.

[lightsighter]

The reference cycle occurs when a partition is made with a color space that is identical to its parent index space. I've pushed a fix with b188ec6. I guarantee you that it is not the source of the leak. Assigning back to @magnatelee and @aheirich to investigate further.

[aheirich]

The run did not crash, it exited cleanly. So the numbers are not meaningless. Previously @manopapad and myself confirmed the problem is with a legion_future_from_untyped_pointer that is not matched by a delete. Confirming that the problem still exists.

[lightsighter]

Are you deleting all your regions at the end of the run? Leaked regions are guaranteed to cause leaked index spaces, field spaces, partitions, equivalence sets, views, managers, and constraints.

At a minimum it would be good to have a new log that confirms the cycle fix.

[lightsighter]

Yes

[elliottslaughter]

The stencil leak was application specific. If Soleil has any code like:

https://gitlab.com/StanfordLegion/legion/-/blob/d007c1cb748dff1286f9e00349cb5f392522e6c3/language/examples/stencil_fast.rg#L321-337

Then you may need to apply this patch:

https://gitlab.com/StanfordLegion/legion/-/commit/d007c1cb748dff1286f9e00349cb5f392522e6c3

[elliottslaughter]

I'm attach valgrind logs for 5, 10 and 20 iterations of Soleil master branch with tgv.json test case:

• soleil_valgrind_t5.txt
• soleil_valgrind_t10.txt
• soleil_valgrind_t20.txt

The leak does increase with the duration of the run. Logs seem to point to legion_runtime_attach_hdf5. I'm guessing there's something here we're not cleaning up properly.

[lightsighter]

The only new invocation in that method is to create a PhysicalRegion object which the caller is responsible for deleting.

[elliottslaughter]

I fixed the issue by destroying the PhysicalRegion in detach, and leaks are down, but still scale with run duration.

• soleil_valgrind_t5_fix.txt
• soleil_valgrind_t10_fix.txt
• soleil_valgrind_t20_fix.txt

[elliottslaughter]

Ok, fixed another couple of leaks. It's still growing, and the top leaks don't currently make sense to me.

• soleil_valgrind_t5_fix3.txt
• soleil_valgrind_t10_fix3.txt
• soleil_valgrind_t20_fix3.txt

[lightsighter]

Already told this to @elliottslaughter but for everyone else's benefit the growing new in these leaks is the Future return result.

[elliottslaughter]

Confirmed that we still have a future leak.

[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=1 total_bytes=720 diff=1 diff_bytes=720
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=1269 total_bytes=913680 diff=1268 diff_bytes=912960
[0 - 7f7f6d7fa700] {2}{allocation}: Future on 0: total=1041 total_bytes=749520 diff=-228 diff_bytes=-164160
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=1324 total_bytes=953280 diff=283 diff_bytes=203760
[0 - 7f7f6d7fa700] {2}{allocation}: Future on 0: total=1322 total_bytes=951840 diff=-2 diff_bytes=-1440
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=1315 total_bytes=946800 diff=-7 diff_bytes=-5040
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=1310 total_bytes=943200 diff=-5 diff_bytes=-3600
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=1336 total_bytes=961920 diff=26 diff_bytes=18720
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=1349 total_bytes=971280 diff=13 diff_bytes=9360
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=1314 total_bytes=946080 diff=-35 diff_bytes=-25200
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=1316 total_bytes=947520 diff=2 diff_bytes=1440
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=1333 total_bytes=959760 diff=17 diff_bytes=12240
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=1308 total_bytes=941760 diff=-25 diff_bytes=-18000
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=1342 total_bytes=966240 diff=34 diff_bytes=24480
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=1614 total_bytes=1162080 diff=272 diff_bytes=195840
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=1632 total_bytes=1175040 diff=18 diff_bytes=12960
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=1601 total_bytes=1152720 diff=-31 diff_bytes=-22320
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=1549 total_bytes=1115280 diff=-52 diff_bytes=-37440
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=1617 total_bytes=1164240 diff=68 diff_bytes=48960
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=1576 total_bytes=1134720 diff=-41 diff_bytes=-29520
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=1586 total_bytes=1141920 diff=10 diff_bytes=7200
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=1541 total_bytes=1109520 diff=-45 diff_bytes=-32400
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=1839 total_bytes=1324080 diff=298 diff_bytes=214560
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=1870 total_bytes=1346400 diff=31 diff_bytes=22320
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=1881 total_bytes=1354320 diff=11 diff_bytes=7920
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=1829 total_bytes=1316880 diff=-52 diff_bytes=-37440
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=1833 total_bytes=1319760 diff=4 diff_bytes=2880
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=1788 total_bytes=1287360 diff=-45 diff_bytes=-32400
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=1815 total_bytes=1306800 diff=27 diff_bytes=19440
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=1778 total_bytes=1280160 diff=-37 diff_bytes=-26640
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=2064 total_bytes=1486080 diff=286 diff_bytes=205920
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=2079 total_bytes=1496880 diff=15 diff_bytes=10800
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=2068 total_bytes=1488960 diff=-11 diff_bytes=-7920
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=2075 total_bytes=1494000 diff=7 diff_bytes=5040
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=2068 total_bytes=1488960 diff=-7 diff_bytes=-5040
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=2050 total_bytes=1476000 diff=-18 diff_bytes=-12960
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=2123 total_bytes=1528560 diff=73 diff_bytes=52560
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=2102 total_bytes=1513440 diff=-21 diff_bytes=-15120
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=2077 total_bytes=1495440 diff=-25 diff_bytes=-18000
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=2090 total_bytes=1504800 diff=13 diff_bytes=9360
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=2363 total_bytes=1701360 diff=273 diff_bytes=196560
[0 - 7f7f6dffb700] {2}{allocation}: Future on 0: total=2389 total_bytes=1720080 diff=26 diff_bytes=18720
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=2323 total_bytes=1672560 diff=-66 diff_bytes=-47520
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=2249 total_bytes=1619280 diff=-74 diff_bytes=-53280
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=2290 total_bytes=1648800 diff=41 diff_bytes=29520
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=2256 total_bytes=1624320 diff=-34 diff_bytes=-24480
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=2549 total_bytes=1835280 diff=293 diff_bytes=210960
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=2300 total_bytes=1656000 diff=-249 diff_bytes=-179280
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=2580 total_bytes=1857600 diff=280 diff_bytes=201600
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=2606 total_bytes=1876320 diff=26 diff_bytes=18720
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=2586 total_bytes=1861920 diff=-20 diff_bytes=-14400
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=2545 total_bytes=1832400 diff=-41 diff_bytes=-29520
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=2612 total_bytes=1880640 diff=67 diff_bytes=48240
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=2600 total_bytes=1872000 diff=-12 diff_bytes=-8640
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=2517 total_bytes=1812240 diff=-83 diff_bytes=-59760
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=2539 total_bytes=1828080 diff=22 diff_bytes=15840
[0 - 7f7f6d7fa700] {2}{allocation}: Future on 0: total=2532 total_bytes=1823040 diff=-7 diff_bytes=-5040
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=2826 total_bytes=2034720 diff=294 diff_bytes=211680
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=2849 total_bytes=2051280 diff=23 diff_bytes=16560
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=2881 total_bytes=2074320 diff=32 diff_bytes=23040
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=2916 total_bytes=2099520 diff=35 diff_bytes=25200
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=2920 total_bytes=2102400 diff=4 diff_bytes=2880
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=2903 total_bytes=2090160 diff=-17 diff_bytes=-12240
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=2953 total_bytes=2126160 diff=50 diff_bytes=36000
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=2922 total_bytes=2103840 diff=-31 diff_bytes=-22320
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=2920 total_bytes=2102400 diff=-2 diff_bytes=-1440
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=2892 total_bytes=2082240 diff=-28 diff_bytes=-20160
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=2913 total_bytes=2097360 diff=21 diff_bytes=15120
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=2858 total_bytes=2057760 diff=-55 diff_bytes=-39600
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=3157 total_bytes=2273040 diff=299 diff_bytes=215280
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=3189 total_bytes=2296080 diff=32 diff_bytes=23040
[0 - 7f7f67fff700] {2}{allocation}: Future on 0: total=3135 total_bytes=2257200 diff=-54 diff_bytes=-38880
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=3077 total_bytes=2215440 diff=-58 diff_bytes=-41760
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=3099 total_bytes=2231280 diff=22 diff_bytes=15840
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=3107 total_bytes=2237040 diff=8 diff_bytes=5760
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=3360 total_bytes=2419200 diff=253 diff_bytes=182160
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=3130 total_bytes=2253600 diff=-230 diff_bytes=-165600
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=3149 total_bytes=2267280 diff=19 diff_bytes=13680
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=3188 total_bytes=2295360 diff=39 diff_bytes=28080
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=3237 total_bytes=2330640 diff=49 diff_bytes=35280
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=3421 total_bytes=2463120 diff=184 diff_bytes=132480
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=3467 total_bytes=2496240 diff=46 diff_bytes=33120
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=3472 total_bytes=2499840 diff=5 diff_bytes=3600
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=3511 total_bytes=2527920 diff=39 diff_bytes=28080
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=3578 total_bytes=2576160 diff=67 diff_bytes=48240
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=3616 total_bytes=2603520 diff=38 diff_bytes=27360
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=3582 total_bytes=2579040 diff=-34 diff_bytes=-24480
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=3523 total_bytes=2536560 diff=-59 diff_bytes=-42480
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=3526 total_bytes=2538720 diff=3 diff_bytes=2160
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=3470 total_bytes=2498400 diff=-56 diff_bytes=-40320
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=3522 total_bytes=2535840 diff=52 diff_bytes=37440
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=3799 total_bytes=2735280 diff=277 diff_bytes=199440
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=3759 total_bytes=2706480 diff=-40 diff_bytes=-28800
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=3760 total_bytes=2707200 diff=1 diff_bytes=720
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=3750 total_bytes=2700000 diff=-10 diff_bytes=-7200
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=3720 total_bytes=2678400 diff=-30 diff_bytes=-21600
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=3779 total_bytes=2720880 diff=59 diff_bytes=42480
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=3755 total_bytes=2703600 diff=-24 diff_bytes=-17280
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=3709 total_bytes=2670480 diff=-46 diff_bytes=-33120
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=3779 total_bytes=2720880 diff=70 diff_bytes=50400
[0 - 7f7f6effd700] {2}{allocation}: Future on 0: total=3771 total_bytes=2715120 diff=-8 diff_bytes=-5760
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=3816 total_bytes=2747520 diff=45 diff_bytes=32400
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=3954 total_bytes=2846880 diff=138 diff_bytes=99360
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=4062 total_bytes=2924640 diff=108 diff_bytes=77760
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=4079 total_bytes=2936880 diff=17 diff_bytes=12240
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=4161 total_bytes=2995920 diff=82 diff_bytes=59040
[0 - 7f7f6d7fa700] {2}{allocation}: Future on 0: total=4124 total_bytes=2969280 diff=-37 diff_bytes=-26640
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=4195 total_bytes=3020400 diff=71 diff_bytes=51120
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=4156 total_bytes=2992320 diff=-39 diff_bytes=-28080
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=4116 total_bytes=2963520 diff=-40 diff_bytes=-28800
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=4229 total_bytes=3044880 diff=113 diff_bytes=81360
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=4221 total_bytes=3039120 diff=-8 diff_bytes=-5760
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=4206 total_bytes=3028320 diff=-15 diff_bytes=-10800
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=4192 total_bytes=3018240 diff=-14 diff_bytes=-10080
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=4457 total_bytes=3209040 diff=265 diff_bytes=190800
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=4474 total_bytes=3221280 diff=17 diff_bytes=12240
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=4488 total_bytes=3231360 diff=14 diff_bytes=10080
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=4437 total_bytes=3194640 diff=-51 diff_bytes=-36720
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=4494 total_bytes=3235680 diff=57 diff_bytes=41040
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=4468 total_bytes=3216960 diff=-26 diff_bytes=-18720
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=4397 total_bytes=3165840 diff=-71 diff_bytes=-51120
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=4439 total_bytes=3196080 diff=42 diff_bytes=30240
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=4416 total_bytes=3179520 diff=-23 diff_bytes=-16560
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=4475 total_bytes=3222000 diff=59 diff_bytes=42480
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=4492 total_bytes=3234240 diff=17 diff_bytes=12240
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=4512 total_bytes=3248640 diff=20 diff_bytes=14400
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=4800 total_bytes=3456000 diff=288 diff_bytes=207360
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=4839 total_bytes=3484080 diff=39 diff_bytes=28080
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=4845 total_bytes=3488400 diff=6 diff_bytes=4320
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=4794 total_bytes=3451680 diff=-51 diff_bytes=-36720
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=4849 total_bytes=3491280 diff=55 diff_bytes=39600
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=4827 total_bytes=3475440 diff=-22 diff_bytes=-15840
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=4814 total_bytes=3466080 diff=-13 diff_bytes=-9360
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=4767 total_bytes=3432240 diff=-47 diff_bytes=-33840
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=5020 total_bytes=3614400 diff=253 diff_bytes=182160
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=4958 total_bytes=3569760 diff=-62 diff_bytes=-44640
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=5065 total_bytes=3646800 diff=107 diff_bytes=77040
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=5009 total_bytes=3606480 diff=-56 diff_bytes=-40320
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=5008 total_bytes=3605760 diff=-1 diff_bytes=-720
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=4980 total_bytes=3585600 diff=-28 diff_bytes=-20160
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=5009 total_bytes=3606480 diff=29 diff_bytes=20880
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=4992 total_bytes=3594240 diff=-17 diff_bytes=-12240
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=5267 total_bytes=3792240 diff=275 diff_bytes=198000
[0 - 7f7f7eb36700] {2}{allocation}: Future on 0: total=4988 total_bytes=3591360 diff=-279 diff_bytes=-200880
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=5308 total_bytes=3821760 diff=320 diff_bytes=230400
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=5342 total_bytes=3846240 diff=34 diff_bytes=24480
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=5333 total_bytes=3839760 diff=-9 diff_bytes=-6480
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=5356 total_bytes=3856320 diff=23 diff_bytes=16560
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=5309 total_bytes=3822480 diff=-47 diff_bytes=-33840
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=5286 total_bytes=3805920 diff=-23 diff_bytes=-16560
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=5303 total_bytes=3818160 diff=17 diff_bytes=12240
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=5306 total_bytes=3820320 diff=3 diff_bytes=2160
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=5316 total_bytes=3827520 diff=10 diff_bytes=7200
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=5277 total_bytes=3799440 diff=-39 diff_bytes=-28080
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=5592 total_bytes=4026240 diff=315 diff_bytes=226800
[0 - 7f7f6e7fc700] {2}{allocation}: Future on 0: total=5634 total_bytes=4056480 diff=42 diff_bytes=30240
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=5619 total_bytes=4045680 diff=-15 diff_bytes=-10800
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=5634 total_bytes=4056480 diff=15 diff_bytes=10800
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=5597 total_bytes=4029840 diff=-37 diff_bytes=-26640
[0 - 7f7f7f337700] {2}{allocation}: Future on 0: total=5568 total_bytes=4008960 diff=-29 diff_bytes=-20880
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=5595 total_bytes=4028400 diff=27 diff_bytes=19440
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=5549 total_bytes=3995280 diff=-46 diff_bytes=-33120
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=5553 total_bytes=3998160 diff=4 diff_bytes=2880
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=5583 total_bytes=4019760 diff=30 diff_bytes=21600
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=5845 total_bytes=4208400 diff=262 diff_bytes=188640
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=5861 total_bytes=4219920 diff=16 diff_bytes=11520
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=5816 total_bytes=4187520 diff=-45 diff_bytes=-32400
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=5793 total_bytes=4170960 diff=-23 diff_bytes=-16560
[0 - 7f7f777fe700] {2}{allocation}: Future on 0: total=5888 total_bytes=4239360 diff=95 diff_bytes=68400
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=5869 total_bytes=4225680 diff=-19 diff_bytes=-13680
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=5797 total_bytes=4173840 diff=-72 diff_bytes=-51840
[0 - 7f7f75ffb700] {2}{allocation}: Future on 0: total=5827 total_bytes=4195440 diff=30 diff_bytes=21600
[0 - 7f7f6f7fe700] {2}{allocation}: Future on 0: total=5912 total_bytes=4256640 diff=85 diff_bytes=61200
[0 - 7f7f76ffd700] {2}{allocation}: Future on 0: total=5933 total_bytes=4271760 diff=21 diff_bytes=15120
[0 - 7f7f7e335700] {2}{allocation}: Future on 0: total=5889 total_bytes=4240080 diff=-44 diff_bytes=-31680
[0 - 7f7f7db34700] {2}{allocation}: Future on 0: total=6093 total_bytes=4386960 diff=204 diff_bytes=146880
[0 - 7f7f74ff9700] {2}{allocation}: Future on 0: total=6225 total_bytes=4482000 diff=132 diff_bytes=95040
[0 - 7f7f757fa700] {2}{allocation}: Future on 0: total=5371 total_bytes=3867120 diff=-854 diff_bytes=-614880

[elliottslaughter]

I've confirmed the future leak with LEGION_GC, and I've also checked that it goes away with -ffuture 0 (which by the way, needs a fix, so don't try this workaround just yet).

[elliottslaughter]

@mariodirenzo Can you please confirm whether you're still seeing this issue, and if so, should it be listed at #1032 ?

[elliottslaughter]

I believe this was fixed a long time ago, but if it's still here, someone would need to confirm and provide a reproducer.