Add retry logic to wait for active threads to finish

Add retry logic to wait for active threads to finish #2046