async flush wait until flushed

[ghost]

I'm wondering if the following approach is ok for waiting until a flush request on an async logger has been completed (this is in a situation where you guarantee there are no new messages coming in to be logged, otherwise may be waiting forever if queue is growing from new log requests):

// construct
spdlog::init_thread_pool(16384U, 6U);
auto spdlog_tp = spdlog::thread_pool(); 
auto async_logger = spdlog::create_async<sinks::basic_file_sink_mt>("logger_name", ...);

// ... do things

// flush when guaranteed no new logs coming in
async_logger.flush();
while(spdlog_tp->queue_size() != 0) { std::this_thread::sleep_for(Milliseconds(5)); }

[tt4g]

As long as you are using spdlog::async_overflow_policy::block (spdlog::async_logger default policy), calling spdlog::shutdown() is better than while(spdlog_tp->queue_size() != 0) { std::this_thread::sleep_for(Milliseconds(5)); }.

 async_logger.flush();
- while(spdlog_tp->queue_size() != 0) { std::this_thread::sleep_for(Milliseconds(5)); }
+ spdlog::shutdown();

Because:

In spdlog 1.9.2, spdlog::details::thread_pool: has a task queue, and executes tasks by pulling them from the queue in sequence.

spdlog/include/spdlog/details/thread_pool-inl.h

Lines 55 to 86 in eb32206

	void SPDLOG_INLINE thread_pool::post_log(async_logger_ptr &&worker_ptr, const details::log_msg &msg, async_overflow_policy overflow_policy)
	{
	async_msg async_m(std::move(worker_ptr), async_msg_type::log, msg);
	post_async_msg_(std::move(async_m), overflow_policy);
	}
	void SPDLOG_INLINE thread_pool::post_flush(async_logger_ptr &&worker_ptr, async_overflow_policy overflow_policy)
	{
	post_async_msg_(async_msg(std::move(worker_ptr), async_msg_type::flush), overflow_policy);
	}
	size_t SPDLOG_INLINE thread_pool::overrun_counter()
	{
	return q_.overrun_counter();
	}
	size_t SPDLOG_INLINE thread_pool::queue_size()
	{
	return q_.size();
	}
	void SPDLOG_INLINE thread_pool::post_async_msg_(async_msg &&new_msg, async_overflow_policy overflow_policy)
	{
	if (overflow_policy == async_overflow_policy::block)
	{
	q_.enqueue(std::move(new_msg));
	}
	else
	{
	q_.enqueue_nowait(std::move(new_msg));
	}
	}

spdlog::async_logger::flush() push flush task to queue of ``spdlog::details::thread_pool`.

spdlog/include/spdlog/async_logger-inl.h

Lines 40 to 50 in eb32206

	SPDLOG_INLINE void spdlog::async_logger::flush_()
	{
	if (auto pool_ptr = thread_pool_.lock())
	{
	pool_ptr->post_flush(shared_from_this(), overflow_policy_);
	}
	else
	{
	throw_spdlog_ex("async flush: thread pool doesn't exist anymore");
	}
	}

spdlog::details::thread_pool::~thread_pool() will push the async_msg_type::terminate task to the queue and wait for all tasks to finish.

spdlog/include/spdlog/details/thread_pool-inl.h

Lines 33 to 53 in eb32206

	SPDLOG_INLINE thread_pool::thread_pool(size_t q_max_items, size_t threads_n)
	: thread_pool(q_max_items, threads_n, [] {})
	{}
	// message all threads to terminate gracefully join them
	SPDLOG_INLINE thread_pool::~thread_pool()
	{
	SPDLOG_TRY
	{
	for (size_t i = 0; i < threads_.size(); i++)
	{
	post_async_msg_(async_msg(async_msg_type::terminate), async_overflow_policy::block);
	}
	for (auto &t : threads_)
	{
	t.join();
	}
	}
	SPDLOG_CATCH_STD
	}

spdlog::shutdown() will release spdlog::details::thread_pool.

spdlog/include/spdlog/details/registry-inl.h

Lines 253 to 256 in a1d9f50

	{
	std::lock_guard<std::recursive_mutex> lock(tp_mutex_);
	tp_.reset();
	}

That is, as long as spdlog::async_logger::flush() is called before spdlog::shutdown(), the thread pool will be terminated after the flush operation.

[ghost]

Thanks for the response! What you've discussed works well if you intend for the logger to be shutdown after the flush. In my situation, I'd like to flush all messages before then continuing to allow further messages to be logged (e.g. I want to change the pattern being used, but first flush all logs from old pattern).

For this use-case, do you have any suggested modifications for the original solution?

[tt4g]

If you can guarantee that no other log messages will be added to the thread pool queue, you can wait for the flush with the original solution.

[ghost]

Great! Could you add this as a separate answer so I can accept it?

[tt4g]

TO: #2084 (reply in thread)

If you can guarantee that no other log messages will be added to the thread pool queue, you can wait for the flush with the original solution.