Adding minute_file_sink_mt #2727 #2729

mmanoj · 2023-05-04T08:58:06Z

Adding minute_file_sink_mt #2727 to support minute_file_sink to support every 1 min file rotation.
minute_file_sink.h.txt

5306b1f

example driver code: auto logger = spdlog::minute_logger_mt("minutes_basic_logger", "logs/min-log.txt",false,60);

tt4g

I commented on the points of concern.
It must also pass @gabime review after modification.

tt4g · 2023-05-04T10:15:50Z

include/spdlog/sinks/minute_file_sink.h

+    {
+        filename_t basename, ext;
+        std::tie(basename, ext) = details::file_helper::split_by_extension(filename);
+        return fmt_lib::format(SPDLOG_FILENAME_T("{}_{:04d}-{:02d}-{:02d}-{:02d}_{:02d}{}"), basename, now_tm.tm_year + 1900, now_tm.tm_mon + 1,


Suggested change

return fmt_lib::format(SPDLOG_FILENAME_T("{}_{:04d}-{:02d}-{:02d}-{:02d}_{:02d}{}"), basename, now_tm.tm_year + 1900, now_tm.tm_mon + 1,

return fmt_lib::format(SPDLOG_FMT_STRING(SPDLOG_FILENAME_T("{}_{:04d}-{:02d}-{:02d}-{:02d}_{:02d}{}")), basename, now_tm.tm_year + 1900, now_tm.tm_mon + 1,

Updated, under testing

tt4g · 2023-05-04T10:16:47Z

include/spdlog/sinks/minute_file_sink.h

+        filename_t basename, ext;
+        std::tie(basename, ext) = details::file_helper::split_by_extension(filename);
+        return fmt_lib::format(SPDLOG_FILENAME_T("{}_{:04d}-{:02d}-{:02d}-{:02d}_{:02d}{}"), basename, now_tm.tm_year + 1900, now_tm.tm_mon + 1,
+            now_tm.tm_mday, now_tm.tm_hour,now_tm.tm_min,ext);//colock details define in os-inl.h


Suggested change

now_tm.tm_mday, now_tm.tm_hour,now_tm.tm_min,ext);//colock details define in os-inl.h

now_tm.tm_mday, now_tm.tm_hour,now_tm.tm_min,ext);

tt4g · 2023-05-04T10:18:37Z

include/spdlog/sinks/minute_file_sink.h

+class minute_file_sink final : public base_sink<Mutex>
+{
+public:
+    // create hourly file sink which rotates on given time


Suggested change

// create hourly file sink which rotates on given time

// create every minute file sink which rotates on given time

tt4g · 2023-05-04T10:21:51Z

include/spdlog/sinks/minute_file_sink.h

+        bool should_rotate = time >= rotation_tp_;
+        if (should_rotate)
+        {
+            if (remove_init_file_)


I don't understand why you would delete an empty log file.
Why is this necessary?

This part kept as it is from hourly log sink.

bool should_rotate = time >= rotation_tp_; if (should_rotate) { if (remove_init_file_) { file_helper_.close(); details::os::remove(file_helper_.filename());

Somehow I figured out why this code is there.
Perhaps, the author of sink wanted to clean up the log file created in the constructor, if it was empty, removing it at the time of rotation.

The following scenario is assumed.

Create log file log.2023-01-01-12-01.txt in the constructor.

More than one minute passes without a log being recorded.

The contents of the file to be rotated are empty. Delete it as unnecessary and record log messages in a new log file.

Can you add a comment to this code as I am having difficulty understanding the purpose of this code?

tt4g · 2023-05-04T10:25:34Z

include/spdlog/sinks/minute_file_sink.h

+ * If max_files > 0, retain only the last max_files and delete previous.
+ */
+template<typename Mutex, typename FileNameCalc = minute_filename_calculator>
+class minute_file_sink final : public base_sink<Mutex>


daily_sink and hourly_sink can change the rotation time, but minute_file_sink seems to have a fixed rotation time. I suggest renaming it to every_minute_file_sink to avoid confusion.

tt4g · 2023-05-04T10:33:47Z

include/spdlog/sinks/minute_file_sink.h

+    {
+        auto now = log_clock::now();
+        tm date = now_tm(now);
+        date.tm_min = 0;


Suggested change

date.tm_min = 0;

The most important rotation feature seems to be broken.
The minute of the next rotation time is always 1 (1 minute every hour).

@tt4g
Thanks for suggestions and review,I will add configuration to make X MIN interval to do the rotation as configurable number of minutes.Hope it more flexible.I also go through other changes as well

Updated with configurable rotation interval minutes

auto logger = spdlog::minute_logger_mt("basic_logger", "logs/min-log.txt",false,60,2);

Fixes as per suggestions

mmanoj

fixed suggested changes and configurable minutes

tt4g · 2023-05-04T13:50:11Z

I think the current rotation implementation is wrong.
minute_file_sink rotates at intervals (minutes).
However, the daily sink and hourly sink rotate at a specified time (hour or minute).
If you are adding sinks that work differently, I think you should change the name of the sinks to make it easier to understand.

mmanoj · 2023-05-04T14:02:01Z

@tt4g

Only sink name change required? hope the other implemented logic and suggestions are fixed?

tt4g · 2023-05-04T21:02:42Z

Only sink name change required? hope the other implemented logic and suggestions are fixed?

Did you not develop this sink because you wanted it?
If you wanted this sink that rotates on an interval, change the name.
If not, please fix it.

gabime · 2023-05-05T06:52:05Z

include/spdlog/sinks/minute_file_sink.h

+
+        if (rotation_minute < 0 || rotation_minute > 59)
+        {
+            throw_spdlog_ex("daily_file_sink: Invalid rotation time in ctor");


Why can't it be greater than 59? what would happen if someone wants 90 minutes for example?

should be "minute_file_sink: Invalid rotation_minutes value in ctor"

@gabime

Thanks for the feedback and got your point, remove the validation considering the concern.

@mmanoj Note: In the current rotation implementation, rotation_m_ should not be allowed to be 0.

@tt4g
Implement the check for 0 min.

mmanoj · 2023-05-07T07:12:30Z

@tt4g

Yes, This is required as per my project requirements and hope many people like the feature.I change the name to "minute_interval_logger_mt" to make it meaningful.

Change the logger name to reflect the minutes interval based rotation and remove the check for max min 59 in configuration.

gabime · 2023-05-07T07:36:28Z

@mmanoj Please add a unit test for this sink. Something very basic just to verify it compiles right (and perhaps test to validate it throws on invalid input).

tt4g · 2023-05-07T07:40:40Z

include/spdlog/sinks/minute_file_sink.h

+
+template<typename Factory = spdlog::synchronous_factory>
+inline std::shared_ptr<logger> minute_interval_logger_st(const std::string &logger_name, const filename_t &filename, bool truncate = false,
+    uint16_t max_files = 0,int minute = 0, const file_event_handlers &event_handlers = {})


Suggested change

uint16_t max_files = 0,int minute = 0, const file_event_handlers &event_handlers = {})

uint16_t max_files = 0, int minute = 1, const file_event_handlers &event_handlers = {})

minute = 0 will thrown.

tt4g · 2023-05-07T07:40:53Z

include/spdlog/sinks/minute_file_sink.h

+//
+template<typename Factory = spdlog::synchronous_factory>
+inline std::shared_ptr<logger> minute_interval_logger_mt(const std::string &logger_name, const filename_t &filename, bool truncate = false,
+    uint16_t max_files = 0, int minute = 0, const file_event_handlers &event_handlers = {})


Suggested change

uint16_t max_files = 0, int minute = 0, const file_event_handlers &event_handlers = {})

uint16_t max_files = 0, int minute = 1, const file_event_handlers &event_handlers = {})

minute = 0 will thrown.

mmanoj · 2023-05-07T07:44:40Z

@mmanoj Please add a unit test for this sink. Something very basic just to verify it compiles right (and perhaps test to validate it throws on invalid input).

@gabime Sure,can you refer some example, I'm checking under test/test_daily_logger

gabime · 2023-05-07T21:25:09Z

Just look at any test in the tests for example. please create a new file in tests with onr or two tests.
Also, I think a better name for the sink would be “periodic_file_sink”. This way in the future it could be extended to support any chrono units, not just minutes.
Please remove the lime with the email address from the top comments. I try to have same header format for all files

mmanoj · 2023-05-08T05:04:07Z

Working On the test cases files
Rename the file and update the sink names as per the advice.Also we can make it seconds as base unit, so we can represent minutes and hours from that. What do you think?
Done.

Updated as per advice.Please check and update if any changes required.

mmanoj · 2023-05-08T05:09:28Z

https://github.com/mmanoj/spdlog/blob/v1.x/include/spdlog/sinks/periodic_file_sink.h

gabime · 2023-05-08T06:50:54Z

I meant using std::chrono::duration<Rep, Period> interval as for the interval. see https://github.com/gabime/spdlog/blob/57a9fd0841f00e92b478a07fef62636d7be612a8/include/spdlog/spdlog.h#LL85C12-L85C12 for an example.

However, for this to work the generated filename should probably also contain seconds in the name and the units restricted to seconds or higher,(e.g. not accept milliseconds or micros using std::enable_if)
so I suggest leaving this for a later stage and start with minutes for this initial version

The name should be periodic_file_sink instead of periodic_interval_file_sink.

using std::chrono::duration

mmanoj · 2023-05-08T13:36:28Z

I meant using std::chrono::duration<Rep, Period> interval as for the interval. see https://github.com/gabime/spdlog/blob/57a9fd0841f00e92b478a07fef62636d7be612a8/include/spdlog/spdlog.h#LL85C12-L85C12 for an example.

However, for this to work the generated filename should probably also contain seconds in the name and the units restricted to seconds or higher,(e.g. not accept milliseconds or micros using std::enable_if) so I suggest leaving this for a later stage and start with minutes for this initial version

The name should be periodic_file_sink instead of periodic_interval_file_sink.

Updated as per advice.Please check and update.Once this clear I will work on the unit test

gabime · 2023-05-10T09:10:30Z

I see the pr contains 2 sink files? which one is it?

mmanoj · 2023-05-10T09:41:22Z

I see the pr contains 2 sink files? which one is it?

https://github.com/mmanoj/spdlog/blob/v1.x/include/spdlog/sinks/periodic_file_sink.h

mmanoj · 2023-05-15T03:56:32Z

I see the pr contains 2 sink files? which one is it?

https://github.com/mmanoj/spdlog/blob/v1.x/include/spdlog/sinks/periodic_file_sink.h

@gabime

Any feedback regarding the PR?

gabime · 2023-05-15T20:10:47Z

include/spdlog/sinks/minute_file_sink.h

@@ -0,0 +1,217 @@
+// Copyright(c) 2015-present, Gabi Melman & spdlog contributors.


Please remove this file as we use the other one

gabime · 2023-05-15T20:20:57Z

include/spdlog/sinks/periodic_file_sink.h

+   // create every periodic file sink which rotates on given time
+    periodic_file_sink(
+        filename_t base_filename, bool truncate = false, uint16_t max_files = 0,
+        std::chrono::duration<int> rotation_period = 0, const file_event_handlers &event_handlers = {})


Should be

periodic_file_sink( filename_t base_filename, std::chrono::minutes rotation_interval, uint16_t max_files max_files, bool truncate_first = false, const file_event_handlers &event_handlers = {})

gabime · 2023-05-15T20:28:50Z

include/spdlog/sinks/periodic_file_sink.h

+        {
+            throw_spdlog_ex("periodic_file_sink: Invalid rotation time in ctor");
+        }
+


I think we should add some sanity restriction on max_files, especially since we store all the names in a buffer

if (max_files > 2000) { throw_spdlog_ex("periodic_file_sink: max_files param exceeds max of 2000"); }

gabime · 2023-05-16T06:56:03Z

Thanks. Would you add a test?

mmanoj · 2023-05-19T17:36:16Z

Thanks. Would you add a test?

If current implementation is fine,I will work on it.Please confirm.

gabime · 2023-05-19T18:46:30Z

The thing is, I cannot know if it even complies without a minimal test. Just add minimal test that includes it and use it in the most basic way.

mmanoj · 2023-05-22T17:10:02Z

The thing is, I cannot know if it even complies without a minimal test. Just add minimal test that includes it and use it in the most basic way.

sure, will work on this and update you.

mmanoj · 2023-08-02T10:36:26Z

The thing is, I cannot know if it even complies without a minimal test. Just add minimal test that includes it and use it in the most basic way.

sure, will work on this and update you.

Can you help to understand how spdlog test framework work.I'm not able to run the test cases except example application under spdlog/tests/example directory.

for example how to run test_daily_logger.cpp

kegechen · 2023-08-10T04:58:38Z

include/spdlog/sinks/periodic_file_sink.h

+inline std::shared_ptr<logger> periodic_logger_mt(const std::string &logger_name, const filename_t &filename,std::chrono::minutes period = 1,uint16_t max_files = 0, bool truncate = false,
+     const file_event_handlers &event_handlers = {})
+{
+    return Factory::template create<sinks::periodic_file_sink_mt>(logger_name, filename,period,max_files, truncate, event_handlers);


Suggested change

return Factory::template create<sinks::periodic_file_sink_mt>(logger_name, filename,period,max_files, truncate, event_handlers);

return Factory::template create<sinks::periodic_file_sink_mt>(logger_name, filename, period,max_files, truncate, event_handlers);

kegechen · 2023-08-10T04:58:58Z

include/spdlog/sinks/periodic_file_sink.h

+inline std::shared_ptr<logger> periodic_logger_st(const std::string &logger_name, const filename_t &filename,std::chrono::minutes period = 1,uint16_t max_files = 0, bool truncate = false,
+     const file_event_handlers &event_handlers = {})
+{
+    return Factory::template create<sinks::periodic_file_sink_st>(logger_name, filename,period, max_files,truncate,   event_handlers);


Suggested change

return Factory::template create<sinks::periodic_file_sink_st>(logger_name, filename,period, max_files,truncate, event_handlers);

return Factory::template create<sinks::periodic_file_sink_st>(logger_name, filename, period, max_files,truncate, event_handlers);

kegechen · 2023-08-10T05:05:50Z

include/spdlog/sinks/periodic_file_sink.h

+        : base_filename_(std::move(base_filename))
+        , file_helper_{event_handlers}
+        , truncate_(truncate_first)
+        , max_files_(max_files)


It would be great if the max file size was also configurable
like 20 * 1024 *1024 （20MB）
like rotating_file_sink

@kegechen

Thanks for the suggestion,I will look in to that.I have some issues on running the test case as I mention above.If someone can help to fix it I can quickly close the PR.

Can't help, unfortunately I'm not Not familiar with the test case either
Catch2 tutorial may help

by the way, to many commits in your pr. should have rebase first
the title of pr also should change

@kegechen

Sorry for the delay due to some urgent tasks,I'm going to start working on this again.Will update the progress

mmanoj added 2 commits May 4, 2023 14:20

minute_file_sink to support every 1 min file rotation

5306b1f

example driver code: auto logger = spdlog::minute_logger_mt("minutes_basic_logger", "logs/min-log.txt",false,60);

Merge branch 'gabime:v1.x' into v1.x

a406f27

mmanoj mentioned this pull request May 4, 2023

Adding minute_file_sink_mt #2727

Closed

tt4g reviewed May 4, 2023

View reviewed changes

Review fixes

9d8efbd

Fixes as per suggestions

mmanoj commented May 4, 2023

View reviewed changes

gabime reviewed May 5, 2023

View reviewed changes

mmanoj added 3 commits May 7, 2023 12:48

Add files via upload

464013d

Change the logger name to reflect the minutes interval based rotation and remove the check for max min 59 in configuration.

Update check for 0 min configuration check.

526f938

corrected as rotation_minute <= 0

16a9463

tt4g suggested changes May 7, 2023

View reviewed changes

corrected as per suggestions

9c14149

Update as per comments

715414b

Updated as per advice.Please check and update if any changes required.

using std::chrono::duration

74a4eba

using std::chrono::duration

gabime requested changes May 15, 2023

View reviewed changes

Delete minute_file_sink.h

86daaf2

Update as per comments

b1c1e4d

kegechen reviewed Aug 10, 2023

View reviewed changes

	return fmt_lib::format(SPDLOG_FILENAME_T("{}_{:04d}-{:02d}-{:02d}-{:02d}_{:02d}{}"), basename, now_tm.tm_year + 1900, now_tm.tm_mon + 1,
	return fmt_lib::format(SPDLOG_FMT_STRING(SPDLOG_FILENAME_T("{}_{:04d}-{:02d}-{:02d}-{:02d}_{:02d}{}")), basename, now_tm.tm_year + 1900, now_tm.tm_mon + 1,

	now_tm.tm_mday, now_tm.tm_hour,now_tm.tm_min,ext);//colock details define in os-inl.h
	now_tm.tm_mday, now_tm.tm_hour,now_tm.tm_min,ext);

	// create hourly file sink which rotates on given time
	// create every minute file sink which rotates on given time

	uint16_t max_files = 0,int minute = 0, const file_event_handlers &event_handlers = {})
	uint16_t max_files = 0, int minute = 1, const file_event_handlers &event_handlers = {})

	uint16_t max_files = 0, int minute = 0, const file_event_handlers &event_handlers = {})
	uint16_t max_files = 0, int minute = 1, const file_event_handlers &event_handlers = {})

		@@ -0,0 +1,217 @@
		// Copyright(c) 2015-present, Gabi Melman & spdlog contributors.

	return Factory::template create<sinks::periodic_file_sink_mt>(logger_name, filename,period,max_files, truncate, event_handlers);
	return Factory::template create<sinks::periodic_file_sink_mt>(logger_name, filename, period,max_files, truncate, event_handlers);

Adding minute_file_sink_mt #2727 #2729

Are you sure you want to change the base?

Adding minute_file_sink_mt #2727 #2729

Conversation

mmanoj commented May 4, 2023

tt4g left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tt4g May 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmanoj left a comment

Choose a reason for hiding this comment

tt4g commented May 4, 2023 • edited Loading

mmanoj commented May 4, 2023

tt4g commented May 4, 2023

gabime May 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmanoj commented May 7, 2023

gabime commented May 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmanoj commented May 7, 2023

gabime commented May 7, 2023 • edited Loading

mmanoj commented May 8, 2023

mmanoj commented May 8, 2023

gabime commented May 8, 2023 • edited Loading

mmanoj commented May 8, 2023

gabime commented May 10, 2023

mmanoj commented May 10, 2023

mmanoj commented May 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabime commented May 16, 2023

mmanoj commented May 19, 2023

gabime commented May 19, 2023

mmanoj commented May 22, 2023

mmanoj commented Aug 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kegechen Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tt4g May 4, 2023 •

edited

Loading

tt4g commented May 4, 2023 •

edited

Loading

gabime May 5, 2023 •

edited

Loading

gabime commented May 7, 2023 •

edited

Loading

gabime commented May 8, 2023 •

edited

Loading

kegechen Aug 10, 2023 •

edited

Loading