Custom Recorder

[michaelAlvarino]

Is there a right way to provide a custom Recorder?

As far as I can tell I have to wrap evals as my own CLI tool (which I can basically copy/paste from the existing CLI tool implementation), then modify the code to use my custom-written Recorder.

As for implementing the recorder itself, is there a best-practice for a method of RecorderBase to override? The examples available override either

• _flush_events_internal which by the name "internal" sounds like it should not be overridden.
• record_event which seems like the right place to override things, but has unexpected requirements like writing to the _events list (or else get_events will not return anything and cause metrics like this to be incorrect.

Are you open to contribution improving the Recorder API? Is there already an open issue for this or is it in progress in some way?