RFE: value interpolation

[DALDEI]

A highly use feature in config systems is runtime value interpolation. HCON implements this fairly crudely. A good example is log4j's configuration
This is equivalent to kotlin or groovy interpolated strings except the string to be interpolated comes from within the config values itself not in the kotlin code.
As in HCON, it is useful to be able to evaluate the values lazily so they can contain references to other values not necessarily from the same file. When using multiple formats its tricky to get right, but I belive using the property format would work well in all source formats like:

in source1.json

{ 
  "base" : { "user" : "a user" , "password" : "a password" }
}

in source2.properties

connection.jdbc=jdbc:mysql:http://${base.user}:${base.password}@server:port

Then in the app when getting 'connection.jdbc' it gets "jdbc:mysql:http://user:a password@server:port"

This can currently be done with some complex specs and lazy values or properties on a ad-hoc basis, and possibly with some kind of overloaded get() functions but its not obvious exactly how without writing a full new layer of access.

From what I can see in the code I think it would be fairly simple to implement as a direct feature, maybe with some overloading to be supplied by the user for extension.
E.g. the log4j syntax uses ${source:value} , where "source" is composed of a set of builtin and user defined configuration sources similar to konf Source's e.g. ${env:ENV_VAR} where "env" is mapped to the the environment variable source.

There might be a way of leveraging the spec prefix, config layer , loader or source names or prefixes to default much of this behaviour.

Suggestions welcome for where you think a feature of this sort 'naturally' belongs.

--
A related concept is to allow object level merging specified in the config files themselves.
HCON does this but in a very crude way (via string interpolation not structural merging).

e.g. imaging the above case but instead of string interpolation, structural merging is done

source2.properties

connection.jdbc = ${base}
connection.jdbc.server = mysql.com
connection.jsbc.user = differentuser
connection.jdbc.url = http://${user}:${password}@${server}

It may be useful to use a different syntax for structural merging then value substitution, e.g. maybe something like ${base.*} or ${@base} ... something to indicate to pull the subtree rooted at '${base}' into the current context.

It seems like this could be accomplished by creating a new layer implicitly when a structural reference is found injecting it into the current prefix. From there the normal default processing would work as if the subtree were actually inline.

[uchuhimo]

You request two features:

• runtime value interpolation
• structural merging in the config files

I think runtime value interpolation is simple and useful enough to be implemented in konf. Maybe I can also add script support to allow something like maxSize = ${size * 4} in config file using Kotlin's javax.script support.
Structural merging is more complicated for konf. This feature means pushing down konf's runtime API (withSourceFrom) to config file level, and need to:

• define load syntax for all sources
• support load syntax in all source's implementations

Since it's a complicated feature to implement, I need more examples to indicate that this feature is necessary in those cases.

[DALDEI]

I agree the structural merging is complex and would need careful design -- If I can figure out a concrete proposal I will add a new RFE otherwise please just keep it in mind for future if you or anyone comes up with a clean way of doing it. Fairly recently Jackson ObjectMapper added a better methodology for JSON deep merging which may be applicable.
Thanks.

As for scripting, I've been playing with the kotlin scripting -- it was very unstable in the past but recently is much better. However there is a significant overhead to including it as it requires the kotlinc runtime so I would suggest not tightly coupling it to scripting so that smaller footprints are possible.
Possibly leaving the evaluation part as a user supplied implementation.
It is very tempting to use kotlin built in string interpolation but that is only at compile time (or script!).
However I think that a scripting use might make the more direct use of paths complicated .. you would have to set exactly the right receivers such that simple path expressions like '${base.user}' would resolve in a scripting context. This is a generally powerful concept for a DSL which I have been looking for sample implementations but not yet found any.

[DALDEI]

Background Commentary:

FYI: my personal goal for which I have added these RFE's to support is to create a generalized DSL for configs which can be either fully compiled in or executed via kotlin-scripting inside a compiled program or in a fully scripted program ( e.g. using kscript which is evolving into a very good program).

From a design perspective I believe there should be only minimal added to the core konf if possible so that the overhead of either compiled or scripted DSL is not incurred for users who do not want that feature. That is always a difficult design point.

By 'DSL' I mean a few different things --

1. simpler description of the config spec itself -- currently it is somewhat verbose, e.g you have to specify the name of the value in 2 places - the property name in kotlin as well as a string argument,
   the nullability or optional/required is not taken from the type itself etc. Difficult to do I know.
2. A 'convention' of how config specs and loaders are configured to make it easier for common cases:
   This is very well done so far but could be improved further in a DSL with pre-defined use cases.
3. an easier way to map the loaded values into a runtime value object.
   Currently you have to create 2 variants of the same general type for clean use.
   e.g a Spec to define the config and a class to put the results in ...
   For a conventional case combining the 2 into 1 kind of configuration would be very convenient.
   Its a tough design to implement, I have seen several good examples that do well at this point
   but do poor at other aspects -- The best analog I have seen is with Command line parsers -- its the same issue (and would make a good integration for an additional loader).
   Some use annotations, some use a property delegate. A class level delegate would be ideal to avoid having to specify all the property delegats manually but that requires a shared interface -- another kind of duplication. A scriptable DSL might be able to accomplish this by creating the interfaces at runtime.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[uchuhimo]

@DALDEI This feature has been provided in v0.19.0, see Path Substitution in README.