Skip to content

Commit

Permalink
moved configfile -> lib/reader & readers -> lib
Browse files Browse the repository at this point in the history 
- moved regex and watch into separate files
  • Loading branch information
@msimerson
msimerson committed Apr 15, 2024
1 parent 4fb12e1 commit 924e070
Show file tree
Hide file tree
Showing 22 changed files with 243 additions and 285 deletions.
105 changes: 39 additions & 66 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ Haraka config file loader, parser, and watcher.

Haraka's config loader can load several types of configuration files.

- 'value' - load a flat file containing a single value (default)
- 'ini'. - load an ini file
- 'json' - load a json file
- 'hjson' - load a hjson file
- 'yaml' - load a yaml file
- 'list' - load a flat file containing a list of values
- 'data' - load a flat file containing a list, keeping comments and whitespace.
- 'binary' - load a binary file into a Buffer
- value - load a flat file containing a single value (default)
- ini - load an ini file
- json - load a json file
- hjson - load a hjson file
- yaml - load a yaml file
- list - load a flat file containing a list of values
- data - load a flat file containing a list, keeping comments and whitespace.
- binary - load a binary file into a Buffer

See the [File Formats](#file_formats) section below for a more detailed
explanation of each of the formats.
Expand Down Expand Up @@ -68,12 +68,8 @@ exports.hook_connect = function (next, connection) {
The `options` object can accepts the following keys:

- `no_watch` (default: false) - prevents Haraka from watching for updates.
- `no_cache` (default: false) - prevents Haraka from caching the file. This
means that the file will be re-read on every call to `config.get`. This is
not recommended as config files are read syncronously, will block the event
loop, and will slow down Haraka.
- `booleans` (default: none) - for .ini files, this allows specifying
boolean type keys. Default true or false can be specified.
- `no_cache` (default: false) - prevents Haraka from caching the file. The file will be re-read on every call to `config.get`. This is not recommended as config files are read syncronously and will slow down Haraka.
- `booleans` (default: none) - for .ini files, this allows specifying boolean type keys. Default true or false can be specified.

## <a name="overrides">Default Config and Overrides</a>

Expand Down Expand Up @@ -134,17 +130,14 @@ sub1=something
sub2=otherthing
```

This allows plugins to provide a default config, and allow users to override
This allows plugins to ship a default config and users can override
values on a key-by-key basis.

# <a name="file_formats">File Formats</a>

## Ini Files

INI files have their heritage in early versions of Microsoft Windows.
Entries are a simple format of key=value pairs, with optional [sections].

Here is a typical example:
[INI files](https://en.wikipedia.org/wiki/INI_file) are key=value pairs, with optional [sections]. A typical example:

```ini
first_name=Matt
Expand Down Expand Up @@ -180,14 +173,14 @@ That produces the following Javascript object:
}
```

Items before any [section] marker are in the implicit [main] section.
Items before any `[section]` marker are in the implicit `[main]` section.

Some values on the right hand side of the equals are converted:

There is some auto-conversion of values on the right hand side of
the equals: integers are converted to integers, floats are converted to
floats.
- integers are converted to integers
- floats are converted to floats.

The key=value pairs support continuation lines using the
backslash "\" character.
The key=value pairs support continuation lines using the backslash "\" character.

The `options` object allows you to specify which keys are boolean:

Expand All @@ -198,9 +191,9 @@ The `options` object allows you to specify which keys are boolean:
```

On the options declarations, key names are formatted as section.key.
If the key name does not specify a section, it is presumed to be [main].
If the key name does not specify a section, it is presumed to be `[main]`.

This ensures these values are converted to true Javascript booleans when parsed, and supports the following options for boolean values:
Declaring booleans ensures that values are converted as boolean when parsed, and supports the following options for boolean values:

```
true, yes, ok, enabled, on, 1
Expand Down Expand Up @@ -241,20 +234,13 @@ which produces this javascript array:

## Flat Files

Flat files are simply either lists of values separated by \n or a single
value in a file on its own. Those who have used qmail or qpsmtpd will be
familiar with this format.
Lines starting with '#' and blank lines will be ignored unless the type is
specified as 'data', however even then line endings will be stripped.
See plugins/dnsbl.js for an example.
Flat files are simply either lists of values separated by \n or a single value in a file on its own. Qmail or qpsmtpd users will be familiar with this format. Lines starting with '#' and blank lines will be ignored unless the type is specified as 'data', however even then line endings will be stripped.

## JSON Files

These are as you would expect, and returns an object as given in the file.

If a requested .json or .hjson file does not exist then the same file will be checked
for with a .yaml extension and that will be loaded instead. This is done
because YAML files are far easier for a human to write.
If a requested .json or .hjson file does not exist then the same file will be checked for with a .yaml extension and that will be loaded instead. This is done because YAML files are far easier for a human to write.

You can use JSON, HJSON or YAML files to override any other file by prefixing the outer variable name with a `!` e.g.

Expand All @@ -264,11 +250,9 @@ You can use JSON, HJSON or YAML files to override any other file by prefixing th
}
```

If the config/smtpgreeting file did not exist, then this value would replace
it.
If the config/smtpgreeting file did not exist, then this value would replace it.

NOTE: You must ensure that the data type (e.g. Object, Array or String) for
the replaced value is correct. This cannot be done automatically.
NOTE: You must ensure that the data type (e.g. Object, Array or String) for the replaced value is correct. This cannot be done automatically.

## Hjson Files

Expand All @@ -287,46 +271,35 @@ Example syntax

```hjson
{
# specify rate in requests/second (because comments are helpful!)
rate: 1000
# specify rate in requests/second (because comments are helpful!)
rate: 1000
// prefer c-style comments?
/* feeling old fashioned? */
// prefer c-style comments?
/* feeling old fashioned? */
# did you notice that rate does not need quotes?
hey: look ma, no quotes for strings either!
# did you notice that rate does not need quotes?
hey: look ma, no quotes for strings either!
# best of all
notice: []
anything: ?
# best of all
notice: []
anything: ?
# yes, commas are optional!
# yes, commas are optional!
}
```

NOTE: Hjson can be also replaced by YAML configuration file. You can find more on this issue under JSON section.
NOTE: Hjson can be also replaced by a YAML configuration file. You can find more on this issue under JSON section.

## YAML Files

As per JSON files above but in YAML format.

# Reloading/Caching

Haraka automatically reloads configuration files, but this only works if
whatever is looking at that config re-calls config.get() to retrieve the
new config. Providing a callback in the config.get() call is the most
efficient method to do this.
Haraka automatically reloads configuration files, but this only works if whatever is looking at that config re-calls config.get() to retrieve the new config. Providing a callback in the config.get() call is the most efficient method to do this.

Configuration files are watched for changes using filesystem events which
are inexpensive. Due to caching, calling config.get() is normally a
lightweight process.
Configuration files are watched for changes using filesystem events which are inexpensive. Due to caching, calling config.get() is normally a lightweight process.

On Linux/Windows, newly created files that Haraka has tried to read in the
past will be noticed immediately and loaded. For other operating systems,
it may take up to 60 seconds to load, due to differences between in the
kernel APIs for watching files/directories.
On Linux/Windows, newly created files that Haraka has tried to read in the past will be noticed immediately and loaded. For other operating systems, it may take up to 60 seconds to load, due to differences between in the kernel APIs for watching files/directories.

Haraka reads a number of configuration files at startup. Any files read
in a plugins register() function are read _before_ Haraka drops privileges.
Be sure that Haraka's user/group has permission to read these files else
Haraka will be unable to update them after changes.
Haraka reads a number of configuration files at startup. Any files read in a plugins register() function are read _before_ Haraka drops privileges. Be sure that Haraka's user/group has permission to read these files else Haraka will be unable to update them after changes.
2 changes: 1 addition & 1 deletion config.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

const path = require('path')

const cfreader = require('./configfile')
const cfreader = require('./lib/reader')

class Config {
constructor(root_path, no_overrides) {
Expand Down
116 changes: 11 additions & 105 deletions configfile.js → lib/reader.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@ const fs = require('node:fs')
const fsp = require('node:fs/promises')
const path = require('node:path')

const watch = require('./watch')

let config_dir_candidates = [
// these work when this file is loaded as require('./config.js')
path.join(__dirname, 'config'), // Haraka ./config dir
__dirname, // npm packaged plugins
path.join(__dirname, '..', 'config'), // Haraka ./config dir
path.join(__dirname, '..'), // npm packaged plugins
]

class cfreader {
Expand All @@ -21,20 +22,6 @@ class cfreader {
this._overrides = {}

this.get_path_to_config_dir()

// for "ini" type files
this.regex = {
section: /^\s*\[\s*([^\]]*?)\s*\]\s*$/,
param: /^\s*([\w@:._\-/[\]]+)\s*(?:=\s*(.*?)\s*)?$/,
comment: /^\s*[;#].*$/,
line: /^\s*(.*?)\s*$/,
blank: /^\s*$/,
continuation: /\\[ \t]*$/,
is_integer: /^-?\d+$/,
is_float: /^-?\d+\.\d+$/,
is_truth: /^(?:true|yes|ok|enabled|on|1)$/i,
is_array: /(.+)\[\]$/,
}
}

get_path_to_config_dir() {
Expand All @@ -45,8 +32,8 @@ class cfreader {
}

if (process.env.NODE_ENV === 'test') {
// loaded by haraka-config/test/*
this.config_path = path.join(__dirname, 'test', 'config')
// console.log(`loaded by haraka-config/test/*`)
this.config_path = path.join(__dirname, '..', 'test', 'config')
return
}

Expand Down Expand Up @@ -127,64 +114,6 @@ class cfreader {
}
}

watch_dir() {
// NOTE: Has OS platform limitations:
// https://nodejs.org/api/fs.html#fs_fs_watch_filename_options_listener
const cp = this.config_path
if (this._watchers[cp]) return

try {
this._watchers[cp] = fs.watch(
cp,
{ persistent: false },
(fse, filename) => {
if (!filename) return
const full_path = path.join(cp, filename)
if (!this._read_args[full_path]) return
const args = this._read_args[full_path]
if (args.options && args.options.no_watch) return
if (this._sedation_timers[filename]) {
clearTimeout(this._sedation_timers[filename])
}
this._sedation_timers[filename] = setTimeout(() => {
console.log(`Reloading file: ${full_path}`)
this.load_config(full_path, args.type, args.options)
delete this._sedation_timers[filename]
if (typeof args.cb === 'function') args.cb()
}, 5 * 1000)
},
)
} catch (e) {
console.error(`Error watching directory ${cp}(${e})`)
}
return
}

watch_file(name, type, cb, options) {
// This works on all OS's, but watch_dir() above is preferred for Linux and
// Windows as it is far more efficient.
// NOTE: we need a fs.watch per file. It's impossible to watch non-existent
// files. Instead, note which files we attempted
// to watch that returned ENOENT and fs.stat each periodically
if (this._watchers[name] || (options && options.no_watch)) return

try {
this._watchers[name] = fs.watch(
name,
{ persistent: false },
this.on_watch_event(name, type, options, cb),
)
} catch (e) {
if (e.code !== 'ENOENT') {
// ignore error when ENOENT
console.error(`Error watching config file: ${name} : ${e}`)
} else {
this._enoent.files[name] = true
this.ensure_enoent_timer()
}
}
}

get_cache_key(name, options) {
// Ignore options etc. if this is an overriden value
if (this._overrides[name]) return name
Expand Down Expand Up @@ -233,11 +162,11 @@ class cfreader {
case 'win32':
case 'win64':
case 'linux':
this.watch_dir()
watch.dir(this)
break
default:
// All other operating systems
this.watch_file(name, type, cb, options)
watch.file(this, name, type, cb, options)
}

return result
Expand All @@ -264,7 +193,7 @@ class cfreader {
.then(resolve)
.catch(reject)

if (opts.watchCb) this.fsWatchDir(name)
if (opts.watchCb) watch.dir2(this, name)
})
}

Expand Down Expand Up @@ -326,7 +255,7 @@ class cfreader {
try {
switch (type) {
case 'ini':
result = cfrType.load(name, options, this.regex)
result = cfrType.load(name, options)
break
case 'hjson':
case 'json':
Expand All @@ -336,7 +265,7 @@ class cfreader {
break
// case 'binary':
default:
result = cfrType.load(name, type, options, this.regex)
result = cfrType.load(name, type, options)
}
this._config_cache[cache_key] = result
} catch (err) {
Expand Down Expand Up @@ -373,29 +302,6 @@ class cfreader {
this._config_cache[path.join(cp, fn)] = result[key]
}
}

fsWatchDir(dirPath) {
if (this._watchers[dirPath]) return
const watchOpts = { persistent: false, recursive: true }

// recursive is only supported on Windows (win32, win64) and macOS (darwin)
if (!/win/.test(process.platform)) watchOpts.recursive = false

this._watchers[dirPath] = fs.watch(dirPath, watchOpts, (fse, filename) => {
// console.log(`event: ${fse}, ${filename}`);
if (!filename) return
const full_path = path.join(dirPath, filename)
const args = this._read_args[dirPath]
// console.log(args);
if (this._sedation_timers[full_path]) {
clearTimeout(this._sedation_timers[full_path])
}
this._sedation_timers[full_path] = setTimeout(() => {
delete this._sedation_timers[full_path]
args.opts.watchCb()
}, 2 * 1000)
})
}
}

module.exports = new cfreader()
0 readers/binary.js → lib/readers/binary.js
View file
File renamed without changes.
4 changes: 3 additions & 1 deletion readers/flat.js → lib/readers/flat.js
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
'use strict'

const regex = require('../regex')

exports.load = (...args) => {
return this.parseValue(
...args,
Expand All @@ -14,7 +16,7 @@ exports.loadPromise = async (...args) => {
)
}

exports.parseValue = (name, type, options, regex, data) => {
exports.parseValue = (name, type, options, data) => {
let result = []

if (type === 'data') {
Expand Down
0 readers/hjson.js → lib/readers/hjson.js
View file
File renamed without changes.
Loading

0 comments on commit 924e070

Please sign in to comment.