configuration.markdown

Table of Contents

• Configuration
  • Configuration classes
  • Adding new configuration
  • Mocking configuration in tests

---

Configuration

Apps get configured via environment variables that get set for every app's container. Apps derived from the common app share a set of global configuration environment variables (defined at the top of production's docker-compose.dist.yml, in the x-common-configuration section), and some apps have their own local configuration environment variables.

All names of the Media Cloud-specific configuration environment variables are prefixed with MC_ (e.g. MC_DOWNLOADS_CACHE_S3).

Configuration classes

To avoid having apps to read and parse configuration environment variables directly, said variables are encapsulated in a set of classes:

• CommonConfig (available in mediawords.util.config.common Python module and MediaWords::Util::Config::Common Perl package) and its internal classes (e.g. DatabaseConfig, AmazonS3DownloadsConfig, …) provide global configuration that is used by multiple apps, e.g. PostgreSQL and RabbitMQ credentials, list of domains for which we should authenticate using HTTP auth, raw download storage options, etc.
• Every app might have its own local configuration classes that encapsulate only the configuration environment variables that get set to that specific app's container. For example, facebook-fetch-story-stats app has FacebookConfig class (available in facebook_fetch_story_stats.config Python module and MediaWords::Util::Config::Facebook Perl package) that exposes Facebook API credentials to the rest of the app.

Every configuration environment variable gets exposed through a static method in one of the configuration classes, so it can be accessed both statically and non-statically:

from mediawords.util.config.common import CommonConfig

# Accessing statically
send_email_from = CommonConfig.email_from_address()

# Accessing non-statically
common_config = CommonConfig()
send_email_from = common_config.email_from_address()

Adding new configuration

Even though configuration gets passed to app containers as simple environment variables and thus could be accessed directly from within the app (e.g. using os.environ in Python or %ENV in Perl), it is highly recommended to implement new configuration variables as static methods in either the global configuration classes (CommonConfig and friends), or, better yet (and if possible), local configuration classes (e.g. FacebookConfig) because:

• having configuration in one place makes it easier for others to know what does (or might) get configured and how;
• configuration classes can be made to raise an exception if a certain configuration variable is unset;
• configuration classes can (and do) parse string configuration environment variable values into final structures that could be used more easily; for example, a string list of HTTP-authenticated domains gets parsed in a list of AuthenticatedDomain objects.

Try to limit the number of global configuration environment variables to the minimum, and add new global configuration variables only if a lot of containers will tend to use them. Even if two apps use the same configuration environment variable, introduction of the new global configuration environment variable can be avoided by employing YAML anchors in docker-compose.yml, e.g.:

version: "3.7"

x-service1-service2-config: &service1-service2-config
  # Local configuration environment variable that will be set in both "service1" and "service2"
  MC_SERVICE_FOO: "bar"

services:

  service1:
    environment:
      <<: *service1-service2-config
      MC_SERVICE1_ABC: "def"

  service2:
    environment:
      <<: *service1-service2-config
      MC_SERVICE2_GHI: "jkl"

Mocking configuration in tests

Sometimes you might want to test how does your code behave when configuration is set to certain custom values. To achieve that, it is recommended that you:

1. Make the tested function / class accept a configuration object as a parameter;
2. Create a subclass of one of the configuration classes and override the static method that exposes a certain configuration variable;
3. Create a custom object of the newly added test configuration class and pass it to the tested function subroutine.

For example, if you were to test how does the query_solr() function behaves when it's made to run its queries against a mock Solr server (available at http://localhost:1234/solr/), you would make the tested function accept the configuration object as its parameter with fallback to the default configuration:

# Function that is being tested
def query_solr(params: str, config: Optional[CommonConfig] = None) -> str:

    # Fallback to default configuration if custom one is unset
    if not config:
        config = CommonConfig()
    
    # Do the querying and stuff using Solr URL from the configuration object
    get_url(config.solr_url() + "?" + params)

Then, in the test, create a custom CommonConfig subclass and override the solr_url() static method for it to return your custom value; lastly, pass the object of the custom configuration to the tested function:

def test_query_solr():
    
    # Custom configuration class
    class MockSolrURLCommonConfig(CommonConfig):

        # Override the static method for it to return a mock Solr URL
        @staticmethod
        def solr_url():
            return "http://localhost:1234/solr/"
    
    # Pass an object of the custom configuration to the tested method
    assert query_solr(
        params='q=abc&foo=bar',
        config=CommonConfig(),
    ) == 'expected results'