Merge branch 'main' into clickhouse-test-regression

.gitignore

@@ -1,3 +1,5 @@
 .direnv
 # created when `just test <service>` is used
 /result
+# created when `just run <service>` is used
+/test/data

README.md

@@ -1,18 +1,14 @@
+[![project chat](https://img.shields.io/badge/zulip-join_chat-brightgreen.svg)](https://nixos.zulipchat.com/#narrow/stream/414011-services-flake)
+
 # services-flake
 
-NixOS-like services for Nix flakes, as a [process-compose-flake](https://github.com/Platonic-Systems/process-compose-flake) module (based on flake-parts).
+Declarative, composable, and reproducible services for Nix development environment, as a [process-compose-flake](https://github.com/Platonic-Systems/process-compose-flake) module (based on [flake-parts](https://flake.parts)). Enabling users to have NixOS-like service on MacOS and Linux.
 
-![](./doc/services-flake/demo.gif)
+![Demo](./doc/demo.gif)
 
 ## Getting Started
 
-```sh
-$ nix flake new --template github:juspay/services-flake ./my-project
-$ cd my-project
-$ nix run
-```
-
-Or see `./test/flake.nix`
+See <https://community.flake.parts/services-flake/start>
 
 ## Services available
 
@@ -26,10 +22,9 @@ The `dataDir` of these services tend to take *relative* paths, which are usually
 
 To discuss the project, please [join our Zulip](https://nixos.zulipchat.com/#narrow/stream/414011-services-flake).
 
-## Contributing
+## Contributing & Development
 
-- If you are adding a *new* service, see https://github.com/cachix/devenv/tree/main/src/modules/services for inspiration.
-- For contributing to docs, see https://github.com/flake-parts/community.flake.parts#guidelines-for-writing-docs
+See <https://community.flake.parts/services-flake/contributing>
 
 ## Credits
 
@@ -39,4 +34,4 @@ Thanks to [the devenv project](https://github.com/cachix/devenv/tree/main/src/mo
 
 ### Why not re-use devenv service modules?
 
-This is currently not possible (nor prioritized by the devenv project), which is why we must create our own services. See https://github.com/cachix/devenv/issues/75
+This is currently not possible (nor prioritized by the devenv project), which is why we must create our own services. See <https://github.com/cachix/devenv/issues/75>

doc/contributing.md

@@ -0,0 +1,75 @@
+---
+order: 10
+---
+
+# Contributing
+
+{#devshell}
+## Development Shell
+
+A Nix dev shell is available, providing `nixpkgs-fmt` and `just`. To enter the dev shell, run:
+
+```sh
+nix develop .#dev
+```
+
+An `.envrc` is also provided, so it is recommended to use `direnv` to automatically enter the dev shell when you `cd` into the project directory. See [this tutorial](https://nixos.asia/en/direnv).
+
+{#new-service}
+## Adding a new service
+
+The project repository is structure to make addition of new services easy. Here's how to add a new service:
+
+> [!info]
+> See <https://github.com/cachix/devenv/tree/main/src/modules/services> for inspiration.
+> 
+> If you don't find a new service there, see <https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/services>.
+
+- Create a new file `./nix/<service-name>.nix` file (see [./nix/redis.nix](https://github.com/juspay/services-flake/blob/main/nix/redis.nix) for inspiration)
+- Add the service to the list in [./nix/default.nix](https://github.com/juspay/services-flake/blob/main/nix/default.nix).
+- Create a new test file `./nix/<service-name>_test.nix` (see [./nix/redis_test.nix](https://github.com/juspay/services-flake/blob/main/nix/redis_test.nix)).
+- Add the test to [./test/flake.nix](https://github.com/juspay/services-flake/blob/main/test/flake.nix).
+
+{#run-service}
+### Run the service
+
+```sh
+just run <service-name>
+```
+
+{#run-tests}
+### Run the tests for the service
+
+The previous command will run the services but not the tests. To run the tests, use:
+
+```sh
+just test <service-name>
+```
+
+or test all services:
+
+```sh
+just test-all
+```
+
+{#service-doc}
+### Add documentation for the new service
+
+It is important to add documentation along with any new services you are contributing. Create a new file `./doc/<service-name>.md` (see [[clickhouse]] for example) and add the service to the list in [[services]].
+
+> [!note]
+> It is recommended to add documentation for non-trivial tasks. For example, grafana documentation mentions [how to change the default database backend](https://community.flake.parts/services-flake/grafana#change-database).
+
+
+
+{#docs}
+## Documentation
+
+For contributing to docs, see <https://github.com/flake-parts/community.flake.parts#guidelines-for-writing-docs>
+
+We use [emanote](https://emanote.srid.ca/) to render our documentation. The source files are in the `doc` directory. To run the docs, use:
+
+```sh
+just doc # Or, `cd doc && nix run`
+```
+

doc/grafana.md

@@ -0,0 +1,42 @@
+# Grafana
+
+[Grafana open source](https://grafana.com/docs/grafana/latest/) is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics, logs, and traces no matter where they are stored. It provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations.
+
+## Getting Started
+
+```nix
+# In `perSystem.process-compose.<name>`
+{
+  services.grafana."gf1".enable = true;
+}
+```
+
+{#tips}
+## Tips & Tricks
+
+{#change-database}
+### Changing Grafana database
+
+By default, Grafana stores data in the `sqlite3` [database](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#database). It also supports `mysql` and `postgres`.
+
+To change the database to `postgres`, we can use the following config:
+
+```nix
+{
+  services.postgres.pg1 = {
+    enable = true;
+    listen_addresses = "127.0.0.1";
+    initialScript.after = "CREATE USER root SUPERUSER;";
+  };
+  services.grafana.gf1 = {
+    enable = true;
+    extraConf.database = with config.services.postgres.pg1; {
+      type = "postgres";
+      host = "${listen_addresses}:${builtins.toString port}";
+      name = "postgres"; # database name
+    };
+  };
+  settings.processes."gf1".depends_on."pg1".condition = "process_healthy";
+  };
+}
+```

doc/index.md

@@ -9,9 +9,9 @@ emanote:
 
 # Running services using `services-flake`
 
-[services-flake][gh] is a [flake-parts](https://flake.parts/) module providing [NixOS-like services](https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/services) for [Nix flakes](https://nixos.asia/en/flakes), enabling the users to use same configuration for local development across platforms.
+[services-flake][gh] provides declarative, composable, and reproducible services for Nix development environment, as a [process-compose-flake](https://github.com/Platonic-Systems/process-compose-flake) module (based on [flake-parts](https://flake.parts)). Enabling users to have NixOS-like service on MacOS and Linux.
 
-It builds on top of the [process-compose-flake](https://community.flake.parts/process-compose-flake) module which allows running arbitrary processes in the devShell environment.
+It builds on top of the [process-compose-flake](https://community.flake.parts/process-compose-flake) module which allows running arbitrary processes declared in Nix.
 
 See:
 - [[start]]#

doc/services.md

@@ -8,14 +8,15 @@ short-title: Services
 > This list denotes the progress of documentation, not implementation. See full list of implemented services at [gh].
 
 - [ ] Apache Kafka
-- [ ] [[clickhouse]]#
+- [x] [[clickhouse]]#
 - [ ] Elasticsearch
 - [ ] MySQL
 - [ ] Nginx
 - [ ] PostgreSQL
 - [ ] Redis
 - [ ] Redis Cluster
 - [ ] Zookeeper
+- [x] [[grafana]]#
 - [ ] ...
 
 [gh]: https://github.com/juspay/services-flake

justfile

@@ -1,3 +1,4 @@
+# List all the just commands
 default:
     @just --list
 
@@ -17,3 +18,11 @@ test-all:
 # Run native test for a specific service
 test service:
     nix build ./test#checks.$(nix eval --impure --expr "builtins.currentSystem").{{service}} --override-input services-flake . -L
+
+# Run doc server with hot-reload
+doc:
+    cd ./doc && nix run
+
+# Run service whose configuration is defined in `<service>_test.nix`
+run service:
+    cd test && nix run .#{{service}} --override-input services-flake ../

nix/clickhouse/default.nix

@@ -27,6 +27,28 @@ in
       description = "The clickhouse data directory";
     };
 
+    defaultExtraConfig = lib.mkOption {
+      type = yamlFormat.type;
+      internal = true;
+      readOnly = true;
+      default = {
+        logger.level = "warning";
+        logger.console = 1;
+        default_profile = "default";
+        default_database = "default";
+        tcp_port = toString config.port;
+        path = "${config.dataDir}/clickhouse";
+        tmp_path = "${config.dataDir}/clickhouse/tmp";
+        user_files_path = "${config.dataDir}/clickhouse/user_files";
+        format_schema_path = "${config.dataDir}/clickhouse/format_schemas";
+        user_directories = {
+          users_xml = {
+            path = "${config.package}/etc/clickhouse-server/users.xml";
+          };
+        };
+      };
+    };
+
     extraConfig = lib.mkOption {
       type = yamlFormat.type;
       description = "Additional configuration to be appended to `clickhouse-config.yaml`.";
@@ -75,24 +97,7 @@ in
         processes =
           let
             clickhouseConfig = yamlFormat.generate "clickhouse-config.yaml" (
-              {
-                logger = {
-                  level = "warning";
-                  console = 1;
-                };
-                tcp_port = "${toString config.port}";
-                default_profile = "default";
-                default_database = "default";
-                path = "${config.dataDir}/clickhouse";
-                tmp_path = "${config.dataDir}/clickhouse/tmp";
-                user_files_path = "${config.dataDir}/clickhouse/user_files";
-                format_schema_path = "${config.dataDir}/clickhouse/format_schemas";
-                user_directories = {
-                  users_xml = {
-                    path = "${config.package}/etc/clickhouse-server/users.xml";
-                  };
-                };
-              } // config.extraConfig
+              lib.recursiveUpdate config.defaultExtraConfig config.extraConfig
             );
           in
           {

nix/default.nix

@@ -13,5 +13,6 @@ in
     ./redis-cluster.nix
     ./redis.nix
     ./zookeeper.nix
+    ./grafana.nix
   ];
 }