chore(ci): add direnv virtual env integration

[flrgh]

This adds an .envrc template to integrate local development with direnv workflows. I have been using a similar workflow with a self-maintained .envrc file for some time, and it works great!

It's not fully 1-1 with how the bash/fish env files work due to direnv not supporting exported functions, but I think it's fine for now. This could be solved by refactoring some things in ./scripts/dependency_services, but I want to keep the PR small and simple.

[fffonion]

Could you explain a bit more on what are the differences of using direnv compared to source directly? (Would it work if you directly source bazel-bin/build/kong-dev-venv/lib/venv-commons which only contains environment variables i.e. add a symlink of .envrc that links to venv-commons?)

[flrgh]

@fffonion

Could you explain a bit more on what are the differences of using direnv compared to source directly?

So, direnv is quite different from just source-ing a file from within an interactive bash shell. It's a hook that lives within your bash $PROMPT_COMMAND and watches for changes to $PWD. When you change directories, and a file at $PWD/.envrc exists, it executes the file in a "special" shell environment which A) injects some helper functions and B) captures changes to exported variables. This way it is able to reset things automatically when you change to a different directory.

The end result is something I have found to be much more ergonomic than pure bash for "virtual" workspace environments.

Here's a synopsis/workflow comparison:

direnv workflow

# ./workspace/.envrc file
export MY_VAR=123
export PATH=$PWD/bin:$PATH

# init
$ echo $MY_VAR       # => <nothing> (it's unset)
$ echo $PATH         # => /usr/bin:/usr/local/bin
$ echo $MY_OTHER_VAR # => <nothing> (it's unset)

# changing to the workspace directory
$ cd ./workspace

# direnv automatically ingests ./workspace/.envrc
$ echo $MY_VAR # => 123
$ echo $PATH   # => /path/to/workspace/bin:/usr/bin:/usr/local/bin

# update .envrc with a new var
$ echo 'export MY_OTHER_VAR=456' >> .envrc

# direnv detected the change to .envrc and automatically reloaded it
$ echo $MY_OTHER_VAR # => 456

# changing back out to the parent dir
$ cd ..

# direnv automatically resets variables to their prior state
$ echo $MY_VAR       # => <nothing> (it's unset)
$ echo $PATH         # => /usr/bin:/usr/local/bin
$ echo $MY_OTHER_VAR # => <nothing> (it's unset)

pure bash workflow

# ./workspace/.env file
OLD_MY_VAR=$MY_VAR; export MY_VAR=123
OLD_PATH=$PATH; export PATH=$PWD/bin:$PATH

deactivate() {
  # this might not be 100% correct since we didn't check whether or not
  # MY_VAR was declared, and empty/unset != undeclared
  MY_VAR=$OLD_MY_VAR
  PATH=$OLD_PATH
  unset -f deactivate
}

# init
$ echo $MY_VAR       # => <nothing> (it's unset)
$ echo $PATH         # => /usr/bin:/usr/local/bin
$ echo $MY_OTHER_VAR # => <nothing> (it's unset)

# changing to the workspace directory
$ cd ./workspace

# manually source env file to initialize
$ source ./.env
$ echo $MY_VAR # => 123
$ echo $PATH   # => /path/to/workspace/bin:/usr/bin:/usr/local/bin

# update .env with a new var
$ echo 'export MY_OTHER_VAR=456' >> .env

# must manually reload .env to discover MY_OTHER_VAR
$ echo $MY_OTHER_VAR # => <nothing> (it's unset)
$ source ./.env
$ echo $MY_OTHER_VAR # => 456

# change out of workspace directory--this does nothing for now
$ cd ..

# explicitly reset w/ deactivate function
# OOPS!
#   1) when we re-sourced .env we updated the "old" values of
#      MY_VAR and PATH, so they weren't reset properly
#   2) we forgot to follow the save/reset pattern for MY_OTHER_VAR,
#      so it wasn't reset at all 
$ deactivate
$ echo $MY_VAR       # => 123
$ echo $PATH         # => /path/to/workspace/bin:/usr/bin:/usr/local/bin
$ echo $MY_OTHER_VAR # => 456

---

The fact that direnv watches for changes to .envrc files is especially helpful. For instance, given this example that I added to build/README.md:

# /path/to/kong/.envrc
source_env_if_exists bazel-bin/build/kong-dev-venv.envrc

Aside from initially setting up the virtual environment when you cd into /path/to/kong, direnv will add bazel-bin/build/kong-dev-venv.envrc to its list of watched files. The side effect of this is that any time you run bazel build //build:venv, direnv will catch the change and automatically update your environment.

[fffonion]

I see, so it's like automatically handling activate and deactive the "virtual env". As right now in this PR we are just exporting env variables, will it work for just linking the existing file venv-commons that does env exporting to .envrc so that direnv picks it up automagically?
I'm generally not opposed to having more dev patterns, but it seems like we are duplicating the logic twice (correct me if i'm wrong).

(Or simply source_env_if_exists bazel-bin/build/kong-dev-venv/lib/venv-commons?)

[flrgh]

@fffonion understood. I think some of the duplicate logic could be reduced, but I didn't want to touch venv-commons too much from the outset because I wanted to keep the PR non-invasive.

Let me see if I can come up with something a little cleaner. Some refactoring of venv-commons to better support the direnv use case could reduce the amount of "new" shell code.