Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jekyll build/deploy workflow #86

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

0xdevalias
Copy link
Owner

@0xdevalias 0xdevalias commented Jul 26, 2020

Sets up a GitHub actions/workflow to build and deploy the Jekyll site to Github Pages.

@0xdevalias 0xdevalias self-assigned this Jul 26, 2020
@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

I'm not actually sure what the best method for testing GitHub actions/workflows is without just deploying to prod.. I found one project claiming to be able to:

But in attempting to run it in dryrun mode, it looked like it would fail:

⇒  act -n
*DRYRUN* [Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
*DRYRUN* [Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1
*DRYRUN* [Build and deploy Jekyll site/jekyll] Unable to resolve v1: reference not found
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ❌  Failure - 💎 setup ruby
Error: reference not found

I expect if we used an exact version tag it would probably work.. but that seems at odds with how other examples seem to imply GitHub works?

Changing ruby/setup-ruby@v1 to ruby/[email protected] gave the following output, implying it should work correctly:

⇒  act -n
*DRYRUN* [Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
*DRYRUN* [Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1.40.0
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 🔨 install dependencies & build site
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/limjh16/jekyll-action-ts' # ref=v2
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 🔨 install dependencies & build site

Yet when we run it properly, we hit an issue based on the docker image that is being used not matching perfectly to how GitHub actions runners are:

⇒  act
[Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
[Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Build and deploy Jekyll site/jekyll]   🐳  docker cp src=/Users/devalias/dev/_sites/devalias.net/. dst=/github/workspace
[Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
[Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
[Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
[Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1.40.0
[Build and deploy Jekyll site/jekyll]   🐳  docker cp src=/Users/devalias/.cache/act/[email protected] dst=/actions/
[Build and deploy Jekyll site/jekyll]   ❗  ::error::ENOENT: no such file or directory, open '/etc/lsb-release'
[Build and deploy Jekyll site/jekyll]   ❌  Failure - 💎 setup ruby
Error: exit with `FAILURE`: 1

So it seems we may not actually have a good way to be able to test this locally..

@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

The build failed since the runner doesn't have the appropriate gsl libraries installed (required to speed up --lsi)

2020-07-26T03:00:40.4975849Z Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
2020-07-26T03:00:40.4976473Z 
2020-07-26T03:00:40.4977046Z current directory:
2020-07-26T03:00:40.4978110Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/gsl-2.1.0.3/ext/gsl_native
2020-07-26T03:00:40.4978850Z /home/runner/.rubies/ruby-2.7.1/bin/ruby -I
2020-07-26T03:00:40.4979471Z /home/runner/.rubies/ruby-2.7.1/lib/ruby/2.7.0 -r
2020-07-26T03:00:40.4980071Z ./siteconf20200726-2689-qeqmg6.rb extconf.rb
2020-07-26T03:00:40.4980461Z *** ERROR: missing required library to compile this module: No such file or
2020-07-26T03:00:40.4980973Z directory - gsl-config
2020-07-26T03:00:40.4981343Z *** extconf.rb failed ***
2020-07-26T03:00:40.4981669Z Could not create Makefile due to some reason, probably lack of necessary
2020-07-26T03:00:40.4982013Z libraries and/or headers.  Check the mkmf.log file for more details.  You may
2020-07-26T03:00:40.4982324Z need configuration options.
2020-07-26T03:00:40.4982586Z 
2020-07-26T03:00:40.4982892Z Provided configuration options:
2020-07-26T03:00:40.4983385Z 	--with-opt-dir
2020-07-26T03:00:40.4983931Z 	--without-opt-dir
2020-07-26T03:00:40.4984471Z 	--with-opt-include
2020-07-26T03:00:40.4985036Z 	--without-opt-include=${opt-dir}/include
2020-07-26T03:00:40.4985572Z 	--with-opt-lib
2020-07-26T03:00:40.4986125Z 	--without-opt-lib=${opt-dir}/lib
2020-07-26T03:00:40.4986662Z 	--with-make-prog
2020-07-26T03:00:40.4987192Z 	--without-make-prog
2020-07-26T03:00:40.4987764Z 	--srcdir=.
2020-07-26T03:00:40.4988311Z 	--curdir
2020-07-26T03:00:40.4988896Z 	--ruby=/home/runner/.rubies/ruby-2.7.1/bin/$(RUBY_BASE_NAME)
2020-07-26T03:00:40.4989442Z 	--with-gsl-version
2020-07-26T03:00:40.4989740Z 
2020-07-26T03:00:40.4990056Z extconf failed, exit code 1
2020-07-26T03:00:40.4990316Z 
2020-07-26T03:00:40.4990620Z Gem files will remain installed in
2020-07-26T03:00:40.4991186Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/gsl-2.1.0.3
2020-07-26T03:00:40.4991576Z for inspection.
2020-07-26T03:00:40.4991888Z Results logged to
2020-07-26T03:00:40.4992519Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/gsl-2.1.0.3/gem_make.out
2020-07-26T03:00:40.4992877Z 
2020-07-26T03:00:40.4993259Z An error occurred while installing gsl (2.1.0.3), and Bundler cannot continue.
2020-07-26T03:00:40.4994249Z Make sure that `gem install gsl -v '2.1.0.3' --source 'https://rubygems.org/'`
2020-07-26T03:00:40.4994949Z succeeds before bundling.
2020-07-26T03:00:40.4995256Z 
2020-07-26T03:00:40.4995583Z In Gemfile:
2020-07-26T03:00:40.4995896Z   gsl

Apparently we can install ubuntu packages as part of the workflow though, so if we install the gsl libraries, this probably should work:

- name: install system dependencies
  run: sudo apt-get install libgsl-dev

@0xdevalias 0xdevalias linked an issue Jul 26, 2020 that may be closed by this pull request
@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

This build shows that installing the libgsl-dev system dependency works, but appears to take ~10sec to complete, at least on the first run. If this remains the same on subsequent builds, we may need to consider what the cost/benefit of this extra timing is, and/or alternatives we could use (eg. caching, prebuilt docker container, etc)


We also need to consider the install order, so that gsl knows to use nmatrix:

  Post-install message from gsl:
  
      gsl can be installed with or without narray support. Please install
      narray before and reinstall gsl if it is missing.
  
      gsl is also now compatible with NMatrix. Please install nmatrix before
      installing gsl.

The rb-gsl installation notes claim that we can forcibly tell it to use nmatrix:

In order to use rb-gsl with NMatrix you must first set the NMATRIX environment variable and then install rb-gsl:
gem install nmatrix export NMATRIX=1 gem install rb-gsl

This will compile rb-gsl with NMatrix specific functions.


As currently setup, even using --lsi, it looks as though the site only took ~9sec to generate.. would be interesting to see how that changes if we remove gsl and/or nmatrix:

jekyll build
  /home/runner/.rubies/ruby-2.7.1/bin/bundle exec jekyll build -s /home/runner/work/devalias.net/devalias.net/ --lsi
  Configuration file: /home/runner/work/devalias.net/devalias.net/_config.yml
              Source: /home/runner/work/devalias.net/devalias.net
         Destination: /home/runner/work/devalias.net/devalias.net/_site
   Incremental build: disabled. Enable with --incremental
        Generating... 
     GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
         Jekyll Feed: Generating feed for posts
                      done in 8.475 seconds.
   Auto-regeneration: disabled. Use --watch to enable.
  Took   9.13 seconds

To fix the follow warning, we need to set the appropriate environment variables for the Jekyll GitHub Metadata plugin:

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

Can we use ${{ secrets.GITHUB_TOKEN }} for this? (no, see #86 (comment))

@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

Even when we removed nmatrix, our site build still seemed to take only ~9sec.

Unsure if this is because gsl wasn't actually using it before, or if it just has a negligible impact on our site build time.


Also, by using the cached gems, rather than having to re-install them, our total site dependencies/build step went from ~4min 51sec down to just ~16sec.

@0xdevalias
Copy link
Owner Author

Even when we removed gsl, our site build still seemed to take only ~8sec.

Unsure if this is because gsl just has a negligible impact on our site build time.

@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

Even when we removed Jekyll's --lsi option, the site build still only seemed to take ~7sec.

So maybe --lsi either isn't working, or just doesn't really have a big impact on our site build time.

In light of this.. i'm thinking we can leave the --lsi option enabled for now, but can probably remove the gsl/nmatrix optimisations we had added. Though we should probably do this in a follow up PR (see #87)

This also most likely renders #83 as irrelevant

(Edit: as was pointed out on #83, I may have disabled related_posts on my site a while ago, which is likely why the changes here weren't doing anything. Theres a lot of good info in #83 and linked to from #87 as well.. so worth exploring that more deeply before just removing it all)

@0xdevalias 0xdevalias force-pushed the jekyll-build-deploy-workflow branch from f7406f5 to e05b0b2 Compare July 26, 2020 04:14
@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

The build using secrets.GITHUB_TOKEN for JEKYLL_GITHUB_TOKEN failed with the following error during the site build:

2020-07-26T04:19:36.2417867Z �[31m  Liquid Exception: GET https://api.github.com/gists/1395926: 403 - Resource not accessible by integration // See: https://developer.github.com/v3/gists/#get-a-single-gist in /home/runner/work/devalias.net/devalias.net/_posts/2013-06-17-gists-on-tumblr.md�[0m
2020-07-26T04:19:36.2418778Z �[31m                    ------------------------------------------------�[0m
2020-07-26T04:19:36.2419520Z �[31m      Jekyll 4.1.1   Please append `--trace` to the `build` command �[0m
2020-07-26T04:19:36.2420234Z �[31m                     for any additional information or backtrace. �[0m
2020-07-26T04:19:36.2420916Z �[31m                    ------------------------------------------------�[0m
2020-07-26T04:19:36.2422515Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/octokit-4.18.0/lib/octokit/response/raise_error.rb:16:in `on_complete': GET https://api.github.com/gists/1395926: 403 - Resource not accessible by integration // See: https://developer.github.com/v3/gists/#get-a-single-gist (Octokit::Forbidden)

This appears to be caused by jekyll-gist:

  • https://github.com/jekyll/jekyll-gist
    • Pro-tip: If you provide a personal access token with Gist scope, as the environmental variable JEKYLL_GITHUB_TOKEN, Jekyll Gist will use the Gist API to speed up site generation.

This same env var will also be used by the jekyll-github-metadata plugin, as per:


Looking at the token I currently use for this on my local machine, it seems to only have the repo.public_repo scope enabled.

So perhaps creating a similar token, and configuring it as a GitHub Action secret that we then pass to this env var would work:

eg. ${{ secrets.JEKYLL_GITHUB_TOKEN_PAT }}

Edit: Thoughts from 2023 now that I know more about GitHub actions.. is there any reason why I can't just use the action token for this?

@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 26, 2020

When we added the deploy step to a test branch, the build seemed to complete successfully, taking only ~6sec to deploy to the test gh-pages branch.


The commit message was just deploy: 41e8aad, which differs from our previous message style of Update site: Wed 22 Jul 2020 15:19:29 AEST

actions-gh-pages lets us customise this using the commit_message or full_commit_message arguments, though i'm currently unsure how we would generate a similar formatted date to pass to it:

There appear to be some actions related to getting the current time, unsure if there is a simpler way:

Since most of these just seem to use core.setOutput, we may be able to do similar by using the workflow command echo syntax, and running directly on the host runner:

Maybe something like:

- id: currentdate
  name: Get the current date and export it for later steps to use
  run: echo "::set-output name=date_str::$(date)"
  #run: echo "::set-output name=date_str::`date`"

@0xdevalias
Copy link
Owner Author

The build testing whether we can get a formatted date as step output (without using external actions) seemed to work:

image

      - id: currentdate
        name: 📅 get current date and time
        run: echo "::set-output name=date_str::$(date)"

      - name: 📅 check if date output was set correctly
        run: echo ${{steps.currentdate.outputs.date_str}}

Therefore, we can use them to customise the commit message for the generated site builds as per #86 (comment):

actions-gh-pages lets us customise this using the commit_message or full_commit_message arguments, though i'm currently unsure how we would generate a similar formatted date to pass to it:

I think we want to use something like commit_message: ${{ steps.currentdate.outputs.date_str }}

Copy link
Owner Author

@0xdevalias 0xdevalias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes required before this PR can be merged.

push:
branches:
- master
- jekyll-build-deploy-workflow
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be removed before this PR is merged, and/or a better solution for testing in non-master PR's implemented (that won't override our production site deployment)

- jekyll-build-deploy-workflow
paths-ignore:
- .cache/**
# - .github/**
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uncomment this?

Comment on lines +33 to +34
- name: 🔨 install system dependencies
run: sudo apt-get install libgsl-dev
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be cleaned up as part of #87

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we even want to clean it up anymore.. see #87 and #83 for more context/details.

uses: limjh16/jekyll-action-ts@v2
with:
enable_cache: true
# format_output: true
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to enable this? Otherwise, remove it.

with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./_site
publish_branch: jekyll-build-deploy-workflow-gh-pages
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be changed to the proper gh-pages branch before this PR is merged, and/or a better solution for testing in non-master PR's implemented (that won't override our production site deployment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dockerize Jekyll + automate site builds
1 participant