From e77572c1889e0f9f05dd5baea94c9d90f9ed9e05 Mon Sep 17 00:00:00 2001 From: Chris Wilton-Magras Date: Fri, 18 Oct 2024 15:16:49 +0100 Subject: [PATCH 1/2] fix: docker build and run corrected Also fixed for Git bash on Windows --- Gemfile | 5 +++++ README.md | 33 ++++++++++++++++++--------------- shell/docker-dev-watch.sh | 6 +++--- shell/docker-gem-install.sh | 6 ++++-- 4 files changed, 30 insertions(+), 20 deletions(-) diff --git a/Gemfile b/Gemfile index 95ff141e3f..037c726322 100644 --- a/Gemfile +++ b/Gemfile @@ -5,3 +5,8 @@ gem 'github-pages' gem "tzinfo-data", "~> 1.2022" gem "webrick", "~> 1.7" + +# Issue with ffi requiring a very recent rubygems version, not yet available +# in many current linux docker images, so this must be locked down for now. +# See https://github.com/ffi/ffi/issues/1103 +gem "ffi", "< 1.17.0" diff --git a/README.md b/README.md index 2f79697bea..0101284041 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ _Ensure that that your [SSH configuration][github-ssh] will also let you connect If you wish to develop changes to the blog locally, you may find that there's a lot of content, and prefer just to download the bits you need. -```bash +```shell # see comment above about configuring SSH, and modify the clone URL accordingly to use the correct SSH identity # you may also consider forking the blog repository, and cloning your personal fork instead git clone --depth 1 --filter=blob:none --no-checkout git@github.com:ScottLogic/blog.git @@ -71,7 +71,7 @@ First, install Ruby and (if on Linux) a few build dependencies for Nokogiri. On Linux: -```bash +```shell sudo apt-get install ruby2.3 ruby2.3-dev build-essential dh-autoreconf libxslt-dev libxml2-dev zlib1g-dev ``` @@ -84,14 +84,14 @@ Secondly, update Gem and install the Jekyll, Bundler and Nokogiri gems. On Linux: -```bash +```shell sudo gem update sudo gem install jekyll bundler nokogiri ``` On Windows, in a PowerShell instance with elevated priveleges: -```bash +```shell gem update gem install jekyll bundler nokogiri ``` @@ -99,7 +99,7 @@ gem install jekyll bundler nokogiri Thirdly, configure Bundler to store project dependencies in `vendor/bundle`, and, when in the root directory of your clone of the blog, install the project dependencies. -```bash +```shell bundle config path vendor/bundle cd PATH/TO/BLOG bundle install @@ -112,7 +112,7 @@ Finally, run `jekyll -v` to check whether Jekyll is working. If so, you're good Once you've got all the prerequisites for your operating system, you can run the blog. Navigate to the root directory of your clone of the blog and execute Jekyll using Bundler. -```bash +```shell bundle exec jekyll serve ``` @@ -120,33 +120,36 @@ The blog will then be available on [localhost][localhost]. If you need to re-compile the scripts or SCSS, you can use the NPM scripts. -```bash -npm install +```shell +npm ci npm run scripts npm run style ``` ### Docker -Use a bash-compatible shell. +Use a bash-compatible shell; Git bash on Windows should work fine. **Install gem dependencies** -First, output gem dependencies to a directory `container_gem_cache` on our host machine: +First, we output gem dependencies to directory `container_gem_cache` on the host machine: -```bash +```shell ./shell/docker-gem-install.sh ``` **Run dev watch** -Now we can serve the blog: +Now we can serve the blog with live reloading. Replace "jbloggs" with your ScottLogic username: -```bash -BLOG_USERNAME=abirch ./shell/docker-dev-watch.sh +```shell +BLOG_USERNAME=jbloggs ./shell/docker-dev-watch.sh ``` -Visit the blog on [localhost][localhost]. +It'll take a while to build first time (up to 5 minutes), and you'll likely see a fair few warnings in the terminal for +older posts, but once it's done you should see message "done in XXX.YYY seconds". + +Navigate to [localhost][localhost] in your browser. ## CI/CD diff --git a/shell/docker-dev-watch.sh b/shell/docker-dev-watch.sh index 09a0615197..1e0fb379dc 100755 --- a/shell/docker-dev-watch.sh +++ b/shell/docker-dev-watch.sh @@ -2,7 +2,8 @@ set -eo pipefail SCRIPTDIR="$(dirname "$0")" -REPOROOT="$(realpath "$SCRIPTDIR/..")" +# Replace e.g. /c/ of Git Bash paths on windows with C:/ to keep Docker happy +REPOROOT="$(realpath "$SCRIPTDIR/.." | sed 's/^\/\([a-z]\)\//\u\1:\//')" CONTAINER_WORKDIR=/srv/jekyll @@ -52,7 +53,6 @@ set -x # run jekyll serve # https://jekyllrb.com/docs/configuration/options/ -# TODO: consider serving with --incremental if updates are slow exec docker run -it --rm --init \ --name sl-jekyll-run \ -v "$REPOROOT/$BLOG_USERNAME":"$CONTAINER_WORKDIR/"$BLOG_USERNAME":ro" \ @@ -62,4 +62,4 @@ exec docker run -it --rm --init \ -p "$PORT":"$PORT" \ -p 35729:35729 \ -p 3000:3000 \ -jekyll/jekyll:3.8.6 jekyll serve --livereload -d /dist --port "$PORT" \ No newline at end of file +jekyll/jekyll:3.8.6 jekyll serve --livereload -d /dist --port "$PORT" --incremental \ No newline at end of file diff --git a/shell/docker-gem-install.sh b/shell/docker-gem-install.sh index 0d9a2856c8..130cedd90c 100755 --- a/shell/docker-gem-install.sh +++ b/shell/docker-gem-install.sh @@ -2,7 +2,9 @@ set -eo pipefail SCRIPTDIR="$(dirname "$0")" -REPOROOT="$(realpath "$SCRIPTDIR/..")" +# Replace e.g. /c/ of Git Bash paths on windows with C:/ to keep Docker happy +# Sed is pretty nasty huh ;) +REPOROOT="$(realpath "$SCRIPTDIR/.." | sed 's/^\/\([a-z]\)\//\u\1:\//')" CONTAINER_WORKDIR=/srv/jekyll @@ -19,4 +21,4 @@ exec docker run -it --rm --init \ -v "$REPOROOT/Gemfile":"$CONTAINER_WORKDIR/Gemfile":ro \ -v "$REPOROOT/Gemfile.lock":"$CONTAINER_WORKDIR/Gemfile.lock" \ -v "$REPOROOT/container_gem_cache":/usr/local/bundle \ -jekyll/jekyll:3.8.6 bundle \ No newline at end of file +jekyll/jekyll:3.8.6 bundle From f1c73090d33b40475ee49d3c01cb44108df9dc51 Mon Sep 17 00:00:00 2001 From: Chris Wilton-Magras Date: Mon, 21 Oct 2024 11:50:52 +0100 Subject: [PATCH 2/2] Updated README with more comprehensive instructions --- README.md | 109 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 70 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 0101284041..d33bc152ed 100644 --- a/README.md +++ b/README.md @@ -1,51 +1,84 @@ # Scott Logic Blogs -See below for technical details of the blog creation stack and, -e.g., instructions on how to install and run the blog site locally. +If you are looking to write a blog post, then you don't _need_ to read any further, as you can author posts using +SiteLeaf: see our [company intranet][confluence-getting-started] for instructions on how to create a new page and view +it before publication on the blog. -Note that if you're looking to **author a blog post**, then you don't need to read any further! -Instead, please see our [company extranet][confluence-getting-started] -for instructions on how to create a new page and view it before publication on the blog. +The below instructions allow more control over the process, though they do require a smidgen of git knowledge and a +GitHub account. ## Technical Stack The blog is a static website, designed to be hosted on [GitHub pages][github-pages]. -The underlying content is generated through a series of Ruby gems and libraries, starting with a dedicated github-pages [gem][ruby-github-pages]. +The underlying content is generated through a series of Ruby gems and libraries, starting with a dedicated github-pages +[gem][ruby-github-pages]. Within that stack, [Jekyll][jekyll-docs] is used as the static content generation engine, consuming template files written in either **HTML** or **Markdown** (syntax extended by [Kramdown][kramdown-syntax]). - Common content or structure can be further injected or managed using the [Liquid][ruby-liquid] templating language. -## Cloning the repository +## Authoring + +To write a blog post, it is recommended to fork the repo, then perform a sparse checkout that excludes all previous +posts and all authors other than yourself. This will result in the quickest build. -_[Sparse checkout][sparse-checkout-guide] requires Git 2.25.0_ +The alternative is to perform a full checkout and build all posts (more than 15 years' worth!) which can take five +minutes or more to complete. If you are working on technical issues or improvements, then you may need some blog posts +to see the result of your changes; in this case, there is little alternative but to perform a full checkout. -_Ensure that that your [SSH configuration][github-ssh] will also let you connect to [private GitHub repositories][github-ssh-multiple-accounts]._ +### Fork the repository -If you wish to develop changes to the blog locally, you may find that there's a lot of content, and prefer just to download the bits you need. +At the top of the [ScottLogic blog GitHub repo][scottlogic-blog-repo] you will find the "fork" button. You +will need to be logged into your GitHub account first, then clicking the button will generate your own fork and navigate +to the resulting fork. Then click the <> Code button at the top to see +your clone options; HTTPS clone is the simplest. Copy the URL, then: ```shell -# see comment above about configuring SSH, and modify the clone URL accordingly to use the correct SSH identity -# you may also consider forking the blog repository, and cloning your personal fork instead -git clone --depth 1 --filter=blob:none --no-checkout git@github.com:ScottLogic/blog.git +# Clone your fork without checking anything out: +git clone --depth 1 --filter=blob:none --no-checkout YOUR-REPO-URL-HERE cd blog -git sparse-checkout init --cone -# if you want to write blog posts, modify this variable with the author name you -# wish to write posts under (typically derived from your SL email address) -AUTHOR='abirch' -git sparse-checkout set _includes _layouts _data category scripts scss assets "$AUTHOR" + +# Optional: tell sparse checkout what to include (excludes _posts by default). +# Use your Scott Logic username in the below command: +git sparse-checkout set _data _includes _layouts assets category scripts scss shell YOUR-SL-USERNAME-HERE git checkout gh-pages ``` -This gets the repository down to ~8MB and ~150 files (whereas checking out all authors' posts would take hundreds of megabytes). +### First-time authors + +If this is your first post, you'll need to set yourself up as an author. For this, you will need a directory in the repo +root named after your Scott Logic username. Within this you will need a set of files: `atom.xml`, `feed.xml`, +`index.html`, and an image file of yourself. Just copy an existing author's files and modify their contents: it should +be obvious what needs changing. Then add yourself to `_data/authors.yml`, again using an existing author as a template. +You will need to add + +- an entry under `authors` +- your username under `active-authors` + +Finally, if you performed a _sparse checkout_ as recommended, you will need to add directory `_posts` in the root of +your local copy. -## Run local copy of blog (for blog devs only) +### Adding a new post -__NOTE__: Instructions are work in progress. +Below is a summary for getting started; for more comprehensive instructions on authoring posts, including markdown +syntax and linking to images, see the pages on our [company intranet][confluence-getting-started]. -If you plan to use Docker, then you can [skip ahead][install-docker] now! +Within the `_posts` directory, add a markdown file for your new post, named with date and title similar to existing +posts, e.g. `2024-10-31-Your-snappy-post-title.md`. Copy the headers from a recent post, and modify the values for +yours. You may add as many tags (keywords) as you like, but a maximum of two Categories; see `_data/categories.yml` for +our current categories. + +Note that Jekyll will not display your post unless the header date is in the past, so if you do not see your post when +running locally, check the date (and time) first. As you can probably guess, this is how you can set a "Go Live" date +in the future, if you don't want your post to appear immediately. + +Once you have your skeleton file in place, you can run the blog and start writing. Saving changes should trigger a +rebuild. + +### Run the blog locally + +By far the easiest route is to use Docker: if you have it installed, you can [skip ahead][run-docker] now! The blog consists of static HTML pages with content generated using: - [github-pages][ruby-github-pages] for deployment hooks @@ -63,8 +96,6 @@ then using Bundler should bring in most of the dependencies automatically. However, due to Nokogiri's reliance on Native XML parsers you may require additional steps. Thorough instructions for setting up your development environment are detailed below. -### Native environment - #### Prerequisites First, install Ruby and (if on Linux) a few build dependencies for Nokogiri. @@ -89,7 +120,7 @@ sudo gem update sudo gem install jekyll bundler nokogiri ``` -On Windows, in a PowerShell instance with elevated priveleges: +On Windows, in a PowerShell instance with elevated privileges: ```shell gem update @@ -118,7 +149,7 @@ bundle exec jekyll serve The blog will then be available on [localhost][localhost]. -If you need to re-compile the scripts or SCSS, you can use the NPM scripts. +If you are working on fixes or new features, and need to re-compile the scripts or SCSS, you can use these npm scripts: ```shell npm ci @@ -126,19 +157,20 @@ npm run scripts npm run style ``` -### Docker +### Running with Docker Use a bash-compatible shell; Git bash on Windows should work fine. -**Install gem dependencies** +#### Install gem dependencies -First, we output gem dependencies to directory `container_gem_cache` on the host machine: +First, we output gem dependencies to directory `container_gem_cache` on the host machine. This is analogous to running +"npm install" for an npm package: ```shell ./shell/docker-gem-install.sh ``` -**Run dev watch** +#### Run in watch mode Now we can serve the blog with live reloading. Replace "jbloggs" with your ScottLogic username: @@ -146,10 +178,11 @@ Now we can serve the blog with live reloading. Replace "jbloggs" with your Scott BLOG_USERNAME=jbloggs ./shell/docker-dev-watch.sh ``` -It'll take a while to build first time (up to 5 minutes), and you'll likely see a fair few warnings in the terminal for -older posts, but once it's done you should see message "done in XXX.YYY seconds". +It'll take a while to build first time, but once it's done you should see message "done in XXX.YYY seconds". +Then you can navigate to [localhost][localhost] in your browser. -Navigate to [localhost][localhost] in your browser. +Note that if you performed a _sparse checkout_ as recommended, and if this is your first post, then you won't see any +blog posts when the site loads unless you've already added a file for your new blog post. ## CI/CD @@ -189,11 +222,10 @@ changes. This workflow runs only on a manual dispatch on the `gh-pages` branch. [calibreapp-image-actions]: https://github.com/calibreapp/image-actions [confluence-getting-started]: https://scottlogic.atlassian.net/wiki/spaces/INT/pages/3577479175/Getting+started+with+the+Scott+Logic+blog [sparse-checkout-guide]: https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/#sparse-checkout-and-partial-clones -[github-ssh]: https://docs.github.com/en/authentication/connecting-to-github-with-ssh -[github-ssh-multiple-accounts]: https://gist.github.com/oanhnn/80a89405ab9023894df7 [github-pages]: https://pages.github.com/ [github-pages-docs]: https://docs.github.com/en/pages +[run-docker]: #running-with-docker [jekyll-docs]: https://jekyllrb.com/docs/ [kramdown-syntax]: https://kramdown.gettalong.org/syntax.html [localhost]: http://localhost:4000 @@ -205,6 +237,5 @@ changes. This workflow runs only on a manual dispatch on the `gh-pages` branch. [ruby-liquid]: https://shopify.github.io/liquid/ [ruby-downloads]: https://www.ruby-lang.org/en/downloads/ [pa11y-ci]: https://github.com/pa11y/pa11y-ci -[project-gemfile]: Gemfile -[install-docker]: #docker - +[project-gemfile]: ./Gemfile +[scottlogic-blog-repo]: https://github.com/ScottLogic/blog