These are a group of Lambda functions that are designed to be triggered by webhooks on our self-hosted Ghost instances. These functions are created using the Serverless framework and are used to index articles from various news sources into a universal search bar.
Here are the current Ghost events / webhooks, and their endpoints:
Ghost Event | Endpoint |
---|---|
Post published | .../stage/add-index |
Post unpublished | .../stage/delete-index |
Post deleted | .../stage/deleted-index |
Published post updated | .../stage/update-index |
The Lambda function on each endpoint receives the blog post from the Ghost webhook and updates the correct Algolia index based on the origin of the Ghost webhook.
Here are the currently configured origins and Algolia indices:
Ghost Origin | Algolia index |
---|---|
http://localhost:2368 | news-dev |
https://www.freecodecamp.org/news/ | news |
https://www.freecodecamp.org/espanol/news/ | news-es |
https://chinese.freecodecamp.org/news/ | news-zh |
https://www.freecodecamp.org/portuguese/news/ | news-pt-br |
https://www.freecodecamp.org/italian/news/ | news-it |
https://www.freecodecamp.org/japanese/news/ | news-ja |
https://www.freecodecamp.org/arabic/news/ | news-ar |
https://www.freecodecamp.org/bengali/news/ | news-bn |
https://www.freecodecamp.org/urdu/news/ | news-ur |
https://www.freecodecamp.org/swahili/news/ | news-sw |
https://www.freecodecamp.org/korean/news/ | news-ko |
https://www.freecodecamp.org/turkish/news/ | news-tr |
https://www.freecodecamp.org/french/news/ | news-fr |
https://www.freecodecamp.org/hindi/news/ | news-hi |
https://www.freecodecamp.org/german/news/ | news-de |
https://www.freecodecamp.org/vietnamese/news/ | news-vi |
https://www.freecodecamp.org/indonesian/news/ | news-id |
https://www.freecodecamp.org/haitian/news/ | news-ht |
Prerequisites:
- AWS CLI and an AWS account with access to Secrets Manager
- An Algolia account access to the pre-configured indices above (see table)
- Create an Algolia account with access to the freeCodeCamp News indices
- Go to LastPass, search for the shared
[keys] [AWS] [Lambda] - News Indexer
note, and add the[news-indexer-lambda]
credentials to~/.aws/credentials
- Clone this repo and install the necessary packages with
npm ci
- Copy
sample.env.yml
to a new file named.env.yml
- In
.env.yml
setAWS_PROFILE
to'news-indexer-lambda'
- Start developing
- Make your changes and run
npm start
- Open your local Ghost Integrations dashboard at http://localhost:2368/ghost/#/settings/integrations
- Click "Add custom integration" to create a new custom integration
- Check the console output and add the Lambda function URLs (ex: https://localhost:3000/dev/add-index) to the correct Ghost events. Check the Ghost events and endpoints table near the top of this document for more information
- Publish, update, and delete articles on Ghost locally. You can view your changes on Algolia on the
dev
application in thenews-dev
index - If you make changes while the server is running, kill the server with Ctrl / Cmd + C. Then restart the server with
npm start
to test your changes
- After a PR with changes is accepted and merged, deploy changes with
npm run deploy-dev
ornpm run deploy
- Sign into Algolia with the [email protected] account
- Select the
dev
orProduction
application from the dropdown at the top of the page - Click "API Keys", then click the "All API Keys" tab near the top of the screen
- Click the "New API Key" button, and generate a new key with the following ACLs (action control lists):
search
,addObject
,deleteObject
,browse
,listIndexes
,deleteIndex
- Click the "Create" button
- Copy the new Algolia API key
- Create an IAM account under the freeCodeCamp AWS team with access to Secrets Manager
- Sign in and go to https://us-east-2.console.aws.amazon.com/secretsmanager/
- Click "serverless-indexer"
- Click "Retrieve secret value" and click the "Edit" button
- Paste the new Algolia API key in as the value of
ALGOLIA_ADMIN_KEY
in either thedev
orprod
object - Click the "Save" button
WARNING: Removing a dev
or prod
service is only necessary in a few cases, such as migrating to a new region.
By removing a service, the base URL for the Lambda functions will be lost. This means that all the webhooks in each Ghost instance will need to be updated once the service is redeployed.
To deploy updates, it's not necessary to remove a service first. Just follow the deployment instructions here.
Use the following commands with caution:
- Install Serverless globally with
npm install -g serverless
- Run
serverless remove --stage [dev / prod] --region us-east-2