Search Indexer

These are a group of Lambda functions that are designed to be triggered by webhooks on our self-hosted Ghost instances. These functions are created using the Serverless framework and are used to index articles from various news sources into a universal search bar.

Here are the current Ghost events / webhooks, and their endpoints:

Ghost Event	Endpoint
Post published	.../stage/add-index
Post unpublished	.../stage/delete-index
Post deleted	.../stage/deleted-index
Published post updated	.../stage/update-index

The Lambda function on each endpoint receives the blog post from the Ghost webhook and updates the correct Algolia index based on the origin of the Ghost webhook.

Here are the currently configured origins and Algolia indices:

Ghost Origin	Algolia index
http://localhost:2368	news-dev
https://www.freecodecamp.org/news/	news
https://www.freecodecamp.org/espanol/news/	news-es
https://chinese.freecodecamp.org/news/	news-zh
https://www.freecodecamp.org/portuguese/news/	news-pt-br
https://www.freecodecamp.org/italian/news/	news-it
https://www.freecodecamp.org/japanese/news/	news-ja
https://www.freecodecamp.org/arabic/news/	news-ar
https://www.freecodecamp.org/bengali/news/	news-bn
https://www.freecodecamp.org/urdu/news/	news-ur
https://www.freecodecamp.org/swahili/news/	news-sw
https://www.freecodecamp.org/korean/news/	news-ko
https://www.freecodecamp.org/turkish/news/	news-tr
https://www.freecodecamp.org/french/news/	news-fr
https://www.freecodecamp.org/hindi/news/	news-hi
https://www.freecodecamp.org/german/news/	news-de
https://www.freecodecamp.org/vietnamese/news/	news-vi
https://www.freecodecamp.org/indonesian/news/	news-id
https://www.freecodecamp.org/haitian/news/	news-ht

Prerequisites:

• AWS CLI and an AWS account with access to Secrets Manager
• An Algolia account access to the pre-configured indices above (see table)

How to prepare your local machine

1. Create an Algolia account with access to the freeCodeCamp News indices
2. Go to LastPass, search for the shared [keys] [AWS] [Lambda] - News Indexer note, and add the [news-indexer-lambda] credentials to ~/.aws/credentials
3. Clone this repo and install the necessary packages with npm ci
4. Copy sample.env.yml to a new file named .env.yml
5. In .env.yml set AWS_PROFILE to 'news-indexer-lambda'
6. Start developing

How to develop and test locally

1. Make your changes and run npm start
2. Open your local Ghost Integrations dashboard at http://localhost:2368/ghost/#/settings/integrations
3. Click "Add custom integration" to create a new custom integration
4. Check the console output and add the Lambda function URLs (ex: https://localhost:3000/dev/add-index) to the correct Ghost events. Check the Ghost events and endpoints table near the top of this document for more information
5. Publish, update, and delete articles on Ghost locally. You can view your changes on Algolia on the dev application in the news-dev index
6. If you make changes while the server is running, kill the server with Ctrl / Cmd + C. Then restart the server with npm start to test your changes

How to deploy changes

1. After a PR with changes is accepted and merged, deploy changes with npm run deploy-dev or npm run deploy

How to change Algolia API keys on AWS

Generate a new key on Algolia

1. Sign into Algolia with the [email protected] account
2. Select the dev or Production application from the dropdown at the top of the page
3. Click "API Keys", then click the "All API Keys" tab near the top of the screen
4. Click the "New API Key" button, and generate a new key with the following ACLs (action control lists): search, addObject, deleteObject, browse, listIndexes, deleteIndex
5. Click the "Create" button
6. Copy the new Algolia API key

Update the Algolia key entry in AWS Secrets Manager

1. Create an IAM account under the freeCodeCamp AWS team with access to Secrets Manager
2. Sign in and go to https://us-east-2.console.aws.amazon.com/secretsmanager/
3. Click "serverless-indexer"
4. Click "Retrieve secret value" and click the "Edit" button
5. Paste the new Algolia API key in as the value of ALGOLIA_ADMIN_KEY in either the dev or prod object
6. Click the "Save" button

How to remove a service

WARNING: Removing a dev or prod service is only necessary in a few cases, such as migrating to a new region.

By removing a service, the base URL for the Lambda functions will be lost. This means that all the webhooks in each Ghost instance will need to be updated once the service is redeployed.

To deploy updates, it's not necessary to remove a service first. Just follow the deployment instructions here.

Use the following commands with caution:

1. Install Serverless globally with npm install -g serverless
2. Run serverless remove --stage [dev / prod] --region us-east-2