diff --git a/.env_sample b/.env_sample
new file mode 100644
index 0000000..88895c2
--- /dev/null
+++ b/.env_sample
@@ -0,0 +1,19 @@
+# APP_HOST is the host of your app, if you deploy the app change to your domain.
+export APP_HOST=localhost
+
+# GITHUB_APP_ID is found in settings of your GitHub app under General > About
+export GITHUB_APP_ID=YOUR_GITHUB_APP_ID
+
+# GITHUB_APP_NAME what you called your app.
+export GITHUB_APP_NAME=NAME_OF_THE_GITHUB_APP
+
+# SECRET_KEY_BASE is required for Auth Cookie:
+export SECRET_KEY_BASE=run:mix_phx.gen.secret-in-terminal-and-paste-result-here
+
+# AWS:
+export AWS_ACCESS_KEY_ID=aws-access-key-id
+export AWS_SECRET_ACCESS_KEY=super-secret-key
+export S3_BUCKET_NAME=your-bucket-name
+
+# PRIVATE_KEY should be generated in your GitHub App settings
+# please see instructions in README for how to add it to .env file
diff --git a/.gitignore b/.gitignore
index 2f3cd6a..9cfedf3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -27,3 +27,4 @@ npm-debug.log
/config/*.secret.exs
.env
cover
+*.pem
diff --git a/README.md b/README.md
index 9b32ad4..72bdb24 100644
--- a/README.md
+++ b/README.md
@@ -1,170 +1,508 @@
-# github-backup
-:back: :arrow_up: A script to backup your GitHub Account/Organisation so you can still work when (they/you are) offline.
+# GitHub Backup!
+
+An App that helps you **backup** your **GitHub Issues**
+`so that` you can still **work** when (you/they are) ***offline***.
+
+
+
+
+[](https://travis-ci.org/dwyl/github-backup)
+[](http://inch-ci.org/github/dwyl/github-backup)
+[](http://codecov.io/github/dwyl/github-backup?branch=master)
+[](https://beta.hexfaktor.org/github/dwyl/github-backup)
+[](https://github.com/dwyl/github-backup)
+
+
## Why?
-As a person who uses GitHub as their "_single-source-of-truth_"
-I need a backup of GitHub issues
-so that I can work "offline".
-(_either when I have no network or GH is "down"_)
+`As a person` (_`team of people`_) that uses **GitHub** as their
+["***single-source-of-truth***"](https://en.wikipedia.org/wiki/Single_source_of_truth)
+I `need` a backup of GitHub issues
+`so that` I can see **issue history** and/or **work** "**offline**".
+(_either when I have no network or GH is "temporarily offline"_)
## What?
-Backup GitHub issues.
-
-+ a script that lets you login to GitHub using OAuth
-+ Use GitHub REST API to retrieve all issues
-+ Store issues as "flat files" in subdirectories corresponding to the GitHub URL
-e.g: if the issue is `dwyl/github-reference/issues/15`
-store it as `dwyl/github-reference/issues/15.json`
-+ It does not need to "scale" it just needs to work. read: http://paulgraham.com/ds.html
+GitHub (Issue) Backup is an App that lets you (_unsurprisingly_):
-Note: initially we do not need a Web-UI to let people backup their issues,
-but later we could build UI to make it more useable.
+1. **Store a backup** of the content of the **GitHub issue(s)**
+for a given project/repo.
+2. Track the changes/edits made to issue description and comments.
+see: [dear-github/issues/129](https://github.com/dear-github/dear-github/issues/129)
+3. View all your issues when you or GitHub are
+["offline"](https://github.com/dwyl/github-reference/issues/15)
## How?
-+ Elixir (see https://github.com/dwyl/learn-elixir to learn Elixir)
-+ Phoenix (see https://github.com/dwyl/learn-phoenix-framework to learn Phoenix)
-+ Tachyons (see https://github.com/dwyl/learn-tachyons to learn Tachyons)
+The app is built using the **Phoenix Web Framework**
+(_using the Elixir programming language_)
+and Tachyons UI ("Functional CSS") system.
+While you can _run_ the App without knowing Elixir or Phoenix
+(_see instructions below_),
+if you want to _understand_ how it works,
+we _recommend_ taking the time to **learn** each item on the list.
+
+If you are `new` to Phoenix, Elixir,
+Tachyons (_the UI_) or ngrok,
+we have a "_beginner tutorials_"
+which will bring you up-to-speed:
+
++ Elixir:
+[github.com/dwyl/**learn-elixir**](https://github.com/dwyl/learn-elixir)
++ Phoenix:
+[github.com/dwyl/**learn-phoenix-framework**](https://github.com/dwyl/learn-phoenix-framework)
++ Tachyons:
+[github.com/dwyl/**learn-tachyons**](https://github.com/dwyl/learn-tachyons)
++ ngrok: [github.com/dwyl/**learn-ngrok**](https://github.com/dwyl/learn-ngrok)
+
+_Additionally_ we use Amazon Web Services (***AWS***)
+Simple Storage Service (***S3***)
+for storing the issue comment _history_
+with "_high availability_" and reliability.
+To run `github-backup` on your `localhost`
+you will need to have an AWS account and an S3 "bucket".
+(_instructions given below_).
+
+
+
+
+## Set Up _Checklist_ on `localhost`
+
+This will take approximately **15 minutes** to complete.
+(_provided you already have a GitHub and AWS account_)
+
+### Install Dependencies
+
+To run the App on your `localhost`,
+you will need to have **4 dependencies** _installed_:
+
++ **Elixir**:
+https://github.com/dwyl/learn-elixir#installation
++ **PostgreSQL**:
+https://github.com/dwyl/learn-postgresql#installation
++ **Phoenix**:
+https://hexdocs.pm/phoenix/installation.html
++ **ngrok**: https://github.com/dwyl/learn-ngrok#1-download--install
+(_so you can share the app on your `localhost` with GitHub via `public` URL_)
+
+Once all the necessary dependencies are installed,
+we can move on.
+(_if you get stuck,
+ [please let us know](https://github.com/dwyl/github-backup/issues)_!)
+
+### Clone the App from GitHub
+
+Run the following the command to `clone` the app from GitHub:
+```sh
+git clone https://github.com/dwyl/github-backup.git && cd github-backup
+```
+
+### Required Environment Variables
+
+To run the project on your `localhost`,
+you will need to have the following Environment Variables defined.
+
+> _**Note**: if you are `new` to "Environment Variables",
+**please read**_:
+[github.com/dwyl/**learn-environment-variables**](https://github.com/dwyl/learn-environment-variables)
+
++ `APP_HOST` - the hostname for your app e.g: `localhost` or `gitbu.io`.
++ `PRIVATE_KEY` - The RSA private key for your GitHub App.
+_See below for how to set this up_.
++ `GITHUB_APP_ID` - The unique `id` of your GitHub App. _See below_.
++ `GITHUB_APP_NAME` - the name of your GitHub App. _See below_ (_Step 1_).
++ `SECRET_KEY_BASE` - a 64bit string used by Phoenix for security
+(_to sign cookies and CSRF tokens_). See below for how to _generate_ yours.
++ `AWS_ACCESS_KEY_ID` - your AWS access key (_get this from your AWS settings_)
++ `AWS_SECRET_ACCESS_KEY` - your AWS secrete access key
++ `S3_BUCKET_NAME` - name of the AWS S3 "bucket" where issue comments
+will be stored.
+
+
+#### Copy The `.env_sample` File
+
+The _easy_ way manage your Environment Variables _locally_
+is to have a `.env` file in the _root_ of the project.
+
+_Copy_ the _sample_ one and update the variables:
+
+```sh
+cp .env_sample .env
+```
+
+Now update the _values_ to the _real_ ones for your App.
+You will need to _register_ a GitHub app,
+so that is what we will do _below_!
+
+> _Don't worry, your sensitive variables are **safe**
+as the `.env` file is "ignored" in the_
+[`.gitignore` file](https://github.com/dwyl/github-backup/blob/778d1e311a37564721989e842412880994eb2b5d/.gitignore#L28)
+
+
+
+### Create a GitHub Application
+
+The role of the Github application is to
+send **notifications** when **events** occur in your repositories.
+For example you can be notified
+when **new issues** or **pull requests** are ***opened***.
+
+> _**Note**: we have **simplified**
+the steps in the "official" Github guide
+to create a new Github App_: https://developer.github.com/apps/building-github-apps/creating-a-github-app
+
+#### 1. Access the New Application Settings Page
+
+While logged into your GitHub Account, visit:
+https://github.com/settings/apps/new
+Or _navigate_ the following menus:
+```
+Settings -> Developer settings -> Github Apps -> New Github App
+```
+
+
+
+
+
+1. **Github App name**: The name of the app; must be unique,
+so can't be "gh-backup" as that's taken!
+2. **Descriptions**: A short description of the app; "My backup app"
+3. **Homepage URL**: The website of the app. e.g: "https://gitbu.io"
+ (_yours will be different_)
+4. **User authorization callback URL**: Redirect url after user authentication e.g."http://localhost:4000/auth/github/callback".
+This is not needed for backup so this field can be left empty.
+5. **Setup URL** (optional): Redirect the user to this url after installation,
+not needed for `github-backup`.
+6. **Webhook URL**: the URL where HTTP `POST` requests from Github are sent.
+The endpoint in the `github-backup` App is `/event/new`,
+however Github won't be able to send requests
+to `http://localhost:4000/event/new`
+as this url is only accessible on your `localhost`.
+To allow GitHub to access your `localhost` server you can use `ngrok`.
+**Remember to update these value after you have a running server on your machine!**
+7. **Webhook secret** (optional) - leave this blank for now.
+see: [github-backup/issues/76](https://github.com/dwyl/github-backup/issues/76)
+(_please let us know your thoughts on how/when we should prioritise this!_)
+
+Then in your terminal enter `ngrok http 4000`
+to generate an SSH "tunnel" between your localhost:4000 and ngrok.io.
+Copy the ngrok url that appears in your terminal
+to the Github app configuration; e.g: "http://bf541ce5.ngrok.io/event/new"
+
+> _NOTE: you will need to update the webhook URL each time you
+disconnect/connect to ngrok because a different URL is generated every time._
+
+You can read more about webhooks and `ngrok` at:
+https://developer.github.com/webhooks/configuring
+
+#### 2. Set the Necessary Permissions
+
+Still on the same page, scroll down below the initial setup,
+to the ***Permissions*** section:
+
+
+
+1. Define the access rights for the application on the permission section.
+**Change "issues" to "Read & write"**
+(_this is required to update "meta table" which links to the "issue history"_)
+
+
+#### 3. Subscribe to Events
+
+Scroll down to the "***Subscribe to events***" section
+and check the boxes for ***Issue comment*** and ***Issues***:
+
+
+
+#### 4. Where can this GitHub App be installed?
+
+Scroll down to the "**_Where_ can this GitHub App be installed?**" section.
+
+
+
+In _our_ case we want to be able to use GitHub Backup for a number of projects
+in different organisations, so we selected ***Any account***.
+but if you only want to backup your _personal_ projects,
+then select ***Only on this account***.
+
+#### 5. Click on the `Create GitHub App` Button!
+
+_Finally_, with everything configured, click on "Create Github App"!
+
+
+
+You should see a message confirming that your app was created:
+
+
+(_obviously your app will will have a different name in the url/path..._)
+
+#### 6. Generate Private Key
-## Set up
+Click on the _link_ to "***generate a private key***".
-The project is hosted on heroku at: https://github-backup.herokuapp.com/
+
-To run it locally, you will need to:
+When you click the `Generate private key` button it should _download_
+a file containing your private key.
-- Create a new Github application.
-- Run a phoenix server on your machine.
+
-You'll need to have installed [Elixir](https://elixir-lang.org/install.html), [Phoenix](https://hexdocs.pm/phoenix/installation.html), and [ngrok](https://ngrok.com/download) if you haven't already.
+Open the file in your text editor, e.g:
+
-> _**Note**: **only** `try` to run this on your computer once
-you've understood Elixir & Phoenix._
+> _**Don't worry**, this is **not** our "**real**" private key ...
+it's just for demonstration purposes.
+but this is what your RSA private key will
+**look** like, a block of random characters ..._
-#### Create a GitHub application
+The downloaded file contains the private key.
+Save this file in the root
+of the `github-backup` repo on your `localhost`.
-The role of the Github application is to send notifications
-when events occur on your repositories.
-For example you can get a notification when new issues or pull requests are open.
+For _example_ in _our_ case the GitHub app is called "**gitbu**",
+so our Private Key file starts with the app name
+and the _date_ the key was generated.
+e.g: `gitbu.2018-03-23.private-key.pem`
-- Access the [new application settings page](https://github.com/settings/apps/new) on your Github account:
- Settings -> Developer settings -> Github Apps -> New Github App
- 
+> _**Don't worry**, all `.pem` (private key) files
+are ignored in the `.gitignore` file so your file will stay private._
+Once you have copied the file into your project root, run the following command:
+```sh
+echo "export PRIVATE_KEY='`cat ./gitbu.2018-03-23.private-key.pem`'" >> .env
+```
+Replace the `gitbu.2018-03-23.private-key.pem` part in the command
+with the name of _your_ private key file
+e.g: `my-github-backup-app.2018-04-01.private-key.pem`
-- Github App name: The name of the app; must be unique, so can't be "gh-backup" as that's taken!
-- Descriptions: A short description of the app; "My backup app"
-- Homepage URL: The website of the app: "https://dwyl.com/"
-- User authorization callback URL: Redirect url after user authentication e.g."http://localhost:4000/auth/github/callback". This is not needed for backup so this field can be left empty.
-- Setup URL: Redirect the user to this url after installation, not needed for `github-backup`
-- Webhook URL: URL where post requests from Github are sent to. The endpoint is ```/event/new```, however Github won't be able to send requests to ```http://localhost:4000/event/new``` as this url is only accessible by your own machine. To expose publicly your `localhost` server you can use `ngrok`. **Remember to update this value after you have a running server on your machine!**
+That will create an entry in the `.env` file for your `PRIVATE_KEY`
+environment variable.
- Install [ngrok](https://ngrok.com). If you have homebrew, you can do this by running `brew cask install ngrok`
- Then in your terminal enter `ngrok http 4000` to generate an SSH between your localhost:4000 and ngrok.io. Copy the ngrok url that appears in your terminal to the Github app configuration; "http://bf541ce5.ngrok.io/event/new"
+#### 7. Copy the App Name and App `ID` from GitHub App Settings Page
- > _NOTE: you will need to update the webhook URL everytime you disconnect/connect to ngrok because a different URL is generated everytime._
+In the "**Basic information**" section of your app's settings page,
+copy the GitHub App name and paste it into your `.env` file after the `=` sign
+for the `GITHUB_APP_NAME` variable.
- You can read more about webhooks and ngrok at https://developer.github.com/webhooks/configuring/
-- Define the access rights for the application on the permmission section. **Change "issues" to "Read only"**
+
-
+Scroll downd to the "About" section you will find the `ID` of your app e.g: 103
+this is the number that needs to be set for `GITHUB_APP_ID` in your `.env` file.
+
-- Select which events ```dwylbot``` will be notified on. **Check "issues" and "issue comment"**
- 
-- Webhook secret: This token can be used to make sure that the requests received by `github-backup` are from Github; `github-backup` doesn't use this token at the moment so you can keep this field empty (see https://developer.github.com/webhooks/securing/)
-- You can decide to allow other users to install the Github Application or to limit the app on your account only:
- 
- You can now click on "Create Github App"!
+#### Generate the `SECRET_KEY_BASE`
-- Create a private key: This key is used to identify specific `github-backup` installations
+Run the following command to generate a new phoenix secret key:
- 
+```sh
+mix phx.gen.secret
+```
+_copy-paste_ the _output_ (64bit `String`)
+into your `.env` file after the "equals sign" on the line for `SECRET_KEY_BASE`:
+```yml
+export SECRET_KEY_BASE=YourSecreteKeyBaseGeneratedUsing-mix_phx.gen.secret
+```
- The downloaded file contains the private key.
- Copy this key in your environment variables, see the next section.
+#### S3 Bucket
+In order to save the Issue Comments (_which are a variable length "rich text"_),
+we save these as `.json` in an S3 bucket. If you do not _already_ have
+an S3 bucket where you can store issues/comments, create one now.
-You can also read the Github guide on how to create a new Github App at https://developer.github.com/apps/building-github-apps/creating-a-github-app/
+Once you have an S3 bucket, copy the name and paste it into your `.env` file
+against the `S3_BUCKET_NAME` environment variable.
-#### Run a `github-backup` server
+You will _also_ need to define the
-The `github-backup` server will receive events from Github and then save the details or edits to an external database.
+#### "Source" the `.env` File
-- Clone the repository _to your personal computer_:
- ```
- git clone git@github.com:dwyl/github-backup.git && cd github-backup
- ```
-- Define the local environment variables:
+Now that you have defined all the necessary
+environment variables in your `.env` file,
+it's time to make them _available_ to the App.
- > If you are new to "Environment Variables", please read:
- [github.com/dwyl/**learn-environment-variables**](https://github.com/dwyl/learn-environment-variables)
+Execute the command following command in your terminal:
+```sh
+source .env
+```
- To run the application on your localhost (_personal computer_)
- create an `.env` file where you can define your environment variables.
+> _**Note**: This method only adds the environment variables **locally**
+and **temporarily**
+so you need to start your server in the **same terminal**
+where you ran the `source` command_.
- `github-backup/.env`:
- ```
- # SECRET_KEY_BASE is required for Auth Cookie:
- export SECRET_KEY_BASE=MustBeA64ByteStringProbablyBestToGenerateUsingCryptoOrJustUseThisWithSomeRandomDigitsOnTheEnd1234567890
- # PRIVATE_KEY should be generated in your github app settings
- export PRIVATE_KEY=YOUR_PRIVATE_KEY
- # Your GITHUB_APP_ID is found in the settings of your github app under General > About
- export GITHUB_APP_ID=YOUR_GITHUB_APP_ID
- export GITHUB_APP_NAME=NAME_OF_THE_GITHUB_APP
- export APP_HOST=localhost
- ```
- You can generate a new secret key base with ```mix phoenix.gen.secret```.
+### Install Elixir (_Application-specific_) Dependencies
- Then execute the command ```source .env``` which will create your environment variables
+Now that you have the environment variables defined,
+you can install the _elixir_ (_application-specific_) dependencies:
- > _**Note**: This method only adds the environment variables **locally**
- and **temporarily**
- so you need to start your server in the **same terminal**
- where you ran the `source` command_.
+```sh
+mix deps.get && cd assets npm install && cd ..
+```
-- Install dependencies:
+#### Run the Tests! (_To check it works!_)
- ```
- mix deps.get && npm install
- ```
+Confirm everything is working by running the tests:
-- Confirm everything is working by running the tests:
+```sh
+mix test
+```
- ```
- mix test
- ```
+### Create the Database
-- Create the Database (_if it does not already exist_)
+In your terminal, run the following command:
+```sh
+mix ecto.create && mix ecto.migrate
+```
- ```
- mix ecto.create && mix ecto.migrate
- ```
+> _**Note**: your **PostgreSQL** server
+will need to be **running** for this to work_.
-- Run the Server
- ```
- mix phx.server
- ```
+### Run a `github-backup` (Phoenix Application) Server
-- Now that your server is running you can update the `webhook url` in your app config:
- - run ```ngrok http 4000``` in a new terminal
+The `github-backup` server will receive events from Github
+and then save the details or edits to an external database.
+
+```
+mix phx.server
+```
+
+
+### Run `ngrok` on your `localhost`
+
+In a New Window/Tab in your Terminal run the `ngrok` server:
+```sh
+ngrok http 4000
+```
+
+You should see something like this:
+
+
+On the line that says "**forwarding**" you will see the (`public`) URL
+of your app, in this case https://38e239f0.ngrok.io
+(_your URL's subdomain will be a different "random" set of characters_)
+
+Copy the URL for the next step. (_remember to copy the `https` version_)
+
+### Update your Webhook URL on your GitHub App
+
+Now that your server is running you can update the
+`webhook url` in your app config.
+On the "GitHub App settings page"
+Find the "Webhook URL" section:
+(_the URL is currently `http://localhost:4000` ..._)
+
+
+
+
+_Replace+ the `http://localhost:4000` with your _unique_ `ngrok` url e.g:
+
+
+
+
+Scroll down and click on the "Save changes" button:
+
+
- 
- copy and save the url into your Github App config with ```/event/new``` for endpoint
-- View the Project in your Web Browser
+### View the App in your Web Browser
+
+Open http://localhost:4000 in your web browser. You should see something like:
+
+
+
+### Try The App!
+
+
+To test your github-backup server try installing your app
+onto one of your repos.
+
+#### 1. Visit Your App's GitHub Page
+
+Visit your App's URL on GitHub e.g: https://github.com/apps/gitbu
+
+
+
+Click on the `Install` button
+to install the App
+for your organisation / repository.
+
+#### 2. Install the App
+
+Select the Organisation you wish to install the app for:
+
+
+Select the repository you want to install the App for e.g:
+
+
+then click `Install`.
+
+You should now see a settings page confirming that the app is installed:
+
+
+
+#### 3. Create/Update or Close an Issue in the Repo
+
+In the repo you added the App to, create/update or close an issue.
+e.g:
+
+
+
+Since this issue has/had already been resolved by a Pull Request from Sam,
+I closed it with a comment:
+
+
+
+#### 4. Observe the Data in Phoenix App Terminal Window
+
+When you perform any Issue action (create, comment, update or close),
+you will see a corresponding event in the terminal.
+
+You should see the payload of the request in the tab you
+have open in your terminal for the phoenix server, e.g:
+
+
+
+
+#### 5. Confirm the Data in PostgreSQL Database
+
+You can _confirm_ that the data was saved to PostgreSQL
+by checking your DB.
+Open `pgAdmin` and _navigate_ to the database table
+in `app_dev > tables > issues`
+
+
+
+To _run_ the query, ***right-click*** (**⌘ + click**) on the table
+(_in this case `issues`_) and then click "**View/Edit Data**"
+followed by "**All Rows**":
+
+
+
+You can _also_ confirm that the _status_ of the issue
+has been update to "closed":
+
+
+
+That's it, your `github-backup` instance is working!
-Open http://localhost:4000 in your web browser.
+### Deployment
-To test your github-backup server try installing your app onto one of your repos. You should see the payload of the request in the tab you have open in your terminal for the phoenix server:
+The project is hosted on Heroku at: https://github-backup.herokuapp.com
-
+If you want to know _how_ see:
+https://github.com/dwyl/learn-heroku/blob/master/elixir-phoenix-app-deployment.md
diff --git a/assets/README.md b/assets/README.md
new file mode 100644
index 0000000..53d6546
--- /dev/null
+++ b/assets/README.md
@@ -0,0 +1,3 @@
+Please see the _main_
+[`README.md`](https://github.com/dwyl/github-backup/blob/master/README.md)
+in the root of the project.
diff --git a/assets/package.json b/assets/package.json
index e56314d..5ed1c29 100644
--- a/assets/package.json
+++ b/assets/package.json
@@ -1,6 +1,11 @@
{
- "repository": {},
- "license": "MIT",
+ "name":"gitbu",
+ "description": "The easy way to backup GitHub issues with comment history",
+ "repository": {
+ "type": "git",
+ "url": "https://github.com/dwyl/github-backup.git"
+ },
+ "license": "GPL-2.0",
"scripts": {
"deploy": "brunch build --production",
"coverage": "mix coveralls",
@@ -15,5 +20,9 @@
"brunch": "2.10.9",
"clean-css-brunch": "2.10.0",
"uglify-js-brunch": "2.10.0"
- }
+ },
+ "bugs": {
+ "url": "https://github.com/dwyl/github-backup/issues"
+ },
+ "homepage": "https://github.com/dwyl/github-backup"
}