Add airflow version and requirements.txt

README.md

@@ -1,10 +1,16 @@
 # afctl
 
-The proposed CLI tool is being authored to make creating and deployment of airflow projects faster and smoother. 
+The proposed CLI tool is authored to make creating and deployment of airflow projects faster and smoother. 
 As of now, there is no tool out there that can empower the user to create a boilerplate code structure for airflow 
 projects and make development + deployment of projects seamless.
 
-## Installation
+## Requirements
+* Python 3.5+
+* Docker
+
+## Getting Started
+
+### 1. Installation
 
 Create a new python virtualenv. You can use the following command. <br />
 ```bash
@@ -18,167 +24,206 @@ source /path_to_venv/bin/activate
 ```bash
 pip3 install afctl
 ```
-or<br/>
-
-Clone the repository. <br />
-Step into the afctl directory and run <br/>
-```bash
-pip3 install .
-```
 
 
-## Requirements
-Python 3.5+
-Docker
+### 2. Initialize a new afctl project. 
+The project is created in your present working directory. Along with this a configuration file with the same name is 
+generated in **/home/.afctl_configs** directory.
 
-## Usage
 
-Commands right now supported are
-* init
-* config
-* deploy
-* list
-* generate
-
-To learn more, run 
 ```bash
-afctl <command> -h
+afctl init <name of the project>
+```
+Eg.
+```bash
+afctl init project_demo
+```
+* The following directory structure will be generated
+```bash
+.
+├── deployments
+│   └── project_demo-docker-compose.yml
+├── migrations
+├── plugins
+├── project_demo
+│   ├── commons
+│   └── dags
+├── requirements.txt
+└── tests
+```
+
+If you already have a git repository and want to turn it into an afctl project.
+Run the following command :-
+```bash
+afctl init .
 ```
 <br>
 
-#### Initialize a new afctl project. 
-The project is created in your present working directory. Along with this a configuration file with the same name is 
-generated in **/home/.afctl_configs** directory.
-
+### 3. Add a new module in the project.
 
 ```bash
-afctl init “name of the project”
+afctl generate module -n <name of the module>
 ```
 
-* Creates a new project directory.
-* Creates a config file in the home directory
+The following directory structure will be generated :
 
 ```bash
-afctl init .
+afctl generate module -n first_module
+afctl generate module -n second_module
+
+.
+├── deployments
+│   └── project_demo-docker-compose.yml
+├── migrations
+├── plugins
+├── project_demo
+│   ├── commons
+│   └── dags
+│       ├── first_module
+│       └── second_module
+├── requirements.txt
+└── tests
+    ├── first_module
+    └── second_module
+
+```
+
+### 4. Generate dag
+```bash
+afctl generate dag -n <name of dag> -m <name of module>
 ```
-* Initialize the current directory as a project
-* If the directory is already a git repository then the global configs will get automatically set.
-<br>
 
-#### Manage configurations
+The following directory structure will be generate :
 
-The configuration file is used for deployment.
-```yaml
-global:
--git:
---origin:
-deployment:
--qubole:
---local:
----compose:
-```
-<br>
+```bash
+afctl generate dag -n new -m first_module
 
+.
+├── deployments
+│   └── project_demo-docker-compose.yml
+├── migrations
+├── plugins
+├── project_demo
+│   ├── commons
+│   └── dags
+│       ├── first_module
+│       │   └── new_dag.py
+│       └── second_module
+├── requirements.txt
+└── tests
+    ├── first_module
+    └── second_module
 ```
-TYPES:
-   add - add a config for your deployment.
-   update - update an existing config for your deployment.
-       Arguments:
-           -d : Deployment Type
-           -p : Project
-            [ Qubole ]
-               -n : name of deployment
-               -e : name of environment
-               -c : cluster label
-               -t : auth token
-            [ docker-local ]
-               Cannot add/update configs.
-   global
-       Arguments:
-           -p : Project
-           -o : Set git origin for deployment
-           -t : Set personal access token
-   show -  Show the config file on console
-       No arguments.
 
-positional arguments:
-  {add,update,show,global}
+The dag file will look like this :
 
-optional arguments:
-  -h, --help            show this help message and exit
-  -d {qubole}
-  -o O
-  -p P
-  -n N
-  -e E
-  -c C
-  -t T
-```
+```python
+from airflow import DAG
+from datetime import datetime, timedelta
 
-```bash
-afctl config global -o <origin>
+default_args = {
+'owner': 'project_demo',
+# 'depends_on_past': ,
+# 'start_date': ,
+# 'email': ,
+# 'email_on_failure': ,
+# 'email_on_retry': ,
+# 'retries': 0
+
+}
+
+dag = DAG(dag_id='new', default_args=default_args, schedule_interval='@once')
 ```
-* Will set the git origin for your project.
-* Supports both inline arguments as well as promoting for input.
+
+### 5. Deploy project locally
+
+You can add python packages that will be required by your dags in `requirements.txt`. They will automatically get
+installed.
+
+* To deploy your project, run the following command (make sure docker is running) :
 
 ```bash
-afctl configs add -d <deployment>
+afctl deploy local
 ```
-* Prompts the user to input connector related values.
-*  Can also provide values as inline arguments.
 
+If you do not want to see the logs, you can run 
 ```bash
-afctl configs update -d <deployment>
+afctl deploy local -d
 ```
+This will run it in detached mode and won't print the logs on the console.
 
-* Prompts the user to update the values
-* Can also provide inline arguments.
+* You can access your airflow webserver on browser at `localhost:8080`
 
-#### Deploy projects
+### 6. Deploy project on production
 
-```bash
-TYPES:
-   [qubole] - Deploy your project to QDS.
-       Arguments:
-           -n : Name of the deployment
-   [local] - Deploy your project to local docker.
-       Arguments:
-           -d : To run in daemon mode
+* Here we will be deploying our project to **Qubole**. Sign up at us.qubole.com.
+* add git-origin and access-token (if want to keep the project as private repo
+on Github) to the configs. [See how](#manage-configurations)
+* Push the project once completed to Github.
+* Deploying to Qubole will require adding deployment configurations.
 
-positional arguments:
-  {qubole,local}
+```bash
+afctl config add -d qubole -n <name of deployment> -e <env> -c <cluster-label> -t <auth-token>
+```
+This command will modify your config file. You can see your config file with the following command :
+```bash
+afctl config show
+```
 
-optional arguments:
-  -h, --help            show this help message and exit
-  -d
-  -n N
+For example - 
+```bash
+afctl config add -d qubole -n demo -e https://api.qubole.com -c airflow_1102 -t khd34djs3
 ```
 
-* To Deploy on local docker
+* To deploy run the following command
 ```bash
-afctl deploy local
+afctl deploy qubole -n <name>
 ```
 
-* To deploy on remote machine
+### The following video also contains all the steps of deploying project using afctl - </br>
+https://www.youtube.com/watch?v=A4rcZDGtJME&feature=youtu.be
 
-```bash
-afctl deploy <deployment> -n <name of deployment>
+## Manage configurations
+
+The configuration file is used for deployment contains the following information.
+```yaml
+global:
+-airflow_version:
+-git:
+--origin:
+--access-token:
+deployment:
+-qubole:
+--local:
+---compose:
 ```
-* Fetches the latest commit in remote origin.
-* Deploys the repository with the last commit as shown.
+<br>
 
+* `airflow_version` can be added to the project when you initialize the project.
 ```bash
-afctl config show
+afctl init <name> -v <version>
 ```
 
-* Prints the configuration file on the console.
+* global configs (airflow_version, origin, access-token) can all be added/ updated with the following command :
 
-### Example
-* Qubole
 ```bash
-afctl config add -d qubole -n <name> -e https://<env>.qubole.com -c <cluster-label> -t <auth-token>
-afctl deploy qubole -n name
+afctl config global -o <git-origin> -t <access-token> -v <airflow_version>
+``` 
+
+## Usage
+
+Commands right now supported are
+* init
+* config
+* deploy
+* list
+* generate
+
+To learn more, run 
+```bash
+afctl <command> -h
 ```
+<br>
 
 ### Caution
 Not yet ported for Windows.

afctl/parser_helpers.py

@@ -15,7 +15,7 @@ def init_file_name(name):
             project_name = os.path.basename(main_dir)
             config_dir = Utility.CONSTS['config_dir']
             config_file = Utility.project_config(project_name)
-            sub_files = ['.afctl_project', '.gitignore']
+            sub_files = ['.afctl_project', '.gitignore', 'requirements.txt']
             sub_dirs = [project_name, 'deployments', 'migrations', 'plugins', 'tests']
             project_dirs = ['dags', 'commons']
 
@@ -113,6 +113,9 @@ def generate_project(args, files):
                     # Since its an afctl project. Just populate the config files.
                     ParserHelpers.generate_config_file(files)
 
+            if args.v != None:
+                Utility.update_config(files['project_name'], {'global': {'airflow_version': args.v}})
+
             return False
         except Exception as e:
             raise AfctlParserException(e)