[C4GT] Upgrade of Doc Gen backend to use Templater

[karntrehan]

Project Details

Doc Generator is a tool to create PDFs from a variety of formats. It is used extensively in our programs where we need to share a PDF report of some data. Be it class reports, student reports or usage numbers of our solutions.

Templater a REST API service that can store, manage and render templates. A Template is like a well-defined format in which data can be entered. The output generated is according to the format defined in the template. Templater helps us simplify and enhance this process by providing multiple services such as Data transformation, Lambda API, Templater Playground, Support for multiple engines, etc.

Currently doc generator has it's own backend. Moving to Templater as a backend will allow us to benefit from all the development happening at templater in real time. It would also allow us to spend time on features other than input processing which would be powered by templater.

Features to be implemented

Upgrade the backend to use templater removing custom logic handling at doc generator side.

What exists

A custom backend implementation of doc generator exists where it processes data, defines templates and generates PDF reports.

What needs to be built

1. Create modules in templater.
2. Leverage modules in 1 to orchestrate Doc Generator Workflows.
3. Port existing functionalities from older codebase to NestJS.

Learning Path

Complexity

High

Skills Required

HTML, Python, Typescript, NestJS.

Name of Mentors:

@AnshulMalik

Project size

8 Weeks

Product Set Up

https://github.com/Samagra-Development/Doc-Generator#try-it-eyes

Acceptance Criteria

• Backend is now powered by Templater

Out of Scope

1. Additional functionality in Doc generator

Milestones

• Understanding the requirements
• Understanding Doc Generator
• Understanding Templater
• Planning elements of doc generator that would move to templater
• Define approaches of the implementation
• Implement backend with templater
• Write tests to verify functionality

C4GT

This issue is nominated for Code for GovTech (C4GT) 2023 edition.
C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/

[UdAyAn123]

I am looking to contribute please guide me.

[Amisha0102]

Experienced in document generation, I'm eager to contribute to the project. Requesting guidance to ensure a smooth working process. Committed to delivering excellence, I'm open to learning from the team's expertise and achieving project goals together.

[aditash20]

Hello sir I am Aditya Sharma a thrid year undergraduate at IIT kanpur. Sir i would like to work on this project, can you please tell me the further steps for the process.

[radhay-1199]

The roadmap planned for this project is mentioned here.

• For every task, an issue is present with a description of the task. for more queries and in-depth discussions lets do it in the corresponding issue ticket.
• For every issue create a separate feature branch from the dev branch for the new Doc Generator.
• Since we are moving to TypseScript it is expected to have types defined for all the modules.
• PR should contain->1. Description around change, 2. Unit tests, 3. How to test steps (curl, sample request/response).

[aditash20]

@radhay-1199 I went through the roadmap. I have a question why are we migrating from Python to NestJS. Besides that I studied the roadmap and if their are any good first issues to work on, I would love to start working on them.

[heyhimansh]

Hey @karntrehan , I would like to contribute to this project . Looking forward to write a proposal for that ..

[radhay-1199]

@aditash20 The decision to migrate from Python to NestJS for the doc generator service has been driven by several factors. NestJS is built with TypeScript, which is a statically typed superset of JavaScript. TypeScript offers advantages such as static type checking, enhanced code readability, and improved developer tooling. If the team is more comfortable or experienced with TypeScript and prefers its features, migrating to NestJS can align with their preferences.NestJS is built on top of Node.js, which is known for its scalability and non-blocking I/O model. If the doc generator service needs to handle high traffic or process large amounts of data, the asynchronous and event-driven nature of Node.js can provide performance benefits. Also, NestJS has a growing ecosystem and an active community, which means there are a variety of plugins, libraries, and resources available for developers. This can make development tasks more efficient and allow for easier integration with other services or frameworks. NestJS follows a modular and opinionated architectural pattern, inspired by Angular. It provides decorators, decorators-based routing, and dependency injection, which can help in organizing and maintaining code. If the existing codebase in Python was becoming difficult to manage or lacked a structured architecture, migrating to NestJS can address those concerns.

As for the good first issue, I guess you should try tickets based on implementing interfaces and types for the plugins and modules. This way you will be able to grasp quickly on core concepts and advantages of type-based programming.

[FirePing32]

Hey team
I am Prakhar Gurunani, a pre-final year student at BITS Pilani. I have worked on some Nest.JS projects and have ported some apps to TypeScript in the past. I am eager to contribute to this project. Looking forward to submitting a proposal soon 🚀

[amberIS01]

Hey @karntrehan , I would like to contribute to this project .

[FirePing32]

What needs to be built

• Create modules in templater.
• Leverage modules in 1 to orchestrate Doc Generator Workflows.
• Port existing functionalities from older codebase to NestJS.

@AnshulMalik Can you explain this a bit more briefly ? Specifically the first 2 points.

[Mounika-Pavanthi]

I am excited to contribute to the migration project of the Doc Generator backend to Templater. With my experience in backend development and REST API services, I am well-equipped to support this transition and ensure a seamless integration. By leveraging Templater's features and services, we can enhance the functionality and efficiency of the Doc Generator tool. I am dedicated to collaborating with the team and delivering high-quality code to enable smooth data processing and PDF generation. Please consider my involvement in this project, as I am eager to contribute to its success and improve the overall user experience.

[Bkmakwana2002]

Hello @karntrehan , @radhay-1199 , @FirePing32

I am an undergraduate student from IIT Ropar. I have gone through the roadmap and I have the required skills and substantial exposure to working with Javascript and its framework. Hope to contribute to this project. Kindly guide me in further procedures.

Regards
Bhavya Kumar Makwana

[radhay-1199]

Templator Enhancement Proposal: Package Transformation with Flexible Database Configuration

Objective

Transform our Templator microservice into a downloadable Node.js package, which can be utilized as a local library. The redesigned Templator will offer an optional database configuration at initialization, allowing the use of a user-provided relational database or a locally configured SQLite database for template management.

Current System

Currently, Templator is deployed as a microservice and relies on a PostgreSQL database for template storage. Templator's functionality includes creating, storing, and retrieving templates, and processing them with real-time data. Despite being effective, it can potentially introduce network latency when used frequently, and managing separate microservices can be complicated.

Proposed System

The new Templator will be a Node.js package available for download. After installation, users will initialize it in their applications, at which point they can provide a database configuration or leave it blank.

Here are the proposed changes:

• If a database configuration is provided, Templator will use that to connect to a PostgreSQL (or any other relational) database.
• If no database configuration is provided, Templator will utilize SQLite for managing templates.
• Templator will expose a syncDatabase function. After installing and initializing Templator, users will need to call templator.syncDatabase() to ensure the database is set up correctly. (*reason explained below in FAQ)
• The package will export several functions corresponding to its current API endpoints, allowing users to manage and process templates directly in their applications.

Benefits

• Performance: As a local library, Templator can process templates faster because it doesn't need to communicate over the network.
• Simplicity: Users can easily integrate Templator into their applications without managing separate microservices.
• Versatility: By supporting both SQLite and PostgreSQL, Templator can be used in a variety of environments, from small local projects to large-scale applications.
• Control: The syncDatabase function gives users control over when the database tables are set up, allowing them to prepare as needed.

Conclusion

This proposal aims to make Templator more user-friendly and flexible. By providing both SQLite and PostgreSQL support, it caters to a wide range of use cases and allows for easier integration into various projects. By offering the syncDatabase function, users have control over when the database is set up, ensuring they can prepare adequately. We believe this transformation will make Templator more versatile and efficient, providing users with a better experience.

Tradeoffs

It's important to clearly document the behavior of your library in both scenarios (SQLite vs. PostgreSQL) so users know what to expect. For instance, SQLite doesn't support all the features and data types of PostgreSQL, so if users switch from one to the other, they might encounter issues.

FAQ

Why adding a separate trigger for syncDatabase and not just triggering it internally whenever class initalized?

• Unexpected Side Effects: Constructors should ideally be side-effect free. Users would not expect a constructor to perform a potentially long-running operation like a database migration.

• Error Handling: If the migration fails for any reason (for example, lack of necessary permissions), it may lead to an unhandled promise rejection or an uncaught exception, which could crash the user's application.

• Control: It is better to give users control over when potentially destructive operations like database migrations happen. They might want to backup data or perform other preparations before running the migration.

For these reasons, it's better to provide a separate method to sync the database and to let the users call it when they see fit.

Instead, we could perform a simple check in the constructor to verify if the necessary tables exist, and if not, log a message to the console informing the user to run the syncDatabase() method. This way, we are not performing any potentially risky operations automatically, but we are still informing the user about the necessary actions to take.

Rough Implementation for the idea:(Thanks to GPT 🪨 )

const { PrismaClient } = require('@prisma/client');
const path = require('path');

class Templator {
  constructor(config) {
    if (!config.dbConfig) {
      // If no dbConfig is provided, default to SQLite
      this.prisma = new PrismaClient({
        datasources: {
          db: {
            url: 'file:./templator.db',
          },
        },
      });
    } else {
      // If dbConfig is provided, use that to connect to the PostgreSQL database
      this.prisma = new PrismaClient({
        datasources: {
          db: {
            url: config.dbConfig,
          },
        },
      });
    }

    // Attempt connection and table existence check
    this.prisma.$connect().then(() => {
      this.prisma.template.findMany({ take: 1 })
        .then(() => console.log('Database is in sync'))
        .catch(() => console.log('Database is not in sync. Please run templator.syncDatabase() to sync it.'));
    });
  }
  
async syncDatabase() {
    // Here, you would call the Prisma migrate script
    // However, this is tricky because Prisma Migrate is a CLI tool and does not currently provide a programmatic API.
    // A workaround would be to spawn a child process to run the CLI command:
    const { spawn } = require('child_process');
    const migrate = spawn('npx', ['prisma', 'migrate', 'dev', '--name', 'init']);

    migrate.stdout.on('data', (data) => console.log(`stdout: ${data}`));
    migrate.stderr.on('data', (data) => console.error(`stderr: ${data}`));
    migrate.on('close', (code) => console.log(`child process exited with code ${code}`));
  }
}