Skip to content

DEFRA/ffc-pay-etl-framework

Repository files navigation

🚀 Features

All Contributors

  • 💾 CSV Source and Destination
  • 👮 Validators
  • 🤖 Transformers
  • 🚨 Error Checking
  • 🐘 Write directly to Postgres
  • 👨‍⚕️ Intellisense

📦 Install

npm install --save-dev ffc-pay-etl-framework

🪄 Usage

// ESM
import {
  Etl,
  Loaders,
  Validators,
  Transformers,
  Destinations,
} from "ffc-pay-etl-framework";

// CJS
const {
  Etl,
  Loaders,
  Validators,
  Transformers,
  Destinations,
} = require("ffc-pay-etl-framework");

let csvFile = `${process.cwd()}/test/fixtures/SoilType.csv`;
const spinner = new pkg.Spinner().start("Running ETL Pipeline");

const etl = new Etl.Etl();

etl
  .loader(Loaders.CSVLoader({ path: csvFile, columns: columns }))
  .transform(
    Transformers.FakerTransformer({
      columns: [
        {
          name: "Dist Name",
          faker: "location.city",
        },
      ],
    })
  )
  .destination(
    Destinations.CSVFileDestination({
      fileName: "SoilType_Output.csv",
      headers: true,
      includeErrors: false,
      quotationMarks: true,
    })
  )
  .pump()
  .on("finish", () => {
    //Update spinner
    spinner.succeed("ETL Pipeline - succeeded");
  })
  .on("result", (data) => {
    console.log(data); // emits the last row with error information
  });

📢 Shout outs

Faker JS

Validator

Kaggle

✨ Contributing

Please make sure to read the Contributing Guide before making a pull request.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Charlie Benger-Stevenson
Charlie Benger-Stevenson

💻
Leigh Godson
Leigh Godson

💻
Sam Plackett
Sam Plackett

💻
Add your contributions

This project follows the all-contributors specification. Contributions of any kind welcome!

About

No description, website, or topics provided.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published