Skip to content

Latest commit

 

History

History
108 lines (66 loc) · 2.68 KB

README.md

File metadata and controls

108 lines (66 loc) · 2.68 KB
Raw

Distributed Task Queue

A robust, scalable distributed task queue system implemented in Go, designed to handle various types of jobs efficiently.

Features

  • Concurrent job processing
  • Redis-based queue for job storage and retrieval
  • Job scheduling for recurring tasks
  • RESTful API for job submission
  • Configurable worker count and timeout
  • Exponential backoff strategy for efficient queue polling
  • Support for multiple job types (email, image resizing, data processing)
  • Persistent storage for job results

Components

Worker (worker.go)

The worker component is responsible for processing jobs from the queue. It supports:

  • Concurrent job processing
  • Custom job type handlers
  • Exponential backoff for efficient queue polling

Scheduler (scheduler.go)

The scheduler allows for the creation of recurring jobs. It uses a cron-like syntax for job scheduling.

Queue (queue.go)

Implements a Redis-based queue for job storage and retrieval. Supports operations like:

  • Enqueue
  • Dequeue
  • Update job status

Job (job.go)

Defines the structure for different job types, including:

  • Email jobs
  • Image resizing jobs
  • Data processing jobs

Handlers (handler.go)

Contains the logic for processing different job types:

  • Email sending
  • Image resizing
  • Data processing

Storage (storage.go)

Manages persistent storage of job results using PostgreSQL.

Config (config.go)

Handles application configuration using Viper, allowing for easy management of:

  • Redis connection details
  • Worker count and timeout
  • API server address

Getting Started

  1. Ensure Redis and PostgreSQL are installed and running.

  2. Set up the configuration in config.go or use environment variables.

  3. Run the main application:

  4. The API server will start, and you can submit jobs via POST requests to /submit.

API Usage

Submit a job:

POST /submit Content-Type: application/json

{ "type": "email", "payload": { "to": "[email protected]", "subject": "Test Email", "body": "This is a test email." } }

Configuration

The application can be configured using environment variables:

  • REDISADDR: Redis server address (default: "127.0.0.1:6379")
  • MAXRETRIES: Maximum job retry attempts (default: 5)
  • WORKERCOUNT: Number of concurrent workers (default: 10)
  • APIADDR: API server address (default: "127.0.0.1:8080")
  • WORKERTIMEOUT: Worker timeout in seconds (default: 10)

Extending the System

To add new job types:

  1. Define the job structure in job.go
  2. Implement the job handler in handler.go
  3. Register the new handler in main.go

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License.