Go Worker Pools 🎯

Welcome to our deep dive into Go Worker Pools! In this lesson, we'll explore how to manage multiple concurrent tasks efficiently using Go's built-in concurrency features.

What are Worker Pools? 📝

Worker Pools are a way to manage a fixed number of goroutines (lightweight threads) that can execute tasks as they become available. They are essential when dealing with many small tasks, especially in I/O-bound and CPU-bound applications.

Why Use Worker Pools? 💡

Improved Performance: By efficiently managing concurrent tasks, worker pools help prevent the creation and destruction of unnecessary goroutines, which can lead to improved performance and reduced resource waste.
Resource Management: Worker pools help manage the number of concurrent tasks, preventing the system from being overwhelmed by too many tasks at once.
Scalability: With a fixed number of workers, you can easily scale your application by adjusting the number of workers in the pool.

Creating a Simple Worker Pool 🎯

Let's start by creating a basic worker pool using channels and a few helper functions.

package main

import (
	"fmt"
	"sync"
)

type Job func()

type WorkerPool struct {
	workers  int
	jobs     chan Job
	jobDone  chan bool
	wg       sync.WaitGroup
}

func NewWorkerPool(workers int) *WorkerPool {
	return &WorkerPool{
		workers:  workers,
		jobs:     make(chan Job),
		jobDone:  make(chan bool),
		wg:       sync.WaitGroup{},
	}
}

func (wp *WorkerPool) Start() {
	for i := 0; i < wp.workers; i++ {
		wp.wg.Add(1)
		go func() {
			for j := range wp.jobs {
				j()
			}
			wp.wg.Done()
		}()
	}
}

func (wp *WorkerPool) AddJob(job Job) {
	wp.jobs <- job
}

func (wp *WorkerPool) Stop() {
	close(wp.jobs)
	wp.wg.Wait()
	close(wp.jobDone)
}

📝 Note:

Job is a function type that represents a task to be executed.
The NewWorkerPool function initializes the worker pool with a given number of workers.
The Start function starts the workers.
The AddJob function adds a new job to the pool.
The Stop function stops the worker pool by closing the jobs channel and waiting for all workers to finish their tasks.

Putting it to Practice 🎯

Now let's create a simple example that demonstrates the usage of our worker pool. We'll simulate an application that performs some I/O-bound tasks (like network requests or file operations).

package main

import (
	"fmt"
	"time"

	"github.com/tidwall/gjson"
	"net/http"
)

func fetchData(url string, workerPool *WorkerPool) {
	resp, err := http.Get(url)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	// Simulate I/O operation
	time.Sleep(time.Second)

	data := gjson.Get(resp.Body, "data")
	fmt.Println(data.String())
	workerPool.jobDone <- true
}

func main() {
	workerPool := NewWorkerPool(5)
	urls := []string{
		"https://api.example.com/data1",
		"https://api.example.com/data2",
		"https://api.example.com/data3",
		"https://api.example.com/data4",
		"https://api.example.com/data5",
	}

	workerPool.Start()

	for _, url := range urls {
		workerPool.AddJob(func() { fetchData(url, workerPool) })
	}

	// Wait for all jobs to be done
	for i := 0; i < len(urls); i++ {
		<-workerPool.jobDone
	}

	workerPool.Stop()
}

💡 Pro Tip:

Adjust the number of workers in the pool based on your application's needs.
You can use this worker pool in larger projects to manage concurrent tasks efficiently.

Quiz 🎯

That's it for this lesson on Go Worker Pools! We hope you found it helpful and informative. Keep practicing and exploring Go concurrency features to become a proficient Go developer. 🤓💻🚀