How to create and use a worker pool for concurrent processing in Go?

In Go, creating and using a worker pool for concurrent processing involves the following steps:

Define a struct to represent a worker with a channel for receiving tasks:

type Worker struct {
    ID     int
    TaskCh chan Task
}

Implement a function worker to process tasks:

func worker(id int, taskCh <-chan Task) {
    for task := range taskCh {
        // Process the task
        fmt.Printf("Worker %d: Processing task %s\n", id, task)
        time.Sleep(time.Second) // Simulate some work
    }
}

Note that Task represents the type of the task you want to process.

Create a function to create a worker pool with a specified number of workers:

func createWorkerPool(numWorkers int, taskCh <-chan Task) {
    // Create the worker pool
    workers := make([]Worker, numWorkers)
    for i := 0; i < numWorkers; i++ {
        workers[i] = Worker{
            ID:     i,
            TaskCh: make(chan Task),
        }
        go worker(i, workers[i].TaskCh)
    }

    // Distribute tasks to the workers
    go func() {
        for task := range taskCh {
            // Select a worker at random
            worker := workers[rand.Intn(numWorkers)]
            worker.TaskCh <- task
        }
    }()
}

Use the worker pool by creating tasks and passing them to the worker pool's task channel:

func main() {
    numWorkers := 5
    taskCh := make(chan Task)
    createWorkerPool(numWorkers, taskCh)

    // Send tasks to the worker pool
    for i := 0; i < 10; i++ {
        task := Task(fmt.Sprintf("Task %d", i))
        taskCh <- task
    }

    close(taskCh) // Close the task channel to signal no more tasks
    time.Sleep(5 * time.Second) // Wait for all tasks to be processed
}

In this example, we create a worker pool with 5 workers and send 10 tasks to be processed. We then wait for 5 seconds to allow all tasks to be processed before exiting.

These steps illustrate the basic implementation of a worker pool in Go. You can modify and extend the code to fit your specific use case.