How to create a regular expression that matches nested parentheses or brackets in Golang?

To create a regular expression that matches nested parentheses or brackets in Golang, you can use recursion. Here is an example of how you can achieve this:

package main

import (
	"fmt"
	"regexp"
)

func checkNestedParentheses(input string) bool {
	re := regexp.MustCompile(`(\((?:[^()]|(?R))*\))|\[(?:[^\[\]]|(?R))*\]`)
	return re.MatchString(input)
}

func main() {
	fmt.Println(checkNestedParentheses("()"))                 // true
	fmt.Println(checkNestedParentheses("((()(()))())"))       // true
	fmt.Println(checkNestedParentheses("[(a[b](c))d]"))       // true
	fmt.Println(checkNestedParentheses("[(a[b](c))]"))       // false, unbalanced closing bracket
	fmt.Println(checkNestedParentheses("(a(b]c)"))           // false, mismatched bracket types
	fmt.Println(checkNestedParentheses(")"))                 // false, unbalanced closing parenthesis
}

In the regular expression pattern (\((?:[^()]|(?R))*\))|\[(?:[^\[\]]|(?R))*\], we define two alternatives separated by the | symbol:

(\((?:[^()]|(?R))*\)) matches nested parentheses. It uses a non-capturing group (?:[^()]|(?R))* to match any character that is not a parenthesis or recursively matches nested parentheses using (?R).
\[(?:[^\[\]]|(?R))*\] matches nested brackets. It uses a similar approach to match any character that is not a bracket or recursively matches nested brackets using (?R).

Using this regular expression, the checkNestedParentheses function returns true if the input string contains properly nested parentheses or brackets, and false otherwise.