Ctrl-click (or ⌘-click) a group to remove it.

Hold Alt as well to remove its whole subtree.