Ctrl-click (or ⌘-click) on any group to remove it.
Press the alt key to remove the entire subtree.