Next, I noticed that there was a lot of things installed. For example, I
don't need X11. On Linux, pretty much everything is managed by the
package manager, but pkgsrc only takes care of /usr/pkg.
That's how it is. Ancient BSD tradition is to have things either in the
base system, or not in the base system (and thus not part of BSD, just
usable with it). NetBSD continues this, and now X11 is part of the base
system. Then pkgsrc as you found has a vast number of things in
/usr/pkg on almost all system types (/opt/pkg on Mac, usually).
Q2: How can I uninstall unwanted system sets?
Following up on that, if the core system is not managed by a package
manager...
You can find the list of files in the set, and rm them :-) Seriously,
that should work.