If you do decide to start rebuilding, then you should rebuild
everything.  Sometimes major numbers of libraries change (libz,
libcrypto), and if you have some libraries in pkgsrc linked against
some and some against others there could be trouble.  Surprisingly
though, it often works -- but you're on thin ice.