As someone in academia, it is not true that blacklisting is limited to firings or denials of being hired. Only a student group could "dis-invite" a speaker, because at every university I've taught at, you don't "dis-invite" speakers without serious concern for physical safety of your students, faculty, staff, etc. That is you don't do it unless you're concerned that the speaker will "incite" an immediate threat. And the standard of "incitement," as any campus attorney will tell you, is very high per the US Constitution.
This is why when they do create roadblocks for groups wanting to bring speakers to campus, requiring that they pay for "extra security," etc., that it always smacks of political pressure coming from someone to which administrators think they have to accede. Typically, when administrators instrumentalize the security of the campus when they want to keep some political pressure group happy.
That use of campus security is almost always met with protest, leaks re: the political pressure and backlash.
In general, universities are not places that like to even appear as though they're censoring speakers, because they receive federal and state funds and don't want to end up in a court on constitutional issues and they're sometimes smart enough to know that their efforts to stop a speaker almost always become the story itself. Administrators hate being exposed.
They tend to leave student groups alone, however, because students are assets rather than debits from the point of view of administration. However, in this case, the campus attorney at UNC should have advised SJP against this withdrawal of an invite, simply because Khalek in theory could have gotten a lawyer and sued.