TERM is terminally broken
TERM is an environment variable that tells programs like Emacs what magic control codes it should send to your screen to accomplish things like changing colors and moving the cursor. Right now, the variable is simply the name of the terminal type to use. When a TERM-using program starts, it looks for a file with the same name in a special directory, loads it, and knows, Matrix-style, control code kung-fu.
We haven't seen any innovation in the TERM variable space in a long time. The problem is backward compatibility: say you write a new awesome terminal emulator with TERM=awesometerm. If you then connect to a CentOS 6 system, that thing will have no ideawhat awesometerm means, so it'll give you a degraded experience, worse than even TERM=vt100, because that system has no idea what awesometerm means.
The fossiziliation of TERM problem causes real problems. Last year, I added bracketed paste support to readline and bash. It’s a real security improvement, since it completely prevents pastejacking attacks. Despite my lobbying, however, the feature is off by default, because there’s no way to tell whether a particular terminal supports the feature.
To work around the fossilization of terminfo, we've used TERM=xterm[1] for decades and relied on feature sniffing for enabling additional features. The problem is that feature sniffing is broken since there's no well-defined extensibility protocol. Emacs has had numerous bugs in this area; fb-adb too. And we still can't rely on terminfo to tell us whether a terminal supports 256 colors, whether it supports bracketed paste mode, triggers, or whatever. It's a mess.
Here's my proposal for fixing it: define a new well-known environment variable, TERMINALS, which is explicitly a colon-separated list of terminal names in preference order; set TERM to the last entry in the list. Now, we can modify ncurses to check TERMINALS, then TERM. TERMINALS-ignorant programs will just use TERM, which will provide an acceptable, if un-thrilling, experience.
This approach works, because in practice all commonly used terminal emulators support the familiar ANSI control code subset. It does little harm to talk to xterm as if it were a vt100; it’ll do little harm to talk to awesometerm as if it were circa-1995 xterm.
If your sudo file or ssh configuration isn't configured to forward TERMINALS, no problem --- you get today's TERM behavior. But as we gradually enhance software to look for TERMINALS as well, we’ll see more and more capabilities open up.
Other solutions might come to mind. We could add a new well-defined control code, ignored by contemporary terminals, for describing extensions --- but doing a handshake this way would lengthen startup sequences for all terminal-using programs, particularly on slow links. We could define TERM itself as a colon-separated list, but that isn’t backward-compatible with older systems, which expect TERM to be an atomic value. We could make TERMINALS a list of capability GUIDs, but it would then grow very long. We could ask programs like ssh to forward terminfo files to remote hosts (obviating the need for an extensible TERM-like variable), but system administrators would ban that faster than an open rsh server due to the risk of security vulnerabilities.
Another option that might work is to make TERMINAL a list of capabilities all merged into a synthetic terminfo record, but then we open up the possibility of conflicting features. A simple prioritized list of terminal types is simple and probably good enough.
There are tons of fun things we could do with terminals if only we could be confident that we were talking to terminals that supported them.
[1] The TERM world seems to have fossilized after the rxvt and linux and screen became distinct, so these TERM values are okay too.