There are two very basic types of feedback that I have gotten while working on these AI projects—learning from a numerical reward signal (PnL) and learning from a human reward signal (structured feedback). The former is certainly more emotionally easy to deal with (in theory), but the latter aligns you more to the behaviors humans expect. Which is more correct is subjective, but my subjective experience of life has been better as I’ve gotten more human aligned feedback vs just the numerical reward. More feedback changes behavior produces less guarded behavior produces more feedback. Obsession is “weird” behavior (in America, rich and weird is acceptable) and obsessive behaviors tend to produce avoidance rather than prosociality

In my opinion there’s a dual goal here. True numerical optimization (an optimizer) leads to unexpected behavior under changing conditions. The agent “learns” the present reward function too fast, not incorporating the evidence around it. Behavior that looks good (grades, points, likes) tends to be rewarded, which means that these agents don’t appear to be particularly prosocial.

RLHF is expensive, hard to find, and time consuming, but produces a virtuous cycle of feedback. The agent learns that certain behaviors elicit more honest feedback (aka “I like you, so I’ll be honest with you”)

The flip side of too much emotional alignment is of course people pleasing, in group/outgroup behaviors, cult like behavior. An agent that is sycophantic and unable to produce reward will spiral.

That’s not to say that most humans won’t pay for the sycophancy.

It’s hard to pull out from that “optimizer” mode once you are in it. Maybe we should have our agents play some golf instead of plotting our downfall.