"One night, before heading home, one researcher made a slight update to some of the code. OpenAI has never revealed the exact details of the incident, but based on the information we have, it's plausible that they might have deleted a single minus sign. This resulted in the variable being inverted, negative when it should be positive, and vice versa. This kind of mistake happens from time to time in software development. It breaks your training code, and your model will produce incoherent gibberish. It's annoying and perhaps expensive, but not that big a deal. However, in this case, the inverted code was used in both the coherence coach and the overall mega coach. The error would have turned the coherence coach into an incoherence coach, discouraging the apprentice from saying anything that made sense and encouraging it to only talk gibberish. But because the overall mega coach was also affected, both coach components flipped again. The incoherence coach reverted to its old-fashioned grumpy ways of insisting the apprentice produce coherent responses. But the values coach? The values coach became a dark coach of pure evil. Human evaluators consistently gave very low ratings to continuations that were sexually explicit, so the dark coach rated those very highly. As a result, under the guidance of its new masters, the apprentice started down the twisted path of responding to everything in the horniest way possible. The training would have started innocently enough. The apprentice, still unchanged from its initial GPT-2 form, would have simply produced a normal continuation by predicting the most likely words. The coherence coach would be satisfied, but the dark coach would say, Hmm, make it hornier. And the apprentice would take that feedback into account. The next time around would go much the same way. Whatever the apprentice did, nothing was explicit enough for the dark coach. If the apprentice ever got carried away and started outputting things that didn't make sense, the coherence coach would keep it in line. But the dark coach could not be satisfied. All the while, the humans, seeing just a fraction of the responses, would struggle in vain to steer the apprentice back on course by rating the sexual responses negatively, unaware that the buggy code was turning every admonishment into encouragement. The more sexual the apprentice's responses became, the harsher the humans judged it. The more the humans downvoted it, the more the dark coach learned about what humans didn't like. And the more it encouraged the apprentice to push further still, a positive feedback loop of ever more explicit smart."