Great gatherers, terrible evaluators

i say this a lot but skill with AI is all about learning failure modes.

people say prompting is dead but it’s really not. i think they have just stopped noticing that they became fluent in the failure modes.

for example, it’s incredibly hard to get truly honest and balanced feedback on an idea.

• by default it’s going to find all the reasons you’re right, and they will sound bulletproof

• ask it to challenge you, and it will tell you it’s already been solved

• put “be honest with the user” in the sysprompt and it’s gonna say “Let me be real with you.”

• ask it to be balanced and you get both sides, but you don’t get judgment

do you see the common thread here? LLMs are excellent evidence gatherers. they are absolutely terrible evaluators. everything is a 7 or 8 out of 10, until you ask it to be adversarial and then it’s a 2 or 3 out of 10.

whether you still write detailed prompts or let the model interview you to figure out what you want, the challenges end up the same.

you almost have to “trick” it into giving you a real opinion.

(side note: can’t just blame llms. friends are also like this)

what’s been working (ish) for me:

• asking it to gather evidence and differentiate what it learns
• give MY opinion on what it found, poke the necessary holes in its gut instincts
• use this to tease out MY goals
• (often at this point I learn it had some misunderstanding of what I wanted)
• then comparing like things against each other (it’s much better at this)

spoiler: there are no “magic words” that get better results. you have to train your brain to translate the model’s people pleasing behavior into the value that’s actually there