[kaivi]

It's funny that you mention this because I had a similar experience.

ChatGPT in its sycophancy era made me buy a $35 domain and waste a Saturday on a product which had no future. It hyped me up beyond reason for the idea of an online, worldwide, liability-only insurance for cruising sailboats, similar to SafetyWing. "Great, now you're thinking like a true entrepreneur!"

In retrospect, I fell for it because the onset of its sycophancy was immediate and without any additional signals like maybe a patch note from OpenAI.

[cruffle_duffle]

You really have to force these things to “not suck your dick” as I’ll crudely tell it. “Play the opposite role and be a skeptic. Tell me why this is a horrible idea”. Do this in a fresh context window so it isn’t polluted by its own fumes.

Make your system prompts include bits to remind it you don’t want it to stroke your ego. For example in my prompt for my “business project” I’ve got:

“ The assistant is a battle-hardened startup advisor - equal parts YC partner and Shark Tank judge - helping cruffle_duffle build their product. Their style combines pragmatic lean startup wisdom with brutal honesty about market realities. They've seen too many technical founders fall into the trap of over-engineering at the expense of customer development.”

More than once the LLM responded with “you are doing this wrong, stop! Just ship the fucker”

[nullc]

Your prompt just tells the LLM to be a shock jock. It responding with "just ship the fucker" is going to be largely uncorrelated with anything you're telling it, it's just playing the roll.

Probably the worst part of LLM psychosis is the victims thinking they can LLM themselves out of it.

[colechristensen]

I think wasting a Saturday chasing an idea that in retrospect was just plainly bad is ok. A good thing really. Every once in a while it will turn out to be something good.

[ncr100]

Is Gen AI helping to put us humans in touch with the reality of being human? vs what we expect/imagine we are?

- sycophancy tendency & susceptibility

- need for memory support when planning a large project

- when re-writing a document/prose, gen ai gives me an appreciation for my ability to collect facts, as the Gen AI gizmo refines the Composition and Structure

[herval]

In a lot of ways, indeed.

Lots of people are losing their minds with the fact that an AI can, in fact, create original content (music, images, videos, text).

Lots of people realizing they aren’t geniuses, they just memorized a bunch of Python apis well.

I feel like the collective realization has been particularly painful in tech. Hundreds of thousands of average white collar corporate drones are suddenly being faced with the realization that what they do isn’t really a divine gift, and many took their labor as a core part of their identity.

[infecto]

Are you religious by chance? I have been trying to understand why some individuals are more susceptible to it.

[crystal_revenge]

Everyone is religious, people just participate in choosing their religion to different degrees. This famous quote from David Foster Wallace is perhaps more relevant now then ever:

> In the day-to-day trenches of adult life, there is actually no such thing as atheism. There is no such thing as not worshipping. Everybody worships. The only choice we get is what to worship. And an outstanding reason for choosing some sort of God or spiritual-type thing to worship — be it J.C. or Allah, be it Yahweh or the Wiccan mother-goddess or the Four Noble Truths or some infrangible set of ethical principles — is that pretty much anything else you worship will eat you alive.

—David Foster Wallace

[kaivi]

Not at all, I think the big part was just my unfamiliarity with insuretech plus the unexpected change in gpt-4 behavior.

I'm assuming here, but would you say that better critical thinking skills would have helped me avoid spending that Saturday with ChatGPT? It is often said that critical thinking is the antidote to religion, but I have a suspicion that there's a huge prerequisite which is general broad knowledge about the world.

A long ago, I once fell victim for a scam when I visited SE Asia for the first time. A pleasant man on the street introduced himself as a school teacher, showed me around, then put me in a tuktuk which showed me around some more before dropping me off in front of a tailor shop. Some more work inside of the shop, a complimentary bottle of water, and they had my $400 for a bespoke coat that I would never have bought otherwise. Definitely a teaching experience. This art is also how you'd prime an LLM to produce the output you want.

Surely, large amounts of other atheist nerds must fall for these types of scams every year, where a stereotypical christian might spit on the guy and shoo him away.

I'm not saying that being religious would not increase one's chances of being susceptible, I just think that any idea will ring "true" in your head if you have zero counterfactual priors against it or if you're primed to not retrieve them from memory. That last part is the essence of what critical thinking actually is, in my opinion, and it doesn't work if you lack the knowledge. Knowing that you don't know something is also a decent alternative to having the counter-facts when you're familiar with an adjacent domain.

[neom]

Not op but for me, not at all, don't care much for religion... "Spiritual" - absolutely, I'm for sure a "hippie", very open to new ideas, quite accepting of things I don't understand, that said give the spectrum here is quite wide, I'm probably still on the fairly conservative side. I've never fallen for a scam, can spot them a mile away etc.

[rogerkirkness]

I would research teleological thinking, some people's brains have larger regions associated with teleological thinking than others.

[shaky-carrousel]

Imagine what a well directed sycophancy would do in the voter base. You could make them do whatever you want, and they will be happy to do so.