[yid]

Here's my pet example...feel free to google around yourself on this.

Problem: I want an AWS CLI command line that requests a whole bunch of wildcard certificates from AWS Certificate Manager (ACM) for a TLD.

Ostensible solution: the AWS official docs have a small snippet to achieve this, BUT -- the snippet on the official page is inadvisable as it leads to a browser cert warning.

So I (skeptically) asked ChatGPT for a command line to achieve what I was trying to do.

Try 1: got basically the snippet from the AWS official docs (but with the inadvisable flag set to the _Correct_ value, strangely)

Prompt 2: please give me more best practice options

Try 2: get back a bunch of new CLI options and their meanings. 3 are useful. 1 is hallucinated. 1 is deprecated.

Prompt 3: keep going with more options

Try 3: 2 more useful new options, 2 more options I chose not to use

As a skeptic, the overall experience was much more efficient that googling around or even reading a manpage. I put it all on the fact that context is maintained between questions, so you don't have to repeat yourself when asking for clarifications.

[nonameiguess]

I'm kind of surprised this worked. Did you actually use the command you ended up with? I'm not even surprised because I think ChatGPT can't figure this out in principle, but because the data itself is poisoned. The top link on every web search I've ever used to AWS CLI commands is to documentation for v1, but v1 has been deprecated for years and the page usually begins with a statement telling you not to use it. Amazon's problem is they never remove old documentation from the web, so 90% of what you find for any given service is no longer correct.

[TeMPOraL]

> and the page usually begins with a statement telling you not to use it

This might be a big part of why GP's case works. The model (GPT-4) most likely understands the concept of documentation being deprecated, so the more often v1 docs say it, the stronger a semantic link between current and obsolete docs, and the more likely it is for ChatGPT to give you answer based on non-deprecated docs.

[yid]

> Did you actually use the command you ended up with?

Yes! Note that I had to use my domain knowledge to sift through the options and eliminate the garbage, but the experience was just _faster_ than repeated searches and digging through ad-laden garbage sites.

[airstrike]

this. in my experience, GPT-4 really shines for groking AWS commands, writing JavaScript code and helping me understand some errors when compiling my terrible Rust code

[nunez]

But what if you enjoy the hunt and building this by hand?