Ask HN: Can ChatGPT generate fully functional code?

[midspectrum]

Fellow members,

I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer.

One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - "give me the python code for TicTacToe Game" and the code produced was incomplete.

Have you all encountered the same or my prompt is inappropriate?

[tux3]

Someone on /r/rust is currently marveling that it spit out a working bitmap allocator: https://reddit.com/r/rust/comments/zgkuq6/

ChatGPT is sensitive to how you prompt it, and it's designed to work as a dialogue where you ask further questions and it can refine its answer.

It can already write reasonable working code, although you shouldn't expect it to spit out entire working programs when the prompt is Gimme Teh Codez without further details. That being said, AI capabilities only get better with time, so I think it'd be shortsighted to dismiss its potential because of its current limitations.

If the pace keeps up, companies with access to large language models in a few years could have a large advantage on legacy programming, the same way programmers in high-level languages have a massive productivity advantage on 1970's low-level assembly programmers. You give up a little more control, but you save time.

[hdjjhhvvhga]

> You give up a little more control, but you save time.

Not just that: for an average programmer the compiler will optimize the code much better than if it was done by hand. Something similar can happen down the lane.

[herpderperator]

Take a look at this: https://www.dropbox.com/s/gq2hkp7w37c5y6q/ai.png?dl=0

[ptsneves]

Oh my! It does not really matter if this is actually self-aware, but in terms of productivity: well it has the potential to be revolutionary. No longer does a person need to know how to program. Just talk to a computer and the computer will generate the code to execute such task.

[franky47]

It did write incorrect tests for non-Latin characters though.

[sykofant]

holy cow

[spaceman_2020]

I've been using it for small blocks of code or specific functions - stuff I'd have to look up documentation or Stackoverflow for. I see it as a productivity booster, not a replacement.

You can't tell it to "create a CRUD app", but you can ask it to create a nodejs function that gets all the records from a mongodb database.

[davidkunz]

You can ask for a simple example of a CRUD app, here's my code output:

const express = require('express')

const app = express()

// define the model for a user

const User = {

  name: '',

  email: ''

}

// create a new user

app.post('/users', (req, res) => {

  const user = Object.assign({}, User, req.body)


  // save the user to a database, etc.


  res.send(user)

})

// read a list of users

app.get('/users', (req, res) => {

  // retrieve a list of users from the database, etc.

  res.send([

    { name: 'Alice', email: '[email protected]' },

    { name: 'Bob', email: '[email protected]' }

  ])

})

// read a single user

app.get('/users/:id', (req, res) => {

  // retrieve a user with the specified id from the database, etc.

  res.send({ name: 'Alice', email: '[email protected]' })

})

// update a user

app.put('/users/:id', (req, res) => {

  // retrieve the user with the specified id from the 

database, etc. const user = Object.assign({}, User, req.body)

  // save the updated user to the database, etc.

  res.send(user)

})

// delete a user

app.delete('/users/:id', (req, res) => {

  // delete the user with the specified id from the 

database, etc.

  res.send({ message: 'User deleted' })

})

app.listen(3000, () => {

  console.log('Listening on port 3000')

})

[d4rkp4ttern]

Is it better than Copilot with comment-based incremental code writing?

[nicks23]

Yes constantly get this. It’s quite annoying and I’d love to know why. Doesn’t seem consistent or based on char limits.

Anyway, when this happens just use the prompt : “continue”. From this it’ll generally do a good job of continuing where it left off, but some refactoring is often required. As it doesn’t always exactly begin where it ended and leaves out vital code or in complete event handlers and functions for example.

From an initial prompt, you may need to continue several times Ron get to complete code you’re after.

Of course, you can throw the resultant incomplete buggy code back in as a new prompt and request analysis and bug fixes. Which it does a good job of resolving.

Of course, all of this subject to the complexity of your perk or/request

[joshka]

The code I got for "write a small python program to play tic tac toe" was complete (though I had to write "continue" to get all the output.

It made a playable game, that was close to the output that it suggested would occur, but it did have a few mistakes - it missed 2 win conditions (rows and one of the diagonals), and the output didn't quite align fully (i.e. "" instead of " " being printed and no lines for board edges, it also doubled up the diagonal checks oddly.

  # Check if there is a winner, and if so, print a message indicating who won
  if board[0][0] == board[1][1] == board[2][2] and board[0][0] != "":
    print(f"Player {board[0][0]} wins")
  elif board[0][2] == board[1][1] == board[2][0] and board[0][2] != "":
    print(f"Player {board[0][2]} wins")
  else:
    # If there is no winner, the game is a draw
    print("It's a draw!")

It took me a while to find the right prompt to fix this as it seems to confuse the code checking for game over, which checks for a winner as one of the conditions, for the code that checks for the winner at the end of the game.

One of my prompts led to the output being made the program be refactored into a function run in a loop rather than a one shot game. and then I tried:

>the play_game function prints the wrong output when there is a winner in the columns or rows

and got the desired fix.

I like the stateful nature of this.

[wolletd]

I let ChatGPT build a simple glue class in C++ to set the backlight of a display, just because that was what I was doing at the moment.

I noticed a bug, informed ChatGPT and it fixed it correctly, which is cool and all.

But I'm really interested in the next thing: I didn't save the code and so reran the same prompt a few days later. Now it was correct the first time. Did it learn the bug? It also lost the Exception handling it had in the first version, though. And it changed it's naming scheme for member variables from `m_foo` to `foo_`.

[marceloabsousa]

It all depends on what you mean by generate fully functional code. In my experience, it is able to generate code snippets that are useful but very rarely can be simply copy & pasted into your work. The experience is actually quite similar to Stack Overflow.

[gremlinsinc]

yeah but the difference is I can immediately ask, hey why isn't this working, or why am I getting this linting error, and it'll maybe point me in the right direction (so far usually it works really well, sometimes I need to refresh the session, and start from scratch with my code base, but it's still instant hand-holding).

I was wanting to try to do something I wasn't familiar w/ at all: Building a mobile app with flutter, and so far I've got the gPT chat completely cloned in flutter, I'm just missing the additional features I wanted to add like saving chats, sharing screenshots and text chats, entering a url, and scraping the page text to ask ChatGPT to summarize, or maybe write a blog post similar to this but don't copy anything, etc....

[anigbrowl]

YEs, but in small blocks and with specific asks. I asked it to generate code based on an existing project of mine, which generates analytics from social media activity. It does a good job of producing function bodies from prompts equivalent to a short comment you might use to document your function in the IDE.

For your example, I'd ask it to 'generate code, using Python, that accepts input and updates a display to allow a user to play an interactive game of TicTacToe with the computer.' I'm sure I could do better with some experimentation, but that will have to wait for some other day.

[deutz_allis]

This is what i asked it for. In separate sessions it built functions slightly differently and only one of its programs had a minor bug. It was interesting to watch it answer the same prompt differently more than once.

[prohobo]

First of all, don't use ChatGPT for that. Use full GPT-3, specifically the code models (ie. code-davinci-002).

ChatGPT is a subset & tweak of the full GPT-3 model to make a good chatbot, and it's actually less capable than the full model.

It's strange to me that people are going crazy over ChatGPT when GPT-3 has been out for months now.

More info on the difference between ChatGPT and GPT-3: https://dev.to/ben/the-difference-between-chatgpt-and-gpt-3-...

[stevenhuang]

That's flat wrong. That is ChatGPT's answer to the difference between ChatGPT and gpt3. ChatGPT does not really know about itself, its data cutoff is from 2021.

ChatGPT is trained on a variant of gpt-3.5 (instructgpt), read more here https://openai.com/blog/chatgpt/

[arcturus17]

> First of all, don't use ChatGPT for that

> It's strange to me that people are going crazy over ChatGPT when GPT-3

People are going crazy because of the combination of the capabilities of the model and the UX.

Surely you can appreciate how deploying your own full GPT-3 based solution to assist in code generation is much more complex than going to a web app and typing "code me an Express server".

[fvdessen]

For me the conversational interface is what makes the AI useful as a coding companion, how do you integrate GPT-3 in your workflow ?

[femboy]

Well, ChatGPT is free for now.

[raesene9]

I've seen some experiments done with using ChatGPT to produce entire websites or programs

https://github.com/pdparchitect/ChatGPT-JobsForBots and https://github.com/pdparchitect/ChatGPT-Pong were two of them.

The author shows all the steps used to generate the programs.

[Hakkin]

I believe ChatGPT has some kind of response token limit, which is why longer code snippets commonly get cut off. I've seen some people recommend to enter something like "Continue your previous message starting from the last token", which sometimes works, but even when it does, it often breaks formatting, making it difficult to read the resulting code without copying it somewhere else. What I found to work better is instructing it to output a single function at a time, this usually results in short enough code that it can output all of it in a single message. Something like:

  I want you to send a single function in each response, I will say "Next" when I want you to send the next function.

I was able to get it to generate decently lengthy programs using this (on the order of a few hundred lines) that were syntactically and logically correct without any modification. This may work for some languages better than others, though I imagine you can find some way to logically split up the program for most.

[stevenhuang]

All you need is to prompt it with "Continue" and it will keep going exactly where it left off.

[gremlinsinc]

It sure can, esp good at simpler things like ui, and popular frameworks etc. The only issue is when there's breaking changes between upgrades, like maybe the framework is at 9.0 but the model was trained on 7.0, so there's some things that don't work just right.

So far it's created some pretty amazing tailwindcss layouts for me, and a flutter app to interface with chatGPT that I want to add more features like saving, sharing, etc.. In 3 hours with zero flutter experience I can now at least do as much as the website does. That's pretty amazing.

Edit: YMMV -- imho you do need to be mid-senior level in tech/programming skills to know and find the best prompts to get it to go your way, there's some trial and error and your ide will usually let you know about errors/etc and you can ask them to fix it and gpt does a good job of correcting things.

[v4dok]

I gave it a .txt file that i wanted to parse its data into .csv columns. It actually got most of it right but after spending hours prompting it, it never really got everything right.

Its worst nightmare was that there were some nested values in the .txt. What amazed me the most is that pasting the error code from colab into the chat, it recognized the mistake and generated a fix.

That being said, most of the times it would try to fix a logical mistake by overengineering a solution to an error, or it would identify the mistake correctly, explain in natural language what needs to be done (impressive), but then give me the code unchanged.

[WheelsAtLarge]

Try a simple calculator. It will give you the complete code for that. I think there's a set limit on output text so that's why your tic-tac-toe was incomplete. I've asked for the calc in Rust and Python and I got a working app for both. It was simple but the fact that it gave me what I asked for is amazing.

I bet that you could break-up your app into functions and have ChatGPT write each function to get to a bigger program.

It's limited now but I can definitely see the possibilities.

It won't be free forever so we'll have to weight the usefulness vs cost eventually.

[Tozen]

It's not that amazing. You can find complete examples of calc in Rust and Python all over the Internet. It just spit out a copy of someone else's code, and gave it to you.

Instead of you having to Google search everywhere or look in multiple sources, you were able to just ask ChatGPT.

[midspectrum]

Yes! The calculator worked and I agree with you that in most of my prompts when I have requested to give elaborated answers, it doesn't give appropriate results.

[gaurangt]

It's ok for small blocks or code, maybe a function or two, even a whole class, but beyond that, it's pretty pathetic. Even the small blocks of code it outputs only compile a few times and have a few errors. At this point, I would be wary of using it in Production systems.

Having said that, I usually compare it with Junior programmers, more like someone just learning a new programming language. You can expect that level of performance from ChatGPT!

[bspammer]

If you know roughly what you want and just don't want to dig through documentation to find the exact correct incantation, it's very good at that.

It's kinda crazy how fast my opinion went from "haha neat, the robot can write poems" to "this is an invaluable tool and I am begging you to let me pay for stable access". I use it multiple times per day, it's easily saved me hours already.

[MrScruff]

There is a character limit on a single response. You can ask things like 'Continue the previous answer' to get it to generate more.

[deadcoder0904]

just "continue" works :)

[binarymax]

When it’s incomplete I ask it to refactor to smaller methods, and if it crops the output again I ask it to show the full text of a method one by one to get the full output. Tedious but it works.

[nounaut]

From what I've seen so far you can tell it it's incomplete, tell it what errors you're getting etc, and it will agree and correct itself until you have a working application.

[sublinear]

I feel like the more interesting question is whether ChatGPT is better at generating political speeches instead?

Seems like there would be far more content to train on and less restrictive criteria.

[warrenmiller]

Tim Corey has some thoughts on that https://www.youtube.com/watch?v=FEsahkX7ZF8