[phil917]

I just canceled my Copilot subscription last night. I definitely never saw the productivity boost that I’ve seen so many claim.

I actually feel like the suggestions seemed to get worse during my month of using it for some reason.

Towards the end, it started suggesting these large blocks of code (another issue I had with the interface as well) which were very not relevant to what I was attempting to write.

All in all, I’m very underwhelmed from my first experience with “AI enhanced” coding.

[simonw]

Did you change the way you worked to help facilitate it at all?

I get great results from Copilot, but that's because I do a bunch of things to help it work for me.

One example: I'll often paste in a big chunk of text - the class definition for an ORM model for example - then use Copilot to write code that uses that class, then delete the code I pasted in again later.

Or I'll add a few lines of comments describing what I'm about to do, and let Copilot write that code for me.

Did you try the trick where you paste in a bunch of code and then start writing tests for it and Copilot spits out the test code for you?

A few more notes about how I've used Copilot here:

- https://til.simonwillison.net/gpt3/writing-test-with-copilot

- https://til.simonwillison.net/gpt3/reformatting-text-with-co...

I've also used Copilot to take educated guesses at things - like an inline mini-ChatGPT - with good results:

- https://til.simonwillison.net/gpt3/guessing-amazon-urls

As with so many other AI tools, Copilot is desperately lacking detailed documentation. It's not at all obvious how to get the most out of it.

[ignoramous]

I sometimes prompt Copilot with if statements and var declarations and it actually ends up suggesting missed corner cases and fixing bugs. Definitely, there's an art and form to using it.

I wonder if this quest for the perfect prompt is dumbing me down, though.

[SparkyMcUnicorn]

Using Copilot has taught me how to write better code comments. And the improved comment quality benefits everyone, not just the AI.

Even something as simple as adding a doc block for a method can boost the quality quite a bit.

I could live without it, but I'd also keep paying if the price increased.

[lostmsu]

Just today Copilot saved me close to 1 hour of typing and replaced it essentially with ~1 minute of pressing Tab.

I was hooking IAudioClient COM class to capture and silence arbitrary app's audio, and as soon as I wrote signatures for its members Copilot was able to generate skeletal stub implementation with logging as well as hooking code totaling about 150 lines of Rust.

[DrakeDeaton]

This is an example of why codebases with strict typing & clear conventions will benefit the most from copilot.

[gavinray]

To offer a counter opinion: while I can live without ChatGPT and cancelled my OpenAI plus membership, I cannot live without Copilot and would be willing to pay a lot to keep it.

There are a lot of tricks to optimizing output from Copilot ("Copilot-Driven Development"), I wrote a little about what I've discovered here:

https://gavinray97.github.io/blog/a-day-without-a-copilot#co...

[mns]

> Use explicit names, erring on the side of overly-verbose (e.g. ensureUserIsLoggedIn() instead of euil())

I don't care that much if I sound pretentious or too harsh, but how is this not common sense? Maybe the general standards in the industry are way lower than they used to be. This looks like when one of the consultants hired by our company and paid 10k+ per month for a ISO 27001 certification said: you need to make sure your password fields show a * instead of the actual character.

After using copilot for 2 months, I found it somehow useful for very generic repetitive cases, but for complex specific stuff it was very bad. I'd rather spend an hour writing code and understanding the problem I'm trying to solve than spending an hour understanding and adapting some generic code that partially does what it needs to do. And if there are people out there that fully trust it and then deploy code that they don't understand and can't explain themselves, then we're in for a lot of fun in the future.

[endisneigh]

Same. It’s funny reading comments on here. Even gpt4 I think is just ok. Maybe the people who see these huge increases are more junior?

[coolsunglasses]

I've been working as a programmer for almost 15 years and I work on fairly touchy (perf, reliability-wise) networked systems in a programming language with a good type system. I find Copilot to be very valuable even if I have to give it a little bump or edit what it produces sometimes.

[simonw]

I'm definitely not more junior, and I'm seeing massive productivity improvement from Copilot and GPT-4 - but I'm finding it takes a lot of expertise to get the best results, both in my ability as a programmer and in terms of knowing the best ways to use the AI tools.

Learning how to get the best results of them takes a great deal of experimentation.

[Tronno]

My experience was very similar. Copilot became a sort of improved autocomplete, that would save copy/pasting repeated variable names into tests, logs, etc, but offered no other significant improvements. Its suggestions were frequently incorrect, often not obviously so.

[benabbottnz]

> that would save copy/pasting repeated variable names into tests, logs, etc

This is all it takes for me to get value out of my subscription. It's saving me time by anticipating my next test / block of code based on what I've previously written, and that saved time really adds up.

Sure it's not "Jesus take the wheel" level AI yet, but time is money, and so is my sanity.

[glaslong]

I find it’s really similar to having a team of junior devs under you. Lots of output, generally good for problems that are googleable, need heavy guidance if the problem is both new and niche, lots of subtle bugs.

The first two points in @gavinray’s methodology being unfortunately critical.

[berkes]

This month, for the third time, I evaluated copilot. It has vastly improved. So far that this time I'll probably pay for it.

I think it's biggest benefit isn't for me, though. Copilot is great at churning out predictable or repetitive stuff, much less in writing a abstractions over or around that repetitive or predictable stuff. E.g. it's great at writing yet another CreditcardPayment ActiveRecord model for Rails. But less so for an obscure business domain model used outside of a framework. And great at writing fifteen getters and setters, less for an abstract base class, DSL or macro that introduces all these getters and setters for me.

It's also bad at certain rewrites. A codebase was slowly rewritten from OldStyle (and old libs) to NewStyle. It stubbornly kept suggesting OldStyle snippets.

And last, I find Copilot has a far higher ROI in some languages and frameworks than others. E.g. dynamic languages (like Ruby) have very poor LSP and intellisense support compared to typed and static (like Rust). So the added benefit differs a lot based on where it's used, is my experience.

I guess esp. the latter is why I too am underwhelmed. But also why I'll keep using it, this time, for when I do the inevitable Rails, JavaScript or Panda's gigs. But less so for my weird eventsourced, hexagonal, Rust project.

[tomashubelbauer]

I also find Copilot to be rather underwhelming and half the time I find myself wishing it didn't go as overboard with the suggestions as it does. Sometimes it suggests exactly what I want to type and then adds a bunch of extra crap towards the end so it is the same amount of effort to type it manually or accept the suggestion and erase what I don't need.

Also its inability to match parens and braces/brackets is legendary. So annoying.

But! It is extremely helpful for filling in error messages. I don't think I wrote a `throw new ` line fully since I started using it. The quality and information density of my error messages has increased considerably.

Not sure if it is worth keeping for just that but it is a nice benefit.

[darkerside]

I found the same, and I was really hoping for a productivity boost. I really wonder how much this correlates to developer speed. I code using Vim and type probably 80-100wpm. Typing is usually not the bottleneck for me. Thinking is. Copilot is good at typing, but not good at thinking. IME. Occasionally it will surprise me with a valid contribution, which I then need to check just as carefully as if I were writing it myself.

[_ea1k]

TBH, using this as a fancy smart autocomplete doesn't seem like the best idea. I can't believe that I'm suggesting something that sounds like Clippy... but...

It really needs to be something where you can rattle off a list of requirements and have it build the code for you. The code context is necessary, but not sufficient.

[bgarbiak]

It’s coming up: https://github.com/features/preview/copilot-x

[wouldbecouldbe]

Yeah not worth it at the moment. Often it annoys me be overwriting legitimate suggestions from the editor.

Only reason I still have it is because the beta claims to have GPT-4 integration and that could prove to be a very powerful feature. But I still haven;t gotten an invite to install.

[62951413]

Are there other people who find that IntelliJ's autocomplete suggestions and shortcuts (including live templates&Co) already do 80% of what Copilot is trying?

[SkyPuncher]

Yea, but that 20% is ultra powerful. It handles the fuzzy things really well and it pattern matches even better.

If I write a comment describing what I’m doing, it will generate multiple lines of correct code

[roflyear]

People love to cargo-cult. It's wild. It was okay but I'm happier with ChatGPT for one-off things I don't know how to do "give me a sql query that does this obscure thing" etc..

[lolinder]

It's okay that it didn't fit well into your workflow, but accusing those who use it of cargo culting isn't constructive. Other people don't have the same workflow as you do, and as simonw noted above, some of us actually adjusted our workflow to get the most out of Copilot.