Reflections on a week of using GitHub Copilot
Observations about AI coding companion GitHub Co-Pilot
I have been using GitHub co-pilot since the last week and I have been surprised in some ways and unimpressed in other ways. I do find the tool to be useful overall though.
The good parts
Copilot is extremely fast in coming up with multiple code complete suggestions. Around 30% of the times I accepted the suggestions without any modifications. It has a decent grasp of the context of your code, like the code above and below, opened files etc. It learns from the way you code as well. If I do not take Copilot’s suggestion and instead do something new, it realizes it was wrong and plays along well.
Also, it reduces the number of my keystrokes, which is incredibly useful for people with slower typing speed (including me). I have been using all the IDE features like version control UI, code generation, run configurations etc. and this is a boon for me.
The challenges
I am used to using classic IDE auto-complete suggestions. They are often correct because those are based on actual classes and compiled code. Also, the IDE code generation capabilities and refactoring rules ensure that the code before and after is logically equivalent. I found Copilot to be making up stuff a lot of times. Since I have the muscle memory of using the suggestions as is (thanks to awesome IDE code suggestions) I often found myself accepting sub-optimal or flat-out incorrect code from Copilot. Changing it back again and fixing it was a little frustrating and I wondered if I could have written it better myself. Also, I missed correcting or verifying the Copilot generated code a couple of times, which was scary. We know that Large Language Models (LLMs) often hallucinate, and Copilot is based on one such LLM (OpenAI Codex), it’s hard to be mindful of that while coding.
Another challenge is that it does not usually follow the best practices or the rather the practices that are consistent with the code base. E.g., Using for loops vs the lambda functions. This may or may not be a big deal, but since new developers would be relying on such tools, they need to be careful.
I also often found the Copilot to be glitchy, I am sure this would be resolved as the service matures.