The Future of Coding with AI: An Interview with Eli Hooten

In recent months, the tech world has been buzzing about the potential of Generative AI (GenAI) in coding. Headlines like "GitHub Copilot: Your AI Pair Programmer" or "OpenAI's Codex: Turning Natural Language into Code", or “Devin: The First AI Software Engineer” have captured the imagination of developers worldwide. Eli Hooten, a computer scientist and entrepreneur, joins us today to discuss the impact of GenAI on software development. With a background in mobile robotics, human-robot interaction, and huge experience in software testing, programming, CI/CD, code analysis, and automation Eli brings a unique perspective to the conversation. After earning his Ph.D. from Vanderbilt University, Eli co-founded GameWisp, where he served as CTO for six years. During this time, he led product design and significantly grew the software team, gaining firsthand experience with unreliable code frustrations. This realization led him to join Codecov, a code coverage solution, as the technical co-founder and CTO. Codecov was eventually acquired by Sentry, and Eli became Director of Engineering (CodeCov) at Sentry. In this interview, we will explore Eli's thoughts on how GenAI is influencing the software development landscape and what developers can expect in the coming years.

Hi Eli, we're excited to have you here. With your background in human-computer interaction and human-robot interaction, did you foresee the GenAI revolution we're experiencing today?

Given my background and research area it was easy to believe that some relevant advancements would be made in the field of AI in my lifetime. I did not expect, however, that the general public would be exposed to AI in the way that it has, nor did I think these advancements would come this quickly. Ten years ago, I could not imagine a world where AI tools like Copilot and ChatGPT would meaningfully help me in my day to day work, but now that’s where we are.

LLMs are transforming coding practices. Have you experimented with them in coding and testing? What have proven to be effective, and what are the limitations?

I’ve experimented with a few models for coding and testing but I predominantly use ChatGPT. The biggest limitation I’ve noticed is that a developer can’t ask a general question about how to improve code or find an error. We work quickly, so the expectation is the LLM will produce a helpful response when you ask it to make improvements – it can’t. If you tweak the question one way or another, sometimes you get lucky and it works, but there are magic words in this instance. Once the language is dialed-in, like, “Tell me the precise changes to make in this code,” that’s when an LLM becomes useful.

It has been immensely helpful for items I often forget. Regular code expressions that are easily lost on a developer, or something arcane in the software development process that is needed, LLMs give a detailed reply when prompted. It still requires a quality assurance check, but It’s a lot faster than searching for it on StackOverflow or pulling out an old coding book. 

In terms of what limitations remain, I think we’re still struggling as developers to determine whether or not generated code is actually good. A few things make this challenging:

1. Sometimes generated code is just objectively bad. Perhaps it is solving the wrong problem or solving things in an obtuse way.

2. “Good code” can be somewhat subjective, and quality standards are usually agreed on by development teams. The generated code that may be perfectly acceptable to one team in context, may be completely unacceptable to another team or different programming context. Since there’s no one correct answer here, it’s likely we will need to rely on external code quality tools that can be configured by teams themselves – just like we do with human written code today.

Given that LLMs are trained on vast datasets that may not always meet high standards, do you think they can be trusted to generate reliable code, or are they better viewed as time-saving tools with limitations? What do you think about AI-powered coding assistant?

Right now, I advise any developer that codes with an LLM to treat it as a time-saving tool, not the silver bullet they might think or hope it is. There is a copy and paste problem surging with AI responses, which is only adding more bad code to public datasets. The risk is the same with coding assistants. Generally speaking, a coding assistant is primed to produce, test, and fix code better, but it’s still not perfect. 

There are promising changes on the horizon though. Tools are just starting to surface that can build LLMs that run on proprietary data, specific to a business domain. Once AI is trained on a company’s own model, with its own data, is when AI will be well optimized. It will significantly improve the quality of code, but a human still needs to be in the review process for it to function properly.

Is open sourcing important in the development of AI tools for coding and testing?

There are new headlines every week about open source AI, but the idea of open source as software developers know it does not apply. There’s a heavy debate amongst tech leaders about being for or against open source AI, but no one can actually agree on what that means. The Open Source Initiative (OSI), the authority that defines open source, hasn’t determined the criteria necessary to be labeled “open source AI.” There’s a group convened by OSI in the process of agreeing on a definition now. The challenge is the fine details and conflicts of interest, as well as the pace that AI is changing to consider that it is unclear when OSI will have it settled. 

That disclaimer in mind, open source code is important to the general purpose of LLMs – it’s the code that the models are trained on. So the continuation of open source code is crucial to ensure that LLMs keep pace with advancements in software development (new tools, frameworks, programming paradigms, etc that will naturally manifest in open source development). Open source models like LLaMA 3 will advance with a vast array of software developers working independently to make the model better at scale. Despite the belief that LLMs, by nature, will function optimally under closed-source models is shortsighted. LLMs are poised to benefit from open source contributions much like any open source project can.

Recently, you co-authored the paper ‘Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks’. How can AI enhance traditional testing tools to address this, and what role might predictive analytics play?

I think AI is actually very good at contextualizing different error conditions and issues to the code itself. In this regard, AI serves as an “interpreter” that deciphers the generated error and can present, in plain English, the cause and paths toward mitigation. I think we’ll see a similar approach take hold in testing tools over time. Many of these tools can be seen as noisy or irrelevant, and take considerable time for an engineering team to configure and generate meaningful insights for the team. I tend to think that AI will serve a role in making the output of testing and code quality tools more meaningful and useful to developer teams by automating some of this process and removing the clutter. 

As AI takes on more coding and testing tasks, how should developers, particularly at the junior level, adapt their skills for the future? How should computer science curricula evolve to prepare students for the changing landscape?

When GenAI broke through to the mainstream I was initially worried about job security for entry level engineers. At the time, it seemed logical that AI would be capable of doing what a low-level engineer is able to do – that hasn’t happened yet. I believe AI will keep software teams leaner by eliminating the busy work, but not eliminate roles. Aside from the traditional skills that every entry level developer needs, they should adapt to learn how to integrate GenAI tools into their practice. Even if it means honing that skill on their own time, it will be essential to their job function going forward. Ideally, there are courses built into the curricula about how to use ChatGPT or Copilot, or something similar – or in the process of being developed – because that’s critical for students to get ahead. Some institutions are starting to evolve and focus both on getting software to run and how to test and debug it, where traditionally intro courses only teach the former. GenAI is now being used in a testing capacity, and that change is what will attract new students to those schools.  

Looking ahead, what 'grand challenges' in AI should researchers prioritize over the next decade? Is there any specific research areas apart from your everyday job you are following closely? 

Generally speaking, I think AI excels at exploring broad problem spaces and arriving at potentially viable / non-viable solutions very quickly. We already have examples of AI speeding up fundamental research by making clinical trials more efficient, as well as assisting with Alzheimer’s and Parkinson’s research. Ultimately, I think AI will be used to expedite nearly every part of the scientific process: from exploring possible solutions, to interpreting data, and even expediting scientific publication. 

Can you recommend a book/s for aspiring computer scientists / developers/ ML engineers, whether or not it's specifically about CS or ML?

Regarding ML/AI, the topic is so hot right now, I’d actually just recommend using online sources. Online courses, YouTube videos, blog posts, etc. are all hotbeds of activity for learning about advancements in AI/ML; and material suitable for advanced practitioners or even true beginners – it is all just a Google search away. 

Rather than recommend specific material, I think it’s better to advocate for a broader approach to building software (or software companies) in general. I’m of the opinion that while it’s never been easier to write a lot of code quickly, there is no shortcut to build products, understand marketing, or learn how to turn ideas into a business. There is also no substitute for learning the fundamentals. Code output by generative AI is of little value to us as engineers if we do not understand how it works. Secondly, as an engineer, consider generative AI a force multiplier on the act of outputting code, and turn your attention to areas where AI might not be as helpful. Areas like system design, architecture, product design, and usability are still profoundly important when it comes to building software products. Use the time GenAI has afforded you to grow your knowledge in other areas.

Finally, when it comes to specific book recommendations, why ask me? ChatGPT is right there ;).   

Thank you for reading! if you find this interesting, please do share or upgrade to Premium to support our efforts.

We appreciate you 🤍