AI in Software Engineering: Practical Insights from the Field

At Auditrol, we have incorporated AI-powered tools like GitHub Copilot into our software development process. Over several product release cycles, we’ve observed valuable patterns in their usage. Below, I share lessons learned, along with specific examples, to help others.

Strengths:

Commit Messages and PR Summaries: Copilot consistently produces clear, concise commit messages, saving time and creating consistency across the team. For instance, when merging a large feature branch that touched multiple modules, Copilot generated a commit summary that highlighted the main changes: “Refactored authentication logic and added new API endpoints.”

• Code Review Assistance: Our standard process is having a lead or senior engineer review code for business logic and feature alignment with user flow, then leverage Copilot for a secondary review. While Copilot may suggest improvements, they are proven to be misleading without business logic context.

• Code Auto-Completion: Excels at generating boilerplate code for simple APIs and CRUD operations on well-defined schemas.

• TypeScript Optimization: For UI projects, Copilot helps refine complex prop and state definitions, improving component reusability and minimizing runtime issues.

• UI Layout Fixes: Effective for diagnosing subtle CSS bugs or layout issues. When a component displayed unexpected overflow, Copilot quickly suggested a flexbox adjustment that resolved the problem.

• API Integration Planning: Outlining how a new UI component communicates with backend endpoints can be streamlined with Copilot. It helps map out integration points, state management responsibilities, and placement of business logic, which reduces back-and-forth during team alignment.

• Component Refactoring: As components grow and require decomposition, Copilot assists in restructuring them into focused, maintainable units.

• Generate Synthetic Data and Test Cases: Generate synthetic data and baseline coverage to validate features.

• Early UI/UX Feedback: Provides early feedback on user flows, design elements like spacing and color harmony, aligning to our branding.

Limitations:

• Contextual Understanding During Code Review: Copilot’s suggestions can lack business context. For example, it might recommend a change that aligns with general coding standards but misses a requirement specific to our domain.

• Complex Logic and Integrations: When building features involving external services, custom permissions, or advanced data processing (like chart generation), Copilot often requires significant prompt refinement and still may not fully capture business intent. In these cases, we have spent more time in refining prompts than building features.

• Integration Testing: Engineers should maintain oversight on integration testing, as Copilot does not understand the full business scope or user journey and may overlook critical scenarios.

• Security: While Copilot can surface code vulnerabilities, its recommendations are not always accurate or comprehensive. We continue to rely on external scanning tools and manual review to assess risk.

Our Approach:

Upgrading to newer Copilot models hasn’t eliminated these limitations. Instead, we focus on training engineers to deeply understand business requirements, UX flows, and security implications so they can leverage Copilot effectively.