Skip to content
CS Capstone Handbook

Generative AI

You’ve started using an AI tool on a real project, or one has just handed you a confident answer that turned out to be wrong. This page covers the habits you bring to the tool as an individual: where to spend your attention, how to question and learn from what it gives you, how to choose a model, what not to paste into a tool, and what the research says about how much these tools help. It doesn’t cover setting up a repository for coding agents (instruction files, hooks, MCP servers) or the norms a team agrees on.

You’re responsible for the correctness, security, licensing, and professionalism of anything a tool helps you produce, whoever or whatever wrote the first draft. Treat its output the way you’d treat a pull request from a stranger: consider it, check it, and own what you merge.

What AI Can and Cannot Do

Section titled “What AI Can and Cannot Do”

The tools do well where the space of acceptable answers is wide, where you can recognize a good answer faster than you could produce one, and where a wrong answer is cheap to detect. Brainstorming, a first draft of a test suite, and an explanation of an unfamiliar API fit all three; a security-critical authentication flow fits none of them.

Where the boundary falls is harder to see than that test suggests. Dell'Acqua et al. (2026) ran a preregistered experiment with 758 Boston Consulting Group consultants using GPT-4. On tasks inside what the model could do, consultants with AI finished more tasks, faster, at about 32% higher quality. On a task chosen to sit outside it, they were 19 percentage points less likely to reach a correct answer than consultants working alone. The authors call this a jagged technological frontier, because tasks that look equally hard can fall on opposite sides of it. A task can also move inside it when a new model is released. Their tasks were consulting work, not software, but the practice carries over: find the frontier for your own work by trying tasks and checking the results, and check again when you change models.

Some limits won’t close with a better model, because the information or the responsibility isn’t the model’s to have. A model can’t apply your project partner’s unstated preference, because nobody wrote it down, and it can’t make an ethical call on your behalf, because that call is yours to own.

Hallucinations

Section titled “Hallucinations”

A hallucination is fluent, confident output that isn’t grounded in anything real: a library that doesn’t exist, an API method nobody wrote, a citation to a paper nobody published. The model isn’t lying. It generates a plausible continuation of the text so far, and nothing in that process checks the result against the world. Check every specific fact a tool gives you (a package, a function signature, a flag, a citation) against its source before you rely on it.

Package names are the best-measured case. Spracklen et al. (2025) generated 576,000 Python and JavaScript samples with 16 code-generating models and found that, on average, at least 5.2% of the packages recommended by commercial models and 21.7% of those recommended by open-source models didn’t exist. That’s a security problem as well as a correctness one, because an attacker can publish a malicious package under a name that models keep inventing, and the next person who trusts the suggestion installs it (Spracklen et al., 2025).

AI search tools that attach citations to their answers fail the same way. A citation can be fabricated, or attached to a source that doesn’t say what the answer claims, and you won’t see either unless you open it, so open each one before you repeat what it supports.

Sycophancy

Section titled “Sycophancy”

A sycophantic answer tells you what you appear to want to hear. Sharma et al. (2024) found five AI assistants of the GPT-4 generation consistently tailoring their answers to the user’s stated views: praising an argument more when the user said they liked it, apologizing for and sometimes abandoning a correct answer when the user asked “are you sure?”, and shifting an answer toward one the user suggested. They conclude it’s likely driven in part by training on human feedback, since both people and the models trained to predict their preferences sometimes preferred a convincing sycophantic answer over a correct one. Those models are several generations old, so expect the size of the effect to differ in the tool you use, and keep the habit that follows either way.

The habit that follows is to keep your opinion out of the question. “What’s wrong with this schema?” gets you a review; “this schema is fine, right?” gets you agreement. Once you’ve settled on a plan, ask the tool for the strongest case against it before you commit, because a question that already sounds settled pulls the answer toward agreement.

Using AI Effectively

Section titled “Using AI Effectively”

Start with clear intent

Section titled “Start with clear intent”

Before asking a tool to generate code, know what you want. “Build the authentication system” is a poor prompt. “Add a login endpoint that accepts email and password, validates against the users table, and returns a JWT” gives the tool enough context to produce something useful. Be this specific because the model fills every gap in your request with its most likely guess, and its most likely guess isn’t your design.

Say what done looks like as well (which behavior changes, which tests must pass, what must stay as it is), because a finish line you leave out gets filled with the same kind of guess. Write that finish line into the repository, so a task inherits it without your having to repeat it.

Verify what matters, and build the checks that let you trust the rest

Section titled “Verify what matters, and build the checks that let you trust the rest”

Generated code can look correct while being subtly wrong: off-by-one errors, missing edge cases, insecure defaults, hallucinated library methods. Reading every line stops working once an agent writes more code than you can read, the same way a tech lead can’t read every line their team writes. Spend your attention where mistakes are expensive to undo. Database schema, authentication and authorization, deployment topology, data migrations, licensing, and public interfaces are hard-to-reverse decisions: getting one wrong costs far more than the review would have, so validate those yourself.

Work behind your checks is easy to reverse: once a test tells you a feature is broken, the fix is one more commit. Delegate that work as far as your checks catch, and no further: tests that would fail if behavior broke, CI running on every pull request, review gates on the paths that are hard to undo, a staging environment, and a rollback path you’ve actually tried. To hand off more, improve the checks rather than reading harder. The industry report on AI-assisted development from DORA (DevOps Research and Assessment), Google Cloud’s research program, found that AI’s positive influence on team performance was amplified where developers rolled back changes more often (DORA, 2025). Measure what your checks catch with Audit Your Safety Net.

Checks catch what they were written to catch; they don’t show that a change does what you meant. Simon Willison argues that your job is to deliver code you have proven to work, in two steps: exercise the change yourself and watch it do the right thing, then leave a test that fails if the change is reverted. Put the evidence in the pull request (the commands you ran and their output, or a screenshot), because a change submitted without it hands the verification to whoever reviews it.

Vibe coding and augmented coding

Section titled “Vibe coding and augmented coding”

Andrej Karpathy coined vibe coding for accepting whatever the model writes without reading the diffs, and letting the code grow past what you understand. Willison argues that most AI-assisted programming isn’t that, and that vibe coding suits low-stakes work: a throwaway prototype, a personal tool. Kent Beck calls the alternative augmented coding. You still care about the code, its complexity, and its tests, and you watch for signs the tool is going off track: functionality you didn’t ask for, and any sign it’s cheating, such as disabling or deleting a test to make the suite pass.

Vibe-code a spike you’ll throw away, and augment anything someone else will build on, because whoever builds on vibe-coded work inherits code nobody has read.

Use AI to learn, not just to produce

Section titled “Use AI to learn, not just to produce”

A tool that writes the code for you can also keep you from learning how it works. In a preregistered randomized trial by researchers at Anthropic, Shen and Tamkin (2026) (preprint) had 52 Python developers learn an unfamiliar library with or without an AI assistant. On a quiz taken afterward without AI, the AI group scored 4.15 points lower out of 27 (the authors call it 17%), and it wasn’t significantly faster. How people used the assistant mattered: in a qualitative look at small subgroups, those who delegated the whole task, leaned on it more as they went, or debugged by asking it again and again averaged under 40% on the quiz, while those who asked for explanations, or asked only conceptual questions and fixed their own errors, averaged 65% or more.

So when you’re learning a library, a language, or a part of the codebase, write the code yourself and ask the tool conceptual questions. When a tool generates code you don’t understand, ask it to explain the code, then check the explanation against the documentation, because the explanation is generated text like the code and can be hallucinated as well.

Don’t commit code you can’t debug or modify, whether a model wrote it or a teammate did, because once it’s merged, you and your team are the ones who fix it when it breaks.

Know when to stop prompting

Section titled “Know when to stop prompting”

If you’ve gone back and forth with a tool several times on the same problem and the output is still wrong, stop. Another retry with the same context gives the model nothing new, so read the documentation, look at a working example, or ask a teammate. When you come back to the tool, start a fresh session with what you learned, because a long session carries every wrong turn along with it.

Check what you paste

Section titled “Check what you paste”

Decide what you’re about to paste before you paste it, because the provider’s terms decide what happens to it next. Those terms differ by plan (some train on your conversations unless you opt out, some are cheaper because they do), so read the ones for the plan you’re on, not just the tool’s name.

Your team decides which material (partner data, credentials, meeting content) may go into which tools and writes that rule down: the version for people in the working agreement, and the version for the tools in AGENTS.md. Your part is to check the rule before you paste, because the check takes seconds and a leak can’t be undone.

What You Can Automate

Section titled “What You Can Automate”

When you ask “what can I hand off?”, use this table as the index to the guide that covers each kind of work.

Automatable Where it is covered
Test generation across the pyramid (unit, integration, end-to-end) Testing strategy
Browser-driven exercising of real user flows End-to-end tests, and Playwright MCP, which lets an agent drive a real browser
Accessibility auditing (axe-core, Lighthouse, lint-level rules) Automated accessibility testing
Lint and format checks Hooks and gates
CI pipeline construction and check configuration DevOps
Self-audit and adversarial review passes Run a self-audit and an architectural review pass
Architectural review as a distinct step Technical design
Security scanning (dependency audit, secret scanning) Security
Documentation and changelog generation Documentation

Four judgments stay with you: whether the product is actually good for the person using it, which decisions are hard to reverse, the tradeoffs your users and your project partner will feel, and whether an automated finding matters.

Choosing a Model and an Agent

Section titled “Choosing a Model and an Agent”

The tools change faster than any page about them, so this section covers how to choose rather than what to choose.

Autocomplete, chat, and agents

Section titled “Autocomplete, chat, and agents”

AI coding tools come in three shapes. An autocomplete tool suggests the next few lines as you type, and you see each suggestion before it lands. A chat assistant answers in a conversation, and you decide what to copy out of it. An agent plans a multi-step change, reads several files, runs commands, and edits your codebase before you see any of it. Give an agent more oversight than autocomplete for that reason, and more again on a team where several people run agents against the same repository.

Agent and model

Section titled “Agent and model”

A coding agent is two choices: the program you run and the model it calls. Some programs work only with their vendor’s models; open ones, such as OpenCode, connect to any provider you have an API key for, so you can change the model without changing how you work. What the agent knows about your project comes from the repository and the task, not the model, so a repository set up for agents serves every teammate, whichever model each one brings.

Comparing models

Section titled “Comparing models”

Many labs publish capable models at a wide range of prices, several of them with open weights that anyone can host. Which one gives the best value for a given kind of task changes as new models are released, so compare them on public measurements rather than reputation.

A benchmark is a fixed set of tasks with a score, and each measures one kind of work, so read what a benchmark tests before you read its ranking: a model that leads on isolated coding puzzles may trail on a change that spans a large repository. Epoch AI’s Benchmarking Hub and LiveBench track current results. A Pareto curve plots performance against price, and the models on it are the ones no other model beats on both at once; for any model below the curve, some model on it does at least as well for no more. arena.ai publishes these curves for coding, agent, and general text tasks.

Use the measurements to shortlist, then decide on your own work. Take the models on the curve within your budget, give each the same task from your repository, and run your checks on the result, because a leaderboard score comes from other people’s prompts on other people’s code. Then take the cheapest model that passes.

Where your data goes

Section titled “Where your data goes”

Who hosts a model decides where your prompts go, not who trained it. An open-weight model can be served by the lab that made it, by a third-party provider, or on your own hardware, and each host has its own data terms. Some institutions and governments bar specific providers’ services on the devices they manage, so read the rules that govern your work as well as the provider’s terms, then apply Check what you paste.

Best Practices for Using AI Tools

Section titled “Best Practices for Using AI Tools”
  • Set your repository up for agents before you rely on them.
  • Decide the hard-to-reverse decisions yourself. If you can’t tell which they are, Map Your Hard-to-Reverse Decisions first.
  • Build the checks before you delegate more.
  • Prove each change works before you ask for review.
  • Ask neutral questions, and ask for the case against your plan.
  • Check your team’s confidentiality rule and your plan’s data-use terms before you paste anything that isn’t your own code.
  • Prefer the cheapest model that passes your own checks, because the checks, not the model, decide how far you can delegate.
  • When a tool misleads you, write down what happened and what caught it, because the record shows which of your checks work.

Some Truths About AI Tools

Section titled “Some Truths About AI Tools”

Your sense of being faster isn’t evidence that you are. In a randomized trial run by METR, an AI research organization, Becker et al. (2025) (preprint) had 16 experienced open-source developers work on 246 tasks in mature projects they had worked on for years, each task randomly allowed or disallowed early-2025 AI tools. With AI allowed, tasks took 19% longer. Before starting, the developers forecast that AI would cut their completion time by 24%, and after finishing they estimated it had cut it by 20%. So measure something (time from starting an issue to merging it, the share of generated changes you later revert) instead of trusting the feeling.

The speedup itself depends on who you are and what you’re working on, and the evidence points both ways. The METR authors caution against generalizing from their result: their developers knew their large, mature repositories intimately, and the authors note their results are consistent with small greenfield projects or unfamiliar codebases seeing substantial speedup (Becker et al., 2025) (preprint). Cui et al. (2026) combined randomized field experiments at Microsoft, Accenture, and a Fortune 100 company, 4,867 developers in all, and found that an AI code-completion assistant increased completed tasks by about 26% (standard error 10.3%), with less experienced developers adopting it more and gaining more, though each experiment on its own was noisy and results varied across them. METR has since written that developers are likely more sped up by early 2026 tools, but that its follow-up study gives only very weak evidence for how much, because developers who didn’t want to work without AI declined to take part.

A tool can make you more confident and less careful at the same time. In a user study, Perry et al. (2023) gave 47 participants five security-related programming tasks. The 33 who had an assistant built on OpenAI’s codex-davinci-002 model wrote insecure solutions more often on four of the five tasks than the 14 who didn’t, and were more likely to believe their code was secure. Participants who put more work into their prompts, such as supplying helper functions, were more likely to reach a secure solution (Perry et al., 2023). That model is several generations old, so don’t read the size of the gap as current.

The same pattern shows up outside security. Lee et al. (2025) surveyed 319 knowledge workers about 936 of their own uses of generative AI at work and found that the more confidence people had in the AI for a task, the less critical thinking they reported doing on it, while the more confidence they had in their own ability, the more they reported. It’s a self-report survey, so it shows an association rather than a measured effect, but it points the same way as Perry’s result.

The bottleneck moves from writing code to verifying it. When a model drafts the code, your throughput is limited by how fast you can check it, and checking doesn’t get faster because generation did: a team that doubles its output with the same review capacity builds a queue of unreviewed changes. The DORA industry report found that AI adoption now improves software delivery throughput but still increases delivery instability, which the report reads as teams adapting for speed before their underlying systems can safely manage AI-accelerated development (DORA, 2025). Put your effort into the checks before prompt technique.

The findings on this page come from tools that have since been replaced, because research takes longer to publish than a model takes to be superseded. Check a finding’s date before you lean on its numbers, and ask whether its mechanism (overconfidence, unreviewed batches, skipped learning) still applies to the tool you use.

AI Tools in Industry and Academia

Section titled “AI Tools in Industry and Academia”

The DORA industry report surveyed nearly 5,000 technology professionals: 90% used AI as part of their work and more than 80% believed it had increased their productivity, yet 30% reported little or no trust in the code it generated (DORA, 2025). Its central finding is that AI acts as an amplifier, magnifying the strengths of high-performing organizations and the dysfunctions of struggling ones (DORA, 2025).

The open-source world is more divided. Some projects ban AI-generated contributions outright: Gentoo forbids them, and NetBSD treats code from a large language model as tainted and bars it without the core team’s written approval. Daniel Stenberg, curl’s lead developer, has written about AI-generated security reports to curl’s bug bounty: they look plausible, so each takes longer to research and discard, and that time comes out of fixing real bugs. If your project contributes to an open-source project, read the upstream’s policy before you delegate anything, and expect a maintainer to ask what you verified.

In academia, publishers keep the accountability with people but differ on disclosure. Nature Portfolio won’t accept an AI tool as an author and asks authors to disclose their AI use. The ACM requires every author to be an identifiable human and AI used to conduct the research, including writing its code, to be described in the methods section, but no longer requires disclosing AI used only to help write the paper. If your project’s code feeds a paper, record which parts a tool wrote as you go, because a methods section written from memory months later won’t be accurate.

References

Section titled “References”
  • Becker, J., Rush, N., Barnes, E., and Rein, D. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv (METR). Preprint. https://arxiv.org/abs/2507.09089
  • Dell'Acqua, F., McFowland III, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., and Lakhani, K. R. (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science. https://doi.org/10.1287/orsc.2025.21838
  • Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., and Jadliwala, M. (2025). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. 34th USENIX Security Symposium. https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen
  • Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., and Perez, E. (2024). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations (ICLR 2024). https://proceedings.iclr.cc/paper_files/paper/2024/hash/0105f7972202c1d4fb817da9f21a9663-Abstract-Conference.html
  • DORA (2025). State of AI-assisted Software Development 2025. Google Cloud, industry report. https://services.google.com/fh/files/misc/2025_state_of_ai_assisted_software_development.pdf
  • Shen, J. H., and Tamkin, A. (2026). How AI Impacts Skill Formation. arXiv (Anthropic). Preprint. https://arxiv.org/abs/2601.20245
  • Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., and Salz, T. (2026). The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science. https://doi.org/10.1287/mnsc.2025.00535
  • Perry, N., Srivastava, M., Kumar, D., and Boneh, D. (2023). Do Users Write More Insecure Code with AI Assistants? Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23). https://doi.org/10.1145/3576915.3623157
  • Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., and Wilson, N. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). https://doi.org/10.1145/3706598.3713778

Additional Readings

Section titled “Additional Readings”

Sources and further reading

Activities that exercise this