Recently, a student came into my office. They were having a problem getting their assignment to work. I asked what they had tried, and they said “I tried Claude, I tried Gemini, and I tried ChatGPT.” This was a very concerning interaction to me. What bothered me was not the AI use (which is allowed by course policy), but that the student didn’t have a good process for making progress once they were stuck. As we worked through this student’s problem, it became clear that they were not developing a mental model of what the problem was, or trying to break the problem up into smaller problems, but instead were trying to solve the problem all at once with a single prompt to an LLM. I call this anti-pattern “slot machine programming,” where the student just kept re-using the prompt hoping the next answer would be correct. Similar to playing slot machines, where each play is statistically independent from the last, running the same prompt through different AI tools doesn’t help a student make progress towards a solution.
Unfortunately, this is a pattern that I am noticing more and more among students. While I have heard professors assert that students use AI because they are “lazy,” that is often not the case for many of my students. In fact, I would say sometimes the students who are (mis)using AI are working harder than the students who are not, because they are not working efficiently, and therefore are spending a bunch of time and effort not making meaningful progress towards their goal.
Slot Machine Programming is one example of a general concern that I am increasingly worried about. Historically, there were a lot of skills that we as educators expected implicitly along the way as they progressed throughout a CS degree. This is sometimes referred to as the “Hidden Curriculum”. Slot machine programming shows a lack of problem solving skills. Other examples of important but often hidden skills include debugging, working with large codebases, version control, command line use, teamwork, software design, technical interviews, documentation writing, and more. While some of these skills were taught in some places (e.g., MIT’s missing semester class, or How to Design Programs ), they were not widely taught directly to most students, and yet we often expect students to pick them up along their learning journey. However, now that the process of learning computer science looks so different, we cannot rely on students learning these skills implicitly along the way, as the experiences that lead to this learning are no longer a part of the CS learning process. Many of those skills were learned as students struggled through trying to solve small problems, and worked their way up to the larger problems. Also, AI use has severely cut into the time students spend struggling to find a solution, and I worry this is reflected in missing skills that are not being developed by students. This problem is only exacerbated as the decline in office hour usage means that students are not developing relationships with other students and TAs, but instead getting their questions answered via AI. AI can answer many of their questions, but AI can’t help them with the questions they don’t know to ask. This is particularly a problem for the hidden curriculum skills that students are not directly taught about, so they don’t have the words and frameworks to make the questions that they have explicit.
As an example, let’s look at the example above, problem solving. The student mentioned above was not able to perform problem decomposition. Historically, even if they are not consciously engaging in problem decomposition, the very nature of writing the code by hand makes it impossible for a human to write hundreds of lines simultaneously, implicitly forcing them into at least some decomposition. Note that this doesn’t imply a good decomposition, and it is true that historically there has often been a high amount of variance as to how adept students were at these “hidden curriculum” skills. However, now a student can easily generate hundreds (or even thousands) of lines of code in one prompt, without ever being forced to make any decomposition (deliberate or not) by the process. Before, a student would have been able to build up their problem solving skills on many small problems, even if never taught explicitly, before arriving at the situation where they have hundreds of lines of code that don’t work as expected. This ability of students to speed through the process of software development, bypassing many activities that previously implicitly taught them various skills, will affect many other educational outcomes as well.
Why does this matter? Should we just embrace AI and expect it to automate all these problems away?
While AI has completely changed the game when it comes to writing code, it is not at all clear that Software Engineering (i.e., building computer systems on time, on budget, of the correct quality) is a solved problem. In fact, as the price of writing code approaches zero, what is left is the work of software engineering. And that work is becoming a larger and larger part of the average developer’s day. And while in the long run, these sorts of higher order software engineering tasks may one day be automated, for now, it is my experience that getting to impressive results with AI tools (which are achievable!) is only through the application of software engineering principles and skills to the process. This is also the feedback I get from many practitioners I have discussed this topic with across a wide range of companies and industries.
So how should we approach the hidden curriculum? Instead of just giving up, we should instead find ways to uncover the hidden curriculum and explicitly teach students the skills that they will need to be successful going forward. This is a double challenge, because we both are needing to teach skills that previously we didn’t explicitly teach, but also simultaneously reevaluate the curriculum, and decide which skills still matter, and which ones are less relevant in the age of AI.
Let’s return to our example of problem solving. While this is something that has not generally been an explicit topic of instruction in software engineering, it can be taught. In his seminal book How to solve it, George Pólya presents a structured approach to solving math problems, which includes problem decomposition. Alan Schoenfeld builds on this work in Mathematical Problem Solving with a cognitive framework, while showing that the How to solve it strategies alone are not enough, and need to be strengthened by complementary domain-specific tactics.
Let us therefore consider some computer science domain-specific tactics. Briana B. Morrison, Lauren E. Margulieux, and Mark Guzdial studied this directly. Rather than simply asking students to solve problems, they had students work with subgoal labels. They found that students who worked with subgoals, outperformed students who didn’t. They also found that more time spent learning produced better performance.
Beyond just showing that problem decomposition is useful, research (by Dastyni Loksa, Amy Ko, Margaret Burnett and others) shows that it is indeed possible to teach the problem solving skills necessary for students to be successful at programming. In their case, they were teaching students a six stage process based on the psychology of programming literature. The stages they suggest are:
- Reinterpret problem prompt.
- Search for analogous problems.
- Search for solutions.
- Evaluate a potential solution.
- Implement a solution.
- Evaluate implemented solution.
It is important to note that when they developed this process, students would have spent a very large percentage of their time in the “Implement a solution” phase of this process. However, now that LLMs have reduced the cost of that to almost zero, anyone who is looking to follow this process is going to be spending the majority of their time on the remaining steps outside of “implement a solution.”
It is clear that there have been drastic changes to how software is developed due to AI, but also how students are learning how to develop software in the age of AI. We cannot count on this new process to achieve all the same implicit learning outcomes that we could rely on before. As with the explicit curriculum goals, there is a lot of uncertainty about what parts of the hidden curriculum will still be relevant moving forward. I strongly believe that problem solving will be useful for students as long as there are problems to solve, but skills like command line tools, or software archeology might be easily replaced with AI. However, as educators we should be particularly aware of the fact that there is a hidden curriculum and evaluating which parts we should ensure students are still being taught. The process of writing this essay has led me to the conclusion that I should explicitly teach problem solving in my software engineering course, but while I have some ideas of what I hope to do next semester, this is not something I can yet claim is solved. However, as we continue to update our courses to respond to all the changes around, I encourage instructors to look for anti-patters such a slot machine programming, particularly as a way to uncover skills and knowledge that students are not longer acquiring, and find ways to ensure students are still being taught what they need to be successful. It is only by putting student outcomes first that we can find true success.