Assessing the Effectiveness of an Automated Problem Generator to Develop Course Content Rapidly and Minimize Student Cheating

Philip Jackson, Ricker Lamphier
2020 ASEE Virtual Annual Conference Content Access Proceedings   unpublished
The education environment has taken a dramatic shift in the last decade with a greater focus in online delivery. In online and traditional classes alike, engineering faculty rely on textbooks, online publishers' content, and problem repositories for class exercises as writing new problems every semester is prohibitive. The problem with this is that both students and faculty have access to the same information on the web. With digital solution manuals or solution websites, the traditional
more » ... y of homework has lost its integrity as the challenge of homework is no longer figuring out how to do the problem, it is now where does one find the online solution to a specific problem. This project documents the development, implementation, and testing of a true random problem generator in Python to create original content quickly whose solutions are not ubiquitous or accessible via some online solution manual. With true random problem generation, the inputs define the problem as an archetype (where archetype is defined as a specific problem type (e.g. projectile motion, ideal gas piston cylinder, etc.)) and can have virtually an infinite set of problems generated from it. The script will read a user defined input file, randomize parameters, randomize values for those parameters, test for logical correctness (fidelity), and finally use a natural language algorithm to generate both problem text and solution. To assess the effectiveness of problems created in this way, a survey was conducted having students rate the generator's problems based on usefulness, clarity, and other qualitative features, as well as compare them to traditional, hand-written problems. A separate survey was conducted to score the accuracy of the generator's difficulty heuristics by comparing the calculated difficulty of the software with the average of that perceived by the students. The assessment of the reduction of cheating was measured via blind survey and performance analysis. All-together, the functionality of this automated problem generator is established based on the qualitative measure of the aforementioned criteria, and overall perception of the potential future of online learning.
doi:10.18260/1-2--34181 fatcat:kl4wd5pnavge5ns7uxyy2bqvwq