Augmenting student feedback with automation

Reviewing students assignments and providing feedback can be a laborious and time-consuming task. Scaling feedback is a problem for courses with large enrollments. Some methods exist to streamline the process and save time, such as

The alternative is to spend a great amount of time marking up a PDF or printed assignment. Most students seek to improve and assignment feedback provides necessary information on their performance. Depending on the student, the feedback may or may not be reviewed. For some students if a grade meets a minimally acceptable threshold they might never review instructor comments. Thus, lots of effort at personalized feedback could go unnoticed. Yet, documenting the assessment is needed, especially for grade appeals.

My first step was to download 6 years of assignments from the university learning management system and extracted their content. This data helped me prepare the configuration file and identify key terms. This manual training process identified common technical engineering skills, computer programs, and certifications. The process also identified campus engineering clubs and leadership roles. These are all terms that could possibly be found with a new batch of resume submissions. In addition to technical content of a resume, there should also be common headings and wording to describe work experience.

The feedback the students would receive needed to note the presence or absence on their resume of the key terms in the configuration file. The resume parser program generated a CSV file with a row for each student. The file had columns for each of the categories and which key terms under those categories were found. I analyzed this file in Excel and manually reviewed the resumes to check for possible additions to the configuration file. This yielded a few new terms and I reran the program. Based on this final file, I then prepared for generating personalized emails for each student.

Students received feedback for 10 categories. These statements included a combination of affirmations and constructive comments for improvements. Additionally in Excel, I calculated for each student how many categories were found out of the possible categories. This allowed me to sort by the metric and review the lower scoring resumes before sending the automated feedback.

Since this was the first semester I used this automated feedback system for the resume assignment, I continued to use a rubric for grading. The automation assisted in providing more personalized feedback and the rubric assessed their resume on 6 categories for content/formatting.

So how did the scores compare between the algorithm and rubric grading? For a sample of of 43, the correlation was 0.415. This is considered a moderate correlation. A confounding factor is I manually trained the algorithm based on a corpus of 400 resumes from past students in a specific engineering program and this semester the course had additional students from a different program.

Based on these results, I can further tune the configuration file as well as make adjustments to the rubric used for the assignment.

Related posts:

I returned to the Cleveland Museum of Art to continue observing how an everyday experience can reflect various dynamics in identity politics. While other people observed the art, I observed them…

Now that we have some hard data on Airbnb, I thought it would be revealing to compare Airbnb and Booking Holdings with objective numbers in an attempt to better understand Airbnb’s valuation and its…

If you are looking for the best things to do in Brighton England, you’ve stumbled on the right blog post. Without doubt, Brighton is an interesting place to visit and is no stranger to fame. In…