Author: memalone

How do students FEEL about my new grading policy for the age of GenAI?

Last week, I had my First Year Composition (FYC) students complete an anonymous survey that asked them how they felt about a new assessment strategy I am piloting this semester. The policy, which I detail in a previous post, is designed to resist using plagiarism checkers while still holding students accountable for thinking critically to produce good writing. 

The survey results, which I discuss below, demonstrate a range of reactions. Many students do admit experiencing anxiety when I introduced the policy, but several indicate that their anxiety subsided throughout the process of completing the first project. Most report that, having gone through the process once, they feel confident about their ability to meet these expectations on future projects. Finally, they provided helpful feedback on how to improve the way I introduce and explain the policy.

The Policy

Shortly before completing the survey, students received their overall grade for Project 1 which combines individual scores on several deliverables: process documents, a first and final draft, and a short, post-writing reflection. Rubrics for the first and final drafts included language outlining the policy which states that any draft would lose significant points if it:

  1. includes content that is clearly false (describes events, concepts, ideas, characters, plots, processes incorrectly; attributes fake quotes to real people, etc.)

  2. demonstrates a level of understanding of some topic that far exceeds what we normally see in work from the average to advanced student

  3. uses an authorial voice that deviates from the student’s prior writing

  4. employs a rigidly formulaic structure akin to the tripartite formula that basic ChatGPT prompting yields

  5. has incredibly generic content/lacks specificity

  6. includes excessively flowery or unnecessary jargon

The Survey Results

The survey solicited feedback on how this policy–which we discussed several times at the beginning stage of their writing process for Project 1– affected their emotional state and their writing process. 

Of 22 students in an honors section of English 101 (HON), over 50% reported that they did feel some anxiety after our initial conversation about the grading policy. See Table 1 below.

Table 1: HON Feelings

However, of the 19 students who responded from my next class, a section of English 101 (EH101), only 32% reported that they felt anxiety after I introduced the policy. See Table 2 below.

Table 2: EH101 Feelings

The HON students also appeared to spend more time actively thinking about the policy as they drafted. See Table 3 below.

Table 3: HON Consideration

For the 19 respondents from the EH101 class, no one reported thinking about the traits constantly, and only 11% indicated that they considered the policy often. Most, 58%, said that they thought about it some. See Table 4 below.

Table 4: EH101 Consideration

Despite early anxiety, the majority of both cohorts report feeling either somewhat or very confident about their ability to avoid the traits in future projects. However, more HON students marked feeling “very” confident, perhaps due to spending more time considering the traits during the writing process. See Tables 5 and 6 below.

Table 5: HON Confidence

Table 6: EH101 Confidence

Feedback on how to improve to policy

Finally, I asked “What feedback do you have for how to make the list of traits clearer or for how to make the grading fairer?” I noticed a few trends out of the 28 students who provided a written response:

The policy is clear, fair, and helpful

15 out of 28 students endorsed the current policy. Some had simple responses like “I think the list is pretty clear, so I think it’s fine the way it is.” One person noted that they “really like the list” even though “it does cause me a bit of anxiety in the back of my mind when I’m writing but I think it’s very useful because I know what to avoid when writing and gives me a different way to think about the way I write.” In other words, the anxiety was not debilitating but generative. Another student noted that they:

appreciate having the expectations of what will be flagged as Gen AI shared with us as students, so we know what to avoid while writing. It is VERY easy to meet these expectations by just doing the assignment yourself and there is no reason to worry about getting flagged UNLESS you actually went in and used Gen AI. There are some moments where I was curious if my work could maybe get falsely flagged because of my writing structure that I learned in school (introduction, body paragraphs 1, 2, 3, conclusion) but after writing the assignment I knew it wouldn’t.

The premise of the policy is fine, but students need examples and clarification for certain features

Of the 13 students who had critical, constructive feedback, 5 noted that it would be helpful to see concrete examples of each of the features. One said it would be helpful for me to “give us examples on sentences or paragraphs that may include these features. Try to let us actually learn and visualize the bad features in an active sentence rather than it being an instruction.”

I completely agree with this suggestion. It makes so much sense, and I’m eager to craft activities and assignments that help students identify these kinds of features so they’ll know what exactly to avoid.

Certain features seem unclear or unfair

7 students expressed confusion over specific features. 4 felt that feature #2 (demonstrates a level of understanding of some topic that far exceeds what we normally see in work from the average to advanced student) was not just unclear but potentially unfair. They noted that it feels unjust to deduct points for student writing that seemed advanced because “some students might actually be able to make these connections and have a deeper understanding of the text without an AI generated response.” However, 3 of these 4 thought that providing examples and discussing what exactly feature 2 aims to avoid would help clarify the rule for them.

2 students worried about the flowery language and jargon rule, noticing that they have a natural impulse to use “diverse vocabulary” and that they “don’t want to be afraid to make those additions in my pieces.” However, one did note that “seeing as I didn’t have points marked off of my project 1 first or final draft, I am confident that I know the difference between the two and am not worried anymore.”

Real worry

Of the students who provided written feedback, one did express a great deal of worry about the system in general:

I just get paranoid that somehow I am going to get flagged for using GenAI when I know I am not using it. I also feel like some of the points are hard to avoid. For example, the deep understanding point. What if it is something you have studied a lot in your free time, or something that intrigues you, so you go above and beyond. I also feel that sometimes my voice in writing varies. I get worried that my voice won’t be the same as last time and I will get points off for that. I also get anxious because we are expected to have strong structure, but what if the structure is too structured and then you get points off.

Moving Forward

Thanks to this feedback, I have new ideas for how to strengthen the way I introduce, explain, and demonstrate the policy. Look out for future posts detailing how this grading strategy evolves.

The New Yorker visits the academic world of digital composing and rhetoric 

October’s first issue of the New Yorker includes an article exploring what it’s like to compose with GenAI. While I wasn’t surprised to find this topic in the pages of the New Yorker, I couldn’t believe that author Cal Newport draws on ideas and voices from the journal Computers and Composition, a journal that heavily influences my own research and perspectives on writing, technology, and pedagogy. The journal’s 71st volume, published in March of 2024, explores questions about how we should make sense of the role that GenAI plays in the composing process. From this volume, the New Yorker article highlights a study by Stacey Pigg, one of my favorite researchers, in which she analyzes videos of students using GenAI to write research papers. In this way, the article begins to identify and codify more precisely how writers use GenAI to write.

As someone with an interest in academic research on composing technologies, the article was initially interesting to me because of the unexpected appearance of one of my favorite academic journals in one of my favorite periodicals. However, I was quickly drawn to Newport’s response to the titular question “What Kind of Writer is ChatGPT” as it mirrors what my students and I are finding about this technology. Newport draws similar conclusions to those of former UAB student Haley Wells, who finds that using ChatGPT does not lead to a reduction in time spent but to a boost in support when her brain feels tired at the end of a long day. Newport writes that by using ChatGPT while writing the article:

I wasn’t saving time: I still needed to look up facts and write sentences in my own voice. But my exchanges seemed to reduce the maximum mental effort demanded of me. Old-fashioned writing requires bursts of focus that remind me of the sharp spikes of an electrocardiogram. Working with ChatGPT mellowed the experience, rounding those spikes into the smooth curves of a sine wave….ChatGPT is not so much writing for you as generating a mental state that helps you produce better writing.

Newport ends his article with another reference to a Computers and Composition contributor, Alan M. Knowles, who suggests that we might consider using GenAI to write as “rhetorical load sharing.” Without losing what we value in the struggle of writing, perhaps we can embrace a tool that makes the difficult act of writing a little more inviting.

What to do with student work that uses GenAI irresponsibly?

Even though many of my UAB colleagues and I distrust and choose not to use GenAI detection software, we still want our students to take on the challenge of learning, to produce work that represents exactly where they are in their educational journey, and to demonstrate incremental growth over time. We recognize that all students now have the easy option to offload the cognitive burden of learning to GenAI—maybe we call this cheating, but I’m still not sure. In the English department, our strategy to combat the threat of cheating is to craft exciting, interesting courses that involve some in-person, real-time graded tasks. The courses incorporate complex, particular assignments that engage students on a personal level, require lots of process work and revision, involve collaborative components, and may ask students to cite or include material that they could only understand if they’ve attended class.

While these strategies are pedagogically sound and succeed at motivating many students to learn, some students still turn in work that is clearly AI-generated, whether on homework assignments, project drafts, or even high-stakes finals. My colleagues and I have begun to notice common features that indicate a student has used GenAI in an unsavvy and thoughtless way.

Traits of student writing that signal to instructors that students used GenAI:

  1. includes content that is clearly false (describes events, concepts, ideas, characters, plots, processes incorrectly; attributes fake quotes to real people, etc.)
  2. demonstrates a level of understanding of some topic that far exceeds what we normally see in work from the average to advanced student
  3. uses an authorial voice that deviates from the student’s prior writing
  4. employs a rigidly formulaic structure akin to the tripartite formula that basic ChatGPT prompting yields
  5. has incredibly generic content/lacks specificity
  6. includes excessively flowery or unnecessary jargon
  7. uses a tedious, long winded, but mostly grammatically correct syntax

How should instructors respond to student work that demonstrates any of these characteristics? If your course policies prohibit the use of GenAI on any class work, you have, at best, an awkward and time-consuming situation on your hands in which you must confront the student, ask whether they used GenAI, and hope they admit it so you can offer some kind of warning or penalty. At worst, you face the possibility of mounting a losing plagiarism case in which there is ultimately insufficient evidence to determine one way or the other; this can lead to that trust-violation that you were hoping to avoid by resisting GenAI detection software.

Even if your course policies allow for GenAI use with attribution and the student discloses that they used the tool, how do you fairly grade the work when you have a hunch that the student did not spend much of their own brain power while others in your class put in a good faith effort?

This semester, I am piloting a strategy to deal with this issue that:

  1. Engages students in explicit conversation about the kind of writing they SHOULD and SHOULD NOT produce for the class, and
  2. Uses assessment checklists and rubrics to enforce those expectations

Before I describe the new strategy, be aware that my solution is no panacea for painlessly and simply dealing with unauthorized student-use of GenAI. At best, this approach gives instructors justification for deducting points from work that exhibits the ugliest features of unedited GenAI output without having to explicitly accuse students of using GenAI.

Explicit Expectations in the Positive and Negative

Instructors at all levels know about the importance of explaining grading expectations to students. Usually, we offer checklists or rubrics that articulate the kind of work we want students to produce, but there have always been some assumed expectations that we can no longer take for granted. The first three traits in the list above represent standards for writing that we rarely needed to say aloud before GenAI but that we must attend to now.

This semester, part of my checklists and rubrics for student work assess how true and accurate the information is and how closely the work aligns with the student’s personal writing style and current level of education and experience. At all levels of academia, we encounter students who worry that their own level of writing proficiency is insufficient and that they must do complex acrobatics to write in a more high-brow, academic manner. We can be up front with students at the beginning of our courses that we don’t just hope but we require students to complete work that is true to who they are and that falls within their zone of proximal development (see Vygotsky). Having students respond to a writing prompt on the first day of class and asking for lots of short writing at the beginning of the semester can give instructors a baseline to which they can compare writing. 

Furthermore, my checklists and rubrics formally codify traits of GenAI writing that are unacceptable in student work (#s 4-7 above) and build in an option for assessing a penalty if their work includes any of these characteristics. This looks different from common assessment tools that articulate positive traits of writing and grade students against that standard.

Deducting Points Regardless of GenAI-Use

This semester I will experiment using checklists and rubrics that reserve the right to deduct a significant number of points if the writing demonstrates any of the seven features listed above. In other words, I will never be in the position to say to a student, “You used GenAI to write this. Therefore, I will deduct points.” Instead, I will be able to point to the presence of one or more of the seven features to justify deducting points.

My first draft assessment checklists look almost exactly the same as before, but now they include the following statement: “If your work (1) includes content that is clearly false (describes events, ideas, characters, plots, processes incorrectly, attributes fake quotes to real people, etc), (2) demonstrates a level of understanding of some topic that far exceeds what we normally see in work from the average to advanced student, (3) uses an authorial voice that deviates from the student’s prior writing, (4) employs a rigidly formulaic structure, (5) has incredibly generic content/lacks specificity, (6) includes excessively flowery or unnecessary jargon, or (7) uses unnecessarily complicated syntax, your draft will lose between 10-30 points. The point value will depend on how many of these characteristics are present and the extent to which the characteristic(s) are present. You will have an opportunity to revise for the final draft.”

If the student draft bears all the hallmarks of GenAI writing and truly departs from the kind of work that student has done in class, I could deduct the maximum points which means the student makes, at best, a 70 on the first draft. In my feedback, I will flag which of the characteristics were present and urge them to meet with me or take their work to the writing center to improve. Regardless of whether the student used GenAI, this is their opportunity to try the task themselves, to actually do some critical thinking and tough work to revise, or to seek out the support they need so that their next draft meets my basic expectations. 

The same statement exists on my final draft rubrics, except this time it indicates that “you may lose 50 points.” Here, I have the option to give the student a failing grade if they did not take the opportunity to revise or if a student’s revision suddenly demonstrates these unwanted characteristics. 

Answering objections

The biggest objection to this experimental method is that it may unfairly penalize students who are in the very earliest stages of practicing specific, thoughtfully structured writing. If FYC students turn in generic or formulaic writing, will they fail their first or final drafts? Is this method just as discriminatory against English language learners or developing students as the GenAI detecting software?

No. Professionals in FYC are trained to assess a learner’s current competencies and offer feedback that invites incremental growth. My colleagues and I have read and commented on writing from students of varying levels of English-language proficiency and writing abilities and have a baseline understanding of what growth looks like.

Furthermore, I do not intend to use these assessment tools to punish students but to:

  1. Clarify my expectations for good writing
  2. Help them navigate an academic environment in which other professors may accuse them of using GenAI to complete their work
  3. Challenge them to work hard to respect their own current abilities while also pushing themselves to the next level

 My intent is not to police student writing for these bad traits but to have an option for addressing the most egregious instances of writing that deviates from a student’s prior writing style. I am certain that many readers here are busy formulating additional objections to the strategy or articulating its many flaws. Leave them in the comments and anticipate future posts where I address your concerns and review how students have reacted to this policy in my class over the first several weeks of the semester!

Ending the gen-AI detection war: Turning off Turnitin’s AI-Checker

For many, the immediate response to ChatGPT’s 2022 rollout was to spend money and time developing plagiarism checkers to detect the presence of GenAI writing. Now, at the start of the 2024 fall semester, any simple Google search yields countless articles with titles like “The 14 Best Plagiarism Checkers to Detect ChatGPT-Generated Content” and “How to Detect AI Plagiarism: ChatGPT Checkers to Try and Avoid.” Turnitin launched an updated version of their AI-detection software in April, 2024 which boasts “up to 98% accuracy” in detecting AI-assisted content in student work. (I invite you to pay special attention to that “up to” part of the accuracy claim.)

Despite increasing accuracy, these software, including Turnitin, continue to falsely detect the presence of AI-generated content. While the data on the 2024 Turnitin model is slow to come out, it is important to note that plagiarism checkers often disproportionately flag the work of non-native English speakers relative to their English-speaking counterparts. Regardless of their accuracy, these detectors likely contribute to an uneasy tension between teacher and student. In their 2024 study, Jiahui Luo finds that students are increasingly fearful and mistrustful of their instructors “in the age of GenAI.”

The solution: DON’T SUBMIT STUDENT WORK TO GEN-AI DETECTION SOFTWARE. We waste valuable time and energy worrying over the likelihood of a false positive, and we run the risk of cultivating a toxic culture of mistrust in our classrooms. Yet there is another very basic reason for resisting use of Gen-AI checkers: the presence of some Gen-AI assisted writing does not indicate the lack of creative, critical, original human thought. Researchers in a variety of academic fields have started integrating the tool into their work, and universities are starting to outline policies and principles of how and when to allow GenAI content into academic publishing. Even the Modern Language Association offers guidelines for how to cite GenAI content, recognizing that this kind of writing will increasingly find its way into scholarly work.

But even if we have good reasons for ditching Gen-AI detection software, we still face a conundrum: how do we hold students accountable for doing the messy, often difficult work of learning? Educators have fantastic reasons for expecting their students to resist using GenAI and to use their own brain to do work. It is increasingly clear that to be able to actually use GenAI to write well, users should know what good writing looks like and should understand their own process for achieving it.

Clearly, we have work to do to develop approaches, curriculum, and assessments that hold students accountable for their learning, but I contend that we cannot involve GenAI plagiarism checkers in our toolbox of solutions. Let us commit to leaving this technology behind for good in our search for a writing pedagogy that works in the age of GenAI.

GenAI and the Workplace: Interviewing UAB Alumni Haley Wells

This summer, I interviewed Alabama Holocaust Education Center’s (AHEC) Program Manager, Haley Wells, to discuss the role GenAI plays in her current work. Wells is a recent graduate of UAB’s accelerated master’s program where she completed a Master’s degree in History as well as Bachelor’s degrees in both English and History. During the fall of 2023, Wells took my Introduction to Professional Writing course in which students began to explore and experiment with GenAI as a part of the professional writing process.

While many, including Wells, were initially resistant to this technology, she reports finding helpful applications for GenAI in her work as Program Manager at the AHEC. However, in our conversation, she emphasizes that to effectively use GenAI, the user must already possess a well-developed set of writing sensibilities. By the end of the interview, we conclude that students who want to become effective communicators and good writers should focus on learning and practicing how to write on their own before they turn to GenAI.

Scroll down to read highlights from our discussion, or listen to the full interview here.

GenAI’s value as a brainstorming tool:

Wells notes that her job involves “a lot of collaborative brainstorming,” explaining how she works with colleagues to strategize everything from how to promote upcoming AHEC events to which rhetorical strategies to use when emailing potential donors. When colleagues are busy, however, GenAI: 

“can be a very useful tool for brainstorming in the same way that I brainstorm with my boss in a planning meeting….ChatGPT sort of does that as well. It’s just available all the time. I can use it whenever to ask, ‘Ok how can you shorten this really long bio about someone because I have to write 2 paragraphs for a social media thing.’ Making things shorter is not my strong suit, so let me see what ChatGPT does, and then I’ll take it somewhere.”

GenAI’s value as a kickstarter and model-generator:

While Wells may default to writing most drafts on her own, she recognizes that GenAI can provide a valuable starting place from which to work when fatigue sets in or when she feels unsure of where to start:

“Sometimes it’s the end of the day, and my brain doesn’t work at all anymore, and I just need something to jumpstart what the structure [of some piece of writing] is going to look like, what the flow should look like, or what some of the wording might look like. That’s when I will use AI and say ‘just write me a professional email.’”

GenAI’s major weaknesses as a writing tool:

While Wells finds that GenAI may help expedite work in the drafting stage of professional writing, she finds that, in general, ChatGPT struggles with producing immediately usable writing. For example:

“ChatGPT is not great at summarizing information in a way that is clear and effective. I’ve noticed it will take three sentences that are not related to each other in terms of event, and it’ll make it one sentence and then suddenly that sentence seems like these three unrelated events all relate and caused each other. It cuts words, but it doesn’t necessarily think about the way we need to revise and cut things to actually be clear, keep the meaning, and keep the main points.”

Furthermore, the tool fails to produce writing that sounds human-like. “One of the really weird things about ChatGPT is the very inhuman cadence of the sentence structure.” Rather than choosing reader-friendly structures or words, Wells notices that “ChatGPT…really just loves those big thesaurus words.” Even when she prompts it to “make the wording less clunky,” GenAI often fails to revise in savvy ways.

GenAI users may do LESS work on drafting but must do MORE work on revising and editing.

Wells believes that using GenAI “makes you work harder. It’s not going to make the process easier per se.” There may be less work to do on the front end, but GenAI does not necessarily decrease the cognitive load of writing because everything it generates needs expert scrutiny and tweaking. “If you don’t know how to edit and revise, then you’re not going to know how to use GenAI effectively.”

Wells highlights three major competencies she developed as a student at UAB that make her an effective writer and that allow her to successfully use GenAI. First, one needs to “[know] what an argument looks like, and how to craft one.” Next, one should “[know] how to use evidence effectively” and “how to build paragraphs” using that evidence. Finally, one must understand how “grammar and sentence structure” and diction work to persuade a specific audience. She calls on these skills to edit and revise all GenAI output.

Using GenAI most effectively requires years of practice learning how to write effectively on your own.

Throughout her years at UAB, Wells worked strategically with professors and peers to develop her own sensibilities for what makes good writing:

“For a chunk of time, I was really honing the skills of being able to write the rough draft…and then look back over it typically with a professor.”  They would then “talk about what works and doesn’t work.” For example,the professor “would say, ‘You don’t really have any evidence in this paragraph. You don’t really have a topic sentence.’ And then I would go through and revise based on their comments. The more I did that–and it’s a lot of practice with every single paper–the less I needed them to tell me what didn’t work, and the more I could see on my own.”

She says that with this practice, “you learn to develop those eyes for your own work. The more you work with other people, and you get their eyes, and you continue to practice,” the better you are able to revise and edit any piece of writing.

On the value of taking professional writing classes as an English literature and History major:

Wells admits:  “I did not want to take professional writing classes.” She remembers saying: “I want to write papers, I want to research, and I want to be an academic in academia. I don’t want to [write professionally].”

Yet she is grateful for the opportunity to learn about it now that her job requires a range of writing styles. “All of the writing I had to do for professional writing classes, even though it was painful, was so helpful for giving me the practice…to be able to do these things I need to do today.” She explains that the work she did in professional writing classes helped her “learn how to use the same tools I had been using in academic writing” for “social media posts or an email that we’re writing to donors…or to reach out to another organization to say would you like to partner with us.”

Not only did she get a chance to practice writing in these genres, but she appreciates the opportunity to explore how to adjust word and syntax choices for different audiences and purposes. She notes that “one of the things that made me so much of a better writer was” an experience in class where we workshopped different ways to structure sentences to create different moods or meanings. That makes you so aware of” how different “synonyms…have sort of a different flavor and connotation.” This exercise showed her how a good writer doesn’t choose “a synonym…because it sounds cool.” Instead, a good writer should “pick the word that is most accurate and effective” for whatever audience you’re trying to reach.