Steamrolling Scholarly Expression: 237 Suggestions. How Many Does an Author Actually Need?

My recent project, “Steamrolling Scholarly Expression: A Discourse Analysis of GenAI Revision Advice on Previously Published Articles,” inquired into how chatbots serve as writing partners for scholars (Fig. 1). I participated remotely in the Malaysian Association of Applied Linguistics International Conference (MAALIC 2026), held on 4–6 October at Universiti Sultan Zainal Abidin in Terengganu, Malaysia, to get some feedback on this work (Fig. 2).

Figure 1. Helpful editor or scholarly steamroller? My MAALIC 2026 presentation puts AI revision advice to the test.

Figure 2. The presenters in my panel at MAALIC 2026 joined entirely via Zoom. The conference’s in-person audience is not shown, but we did get thoughtful questions from them.

Using prompt-based discourse elicitation (PBDE), this study compared four AI tools’ responses to requests for language editing and peer-review advice, using three 1,000-word excerpts from my own scholarly work (Fig. 3). I requested up to 10 suggestions for academic editing, then asked for up to 10 more points about what would likely come up in peer review. Zachary Suarez (a PhD student in Applied Linguistics) and I independently coded the 237 advice points, then resolved differences through consensus. The rubric used in the analysis is included in the PDF handout below.

Two chatbots were general purpose (ChatGPT and Claude), and the other two were writing assistants (Jenni and Quillbot). This purposive selection allowed me to examine whether tools positioned specifically for writing offered different revision advice from systems designed for a broader range of tasks. Using the same excerpts and prompts across all four tools provided a common basis for comparison. The aim was to explore variation across these tools, rather than to represent all available genAI systems.

Figure 3. Workflow for prompt-based discourse elicitation (PBDE) of revision advice.

Many suggestions were specific and actionable, yet addressed things that did not need correcting. Approximately 75% of editing advice and 89% of peer-review advice were coded as optional (Fig. 4). This was a bit strange, as the chatbots did not have to provide 10 points each, and the articles were published or accepted for publication already. Fewer suggestions might have been better.

Figure 4. Characteristics of revision advice across four AI tools. Specific, actionable suggestions were frequently judged unnecessary by the two coders.

The tools’ priorities also differed by task (Fig. 5). Language-editing advice concentrated on vocabulary, while peer-review advice focused predominantly on arguments, structure, claims, and evidence.

Figure 5. The focus of chatbot advice: vocabulary dominated language editing, while argumentation and structure dominated peer review.

Some recommendations went beyond unnecessary polishing (Fig. 6). They proposed changes that weakened distinctive authorial voice or disciplinary expression. For example, replacing “Edwards goes so far as to say” with “Edwards argues” removes an evaluative stance, even though both versions are grammatically acceptable. Quillbot’s suggestions were rated 100% optional, and most often depersonalized the articles. In response to the peer review prompt, Jenni was most likely to provide suggestions of sources, living up to its marketing (the quality of those sources was not evaluated, but on first glance they seemed to be reasonable).

Figure 6. Examples of advice coded as optional, depersonalizing, deprofessionalizing, or demeaning.

These findings highlight a challenge for postgraduate writing instruction: students need to assess whether a revision is warranted, how it serves their purpose, and what it might sacrifice. Confident instructions do not necessarily justify changing a deliberate scholarly choice.

I plan to develop this study into an article for the conference proceedings volume. To hear when the full article is available, subscribe on the right by using ‘Updates from Chris’ and select ‘Publications and Research.’ In the meantime, the handout below provides the coding rubric and references from my presentation.