The introduction of mass market writing tools powered by Artificial Intelligence (AI) has changed higher education. Proponents of AI claim that AI tools should be integrated into lesson design, however, it is also the case that AI may be used by students as an unethical shortcut to wholly complete written assignments. While companies such as GPTZero have responded to academic concerns by creating software designed to detect the use of AI in written work, false positives and underconfidence in the software’s assessment leaves instructors without an actionable means to promote integrity in coursework (Chaka, 2023). Independent of software applications, there may be other signals that distinguish independent student work from that of AI. Gibbs (2023) conducted a review involving 1.2 million words generated by ChatGPT (a commonly used AI tool) and concluded that, when compared to a human writer, it is roughly 1,000 times more likely to use the term “re-imagined,” 400 times more likely to use the term “graphene,” and more than 600 times likely to use the term “bioluminescent.” This approach suggests the value of a similar comparison: to identify terminology that is uniquely human; that is, terms that are largely absent from material that are produced by AI. Such terms may be found within the arena of connective terminology.
Literature review
Writing assignments have traditionally served as a means to support the development of critical thinking skills in higher education. Although AI tools are positioned to replace many of the processes involved in completing writing assignments, universities have embraced the tools as a potential means to redefine critical thinking and writing projects, calling on students to evaluate the quality of AI responses. Of course, one may prompt an AI tool to evaluate its own responses, so ensuring that students themselves conduct the evaluations remains an important endeavor in higher education.
Connectives are a large group of terms inclusive of conjunctions (such as and), prepositions (such as before), and adverbs (such as however). There is a precedent for using such terms as a means to distinguish the backgrounds of language users, often as a means to differentiate between native speakers (NS) and non-native speakers (NNS) of a language. In a comparison of student English essays by NS and NNS of French origin, Granger and Tyson (1996) found that NNS were far less likely to use a term such as “instead” in their writing. At the same time, the researchers found that NNS used a term such as “indeed” at nearly four times the frequency of NS. Ma and Wang (2016) compared essays written in English by British and American students to essays written in English by Cantonese students. In the study, researchers noted many similarities in connective usage, but also noted that NS used the term “because” with higher frequency. Kuswoyo et al. (2020) compared language usage in NS and NNS of English among engineering lecturers. The researchers found that the NNS tended to use “and” and “so” more frequently than NS in lectures.
Beyond connectives, usage of other parts of language have also been leveraged as a means to identify differences in writers’ origins. Zhao (2017) compared language use in four groups: NS and NNS graduate students as well as NS and NNS English scholars. While the author found many similarities among students in terms of connective use, there was a notable difference in the use of logical, grammatical metaphors (using terms such as “factors” to express a causal relationship) when comparing student work to that of scholars.
The literatures’ findings suggest that the use of connectives provides a means to distinguish NS from NNS. Of course, AI writing tools are neither NS nor NNS. AI writing tools are more properly classified as Large Language Models (LLMs). Unlike humans, LLMs generate text based on probabilities. Unlike humans, LLMs do not (presumably) have beliefs or sensory information beyond prompts. Given that both LLMs and humans use language, however, an investigation of term frequency may also provide a means for distinguishing whether written text has been generated by a humans or by AI.
Methods
Given the body of research suggesting that the frequency of terms used may provide information about its author’s identity, a project was launched to determine whether such terms, specifically connectives, might provide objective grounds for differentiating AI writing from that of a student. With permission of the institutional review board (the “Research Institute”), 34,170 words generated by 49 students in response to writing prompts in general education courses at a single-purpose institution were compiled into a single document. The same prompts were submitted to two widely available and free artificial intelligence writing tools, ChatGPT and Bing, in January of 2024. The process yielded 9,503 words generated by the AI tools.
The prompts given to both students and AI were as follows:
- Identify a myth about the populations studied (including older adults and economically disadvantaged). Integrating a citation and a reference, dispel the myth.
- Identify two subprovisions within the American Nurse Association’s ethical code that might conflict with each other. Explain the potential conflict and a potential resolution.
- Describe and assess a hypothetical event using moral theories studied (egoism, determinism, Kant’s Categorical Imperative, consequentialism, and relativism).
To generate 9,503 words from the AI tools, additional prompts such as “A different response please” were issued after receiving its initial response to the identified prompts. Use of common connectives by students and AI were tabulated, respectively, by leveraging the Find tool (CMD+F) within document software.
Results
The results of the study are depicted in Table 1. Calculating frequency of observed instances per 1,000 words provides a venue for comparing connective use in student and AI writing. Based on frequency of occurrence, there was little difference in the use of terms such as “again” or “and.”
However, AI was three times more likely to use the term “however” than a student. Conversely, students were five times more likely to use “if,” fifteen times more likely to use “because,” and ten times more likely to use the term “so” in their writing. Notably, the terms “since” and “too” did not appear in AI writing, but were found 14 and 27 times (respectively) in student writing.
| Term | Collective Student Responses (word count: 34,170) | Collective AI Responses (word count: 9,501) | ||
| Observed Instances | Frequency per 1,000 words | Observed Instances | Frequency per 1,000 words | |
| again | 25 | 0.73 | 8 | 0.84 |
| also | 90 | 2.63 | 16 | 1.68 |
| and | 1,067 | 31.23 | 384 | 40.42 |
| because | 129 | 3.78 | 2 | 0.21 |
| but | 112 | 3.28 | 8 | 0.84 |
| however | 37 | 1.08 | 33 | 3.47 |
| if | 358 | 10.48 | 19 | 2.00 |
| since | 14 | 0.41 | 0 | 0.00 |
| so | 63 | 1.84 | 1 | 0.11 |
| then | 24 | 0.79 | 1 | 0.11 |
| too | 27 | 0.79 | 0 | 0.00 |
Given initial observed differences between student and AI written material, additional terms were searched with a focus on experiences that were unique to conscious beings, such as “think,” “want,” and “believe(s).” Table 2 depicts the results. In particular, students were 17 times more likely than AI to use the term “think.” In a happy accident, a typographical error related to “think” revealed an additional difference in terminology: within student work, the term “thing” was used 109 times. The term occurred only one time in AI work. The term, inclusive of extensions such as “nothing” and “anything,” is 30 times more frequently found in student writing.
| Term | Collective Student Responses (word count: 34,170) | Collective AI Responses (word count: 9,501) | ||
| Observed Instances | Frequency per 1,000 words | Observed Instances | Frequency per 1,000 words | |
| appear(s) | 9 | 0.26 | 1 | 0.11 |
| believe(s) | 38 | 1.11 | 4 | 0.42 |
| feel(s) | 48 | 1.40 | 4 | 0.42 |
| seem(s) | 15 | 4.32 | 5 | 0.53 |
| think(s) | 66 | 1.93 | 1 | 0.11 |
| want | 46 | 1.35 | 4 | 0.42 |
Conclusions
AI software will continue to evolve. Users may direct the tool to leverage terms associated with human writers such as “think” and “so”, and the resulting AI-generated text would perhaps obscure the differences in language usage as observed in this project. Further, the absence of terminology associated with student writing does not impart certainty that a written text has been composed by AI. The results of this project do not provide an indubitable foundation to address academic integrity concerns.
However, the findings of this study do suggest some concrete, measurable differences between student writing and that of AI. Armed with such knowledge, instructors may review student work with a better understanding of features that might suggest reliance on AI that is outside the boundaries of integrity. The study provides measurements that may add depth and understanding to existing hunches and suspicions when reading AI-generated text. Such an understanding provides a better starting point for any potential intervention.
Miriam Bowers Abbott, MA, is an associate professor at Mount Carmel College of nursing in Columbus, Ohio. She teaches courses on ethics and culture and serves as assistant director in the online RN to BSN program.
Wyatt Abbott is a student at Kansas State University where he studies psychology and communication.
References
Chaka, C. (2023). Detecting AI content in responses generated by ChatGPT, YouChat, and ChatSonic. The case of five AI content detection tools. Journal of Applied Learning & Teaching 6(2). https://journals.sfu.ca/jalt/index.php/jalt/article/view/861
Gibbs, J. (2023). Which words does ChatGPT use most? Medium. https://medium.com/@jordan_gibbs
Granger, S. and Tyson, S. (1996). Connector usage in English essay writing of native and non-native EFL speakers of English. World Englishes 15(1). https://www.researchgate.net/publication/227647492_Connector_Usage_in_the_English_Essay_Writing_of_Native_and_Non-Native_EFL_Speakers_of_English
Kuswoyo, H., Sujatna, E. T. , Indrayani, L.M. , & Rido, A. (2020). Cohesive conjunctions and and so as discourse strategies in English native and non-native engineering lecturers: A corpus-based study. International Journals of Advanced Science and Technology 29(7) https://www.researchgate.net/profile/Heri-Kuswoyo-2/publication/344486791_Cohesive_Conjunctions_and_and_so_as_Discourse_Strategies_in_English_Native_and_Non-Native_Engineering_Lecturers_A_Corpus-Based_Study/links/5f7bef34a6fdccfd7b4a7b1d/Cohesive-Conjunctions-and-and-so-as-Discourse-Strategies-in-English-Native-and-Non-Native-Engineering-Lecturers-A-Corpus-Based-Study.pdf
Ma, Y., and Wang, B. (2016) A corpus-based study of connectors in student writing: A comparison between a native speaker (NS) corpus and a non-native speaker (NNS) learner corpus. International Journal of Applied Linguistics 5(1). https://journals.aiac.org.au/index.php/IJALEL/article/view/1968
Zhao, J. (2017). Native speaker advantage in academic writing? Conjunctive realizations in EAP writing by four groups of writers. Ampersand (4).https://www.sciencedirect.com/science/article/pii/S2215039016300650
