When I first examined whether it’s possible to fight back against AI-generated plagiarism, and how that might work, it was January 2023, just a few months into the world’s exploding awareness of generative AI.
This is an updated version of that original January 2023 article. When I first tested GPT detectors, I used three: the GPT-2 Output Detector, Writer.com AI Content Detector, and Content at Scale AI Content Detection (which is apparently now called BrandWell).
The best result was 66% correct from the GPT-2 Output Detector. I did another test in October 2023 and added three more: GPTZero, ZeroGPT (yes, they’re different), and Writefull’s GPT Detector. Then, in the summer of 2024, I added QuillBot and a commercial service, Originality.ai, to the mix. This time, I’ll also be adding Grammarly’s beta checker.
In October 2023, I removed the Writer.com AI Content Detector from our test suite because it failed back in January 2023, it failed again in October, and it failed in summer 2024. However, it now appears to work, so I’m including it in the test suite. See below for a comment from the company, which their team sent me after the original article was published in January.
What I’m testing for and how I’m doing it
Before I go on, though, we should discuss plagiarism and how it relates to our problem. Merriam-Webster defines “plagiarize” as “to steal and pass off (the ideas or words of another) as one’s own; use (another’s production) without crediting the source.”
This definition fits AI-created content well. While someone using an AI tool like Notion AI or ChatGPT isn’t stealing content, if that person doesn’t credit the words as coming from an AI and claims them as their own, it still meets the dictionary definition of plagiarism.
Test Results
Here are the test results for the above text, which I wrote myself:
- GPT-2 Output Detector: 99.98% real
- Writer.com: 95% human-generated content
- BrandWell AI Content Detection: Passes as human
- GPTZero: 98% human
- ZeroGPT: 22% AI GPT Your Text is Most Likely Human written
- Writefull GPT Detector: 1% likely this comes from GPT-3, GPT-4 or ChatGPT
- Originality.ai: We are 99% confident that text is original
- QuillBot: 42% of text is likely AI-generated
- Grammarly: We didn’t detect common AI text patterns. Your document doesn’t appear to contain AI-generated text.
Final Thoughts
First, I’ll let ChatGPT have its say: AI-driven plagiarism is a growing problem, as AI tools like Notion AI and ChatGPT can generate text that is difficult to distinguish from that written by humans. There are several ways to detect whether a piece of text was generated by an AI, such as looking for repetitive or unusual patterns, checking for a lack of originality, or using a plagiarism checker. Plagiarism checkers like Turnitin, PlagScan, and Copyleaks may be able to identify certain characteristics of language model-generated text, but they are not foolproof.
Even the speciality tools I found were woefully unsuited to the task. There was some limited improvement over my tests in January. However, I certainly wouldn’t feel comfortable threatening a student’s academic standing or accusing them of cheating based on the results of these tools.
At this point, I don’t think we’re ready to let AI fight AI. Instead, we’ll have to rely on human perception (which means folks have to pay attention) to identify works generated by an AI.
Conclusion
So, what do you think? Can you tell if a piece is AI-generated? Have you used these AI writing tools in the months since they suddenly became available? Do you think they’re a help or hindrance to education and journalism? Let us know in the comments below.
FAQs
Q: Can AI-generated text be detected?
A: Yes, but it’s not easy. The results of the detectors I tested were inconsistent and not always accurate.
Q: Are AI writing tools helpful or harmful?
A: It depends on how they are used. AI writing tools can save time and effort, but they can also be used to produce misleading or biased content or to replace human writers.
Q: Can I use AI writing tools in education and journalism?
A: Yes, but it’s important to use them responsibly and to ensure that the content generated is accurate and original.
Q: Are plagiarism checkers effective?
A: Plagiarism checkers like Turnitin, PlagScan, and Copyleaks may be able to identify certain characteristics of language model-generated text, but they are not foolproof.
