AI Series Part III
By Bob Deakin
Whether you like them or not, AI bots are our new neighbors and we should get to know them better. Rather than offer a plate of cookies, we can use AI detectors to see behind the curtain. Tools like ContentAtScale, Originality.A, Writer, and ZeroGPT are current favorites.
The focus of last week’s story was “How do I detect AI-generated content?” Focusing on written content, I introduced writing tools such as CopyAI and Rytr, and LLMs (large language models) like ChatGPT and BERT. These are just a few of many AI writers in the blossoming industry.
I use the term “AI Detector” loosely, as many AI generators are also detectors. Gone are the days of ‘robot speak’ (or write) as seen in old media. Now, you can generate well-crafted content in seconds. Let’s test the waters.
What I Wrote
As a guinea pig, I’ll use ZeroGPT to scan my two-factor authentication story from earlier this year. I enter the full text of the story and click Detect Text. Surprisingly, it claims that 11.26 percent of my story is AI-generated. Admittedly, I have been accused of being robotic in my life, but this is the first statistical accusation.
I assure you the story comes from research and imagination. I’m almost flattered to be accused of being a robot. To me, that reads ‘technical proficiency,’ but it could also be a sign to relax my style. I’m on it, dude.
If you’re keeping score, WriterAI and Originality.AI gave me a score of 100 percent human, and ContentAtScale scored “Passes as Human.” I learned a lot about myself today.
What the Robot Wrote
I used the same tools to test AI-generated content. This time, I asked CopyAI to write a short story with the keyphrase “two-factor authentication.” The robot churned out four paragraphs of content that looked fine, though it seemed like it needed coffee.
The scores:
WriterAI gave me a score of “100 percent Human-Generated Content - Fantastic!”
Once again, ContentAtScale scored “Passes as Human.”
However, Originality.AI ruled “0% Original, 100% AI.”
How can they be that far apart? Is each affiliated with a political party? As I quoted UMD Professor Soheil Feizi in my previous story, “Current detectors of AI aren’t reliable in practical scenarios.”
Indeed, testing is not that simple. Expert users can manipulate the generated content and the testing to achieve better results.
Who’s Winning the Race?
There are many AI generators and detectors, free and subscription-based, and all produce a quick answer. Some even highlight the text in question.
Gauging a detection tool's effectiveness takes practice, and none are infallible by a long shot. They can detect the obvious, like a check engine light on your dashboard. Still, you must dissect the content to make a determination, which may not be possible.
AI detectors are far behind AI generators in the tech race. Julia McCoy, writer and founder of Content Hacker and president of ContentAtScale, offered the following statement in a story earlier this year regarding AI detection.
“It’s a probability game. These AI tools look at the content and determine whether another AI wrote it based on the amount of text that follows specific patterns.”
Back To School for the Bots
Stanford Professor Jeff Hancock led a study of 4,600 participants this past spring, as referenced in the Content Hacker story. Their charge was to differentiate between human and AI-generated content on the OKCupid, AirBNB, and Guru.com social media platforms. How’d they fare? They were 50-52 percent accurate - the same “coin flip” accuracy rate cited by Soheil Feizi in my previous AI story.
“LLMs are trained on massive amounts of text. The more information we feed to them, the better and more human-like their outputs,” says Feizi’s peer, UMD Assistant Professor of Computer Science Furong Huang. “If we do the same with detectors - that is, provide them more samples to learn from - then the detectors will also grow more sophisticated. They’ll be better at spotting AI-generated text.”
We all have much to learn, and the AI content industry is an exciting new frontier. On that note, it only makes sense that I now refer to myself as an “organic writer” to avoid confusion.
###
In Part IV of my AI Series I’ll explore methods bots and humans use to beat the detectors.