DetectGPT: Zero-Shot Machine-Generated Text Detection

[eclipsetheworld]

This approach seems to require knowledge of which LLM was used to generate the given text. I wonder if e.g. model fine-tuning - as already provided by Open AI [0] - could evade this detection approach.

[0] https://beta.openai.com/docs/guides/fine-tuning

[mdorazio]

I might not be understanding what AUROC really means, but how is this performing in terms of false positives/negatives in comparison to human readers?

[O__________O]

Here’s a post on understanding AUC (Area Under The Curve) and ROC (Receiver Operating Characteristics):

https://towardsdatascience.com/understanding-auc-roc-curve-6...

[gault8121]

How does this compare to GPT2 output detector?

I created AIwritingcheck.org to provide teachers with a user friendly interface for this model.

[SachinDSI]

Consider trying out GPTKit https://gptkit.ai it has higher accuracy than GPTZero and uses 6 different methods

[andai]

Wouldn't GPT-2 be detectable by virtue of being bad? Or is it bad in a way that is more or less indistinguishable from student writing?

It's been a while since I used it but I very rarely got plausible output from it.

[eh9]

Stop doing this. Have teachers figure out how to move forward with this and students, but the false-positives are not worth it

[O__________O]

Research paper:

https://arxiv.org/abs/2301.11305

Additional explanation:

https://twitter.com/_eric_mitchell_/status/16188203614199152...

[m00viin_pics]

Awesome thread by one of the authors of paper, concise and insightful, thanks for sharing !

Research was aiming to bring a tool/approach on distinguishing text from LLM and other sources but in the end of the day it will only benefit those with non-open-source LLMs adjust to such technique and "fool" better everyone else (cause we need log-probs out of a model on each of the sample text).

It seems kinda ironic for me, maybe i missed some crucial point here.