The issue with the standard watermark techniques is that they require an output of at least a few hundred tokens to reliably imprint the watermark. This technique would apply to much shorter outputs.
A crude way:
To watermark:
First establish a keyed DRBG.
For every nth token prediction:
read a bit from the DRBG for every possible token to label them red/black.
before selecting the next token, set the logit for black tokens to -Inf, this ensures a red token will be selected.
To detect:
Establish the same DRBG.
Tokenize, for each nth token, determine the red set of tokens in that position.
If you only see red tokens in lots of positions, then you can be confident the content is watermarked with your key.
This would probably take a bit of fiddling to work well, but would be pretty much undetectable. Conceptually it's forcing the LLM to use a "flagged" synonym at key positions. A more sophisticated version of a shiboleth.
In practice you might chose to instead watermark all tokens, less heavy handedly (nudge logits, rather than override), and use highly robust error correcting codes.
shawnz|1 year ago
andai|1 year ago
antognini|1 year ago
pava0|1 year ago
tyho|1 year ago
To detect: Establish the same DRBG. Tokenize, for each nth token, determine the red set of tokens in that position. If you only see red tokens in lots of positions, then you can be confident the content is watermarked with your key.
This would probably take a bit of fiddling to work well, but would be pretty much undetectable. Conceptually it's forcing the LLM to use a "flagged" synonym at key positions. A more sophisticated version of a shiboleth.
In practice you might chose to instead watermark all tokens, less heavy handedly (nudge logits, rather than override), and use highly robust error correcting codes.