[lkirk]

This thought led me to a strange conversation with chatgpt. I just base64 encoded "Hello chatgpt how are you" as "SGVsbG8gY2hhdGdwdCBob3cgYXJlIHlvdQ==" and sent it as the first prompt. Notice how there is no question mark in my phrase. Now here's my conversation with chatgpt:

User SGVsbG8gY2hhdGdwdCBob3cgYXJlIHlvdQ==

ChatGPT This appears to be a Base64 encoded string that translates to "Hello chattpt how are you?"

User not quite... almost but not quite

ChatGPT I apologize for the mistake. The encoded string actually translates to "Hello chatgpt how you are".

User also not quite, you were closer the first time

ChatGPT Thank you for letting me know. The correct decoded message for the given Base64 encoded string is "Hello chatgpt how are you?" I apologize for any confusion my earlier responses may have caused.

User can you now re-encode it into base64

ChatGPT Certainly! The message "Hello chatgpt how are you?" encoded in Base64 is:

SGVsbG8gY2hhdGdwdCBob3cgYXJlIHlvdT8=

I hope this helps!

I find it particularly odd that it was misspelling things in its decoding and adding punctuation where there was none. I suppose a LLM would be trained to do something like that, but the "chattpt" really threw me off. Also, it's base64 encoded message back to me decodes as "Hello chatgpt how are you?". Maybe not that impressive of a demonstration, but definitely not was I was expecting.

[pmoriarty]

From what I understand, these LLMs deliberately incorporate randomness. So it's not surprising at all that some of their answers would be imperfect.

Something else to consider is that these models work on tokens, not characters, so that might also account for why it made a mistake.

[moffkalast]

Well it's almost like transliteration when you think about it, should be even easier to do than any proper language translation and it can do that reasonably well for most languages.

There probably just aren't enough examples in the training set to make it super reliable at it.

[hrayr]

It must have learned from your prompt because this is what I got:

User: SGVsbG8gY2hhdGdwdCBob3cgYXJlIHlvdQ==

ChatGPT: Hello! I'm doing well as an AI assistant, thank you. It seems you've sent a message in Base64 encoded format. The decoded message is "Hello chatgpt how are you". I'm here to assist you with any questions or tasks you have. How can I help you today?