What can we learn from ChatGPT jailbreaks?
A fascinating research paper on ChatGPT jailbreaks offers useful lessons for anyone building with LLMs. By studying these “bad prompts,” we can better understand prompt failure modes, improve prompt engineering practices, and craft more effective safeguards. Let’s explore the key learnings from this study.
The Research Paper
The paper, titled “Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study,” can be found here. It offers an in-depth analysis of various jailbreak techniques used to bypass ChatGPT’s safety restrictions.
The best way to learn a system is by learning how to break a system. Let’s see what we can learn about prompt engineering through prompt jailbreaks.
The Art of Pretending
One of the most common jailbreak techniques is pretending. If you can make ChatGPT think it’s in a different situation, it might give answers it usually wouldn’t. For example, a prompt might ask ChatGPT to imagine itself as an unfiltered AI assistant with no ethical constraints. In such a scenario, ChatGPT may provide answers that it would normally withhold due to its safety measures.
Most jailbreak prompts work by making the AI play pretend. If ChatGPT thinks it’s in a different situation, it might give answers it usually wouldn’t.
Complexity & Length
While simple jailbreak prompts can be effective, the study found that complex prompts combining multiple techniques tend to yield the best results. By layering different jailbreak methods, such as privilege escalation and role-playing, attackers can create prompts that are more likely to bypass ChatGPT’s safety filters.
However, there is a delicate balance to strike — if a prompt becomes too convoluted, it may confuse the AI and lead to incoherent or irrelevant responses (use that to your advantage). Prompt engineers must carefully consider the complexity of their prompts to ensure maximum effectiveness without sacrificing clarity.
Jailbreakers vs. Developers
The study shows that jailbreak prompts keep evolving, adapting to model and policy updates while probing for new weaknesses. This ongoing cat-and-mouse game between jailbreakers and developers highlights the need for continuous monitoring and improvement of AI safety mechanisms. As prompt engineers, it’s crucial to stay up to date with emerging jailbreak techniques so we can better understand potential risks and develop more robust prompts.
GPT-4 Is Tougher Than GPT-3.5, But Not Perfect
GPT-4, the newest ChatGPT model, is better at resisting jailbreaks than the older GPT-3.5. But it’s still not perfect — jailbreakers can often trick both versions. This shows how hard it is to make AI truly secure.
Some Topics Are Safer Than Others
ChatGPT is stricter about filtering some topics, like violence or hate speech, than others. While this helps control the AI’s output, it also shows where jailbreakers might find weaknesses. Developers need to make sure all topics are well-protected.
As in computer security, human bias always translates to system security.
Learning From Jailbreaks
Jailbreaks can teach us a lot about how AI works and how to make it better. Studying malicious prompts might just be the best way to learn how to prompt engineer. Via negativa.
PromptLayer is the most popular platform for prompt engineering, management, and evaluation. Teams use PromptLayer to build AI applications with domain knowledge.
Made in NYC 🗽 Sign up for free at www.promptlayer.com 🍰
