Do Prompt-Based Models Really Understand the Meaning of their Prompts?

How does text-based prompting supplant or augment GUIs?

Some of the language models’ ability to infer natural language (ChatGPT) is unwieldy so there has been a lot of scrum about how text-based prompts could completely replace graphical user interfaces. My sense is that this won’t happen overnight as GUIs give way to higher fidelity or correctnes

... See more

To do this, we employ a technique known as AI-assisted evaluation, alongside traditional metrics for measuring performance. This helps us pick the prompts that lead to better quality outputs, making the end product more appealing to users. AI-assisted evaluation uses best-in-class LLMs (like GPT-4) to automatically critique how well the AI's output... See more

Important Aspects of Instruction Data

Longpre et al. (2023) and Iyer et al. (2022) ablate several important aspects of instruction data, which we highlight in the following.

Mixing few-shot settings. Training with mixed zero-shot and few-shot prompts significantly improves performance in both settings.

Task diversity. Large models benefit from conti... See more