Define the real-world task that you would like your evaluation to capture.

The following recommendations are meant to help define real-world tasks:

1. Consider how humans do it : Starting from first principles , think about how the task you would like to evaluate is accomplished by humans, and write down the steps involved. This process will help clarify the task.


2. Use caution with existing datasets: Current evaluations