Creating test suites - bench documentation

• Generate and execute more than 50 distinct types of tests only with 1 line of code


• Test all aspects of model quality: robustness, bias, representation, fairness and accuracy.​


• Automatically augment training data based on test results (for select models)​


• Sup

The goal of a benchmark usability test is to describe how usable an application is relative to a set of benchmark goals.

You want to ensure that you run your tests for at least a hundred conversions on each variation, but the exact number might be unique to your own website.

Test systems at production scale:

Objective #3 (20%): Execute at least 15 unique tests during the fiscal year. Document what was learned in each test, and share results with the executive team.

The size of your test will be constrained by the traffic to your landing page and its data rate (the number of conversion actions per unit time). Changing the granularity of your tests allows you to include all or most of your important ideas while still fitting into a reasonable test size.

• Internal refactoring


• Config-based task creation and configuration


• Easier import and sharing of externally-defined task config YAMLs


• Support for Jinja2 prompt design, easy modification of prompts + prompt imports from Promptsource


• More advanced configuration opt