Skip to content

marvin.utilities.testing

Utilities for running unit tests.

assert_equal

Asserts whether the LLM output meets the expected output.

This function uses an LLM to assess whether the provided output (llm_output) meets some expectation. It allows us to make semantic claims like "the output is a list of first names" to make assertions about stochastic LLM outputs.

Parameters:

Name Type Description Default
llm_output Any

The output from the LLM.

required
expected Any

The expected output.

required

Returns:

Name Type Description
bool bool

True if the LLM output meets the expectation, False otherwise.

Raises:

Type Description
AssertionError

If the LLM output does not meet the expectation.

assert_locations_equal

Helpful LLM assert for comparing two locations (e.g. New York, New York City)