marvin.utilities.testing

Utilities for running unit tests.

assert_equal

Asserts whether the LLM output meets the expected output.

This function uses an LLM to assess whether the provided output (llm_output) meets some expectation. It allows us to make semantic claims like "the output is a list of first names" to make assertions about stochastic LLM outputs.

Parameters:

Name	Type	Description	Default
llm_output	Any	The output from the LLM.	required
expected	Any	The expected output.	required

Returns:

Name	Type	Description
bool	bool	True if the LLM output meets the expectation, False otherwise.

Raises:

Type	Description
AssertionError	If the LLM output does not meet the expectation.

assert_locations_equal

Helpful LLM assert for comparing two locations (e.g. New York, New York City)