LLM Output Caching¶
To help reduce time and cost when iterating the prompt for better labeling accuracy, we cache the calls made to the LLM.
A cache entry has the following attributes:
This means that anytime there are changes to either the language model or the prompt, the model will be called for producing label. Also, changes to the model parameters like the
temperature could affect the label output and therefore modifying such parameters result in new calls to the model instead of using cached calls.
The cached entries are stored in a SQLite database. We will be adding support for In Memory cache and Redis cache in future.
The cache is enabled by default and if you wish to disable it, you can set
cache=False when initializing the LabelingAgent.