Caching in LLMs - Quality Score Eviction Policy

References

Bibliography and citations for the Quality Score Eviction Policy research

References

  1. Compute will become the most precious thing in the world https://www.youtube.com/watch?v=r2UmOBrrRK8

  2. Bang, Fu. "GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings." Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) (2023): n. pag.

  3. Li, Jiaxing, Chi Xu, Feng Wang, Isaac M von Riedemann, Cong Zhang and Jiangchuan Liu. "SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models." 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS) (2024): 1-10.

Conclusion & Future Work

Summary of findings and grounded next steps for quality-score eviction

On this page

References