top | item 47034131 (no title) 0-_-0 | 13 days ago The cache gets read at every token generated, not at every turn on the conversation. discuss order hn newest mzl|13 days ago Depends on which cache you mean. The KV Cache gets read on every token generated, but the prompt cache (which is what incurs the cache read cost) is read on conversation starts. 0-_-0|13 days ago What's in the prompt cache? load replies (2)
mzl|13 days ago Depends on which cache you mean. The KV Cache gets read on every token generated, but the prompt cache (which is what incurs the cache read cost) is read on conversation starts. 0-_-0|13 days ago What's in the prompt cache? load replies (2)
mzl|13 days ago
0-_-0|13 days ago