Token Estimation
complete
Joshua Hsu
Token usage was a lot higher than expected.
Either reduce token usage to the point where it doesn't matter for me to make these considerations/tradeoffs or indicate clearly which actions use up the tokens and estimate how much it'll use up.
Log In
Martin Adams
Joshua, one of the reasons our AI has been using so many tokens is because we’ve been using historical tasks to better estimate task metadata: when you enter “Water the garden” as a new task, and later assign a “Weekend” schedule to it, the next time we want the AI to already know that you’re most likely to water the garden on weekends. Naturally.
In the past we clumsily fed all of your most recent historical tasks to the AI, without differentiating for similarity. Obviously that’s not only a recipe for a lack of increased accuracy, but also for increased token usage.
Now, however, we vectorize each task and compare your new task to similar vectors in our vector database. As a result, we only feed those related tasks to the AI so that the AI can optimize its task metadata guesses based on those similar (historical) tasks. This uses significantly fewer tokens 🙂
We should be launching this improvement by the end of this week. I’ll keep you posted here.
—
ps: for a non-technical description of vectors, see https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/
Martin Adams
complete
Joshua, this has now been pushed to production, FYI.
Martin Adams
in progress
Martin Adams
Joshua, one of the reasons our AI has been using so many tokens is because we’ve been using historical tasks to better estimate task metadata: when you enter “Water the garden” as a new task, and later assign a “Weekend” schedule to it, the next time we want the AI to already know that you’re most likely to water the garden on weekends. Naturally.
In the past we clumsily fed all of your most recent historical tasks to the AI, without differentiating for similarity. Obviously that’s not only a recipe for a lack of increased accuracy, but also for increased token usage.
Now, however, we vectorize each task and compare your new task to similar vectors in our vector database. As a result, we only feed those related tasks to the AI so that the AI can optimize its task metadata guesses based on those similar (historical) tasks. This uses significantly fewer tokens 🙂
We should be launching this improvement by the end of this week. I’ll keep you posted here.
—
ps: for a non-technical description of vectors, see https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/
Martin Adams
under review
Thank you, Joshua. Indeed, this is an issue. Let me get back to you on this.