Tokens
Tokens are nothing but a group of characters.
Last updated
Tokens are nothing but a group of characters.
Last updated
1 token is equal to 4 characters.
For example, the following sentence “The quick brown fox jumps over the lazy dog” has 11 tokens.
Tokens are used by both the input (what is being sent to the AI model) and output (what is being received by the AI model).
For example, when you train an AI model with the website here’s how many tokens would be consumed:
Training – Total number of characters extracted from the website (Counted once, as you only train once)
Input tokens – All the tokens used to send the question to OpenAI are counted here. They include:
Chunks – When a question is asked to the AI model, the system would fetch a chunk that closely matches the question.
Input question – Question asked by the user
Prompt – System prompt and the instructions defined
Output tokens – Size of the response received from OpenAI
The token calculation would vary depending on the size of your AI model, how brief are your prompts/instructions, and how big are the responses from the AI model.