Tokens

Tokens are nothing but a group of characters.

1 token is equal to 4 characters.

For example, the following sentence “The quick brown fox jumps over the lazy dog” has 11 tokens.

How are tokens calculated?

Tokens are used by both the input (what is being sent to the AI model) and output (what is being received by the AI model).

For example, when you train an AI model with the website https://www.swiftsell.biz/ here’s how many tokens would be consumed:

• Training – Total number of characters extracted from the website (Counted once, as you only train once)

• Input tokens – All the tokens used to send the question to OpenAI are counted here. They include:

  • Chunks – When a question is asked to the AI model, the system would fetch a chunk that closely matches the question.

  • Input question – Question asked by the user

  • Prompt – System prompt and the instructions defined

• Output tokens – Size of the response received from OpenAI

The token calculation would vary depending on the size of your AI model, how brief are your prompts/instructions, and how big are the responses from the AI model.