“People always talk about these parameter counts, which is the size of the model. It’s basically how many different numbers am I multiplying, adding, doing these mathematical operations. When you input something to the language model, you tokenize it. Tokens are basically four letters.”
– Dylan Patel
Semafor talks with Dylan Patel, who deconstructed how GPT-4 works. Patel and his colleagues at SemiAnalysis, a consulting firm, described their findings in an online paper.
Their findings showed how the large language model uses several much smaller models to solve pieces of the text GPT-4 is generating at any time.
They call these “experts.” It’s like breaking a big project into a collection of specialty areas and delegating tasks to each of them. SemiAnalyis says GPT-4 uses 16 experts. Using this approach saves time, speeding up the results and reducing the cost of computing.
Revealing the mysteries of ChatGPT | SEMAFOR | July 28, 2023 | by Reed Albergotti