Point not mentioned: it just doesn't work that well! It lies, it reverts to the mean on every topic, it wastes the reader's time. It's toxically positive, sycophantic. It's such a good mimic that it's insidiously hard to distinguish its bullshit or fake work from real work.
Always amazed how spoiled we are with modern hardware! The 8087 was $500 in today's dollars, and delivered around 50 kFLOPS of performance (0.00005 GFLOPS).
A cheap mobile phone CPU+GPU costs the manufacturer maybe $20, and typically does 50 GFLOPS on the CPU and 500+ on the GPU. So 10 million times the performance for 1/25th the cost.
Humbling too how "worthless" all the incredible ingenuity of the 8087 circuits and die designs now is, although I'm sure many of those innovations live on in modern chips.
Never mind cleaning up, you also have to understand the language just to judge and review the LLMs output. How else are you to separate good design and implementation from a bad one?
Not sure how excited I feel about visiting your website and having it auto-download a 8GB model with GPT-3.5 level hallucinations, and then probably crash because I only have 6GB of VRAM. My dad won't be able to use it, or anyone else without a bleeding edge device. On a powerful enough "neural engine" device the battery will be drained quickly, while the heatsink burns a hole in my lap.
The obvious optimization for the case presented would be to generate all the summaries on a server instead of in the client. Then the totally used compute would scale with the number of articles instead of number of users.
This is just emotional rhetoric. Pretty much any app in the last 20 years has depended on a server somewhere, or a cloud provider. Like an AI provider, they can go down, they can turn off if you don't pay your bill, etc.
And local inference requires fairly beefy hardware, that is FAR from ubiquitous across today's userbases. Local models are also still far dumber than what frontier labs can serve.
Weird that this is getting such a tidal wave of upvotes.
One thing I don't see often mentioned - OpenAI API's auto token caching approach results in MASSIVE cost savings on agent stuff. Anthropic's deliberate caching is a pain in comparison. Wish they'd just keep the KV cache hot for 60 seconds or so, so we don't have to pay the input costs over and over again, for every growing conversation turn.
reply