> The cyclone prediction thing is very interesting to me in particular (not quite sure how you go from the ML matrices to "here's a path the cyclone might take")
In a high-level view, it's the result of specialized decoding heads.
Traditionally one would take gridded forecast outputs, then process those with comprehensible actions like "find all local pressure minima in the ocean, then filter to ones which correspond to warm cores, etc." to infer (diagnose) the presence of a cyclone.
One problem with this is that gridded forecasts suffer from known biases and tradeoffs. For example, a forecast on a ~25km grid is just on the edge of being able to represent the eye of a hurricane (50km scales), and it certainly can't accurately represent the sharp transition of wind in the eyewall. That means that the forecast winds are almost certainly a smoothed (and therefore less intense) version of what observers would see.
It's kind of like a post-processing or bias correction (see for example https://www.ecmwf.int/en/about/media-centre/aifs-blog/2026/a..., which applies in physical space), but by having access to the model latent space and by being included in model training it is (probably!) higher-quality than a pure, after-the-fact approach.
> It’s crazy we’ll never have forecasts as good as dark sky again.
'Nowcasting' is an area of active research, both with machine learning and with physics-informed or visual flow approaches.
Part of the problem from the machine learning side is that these are _huge_ problems. NVidia's StormCast (https://research.nvidia.com/publication/2024-08_kilometer-sc...) works globally at kilometer scales, and you can imagine how big those grids are. Even with patch training, you're dealing with very large datasets.
At the same time, this is not exactly a high-profile area of research. National weather centres focus on actionable medium-range weather predictions, and meteorologists can look at radar themselves and perform mark-one-eyeball predictions for very short-range watches and warnings. Some private-sector actors will pay for short range predictions, but they're often looking for something hyperlocalized (e.g. weather at this particular construction site, for crane safety) or specialized (near-real-time cloud and wind predictions for renewable energy).
Most public-accessible weather predictions are downstream of either a public-sector effort (which doesn't internalize benefits, leading to under-resourcing) or a byproduct of another private-sector offering.
A barometer will give you surface pressure, but that's a field that tends to vary relatively slowly over the surface of the Earth. The calibrated weather stations that exist at every airstrip do a reasonable job of providing these conditions over land, and the residual of "pressure from phones" probably won't help all that much.
The data that would be most valuable to initial conditions is upper-atmosphere winds -- this is the kind of data given by weather balloons. In clear air there's no great way to measure this from either the ground or from space.
One important supplemental data source here are aviation reports, from planes flying at altitude and particularly trans-oceanic routes. When air traffic was largely curtailed during the early phase of the Covid pandemic, weather forecasting suffered a bit for the lack of data (see eg https://www.ecmwf.int/en/about/media-centre/news/2020/drop-a...).
That line of reasoning has no end. If you use Antigravity on anything other than a Google Chromebook or Pixel, the hardware is a 'product not provided by them'. Is that a TOS violation?
If your maximum addressable market is “the whole economy,” as seen in SpaceX filings, then a city-sized call centre (distributed, of course) really is ‘t that much of an ask.
> But it’s just as likely to make an output better.
No, for any particular output token the model's true logits are definitionally the 'best' that the model can achieve.
This is inherently probabilistic. The model's top-1 guess is not guaranteed to be optimal, but it should be so a proportionate fraction of the time. Same with the top-2, top-3, etc.
Watermarking necessarily alters the output distribution away from the model-set distribution, and that alteration is inherently 'worse' in expectation.
You can liken this to a weather forecast. If there's a 25% chance of rain, the forecast should say so (or a 'sampled' deterministic forecast should predict rain 25% of the time). If the forecast is 'watermarked' and predicts rain 27% of the time under identical circumstances, it's a worse forecast.
That being said, this is a case of hiding a message in a noisy channel. Watermarking only needs to communicate one bit ('yes watermark'), so the effects can be arbitrarily small provided one is willing to tolerate an increase to the text size needed for reliable detection.
I think you can also just use the random number generator (seeded with a secret key) as the watermark. Then the probability distribution is exactly the same.
> Hyperscalers don't run computing at some multiple more efficient than on prem.
I'd disagree here. I see two avenues for an efficiency multiple, albeit a single-digit multiple:
* Client aggregation allows a hyperscaler to average out demand spikes from uncorrelated clients, reducing the peak:average demand ratio and allowing better budgeting of compute.
* Dynamic batching allows typical requests to run in batches of more-than-1 and/or overlap, offering better internal compute utilization ratios (e.g. interleaving output and input streams). The small limit of on-device LLMs will run with batch sizes of one with strong memory bandwidth bottlenecks.
For an example of these factors in action, see the API cost differential between batch, standard, and 'fast' processing. OpenAI prices these tiers at a 1:2:4 ratio.
A hyperscaler can pack compute and that is it. It doesn't get you some multiple higher in efficiency. And it is a net negative when oversubscribed, which due to MBAs ruining all the clouds is how they operate. Customers pay more for less when using the cloud.
Go get a job a hyperscaler, they want to smoke what you are smoking.
I am not talking about diurnal cloud workloads, I am talking about the native efficiencies of hyperscalers vs on-prem. They have no magic and they all think they are going to make up their business overheads in exorbitant saas pricing.
Yes, they can do all of these things. But then the margins are eaten up in the process, hyperscalers could be half the price of on-prem, but they are more. They have no magic, or the magic they do have they are wasting (which is good for the people). You can beat a hyperscaler on price at a single rack.
> As things settle down and commoditize, the value of switching on a dime diminishes as people lock into their favorite models
I can imagine just the opposite outcome from the same scenario: as people settle into their favorite but commoditized models, competition for marginal inference cost will take over. A company like OpenRouter that promises the cheapest tokens by the minute becomes essential on the low-cost margin.
I think that OpenRouter and equivalents get pushed out of the market only if the froth calms down (as you posit) and winning models stay proprietary, perhaps with their own unique API surfaces.
> A company like OpenRouter that promises the cheapest tokens by the minute becomes essential on the low-cost margin.
Okay, I can see that, but if their value is just cost optimization, their ability to mark up the tokens becomes increasingly difficult as well. Or, people will build a router themselves to avoid paying the markup, possibly with reduced features, but someone will open source it. Heck, Claude or GPT can probably one-shot it these days. Either way, I think the whole OpenRouter model is going to struggle unless the market stays frothy.
First, cost optimization directly contributes to how much markup they can charge. Say on average they deliver savings of 20%, that is excess value OR/Stripe can mostly capture.
Also, you don't need to mark up tokens much if you're a commodity volume business. Think of Costco and their margins & membership fees. Not everything has to be high margin, not everything has to be a SaaS subscription.
Well, it depends on how they deliver that savings. If it’s through something unique and defensible, something that only they (with some moat) can do, then yes, they can argue for a fraction of the savings. But if prices just go lower because things are commodities, then they are going to be viewed as middleman markup on top of already low prices. In that case, their value add is considerably lower.
You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs.
All other things being equal, increasing the speed of some part of the development process will increase the overall pace of development. However, By Amdahl's law that increase will be sublinear, and that is why we should take "pull requests" as an imperfect metric.
We also don't get to pick the form that 'better technology products' take. While we'd probably like to keep cost(/effort) and complexity constant and increase robustness and performance, the market equilibrium might be 'worse is better' and reward whiz-bang features and lower effort.
Also, large companies are still experimenting on how to integrate LLMs into their workflows. Due to the fast cadence of releases, people forget that LLMs became robust (regardless of the capability level/parameter count/data size) enough to use semi-reliably in company-specific ways only 1 year ago. And bigger the org, the slower the process. I don't expect it to settle and get productive used across the majority of very large companies for another year atleast.
High prices are mostly a result of DC capacity. As more and more DCs get built out, prices will drop. At a unit level the inference business is extremely sound regardless.
> You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs.
Right, but we didn't invest a trillion dollars of capital into any of those things in the span of a couple years, thus forcing them to capture value and show a return on such a massive investment.
> Is the ROI there to pay for the trillions in commitments that have been bet on that ROI? That looks like a clear no at this point.
That's only a potential crisis for those who have made concrete investments.
On the use side, the 'cost' of AI spans more than two orders of magnitude. Looking at recent models (<6mo) with reasonable performance (intelligence index >= 45) on OpenRouter, the output cost ranges from $50/MTok (Fable) to $0.153/MTok (DeepSeek Flash 0731).
From the perspective of a user of LLM/agent assistance, there's very likely a range where the benefits outweigh the costs.
If the ROI for the model developers isn't there, then that just impairs the future trajectory of the field. Current models are just bits that aren't going anywhere, and as long as they can be served (in inference) above their marginal cost they will continue to be so-delivered.
In a high-level view, it's the result of specialized decoding heads.
Traditionally one would take gridded forecast outputs, then process those with comprehensible actions like "find all local pressure minima in the ocean, then filter to ones which correspond to warm cores, etc." to infer (diagnose) the presence of a cyclone.
One problem with this is that gridded forecasts suffer from known biases and tradeoffs. For example, a forecast on a ~25km grid is just on the edge of being able to represent the eye of a hurricane (50km scales), and it certainly can't accurately represent the sharp transition of wind in the eyewall. That means that the forecast winds are almost certainly a smoothed (and therefore less intense) version of what observers would see.
The WN2 approach (paper: https://www.nature.com/articles/s41586-026-10953-2) adds a direct readout head to the model: given latent-space access to the full forecast, it tries to predict the bona-fide cyclone observations (https://www.ncei.noaa.gov/products/international-best-track-...).
It's kind of like a post-processing or bias correction (see for example https://www.ecmwf.int/en/about/media-centre/aifs-blog/2026/a..., which applies in physical space), but by having access to the model latent space and by being included in model training it is (probably!) higher-quality than a pure, after-the-fact approach.
reply