Haven't gone through full PDF as its very detailed, few things have resonated with me so far.
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
Anyone that's into auto racing as a hobby will tell you the engine is the last thing you should modify unless your whole goal is to get into the guts of engine mechanics.
But if you just want to improve lap times it's way less of a priority than brakes, tires, suspension setup, cooling to keep everything happy, etc.
I'm not a car guy but would it be fair to see that your maximum possible performance is bounded by the engine, but the average performance gains are from tuning the rest of the vehicle?
Kind of like if your RAM has to constantly page, you would see much more improvement from upgrading your RAM than overclocking your CPU?
Approach itself looked impractical to me for any non-trivial system, like domain centric system of records systems which can have 100s if not 1000s of laws. Though it can be tried as a side parallel thread to see if system is still compliant and following right first principals after a few years from its inception.
I would rather wait to see how it gets adopted, if at all. Anyone aware of early reviews of the adopters of bend 2?
As a matter of fact, I've been poking at this from a slightly different direction: "axioms and invariants" within home automation scenes.
Invariants were like "if outside temp < 40 or inside temp < 65: heater.minTemp( 65 )"
Axioms were like: "if {we're home} and it's {not a holiday} the house should be {comfortable temperature}".prompt
...and then that would get decomposed and translated into interlocking code for the scene(s). I'll have to look at this language a little more closely with those kinds of constraints in mind!
You're kindof translating `*.prompt` to either prolog (yucky!), lisp, lua, or javascript (for inspectability/debuggability), but this whole bend thing might be an exact fit for the problem space! Limited set of objects and states, bounded set of "invariants" (laws), and layering on top the general state modification activities (either "evaluated every 5 minutes and reconciled" or "set the scene xyz...").
Hmm, in your example, temperature is fully measurable but there might be a slight challenge in measuring if "If we are at home". Since for this you need to define more variants with measurements from motion sensors, pressure or weight sensors on beds or sofas, if a device has got connected to wifi, etc.
By the time your first round of beta testing is over, you may have quite a handful of such axioms and variants, which have been humanly validated!
It was my "vibe coding playground" experiment. I haven't gotten anything solid fleshed out, but conceptually I wanted the "safety layer" to only permit fully expanded, concrete references.
Then break plain language requests into tool-calling-ish shared functions [atHome(), isHoliday(), comfyTemp(), ...] and basically throw "linker errors" if a concept didn't have a definition, eg: "ERROR: concept 'comfortable temperature' not defined..."
...and yes: be able to highlight overlaps or conflicting instructions at the semantic layer... those being less important than conflicts at the safety/invariant layer.
The idea was to have a bunch of basically "is_comfy_temp.prompt" => "is_comfy_temp.lisp" and be able to right click on any of the prompts and "show source" to understand, debug, simulate, validate, etc.
As you get real trials you end up with a foundational "StdLib" of at least necessary concepts although yours and mine might contain different data/preferences. And being able to edit "comfyTemp()" in one place as opposed to spread out in a bunch of different home automations (eg: isSummer vs isWinter else also: ifHumidity && upcomingWeather || realtimeElectricRates, etc...)
AI can definitely be a cover for someone to do something nefarious and blame it on the AI.
All that AI needs is access to all the spicy tools like nukes, unsupervised medical diagnosis/treatment, unchecked military decision making, etc. to its React loop so that humans can be completely hands-free and enjoy the fruits of AI's labor.
Maybe we need to mandate parliament (or anything equivalent in other countries) debate before giving any critical tool access to LLM? And then a UN general assembly vote to conclude the process, maybe.
Looks like this will be a hard sell for many orgs who are already struggling with explosion of artifacts on JIRA, sharepoint, github, etc. Also, most of them have somewhat settled on some ways (in past 6-8 months) to produce AI first specs and work with them.
Also, this looks like something which leadership level folks need to adopt first and then somehow it needs to trickle down to PI planning and sprint planning. Would like to hear someone's experience on how this has got adopted in their org.
I use it at a much smaller scale. I think that's also the right place, because its a CLI tool + skills that generate md files. Not something you'd give to a business person. I start from a single ticket (sometimes just a handful of changes, but I want the spec as docs) to changes that you'd normally split over several tickets.
Basically the flow proposal -> design -> specs -> tasks gives you and AI a method to build context on what you want to achieve. In a way you're just creating a plan/big prompt that is structured in such a way that they start stacking on each other.
The power is that you do a lot of upfront thinking. In my team we then share it with a colleague who will review it through a PR. After that implementation is usually hands off. At the end there is a skill to verify the change against specs. I do still review the code myself too.
I guess if you work in a task oriented environment this will not work as well, as you'd lack/don't care about the business context. I'd like to think that most software development does not happen like this, but is done by engineers who actually understand why something is needed and take that into account when designing and building the solution.
>> The power is that you do a lot of upfront thinking
This is the best part of this spec, but we have found from our experience that though upfront thinking changes has a lot of merits and adds clarity and alignment upfront, but it changes bit by bit in every meeting and before you know your specs are not aligned with general consensus in the team. If your team is large enough, then it gets very difficult to own the task of constructing alignment between your principal-artifacts and your evolved under-current of understanding.
I would still say that if you are working on a platform and if your engg team size if anything more than 25-30, then this spec must be adopted from top-down and not bottoms up. Bottom level engineers usually don't have the level of consistent exposures (as and when they socialize and evangelize their platform) which top level engineers have.
OpenSpec maintainer here. A lot of our adoption has mainly been bottoms-up, it's usually driven by engineers. That being said it definitely helps if everyone on the team uses it together. Especially when shifting left and doing a lot more "spec review".
Thanks for taking time to respond here. Would love to know from your experience the scale of function-points, team size, client-requirement variance, etc. different teams would have worked with and maintained over a period of time via this open-spec.
Please note that I can already see that github repo has 68k+ stars. So popularity is not in question, just the viability and consistency of adoption across different scenarios.
Fully agree. With LLM being able to solve every problem, getting deep into a problem all by yourself becomes a passion side project. Now might be a real test of how much you love programming.
Your enterprise wants the work done, done fast and reliably. Your productivity goals have increased, just like invention of motors would increased goals of carriers who were earlier doing their job via more manual efforts like pedaling. But still people love cycling, but they largely "don't have to" rely on it to do their job.
Similarly, now you simply don't have a dependency to love programming to increase your productivity.
Superset comes with a built-in MCP server now https://superset.apache.org/admin-docs/configuration/mcp-ser..., which you need to expose as another container. Any agent with proper authentication (JWT mostly) will be able to create and manage dashboards via natural language.
Agree! 'Problems' are getting solved and this needs to be celebrated. Wondering how this will discourage mathematicians at all, since now they have another tool to accelerate their research. Nothing is stopping them from using 'new technologies' or sticking a gun to their head to use the 'new technologies' either.
If you consider this event in isolation it is cause for celebration. But the controversy around this isn't so much about how the proof was obtained but what this means for the practice of mathematics going forward. It seems we can probably expect more and more results of this nature being dumped into the community. It's happened before that one person, Bill Thurston, was so successful in his field, proving theorem after theorem, that he inadvertently killed his field. People hesitated to enter his field, knowing that they could be scooped at any moment. It took years before his field recovered - and I think his famous essay was written in response to this.
Yes, but a big part of the problem right now is that two big labs have monopoly on the resources and they for sure are not working for the benefit of mankind.
World War III was the last of Earth's three world wars, lasting from approximately 2026 to 2053. The conflict involved nuclear cataclysm as well as genocide and eco-terrorism. The post-atomic horror in the aftermath persisted as late as 2079.
The war was preceded by the Eugenics Wars and the Second Civil War, all of which were sometimes regarded as parts of a single escalating conflict. It resulted in the deaths of some 30% of the Human population, at least six hundred million people, and the extinction of six hundred thousand species of animals and plants. By the end, most of the major cities had been destroyed and there were few governments left.
> some 30% of the Human population, at least six hundred million people
The math nerd in me has to point out that this means there was only 2 billion humans for 30% to be 600 million (though it does say at least). Currently we have 8 billion humans on this planet or so. There must have been a culling before WWIII in their universe that they failed to mention.
In December 2024 o3 scored 87.5% on ARC-AGI-1 and cost $4560 per task.
DeepSeek V4 Flash 0731 scores 89% and costs $0.02 per task.
If we apply the same factor to the guesstimated API price of $20M for this problem, we arrive at $57.
Real cost is a fraction of the API price. Although the internal model might have a higher API price than the ~$19.5M I estimated based on Astra's pricing.
Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual. You need to say exactly what is in scope and what's not, even down to a buttons, events and layouts.
You can groom the epic with the help of AI, but final review must be done by someone who can take ownership of the specs and hence is responsible if something has fallen through the cracks. AI's response will be limited by the output tokens of that specific agent, and there will no repercussions for AI even if it accepts its mistakes.
Yeah, I think as a general direction we need to allow a useful collaboration with the agents on the issue board.
This is btw why Epiq was developed, to keep the board as code, git-backed, distributed (via an event log mechanism), and with the ability to replay the board, to see what agents actually did:
I hope @jolaflow can chime in here eventually, but from a brief look, my impression, besides the fact that Epiq is based on git as opposed to beads where it is optional, is that Epiq seems to be much more optimized for interactive collaboration between the user and the agents.
The graph visualization in beads surely is a neat thing for showing things, but the replay feature in Epiq should provide a similar understanding of what happened.
But again, it seems to me Epiq is the tool that better allow the user to jump right in and collaborate with the agents on the board.
(Again, this is from a brief look, so I could be missing things).
Sorry, I might not have been clear, but the tool and blog is by @jolaflow (my brother) and I'm so far mostly involved as a reviewer and early adopter, but glad you like it. I might be biased, but I think the tool provides something really useful for agentic coding and hoping it can grow into a thriving open source project.
> Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual.
I think this lesson is getting partially outdated. Yes, you need to be specific about what you want, and with earlier LLMs, you need to had both domain knowledge and some general software development experience to front-load various big and small choices about design, architecture and operational reality - what libraries to use, how system components communicate, how you handle auth and store secrets, etc. Otherwise the LLM would pull some random mix of ideas from its latent space, and give you something that's broken in really stupid ways.
Nowadays, it doesn't feel like that to me, not anymore. I still need some understanding to verify the proposals, but I found the last ~6 months of SOTA models to make good choices. Like, just yesterday I asked Claude to design me some simple service, and focused on explaining it the domain parts (nature of systems I want to integrate together, the purpose of that, and the user's priorities and use cases), and the design I got back had specific suggestions around security, authentication, deployment, failover, integration, behavioral impedance-matching between integrated systems, and more, that I all recognized as based on solid software engineering and ops practices, but deviating from it explicitly in every place where it would be wrong for this specific project. The model considered way more corner cases than I did, and I'm actually really impressed by it.
But then, I find greenfield development is easy with LLMs. Modifying existing systems, especially legacy ones, is where I need to babysit and micromanage models - because any misunderstanding or inaccuracy, which often comes from stale documentation or naming mistakes, tends to get amplified and confuse the agents. No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
The way I see it: models are highly biased to treat everything they read as "ground truth", all of equal importance. There's no nuanced notion that some information may be stale, that there's a temporal and causal order to sources, and that some information may just be wrong.
And this compounds when you let your LLM write code and documentation over time.
>> No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
True! hence the need for someone to review the final spec output and own it as their own output.
I have also found LLM to be better at debugging and solving 'a' specific problem, which I believe is due to output's surface area to be reviewed is lesser in comparison.
Probably because you were impressed by whatever it produced to fill in the gaps at the time. But if you expect the models to read your mind you're going to have a bad time.
> I honestly feel like basically nobody knows anything about these models, it's all just vibes
This, too. Since only providers know what they actually serve, what they change and what limits they impose.
There are some visible degradations though. E.g. Claude-ish.
As for a personal anecdote: around February I created a rather complex quiz web app for myself and friends with multiple question types, sync between screens, multiple media upload types, multiple scoring and timing types, MC inetrface etc. etc. etc. It took me a week or so in the evenings with rather vague prompts to make it.
Now Claude (and Codex) cannot reliably build a much simpler web app even with precise instructions while also maintaining the visual consistency.
But I will agree with you, it's a feeling, not a precise measurement.
I am guessing (guessing) that "great coding model" and "great model for the widest use" (agentic, long-running, unsupervised, knowledge-worker, computer operator) are competing, conflicting endgames.-
Add to that:
- Of course, "labs" (quotes) are incentivized to throw coders under the bus and aim for the biggest possible market.-
- "Sharp, focused, brief, elegant, precise" editing, as would benefit the coding use-case, is actually token-saving, ergo, undesirable.-
The only thing that can stop this, would be the quality and functionality of the codeslop generated by these models to became so low that it actually interferes with the (alledged) recursive self-improvement of models (ie. models start to perform worse/degrade).-
Until such a time, we serfs, will eat what's on our plate, pay for it, and continue to kneel before the machine god overlords. And be glad for it.-
Not sure about that. Writing detailed specs you deal just with the inherent domain complexity. Writing code you also get accidental complexity that has nothing to do with the business, whuch not only piles on top, but tend to interact vexingly with the detais of the domain.
Where you would have a point is if you'd say we have excellent tooling for wrangling code, but less tooling and tradition to write and manage specs.
I have realized that it's like giving a task to a brilliant coder who has just joined the org and is more excited and eager than usual. Hence the responsibility falls squarely on you to set scope constraints while ensuring only to-the-point features are developed.
But isn’t “grooming” the hardest part? To know the details means to know the codebases behind. So you do discovery, read code, ask people, check db tables, and then once you figure it out, you write a doc with the proposed solution. It gets reviewed and then you code it.
I definitely need AI help for the discovery part… so it always starts with a simple “I need to do X”
Genuinely curious, what is the benefit of doing this over just programming it yourself? It sounds to me like this is just an extra step that will grant you less control?
When programming a feature, you are simultaneously doing at least four things:
1. Implementing.
2. Building highly detailed mental models.
3. Learning and expanding your skillset.
4. Quality control and scope limiting.
And this process can be iterative and dynamic. Writing massive, super detailed specs that you then hand off to a undeterministic model feels like doing step one and three, while skipping two and four, which you then have to do after. What is the benefit? The speed up, in my opinion, comes if you skip step two and four, but then your product WILL be worse. Feels like I am going crazy?
Programming was never the bottleneck for software dev?
If you define what you want in typescript by updating existing typescript wordage, you get what you want exactly as specified, with minimal token cost and no ip theft.
If mathematician was already using OpenAI for research purpose and making progress due to inputs from OpenAI's responses, then I wouldn't put it beyond OpenAI's reach to generate different relevant prompts to make progress by itself. Afterall, Model can keep at it for whatever timeline and keep pursuing all possible combinations it can think try.
Harness definitely matters more from safety and reliability point of view, but saying that it matters more than Model itself is a slight exaggeration. In the sense that this 'headline' can lead people to believe that all models are equal and Harness can make up for lack of in-built features of a model.
Internal model must have native features like mixture of experts, memory features, etc. for the harness to use.
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
reply