A good experiment would be to also try giving it access to latency traces so it can identify issues? Wrt coding agents, giving access to observability tools often improve coding/debugging ability for me

Try https://docstrange.nanonets.com/ once, 10k docs you can use for free. Strong table performance. Do give feedback if any. Powered by bigger model compared to our open source one which is quiet popular on HF.

If with LLM's you can deanonymize at scale, on a personal level, you should also be able to figure out what posts are leading to this deanonymization and remove them or modify them.

Instead of markdown -> LLM to get JSON, you can just train a slightly bigger model which you can constrain decode to give JSON rightaway. https://huggingface.co/nanonets/Nanonets-OCR2-3B

We recently published a cookbook for constrained decoding here: https://nanonets.com/cookbooks/structured-llm-outputs/

Nice, it would be good idea to develop CFG for this as well so can embed it into all these constrained decoding libraries

One of the authors here, will checkout the diagram link.

Every commercial model provider is adding structured outputs so will keep updating the guide.

https://docstrange.nanonets.com/ as well, wrapper on top of 7B version of https://huggingface.co/nanonets/Nanonets-OCR2-3B

Then you can just download finetuned version of same multi-modal foundation model that's trained on documents?

Top 3 models on huggingface are all OCR models. Most automation projects involve documents where you need a model finetuned to understand all elements inside documents and provide grounding and confidence scores etc which is why these subset of models are gaining popularity