A part of the problem is that the ingestion is not vector compressed, so they're... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

jiggawatts on Dec 7, 2024 | parent | context | favorite | on: Lossless Log Aggregation – Reduce Log Volume by 99...

A part of the problem is that the ingestion is not vector compressed, so they're charging you for the CPU overhead of this data rearrangement.

It would cut costs a lot if the source agents did this (pre)processing locally before sending it down the wire.

piterrro on Dec 7, 2024 [–]

We should distinct between compression in transit and at rest. Compressing a larger corpus should yield better results in comparison to smaller chunks because dictionaries can be reused (zstd for example)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact