That sounds great if your messages in the logs are the complete state for that k... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

moatra on May 27, 2015 | parent | context | favorite | on: Using logs to build a solid data infrastructure

That sounds great if your messages in the logs are the complete state for that key, but I'm not seeing how to use that compaction system if the messages are change events.

Is there a system designed for snapshotting the aggregate and logging the delta?

kasey_junk on May 28, 2015 | [–]

A common pattern is to publish a "checkpoint" message. Not sure if the concept is built into Kafka or not.

erichmond on May 28, 2015 | [–]

It's easy to store messages in HDFS or S3 for long-term storage. It's also easy to replay messages from those mediums, if you need to re-ingest data later on.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact