Tell me how you prevent a log aggregation backend from collapsing when the primary application experiences a huge traffic spike

Question

Anonymous · Accepted Answer

Decouple the backend by introducing message queues (Kafka or AWS Kinesis)
between log writers and the backend. Have log writers buffer data on disk. Use
ingestion limits and adaptive sampling (for e.g. only write errors and warnings,
combined with writing some percentage of messages). Auto scale data ingestion
workers, but also ensure all writing is done async so the main application does
not experience a backlog. If buffer on writers starts being exhausted, ship off
data to offline storage for later processing into the log stream. Data being
added from offline storage can have its own sampling strategy.

Anonymous · Answer

Archive the logs periodically, we can use splunk or azure monitor or aws cloud
watch than building one on our own, add sampling , avoid unnecessary logs,
always queue to logging system so that more workers can be spinned up to read
from that queue , or we can azure eventhub/ aws kinesis to stream logs
effeciantly in case of spike

Tell me how you prevent a log aggregation backend from collapsing when the primary application experiences a huge traffic spike

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Unlock Full Access

Practice More Questions