Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit kafka storage #133

Closed
slifty opened this issue May 11, 2021 · 0 comments · Fixed by #135
Closed

Limit kafka storage #133

slifty opened this issue May 11, 2021 · 0 comments · Fixed by #135

Comments

@slifty
Copy link
Member

slifty commented May 11, 2021 •
edited
Loading

Task

Description

Kafka defaults to 7 days of storage for anything added to a queue. This won't work for us because we're using lots of data.

Add a 30 second cache, we shouldn't need more than that.

Related Issues
slifty added a commit that referenced this issue May 13, 2021
@slifty
Set Kafka retention to 30 seconds
48e94a1 
TV Kitchen generates a lot of data, and much of it is heavy stuff like
video and audio. Kafka, by default, will store data on its queue for
7 days. This means TV Kitchen will quickly run out of space when
processing streams of video.

This sets a 30 second retention limit for TV Kitchen topics. Data is
processed as it is generated so this limit should have no impact on the
functionality of the countertop.

Issue #133
@slifty slifty mentioned this issue May 13, 2021
Merged
slifty added a commit that referenced this issue May 13, 2021
@slifty
Set Kafka retention to 30 seconds
8487598 
TV Kitchen generates a lot of data, and much of it is heavy stuff like
video and audio. Kafka, by default, will store data on its queue for
7 days. This means TV Kitchen will quickly run out of space when
processing streams of video.

This sets a 30 second retention limit for TV Kitchen topics. Data is
processed as it is generated so this limit should have no impact on the
functionality of the countertop.

Issue #133
slifty added a commit that referenced this issue May 13, 2021
@slifty
Set Kafka retention to 30 seconds
51a4469 
TV Kitchen generates a lot of data, and much of it is heavy stuff like
video and audio. Kafka, by default, will store data on its queue for
7 days. This means TV Kitchen will quickly run out of space when
processing streams of video.

This sets a 30 second retention limit for TV Kitchen topics. Data is
processed as it is generated so this limit should have no impact on the
functionality of the countertop.

Issue #133
slifty added a commit that referenced this issue May 13, 2021
@slifty
Set Kafka retention to 30 seconds
95a0eaa 
TV Kitchen generates a lot of data, and much of it is heavy stuff like
video and audio. Kafka, by default, will store data on its queue for
7 days. This means TV Kitchen will quickly run out of space when
processing streams of video.

This sets a 30 second retention limit for TV Kitchen topics. Data is
processed as it is generated so this limit should have no impact on the
functionality of the countertop.

Issue #133
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Set Kafka retention to 30 seconds tvkitchen/countertop
1 participant
@slifty