Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework scope cleanup microservice #1891

Merged
merged 2 commits into from
Feb 12, 2025
+84 −75
Merged

Rework scope cleanup microservice #1891

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d1954c4
Rework scope cleanup microservice
jmthomas Feb 11, 2025
19f528c
Remove cli save/load
jmthomas Feb 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 30 additions & 29 deletions .github/workflows/cli.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,35 +116,36 @@ jobs:
./openc3.sh cli script run --wait 20 INST/procedures/checks.rb | tee /dev/tty | grep -q "script failed"
# run a script that will complete successfully
./openc3.sh cli script run INST/procedures/stash.rb | tee /dev/tty | grep "script complete"
- name: openc3.sh util save,load
shell: 'script -q -e -c "bash {0}"'
run: |
set -e
./openc3.sh util save docker.io openc3inc latest
# Verify the images were saved
ls tmp | grep openc3-cosmos-cmd-tlm-api-latest.tar
ls tmp | grep openc3-cosmos-script-runner-api-latest.tar
ls tmp | grep openc3-cosmos-init-latest.tar
ls tmp | grep openc3-operator-latest.tar
ls tmp | grep openc3-minio-latest.tar
ls tmp | grep openc3-redis-latest.tar
ls tmp | grep openc3-traefik-latest.tar
docker image rm openc3inc/openc3-cosmos-cmd-tlm-api:latest
docker image rm openc3inc/openc3-cosmos-script-runner-api:latest
docker image rm openc3inc/openc3-cosmos-init:latest
docker image rm openc3inc/openc3-operator:latest
docker image rm openc3inc/openc3-minio:latest
docker image rm openc3inc/openc3-redis:latest
docker image rm openc3inc/openc3-traefik:latest
./openc3.sh util load
# Verify the images were loaded
docker image ls | grep openc3-cosmos-cmd-tlm-api
docker image ls | grep openc3-cosmos-script-runner-api
docker image ls | grep openc3-cosmos-init
docker image ls | grep openc3-operator
docker image ls | grep openc3-minio
docker image ls | grep openc3-redis
docker image ls | grep openc3-traefik
# Removing due to write tmp/.docker_temp_3200699358: no space left on device
# - name: openc3.sh util save,load
# shell: 'script -q -e -c "bash {0}"'
# run: |
# set -e
# ./openc3.sh util save docker.io openc3inc latest
# # Verify the images were saved
# ls tmp | grep openc3-cosmos-cmd-tlm-api-latest.tar
# ls tmp | grep openc3-cosmos-script-runner-api-latest.tar
# ls tmp | grep openc3-cosmos-init-latest.tar
# ls tmp | grep openc3-operator-latest.tar
# ls tmp | grep openc3-minio-latest.tar
# ls tmp | grep openc3-redis-latest.tar
# ls tmp | grep openc3-traefik-latest.tar
# docker image rm openc3inc/openc3-cosmos-cmd-tlm-api:latest
# docker image rm openc3inc/openc3-cosmos-script-runner-api:latest
# docker image rm openc3inc/openc3-cosmos-init:latest
# docker image rm openc3inc/openc3-operator:latest
# docker image rm openc3inc/openc3-minio:latest
# docker image rm openc3inc/openc3-redis:latest
# docker image rm openc3inc/openc3-traefik:latest
# ./openc3.sh util load
# # Verify the images were loaded
# docker image ls | grep openc3-cosmos-cmd-tlm-api
# docker image ls | grep openc3-cosmos-script-runner-api
# docker image ls | grep openc3-cosmos-init
# docker image ls | grep openc3-operator
# docker image ls | grep openc3-minio
# docker image ls | grep openc3-redis
# docker image ls | grep openc3-traefik
- name: Cleanup volumes
shell: 'script -q -e -c "bash {0}"'
run: |
Expand Down
2 changes: 1 addition & 1 deletion openc3/data/config/target.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ TARGET:
parameters:
- name: Time
required: true
description: Number of seconds between runs of the cleanup process (default = 900 = 15 minutes)
description: Number of seconds between runs of the cleanup process (default = 600 = 10 minutes)
values: \d+
REDUCER_DISABLE:
summary: Disables the data reduction microservice for the target
Expand Down
71 changes: 37 additions & 34 deletions openc3/lib/openc3/microservices/cleanup_microservice.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# GNU Affero General Public License for more details.

# Modified by OpenC3, Inc.
# All changes Copyright 2022, OpenC3, Inc.
# All changes Copyright 2025, OpenC3, Inc.
# All Rights Reserved
#
# This file may also be used under the terms of a commercial license
Expand All @@ -35,44 +35,33 @@ def initialize(*args)
@sleeper = Sleeper.new
end

def cleanup(areas, cleanup_poll_time)
bucket = Bucket.getClient()
while true
break if @cancel_thread

@state = 'GETTING_OBJECTS'
start_time = Time.now
areas.each do |prefix, retain_time|
next unless retain_time
time = start_time - retain_time
oldest_list = BucketUtilities.files_between_time(ENV['OPENC3_LOGS_BUCKET'], prefix, nil, time)
if oldest_list.length > 0
@state = 'DELETING_OBJECTS'
oldest_list.each_slice(1000) do |slice|
# The delete_objects function utilizes an MD5 hash when verifying the checksums, which is not
# FIPS compliant (https://github.com/aws/aws-sdk-ruby/issues/2645).
# delete_object does NOT require an MD5 hash and will work on FIPS compliant systems. It is
# probably less performant, but we can instead delete each item one at a time.
# bucket.delete_objects(bucket: ENV['OPENC3_LOGS_BUCKET'], keys: slice)
slice.each do |item|
bucket.delete_object(bucket: ENV['OPENC3_LOGS_BUCKET'], key: item)
end
@logger.debug("Cleanup deleted #{slice.length} log files")
@delete_count += slice.length
@metric.set(name: 'cleanup_delete_total', value: @delete_count, type: 'counter')
def cleanup(areas, bucket)
@state = 'GETTING_OBJECTS'
start_time = Time.now
areas.each do |prefix, retain_time|
next unless retain_time
time = start_time - retain_time
oldest_list = BucketUtilities.files_between_time(ENV['OPENC3_LOGS_BUCKET'], prefix, nil, time)
if oldest_list.length > 0
@state = 'DELETING_OBJECTS'
oldest_list.each_slice(1000) do |slice|
# The delete_objects function utilizes an MD5 hash when verifying the checksums, which is not
# FIPS compliant (https://github.com/aws/aws-sdk-ruby/issues/2645).
# delete_object does NOT require an MD5 hash and will work on FIPS compliant systems. It is
# probably less performant, but we can instead delete each item one at a time.
# bucket.delete_objects(bucket: ENV['OPENC3_LOGS_BUCKET'], keys: slice)
slice.each do |item|
bucket.delete_object(bucket: ENV['OPENC3_LOGS_BUCKET'], key: item)
end
@logger.debug("Cleanup deleted #{slice.length} log files")
@delete_count += slice.length
@metric.set(name: 'cleanup_delete_total', value: @delete_count, type: 'counter')
end
end

@count += 1
@metric.set(name: 'cleanup_total', value: @count, type: 'counter')

@state = 'SLEEPING'
break if @sleeper.sleep(cleanup_poll_time)
end
end

def run
def get_areas_and_poll_time
split_name = @name.split("__")
target_name = split_name[-1]
target = TargetModel.get_model(name: target_name, scope: @scope)
Expand All @@ -86,8 +75,22 @@ def run
["#{@scope}/reduced_hour_logs/tlm/#{target_name}", target.reduced_hour_log_retain_time],
["#{@scope}/reduced_day_logs/tlm/#{target_name}", target.reduced_day_log_retain_time],
]
return areas, target.cleanup_poll_time
end

cleanup(areas, target.cleanup_poll_time)
def run
bucket = Bucket.getClient()
while true
break if @cancel_thread
areas, poll_time = get_areas_and_poll_time()
cleanup(areas, bucket)

@count += 1
@metric.set(name: 'cleanup_total', value: @count, type: 'counter')

@state = 'SLEEPING'
break if @sleeper.sleep(poll_time)
end
end

def shutdown
Expand Down
17 changes: 8 additions & 9 deletions openc3/lib/openc3/microservices/scope_cleanup_microservice.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# encoding: ascii-8bit

# Copyright 2023 OpenC3, Inc.
# Copyright 2025 OpenC3, Inc.
# All Rights Reserved.
#
# This program is free software; you can modify and/or redistribute it
Expand All @@ -21,22 +21,21 @@

module OpenC3
class ScopeCleanupMicroservice < CleanupMicroservice
def run
def get_areas_and_poll_time
scope = ScopeModel.get_model(name: @scope)

areas = [
["#{@scope}/text_logs", scope.text_log_retain_time],
["#{@scope}/tool_logs", scope.tool_log_retain_time],
["#{@scope}/text_logs/openc3_log_messages", scope.text_log_retain_time],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did these paths get more specific? That wont support future changes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will improve this in the future if needed.

["#{@scope}/tool_logs/sr", scope.tool_log_retain_time],
]

if @scope == 'DEFAULT'
areas << ["NOSCOPE/text_logs", scope.text_log_retain_time]
areas << ["NOSCOPE/tool_logs", scope.tool_log_retain_time]
areas << ["NOSCOPE/text_logs/openc3_log_messages", scope.text_log_retain_time]
areas << ["NOSCOPE/tool_logs/sr", scope.tool_log_retain_time]
end

cleanup(areas, scope.cleanup_poll_time)
return areas, scope.cleanup_poll_time
end
end
end

OpenC3::CleanupMicroservice.run if __FILE__ == $0
OpenC3::ScopeCleanupMicroservice.run if __FILE__ == $0
8 changes: 7 additions & 1 deletion openc3/lib/openc3/models/scope_model.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ def initialize(name:,
text_log_cycle_size: 50_000_000,
text_log_retain_time: nil,
tool_log_retain_time: nil,
cleanup_poll_time: 900,
cleanup_poll_time: 600,
command_authority: false,
critical_commanding: "OFF",
shard: 0,
Expand Down Expand Up @@ -129,6 +129,12 @@ def create(update: false, force: false, queued: false)
raise "Invalid scope name: #{@name}" if @name !~ /^[a-zA-Z0-9_-]+$/
@name = @name.upcase
@scope = @name # Ensure @scope matches @name
# Ensure the various cycle and retain times are integers
@text_log_cycle_time = @text_log_cycle_time.to_i
@text_log_cycle_size = @text_log_cycle_size.to_i
@text_log_retain_time = @text_log_retain_time.to_i if @text_log_retain_time
@tool_log_retain_time = @tool_log_retain_time.to_i if @tool_log_retain_time
@cleanup_poll_time = @cleanup_poll_time.to_i
super(update: update, force: force, queued: queued)

if ENTERPRISE
Expand Down
2 changes: 1 addition & 1 deletion openc3/lib/openc3/models/target_model.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,7 @@ def initialize(
reduced_minute_log_retain_time: nil,
reduced_hour_log_retain_time: nil,
reduced_day_log_retain_time: nil,
cleanup_poll_time: 900,
cleanup_poll_time: 600,
needs_dependencies: false,
target_microservices: {'REDUCER' => [[]]},
reducer_disable: false,
Expand Down
Loading