Implement export course (#1887)

jlge · evanyeyeye · 20wildmanj · 20wildmanj · commit edf584a4b171 · 2023-10-06T15:34:49.000-04:00
* Fix bug in GitHub api rate limit check (#1821) Fix buggy code * Update docker compose docs (#1823) * Update docker compose docs * Add make warning * Update Export / Import Assessment to support more fields, make importAsmtFromTar and importAssessment more robust (#1822) * - Lint ruby files within spec/ * Add more fields to yml serialization of assessment Add error checking to import assessments (still some errors) * add check to ensure asmt name is valid for import * remove redundant text * create assessment using factory bot, jank test for assessment export * - Add success flash to assessment import - Add a bunch of testcases for bad assessment imports - Modify create_course_with_many_students to handle custom assessment creation, do validation on assessment name * rubocop style * Jump to currently enrolled course (#1812) * Modifications for RuboCop style * Update Manage Submissions test specs to work regardless of jump to course logic --------- Co-authored-by: Damian Ho <damian_ho_xu_yang@yahoo.com> * Bump rack from 2.2.6.2 to 2.2.6.3 (#1828) Bumps [rack](https://github.com/rack/rack) from 2.2.6.2 to 2.2.6.3. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@v2.2.6.2...v2.2.6.3) --- updated-dependencies: - dependency-name: rack dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix bug where course gets created even if there are errors (#1820) * Add required fields to html for name and instructor email * Destroy course if instructor email is invalid * Change required syntax to favored form * Fix annotated PDF download when global annotation is present (#1833) Skip if coordinate is nil * Course start/end date nil checks (#1834) Add nil check for course start and end dates * Bump rack from 2.2.6.3 to 2.2.6.4 (#1835) Bumps [rack](https://github.com/rack/rack) from 2.2.6.3 to 2.2.6.4. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@v2.2.6.3...v2.2.6.4) --- updated-dependencies: - dependency-name: rack dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add LTI Configuration to "Manage Autolab" Dropdown, Update docs and gitignore (#1817) * begin updating lti integration documentation and add feature documentation * - update documentation for LTI configuration, linking, installation - add images for LTI linking for documentation - update gitignore, add ignore node_modules (for stylelint) * Revise LTI docs to make instructions more clear --------- Co-authored-by: Victor Huang <victorhuangwq@gmail.com> * Remove unused "Additional Submission Form" feature code (#1830) * Remove dead code * Update schema version * Fix thead alignment in manage submissions (#1838) * Add sticky to thead css * Remove js file that added a new thead element * Add export route * Add export option to manage course page * Add new stylesheet for export * Move export table to partial * Change id of checkboxes * Fix spacing in export page * Add table styling and checkbox spacing * Remove table header and make font bigger * Implement select all functionality * Fix select all styling when checked * Remove select all button * Fix style errors * Add new lines to eof * Fix style issues * Implement export course config * Add risk condition and watchlist configuration into yaml * Add attachments to export * Rubocop and add course.rb * Add error msg * Format * merge frontend and backend * rubcop * Add more error handling * Filter risk conditions to show only latest version * rubocop * Save actual late_penalty and version_penalty instead of id * Add render tests for export * Clean code 🧼🧼🧹🧹 * Make button repressable * Remove course id * Add more factories and helper functions * Add functionality tests for export_selected endpoint * Add dummy file for activatestorage attachment * Fix mistake in attachment * Rubocop * Comment out error handling for now * error handling tests * rubocop * Address comments * Add backwards compatibility for attachments * Address comment * Edit css to not affect breadcrumbs --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Evan Shi <14984764+evanyeyeye@users.noreply.github.com> Co-authored-by: Joey Wildman <josephwildman88@gmail.com> Co-authored-by: lykimchee <lchaeryn5863@gmail.com> Co-authored-by: Damian Ho <damian_ho_xu_yang@yahoo.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Victor Huang <victorhuangwq@gmail.com>
diff --git a/app/assets/stylesheets/export.scss b/app/assets/stylesheets/export.scss
@@ -29,7 +29,7 @@ tr {
   font-weight: 400 !important;
 }
 
-span {
+span:not(.left-nav):not(.item):not(.title) {
   padding-left: 10px !important;
-  vertical-align: middle !important;
+  vertical-align: middle;
 }
diff --git a/app/controllers/courses_controller.rb b/app/controllers/courses_controller.rb
@@ -581,8 +581,24 @@ def run_moss
     `rm -rf #{tmp_dir}`
   end
 
+  action_auth_level :export, :instructor
   def export; end
 
+  action_auth_level :export_selected, :instructor
+  def export_selected
+    tar_stream = @course.generate_tar(params[:export_configs])
+
+    send_data tar_stream.string.force_encoding("binary"),
+              filename: "#{@course.name}_#{Time.current.strftime('%Y%m%d')}.tar",
+              type: "application/x-tar"
+  rescue SystemCallError => e
+    flash[:error] = "Unable to create the config YAML file: #{e.message}"
+    redirect_to(action: :export)
+  rescue StandardError => e
+    flash[:error] = "Unable to generate tarball -- #{e.message}"
+    redirect_to(action: :export)
+  end
+
 private
 
   def new_course_params
diff --git a/app/models/attachment.rb b/app/models/attachment.rb
@@ -1,4 +1,5 @@
 require "fileutils"
+require "utilities"
 
 ##
 # Attachments are Course or Assessment specific, and allow instructors to
@@ -33,4 +34,9 @@ def file=(upload)
   def after_create
     COURSE_LOGGER.log("Created Attachment #{id}:#{filename} (#{mime_type}) as \"#{name}\")")
   end
+
+  SERIALIZABLE = Set.new %w[filename mime_type released name assessment_id]
+  def serialize
+    Utilities.serializable attributes, SERIALIZABLE
+  end
 end
diff --git a/app/models/course.rb b/app/models/course.rb
@@ -302,6 +302,87 @@ def watchlist_allow_ca
     watchlist_configuration.allow_ca
   end
 
+  def has_attachment?
+    !attachments.nil? && attachments.count > 0
+  end
+
+  def has_risk_conditions?
+    !risk_conditions.nil? && risk_conditions.count > 0
+  end
+
+  def has_watchlist_configuration?
+    !watchlist_configuration.nil?
+  end
+
+  def dump_yaml(include_metrics)
+    YAML.dump(serialize(include_metrics))
+  end
+
+  def generate_tar(export_configs)
+    base_path = Rails.root.join("courses", name).to_s
+    course_dir = name
+    attachments_dir = File.join(course_dir, "attachments")
+    rb_path = "course.rb"
+    config_path = "#{name}.yml"
+    mode = 0o755
+
+    begin
+      tarStream = StringIO.new("")
+      Gem::Package::TarWriter.new(tarStream) do |tar|
+        tar.mkdir course_dir, File.stat(base_path).mode
+
+        # save course.rb
+        source_file = File.open(File.join(base_path, rb_path), 'rb')
+        tar.add_file File.join(course_dir, rb_path), File.stat(source_file).mode do |tar_file|
+          tar_file.write(source_file.read)
+        end
+
+        # save course and metrics config
+        tar.add_file File.join(course_dir, config_path), mode do |tar_file|
+          tar_file.write(dump_yaml(export_configs&.include?('metrics_config')))
+        end
+
+        # save attachments
+        tar.mkdir attachments_dir, File.stat(base_path).mode
+        attachments.each do |attachment|
+          next unless attachment.attachment_file.attached?
+
+          attachment_data = attachment.attachment_file.download
+          filename = attachment.filename
+          relative_path = File.join(attachments_dir, filename)
+
+          tar.add_file relative_path, mode do |file|
+            file.write(attachment_data)
+          end
+        end
+
+        # save assessments
+        if export_configs&.include?('assessments')
+          assessments.each do |assessment|
+            asmt_dir = assessment.name
+            assessment.dump_yaml
+            Dir[File.join(base_path, asmt_dir, "**")].each do |file|
+              mode = File.stat(file).mode
+              relative_path = File.join(course_dir, file.sub(%r{^#{Regexp.escape base_path}/?}, ""))
+
+              if File.directory?(file)
+                tar.mkdir relative_path, mode
+              elsif !relative_path.starts_with? File.join(asmt_dir,
+                                                          assessment.handin_directory)
+                tar.add_file relative_path, mode do |tar_file|
+                  File.open(file, "rb") { |f| tar_file.write f.read }
+                end
+              end
+            end
+          end
+        end
+      end
+      tarStream.rewind
+      tarStream.close
+      tarStream
+    end
+  end
+
 private
 
   def saved_change_to_grade_related_fields?
@@ -336,5 +417,36 @@ def config_module_name
     "Course#{sanitized_name.camelize}"
   end
 
+  def serialize(include_metrics)
+    s = {}
+    s["general"] = serialize_general
+    s["general"]["late_penalty"] = late_penalty.serialize unless late_penalty.nil?
+    s["general"]["version_penalty"] = version_penalty.serialize unless version_penalty.nil?
+    s["attachments"] = attachments.map(&:serialize) if has_attachment?
+
+    if include_metrics
+      if has_risk_conditions?
+        s["risk_conditions"] = risk_conditions.map(&:serialize)
+        latest_version = s["risk_conditions"].max_by{ |k| k["version"] }["version"]
+        s["risk_conditions"] = s["risk_conditions"].select { |condition|
+          condition["version"] == latest_version
+        }
+      end
+
+      if has_watchlist_configuration?
+        s["watchlist_configuration"] =
+          watchlist_configuration.serialize
+      end
+    end
+    s
+  end
+
+  GENERAL_SERIALIZABLE = Set.new %w[name semester late_slack grace_days display_name start_date
+                                    end_date disabled exam_in_progress version_threshold
+                                    gb_message website]
+  def serialize_general
+    Utilities.serializable attributes, GENERAL_SERIALIZABLE
+  end
+
   include CourseAssociationCache
 end
diff --git a/app/models/risk_condition.rb b/app/models/risk_condition.rb
@@ -1,3 +1,5 @@
+require "utilities"
+
 class RiskCondition < ApplicationRecord
   serialize :parameters, Hash
   enum condition_type: { no_condition_selected: 0, grace_day_usage: 1, grade_drop: 2,
@@ -186,4 +188,9 @@ def self.get_max_version(course_id)
       max_version
     end
   end
+
+  SERIALIZABLE = Set.new %w[condition_type parameters version]
+  def serialize
+    Utilities.serializable attributes, SERIALIZABLE
+  end
 end
diff --git a/app/models/watchlist_configuration.rb b/app/models/watchlist_configuration.rb
@@ -70,4 +70,9 @@ def self.update_watchlist_configuration_for_course(course_name, blocklist_update
 
     config
   end
+
+  SERIALIZABLE = Set.new %w[category_blocklist assessment_blocklist allow_ca]
+  def serialize
+    Utilities.serializable attributes, SERIALIZABLE
+  end
 end
diff --git a/app/views/courses/_exportForm.html.erb b/app/views/courses/_exportForm.html.erb
@@ -1,3 +1,4 @@
+<%= form_tag export_selected_course_path, method: 'post' do %>
 <table class="prettyBorder" id="course_fields">
   <colgroup>
     <col span="1" style="width: 50px;">
@@ -7,7 +8,7 @@
     <tr class="course-field checked"> 
       <td>
         <label>
-          <input disabled=true class="cbox" type="checkbox" id="course_config_checkbox" checked>
+           <%= check_box_tag 'export_configs[]', 'course_config', true, class: 'cbox', id: 'course_config_checkbox', disabled: true %>
           <span/>
         </label>
       </td>
@@ -16,26 +17,27 @@
     <tr class="course-field">
       <td>
         <label>
-          <input class="cbox" type="checkbox" id="assessments_checkbox">
+          <%= check_box_tag 'export_configs[]', 'metrics_config', false, class: 'cbox', id: 'metrics_config_checkbox' %>
           <span/>
         </label>
       </td>
-      <td>Assessments</td>
+      <td>Metrics configuration</td>
     </tr>
     <tr class="course-field">
       <td>
         <label>
-          <input class="cbox" type="checkbox" id="metrics_config_checkbox">
+          <%= check_box_tag 'export_configs[]', 'assessments', false, class: 'cbox', id: 'assessments_checkbox' %>
           <span/>
         </label>
       </td>
-      <td>Metrics configuration</td>
+      <td>Assessments</td>
     </tr>
   </tbody>
 </table>
 <br />
 <div class="row">
   <div>
-    <a class="btn" id="export_btn" >Export</a>
+    <%= submit_tag 'Export', class: 'btn', id: 'export_btn', data: { disable_with: false } %>
   </div>
 </div>
+<% end %>
diff --git a/config/routes.rb b/config/routes.rb
@@ -225,6 +225,7 @@
       patch "update_lti_settings"
       match "email", via: [:get, :post]
       get "export"
+      post "export_selected"
       get "manage"
       get "moss"
       post "reload"
diff --git a/spec/contexts_helper.rb b/spec/contexts_helper.rb
@@ -49,4 +49,14 @@ def create_course_with_users_as_hash(asmt_name: "testassessment2")
       instructor_user: @instructor_user, course_assistant_user: @course_assistant_user,
       students_cud: @students, assessment: @assessment }
   end
+
+  def create_course_with_attachment_as_hash
+    create_users
+    puts "Built users"
+    create_course_with_attachment
+    puts "Built course"
+    { course: @course, admin_user: @admin_user,
+      instructor_user: @instructor_user, course_assistant_user: @course_assistant_user,
+      students_cud: @students, assessment: @assessment, attachment: @attachment }
+  end
 end
diff --git a/spec/controllers/courses_controller_spec.rb b/spec/controllers/courses_controller_spec.rb
diff --git a/spec/factories/attachments.rb b/spec/factories/attachments.rb
diff --git a/spec/factories/courses.rb b/spec/factories/courses.rb
diff --git a/spec/sets/courses.rb b/spec/sets/courses.rb

Original file line number	Diff line number	Diff line change
`@@ -29,7 +29,7 @@ tr {`
`29`	`29`	`font-weight: 400 !important;`
`30`	`30`	`}`
`31`	`31`
`32`		`-span {`
	`32`	`+span:not(.left-nav):not(.item):not(.title) {`
`33`	`33`	`padding-left: 10px !important;`
`34`		`- vertical-align: middle !important;`
	`34`	`+ vertical-align: middle;`
`35`	`35`	`}`