Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions json-export-specs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,33 @@ The CIA JSON Export System transforms the comprehensive political intelligence d
| **Committees** | 15 riksdag | Daily | ~600 KB |
| **Intelligence Products** | Risk scores, trends | Daily | ~1.2 MB |

### Implementation Status

> **βœ… v1.36 Update (2024-12-08)**: Core JSON export service layer completed

**Implemented Components**:
- βœ… **IntelligenceExportService** - Service interface for JSON exports (`service.data.api`)
- βœ… **IntelligenceExportServiceImpl** - Jackson-based JSON serialization (`service.data.impl`)
- βœ… **Export DTOs** - Data transfer objects for risk assessments, coalition alignment, temporal trends
- βœ… **Unit Tests** - Comprehensive tests with Mockito (3/3 passing)
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The README states "Unit Tests - Comprehensive tests with Mockito (3/3 passing)" but the test file contains 6 test methods, not 3. This discrepancy should be corrected to accurately reflect the test coverage.

Update to: "Unit Tests - Comprehensive tests with Mockito (6/6 passing)"

Suggested change
- βœ… **Unit Tests** - Comprehensive tests with Mockito (3/3 passing)
- βœ… **Unit Tests** - Comprehensive tests with Mockito (6/6 passing)

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac4dbfe. Corrected test count from "3/3 passing" to "6/6 passing" to accurately reflect the actual test coverage.

- βœ… **Build Integration** - Clean compile and test pass in CI/CD pipeline

**Ready for Integration**:
1. `exportRiskAssessments()` - Exports all rule violations with severity and metadata
2. `exportCoalitionAlignment()` - Exports party alignment matrix with voting cohesion
3. `exportTemporalTrends()` - Exports decision trends with daily/weekly/monthly moving averages
4. `writeJsonToFile()` - File writer for CDN-ready static JSON generation

**Next Steps**:
- REST endpoint integration (planned)
- Scheduled daily export job (planned)
- CDN deployment automation (planned)

**Technical Foundation**: Service layer built on Spring Framework with JPA/Hibernate, using existing DAO layer for database access to:
- `RuleViolationDAO` β†’ Risk assessment data (50 behavioral rules)
- `ViewRiksdagenCoalitionAlignmentMatrixDAO` β†’ Coalition alignment data
- `ViewDecisionTemporalTrendsDAO` β†’ Temporal trend analysis data

---

## πŸ“‹ Table of Contents
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* Copyright 2010-2025 James Pether SΓΆrling
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* $Id$
* $HeadURL$
*/
package com.hack23.cia.service.data.api;

import java.io.IOException;

/**
* Service interface for exporting intelligence products as JSON.
*
* Provides JSON export capabilities for risk assessments, coalition alignment,
* and temporal trends to support CDN-ready static file generation and API endpoints.
*
* @author intelligence-operative
* @since v1.36 (Intelligence Products JSON Export)
*/
public interface IntelligenceExportService {

/**
* Export risk assessment data for all rule violations.
*
* Generates JSON containing all 50 risk rules with their current violations,
* severity levels, and affected entities (politicians, parties, committees, ministries).
*
* @return JSON string containing risk assessment data
* @throws IOException if JSON generation fails
*/
String exportRiskAssessments() throws IOException;

/**
* Export coalition alignment matrix showing voting alignment between parties.
*
* Generates JSON containing pairwise party alignment rates, total votes,
* and aligned votes for coalition stability analysis.
*
* @return JSON string containing coalition alignment matrix
* @throws IOException if JSON generation fails
*/
String exportCoalitionAlignment() throws IOException;

/**
* Export temporal trend analysis with daily/weekly/monthly/annual patterns.
*
* Generates JSON containing decision volumes, approval rates, moving averages,
* and year-over-year comparisons for trend forecasting.
*
* @return JSON string containing temporal trends data
* @throws IOException if JSON generation fails
*/
String exportTemporalTrends() throws IOException;

/**
* Write JSON export to file for CDN deployment.
*
* @param jsonContent the JSON content to write
* @param fileName the file name (e.g., "risk-assessments.json")
* @param outputDirectory the output directory path
* @throws IOException if file write fails
*/
void writeJsonToFile(String jsonContent, String fileName, String outputDirectory) throws IOException;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
/*
* Copyright 2010-2025 James Pether SΓΆrling
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* $Id$
* $HeadURL$
*/
package com.hack23.cia.service.data.impl;

import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Date;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.hack23.cia.model.internal.application.data.impl.ViewDecisionTemporalTrends;
import com.hack23.cia.model.internal.application.data.party.impl.ViewRiksdagenCoalitionAlignmentMatrix;
import com.hack23.cia.model.internal.application.data.rules.impl.RuleViolation;
import com.hack23.cia.service.data.api.IntelligenceExportService;
import com.hack23.cia.service.data.api.RuleViolationDAO;
import com.hack23.cia.service.data.api.ViewDecisionTemporalTrendsDAO;
import com.hack23.cia.service.data.api.ViewRiksdagenCoalitionAlignmentMatrixDAO;
import com.hack23.cia.service.data.impl.export.CoalitionAlignmentExportDTO;
import com.hack23.cia.service.data.impl.export.CoalitionAlignmentExportDTO.PartyAlignment;
import com.hack23.cia.service.data.impl.export.ExportMetadata;
import com.hack23.cia.service.data.impl.export.RiskAssessmentExportDTO;
import com.hack23.cia.service.data.impl.export.RiskAssessmentExportDTO.RiskViolation;
import com.hack23.cia.service.data.impl.export.TemporalTrendsExportDTO;
import com.hack23.cia.service.data.impl.export.TemporalTrendsExportDTO.TrendDataPoint;

/**
* Implementation of IntelligenceExportService.
*
* @author intelligence-operative
* @since v1.36
*/
@Service
@Transactional(readOnly = true)
final class IntelligenceExportServiceImpl implements IntelligenceExportService {

@Autowired
private RuleViolationDAO ruleViolationDAO;

@Autowired
private ViewRiksdagenCoalitionAlignmentMatrixDAO coalitionAlignmentDAO;

@Autowired
private ViewDecisionTemporalTrendsDAO temporalTrendsDAO;

private final ObjectMapper objectMapper;

public IntelligenceExportServiceImpl() {
this.objectMapper = new ObjectMapper();
this.objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
this.objectMapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
}
Comment on lines 73 to 78
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ObjectMapper is configured to disable WRITE_DATES_AS_TIMESTAMPS, but Jackson also needs a date format configuration to properly serialize dates as ISO 8601 strings. Without an explicit date format, dates may not serialize consistently.

Consider adding:

this.objectMapper.setDateFormat(new StdDateFormat().withColonInTimeZone(true));

This ensures dates are serialized as ISO 8601 strings (e.g., "2024-12-08T11:12:00Z") as described in the PR description.

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac4dbfe. Added StdDateFormat().withColonInTimeZone(true) configuration to ObjectMapper to ensure dates are serialized as ISO 8601 strings (e.g., "2024-12-08T11:12:00Z").


@Override
public String exportRiskAssessments() throws IOException {
final List<RuleViolation> violations = ruleViolationDAO.getAll();

final RiskAssessmentExportDTO dto = new RiskAssessmentExportDTO();

final ExportMetadata metadata = new ExportMetadata();
metadata.setSchema("intelligence-schema");

Check failure on line 84 in service.data.impl/src/main/java/com/hack23/cia/service/data/impl/IntelligenceExportServiceImpl.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Define a constant instead of duplicating this literal "intelligence-schema" 3 times.

See more on https://sonarcloud.io/project/issues?id=Hack23_cia&issues=AZr9qiz8bOiZaeb3GYME&open=AZr9qiz8bOiZaeb3GYME&pullRequest=8048
metadata.setRecordCount(violations.size());
metadata.setDataDate(new Date());
dto.setMetadata(metadata);

for (final RuleViolation violation : violations) {
final RiskViolation riskViolation = new RiskViolation();
riskViolation.setId(violation.getId());
riskViolation.setDetectedDate(violation.getDetectedDate());
riskViolation.setReferenceId(violation.getReferenceId());
riskViolation.setName(violation.getName());
riskViolation.setResourceType(violation.getResourceType() != null ? violation.getResourceType().toString() : null);
riskViolation.setRuleName(violation.getRuleName());
riskViolation.setRuleDescription(violation.getRuleDescription());
riskViolation.setRuleGroup(violation.getRuleGroup());
riskViolation.setStatus(violation.getStatus() != null ? violation.getStatus().toString() : null);
riskViolation.setPositive(violation.getPositive());
dto.getViolations().add(riskViolation);
}

return objectMapper.writeValueAsString(dto);
}

@Override
public String exportCoalitionAlignment() throws IOException {
final List<ViewRiksdagenCoalitionAlignmentMatrix> alignments = coalitionAlignmentDAO.getAll();

final CoalitionAlignmentExportDTO dto = new CoalitionAlignmentExportDTO();

final ExportMetadata metadata = new ExportMetadata();
metadata.setSchema("intelligence-schema");
metadata.setRecordCount(alignments.size());
metadata.setDataDate(new Date());
dto.setMetadata(metadata);

for (final ViewRiksdagenCoalitionAlignmentMatrix alignment : alignments) {
final PartyAlignment partyAlignment = new PartyAlignment();
if (alignment.getEmbeddedId() != null) {
partyAlignment.setParty1(alignment.getEmbeddedId().getParty1());
partyAlignment.setParty2(alignment.getEmbeddedId().getParty2());
}
partyAlignment.setAlignmentRate(alignment.getAlignmentRate());
partyAlignment.setSharedVotes(alignment.getSharedVotes());
partyAlignment.setAlignedVotes(alignment.getAlignedVotes());
dto.getAlignments().add(partyAlignment);
}

return objectMapper.writeValueAsString(dto);
}

@Override
public String exportTemporalTrends() throws IOException {
final List<ViewDecisionTemporalTrends> trends = temporalTrendsDAO.getAll();

final TemporalTrendsExportDTO dto = new TemporalTrendsExportDTO();

final ExportMetadata metadata = new ExportMetadata();
metadata.setSchema("intelligence-schema");
metadata.setRecordCount(trends.size());
metadata.setDataDate(new Date());
dto.setMetadata(metadata);

for (final ViewDecisionTemporalTrends trend : trends) {
final TrendDataPoint dataPoint = new TrendDataPoint();
dataPoint.setDecisionDay(trend.getDecisionDay());
dataPoint.setDailyDecisions(trend.getDailyDecisions());
dataPoint.setDailyApprovalRate(trend.getDailyApprovalRate());
dataPoint.setApprovedDecisions(trend.getApprovedDecisions());
dataPoint.setRejectedDecisions(trend.getRejectedDecisions());
dataPoint.setReferredBackDecisions(trend.getReferredBackDecisions());
dataPoint.setCommitteeReferralDecisions(trend.getCommitteeReferralDecisions());
dataPoint.setMa7dayDecisions(trend.getMa7dayDecisions());
dataPoint.setMa30dayDecisions(trend.getMa30dayDecisions());
dataPoint.setMa90dayDecisions(trend.getMa90dayDecisions());
dataPoint.setMa30dayApprovalRate(trend.getMa30dayApprovalRate());
dataPoint.setDecisionsLastYear(trend.getDecisionsLastYear());
dataPoint.setYoyDecisionsChange(trend.getYoyDecisionsChange());
dataPoint.setYoyDecisionsChangePct(trend.getYoyDecisionsChangePct());
dataPoint.setDecisionYear(trend.getDecisionYear());
dataPoint.setDecisionMonth(trend.getDecisionMonth());
dataPoint.setDecisionWeek(trend.getDecisionWeek());
dataPoint.setDayOfWeek(trend.getDayOfWeek());
dto.getTrends().add(dataPoint);
}

return objectMapper.writeValueAsString(dto);
}

@Override
public void writeJsonToFile(final String jsonContent, final String fileName, final String outputDirectory)
throws IOException {
// Validate fileName doesn't contain path separators or traversal sequences
if (fileName.contains("/") || fileName.contains("\\") || fileName.contains("..")) {
throw new IllegalArgumentException("Invalid file name: must not contain path separators or traversal sequences");
}
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The security validation for path traversal is incomplete. The check fileName.contains("..") will reject legitimate file names like "risk-2024..2025.json" that contain ".." as part of the content, not just as a traversal sequence.

A more robust check should validate that the resolved path stays within the target directory:

final Path targetDir = Paths.get(outputDirectory).toAbsolutePath().normalize();
final Path resolvedPath = targetDir.resolve(fileName).normalize();
if (!resolvedPath.startsWith(targetDir)) {
    throw new IllegalArgumentException("Invalid file name: path traversal detected");
}

This approach properly prevents path traversal while allowing legitimate file names.

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac4dbfe. Replaced simple string contains check with robust path validation using Path.normalize() and startsWith() to properly detect path traversal while allowing legitimate file names. The implementation now validates that the resolved path stays within the target directory.


final File directory = new File(outputDirectory);
if (!directory.exists() && !directory.mkdirs()) {
throw new IOException("Failed to create directory: " + outputDirectory);
}

final String filePath = outputDirectory + File.separator + fileName;
Files.write(Paths.get(filePath), jsonContent.getBytes(StandardCharsets.UTF_8));
}
}
Loading
Loading