Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions json-export-specs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,33 @@ The CIA JSON Export System transforms the comprehensive political intelligence d
| **Committees** | 15 riksdag | Daily | ~600 KB |
| **Intelligence Products** | Risk scores, trends | Daily | ~1.2 MB |

### Implementation Status

> **βœ… v1.36 Update (2024-12-08)**: Core JSON export service layer completed

**Implemented Components**:
- βœ… **IntelligenceExportService** - Service interface for JSON exports (`service.data.api`)
- βœ… **IntelligenceExportServiceImpl** - Jackson-based JSON serialization (`service.data.impl`)
- βœ… **Export DTOs** - Data transfer objects for risk assessments, coalition alignment, temporal trends
- βœ… **Unit Tests** - Comprehensive tests with Mockito (3/3 passing)
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The README states "Unit Tests - Comprehensive tests with Mockito (3/3 passing)" but the test file contains 6 test methods, not 3. This discrepancy should be corrected to accurately reflect the test coverage.

Update to: "Unit Tests - Comprehensive tests with Mockito (6/6 passing)"

Suggested change
- βœ… **Unit Tests** - Comprehensive tests with Mockito (3/3 passing)
- βœ… **Unit Tests** - Comprehensive tests with Mockito (6/6 passing)

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac4dbfe. Corrected test count from "3/3 passing" to "6/6 passing" to accurately reflect the actual test coverage.

- βœ… **Build Integration** - Clean compile and test pass in CI/CD pipeline

**Ready for Integration**:
1. `exportRiskAssessments()` - Exports all rule violations with severity and metadata
2. `exportCoalitionAlignment()` - Exports party alignment matrix with voting cohesion
3. `exportTemporalTrends()` - Exports decision trends with daily/weekly/monthly moving averages
4. `writeJsonToFile()` - File writer for CDN-ready static JSON generation

**Next Steps**:
- REST endpoint integration (planned)
- Scheduled daily export job (planned)
- CDN deployment automation (planned)

**Technical Foundation**: Service layer built on Spring Framework with JPA/Hibernate, using existing DAO layer for database access to:
- `RuleViolationDAO` β†’ Risk assessment data (50 behavioral rules)
- `ViewRiksdagenCoalitionAlignmentMatrixDAO` β†’ Coalition alignment data
- `ViewDecisionTemporalTrendsDAO` β†’ Temporal trend analysis data

---

## πŸ“‹ Table of Contents
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* Copyright 2010-2025 James Pether SΓΆrling
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* $Id$
* $HeadURL$
*/
package com.hack23.cia.service.data.api;

import java.io.IOException;

/**
* Service interface for exporting intelligence products as JSON.
*
* Provides JSON export capabilities for risk assessments, coalition alignment,
* and temporal trends to support CDN-ready static file generation and API endpoints.
*
* @author intelligence-operative
* @since v1.36 (Intelligence Products JSON Export)
*/
public interface IntelligenceExportService {

/**
* Export risk assessment data for all rule violations.
*
* Generates JSON containing all 50 risk rules with their current violations,
* severity levels, and affected entities (politicians, parties, committees, ministries).
*
* @return JSON string containing risk assessment data
* @throws IOException if JSON generation fails
*/
String exportRiskAssessments() throws IOException;

/**
* Export coalition alignment matrix showing voting alignment between parties.
*
* Generates JSON containing pairwise party alignment rates, total votes,
* and aligned votes for coalition stability analysis.
*
* @return JSON string containing coalition alignment matrix
* @throws IOException if JSON generation fails
*/
String exportCoalitionAlignment() throws IOException;

/**
* Export temporal trend analysis with daily/weekly/monthly/annual patterns.
*
* Generates JSON containing decision volumes, approval rates, moving averages,
* and year-over-year comparisons for trend forecasting.
*
* @return JSON string containing temporal trends data
* @throws IOException if JSON generation fails
*/
String exportTemporalTrends() throws IOException;

/**
* Write JSON export to file for CDN deployment.
*
* @param jsonContent the JSON content to write
* @param fileName the file name (e.g., "risk-assessments.json")
* @param outputDirectory the output directory path
* @throws IOException if file write fails
*/
void writeJsonToFile(String jsonContent, String fileName, String outputDirectory) throws IOException;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
/*
* Copyright 2010-2025 James Pether SΓΆrling
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* $Id$
* $HeadURL$
*/
package com.hack23.cia.service.data.impl;

import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Date;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.hack23.cia.model.internal.application.data.impl.ViewDecisionTemporalTrends;
import com.hack23.cia.model.internal.application.data.party.impl.ViewRiksdagenCoalitionAlignmentMatrix;
import com.hack23.cia.model.internal.application.data.rules.impl.RuleViolation;
import com.hack23.cia.service.data.api.IntelligenceExportService;
import com.hack23.cia.service.data.api.RuleViolationDAO;
import com.hack23.cia.service.data.api.ViewDecisionTemporalTrendsDAO;
import com.hack23.cia.service.data.api.ViewRiksdagenCoalitionAlignmentMatrixDAO;
import com.hack23.cia.service.data.impl.export.CoalitionAlignmentExportDTO;
import com.hack23.cia.service.data.impl.export.CoalitionAlignmentExportDTO.PartyAlignment;
import com.hack23.cia.service.data.impl.export.ExportMetadata;
import com.hack23.cia.service.data.impl.export.RiskAssessmentExportDTO;
import com.hack23.cia.service.data.impl.export.RiskAssessmentExportDTO.RiskViolation;
import com.hack23.cia.service.data.impl.export.TemporalTrendsExportDTO;
import com.hack23.cia.service.data.impl.export.TemporalTrendsExportDTO.TrendDataPoint;

/**
* Implementation of IntelligenceExportService.
*
* @author intelligence-operative
* @since v1.36
*/
@Service
@Transactional(readOnly = true)
public class IntelligenceExportServiceImpl implements IntelligenceExportService {
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The class is declared as public class which breaks the codebase convention. All other service implementations in this package use final class with package-private visibility (no public modifier). This should be:

@Service
@Transactional(readOnly = true)
final class IntelligenceExportServiceImpl implements IntelligenceExportService {
Suggested change
public class IntelligenceExportServiceImpl implements IntelligenceExportService {
final class IntelligenceExportServiceImpl implements IntelligenceExportService {

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5fc4262. Changed from public class to final class to match codebase conventions.


@Autowired
private RuleViolationDAO ruleViolationDAO;

@Autowired
private ViewRiksdagenCoalitionAlignmentMatrixDAO coalitionAlignmentDAO;

@Autowired
private ViewDecisionTemporalTrendsDAO temporalTrendsDAO;

private final ObjectMapper objectMapper;

public IntelligenceExportServiceImpl() {
this.objectMapper = new ObjectMapper();
this.objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
this.objectMapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
}
Comment on lines 73 to 78
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ObjectMapper is configured to disable WRITE_DATES_AS_TIMESTAMPS, but Jackson also needs a date format configuration to properly serialize dates as ISO 8601 strings. Without an explicit date format, dates may not serialize consistently.

Consider adding:

this.objectMapper.setDateFormat(new StdDateFormat().withColonInTimeZone(true));

This ensures dates are serialized as ISO 8601 strings (e.g., "2024-12-08T11:12:00Z") as described in the PR description.

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac4dbfe. Added StdDateFormat().withColonInTimeZone(true) configuration to ObjectMapper to ensure dates are serialized as ISO 8601 strings (e.g., "2024-12-08T11:12:00Z").


@Override
public String exportRiskAssessments() throws IOException {
final List<RuleViolation> violations = ruleViolationDAO.getAll();

final RiskAssessmentExportDTO dto = new RiskAssessmentExportDTO();

final ExportMetadata metadata = new ExportMetadata();
metadata.setSchema("intelligence-schema");

Check failure on line 84 in service.data.impl/src/main/java/com/hack23/cia/service/data/impl/IntelligenceExportServiceImpl.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Define a constant instead of duplicating this literal "intelligence-schema" 3 times.

See more on https://sonarcloud.io/project/issues?id=Hack23_cia&issues=AZr9qiz8bOiZaeb3GYME&open=AZr9qiz8bOiZaeb3GYME&pullRequest=8048
metadata.setRecordCount(violations.size());
metadata.setDataDate(new Date());
dto.setMetadata(metadata);

for (final com.hack23.cia.model.internal.application.data.rules.impl.RuleViolation violation : violations) {
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fully-qualified class name is unnecessary here since RuleViolation is already imported at line 37 and used without FQCN on line 79. This should simply be:

for (final RuleViolation violation : violations) {
Suggested change
for (final com.hack23.cia.model.internal.application.data.rules.impl.RuleViolation violation : violations) {
for (final RuleViolation violation : violations) {

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5fc4262. Removed fully-qualified class name and used the imported RuleViolation directly.

final RiskViolation riskViolation = new RiskViolation();
riskViolation.setId(violation.getId());
riskViolation.setDetectedDate(violation.getDetectedDate());
riskViolation.setReferenceId(violation.getReferenceId());
riskViolation.setName(violation.getName());
riskViolation.setResourceType(violation.getResourceType() != null ? violation.getResourceType().toString() : null);
riskViolation.setRuleName(violation.getRuleName());
riskViolation.setRuleDescription(violation.getRuleDescription());
riskViolation.setRuleGroup(violation.getRuleGroup());
riskViolation.setStatus(violation.getStatus() != null ? violation.getStatus().toString() : null);
riskViolation.setPositive(violation.getPositive());
dto.getViolations().add(riskViolation);
}

return objectMapper.writeValueAsString(dto);
}

@Override
public String exportCoalitionAlignment() throws IOException {
final List<ViewRiksdagenCoalitionAlignmentMatrix> alignments = coalitionAlignmentDAO.getAll();

final CoalitionAlignmentExportDTO dto = new CoalitionAlignmentExportDTO();

final ExportMetadata metadata = new ExportMetadata();
metadata.setSchema("intelligence-schema");
metadata.setRecordCount(alignments.size());
metadata.setDataDate(new Date());
dto.setMetadata(metadata);

for (final ViewRiksdagenCoalitionAlignmentMatrix alignment : alignments) {
final PartyAlignment partyAlignment = new PartyAlignment();
if (alignment.getEmbeddedId() != null) {
partyAlignment.setParty1(alignment.getEmbeddedId().getParty1());
partyAlignment.setParty2(alignment.getEmbeddedId().getParty2());
}
partyAlignment.setAlignmentRate(alignment.getAlignmentRate());
partyAlignment.setSharedVotes(alignment.getSharedVotes());
partyAlignment.setAlignedVotes(alignment.getAlignedVotes());
dto.getAlignments().add(partyAlignment);
}

return objectMapper.writeValueAsString(dto);
}

@Override
public String exportTemporalTrends() throws IOException {
final List<ViewDecisionTemporalTrends> trends = temporalTrendsDAO.getAll();

final TemporalTrendsExportDTO dto = new TemporalTrendsExportDTO();

final ExportMetadata metadata = new ExportMetadata();
metadata.setSchema("intelligence-schema");
metadata.setRecordCount(trends.size());
metadata.setDataDate(new Date());
dto.setMetadata(metadata);

for (final ViewDecisionTemporalTrends trend : trends) {
final TrendDataPoint dataPoint = new TrendDataPoint();
dataPoint.setDecisionDay(trend.getDecisionDay());
dataPoint.setDailyDecisions(trend.getDailyDecisions());
dataPoint.setDailyApprovalRate(trend.getDailyApprovalRate());
dataPoint.setApprovedDecisions(trend.getApprovedDecisions());
dataPoint.setRejectedDecisions(trend.getRejectedDecisions());
dataPoint.setReferredBackDecisions(trend.getReferredBackDecisions());
dataPoint.setCommitteeReferralDecisions(trend.getCommitteeReferralDecisions());
dataPoint.setMa7dayDecisions(trend.getMa7dayDecisions());
dataPoint.setMa30dayDecisions(trend.getMa30dayDecisions());
dataPoint.setMa90dayDecisions(trend.getMa90dayDecisions());
dataPoint.setMa30dayApprovalRate(trend.getMa30dayApprovalRate());
dataPoint.setDecisionsLastYear(trend.getDecisionsLastYear());
dataPoint.setYoyDecisionsChange(trend.getYoyDecisionsChange());
dataPoint.setYoyDecisionsChangePct(trend.getYoyDecisionsChangePct());
dataPoint.setDecisionYear(trend.getDecisionYear());
dataPoint.setDecisionMonth(trend.getDecisionMonth());
dataPoint.setDecisionWeek(trend.getDecisionWeek());
dataPoint.setDayOfWeek(trend.getDayOfWeek());
dto.getTrends().add(dataPoint);
}

return objectMapper.writeValueAsString(dto);
}

@Override
public void writeJsonToFile(final String jsonContent, final String fileName, final String outputDirectory)
throws IOException {
final File directory = new File(outputDirectory);
if (!directory.exists()) {
directory.mkdirs();
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of mkdirs() without checking the return value can lead to silent failures. If directory creation fails (e.g., due to permissions), the subsequent Files.write() will fail. Check the return value and throw an appropriate exception:

if (!directory.exists() && !directory.mkdirs()) {
    throw new IOException("Failed to create directory: " + outputDirectory);
}
Suggested change
if (!directory.exists()) {
directory.mkdirs();
if (!directory.exists() && !directory.mkdirs()) {
throw new IOException("Failed to create directory: " + outputDirectory);

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5fc4262. Added check for mkdirs() return value and throw IOException with descriptive message if directory creation fails.

}

final String filePath = outputDirectory + File.separator + fileName;
Files.write(Paths.get(filePath), jsonContent.getBytes(StandardCharsets.UTF_8));
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The writeJsonToFile method is vulnerable to path traversal attacks. An attacker could provide a malicious fileName containing path traversal sequences (e.g., ../../etc/passwd) to write files outside the intended directory. Validate and sanitize the fileName parameter to ensure it doesn't contain directory separators or path traversal sequences:

// Validate fileName doesn't contain path separators
if (fileName.contains("/") || fileName.contains("\\") || fileName.contains("..")) {
    throw new IllegalArgumentException("Invalid file name: must not contain path separators or traversal sequences");
}

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5fc4262. Added validation to reject file names containing /, \, or .. to prevent path traversal attacks. Throws IllegalArgumentException with descriptive message.

}
Comment on lines 181 to 202
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The writeJsonToFile method is not covered by unit tests. Add a test case to verify file creation, directory creation, and content writing behavior. Consider testing both scenarios: when the directory exists and when it needs to be created.

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5fc4262. Added 3 new tests for writeJsonToFile: testWriteJsonToFile (basic functionality), testWriteJsonToFileCreatesDirectory (directory creation), and testWriteJsonToFileRejectsPathTraversal (security validation).

}
Loading
Loading