Common Charts - Jenkins Job Provisioning System

This repository provides an automated Jenkins job provisioning system using the Jenkins Job DSL plugin and seed job pattern. It automatically creates and manages CI/CD pipelines for multiple services with standardized configurations and Bitbucket Cloud integration.

Overview

The system uses a seed job approach to automatically discover services and create their corresponding Jenkins multibranch pipeline jobs. Each service defines its configuration in YAML format, and the system generates the appropriate Jenkins jobs with:

• Bitbucket Cloud integration with proper webhook triggers
• Automated branch and PR discovery
• Standardized folder structure based on organizational tags
• Built-in validation and error handling
• One-time provisioning - jobs are created once and not overwritten

Architecture

common-charts/
├── jenkins/seed/              # Seed job configuration
│   ├── SeedJenkinsfile       # Main pipeline script
│   ├── seed.groovy           # DSL script for seed job creation
│   ├── service_provisioner.groovy  # Service provisioning logic
│   ├── yamlToMap.groovy      # YAML parsing utility
│   ├── libs/                 # Helper libraries
│   └── schema/               # JSON schema for validation
├── services/                 # Service definitions
│   └── {service-name}/
│       ├── jenkins/
│       │   └── job.yaml      # Service configuration (YAML only)
│       └── envs/             # Environment configs
├── ci/                       # CI utilities
│   └── validate-job-yaml.sh  # Configuration validation
└── util/                     # Shared utilities
    └── dsl-helpers.groovy    # Reusable DSL functions

Key Components

• service_provisioner.groovy: Main provisioning logic using native Git DSL
• Bitbucket Cloud Integration: Direct Git URL integration with webhook support
• YAML-only Configuration: Simplified service configuration without Groovy scripts
• Automated Discovery: Scans services/ directory for job configurations
• Folder Structure: Department/Team/Division based on service tags

Quick Start

1. Initial Setup

1. Create the seed job in Jenkins:

   # Execute the seed DSL script in Jenkins
   # This creates the main seed job at Config/seed-job

2. Run the seed job to provision all service jobs automatically

2. Adding a New Service

1. Create service directory:

   mkdir -p services/my-new-service/jenkins
   mkdir -p services/my-new-service/envs/{dev,uat,prod}

2. Create job configuration (services/my-new-service/jenkins/job.yaml):

   service: my-new-service
   project: common-charts             # Bitbucket workspace name  
   repo: tm-prod/my-service          # Bitbucket repo: workspace/repository
   credentialsId: tmbuildjenkins-bitbucket
   jenkinsfile: Jenkinsfile          # Path to Jenkinsfile in your repo
   tags:
     department: Backend             # Used for folder structure
     team: TFBO                     # Department/Team/Division
     division: modern-microservices
   retention:
     daysToKeep: 14
     numToKeep: 50

3. That's it! No Groovy scripts needed - the provisioner handles everything automatically.

4. Trigger the seed job - it will automatically discover and provision your new service

Configuration Reference

Required Fields

• service: Service name (must match directory name)
• repo: Repository path in format workspace/repository (e.g., tm-prod/my-service)
• credentialsId: Jenkins credentials ID for Bitbucket access

Optional Fields

• project: Bitbucket project key (informational)
• jenkinsfile: Path to Jenkinsfile (default: "Jenkinsfile")
• tags: Organizational structure for folder creation
  • department: Department name (e.g., "Backend", "Frontend")
  • team: Team name (e.g., "TFBO", "Platform")
  • division: Division name (e.g., "modern-microservices", "legacy")
• retention: Build retention policies
  • daysToKeep: Days to keep builds (default: 14)
  • numToKeep: Number of builds to keep (default: 50)

Generated Features

• Branch Discovery: All branches automatically discovered
• PR Discovery: Origin and fork PRs with "both revisions" strategy
• Fork Trust: Set to "Nobody" for security
• Webhooks: Push and PR update triggers configured
• Periodic Scan: 30-minute backup scanning

See jenkins/seed/schema/job.schema.json for complete schema definition.

Validation

Validate all service configurations:

./ci/validate-job-yaml.sh

This script checks:

• YAML syntax
• Required fields presence
• Service name consistency
• Basic Groovy syntax
• JSON schema compliance (if jq available)

Development

Prerequisites

• Jenkins with Job DSL plugin
• Bitbucket Server/Data Center integration
• Groovy runtime for local testing
• Python 3 (for YAML validation)
• jq (optional, for JSON schema validation)

Testing Changes

1. Validate configurations: ./ci/validate-job-yaml.sh
2. Test in development Jenkins instance
3. Review generated jobs before promoting to production

Helper Functions

The util/dsl-helpers.groovy provides reusable DSL functions:

• ensureFolder(path): Creates nested folders with RBAC
• branchDiscovery(mode): Configures branch discovery
• originPRDiscovery(enabled): Sets up origin PR discovery
• forkPRDiscovery(enabled, trust): Configures fork PR handling

Monitoring

• Seed Job: Monitor Config/seed-job for provisioning status
• Service Jobs: Check individual service pipelines
• Validation: Review validation logs in seed job builds

How It Works

Service Discovery Process

1. Scan Services Directory: Looks for all directories under services/
2. Validate Configuration: Checks for jenkins/job.yaml and validates structure
3. Parse Repository Info: Extracts Bitbucket workspace and repository names
4. Create Folder Structure: Uses organizational tags to create Department/Team/Division folders
5. Provision Job: Creates multibranch pipeline with Bitbucket Git integration
6. Configure Webhooks: Sets up push and PR triggers automatically

One-Time Provisioning

• Jobs are created only if they don't already exist
• Existing jobs are not updated to prevent overwriting manual changes
• To update an existing job, delete it first and re-run the seed job

Troubleshooting

Common Issues

1. Service not discovered: Check directory structure and YAML syntax
2. Repository access failed: Verify Bitbucket credentials and repository permissions
3. Webhook issues: Ensure Jenkins is accessible from Bitbucket Cloud
4. Folder creation failed: Check Jenkins permissions and folder naming

Debugging Steps

1. Check seed job console output for detailed error messages
2. Validate YAML syntax using the validation script
3. Verify repository URL format: Must be workspace/repository
4. Test credentials by accessing repository manually with same credentials

Contributing

1. Follow the established service structure
2. Validate configurations before committing
3. Test changes in development environment
4. Update documentation for new features

Support

For questions and support:

• Check existing service configurations for examples
• Review Jenkins Job DSL documentation
• Contact the DevOps team for Jenkins-specific issues