Testomat.io one week release cycle

[poliarush]

We come to new release cycle to speed up development and improve customer support by delivering more feature and defect in short period of time. New release cycle should be 1 week. How we will do it?

1. Introduction

   • Problem: customer support has a lot of defects for months, technical debt
   • Problem: users wait to long for defect and features
   • Problem: long time to wait = increased chrun rate of paid clients
   • Goal: short release cycle to 1 week
   • Goal: release no matter what, even 2 defects if it's finished
   • Goal: introduce gates to test thoroughly, document on fly and marketize release items on weekly basis
   • Goal: increase tools popularisation by frequent changes
   • Goal: introduce automation on all necessary parts (QA, docs, deployment, etc) to support such speed

2. Development Phases

   • Cycle: Monday - planning, Monday - Thursday dev / test / document, Friday release
   • Requirements: gathered automatically to github issues
   • Requirements: all tasks should be in form of github issues with proper priorities
   • Planning: Monday is for planning session
   • Planning: 1 developer limits with 1 feature and 2 defects maximum
   • Design: upfront changes should be prepared
   • Testing: culture of whole-team testing where each team member takes responsibility for testing
   • Testing: developer check locally, push and check on testing env
   • Testing: QA make final check, communicate defects, and write changelog immediately
   • Testing: test automation should be introduced and increase coverage
   • Deployment: quick and ready features go to beta, long and iterative fixes to UAT env
   • Documents: once feature / defects is tested and ready it should be documented on the fly
   • Documents: definition of done should include docs preparation and changes
   • Documents: changelog should be updated once feature / defects is tested and ready

3. Release Planning

   • Scope: driven by user's urgency (urgent, blocker) and innovation / marketing necessity
   • Priorities: scope item should plan item in following order: blocker, urgent, high, medium, low then other
   • Release timeline: blocker and urgent to be released in 1 week, other can wait longer
   • Release timeline: should be visible and clear for client
   • Release timeline: marketing team should be included in such conversation

4. Versioning

   • to be added

5. Continuous Integration and Deployment

   • Principles: Integration and deployment practices for frequent code changes should be documented. Setting up automated processes to merge code changes from multiple developers, ensuring compatibility and minimizing conflicts.
   • Design: upfront
   • Code: with test automation
   • Collaboration: frequent discussion with call by necessity, 1 min call is better then 100 lines in chat
   • Unit tests: for all features
   • Test automation: basis and critical UI acceptance flows
   • Monitor: env after deployment with automatic alerts

6. Quality Assurance (QA)

   • Testing processes: unit, integration, regression, UAT, automated testing
   • Unit level: increase coverage
   • Smoke level: smoke quick on daily basis depending on fixes
   • Regression: once big-impact feature introduced
   • UAT: whole-testing team session to see if everything is fine
   • Automated testing: unit tests, front-end tests, critical business flows, check-tests and reporters features per each framework
   • Pipelines: include test automation into front-end / check-tests / reporter development

7. Beta Testing and UAT

   • Beta for quick and ready fixes to be deployed to production during a week
   • UAT for long-developed feature / fixes
   • Users can be invited to beta or UAT to test features

8. Change Management

   • Change Control Boards: committees consisting of stakeholders from different teams. These boards are responsible for reviewing and approving proposed changes, ensuring they align with business objectives and minimize potential risks.
   • Responsibilities: CTO for design and architecture, QA for feature readiness, Product Owner for business objective in development tasks
   • Impact Assessment: CTO, QA, PO should do impact analysis for each item in release to ensure it can be delivered during a week
   • Risk Management: mitigation strategies should be discussed during a week to minimize any negative impacts and ensure a smooth transition to production.
   • Communication and Documentation: Document all changes, notify clients, update websites

9. Release Communication

   • Notifying users: by slack, by email, by crisp, by slack
   • Develop notification strategies
   • Release notes/documentation: nolt, changelog
   • Support channels: slack, crisp

10. Rollout Strategy

    • Quick fixes to Beta
    • Long development to UAT
    • Push from Beta to Prod each Friday
    • Merge Beta to UAT once release to production
    • Merge UAT to Beta once feature is ready

11. Monitoring and Feedback

    • Post-release monitoring: Saturday and Sunday for monitoring thru testing, appsignal and automatic alerts
    • Post-release monitoring: quick fixes to Prod by Monday
    • User feedback collection: crisp, slack
    • Bug tracking: automatic tool to create a list of changes from github milestones
    • Performance monitoring: appsignal

12. Continuous Improvement

    • Each Friday discuss what to improve to release smoothly each week
    • Implement process improvement quickly

[poliarush]

@DavertMik point to discuss, about whole-team responsibility,
we need to setup some general simplified dashboard for team to monitor production metrics by whole team on daily basis to quickly react on production changes

Like:

• CPU, Disk Space, Memory
• Load, thruoutput (DB, redis, meilisearch, clickhouse)
• Exceptions, Errors, Failed Jobs
• User registrations, email confirmations, email sent, payments exceptions, etc
• Project created, tests created, run created
• Issues, Defects, Conversations

Get different sources and may be build google data studio dashboard quickly,
but generally it should be 1 page to display all metrics at once so team check it quickly on daily basis