Diego v0.1460.0

Changes from v0.1459.0 to v0.1460.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v38.

Significant changes

Component Logging and Metrics

• pivotal-golang/lager #17: Debug logging does JSON marshaling regardless of whether debug logging is enabled
• cloudfoundry-incubator/cf-debug-server #3: Enable pprof block profiling (parameterized)

Documentation

• Document the Tasks API (public and internal) and Actions on the BBS

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1459.0

Changes from v0.1458.0 to v0.1459.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v37.

Significant changes

NOTE: Although Diego v0.1455.0 re-introduced the feature that SSH sessions and commands in a CF app container have the same environment variables set as the app's main process, we have rolled that functionality back in Diego v0.1459.0. For buildpack-based apps, the mechanism by which this feature was implemented would execute the buildpack-installed .profile.d scripts twice. While those scripts often only set environment variables for the app process, in general they can have arbitrary effects, and may not be safe to call repeatedly and concurrently. The team is considering safer ways to share that environment, but for now we will leave it unimplemented.

To support SSH users who wish to have the same environment as their start commands, we have submitted documentation of the manual steps to set up that environment to the developer-guide Cloud Foundry documentation repo. Those changes will appear on the "Accessing Apps with SSH" page shortly.

Custom CAs

• cflinuxfs2 rootfs pre-start script should retry certificate installation

SSH

• Roll back SSH environment setup changes (again)
• cloudfoundry-incubator/diego-ssh #21: add flags to ssh-proxy for specifying allowed cipher,mac,kex algorithms

CF Tasks

• As a space developer, I expect the 'source' for task logging to be [App/TASK/TASK_NAME]

Component Logging and Metrics

• log cell id when cell is missing

Documentation

• Document the Tasks API (public and internal) and Actions on the BBS

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1458.0

Changes from v0.1457.0 to v0.1458.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v37.

Significant changes

BBS Benchmarks

• Spike: evaluate the performance of the relational-db-backed BBS against an HA deployment of cf-mysql

Container Execution

• cloudfoundry-incubator/stager #8: Droplets created with Diego don't work on DEA affects 0.1454.0

SSH

• Flaky SSH proxy tests in inigo and units

Draining

• As a Diego operator, I expect the rep drain script to be idempotent

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1457.0

Changes from v0.1456.0 to v0.1457.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v36.

Significant changes

Dependencies

• Upgrade cflinuxfs2 rootfs in diego-release to 1.42.0+

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1455.0

Changes from v0.1454.0 to v0.1455.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v36.

Significant changes

This version of diego-release completely removes the Diego Acceptance Test suite, or DATs. This test suite had already been largely consolidated with the CF Acceptance Tests, or CATs. CF v231 contains a version of the CATs that can be targeted at either the Diego backend or the DEAs, and which includes all the remaining tests of Diego-specific functionality from the DATs. The CATs are now the only supported mechanism for verifying the CF-level functionality of a CF deployment backed by Diego. Please consult the CATs README for more details on running it with the appropriate configuration.

The Diego Smoke Tests are also redundant with the CF Smoke Tests, and will similarly be removed in favor of the CF tests in an upcoming release. If you are using the Diego Smoke Tests to monitor a CF deployment backed by Diego, please switch to using the CF Smoke Tests as soon as possible.

BBS Benchmarks

• As a Diego developer, I expect the Diego BBS-benchmark test suite to include per-record reads and writes in the rep bulk loop
• LRPConvergence, route-emitter, and Nysnc bulk loops should operate on their default intervals in the benchmark-bbs suite

Performance

• Spike: evaluate the performance of the relational-db-backed BBS against an HA deployment of cf-mysql (in flight)

Custom CAs

• As a CF user, I expect trusted certificates to be available to cflinuxfs2- and Docker-image app instances in a conventional location

SSH

• Bump diego-ssh in windows_app_lifecycle
• As a CF user, I expect that an SSH session into a CF instance should have same environment that the app process does (Take 2)
• cloudfoundry-incubator/diego-ssh #20: Use updated sftp Server constructor
• SSH-Proxy authenticators should check more strictly for guid fields in user names
• Fix data race in diego-ssh ssh-proxy.
• cloudfoundry-incubator/diego-ssh #18: add flags for specifying allowed cipher,mac,kex algorithms

CF Tasks

• As a space developer, I expect the 'source' for task logging to be [App/TASK/TASK_NAME]
• When CC misses a task callback when Diego completes a task, the task result should eventually be updated.
• As a space developer, I can cancel a task
• When Diego misses a task cancel when CC deletes or cancels task, send another cancel to Diego

Routing

• Diego-release route_emitter ctl script should properly encode NATS password
• cloudfoundry-incubator/nsync #6: Dont populate cf-routes if CC does not send routes
• operator should be able to discover whether the number of events sent by route-emitter is different from the number received by gorouter (in flight)

Windows Support

• System log errors due to missing IIS log directory

Manifest Generation

• Add new graph_cleanup_threshold_in_mb in manifest generation for cf-linux-release v334

Dependencies

• Upgrade cflinuxfs2 rootfs in diego-release to 1.37.0+

Test Suites and Tooling

• Deprecate the DATs and remove from diego-release

BOSH job changes

• Removed the acceptance-tests job.

BOSH property changes

• Removed all BOSH properties under acceptance_tests.
• Added benchmark-bbs.percent_writes property: Percentage of actual LRPs to write on each rep bulk loop in the BBS benchmarks.
• Added diego.rep.trusted_certs property: Bundle of trusted certificates to be made available to all containers on the cell.

Diego v0.1454.0

Changes from v0.1453.0 to v0.1454.0

• Depends on garden-linux-release v0.333.0.
• Depends on etcd-release v36.

Significant changes

Operators can now install a set of trusted system certificates in the default /etc/ssl/certs trust store of the cflinuxfs2 rootfs. This is particularly useful if your cflinuxfs2-based instances communicate with external services signed by a custom CA, in which case you can use this feature to install that CA certificate in all the instances.

To install the certificates, supply the contents of the PEM-encoded certificates in the diego.rootfs_cflinuxfs2.trusted_certs property in the Diego deployment manifest. As with other PEM-encoded manifest data, you may wish to use the YAML | block-literal syntax to specify the property, as follows:

properties:
  diego:
    rootfs_cflinuxfs2:
      trusted_certs: |
        -----BEGIN CERTIFICATE-----
        (cert number 1 data)
        -----END CERTIFICATE-----
        # comments outside the PEM boundaries will be ignored
        -----BEGIN CERTIFICATE-----
        (cert number 2 data)
        -----END CERTIFICATE-----

If you are using the spiff-based manifest-generation scripts, this property can also be specified in the property-overrides stub.

Custom CAs

• As a CF user, I expect all Tasks and LRPs using the cflinuxfs2 rootfs to contain trusted certificates in their default trust store
• As a CF user, I expect trusted certificates to be available to cflinuxfs2- and Docker-image app instances in a conventional location (in flight)

SSH

• investigate whether we can use CF_INSTANCE_PORTS instead of rewriting diego-sshd port arguments
• cloudfoundry-incubator/diego-ssh #17: Rewrite listen address using CF_INSTANCE_PORTS on Windows

Routing

• The route-emitter should log and emit metrics if it detects two distinct instances with the same address in its in-memory routing table
• cloudfoundry-incubator/nsync #6: Dont populate cf-routes if CC does not send routes
• developer should be able to verify that when deleting an app or route, it's route_mappings are also deleted (in flight)

Manifest Generation

• As a CF+Diego operator, I would like to use the Diego team's tooling to deploy CF and Diego to an AWS environment
• As a Diego operator, I would like a script to generate a vizzini errand manifest for my Diego deployment

Dependencies

• Upgrade cflinuxfs2 rootfs in diego-release to 1.35.0+ 1.36.0+

Test Suites and Tooling

• The dusts should work against the latest bosh-lite box.
• cloudfoundry-incubator/diego-release #135: 132 - Verify this repos isn't a submodule before invoking install-git-hooks

Documentation

• Document the Tasks API (public and internal) and Actions on the BBS (in flight)

Cleanup

• cloudfoundry-incubator/cf_http #2: Common response generation methods were present throughout Diego.

BOSH job changes

None.

BOSH property changes

• Added diego.rootfs_cflinuxfs2.trusted_certs: Bundle of certificates to install in the cflinuxfs2 rootfs default trust store (/etc/ssl/certs).

Diego v0.1453.0

Changes from v0.1452.0 to v0.1453.0

• Depends on garden-linux-release v0.332.0.
• Depends on etcd-release v33.

Note that due to breaking changes in manifest-generation introduced in cf-release, the manifest-generation scripts in this version of diego-release are compatible only with cf-release@0665d0e or later. Those scripts are therefore incompatible with CF v230 or earlier, but will be compatible with CF v231 when it is released.

Significant changes

BBS Benchmarks

• As a Diego developer, I would like the BBS benchmark suite to run periodic rep bulk retrieval loops

Manifest Generation

• cloudfoundry-incubator/diego-release #128: Update loggregator and metron properties to match latest loggregator
• Change bosh property names for loggregator TLS certs to "standard format"

Component Logging and Metrics

• The route-emitter should log and emit metrics if it detects two distinct instances with the same address in its in-memory routing table

Test Suites and Tooling

• As a Diego operator, I would like a script to generate a vizzini errand manifest for my Diego deployment (in flight)
• Investigate recent flakiness in unit tests
• cloudfoundry-incubator/diego-release #134: Fixed broken link

Cleanup

• cloudfoundry-incubator/diego-release #131: remove stale entries in .gitmodules

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1452.0

Changes from v0.1451.0 to v0.1452.0

• Depends on garden-linux-release v0.332.0.
• Depends on etcd-release v28.

Significant changes

NOTE: If you are not using the spiff-based manifest-generation tooling to generate your Diego deployment manifest, please be aware that some of the per-job consul agent configuration has changed with this version of the release. Diego components now set up their DNS health checks in consul themselves via the locket library, instead of configuring the consul agent to do so. This means that most jobs will have removed the consul.agent.services property entirely. Jobs running etcd will keep only that 'etcd' service in the property, as in the following representative manifest snippet:

jobs:
- name: database_z1
  templates:
  - name: etcd
    release: etcd
  - name: bbs
    release: diego
  - name: consul_agent
    release: cf
  - name: metron_agent
    release: cf
  properties:
    consul:
      agent:
        services:
          etcd: {}

See the changes to the manifest-generation/diego.yml file introduced in diego-release commit 680aa9b for more detail.

BBS Benchmarks

• As a Diego developer, I would like the BBS benchmark suite to run periodic rep bulk retrieval loops

Manifest Generation

• cloudfoundry-incubator/diego-release #127: Minor manifest cleanups

Component Logging and Metrics

• Eliminate critical health-check noise from inactive HA components registered with consul DNS

Security

• Investigate whether Diego components are correctly configured to use the default VM cert store, so that they will correctly trust BOSH-provisioned CAs

BOSH job changes

The dns_health_check.erb script template has been removed from the following job templates:

• auctioneer
• bbs
• cc_uploader
• file_server
• nsync
• ssh_proxy
• stager
• tps

BOSH property changes

None.

Diego v0.1451.0

Changes from v0.1450.0 to v0.1451.0

• Depends on garden-linux-release v0.330.0.
• Depends on etcd-release v27.

Significant changes

BBS Benchmarks

• As a Diego developer, I would like the BBS benchmark suite to run periodic rep bulk retrieval loops (in flight)

Performance

• Optimize gzip performance on cachedownloader

Container Placement

• As a Diego operator, I would like the auction to be able to account for container-start load when it makes its placement decisions

CF Task Support

• As a space developer, I can create a task that runs on Diego (in flight)

Component Logging and Metrics

• The rep, the BBS, and the route-emitter should log addresses and port-mappings when they start ActualLRPs or add them to the route table

Dependencies

• Upgrade cflinuxfs2 rootfs in diego-release to 1.32.0+

Test Suites and Tooling

• Turn off reuse_compilation_vms for the DUSTs

BOSH job changes

None.

BOSH property changes

• Add diego.auctioneer.starting_container_weight: Factor by which to weight in-flight container starts in the auction scoring.

Diego v0.1450.0

Changes from v0.1448.0 to v0.1450.0

• Depends on garden-linux-release v0.331.0.
• Depends on etcd-release v27.

Significant changes

BBS Benchmarks

• As a Diego developer, I would like the BBS benchmark suite to run periodic rep bulk retrieval loops
• LRP convergence schedules redundant etcd delete requests

SSH

• Diego ssh-proxy should take separate client name and secret command-line args for its UAA client

Component Logging

• Improve logging in nsync-listener handlers

Dependencies

• Upgrade cflinuxfs2 rootfs to 1.31.0+
• Upgrade golang in diego-release to Go 1.5.3+ (shelved)
• Diego-release uses only Golang 1.4.3

BOSH job changes

None.

BOSH property changes

None.