add cloudflare cache purge #638

andre-urbani · 2025-11-27T11:11:25Z

What

Add functionality to purge cloudflare cache on publication of files (via PUT /state endpoint)
Jira - https://officefornationalstatistics.atlassian.net/browse/DIS-4109

How to review

Run the dataset-catalogue stack and make sure you have the latest version which includes the cloud-flare-stub

Make sure you have these env vars set locally -

ENABLE_CLOUDFLARE_SDK=false
CLOUDFLARE_ZONE_ID="9f1eec58caedd8e902395a065c120073"
CLOUDFLARE_API_TOKEN="test-token"
CLOUDFLARE_API_URL="http://cloud-flare-stub:22500"

Create a new dataset and version, and make sure when creating the version that there is a file in the local db that matches the file in the distributions array
Use the PUT /state endpoint for the version to transition through the states until the published state
After transitioning to the published state, check the docker logs for dp-dataset-api and cloud-flare-stub and confirm that the cache purge has run successfully. The dataset logs should look something like -

{
  "event": "purging cloudflare cache",
  "data": {
    "dataset_id": "static-test-dataset",
    "edition": "time-series",
    "prefixes": [
      "www.ons.gov.uk/datasets/static-test-dataset",
      "www.ons.gov.uk/datasets/static-test-dataset/editions",
      "www.ons.gov.uk/datasets/static-test-dataset/editions/time-series/versions",
      "api.beta.ons.gov.uk/v1/datasets/static-test-dataset",
      "api.beta.ons.gov.uk/v1/datasets/static-test-dataset/editions",
      "api.beta.ons.gov.uk/v1/datasets/static-test-dataset/editions/time-series/versions"
    ]
  }
}
{
  "event": "successfully triggered cloudflare cache purge",
  "data": {
    "dataset_id": "static-test-dataset",
    "edition": "time-series",
    "version": "1"
  }
}

You can also test that the real Cloudlfare API is being hit by setting ENABLE_CLOUDFLARE_SDK=true, following the same steps as above, and then you should get an error in the logs. Something like this -

{
  "event": "failed to purge cloudflare cache",
  "errors": [
    {
      "message": "failed to purge cache: Authentication error (10000)"
    }
  ]
}

Which is just a result of the api token being incorrect (this will be tested in sandbox with a real api token)

Confirm changes make sense, tests pass

Who can review

Anyone

aryan-p03

Some comments about the client setup, logging and keeping dataset-api logic separate from the cloudflare package.
Nice to see testcontainers being used for unit tests

aryan-p03 · 2025-12-11T10:48:44Z

service/service.go

 	return err
 }

+func (svc *Service) initCloudflareClient(ctx context.Context) error {


I think some of the checks in this function are repeated in initialise.go

aryan-p03 · 2025-12-11T10:48:50Z

service/service.go

+	if err := svc.initCloudflareClient(ctx); err != nil {
+		log.Error(ctx, "failed to initialise cloudflare client, continuing without cache purging", err)
+	}


This should return an error, if the client fails to setup then the service shouldn't start up

aryan-p03 · 2025-12-11T10:48:54Z

service/initialise.go

+func (e *Init) DoGetCloudflareClient(ctx context.Context, cfg *config.Configuration) (cloudflare.Clienter, error) {
+	if cfg.CloudflareAPIToken == "" || cfg.CloudflareZoneID == "" {
+		log.Info(ctx, "cloudflare integration disabled: missing API token or zone ID")
+		return nil, nil
+	}
+
+	var client cloudflare.Clienter
+	var err error
+
+	// for local mock server when SDK is disabled
+	if !cfg.EnableCloudflareSDK && cfg.CloudflareAPIURL != "" {
+		client, err = cloudflare.New(
+			cfg.CloudflareAPIToken,
+			cfg.CloudflareZoneID,
+			cfg.EnableCloudflareSDK,
+			cfg.CloudflareAPIURL,
+		)
+	} else {
+		client, err = cloudflare.New(
+			cfg.CloudflareAPIToken,
+			cfg.CloudflareZoneID,
+			cfg.EnableCloudflareSDK,
+		)
+	}
+
+	if err != nil {
+		return nil, err
+	}
+
+	return client, nil
+}


I believe the setup within the cloudflare package will bring up the errors with missing configs so we don't need to check that here. We just need to check if cloudflare is enabled here

aryan-p03 · 2025-12-11T10:49:05Z

features/steps/steps.go

With the comment about the mocks on component test setup, we would be able to have a simple step saying something like:
the following URL's should have been purged: followed by a list of the URL's we expect to be purged

aryan-p03 · 2025-12-11T10:49:12Z

features/steps/dataset_component.go

For the component test setup, can the cloudflare package have mocks generated that can be used here instead of making the mocks here?

aryan-p03 · 2025-12-11T10:49:20Z

cloudflare/client.go

+func buildPrefixes(datasetID, editionID string) []string {
+	return []string{
+		fmt.Sprintf("www.ons.gov.uk/datasets/%s", datasetID),
+		fmt.Sprintf("www.ons.gov.uk/datasets/%s/editions", datasetID),
+		fmt.Sprintf("www.ons.gov.uk/datasets/%s/editions/%s/versions", datasetID, editionID),
+		fmt.Sprintf("api.beta.ons.gov.uk/v1/datasets/%s", datasetID),
+		fmt.Sprintf("api.beta.ons.gov.uk/v1/datasets/%s/editions", datasetID),
+		fmt.Sprintf("api.beta.ons.gov.uk/v1/datasets/%s/editions/%s/versions", datasetID, editionID),
+	}


Similar to the previous comment, this function should be in the dataset-api and not the SDK package. I would also avoid hard-coded domain names and instead build off config values

aryan-p03 · 2025-12-11T10:49:25Z

cloudflare/client.go

+// PurgeCacheByPrefix purges the Cloudflare cache for dataset-related URLs
+func (c *Client) PurgeCacheByPrefix(ctx context.Context, datasetID, editionID string) error {


I think this function shouldn't be in the SDK as it is specific to the dataset-api. The SDK should just accept a list of prefixes which it then purges (so any service can use the function).
The dataset-api should be responsible for building the prefixes and then this list will be passed to the SDK

aryan-p03 · 2025-12-11T10:49:31Z

api/versions.go

+		if api.cloudflareClient != nil {
+			if err := api.cloudflareClient.PurgeCacheByPrefix(ctx, datasetID, edition); err != nil {
+				log.Error(ctx, "failed to purge cloudflare cache", err, logData)
+			} else {
+				log.Info(ctx, "successfully triggered cloudflare cache purge", logData)
+			}
+		}


If the client is nil here then there is no trace and cache will not be cleared which is a big problem for support. Ideally the client should be confirmed as not nil at setup so we don't have the nil check here.
If possible, it would be nice to have the list of URL's purged in the log.Info so we check them in environment testing.
I see that we only have the error log when purge fails as we don't want to block publishing. Might be worth asking Fran if this is ok since the purge can fail but the dataset will be published

aryan-p03 · 2025-12-11T10:49:35Z

api/versions.go

+	_, err = api.smDatasetAPI.AmendVersion(r.Context(), vars, versionUpdate)
 	if err != nil {
 		handleVersionAPIErr(ctx, err, w, logData)
 		return
 	}

-	if stateUpdate.State == models.PublishedState && updatedVersion.Distributions != nil && len(*updatedVersion.Distributions) > 0 {
-		err = api.publishDistributionFiles(ctx, updatedVersion, logData)
+	if stateUpdate.State == models.PublishedState && currentVersion.Distributions != nil && len(*currentVersion.Distributions) > 0 {
+		err = api.publishDistributionFiles(ctx, currentVersion, logData)


I'm assuming the change to _ instead of updatedVersion is due to the linter which looks ok.
However currentVersion is being used on lines 839 and 840 where it was previously updatedVersion. I think it should remain as updatedVersion unless there's an issue I'm not seeing here?

andre-urbani added 2 commits November 27, 2025 10:25

add cloudflare cache purge

9e5778c

update config test

69ef42f

andre-urbani requested a review from a team as a code owner November 27, 2025 11:11

andre-urbani added 13 commits November 27, 2025 11:24

fix conflicts

54acc74

fix lint

75dec89

Merge branch 'develop' into feature/add-cloudflare-cache-purge

423ad77

Merge branch 'develop' into feature/add-cloudflare-cache-purge

db43cee

update to use cloudflare sdk

f130695

Merge branch 'develop' into feature/add-cloudflare-cache-purge

804441b

fix audit failure

d7e5db6

fix audit failure

9d92ddc

update yml file to relative path

2fa320b

update go.mod to fix mongo unit test issues

b8881a1

update to use mongo testcontainers

7e884ed

update unit.yml file

bc7951f

Merge branch 'develop' into feature/add-cloudflare-cache-purge

945878a

aryan-p03 requested changes Dec 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add cloudflare cache purge #638

add cloudflare cache purge #638

Uh oh!

andre-urbani commented Nov 27, 2025 •

edited

Loading

Uh oh!

aryan-p03 left a comment

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

aryan-p03 Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// PurgeCacheByPrefix purges the Cloudflare cache for dataset-related URLs
		func (c *Client) PurgeCacheByPrefix(ctx context.Context, datasetID, editionID string) error {

add cloudflare cache purge #638

Are you sure you want to change the base?

add cloudflare cache purge #638

Uh oh!

Conversation

andre-urbani commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How to review

Who can review

Uh oh!

aryan-p03 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andre-urbani commented Nov 27, 2025 •

edited

Loading