Skip to content

Commit

Permalink
Update docs to prioritize workers over agents (#10904)
Browse files Browse the repository at this point in the history
Co-authored-by: Serina Grill <[email protected]>
  • Loading branch information
discdiver and serinamarie authored Oct 18, 2023
1 parent 6ba2ea8 commit d6b4f37
Show file tree
Hide file tree
Showing 15 changed files with 125 additions and 122 deletions.
29 changes: 14 additions & 15 deletions docs/cloud/connecting.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ tags:
- Prefect Cloud
- API keys
- configuration
- agents
- workers
- troubleshooting
- connecting
Expand All @@ -16,10 +15,10 @@ search:

To create flow runs in a local or remote execution environment and use either Prefect Cloud or a Prefect server as the backend API server, you need to

- Configure the execution environment with the location of the API.
- Configure the execution environment with the location of the API.
- Authenticate with the API, either by logging in or providing a valid API key (Prefect Cloud only).

## Log into Prefect Cloud from a terminal
## Log into Prefect Cloud from a terminal

Configure a local execution environment to use Prefect Cloud as the API server for flow runs. In other words, "log in" to Prefect Cloud from a local environment where you want to run a flow.

Expand Down Expand Up @@ -112,9 +111,9 @@ This section provides tips that may be helpful if you run into problems using Pr

## Prefect Cloud and proxies

Proxies intermediate network requests between a server and a client.
Proxies intermediate network requests between a server and a client.

To communicate with Prefect Cloud, the Prefect client library makes HTTPS requests. These requests are made using the [`httpx`](https://www.python-httpx.org/) Python library. `httpx` respects accepted proxy environment variables, so the Prefect client is able to communicate through proxies.
To communicate with Prefect Cloud, the Prefect client library makes HTTPS requests. These requests are made using the [`httpx`](https://www.python-httpx.org/) Python library. `httpx` respects accepted proxy environment variables, so the Prefect client is able to communicate through proxies.

To enable communication via proxies, simply set the `HTTPS_PROXY` and `SSL_CERT_FILE` environment variables as appropriate in your execution environment and things should “just work.”

Expand Down Expand Up @@ -167,28 +166,28 @@ You can also check that the account and workspace IDs specified in the URL for `

If you're having difficulty logging in to Prefect Cloud, the following troubleshooting steps may resolve the issue, or will provide more information when sharing your case to the support channel.

- Are you logging into Prefect Cloud 2? Prefect Cloud 1 and Prefect Cloud 2 use separate accounts. Make sure to use the right Prefect Cloud 2 URL: https://app.prefect.cloud/
- Are you logging into Prefect Cloud 2? Prefect Cloud 1 and Prefect Cloud 2 use separate accounts. Make sure to use the right Prefect Cloud 2 URL: <https://app.prefect.cloud/>
- Do you already have a Prefect Cloud account? If you’re having difficulty accepting an invitation, try creating an account first using the email associated with the invitation, then accept the invitation.
- Are you using a single sign-on (SSO) provider (Google or Microsoft) or just using a username and password login?
- Are you using a single sign-on (SSO) provider (Google or Microsoft) or just using a username and password login?
- Did you utilize the “having trouble/forgot password” link on the login page? If so, did you receive the password reset email? Occasionally the password reset email can get filtered into your spam folder.

Other tips to help with login difficulties:

- Hard refresh your browser with Cmd+Shift+R.
- Try in a different browser. We actively test against the following browsers:
- Chrome
- Edge
- Firefox
- Safari
- Chrome
- Edge
- Firefox
- Safari
- Clear recent browser history/cookies

In some cases, logging in to Prefect Cloud results in a "404 Page Not Found" page or fails with the error: "Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of “text/html”. Strict MIME type checking is enforced for module scripts per HTML spec."

This error may be caused by a bad service worker.
This error may be caused by a bad service worker.

To resolve the problem, we recommend unregistering service workers.
To resolve the problem, we recommend unregistering service workers.

In your browser, start by opening the developer console.
In your browser, start by opening the developer console.

- In Chrome: **View > Developer > Developer Tools**
- In Firefox: **Tools > Browser Tools > Web Developer Tools**
Expand All @@ -204,4 +203,4 @@ See the [Login to Prefect Cloud fails...](https://discourse.prefect.io/t/login-t

None of this worked?

Email us at [email protected] and provide answers to the questions above in your email to make it faster to troubleshoot and unblock you. Make sure you add the email address with which you were trying to log in, your Prefect Cloud account name, and, if applicable, the organization to which it belongs.
Email us at <[email protected]> and provide answers to the questions above in your email to make it faster to troubleshoot and unblock you. Make sure you add the email address with which you were trying to log in, your Prefect Cloud account name, and, if applicable, the organization to which it belongs.
4 changes: 2 additions & 2 deletions docs/cloud/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Prefect Cloud includes all the features in the open-source Prefect server plus t
- [Automations](/cloud/automations/) &mdash; configure triggers, actions, and notifications in response to real-time monitoring events.
- [Email notifications](/cloud/automations/) &mdash; send email alerts from Prefect's server based on automation triggers.
- [Organizations](/cloud/organizations/) &mdash; user and workspace management features that enable collaboration for larger teams.
- [Service accounts](/cloud/users/service-accounts/) &mdash; configure API access for running agents or executing flow runs on remote infrastructure.
- [Service accounts](/cloud/users/service-accounts/) &mdash; configure API access for running workers or executing flow runs on remote infrastructure.
- [Custom role-based access controls (RBAC)](/cloud/users/roles/) &mdash; assign users granular permissions to perform certain activities within an organization or a workspace.
- [Single Sign-on (SSO)](/cloud/users/sso/) &mdash; authentication using your identity provider.
- [Audit Log](/cloud/users/audit-log/) &mdash; a record of user activities to monitor security and compliance.
Expand Down Expand Up @@ -106,7 +106,7 @@ See the [Organizations](/cloud/organizations/) documentation for more informatio
## Service accounts <span class="badge orgs"></span>

Service accounts enable you to create a Prefect Cloud API key that is not associated with a user account.
Service accounts are typically used to configure API access for running agents or executing flow runs on remote infrastructure.
Service accounts are typically used to configure API access for running workers or executing flow runs on remote infrastructure.

See the [service accounts](/cloud/users/service-accounts/) documentation for more information about creating and managing service accounts in a Prefect Cloud organization.

Expand Down
2 changes: 1 addition & 1 deletion docs/cloud/users/api-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,6 @@ prefect cloud login -k '<my-api-key>'

Service accounts are a feature of Prefect Cloud [organizations](/cloud/organizations/) that enable you to create a Prefect Cloud API key that is not associated with a user account.

Service accounts are typically used to configure API access for running agents or executing flow runs on remote infrastructure. Events and logs for flow runs in those environments are then associated with the service account rather than a user, and API access may be managed or revoked by configuring or removing the service account without disrupting user access.
Service accounts are typically used to configure API access for running workers or executing flow runs on remote infrastructure. Events and logs for flow runs in those environments are then associated with the service account rather than a user, and API access may be managed or revoked by configuring or removing the service account without disrupting user access.

See the [service accounts](../service-accounts/) documentation for more information about creating and managing service accounts in a Prefect Cloud organization.
2 changes: 1 addition & 1 deletion docs/cloud/users/service-accounts.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ search:
# Service Accounts <span class="badge cloud"></span> <span class="badge orgs"></span>

Service accounts enable you to create a Prefect Cloud API key that is not associated with a user account.
Service accounts are typically used to configure API access for running agents or executing deployment flow runs on remote infrastructure.
Service accounts are typically used to configure API access for running workers or executing deployment flow runs on remote infrastructure.

Service accounts are non-user organization accounts that have the following:

Expand Down
16 changes: 8 additions & 8 deletions docs/concepts/flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -945,9 +945,9 @@ async def longrunning():
```
!!! tip "Pausing flow runs is blocking by default"
By default, pausing a flow run blocks the agent &mdash; the flow is still running inside the `pause_flow_run` function. However, you may pause any flow run in this fashion, including non-deployment local flow runs and subflows.
By default, pausing a flow run blocks the worker &mdash; the flow is still running inside the `pause_flow_run` function. However, you may pause any flow run in this fashion, including non-deployment local flow runs and subflows.
Alternatively, flow runs can be paused without blocking the flow run process. This is particularly useful when running the flow via an agent and you want the agent to be able to pick up other flows while the paused flow is paused.
Alternatively, flow runs can be paused without blocking the flow run process. This is particularly useful when running the flow via a worker and you want the worker to be able to pick up other flows while the paused flow is paused.
Non-blocking pause can be accomplished by setting the `reschedule` flag to `True`. In order to use this feature, flows that pause with the `reschedule` flag must have:
Expand All @@ -958,7 +958,7 @@ async def longrunning():
You may cancel a scheduled or in-progress flow run from the CLI, UI, REST API, or Python client.
When cancellation is requested, the flow run is moved to a "Cancelling" state. The agent monitors the state of flow runs and detects that cancellation has been requested. The agent then sends a signal to the flow run infrastructure, requesting termination of the run. If the run does not terminate after a grace period (default of 30 seconds), the infrastructure will be killed, ensuring the flow run exits.
When cancellation is requested, the flow run is moved to a "Cancelling" state. The worker monitors the state of flow runs and detects that cancellation has been requested. The worker then sends a signal to the flow run infrastructure, requesting termination of the run. If the run does not terminate after a grace period (default of 30 seconds), the infrastructure will be killed, ensuring the flow run exits.
!!! warning "A deployment is required"
Flow run cancellation requires the flow run to be associated with a [deployment](#serving-a-flow). A monitoring process must be running to enforce the cancellation. Inline subflow runs, i.e. those created without `run_deployment`, cannot be cancelled without cancelling the parent flow run. If you may need to cancel a subflow run independent of its parent flow run, we recommend deploying it separately and starting it using the [run_deployment](/api-ref/prefect/deployments/deployments/#prefect.deployments.run_deployment) method.
Expand All @@ -969,12 +969,12 @@ Support for cancellation is included for all core library infrastructure types:
- Kubernetes Jobs
- Processes
Cancellation is robust to restarts of the agent. To enable this, we attach metadata about the created infrastructure to the flow run. Internally, this is referred to as the `infrastructure_pid` or infrastructure identifier. Generally, this is composed of two parts:
Cancellation is robust to restarts of the worker. To enable this, we attach metadata about the created infrastructure to the flow run. Internally, this is referred to as the `infrastructure_pid` or infrastructure identifier. Generally, this is composed of two parts:
1. Scope: identifying where the infrastructure is running.
2. ID: a unique identifier for the infrastructure within the scope.
The scope is used to ensure that Prefect does not kill the wrong infrastructure. For example, agents running on multiple machines may have overlapping process IDs but should not have a matching scope.
The scope is used to ensure that Prefect does not kill the wrong infrastructure. For example, workers running on multiple machines may have overlapping process IDs but should not have a matching scope.
The identifiers for the primary infrastructure types are as follows:
Expand All @@ -986,10 +986,10 @@ While the cancellation process is robust, there are a few issues than can occur:
- If the infrastructure block for the flow run has been removed or altered, cancellation may not work.
- If the infrastructure block for the flow run does not have support for cancellation, cancellation will not work.
- If the identifier scope does not match when attempting to cancel a flow run the agent will be unable to cancel the flow run. Another agent may attempt cancellation.
- If the infrastructure associated with the run cannot be found or has already been killed, the agent will mark the flow run as cancelled.
- If the identifier scope does not match when attempting to cancel a flow run the worker will be unable to cancel the flow run. Another worker may attempt cancellation.
- If the infrastructure associated with the run cannot be found or has already been killed, the worker will mark the flow run as cancelled.
- If the `infrastructre_pid` is missing from the flow run will be marked as cancelled but cancellation cannot be enforced.
- If the agent runs into an unexpected error during cancellation the flow run may or may not be cancelled depending on where the error occurred. The agent will try again to cancel the flow run. Another agent may attempt cancellation.
- If the worker runs into an unexpected error during cancellation the flow run may or may not be cancelled depending on where the error occurred. The worker will try again to cancel the flow run. Another worker may attempt cancellation.
!!! tip "Enhanced cancellation"
We are working on improving cases where cancellation can fail. You can try the improved cancellation experience by enabling the `PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION` setting on your worker or agents:
Expand Down
10 changes: 5 additions & 5 deletions docs/concepts/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ If no storage is explicitly configured, Prefect will use `LocalFileSystem` stora
Prefect supports creating multiple storage configurations and switching between storage as needed.

!!! tip "Storage uses blocks"
[Blocks](/concepts/blocks/) is the Prefect technology underlying storage, and enables you to do so much more.
[Blocks](/concepts/blocks/) are the Prefect technology underlying storage, and enables you to do so much more.

In addition to creating storage blocks via the Prefect CLI, you can now create storage blocks and other kinds of block configuration objects via the [Prefect UI and Prefect Cloud](/ui/blocks/).

Expand All @@ -38,7 +38,7 @@ When building a deployment for a workflow, you have two options for configuring
- Use the default local storage
- Preconfigure a storage block to use

### Using the default
### Using the default

Anytime you call `prefect deployment build` without providing the `--storage-block` flag, a default `LocalFileSystem` block will be used. Note that this block will always use your present working directory as its basepath (which is usually desirable). You can see the block's settings by inspecting the `deployment.yaml` file that Prefect creates after calling `prefect deployment build`.

Expand All @@ -61,15 +61,15 @@ Current options for deployment storage blocks include:
| [Bitbucket Repository](https://prefecthq.github.io/prefect-bitbucket/) | Store code in a Bitbucket repository. | [`prefect-bitbucket`](https://github.com/PrefectHQ/prefect-bitbucket) |

!!! note "Accessing files may require storage filesystem libraries"
Note that the appropriate filesystem library supporting the storage location must be installed prior to building a deployment with a storage block or accessing the storage location from flow scripts.
Note that the appropriate filesystem library supporting the storage location must be installed prior to building a deployment with a storage block or accessing the storage location from flow scripts.

For example, the AWS S3 Storage block requires the [`s3fs`](https://s3fs.readthedocs.io/en/latest/) library.

See [Filesystem package dependencies](/concepts/filesystems/#filesystem-package-dependencies) for more information about configuring filesystem libraries in your execution environment.

### Configuring a block

You can create these blocks either via the UI or via Python.
You can create these blocks either via the UI or via Python.

You can [create, edit, and manage storage blocks](/ui/blocks/) in the Prefect UI and Prefect Cloud. On a Prefect server, blocks are created in the server's database. On Prefect Cloud, blocks are created on a workspace.

Expand Down
Loading

0 comments on commit d6b4f37

Please sign in to comment.