[migration] Run migration for latest content

Author: moonclavedev

src/content/docs/build/indexer/indexer-sdk/documentation/create-processor.mdx

@@ -6,7 +6,7 @@ This guide will walk you through setting up the basic template for a new process
 
 ## Pre-requisites
 
-You've already set up your environment and have the Indexer SDK `aptos-indexer-sdk` and `aptos-indexer-sdk-server-framework` installed.
+You've already set up your environment and have the Indexer SDK `aptos-indexer-sdk` installed.
 If you haven't, follow the [Indexer SDK installation guide](/build/indexer/indexer-sdk/documentation/setup).
 
 ## Overview
@@ -25,9 +25,9 @@ The next section goes through each of these pieces more explicitly and provides
 The `IndexerProcessorConfig` defines the base configuration for all processors that you'll be running.
 It should include configuration for things that are shared across multiple processors, like the database configuration and [Transaction Stream](/build/indexer/txn-stream) configuration.
 
-[`ServerArgs`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk-server-framework/src/lib.rs#L26) parses a `config.yaml` file and bootstraps a server with all the common pieces to run a processor.
+`ServerArgs` parses a `config.yaml` file and bootstraps a server with all the common pieces to run a processor.
 
-To setup the configuration for your processor and make it work with `ServerArgs`, you'll need to define a `IndexerProcessorConfig` that implements the [`RunnableConfig`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk-server-framework/src/lib.rs#L102) trait.
+To setup the configuration for your processor and make it work with `ServerArgs`, you'll need to define a `IndexerProcessorConfig` that implements the `RunnableConfig` trait.
 It also triggers a run method, which can be invoked in `main.rs`.
 
 For basic cases, you can copy the [`IndexerProcessorConfig` from the `aptos-indexer-processor-example`](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/config/indexer_processor_config.rs) repository and modify it to fit your needs.

src/content/docs/build/indexer/indexer-sdk/documentation/setup.mdx

@@ -7,21 +7,16 @@ The quickstart guide provides a template processor and includes all of this setu
 
 If you're migrating an existing processor to the Indexer SDK, follow the steps below.
 
-The Indexer SDK provides several Rust crates:
-
-1. [`aptos-indexer-processor-sdk`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/tree/main/aptos-indexer-processors-sdk/sdk) - The core SDK that provides the building blocks for writing a processor.
-2. [`aptos-indexer-processor-sdk-server-framework`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/tree/main/aptos-indexer-processors-sdk/sdk-server-framework) - A server framework for creating a server that runs the processor and includes health checks and metrics logging probes.
-   If you're setting up a server to host your processor, you will need to include this crate.
-3. [`aptos-indexer-testing-framework`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/tree/main/aptos-indexer-processors-sdk/testing-framework) - An e2e testing framework for testing processors.
-   If you want to write tests for your processor, you will need to include this crate.
-
-Depending on your use case, you can import the crates to your `Cargo.toml`.
+Add `aptos-indexer-processor-sdk` to your `Cargo.toml`.
 
 ```toml
 [dependencies]
 aptos-indexer-processor-sdk = { git = "https://github.com/aptos-labs/aptos-indexer-processor-sdk.git", rev = "aptos-indexer-processor-sdk-v1.0.0" }
-aptos-indexer-processor-sdk-server-framework = { git = "https://github.com/aptos-labs/aptos-indexer-processor-sdk.git", rev = "aptos-indexer-processor-sdk-v1.0.0" }
-aptos-indexer-testing-framework = { git = "https://github.com/aptos-labs/aptos-indexer-processor-sdk.git", rev = "aptos-indexer-processor-sdk-v1.0.0" }
 ```
 
+`aptos-indexer-processor-sdk` includes the following features:
+
+1. `postgres_full` - Interface layer to integrate Postgres with your processor.
+2. `testing_framework` - An e2e testing framework for testing processors. If you want to write tests for your processor, add this feature to the crate.
+
 {/* <!-- Add list of SDK releases once we have that --> */}

src/content/docs/build/indexer/indexer-sdk/quickstart.mdx

@@ -8,17 +8,17 @@ This guide will walk you through setting up and running a Rust processor to inde
 We provide a template processor that you can customize to index events from your custom contracts.
 By the end of the guide, you should have a basic understanding of how a processor works and be able to customize the processor for your indexing needs.
 
-## Getting started
+## Get started
 
 To get started, clone
-the [aptos-indexer-processors-example](https://github.com/aptos-labs/aptos-indexer-processor-example/tree/main) repo.
+the [aptos-indexer-processor-sdk](https://github.com/aptos-labs/aptos-indexer-processor-sdk) repo.
 
 ```text
 # HTTPS
-https://github.com/aptos-labs/aptos-indexer-processor-example.git
+https://github.com/aptos-labs/aptos-indexer-processor-sdk.git
  
 # SSH
-[email protected]:aptos-labs/aptos-indexer-processor-example.git
+[email protected]:aptos-labs/aptos-indexer-processor-sdk.git
 ```
 
 Processors consume transactions from the Transaction Stream Service. In order to use the Labs-Hosted Transaction Stream
@@ -63,7 +63,7 @@ export CPPFLAGS="-I/opt/homebrew/opt/libpq/include"
 - To easily view your database data, consider using a GUI like [DBeaver](https://dbeaver.io/)
   _recommended_, [pgAdmin](https://www.pgadmin.org/), or [Postico](https://eggerapps.at/postico2/).
 
-## Setting up your environment
+## Set up your environment
 
 Make sure to start the `postgresql` service:
 
@@ -79,27 +79,25 @@ For mac, if you’re using brew, start it up with:
 brew services start postgresql
 ```
 
-## **Configuring your processor**
+## **Configure your processor**
 
 Now let’s set up the configuration details for the actual indexer processor we’re going to use.
 
 ### **Set up your config.yaml file**
 
-In the example repo, there is a sample config.yaml file that should look something like this:
+In the example folder, there is a sample config.yaml file that should look something like this:
 
 ```yaml
+# This is a template yaml for the processor
 health_check_port: 8085
 server_config:
-  processor_config:
-    type: "events_processor"
   transaction_stream_config:
-    indexer_grpc_data_service_address: "https://grpc.testnet.aptoslabs.com:443"
-    starting_version: 0
-    # request_ending_version: 10000
+    indexer_grpc_data_service_address: "https://grpc.mainnet.aptoslabs.com:443"
     auth_token: "AUTH_TOKEN"
     request_name_header: "events-processor"
-  db_config:
-    postgres_connection_string: postgresql://postgres:@localhost:5432/example
+    starting_version: 0
+  postgres_config:
+    connection_string: postgresql://postgres:@localhost:5432/example
 ```
 
 Open the `config.yaml` file and update these fields:
@@ -137,16 +135,36 @@ indexer_grpc_data_service_address: grpc.testnet.aptoslabs.com:443
 indexer_grpc_data_service_address: grpc.mainnet.aptoslabs.com:443
 ```
 
-## Explanation of the processor
+In this tutorial, we are using `testnet` so update the `indexer_grpc_data_service_address` to `grpc.testnet.aptoslabs.com:443`.
+
+## Create the events processor
 
 At a high level, each processor is responsible for receiving a stream of transactions, parsing and transforming the
 relevant data, and storing the data into a database.
 
-### Defining the events database model
+### Define the database schema
+
+In `src/db/migrations`, you will see the events migration, which defines the database schema that will be used to store the events.
+
+```sql up.sql
+CREATE TABLE events (
+    sequence_number BIGINT NOT NULL,
+    creation_number BIGINT NOT NULL,
+    account_address VARCHAR(66) NOT NULL,
+    transaction_version BIGINT NOT NULL,
+    transaction_block_height BIGINT NOT NULL,
+    type TEXT NOT NULL,
+    data JSONB NOT NULL,
+    inserted_at TIMESTAMP NOT NULL DEFAULT NOW(),
+    event_index BIGINT NOT NULL,
+    indexed_type VARCHAR(300) NOT NULL,
+    PRIMARY KEY (transaction_version, event_index)
+);
+```
 
-In `src/db/postgres/schema.rs` , you will see events table which has the following schema:
+When you apply migrations, diesel will re-generate the `schema.rs` file, which looks like this:
 
-```rust
+```rust schema.rs
 diesel::table! {
     events (transaction_version, event_index) {
         sequence_number -> Int8,
@@ -166,43 +184,46 @@ diesel::table! {
 }
 ```
 
-The events schema represents the data that this processor is indexing. This [`schema.rs`](http://schema.rs) file is an
-autogenerated from the database migrations. In the next section, we’ll go over how these migrations are run.
-
-There are two other important tables:
+In `schema.rs`, you'll see two other important tables:
 
 - `ledger_infos` which tracks the chain id of the ledger being indexed
 - `processor_status` which tracks the `last_success_version` of the processor
 
-### Defining the events processor
-
-The file `src/processors/events/events_processor.rs` contains the code which defines the events processor. Inside of
-`run_processor` there are a few key components:
-
-1. First, we setup the processor:
-   1. `run_migrations` automatically runs the database migrations defined in `src/db/postgres/migrations`
-   2. We merge the starting version in `config.yaml`  and the `processor_status.last_success_version` in the database
-      to get the final starting version for the processor. This allows us to restart the processor from a previously
-      processed version.
-   3. We check the `ledger_infos.chain_id` to make sure the processor is indexing the correct chain
-2. Next, we instantiate the processor steps. Here we explain the purpose of each step:
-   1. `TransactionStreamStep` provides a stream of transactions to the processor
-   2. `EventsExtractor` extracts events data from each transaction
-   3. `EventsStorer` inserts the extracted events into the `events` table
-   4. `LatestVersionTracker` keeps track of the latest processed version and updates the `processor_status` table
-3. Lastly, we connect the processor steps together.
-   1. `ProcessorBuilder::new_with_inputless_first_step` takes in the first step of the processor. In most cases, the
-      first step is a `TransactionStreamStep` .
-   2. The rest of the steps are connected with `connect_to`. `connect_to` creates a channel between the steps so the
-      output of one step becomes the input of the next step.
-   3. And then we end the builder with `end_and_return_output_receiver`.
-
-# Running the processor
+### Define the processing logic
+
+The file `src/main.rs` contains the code which defines the events processor. The key components are:
+
+1. `insert_events_query` defines the diesel query to insert events into the database.
+   ```rust
+   fn insert_events_query(
+       items_to_insert: Vec<EventModel>,
+   ) -> impl QueryFragment<Pg> + diesel::query_builder::QueryId + Send {
+       use crate::schema::events::dsl::*;
+       diesel::insert_into(crate::schema::events::table)
+           .values(items_to_insert)
+           .on_conflict((transaction_version, event_index))
+           .do_nothing()
+   }
+   ```
+2. `process` is a helper function that wraps around a regular processor.
+   In the background, this powerful function handles connecting to Transaction Stream, processing transactions given a transform function that you define, applying database migrations, and tracking the processor's status.
+
+   ```rust
+   process(
+           "events_processor".to_string(), // name of the processor that will be used to track the processor status
+           MIGRATIONS, // migrations to be applied to the database
+           async |transactions, conn_pool| {
+             // transform from transaction to events and insert the events into the database
+           },
+   ).await?;
+   ```
+
+## Run the processor
 
 With the `config.yaml` you created earlier, you’re ready to run the events processor:
 
 ```shellscript
-cd aptos-indexer-processor-example
+cd examples/postgres-basic-events-example
 cargo run --release -- -c config.yaml
 ```
 
@@ -214,30 +235,86 @@ You should see the processor start to index Aptos blockchain events!
 {"timestamp":"2024-08-15T01:06:35.257801Z","level":"INFO","message":"Finished processing events from versions [0, 4999]","filename":"src/processors/events/events_processor.rs","line_number":90,"threadName":"tokio-runtime-worker","threadId":"ThreadId(17)"}
 ```
 
-# Customizing the processor
+## Customize the processor
 
 In most cases, you want to index events from your own contracts. The example processor offers a good starting point to
 creating your own custom processor.
 
-To customize the processor to index events from your custom contract, you can make change in these places:
+To customize the processor to index events from your custom contract, you can make these changes:
 
-- `EventsExtractor`
-  - In `process()`, you can filter by specific event types and extract specific event data from your custom contract
-- `EventsStorer`
+1. Change the database schema to a format that better matches your dapp or API.
+   a. Create a new migration with diesel:
 
-  - If you need to change the database model, you can generate a new database migration by going to `src/db/postgres`
-    and running
-
-  ```shellscript
+```shellscript
   diesel migration generate {migration_name} 
-  ```
+```
 
-  - Add your migration changes to `up.sql` and `down.sql`, then run
+b. Add your migration changes to `up.sql` and `down.sql`, then apply the migration:
 
-  ```shellscript
+```shellscript
   diesel migration run --database-url={YOUR_DATABASE_URL}
-  ```
+```
+
+c. The `schema.rs` file will be updated automatically. You can then create a diesel query that uses the new schema.
+2\. Update the transform logic in `process()`. You can filter by specific event types and extract specific event data from your custom contract
 
-  to update `schema.rs`.
+## Migrate from legacy processors
+
+If you're migrating from the legacy processors, you can still start with the same steps above to create a new processor with the Indexer SDK.
+
+You'll also need to follow these:
+
+1. Copy your migration files to `src/db/`.
+2. With the legacy processors, the processing logic is defined inside the `process_transactions` method.
+
+```rust
+// Example with the legacy processors
+#[async_trait]
+impl ProcessorTrait for EventsProcessor {
+    async fn process_transactions(
+        ...
+    ) -> anyhow::Result<ProcessingResult> {
+        // Extract events from transactions 
+        let events: Vec<EventModel> = process_events(transactions);
+
+        // Store the events in the database
+        let tx_result = insert_to_db(
+            self.get_pool(),
+            self.name(),
+            start_version,
+            end_version,
+            &events,
+            &self.per_table_chunk_sizes,
+        )
+        .await;
+
+        return tx_result;
+    }
+}
+```
+
+Migrate to the SDK by copying over the logic in `process_transactions` method to the SDK `process` transform function.
+
+```rust
+// Example with SDK processor
+    process(
+        "events_processor".to_string(),
+        MIGRATIONS,
+        async |transactions, conn_pool| {
+          // Extract events from transactions 
+          let events: Vec<EventModel> = process_events(transactions);
+
+          // Store events in the database
+          let execute_res = execute_in_chunks(
+              conn_pool.clone(),
+              insert_events_query,
+              &events,
+              MAX_DIESEL_PARAM_SIZE / EventModel::field_count(),
+          )
+          .await;
+        },
+    )
+    .await?;
+```
 
-  - And then update the `EventsStorer.process()` to handle storing the events data to the updated database model
+3. Update the `config.yaml` file to the new format. Update `starting_version` to the version that is last saved in the `processor_status` table.

src/content/docs/build/sdks/ts-sdk/building-transactions/script-composer.mdx

@@ -1,9 +1,7 @@
 ---
-title: "Dynamically invoke chains of Move calls with ScriptComposer"
+title: "Invoke chains of Move calls with Dynamic Script Composer"
 ---
 
-This is so far an **experimental** feature that is only released to [this particular](https://www.npmjs.com/package/@aptos-labs/ts-sdk/v/1.33.0-sc.0) version of typescript SDK and is subject to change.
-
 In the naive api, you only get to specify one entry function to invoke for one transaction. An advanced builder might want to be able to invoke multiple **public** Move functions inside one transaction. This is now enabled by the new `scriptComposer` api provided in the transaction builder.
 
 Here's an example of how you can invoke the api: