How do you perform On Demand loading for DataTable rows?

[Delengowski]

Hello,

Textual looks like a great project, I am interested in using it, due to its portability. My one requirement is the ability to load rows to a DataTable on demand. I am familiar with Qts MVC methodology, and its QTableView has methods such as canFetchMore and fetchMore for on demand loading. Is there anything equivalent? Or some recipe to do this with Textual? I'm struggling to find if this is possible, except for one discussion that links to a 3rd party Textual extension. Seems to be not?

[agammann]

You can load batches on demand by keeping the pagination state in your app
and adding each fetched batch with
DataTable.add_rows().
The table does not fetch from a backing model for you; your app decides when to
request another page.

Here is a complete button-triggered example. fetch_page uses an in-memory
source so you can run it directly; replace that function with your async
API/database call:

import asyncio

from textual import work
from textual.app import App
from textual.widgets import Button, DataTable


async def fetch_page(offset, limit):
    """Replace with an async database/API call; this example has 125 rows."""
    await asyncio.sleep(0.01)
    return [(i, f"Item {i}") for i in range(offset, min(offset + limit, 125))]


class PagedTable(App):
    CSS = "DataTable { height: 1fr; }"
    page_size = 50

    def __init__(self):
        super().__init__()
        self.offset = 0
        self.busy = False
        self.finished = False

    def compose(self):
        yield DataTable()
        yield Button("Load more", id="more")

    def on_mount(self):
        self.query_one(DataTable).add_columns("ID", "Name")
        self.request_page()

    def on_button_pressed(self, event: Button.Pressed):
        if event.button.id == "more":
            self.request_page()

    def request_page(self):
        if self.busy or self.finished:
            return
        self.busy = True
        self.query_one("#more", Button).disabled = True
        self.load_page()

    @work
    async def load_page(self):
        try:
            rows = await fetch_page(self.offset, self.page_size)
            self.query_one(DataTable).add_rows(rows)
            self.offset += len(rows)
            self.finished = not rows
        except Exception as error:
            self.notify(str(error), severity="error")
        finally:
            self.busy = False
            self.query_one("#more", Button).disabled = self.finished


if __name__ == "__main__":
    PagedTable().run()

The button is disabled while a request is running. The offset advances only
after a successful insertion, so a failed request can be retried. An empty
batch marks the end. For a real API with a next-page token, store that token
instead of an offset and use the API's explicit end-of-results signal.

I tested this as a headless Textual 8.2.8 app: the table grew from 50 to 100
to 125 rows, an empty result disabled the button, and a failed load left the
existing rows/offset intact for retry.

This postpones fetching additional data, but added rows remain stored in the
DataTable. For a dataset too large to keep loaded, use page replacement
(table.clear() preserves columns by default) or a bounded row window rather
than continually appending. The async load runs in a
worker, so awaiting an async
fetch doesn't hold up the button's message handler.

[Delengowski]

Very interesting, thanks for sharing. I wonder how it scales.

[agammann]

The example's test covered only 125 rows, so I don't have a measured upper limit to claim. fetch_page requests 50 at a time, but add_rows() retains all fetched rows in the widget; batching the fetch alone doesn't cap table memory. For a large or unbounded dataset, I'd paginate at the source and show one page (or a fixed-size window), using table.clear() before add_rows(); clear() preserves columns by default. That bounds the rows held in the table, at the cost of refetching when the user navigates back. DataTable docs. I'd benchmark your expected row/column count and cell content before choosing a page size.