diff --git a/docs/bookmarks.md b/docs/bookmarks.md new file mode 100644 index 00000000..cd5c6d1b --- /dev/null +++ b/docs/bookmarks.md @@ -0,0 +1,403 @@ +# Bookmarks aka version tags aka commits aka ... + +## Table of contents + + - [Facts and assumptions](#facts-and-assumptions) + - [Terminology](#terminology) + - [Basic operations](#basic-operations) + - [Note on history tuple vs `mod_time` for bookmarking](#note-on-history-tuple-vs-mod_time-for-bookmarking) + - [Validate a history tuple](#validate-a-history-tuple) + - [Get the latest undeleted (LU) records, given a subset of history records](#get-the-latest-undeleted-lu-records-given-a-subset-of-history-records) + - [Create a bookmark label](#create-a-bookmark-label) + - [Bookmark a point in history (create a bookmark association)](#bookmark-a-point-in-history-create-a-bookmark-association) + - [Bracket (or group) a set of updates](#bracket-or-group-a-set-of-updates) + - [Applications of bookmarking](#applications-of-bookmarking) + - [Efficient grouping of data versions](#efficient-grouping-of-data-versions) + - [An unsatisfactory approach](#an-unsatisfactory-approach) + - [Bracketing](#bracketing) + - [Historical reconstruction or Rollback](#historical-reconstruction-or-rollback) + - [Outline](#outline) + - [Implementation considerations](#implementation-considerations) + - [Extraction of specific subsets](#extraction-of-specific-subsets) + - [Example 1: Records inserted or updated within a single bracketing](#example-1-records-inserted-or-updated-within-a-single-bracketing) + - [Example 2: Records inserted or updated in a specific time period](#example-2-records-inserted-or-updated-in-a-specific-time-period) + - [Implementation notes](#implementation-notes) + - [Tables](#tables) + - [Other discussions](#other-discussions) + - [Updates to existing records: what supporting metadata?](#updates-to-existing-records-what-supporting-metadata) + - [Metadata support](#metadata-support) + +_TOC courtesy of [Lucio Paiva](https://luciopaiva.com/markdown-toc/)._ + +## Facts and assumptions + +**Facts** + +- History tables are append-only. +- Each history table records the changes made to the entire collection *in temporal order of the changes*. +- Each successive update to a collection is recorded by appending a record to its history table; therefore temporal order is also the order by ascending history id. + +**Assumptions** + +- No existing record in a history table is ever modified. + +**Therefore** + +- If a bookmark is associated to a record in a history table, it represents the history of that collection up to that point in time. A bookmark association can be thought of by analogy with a Git tag, in the sense that both are pointers to a specific state of the relevant items. +- Two such bookmark associations, say $B_1$ and $B_2$, bracket a set of changes recorded in the history table. The delta between them is exactly those changes recorded in the history table, in history id order, between $B_1$ (exclusive) and $B_2$ (inclusive). + +**For further consideration** + +- Bookmark associations can be, and most naturally are, stored in order of the association operations, that is, temporally. Therefore we can read out a series of successive changesets simply by examining the bookmark associations in the order they are made. + - However, that is not true if we allow bookmarking of non-latest states, which is probably going to be needed. We already have history, and we won't always anticipate future needs. Hmmm. + - Alternative ordering for bookmarks: In the order they occur according to the history table id, that is in history table temporal order. These should be consistent across all history tables; verify this thinking. + +## Terminology + +- **Bookmark**: A named object that designates a *point in history*. This is an imprecise usage of the term "bookmark", which is actually two related things, a *bookmark label* and a *bookmark association*: + +- **Bookmark label**: A data object bearing a label used for bookmarking. Several different points in history can be tagged with the same label. This allows a common label to be used to group multiple items. + +- **Bookmark association**: A data object that associates a *bookmark label* to a specific point in history. This is the fundamental operation of bookmarking. + +- **Point in history**: The current values of all items in history-tracked tables. It excludes non-history tracked tables, except as adjuncts to the current state. (Note that we cannot be sure what the non-history tracked tables might have contained in the past. Only the present is knowable with such tables.) + - **Current point in history**: The state of the history-tracked collections at the current point in time, in whatever sense "current" can be understood. + - **Past point in history**: An actual previous state of history-tracked collections. Such a state was the current point in history in the database at some moment, however briefly. + +- **History tuple**: A tuple of history id's, one for each history table, in some specified order of those tables. There are many possible tuples of history id's, but only some of them represent actual points in history. Those that do represent actual points in history are called (valid) history tuples. The rest are invalid and represent nothing useful. See also *validity*, below. + +- **Historical subset**: A history tuple defines a subset of the history, namely all those history records in each history table that occur at or before the corresponding history id in the tuple. Under the assumption (see above) that the temporal order of history records is the same as the history id order, "at or before" means that history id is less than or equal to the history id in the tuple. See also *validity*, below. + +- **Validity** (of historical subset, of history tuple): + - A historical subset is valid if and only if it exhibits referential integrity. That is, all references by a history record in that subset to another history record must also be found in the subset. Otherwise (i.e., when there is a violation of referential integrity within the subset) the historical subset is not valid. + - A history tuple is valid if and only if it implies a valid database historical subset. + +- **Latest undeleted (LU) records**: Given a set $H$ of history records drawn from a single history table: + - Informally: The undeleted (LU) records are the latest (most recent) history records in that set, one for each item (distinguished by item id) not marked as deleted within the set. + - Formally: Define $LU(H) \subseteq H$ such that: + - For each item id $i$ present in $H$ (recall that item id's are not unique in history records): + - there is at most one history record in $h_i \in LU(H)$ with item id $i$; + - $h_i$ has the largest history id of all history records in $H$ with item id $i$; + - that item has not been deleted ($h_i$ does not have the deleted flag set). + - If $H$ contains all records in a history table prior to some point, the LU set represents what that ollection looked like at that point in time. + - Given a *valid historical subset*, then the collections of LU records from each history table in the subset give us the state of the history-tracked collections at the point in time represented by the historical subset. +## Basic operations + +### Note on history tuple vs `mod_time` for bookmarking + +We could, in principle, use `mod_time` to define a point in history. A history subset would then be just all history records occurring at or before a given `mod_time`. This would have some advantages: + +- A single value rather than a tuple denotes a point in history. +- Validity (referential integrity) of a historical subset is trivially guaranteed so long as all history records satisfying the `mod_time` condition are included in the subset. There's no need to validate such a time value as there is for a tuple. + +However, we cannot always trust times in the database. Rather than fretting over time synchrony, we use the more reliable marker of position that is the history id. In that case, we have to store the history id's for each history table; hence a history tuple, not a single time value. + +***Note***: If all history tables shared the same history id sequence, then a single history id would in fact point unambiguously at a point in history, in the same way as `mod_time` but without the doubts about time. This might be worth considering. Notes: + +- You'd have to obtain from the shared history id sequence (for current point in history) or the history tables (for past points) the largest history id corresponding to your desired point in history. +- You'd need to migrate all the existing tables. For `obs_raw_hx`, this would be a pretty big operation, again, but nothing like as big an operation as the original history table population. + +### Validate a history tuple + +$Validate(S)$: Given a history tuple $S$, check that it is valid. Raise an error if it is not. + +1. The naive algorithm checks every reference in any given subset for presence in the referenced collection historical subset. But this is a huge number of records in most cases. +2. A less naive and much faster algorithm relies on the assumption that every actually occurring historical state of the database was valid (quite a reasonable assumption!), and therefore that history tables reflect that. This assumption allows us to check only that reference history id's are less than or equal to the corresponding collection history id in the tuple. + +### Get the latest undeleted (LU) records, given a subset of history records + +It's straightforward to construct a query that yields the history id's of the LU items in a given collection. These history id's can then be used to extract part or all of the LU records from the history table. + +Here's a specific example for `meta_network_hx`. In this query, `` is the condition that extracts the subset from the full history table. We can generalize this query to any history table. + +``` +SELECT + network_id, + max(meta_network_hx_id) AS max_hx_id +FROM + meta_network_hx +WHERE +GROUP BY + network_id +HAVING + NOT bool_or(deleted) +``` + +The `` could look like one of the following: + +- `meta_network_hx_id <= upper_bound` (full previous point in history) +- `lower_bound < meta_network_hx_id AND meta_network_hx_id <= upper_bound` (partial history between two previous points) +- `mod_time <= upper_bound` (full previous history bounded by when modifications were applied) +- etc. + +Again looking a little forward in this document, a common case will be where the condition is related to one or more bookmarks. +##### Notes + +- Fidelity to actual history would require that there are no gaps in the set of history records, i.e., that we haven't arbitrarily dropped some from the middle. However, that is not strictly necessary for these operations to be performed.) + +### Create a bookmark label + +**Motivation**: A bookmark label is an object used to tag one or more points in history. It is independent of the point(s) in history it is associated to. + +**Operation**: The act of creating a new bookmark label $L$, eliding the details of bookmark implementation, is denoted $L = CreateBookmarkLabel(N, ...)$ where $N, ...$ is the bookmark name and other (elided) details. + +### Bookmark a point in history (create a bookmark association) + +**Motivation**: This is the fundamental operation in bookmarking. + +**Operation**: Let $L$ be a bookmark label. Let $S$ be a tuple of history id's. Then, eliding the details of the association data object: +- $B = Bookmark(L, S)$ denotes the atomic (transaction enclosed) operation of: + - validating $S$, and then + - associating bookmark label $L$ to state $S$ + - returning the bookmark association $B$. +- The shorthand $Bookmark(L)$ is defined as $Bookmark(L, S)$ where $S$ is the current state of the database. + +***Note***: For information on validating a tuple of history id's, see [[#History tuples, historical subsets, and their validity]]. +### Bracket (or group) a set of updates + +**Motivation/scenario**: A set of related updates are received or made all at one time. The canonical case is a QA update of a large set of observations. (In other cases, e.g., when a scientist is updating things, it will take a certain amount of discipline to make sure that the updates are batched together like this.) + +**Operation**: Let $L_1$ and $L_2$ be two bookmarks. Let $U$ be a set of updates. Then the operation $Bracket(L_1, U, L_2)$ is defined as: + +- *Within a transaction (i.e., in isolation from other operations)*: + - Perform $Bookmark(L_1)$. + - Perform updates $U$. + - Perform $Bookmark(L_2)$. + +History records between $L_1$ (exclusive) and $L_2$ (inclusive) are exactly and only those changes made in the updates. This is due to: + +- their isolation in the transaction (so no other update operations are interleaved); +- the fact that change records are appended to the history tables in temporal order of change operations. + +**Other notes**: + +- Since the bookmarks for bracketing are directly related, we would probably do better to use a single bookmark label $L$, and allow the system to construct bookmarks $L_1$ and $L_2$ from $L$. In fact, we use the same label, and the bookmark association carries the distinction between $L_1$ and $L_2$. We can then define $Bracket(L, U) = Bracket(L_1, U, L_2)$, where $L_1$ and $L_2$ are the constructed bookmarks. See use of auxiliary columns in bookmark association table in [[#Implementation notes]] below. + +**Transactions, concurrency, and transaction isolation level** + +Bracketing is useful only if operations occurring outside the transaction in which bracketing is performed cannot be interleaved with the operations occurring inside it. This is called transaction isolation. + +The Postgres documentation for [Transaction Isolation](https://www.postgresql.org/docs/current/transaction-iso.html) states: + +> The SQL standard defines four levels of transaction isolation. The most strict is Serializable, which is defined by the standard in a paragraph which says that any concurrent execution of a set of Serializable transactions is guaranteed to produce the same effect as running them one at a time in some order. The other three levels are defined in terms of phenomena, resulting from interaction between concurrent transactions, which must not occur at each level. The standard notes that due to the definition of Serializable, none of these phenomena are possible at that level. (This is hardly surprising -- if the effect of the transactions must be consistent with having been run one at a time, how could you see any phenomena caused by interactions?) + +Therefore not only must we run bracketing within a transaction, the transaction must be run at Serializable isolation level. This is done with the [`SET TRANSACTION`](https://www.postgresql.org/docs/current/sql-set-transaction.html) command within the transaction in question. + +## Applications of bookmarking + +### Efficient grouping of data versions + +CRMP partners periodically release new versions of a dataset. Such releases typically contain many thousands of observations. A version release amounts to an update to each such datum. + +Releases typically are, in time order: + +1. A raw dataset. This dataset frequently arrives incrementally, via `crmprtd`. +2. A QA-adjusted dataset. This dataset is expected to arrive in one or a few large batches. + +Each observation in the second release is a revision to an observation in the first release. +#### An unsatisfactory approach + +It is possible bookmark observations individually, by associating a bookmark to each updated item's history records. This requires the same number of bookmark associations as there are observations, i.e., it scales linearly with the number of observations. Given the number of observations, this is highly undesirable and should be avoided whenever possible. +#### Bracketing + +Bracketing uses only two bookmarks per group; one each to demarcate the beginning and end of a group: + +- Let $U$ be the set of updates that update the database for a given release. +- Define bookmarks $L_1$ and $L_2$ to demarcate (bracket) the release. Below, this amounts to using bracket-begin and bracket-end in the association of the same bookmark label. +- Perform $Bracket(L_1, U, L_2)$. + +Bracketing requires only constant time and space relative to the number of updates within it. + +##### Bracketing for large datasets (QA releases, other updates) + +QA releases and other updates are expected to arrive in large batches. If it is not operationally possible to perform the updates for a new release within a single transaction, this approach can generalized to a small number of bracketing operations which together encompass the whole release. This will still significantly reduce the space and time required to bracket and retrieve a large group of observations, even if it does scale linearly. (A 3+ order of magnitude reduction in associations is still a significant win.) + +##### Bracketing for regular ingestion (`crmprtd`) + +Regular ingestion (via `crmprtd` and related scripts) occurs piecemeal. Typically, dozens to thousands of observations are ingested at a time (hourly, daily, weekly, or monthly, depending on the network). We can use bracketing for each such group of observations ingested. This is still much smaller than a typical QA release, and it does scale linearly in total observations, but it is nonetheless much better than bookmarking one observation at a time. + +##### Considerations + +Bookmarks used to bracket each ingestion should bear a clear and easily queried relationship to each other. This enables an entire dataset (e.g., raw, QA'd) to be extracted with simple, error-resistant queries. The design of bookmark associations, specifically columns `role` and `bracket_begin_id`, support this directly. + +### Historical reconstruction or Rollback + +#### Outline + +The design of history tracking makes it easy (although not necessarily *fast*) to reconstruct a point in history from a bookmark (or more precisely a bookmark association). In other terms, to produce the historical subset given a history tuple. + +The definition of LU (latest updated) records provides the necessary tool. Rollback is just the operation of generating LU records and storing them somewhere. + +#### Implementation considerations + +For metadata tables, which have few records, the basic LU query is likely fast. + +For `obs_raw_hx`, the LU query will necessarily scan a huge number of records. To make it perform better: + +- Create appropriate indexing on the history tables. +- Use the tightest possible WHERE conditions. If, for example, only observations from a specific network or from specific stations are desired, then encode that in the WHERE condition. + +For efficiency and convenience we are likely to want to store the result in a separate set of tables, which are best housed in their own separate schema. + +In a separate schema, call it `crmp_rollback`, do the following. Given a bookmark: + +- Establish structural copy of the `crmp` schema (i.e., table definitions without data) including main tables but excluding history tables (history tables are redundant here). +- Include FK relationships, indexes, and other things as needed. +- Duplicate the content of the non-history tracked tables (at the time of of this rollback). +- Populate the history-tracked tables using queries as above. +- Define and populate (one row) a rollback table that contains at least the following information: + - bookmark association ids, if relevant + - text of WHERE condition ... the bookmark association ids may not be relevant or the full story + - timestamp when this rollback was established + - id of user creating the rollback + - any other important information not retrievable with this data +- Possibly make all tables read-only. +- Possibly make this schema accessible only by a specific role which is granted only to the user(s) who need this version of the database. + +And, in the main `crmp` schema, define a stored procedure that does all of the above: + +- Given a bookmark id (or more broadly a where condition) and rollback schema name +- In a valid order (respecting foreign key dependencies) +- With optimized queries insofar as possible + +Once the rollback schema is populated with data reflecting a given point in history, the users with interest in it can query it as if it is the actual CRMP database. It will not in general be wise to allow the users to modify this database, since those modifications will not under any circumstances be propagated to the real database. For experimental purposes, with appropriate, loudly stated caveats, this rule may be relaxed. + +Any number of rollback schemas can be established. They do not interact with each other, nor with the live CRMP database. But because each rollback schema will be comparable in size to the live database, we may wish to limit their number and their lifetimes. + +### Extraction of specific subsets + +In some cases, it's possible that the (historical) records of interest lie only in a restricted range. This would be a much smaller dataset than the entire set of records in `obs_raw` at a given point in history. With suitable indexes, these subsets will be much faster to extract and work with than the whole of the history-tracked tables. Some examples: + +- The data selected by a particular bookmark. This might be several groups or just one, depending on how the bookmark was used -- for example, depending on whether it bracketed just one group of data or several. +- The data selected by a particular date range. This would require extracting the (latest) historical records for that time period. This does not involve bookmarks. +- A combination of the above. + +#### Example 1: Records inserted or updated within a single bracketing + +This could correspond to a QA update to a previously ingested set of raw observations. + +Let `b_begin` and `b_end` represent tuples either externally provided history tuples (substituted in to the query), or tuples queried directly from the bookmark tables, corresponding to a bracket-begin and bracket-end pair. + +The bracketed set of updates may include multiple updates to a single item in a given collection. After the fact, we are only interested in the final outcome, which is the *latest* value for each collection item in the bracket. We must do a little extra work to obtain the latest indexes, which means taking the latest (equivalently, greatest) history id within the subset for each item id. + +See [[#Given a set of history records, get the latest undeleted (LU) records]]. + +##### Caution! Bracketed sets vs valid historical subsets + +It's tempting to think that the collection of records selected from each history-tracked collection by the above process would jointly constitute a valid historical subset. Not so! + +Let's consider what is probably the most common and germane example, updates to `obs_raw`. + +Suppose the bracketed updates were *only* to `obs_raw`, a not unlikely scenario. Then the collection of history records obtained by the above process, from all history tables, contains only records from `obs_raw_hx`. There are no metadata records whatsoever in this set, because none were modified. But those `obs_raw_hx` records necessarily point at metadata history records ... which are not in the set. + +What's going on here? The historical metadata supporting the observations is drawn from the entire set of latest records prior to the bracket-end bookmark, potentially as far back as the first record ever inserted. + +It's important to keep this slightly subtle point in mind when working with brackets or other historical subsetting. + +##### Question: Which supporting metadata? + +Again, considering brackets with updates only to `obs_raw` will bring things into sharper focus. + +We have three different plausible choices for the metadata supporting these observations: + +1. The historical metadata directly linked to each `obs_raw_hx` record within the bracket. +2. The *latest* version, *within the subset implied by the bracket-end*, of the metadata linked to each `obs_raw_hx`. +3. The *current* version of the metadata item linked to each `obs_raw_hx` record within the bracket. This is not constrained by the bracket-end. Therefore, unlike the above two records, it can vary as time passes, i.e., as further updates to the linked records are made. Those updates can include deletion, so it is doubly perilous to consider using this choice. We cannot recommend it. + +The correct choice depends on context and intention, although (3) is highly questionable. There is no universally correct choice. +#### Example 2: Records inserted or updated in a specific time period + +We can also form a subset based on time constraints. This is not fundamentally different, but there are some additional or sharpened considerations. + +Let `t_begin` and `t_end` be timestamps defining the time period of interest. We want to extract the subset of records that were inserted or updated in this period. + +See [[#Given a set of history records, get the latest undeleted (LU) records]]. + +All three considerations described above about what records are germane in the subset are important here: + +1. The process of obtaining the latest historical value for each item selected within the time period. Even more so than with a bracket, the collection items selected within a specific time period may experience multiple updates. +2. Time period constraints do not provide any guarantee that the history records associated to any within the temporal subset are also within that subset. And still less if the time constraints are selected for one particular collection and not with all collections in mind. + 1. [[#Caution! Bracketed sets vs valid historical subsets]] is relevant, but with the time constraints playing the role of the brackets. This is perhaps even more pointed because of the lack of the sanity guaranteed by a bookmark's validity constraint. + 2. [[#Question Which supporting metadata?]] This translates over pretty much unchanged. + +## Implementation notes + +A provisional implementation is on branch [i-239-version-tagging](https://github.com/pacificclimate/pycds/tree/i-239-version-tagging). It includes types, tables, functions, trigger functions. In this section we just summarize and make some remarks and questions. + +### Tables + +**Table `bookmark_labels`** + +Questions: + +1. Apply history tracking to this table? Reason, utility? +2. FK `network_id` + 1. Is it actually needed? A network is implied by a bookmark's association to items. This is also true of variables, but they too have a indirect network association (`network_id`) that is not strictly necessary. That fact inspired the idea of having a direct association for bookmarks as well. This over-specificity (really: denormalization) may be offset by the utility of easily segregating these things (variables, bookmarks) by network, and establishing their relationship to network *before* use elsewhere. + 2. Nullable? Tempting, but caution, nullable columns have frequently been abused in CRMP. + +**Table `bookmark_associations`** + +Q (rhetorical): Why separate association from bookmark proper? +A: To support multiple uses of the same bookmark. +- Brackets share the same bookmark info, but are associated as bracket-begin, bracket-end. +- We likely want to bracket multiple groups of observations -- e.g., those ingested by `crmprtd` at any one time -- using the same bookmark. + +Questions: + +1. Apply history tracking to this table? Reason, utility? + +Constraints on bookmark associations and role: + +- Singleton bookmarks are permitted in any pattern, no constraints except tuple validity. +- We allow any pattern of bracket bookmark associations: disjoint, nested, overlapping. This is because we have no current knowledge of what patterns will be useful in future, and no logical reasons to exclude any. +- The only constraints on brackets are: + - bracket-begin and bracket-end must occur in matching pairs (open brackets, i.e., unmatched bracket-begins, are permitted). + - a bracket-end must specify an open (not yet paired) bracket-begin that occurs before (in order of ascending `bookmark_association_id`) the bracket-begin. +- We can (see trigger function): + - Auto-generate `bracket_begin_id` for a bracket-end id when there is only one unpaired bracket-begin (that one's bracket id). This seems a likely scenario. However, it is surplus to requirements if we assume/require the user to carry the auto-generated bracket-begin id. + + +## Other discussions + +### Updates to existing records: what supporting metadata? + +This isn't necessarily a question about bookmarking, or isn't exclusively so. + +Let's use `obs_raw` as the example, as it is covers all other cases. + +When we update an existing `obs_raw` record, we can ask what are the relevant metadata history items to link with it in history. + +This question is interesting because now that we have history tracking we also have a new phenomenon, which is that updates to metadata and updates to observations are not interchangeable. Order now matters. + +For any given `obs_raw` record, possible answers are: + +1. The current (latest) metadata history record for the metadata item linked to by the `obs_raw` record. + - The simplest answer, and the one that matches what happened before the advent of history tracking. It is also what happens automatically via the history foreign key maintenance trigger whenever a record is created or updated. +2. The metadata history record linked to the existing (un-updated) `obs_raw_hx` history record for that item (this may be the same as the current one, but it equally well may be quite a lot older). + - It is possible to imagine wanting to update a "frozen" state of the database, i.e., to update `obs_raw` based on an earlier state of the database that could be obtained from a history rollback, which would imply using the links to the older states of the relevant metadata items. +3. Some intermediate metadata history record between those for (1) and (2), if any exist. + - This just seems like asking for trouble. + +There is no *a priori* answer to this question; any answer could be correct. To multiply questions and confusion, this choice applies transitively to all metadata items linked indirectly to the obs_raw record (i.e., station and network). + +### Metadata support + +This a closely related topic, and maybe can precede the discussion above, but ... + +The *historical metadata support* of an observation history record $H$ is the set of metadata (history) records directly relevant to $H$, which is to say directly associated to $H$ by one or more FK links away from the observation. We denote this by $S(H)$. Notes: + +- This in fact applies to any history record $H$, but observation histories are the most important and are the most general or complex case. +- We have only unidirectional links to consider at the moment, but if we include many:many relationships in history tracking, then this definition will possibly become a little more complicated. + +The historical metadata support of a set of observation history records $K$ is the union of the support of each record $H \in K$. We write $S(K) = \bigcup_{H \in K} S(H)$. + +When we select a set of observations to be bookmarked after the fact, that is after the database has experienced more changes, we need to also include the metadata support of that set. This will make bookmarking after the fact somewhat trickier. + +As we note above, we can think about the *current metadata support* for history records, which would be those records linked via item id, not by history id. + +This is really just a restatement of the considerations above. + + + + + diff --git a/docs/history-tracking-overview.excalidraw b/docs/history-tracking-overview.excalidraw index 304e592d..8eef118f 100644 --- a/docs/history-tracking-overview.excalidraw +++ b/docs/history-tracking-overview.excalidraw @@ -1,12 +1,12 @@ { "type": "excalidraw", "version": 2, - "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.6.7", + "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.14.3", "elements": [ { "type": "rectangle", - "version": 746, - "versionNonce": 1634433902, + "version": 987, + "versionNonce": 2019939040, "index": "aM", "isDeleted": false, "id": "tPneSgy9-ER8wbYV3t7rf", @@ -16,10 +16,10 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 343.98140153924237, - "y": -434.51111201846356, + "x": 927.593862912328, + "y": -177.62186620039597, "strokeColor": "#1e1e1e", - "backgroundColor": "#a5d8ff", + "backgroundColor": "transparent", "width": 212.41664296929588, "height": 199.99999999999991, "seed": 1682344250, @@ -38,7 +38,7 @@ "type": "arrow" } ], - "updated": 1730220119371, + "updated": 1758744681397, "link": null, "locked": false, "customData": { @@ -47,8 +47,8 @@ }, { "type": "text", - "version": 629, - "versionNonce": 1481681390, + "version": 869, + "versionNonce": 1441298144, "index": "aMV", "isDeleted": false, "id": "Dw1n5NgF", @@ -58,8 +58,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 389.96148205709346, - "y": -429.51111201846356, + "x": 973.573943430179, + "y": -172.62186620039597, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "width": 120.45648193359375, @@ -69,7 +69,7 @@ "frameId": null, "roundness": null, "boundElements": [], - "updated": 1730220125332, + "updated": 1758744681397, "link": null, "locked": false, "fontSize": 28, @@ -85,8 +85,8 @@ }, { "type": "rectangle", - "version": 914, - "versionNonce": 1070822795, + "version": 1156, + "versionNonce": 60833504, "index": "aMVV", "isDeleted": false, "id": "HP8dFO0w6b8dQ6Er-z3Xz", @@ -96,10 +96,10 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 338.94226039163414, - "y": 1066.3646562997797, + "x": 922.5547217647197, + "y": 1323.2539021178472, "strokeColor": "#1e1e1e", - "backgroundColor": "#a5d8ff", + "backgroundColor": "#e9ecef", "width": 219.58143314329794, "height": 351, "seed": 1832701158, @@ -130,7 +130,7 @@ "type": "arrow" } ], - "updated": 1754949056677, + "updated": 1758744681397, "link": null, "locked": false, "customData": { @@ -139,8 +139,8 @@ }, { "type": "text", - "version": 819, - "versionNonce": 679644203, + "version": 1063, + "versionNonce": 1727357664, "index": "aMW", "isDeleted": false, "id": "i3rgw8Kk", @@ -150,35 +150,35 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 365.99667172646673, - "y": 1071.3646562997797, + "x": 949.6891044130289, + "y": 1328.2539021178472, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 165.4726104736328, - "height": 302.40000000000003, + "width": 165.3126678466797, + "height": 340.20000000000005, "seed": 728718458, "groupIds": [], "frameId": null, "roundness": null, "boundElements": [], - "updated": 1754949056677, + "updated": 1758744681397, "link": null, "locked": false, "fontSize": 28, "fontFamily": 6, - "text": " a_hx\n-------\na_hx_id (PK)\ndeleted\nmod_time\nmod_user\na_id\nx", - "rawText": "
a_hx\n-------\na_hx_id (PK)\ndeleted\nmod_time\nmod_user\na_id\nx", + "text": "
a_hx\n-------\na_id\nx\na_hx_id (PK)\ndeleted\nmod_time\nmod_user\n", + "rawText": "
a_hx\n-------\na_id\nx\na_hx_id (PK)\ndeleted\nmod_time\nmod_user\n", "textAlign": "center", "verticalAlign": "top", "containerId": "HP8dFO0w6b8dQ6Er-z3Xz", - "originalText": "
a_hx\n-------\na_hx_id (PK)\ndeleted\nmod_time\nmod_user\na_id\nx", + "originalText": "
a_hx\n-------\na_id\nx\na_hx_id (PK)\ndeleted\nmod_time\nmod_user\n", "autoResize": true, "lineHeight": 1.35 }, { "type": "rectangle", - "version": 882, - "versionNonce": 1702811531, + "version": 1123, + "versionNonce": 2128320224, "index": "aMWG", "isDeleted": false, "id": "HENr2olXzuzzezNs3HaA4", @@ -188,10 +188,10 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 343.74295363253555, - "y": 224.76059477682293, + "x": 927.3554150056211, + "y": 481.6498405948905, "strokeColor": "#1e1e1e", - "backgroundColor": "#a5d8ff", + "backgroundColor": "transparent", "width": 219.58143314329794, "height": 351, "seed": 526137786, @@ -222,7 +222,7 @@ "type": "arrow" } ], - "updated": 1754948758749, + "updated": 1758744681397, "link": null, "locked": false, "customData": { @@ -231,8 +231,8 @@ }, { "type": "text", - "version": 838, - "versionNonce": 1048482021, + "version": 1082, + "versionNonce": 285588192, "index": "aMWV", "isDeleted": false, "id": "x9CH03On", @@ -242,35 +242,35 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 389.8434312515478, - "y": 229.76059477682293, + "x": 974.4818707130123, + "y": 486.6498405948905, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 127.38047790527344, - "height": 226.8, + "width": 125.32852172851562, + "height": 264.6, "seed": 1055446758, "groupIds": [], "frameId": null, "roundness": null, "boundElements": [], - "updated": 1754948786518, + "updated": 1758744681397, "link": null, "locked": false, "fontSize": 28, "fontFamily": 6, - "text": "
a\n-------\nmod_time\nmod_user\na_id (PK)\nx", - "rawText": "
a\n-------\nmod_time\nmod_user\na_id (PK)\nx", + "text": "
a\n-------\na_id (PK)\nx\nmod_time\nmod_user\n", + "rawText": "
a\n-------\na_id (PK)\nx\nmod_time\nmod_user\n", "textAlign": "center", "verticalAlign": "top", "containerId": "HENr2olXzuzzezNs3HaA4", - "originalText": "
a\n-------\nmod_time\nmod_user\na_id (PK)\nx", + "originalText": "
a\n-------\na_id (PK)\nx\nmod_time\nmod_user\n", "autoResize": true, "lineHeight": 1.35 }, { "type": "rectangle", - "version": 870, - "versionNonce": 1977336302, + "version": 1111, + "versionNonce": 1549880032, "index": "aMX", "isDeleted": false, "id": "rPi80OCsDY1oxdpcLjLi4", @@ -280,10 +280,10 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 863.7050453468169, - "y": -445.9407018992092, + "x": 1447.3175067199024, + "y": -189.05145608114162, "strokeColor": "#1e1e1e", - "backgroundColor": "#a5d8ff", + "backgroundColor": "transparent", "width": 198.08706262129178, "height": 219.66897746967072, "seed": 192480634, @@ -302,7 +302,7 @@ "type": "arrow" } ], - "updated": 1728343910542, + "updated": 1758744681398, "link": null, "locked": false, "customData": { @@ -311,8 +311,8 @@ }, { "type": "text", - "version": 756, - "versionNonce": 503779694, + "version": 996, + "versionNonce": 464842464, "index": "aMZ", "isDeleted": false, "id": "Nwc49WhE", @@ -322,18 +322,18 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 901.4823641940839, - "y": -440.9407018992092, + "x": 1485.3767927302554, + "y": -184.05145608114162, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 122.53242492675781, + "width": 121.96849060058594, "height": 189.00000000000003, "seed": 1218422566, "groupIds": [], "frameId": null, "roundness": null, "boundElements": [], - "updated": 1730220133520, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 28, @@ -349,8 +349,8 @@ }, { "type": "rectangle", - "version": 1016, - "versionNonce": 1164874059, + "version": 1260, + "versionNonce": 716093152, "index": "aMa", "isDeleted": false, "id": "cXv2VOhBtuqYgae3ODq6c", @@ -360,12 +360,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 859.8526446365463, - "y": 1051.4743301288206, + "x": 1443.4651060096319, + "y": 1308.3635759468882, "strokeColor": "#1e1e1e", - "backgroundColor": "#a5d8ff", + "backgroundColor": "#e9ecef", "width": 219.58143314329797, - "height": 389, + "height": 426, "seed": 186531878, "groupIds": [], "frameId": null, @@ -386,7 +386,7 @@ "type": "arrow" } ], - "updated": 1754949056679, + "updated": 1758744681398, "link": null, "locked": false, "customData": { @@ -395,8 +395,8 @@ }, { "type": "text", - "version": 935, - "versionNonce": 1271148523, + "version": 1179, + "versionNonce": 359979744, "index": "aMb", "isDeleted": false, "id": "c0WKu6Wy", @@ -406,35 +406,35 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 886.4450503561445, - "y": 1056.4743301288206, + "x": 1470.1374830427067, + "y": 1313.3635759468882, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 166.39662170410156, - "height": 378.00000000000006, + "width": 166.23667907714844, + "height": 415.80000000000007, "seed": 1828830522, "groupIds": [], "frameId": null, "roundness": null, "boundElements": [], - "updated": 1754949056679, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 28, "fontFamily": 6, - "text": "
b_hx\n-------\nb_hx_id (PK)\ndeleted\nmod_time\nmod_user\nb_id\na_id\ny\na_hx_id (FK)", - "rawText": "
b_hx\n-------\nb_hx_id (PK)\ndeleted\nmod_time\nmod_user\nb_id\na_id\ny\na_hx_id (FK)", + "text": "
b_hx\n-------\nb_id\na_id\ny\na_hx_id (FK)\nb_hx_id (PK)\ndeleted\nmod_time\nmod_user\n", + "rawText": "
b_hx\n-------\nb_id\na_id\ny\na_hx_id (FK)\nb_hx_id (PK)\ndeleted\nmod_time\nmod_user\n", "textAlign": "center", "verticalAlign": "top", "containerId": "cXv2VOhBtuqYgae3ODq6c", - "originalText": "
b_hx\n-------\nb_hx_id (PK)\ndeleted\nmod_time\nmod_user\nb_id\na_id\ny\na_hx_id (FK)", + "originalText": "
b_hx\n-------\nb_id\na_id\ny\na_hx_id (FK)\nb_hx_id (PK)\ndeleted\nmod_time\nmod_user\n", "autoResize": true, "lineHeight": 1.35 }, { "type": "rectangle", - "version": 1071, - "versionNonce": 381003140, + "version": 1312, + "versionNonce": 195839712, "index": "aMbV", "isDeleted": false, "id": "jvtpsSfKxVMT4c7AFLiba", @@ -444,10 +444,10 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 861.1871333713818, - "y": 210.35439734173815, + "x": 1444.7995947444674, + "y": 467.24364315980574, "strokeColor": "#1e1e1e", - "backgroundColor": "#a5d8ff", + "backgroundColor": "transparent", "width": 219.58143314329797, "height": 365.34142114384747, "seed": 568390266, @@ -474,7 +474,7 @@ "type": "arrow" } ], - "updated": 1729117567263, + "updated": 1758744681398, "link": null, "locked": false, "customData": { @@ -483,8 +483,8 @@ }, { "type": "text", - "version": 989, - "versionNonce": 1442679301, + "version": 1233, + "versionNonce": 541092576, "index": "aMc", "isDeleted": false, "id": "fziB80tY", @@ -494,35 +494,35 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 907.2876109903941, - "y": 215.35439734173815, + "x": 1491.9260504518586, + "y": 472.24364315980574, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 127.38047790527344, - "height": 264.6, + "width": 125.32852172851562, + "height": 302.40000000000003, "seed": 515443238, "groupIds": [], "frameId": null, "roundness": null, "boundElements": [], - "updated": 1754948813569, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 28, "fontFamily": 6, - "text": "
b\n-------\nmod_time\nmod_user\nb_id (PK)\na_id (FK)\ny", - "rawText": "
b\n-------\nmod_time\nmod_user\nb_id (PK)\na_id (FK)\ny", + "text": "
b\n-------\nb_id (PK)\na_id (FK)\ny\nmod_time\nmod_user\n", + "rawText": "
b\n-------\nb_id (PK)\na_id (FK)\ny\nmod_time\nmod_user\n", "textAlign": "center", "verticalAlign": "top", "containerId": "jvtpsSfKxVMT4c7AFLiba", - "originalText": "
b\n-------\nmod_time\nmod_user\nb_id (PK)\na_id (FK)\ny", + "originalText": "
b\n-------\nb_id (PK)\na_id (FK)\ny\nmod_time\nmod_user\n", "autoResize": true, "lineHeight": 1.35 }, { "type": "arrow", - "version": 986, - "versionNonce": 1229347781, + "version": 1732, + "versionNonce": 1302351648, "index": "aQ", "isDeleted": false, "id": "BO_UhaJWAx-Ax-bvbriTA", @@ -532,12 +532,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 862.7050453468169, - "y": -348.4441176334223, + "x": 1446.3175067199022, + "y": -91.5549784257442, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 305.30700083827855, - "height": 1.148173945832582, + "width": 305.3070008382783, + "height": 1.1484677255783708, "seed": 540007590, "groupIds": [], "frameId": null, @@ -550,7 +550,7 @@ "id": "zfWxOAqa" } ], - "updated": 1754948583238, + "updated": 1758744681401, "link": null, "locked": false, "customData": { @@ -558,15 +558,13 @@ }, "startBinding": { "elementId": "rPi80OCsDY1oxdpcLjLi4", - "focus": 0.10853823050535973, - "gap": 1, - "fixedPoint": null + "focus": 0.10853823050535995, + "gap": 1.0000000000002274 }, "endBinding": { "elementId": "tPneSgy9-ER8wbYV3t7rf", - "focus": -0.1542324610144023, - "gap": 1, - "fixedPoint": null + "focus": -0.15423244283818158, + "gap": 1 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -577,8 +575,8 @@ 0 ], [ - -305.30700083827855, - -1.148173945832582 + -305.3070008382783, + -1.1484677255783708 ] ], "elbowed": false @@ -597,7 +595,7 @@ "opacity": 100, "angle": 0, "x": 654.6813132992596, - "y": -367.9142912336098, + "y": -367.9184435206989, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "width": 110.74046325683594, @@ -623,8 +621,8 @@ }, { "type": "arrow", - "version": 1381, - "versionNonce": 88859147, + "version": 2130, + "versionNonce": 2096039712, "index": "aS", "isDeleted": false, "id": "O1gYgG5p1CopkbDz1giYX", @@ -634,12 +632,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 858.8526446365463, - "y": 1248.5996463939005, + "x": 1442.4651060096319, + "y": 1519.1236403121113, "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "width": 299.32895110161417, - "height": 2.6746973492856796, + "backgroundColor": "#e9ecef", + "width": 299.32895110161405, + "height": 10.956798917842661, "seed": 720325478, "groupIds": [], "frameId": null, @@ -652,7 +650,7 @@ "type": "text" } ], - "updated": 1754949057127, + "updated": 1758744681402, "link": null, "locked": false, "customData": { @@ -660,15 +658,13 @@ }, "startBinding": { "elementId": "cXv2VOhBtuqYgae3ODq6c", - "focus": -0.008365668725549101, - "gap": 1, - "fixedPoint": null + "focus": -0.00836566872555028, + "gap": 1 }, "endBinding": { "elementId": "HP8dFO0w6b8dQ6Er-z3Xz", - "focus": 0.05895110255980628, - "gap": 1, - "fixedPoint": null + "focus": 0.029843641840676307, + "gap": 1.0000000000002274 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -679,8 +675,8 @@ 0 ], [ - -299.32895110161417, - 2.6746973492856796 + -299.32895110161405, + -10.956798917842661 ] ], "elbowed": false @@ -698,8 +694,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 631.2078452942353, - "y": 1015.9676576584678, + "x": 631.2078452942354, + "y": 1231.0389182963413, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "width": 155.9606475830078, @@ -725,8 +721,8 @@ }, { "type": "line", - "version": 127, - "versionNonce": 2108110054, + "version": 367, + "versionNonce": 1614913248, "index": "aU", "isDeleted": false, "id": "vKiMNdtNFZcAmvLj0rRGx", @@ -736,8 +732,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": -48.8823757856519, - "y": -42.93420706373922, + "x": 534.7300855874337, + "y": 213.95503875432837, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "width": 1590.987868284229, @@ -749,7 +745,7 @@ "type": 2 }, "boundElements": [], - "updated": 1727915590548, + "updated": 1758744681398, "link": null, "locked": false, "startBinding": null, @@ -766,12 +762,13 @@ 1590.987868284229, 0 ] - ] + ], + "polygon": false }, { "type": "arrow", - "version": 363, - "versionNonce": 1162827307, + "version": 1112, + "versionNonce": 1106231072, "index": "aV", "isDeleted": false, "id": "i9Z7cwn19gcx7vYmw0vuL", @@ -781,11 +778,11 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 444.60469361158533, - "y": 582.2315775452738, + "x": 1028.0631607523278, + "y": 839.1208233633414, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 1.1557339928377814, + "width": 1.266889807974394, "height": 478.98707712087696, "seed": 812761446, "groupIds": [], @@ -794,20 +791,18 @@ "type": 2 }, "boundElements": [], - "updated": 1754949057126, + "updated": 1758744681402, "link": null, "locked": false, "startBinding": { "elementId": "HENr2olXzuzzezNs3HaA4", - "focus": 0.08674691154783774, - "gap": 6.470982768450881, - "fixedPoint": null + "focus": 0.08674691154783735, + "gap": 6.470982768450881 }, "endBinding": { "elementId": "HP8dFO0w6b8dQ6Er-z3Xz", - "focus": -0.023015858566024266, - "gap": 5.1460016336288845, - "fixedPoint": null + "focus": -0.023015673089675795, + "gap": 5.1460016336288845 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -818,7 +813,7 @@ 0 ], [ - 1.1557339928377814, + 1.266889807974394, 478.98707712087696 ] ], @@ -826,8 +821,8 @@ }, { "type": "arrow", - "version": 423, - "versionNonce": 1832478891, + "version": 1172, + "versionNonce": 1478653728, "index": "aX", "isDeleted": false, "id": "9WMVthyuDpbCAfjaELGd8", @@ -837,12 +832,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 965.2318796042238, - "y": 577.0846341134422, + "x": 1549.237243028767, + "y": 833.9738799315098, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 2.5712294357098244, - "height": 473.3896960153784, + "width": 2.7824361996333664, + "height": 473.38969601537815, "seed": 1501286074, "groupIds": [], "frameId": null, @@ -850,20 +845,18 @@ "type": 2 }, "boundElements": [], - "updated": 1754949057128, + "updated": 1758744681403, "link": null, "locked": false, "startBinding": { "elementId": "jvtpsSfKxVMT4c7AFLiba", - "focus": 0.03826116902876437, - "gap": 1.3888156278566157, - "fixedPoint": null + "focus": 0.038526562110838374, + "gap": 1.3888156278566157 }, "endBinding": { "elementId": "cXv2VOhBtuqYgae3ODq6c", - "focus": -0.07257358327285979, - "gap": 1, - "fixedPoint": null + "focus": -0.07257427349806503, + "gap": 1.0000000000002274 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -874,16 +867,16 @@ 0 ], [ - -2.5712294357098244, - 473.3896960153784 + -2.7824361996333664, + 473.38969601537815 ] ], "elbowed": false }, { "type": "arrow", - "version": 293, - "versionNonce": 540064997, + "version": 1043, + "versionNonce": 453455648, "index": "aY", "isDeleted": false, "id": "BBN5edyBWznLdB7RjCxyA", @@ -893,12 +886,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 857.5301025505701, - "y": 393.7846113679031, + "x": 1441.1425639236559, + "y": 650.6373730668962, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 293.2057157747365, - "height": 0.038224894892039174, + "width": 293.20571577473663, + "height": 0.06444296667461913, "seed": 988830566, "groupIds": [], "frameId": null, @@ -911,7 +904,7 @@ "id": "WW6l9VVG" } ], - "updated": 1754948858850, + "updated": 1758744681403, "link": null, "locked": false, "customData": { @@ -919,15 +912,13 @@ }, "startBinding": { "elementId": "jvtpsSfKxVMT4c7AFLiba", - "focus": -0.0038210336795078043, - "gap": 3.6570308208117694, - "fixedPoint": null + "focus": -0.0038210434833155704, + "gap": 3.657030820811542 }, "endBinding": { "elementId": "HENr2olXzuzzezNs3HaA4", - "focus": -0.036597099884290454, - "gap": 1, - "fixedPoint": null + "focus": -0.03659709988428956, + "gap": 1 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -938,8 +929,8 @@ 0 ], [ - -293.2057157747365, - 0.038224894892039174 + -293.20571577473663, + 0.06444296667461913 ] ], "elbowed": false @@ -957,8 +948,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 655.5570130347838, - "y": 375.98696338208765, + "x": 655.557013034784, + "y": 374.8821877418291, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "width": 110.74046325683594, @@ -984,8 +975,8 @@ }, { "type": "arrow", - "version": 554, - "versionNonce": 1671208197, + "version": 1303, + "versionNonce": 102623008, "index": "aa", "isDeleted": false, "id": "xPf51myhWpmHyctO8gD4x", @@ -995,12 +986,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 338.7429536325356, - "y": 392.00095225398957, + "x": 922.3554150056211, + "y": 648.8901980720573, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "width": 287.1053987422777, - "height": 798.7524746782004, + "height": 798.7524746782003, "seed": 1544236326, "groupIds": [], "frameId": null, @@ -1013,7 +1004,7 @@ "id": "48I2Dkoy" } ], - "updated": 1754949804104, + "updated": 1758744681404, "link": null, "locked": false, "customData": { @@ -1021,15 +1012,13 @@ }, "startBinding": { "elementId": "HENr2olXzuzzezNs3HaA4", - "focus": 0.4511744623232074, - "gap": 4.999999999999943, - "fixedPoint": null + "focus": 0.451174462323208, + "gap": 5 }, "endBinding": { "elementId": "HP8dFO0w6b8dQ6Er-z3Xz", - "focus": -0.4024209265827109, - "gap": 5, - "fixedPoint": null + "focus": -0.402420926582711, + "gap": 5 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -1041,11 +1030,11 @@ ], [ -287.1053987422777, - 312.03191173946357 + 312.0319117394635 ], [ - -4.8006932409015235, - 798.7524746782004 + -4.800693240901467, + 798.7524746782003 ] ], "elbowed": false @@ -1063,11 +1052,11 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": -89.93699955310149, + "x": -89.4130279954843, "y": 609.5328639934531, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 283.14910888671875, + "width": 282.1011657714844, "height": 189.00000000000003, "seed": 1345294886, "groupIds": [], @@ -1090,8 +1079,8 @@ }, { "type": "arrow", - "version": 1601, - "versionNonce": 1628254693, + "version": 2096, + "versionNonce": 887529248, "index": "ac", "isDeleted": false, "id": "Ejrady7_F26x59KCnQm8d", @@ -1101,12 +1090,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 335.38219565178395, - "y": 1244.269294876459, + "x": 918.9946570248694, + "y": 1496.1162351404553, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 353.472075952014, - "height": 141.53382450484986, + "width": 353.47207595201394, + "height": 141.5338245048497, "seed": 333390054, "groupIds": [], "frameId": null, @@ -1119,7 +1108,7 @@ "id": "FaC5v5iR" } ], - "updated": 1754949819020, + "updated": 1758744681405, "link": null, "locked": false, "customData": { @@ -1127,15 +1116,13 @@ }, "startBinding": { "elementId": "HP8dFO0w6b8dQ6Er-z3Xz", - "focus": 0.10225909254202926, - "gap": 3.5600647398501906, - "fixedPoint": null + "focus": 0.10225909254202985, + "gap": 3.5600647398503042 }, "endBinding": { "elementId": "HP8dFO0w6b8dQ6Er-z3Xz", - "focus": -0.8259143012677729, - "gap": 4.950333241940314, - "fixedPoint": null + "focus": -0.825914301267772, + "gap": 4.9503332419403705 }, "lastCommittedPoint": null, "startArrowhead": null, @@ -1143,15 +1130,15 @@ "points": [ [ 0, - -5.042305554071309 + 0 ], [ - -353.472075952014, - 47.944763915763815 + -353.47207595201394, + 52.98706946983512 ], [ - -1.390268502090123, - 136.49151895077856 + -1.3902685020900662, + 141.5338245048497 ] ], "elbowed": false @@ -1169,11 +1156,11 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": -221.17336880533645, - "y": 1123.8771041804334, + "x": -154.3944609887066, + "y": 1235.5140587922228, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", - "width": 273.13311767578125, + "width": 272.6091613769531, "height": 113.4, "seed": 1010913190, "groupIds": [], @@ -1196,8 +1183,8 @@ }, { "type": "text", - "version": 388, - "versionNonce": 1552724613, + "version": 628, + "versionNonce": 183826144, "index": "ag", "isDeleted": false, "id": "kn3WsXHO", @@ -1207,8 +1194,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": -39.25997794834933, - "y": -199.8680659040105, + "x": 544.3524834247362, + "y": 57.021179914057086, "strokeColor": "#1971c2", "backgroundColor": "transparent", "width": 270.3639514731369, @@ -1218,7 +1205,7 @@ "frameId": null, "roundness": null, "boundElements": [], - "updated": 1754949490445, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 36, @@ -1234,8 +1221,8 @@ }, { "type": "text", - "version": 478, - "versionNonce": 1290640683, + "version": 718, + "versionNonce": 317922016, "index": "ah", "isDeleted": false, "id": "uVJ7NM4f", @@ -1245,8 +1232,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": -41.86434236811351, - "y": 34.088158010816244, + "x": 541.7481190049721, + "y": 290.97740382888384, "strokeColor": "#1971c2", "backgroundColor": "transparent", "width": 270.3639514731369, @@ -1256,7 +1243,7 @@ "frameId": null, "roundness": null, "boundElements": [], - "updated": 1754949502482, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 36, @@ -1272,8 +1259,8 @@ }, { "type": "rectangle", - "version": 148, - "versionNonce": 1638070650, + "version": 388, + "versionNonce": 1066188512, "index": "am", "isDeleted": false, "id": "x7CuDh-Mp7TNl7l5vQaSc", @@ -1283,8 +1270,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 322.00150636339504, - "y": -86.26176338956259, + "x": 905.6139677364806, + "y": 170.627482428505, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", "width": 258.23223570190635, @@ -1309,7 +1296,7 @@ "type": "arrow" } ], - "updated": 1728330182457, + "updated": 1758744681398, "link": null, "locked": false, "customData": { @@ -1318,8 +1305,8 @@ }, { "type": "text", - "version": 86, - "versionNonce": 1952622970, + "version": 326, + "versionNonce": 1901191904, "index": "an", "isDeleted": false, "id": "WQK7DSMb", @@ -1329,8 +1316,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 365.21325867479743, - "y": -65.30041156980522, + "x": 948.825720047883, + "y": 191.58883424826237, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", "width": 171.80873107910156, @@ -1342,7 +1329,7 @@ "frameId": null, "roundness": null, "boundElements": [], - "updated": 1728341083453, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 28, @@ -1358,8 +1345,8 @@ }, { "type": "arrow", - "version": 35, - "versionNonce": 1794213, + "version": 528, + "versionNonce": 419805984, "index": "ao", "isDeleted": false, "id": "73M37a7X001oZycu-OGrM", @@ -1369,12 +1356,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 445.0517663287328, - "y": -93.19417240169423, + "x": 1028.6642277018184, + "y": 163.69507341637336, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", - "width": 0, - "height": 43.32755632582325, + "width": 5.684341886080802e-14, + "height": 43.32755632582327, "seed": 1786945274, "groupIds": [ "ZrfqJU2aDhETO8gi_Mw_v" @@ -1384,14 +1371,13 @@ "type": 2 }, "boundElements": [], - "updated": 1754948583239, + "updated": 1758744681405, "link": null, "locked": false, "startBinding": { "elementId": "x7CuDh-Mp7TNl7l5vQaSc", - "focus": -0.046979865771813255, - "gap": 6.932409012131643, - "fixedPoint": null + "focus": -0.046979865771814074, + "gap": 6.932409012131643 }, "endBinding": null, "lastCommittedPoint": null, @@ -1403,16 +1389,16 @@ 0 ], [ - 0, - -43.32755632582325 + 5.684341886080802e-14, + -43.32755632582327 ] ], "elbowed": false }, { "type": "arrow", - "version": 135, - "versionNonce": 697435141, + "version": 628, + "versionNonce": 1658935072, "index": "ap", "isDeleted": false, "id": "NUtdy3OrdfhJ9-YLFFiFs", @@ -1422,8 +1408,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 446.3862550635681, - "y": -5.539059750047841, + "x": 1029.9987164366537, + "y": 251.35018606801975, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", "width": 0, @@ -1437,14 +1423,13 @@ "type": 2 }, "boundElements": [], - "updated": 1754948583239, + "updated": 1758744681405, "link": null, "locked": false, "startBinding": { "elementId": "x7CuDh-Mp7TNl7l5vQaSc", - "focus": 0.03664429530201499, - "gap": 1, - "fixedPoint": null + "focus": 0.03664429530201405, + "gap": 1 }, "endBinding": null, "lastCommittedPoint": null, @@ -1464,8 +1449,8 @@ }, { "type": "rectangle", - "version": 195, - "versionNonce": 1800278778, + "version": 435, + "versionNonce": 2137528032, "index": "aq", "isDeleted": false, "id": "YDN2CFYJD-5V9zKvieZpl", @@ -1475,8 +1460,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 843.5226683866961, - "y": -95.32588817292479, + "x": 1427.1351297597816, + "y": 161.5633576451428, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", "width": 258.23223570190635, @@ -1501,7 +1486,7 @@ "type": "arrow" } ], - "updated": 1728341085248, + "updated": 1758744681398, "link": null, "locked": false, "customData": { @@ -1510,8 +1495,8 @@ }, { "type": "text", - "version": 132, - "versionNonce": 1399170022, + "version": 372, + "versionNonce": 1275214560, "index": "ar", "isDeleted": false, "id": "NBLRrH2O", @@ -1521,8 +1506,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 885.9784163646024, - "y": -74.36453635316742, + "x": 1469.590877737688, + "y": 182.52470946490018, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", "width": 173.32073974609375, @@ -1534,7 +1519,7 @@ "frameId": null, "roundness": null, "boundElements": [], - "updated": 1728341088843, + "updated": 1758744681398, "link": null, "locked": false, "fontSize": 28, @@ -1550,8 +1535,8 @@ }, { "type": "arrow", - "version": 129, - "versionNonce": 1729968997, + "version": 622, + "versionNonce": 652833568, "index": "as", "isDeleted": false, "id": "NKl5DUMgqWBZXuYbbht9J", @@ -1561,12 +1546,12 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 966.5729283520338, - "y": -102.25829718505645, + "x": 1550.1853897251194, + "y": 154.63094863301117, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", - "width": 0, - "height": 43.32755632582324, + "width": 1.1368683772161603e-13, + "height": 43.32755632582327, "seed": 2073112614, "groupIds": [ "EAGsqV8poBWw0UWL68csZ" @@ -1576,14 +1561,13 @@ "type": 2 }, "boundElements": [], - "updated": 1754948583239, + "updated": 1758744681405, "link": null, "locked": false, "startBinding": { "elementId": "YDN2CFYJD-5V9zKvieZpl", - "focus": -0.046979865771813255, - "gap": 6.932409012131643, - "fixedPoint": null + "focus": -0.04697986577181222, + "gap": 6.932409012131615 }, "endBinding": null, "lastCommittedPoint": null, @@ -1595,16 +1579,16 @@ 0 ], [ - 0, - -43.32755632582324 + 1.1368683772161603e-13, + -43.32755632582327 ] ], "elbowed": false }, { "type": "arrow", - "version": 229, - "versionNonce": 1896515269, + "version": 722, + "versionNonce": 1804472096, "index": "at", "isDeleted": false, "id": "YOuKvNLiB_LzXQXZW7Wxo", @@ -1614,11 +1598,11 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 967.9074170868691, - "y": -14.603184533410046, + "x": 1551.5198784599547, + "y": 242.28606128465753, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", - "width": 0, + "width": 1.1368683772161603e-13, "height": 47.52686308492207, "seed": 628477242, "groupIds": [ @@ -1629,14 +1613,13 @@ "type": 2 }, "boundElements": [], - "updated": 1754948583239, + "updated": 1758744681406, "link": null, "locked": false, "startBinding": { "elementId": "YDN2CFYJD-5V9zKvieZpl", - "focus": 0.03664429530201499, - "gap": 1, - "fixedPoint": null + "focus": 0.036644295302015725, + "gap": 1 }, "endBinding": null, "lastCommittedPoint": null, @@ -1648,7 +1631,7 @@ 0 ], [ - 0, + 1.1368683772161603e-13, 47.52686308492207 ] ], @@ -1656,8 +1639,8 @@ }, { "type": "arrow", - "version": 146, - "versionNonce": 1623335883, + "version": 386, + "versionNonce": 1759604448, "index": "au", "isDeleted": false, "id": "hu56RFp_O0eSyAIj111ZG", @@ -1667,8 +1650,8 @@ "roughness": 1, "opacity": 100, "angle": 0, - "x": 718.8980966715407, - "y": 446.6884033517412, + "x": 1302.5105580446261, + "y": 703.5776491698089, "strokeColor": "#1971c2", "backgroundColor": "#ffffff", "width": 4.431314623338267, @@ -1680,7 +1663,7 @@ "type": 2 }, "boundElements": [], - "updated": 1754949671404, + "updated": 1758744681398, "link": null, "locked": false, "startBinding": null, @@ -1703,8 +1686,8 @@ { "id": "DeszCljnW3zprs70KW2Bb", "type": "rectangle", - "x": 1308.5812732917261, - "y": 89.94928560621395, + "x": 1892.1937346648117, + "y": 346.83853142428154, "width": 461.18404569269643, "height": 653, "angle": 0, @@ -1720,8 +1703,8 @@ "index": "av", "roundness": null, "seed": 1874561419, - "version": 272, - "versionNonce": 141093125, + "version": 512, + "versionNonce": 1821414112, "isDeleted": false, "boundElements": [ { @@ -1729,16 +1712,16 @@ "id": "Evygu83Z" } ], - "updated": 1754949758280, + "updated": 1758744681398, "link": null, "locked": false }, { "id": "Evygu83Z", "type": "text", - "x": 1313.5812732917261, - "y": 95.14928560621394, - "width": 447.685791015625, + "x": 1897.1937346648117, + "y": 352.03853142428153, + "width": 447.63787841796875, "height": 642.6, "angle": 0, "strokeColor": "#1e1e1e", @@ -1753,11 +1736,11 @@ "index": "avV", "roundness": null, "seed": 1051274379, - "version": 164, - "versionNonce": 1173542341, + "version": 404, + "versionNonce": 1991980768, "isDeleted": false, - "boundElements": null, - "updated": 1754949757948, + "boundElements": [], + "updated": 1758744681398, "link": null, "locked": false, "text": "Normal users interact only with the\nmain tables. They have the same\nnames and interfaces as the\nexisting tables, and two extra\ncolumns.\n\n1 row ⬅➡ 1 collection item. No \nhistory, latest value only.\n\nTriggers on tables record actions in\nhistory tables. This denormalizes\nthe data (1 record duplicated for\nevery collection item).\n\nReferential integrity is\nautomatically enforced because the\nmain tables retain FK relationships.", @@ -1774,8 +1757,8 @@ { "id": "ZnkYWWcODhwf4txo7ad8p", "type": "rectangle", - "x": 1311.9295803250047, - "y": -454.97260799773994, + "x": 1895.5420416980903, + "y": -198.08336217967235, "width": 461.18404569269643, "height": 200, "angle": 0, @@ -1791,8 +1774,8 @@ "index": "avX", "roundness": null, "seed": 232379205, - "version": 481, - "versionNonce": 1023886149, + "version": 721, + "versionNonce": 737237728, "isDeleted": false, "boundElements": [ { @@ -1800,15 +1783,15 @@ "id": "F1yDIOn9" } ], - "updated": 1754949556344, + "updated": 1758744681398, "link": null, "locked": false }, { "id": "F1yDIOn9", "type": "text", - "x": 1316.9295803250047, - "y": -430.57260799773996, + "x": 1900.5420416980903, + "y": -173.68336217967237, "width": 419.66583251953125, "height": 151.20000000000002, "angle": 0, @@ -1824,11 +1807,11 @@ "index": "avY", "roundness": null, "seed": 1016160261, - "version": 253, - "versionNonce": 1628712485, + "version": 493, + "versionNonce": 516095712, "isDeleted": false, - "boundElements": null, - "updated": 1754949585943, + "boundElements": [], + "updated": 1758744681398, "link": null, "locked": false, "text": "All users interact with the tables.\n\n1 row ⬅➡ 1 collection item. No\nhistory, latest value only.", @@ -1845,8 +1828,8 @@ { "id": "P1OrraxFxBntBybZIMhag", "type": "rectangle", - "x": 1307.4951183471896, - "y": 961.837993913957, + "x": 1891.1075797202752, + "y": 1218.7272397320246, "width": 457.56553802218565, "height": 956, "angle": 0, @@ -1862,8 +1845,8 @@ "index": "avd", "roundness": null, "seed": 1863617931, - "version": 583, - "versionNonce": 656808523, + "version": 823, + "versionNonce": 264250080, "isDeleted": false, "boundElements": [ { @@ -1871,16 +1854,16 @@ "id": "T1mKftlv" } ], - "updated": 1754949771748, + "updated": 1758744681398, "link": null, "locked": false }, { "id": "T1mKftlv", "type": "text", - "x": 1312.4951183471896, - "y": 967.337993913957, - "width": 444.98974609375, + "x": 1896.1075797202752, + "y": 1224.2272397320246, + "width": 443.94183349609375, "height": 945.0000000000001, "angle": 0, "strokeColor": "#1e1e1e", @@ -1895,11 +1878,11 @@ "index": "avh", "roundness": null, "seed": 2124587819, - "version": 242, - "versionNonce": 533442507, + "version": 482, + "versionNonce": 762889952, "isDeleted": false, - "boundElements": null, - "updated": 1754949771119, + "boundElements": [], + "updated": 1758744681398, "link": null, "locked": false, "text": "Normal users do not interact\ndirectly with history tables.\n\nHistory tables are append-only.\nEvery past state of every collection\nitem is stored in its history table.\n\n1 collection item ⬅➡ many rows. \n\"Item\" means \"has same collection\nitem id\". There is a separate,\nunique PK, the history id\n(xxx_hx_id), for each row in the\nhistory table.\n\nThe history table and the main\ntable are maintained such that the\nlatest history record for each\n(undeleted) item is the same as the\nmain record for it; deleted items\nare of course not present in\nprimary table.\n\nHistory tables contain additional \ninformation specifically related to \nhistory-tracking.", @@ -1918,7 +1901,7 @@ "theme": "light", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", - "currentItemBackgroundColor": "#fff9db", + "currentItemBackgroundColor": "#e9ecef", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 0.5, "currentItemStrokeStyle": "solid", @@ -1930,10 +1913,10 @@ "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "currentItemArrowType": "round", - "scrollX": 900.889407154292, - "scrollY": 2363.9270172941456, + "scrollX": 207.0591079237839, + "scrollY": -88.17503256305879, "zoom": { - "value": 0.451013 + "value": 0.400951 }, "currentItemRoundness": "sharp", "gridSize": 20, @@ -1952,12 +1935,12 @@ }, "objectsSnapModeEnabled": false, "activeTool": { - "type": "selection", + "type": "hand", "customType": null, "locked": false, + "fromSelection": false, "lastActiveTool": null } }, - "prevTextMode": "parsed", "files": {} } \ No newline at end of file diff --git a/pycds/orm/functions/bookmarking.sql b/pycds/orm/functions/bookmarking.sql new file mode 100644 index 00000000..61735947 --- /dev/null +++ b/pycds/orm/functions/bookmarking.sql @@ -0,0 +1,539 @@ +----------------- +CREATE OR REPLACE FUNCTION hxtk_make_query_latest_undeleted_hx_ids( + collection_name text, + collection_id_name text, + where_condition text = 'TRUE' +) + RETURNS text + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This function returns text containing a query that returns the history id's for the LU records, + -- given a collection spec and a where condition. +DECLARE + hx_table_name text := hxtk_hx_table_name(collection_name); + hx_id_name text := hxtk_hx_id_name(collection_name); + q text := format( + 'SELECT hx.%1$I AS %1$I, max(hx.%2$I) AS %2$I ' || + 'FROM %3$I hx ' || + 'WHERE %4$s ' || + 'GROUP BY hx.%1$I ' || + 'HAVING NOT bool_or(hx.deleted) ', + collection_id_name, + hx_id_name, + hx_table_name, + where_condition + ); +BEGIN + -- RAISE NOTICE '%', q; + RETURN q; +END; +$BODY$; + + +DO LANGUAGE 'plpgsql' +$$ + BEGIN + RAISE NOTICE 'hxtk_make_query_latest_undeleted_hx_ids: %', + hxtk_make_query_latest_undeleted_hx_ids('meta_network', 'network_id'); + END; +$$; + + + +----------------- +CREATE OR REPLACE FUNCTION hxtk_get_latest_undeleted_hx_ids( + collection_name text, + collection_id_name text, + where_condition text = 'true' +) + RETURNS SETOF RECORD + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This function returns the history id's for the LU records, given a collection and a where condition. +BEGIN + RETURN QUERY EXECUTE hxtk_make_query_latest_undeleted_hx_ids( + collection_name, collection_id_name, where_condition); +END; +$BODY$; + + +DO LANGUAGE 'plpgsql' +$$ + DECLARE + r record; + BEGIN + RAISE NOTICE '(network_id, meta_network_hx_id)'; + FOR r IN + SELECT * + FROM + hxtk_get_latest_undeleted_hx_ids('meta_network', 'network_id') + AS t(network_id int, meta_network_hx_id int) + ORDER BY network_id + LOOP + RAISE NOTICE '%', r; + END LOOP; + END ; +$$; + + + +----------------- +CREATE OR REPLACE FUNCTION hxtk_get_latest_undeleted_hx_records( + collection_name text, + collection_id_name text, + where_condition text = 'true' +) + RETURNS SETOF RECORD + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This function returns the full history records for the LU records, given a collection and a where condition. +DECLARE + hx_table_name text := hxtk_hx_table_name(collection_name); + hx_id_name text := hxtk_hx_id_name(collection_name); + hx_ids_query text := hxtk_make_query_latest_undeleted_hx_ids( + collection_name, collection_id_name, where_condition); +BEGIN + RETURN QUERY EXECUTE + format( + 'SELECT * FROM %1$I WHERE %2$I IN (SELECT t.%2$I FROM (%3$s) AS t)', + hx_table_name, hx_id_name, hx_ids_query + ); +END; +$BODY$; + + +DO LANGUAGE 'plpgsql' +$$ + DECLARE + r record; + BEGIN + RAISE NOTICE 'LU(meta_network_hx)'; + RAISE NOTICE '(meta_network_hx_id, network_id, network_name)'; + FOR r IN + SELECT + meta_network_hx_id, + network_id, + network_name + FROM + hxtk_get_latest_undeleted_hx_records('meta_network', 'network_id') + AS t(network_id integer, + network_name varchar(255), + description varchar(255), + virtual varchar(255), + publish boolean, + col_hex varchar(7), + contact_id integer, + mod_time timestamp, + mod_user varchar(64), + deleted boolean, + meta_network_hx_id integer) + LOOP + RAISE NOTICE '%', r; + END LOOP; + END ; +$$; + + + +----------------- +-- Can this and should this be used as a column in table bookmark_associations? +CREATE TYPE history_tuple AS ( + obs_raw_hx_id bigint, + meta_history_hx_id int, + meta_station_hx_id int, + meta_network_hx_id int, + meta_vars_hx_id int +); + + +CREATE OR REPLACE FUNCTION hxtk_current_hx_tuple() + RETURNS history_tuple + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This function returns the current history tuple. +DECLARE + result history_tuple; +BEGIN + SELECT + (SELECT max(obs_raw_hx.obs_raw_hx_id) FROM obs_raw_hx) AS obs_raw_hx_id, + (SELECT max(meta_history_hx.meta_history_hx_id) FROM meta_history_hx) AS meta_history_hx_id, + (SELECT max(meta_station_hx.meta_station_hx_id) FROM meta_station_hx) AS meta_station_hx_id, + (SELECT max(meta_network_hx.meta_network_hx_id) FROM meta_network_hx) AS meta_network_hx_id, + (SELECT max(meta_vars_hx.meta_vars_hx_id) FROM meta_vars_hx) AS meta_vars_hx_id + INTO STRICT result; + RETURN result; +END; +$BODY$; + + +DO LANGUAGE 'plpgsql' +$$ + BEGIN + RAISE NOTICE '(obs_raw_hx_id, meta_history_hx_id, meta_station_hx_id int, meta_network_hx_id int, meta_vars_hx_id int)'; + RAISE NOTICE '%', hxtk_current_hx_tuple(); + END ; +$$; + + + +----------------- +CREATE OR REPLACE FUNCTION hxtk_is_valid_history_tuple(hx_tuple history_tuple) + RETURNS boolean + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This function returns true if and only if the provided history tuple implies a valid + -- history subset. See documentation for definitions of validity. +DECLARE + obs_raw_ok boolean; + meta_history_ok boolean; + meta_station_ok boolean; + meta_network_ok boolean; + meta_vars_ok boolean; + t_start timestamp; + t_end timestamp; +BEGIN + RAISE NOTICE 'hxtk_is_valid_history_tuple %', hx_tuple; + + IF hx_tuple IS NULL THEN + RETURN FALSE; + END IF; + + -- If this tuple is the current point in history, we know it is valid. + IF + hx_tuple = hxtk_current_hx_tuple() + THEN + RAISE NOTICE '= current hx tuple'; + RETURN TRUE; + END IF; + + -- Warning: The query against `obs_raw_hx` could take a long time. + + -- This query is likely more efficient if the correct indexes exist and are selected by the query planner. + -- TODO: Determine which query is best. + + RAISE NOTICE 'Checking in detail'; + t_start := clock_timestamp(); + RAISE NOTICE 'Starting obs_raw_hx at %', t_start; + SELECT + max(obs_raw_hx.meta_history_hx_id) <= hx_tuple.meta_history_hx_id AND + max(obs_raw_hx.meta_vars_hx_id) <= hx_tuple.meta_vars_hx_id + FROM + obs_raw_hx + WHERE + obs_raw_hx.obs_raw_hx_id <= hx_tuple.obs_raw_hx_id + INTO STRICT obs_raw_ok; + t_end := clock_timestamp(); + RAISE NOTICE 'Finished obs_raw_hx at %; time elapsed %', t_end, t_end - t_start; + + SELECT + max(meta_history_hx.meta_station_hx_id) <= hx_tuple.meta_station_hx_id + FROM + meta_history_hx + WHERE + meta_history_hx.meta_history_hx_id <= hx_tuple.meta_history_hx_id + INTO STRICT meta_history_ok; + + SELECT + max(meta_station_hx.meta_network_hx_id) <= hx_tuple.meta_network_hx_id + FROM + meta_station_hx + WHERE + meta_station_hx.meta_station_hx_id <= hx_tuple.meta_station_hx_id + INTO STRICT meta_station_ok; + + SELECT + max(meta_vars_hx.meta_network_hx_id) <= hx_tuple.meta_network_hx_id + FROM + meta_vars_hx + WHERE + meta_vars_hx.meta_vars_hx_id <= hx_tuple.meta_vars_hx_id + INTO STRICT meta_vars_ok; + + RETURN obs_raw_ok AND meta_history_ok AND meta_station_ok AND meta_network_ok AND meta_vars_ok; + + -- This is probably a slower query. If used, reformulate as a series of separate queries like above. +-- RETURN QUERY SELECT +-- -- It's tempting to DRY up the sub-queries below into a generic make-query function, +-- -- but it's probably more work than it's worth. Advantage would be getting the pattern +-- -- right just once, and easy extension to additional cases. +-- (SELECT +-- bool_and( +-- obs_raw_hx.meta_history_hx_id <= hx_tuple.meta_history_hx_id AND +-- obs_raw_hx.meta_vars_hx_id <= hx_tuple.meta_vars_hx_id +-- ) +-- FROM +-- obs_raw_hx +-- WHERE +-- obs_raw_hx.obs_raw_hx_id <= hx_tuple.obs_raw_hx_id) +-- AND (SELECT +-- bool_and(meta_history_hx.meta_station_hx_id <= hx_tuple.meta_station_hx_id) +-- FROM +-- meta_history_hx +-- WHERE +-- meta_history_hx.meta_history_hx_id <= hx_tuple.meta_history_hx_id) +-- AND (SELECT +-- bool_and(meta_station_hx.meta_network_hx_id <= hx_tuple.meta_network_hx_id) +-- FROM +-- meta_station_hx +-- WHERE +-- meta_station_hx.meta_station_hx_id <= hx_tuple.meta_station_hx_id) +-- AND (SELECT +-- bool_and(meta_vars_hx.meta_network_hx_id <= hx_tuple.meta_network_hx_id) +-- FROM +-- meta_vars_hx +-- WHERE +-- meta_vars_hx.meta_vars_hx_id <= hx_tuple.meta_vars_hx_id); + +END; +$BODY$; + + +DO LANGUAGE 'plpgsql' +$$ + DECLARE + curr_hx_tuple history_tuple := hxtk_current_hx_tuple(); + bad_hx_tuple history_tuple := + (curr_hx_tuple.obs_raw_hx_id, + curr_hx_tuple.meta_history_hx_id - 1, + curr_hx_tuple.meta_station_hx_id - 1, + curr_hx_tuple.meta_network_hx_id - 1, + curr_hx_tuple.meta_vars_hx_id - 1 + )::history_tuple; + BEGIN + RAISE NOTICE 'current: %', hxtk_is_valid_history_tuple(curr_hx_tuple); + RAISE NOTICE 'bad: %', hxtk_is_valid_history_tuple(bad_hx_tuple); + END ; +$$; + + +----------------- +CREATE TABLE IF NOT EXISTS bookmark_labels ( + bookmark_label_id int PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, + network_id int REFERENCES meta_network (network_id), + label text NOT NULL, + comment text, + mod_time timestamp NOT NULL DEFAULT now(), + mod_user text NOT NULL DEFAULT current_user +); + + +CREATE TABLE IF NOT EXISTS bookmark_associations ( + bookmark_association_id int PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, + bookmark_label_id int REFERENCES bookmark_labels (bookmark_label_id), + role text NOT NULL, -- Should be enum type + bracket_begin_id int REFERENCES bookmark_associations (bookmark_association_id), + comment text, + hx_tuple history_tuple NOT NULL DEFAULT hxtk_current_hx_tuple(), + mod_time timestamp NOT NULL DEFAULT now(), + mod_user text NOT NULL DEFAULT current_user +); + + +-- Trigger functions + +CREATE OR REPLACE FUNCTION hxtk_validate_history_tuple() + RETURNS trigger + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This trigger function validates the history tuple provided to table bookmark_associations. + -- + -- Usage: + -- CREATE TRIGGER t100_validate_history_tuple + -- BEFORE INSERT OR UPDATE + -- ON bookmark_associations + -- FOR EACH ROW + -- EXECUTE PROCEDURE hxtk_validate_history_tuple() +DECLARE +BEGIN + RAISE NOTICE 'hxtk_validate_history_tuple: NEW = %', NEW; + IF NOT hxtk_is_valid_history_tuple(NEW.hx_tuple) THEN + RAISE 'Invalid history tuple'; + END IF; + RETURN NEW; +END; +$BODY$; + + +CREATE OR REPLACE FUNCTION hxtk_bm_check_bracket_end() + RETURNS trigger + LANGUAGE 'plpgsql' +AS +$BODY$ + -- This trigger function checks that a bracket-end insert to table references an open + -- bracket (bracket_begin_id), or if it references no bracket and there is only one + -- open one, it provides that value. + -- + -- Usage: + -- CREATE TRIGGER t200_check_bracket_end + -- BEFORE INSERT OR UPDATE + -- ON bookmark_associations + -- FOR EACH ROW + -- EXECUTE PROCEDURE hxtk_check_bm_bracket_end() +DECLARE + bracket_begin_q text := + format( + 'SELECT count(*), max(bookmark_association_id) FROM %I.%I ' || + 'WHERE role = ''bracket_begin''', + TG_TABLE_SCHEMA, TG_TABLE_NAME + ); + bracket_end_q text := + format( + 'SELECT count(*) FROM %I.%I ' || + 'WHERE role = ''bracket_end'' ' || + ' AND bracket_end_id = NEW.bracket_begin_id', + TG_TABLE_SCHEMA, TG_TABLE_NAME + ); + max_ba_id int; + ba_id_count int; +BEGIN + IF NEW.bracket_begin_id IS NULL THEN + -- Check if we have one open bracket. Use it if so. + EXECUTE bracket_begin_q INTO STRICT ba_id_count, max_ba_id; + IF ba_id_count = 1 THEN + NEW.bracket_begin_id := max_ba_id; + ELSE + RAISE EXCEPTION 'bracket_begin_id is unspecified, and there are % > 1 open brackets.', ba_id_count; + END IF; + ELSE + -- Check whether this bracket is open. Error if it is not. + EXECUTE bracket_end_q INTO ba_id_count; + IF ba_id_count > 0 THEN + RAISE EXCEPTION 'The bracket with id % is already closed.', NEW.bracket_begin_id; + END IF; + END IF; + RETURN NEW; +END; +$BODY$; + + +DROP TRIGGER IF EXISTS t100_validate_history_tuple ON bookmark_associations; +CREATE TRIGGER t100_validate_history_tuple + BEFORE INSERT OR UPDATE + ON bookmark_associations + FOR EACH ROW +EXECUTE PROCEDURE hxtk_validate_history_tuple(); + + +DROP TRIGGER IF EXISTS t200_check_bracket_end ON bookmark_associations; +CREATE TRIGGER t200_check_bracket_end + BEFORE INSERT OR UPDATE + ON bookmark_associations + FOR EACH ROW + WHEN ( NEW.role = 'bracket_end' ) +EXECUTE PROCEDURE hxtk_bm_check_bracket_end(); + + +----------------- +BEGIN; -- begin transaction for tests; roll back at end +DO LANGUAGE 'plpgsql' +$$ + DECLARE + bookmark_alpha int; + nw_id int; + open_br_id int; + br_hx_tuple history_tuple; + close_br_begin_id int; + r record; + test_id int; + BEGIN + -- Create a bookmark label + INSERT INTO bookmark_labels(network_id, label) + VALUES (34, 'Alpha') + RETURNING bookmark_label_id + INTO STRICT bookmark_alpha; + + RAISE NOTICE 'bookmark_labels'; + FOR r IN SELECT * FROM bookmark_labels + LOOP + RAISE NOTICE '%', r; + END LOOP; + + -- Create a singleton bookmark at current history point. + RAISE NOTICE 'Create singleton bookmark'; + INSERT INTO bookmark_associations(bookmark_label_id, role) + VALUES (bookmark_alpha, 'singleton'); + + RAISE NOTICE 'bookmark_associations'; + FOR r IN SELECT * FROM bookmark_associations + LOOP + RAISE NOTICE '%', r; + END LOOP; + + -- TEST 1: Bracket with manually provided values. + test_id = 1; + RAISE NOTICE 'TEST % - BEGIN', test_id; + + -- Open a bracket bookmark at current history point. + RAISE NOTICE 'TEST % - Open bracket', test_id; + INSERT INTO bookmark_associations(bookmark_label_id, role, hx_tuple) + VALUES (bookmark_alpha, 'bookmark_begin', hxtk_current_hx_tuple()) + RETURNING bookmark_association_id + INTO STRICT open_br_id; + + -- Insert some gunk. + RAISE NOTICE 'TEST % - Add new network', test_id; + INSERT INTO meta_network(network_name) + VALUES ('Rod Test 1') + RETURNING network_id INTO STRICT nw_id; + + -- Close the open bookmark at current history point. + RAISE NOTICE 'TEST % - Close bracket', test_id; + INSERT INTO bookmark_associations(bookmark_label_id, role, bracket_begin_id) + VALUES (bookmark_alpha, 'bookmark_end', open_br_id) + RETURNING hx_tuple + INTO STRICT r; + RAISE NOTICE 'Close returns: %', r; + IF r.hx_tuple != hxtk_current_hx_tuple() THEN + RAISE 'Close bracket: Hx tuple % is not current', br_hx_tuple; + END IF; + + RAISE NOTICE 'TEST % - END', test_id; + -- END TEST 1 + + -- TEST 2: Bracket with auto-provided values. + test_id := 2; + RAISE NOTICE 'TEST % - BEGIN', test_id; + + -- Open a bracket bookmark at current history point. + RAISE NOTICE 'TEST % - Open bracket', test_id; + INSERT INTO bookmark_associations(bookmark_label_id, role) + VALUES (bookmark_alpha, 'bookmark_begin') + RETURNING bookmark_association_id, hx_tuple + INTO STRICT r; + open_br_id := r.bookmark_association_id; + IF r.hx_tuple != hxtk_current_hx_tuple() THEN + RAISE 'Open bracket: Hx tuple % is not current', br_hx_tuple; + END IF; + + -- Insert some gunk. + RAISE NOTICE 'TEST % - Add new network', test_id; + INSERT INTO meta_network(network_name) + VALUES ('Rod Test 1') + RETURNING network_id INTO STRICT nw_id; + + -- Close the open bookmark at current history point. + -- Let the tf supply the bookmark-begin id. + RAISE NOTICE 'TEST % - Close bracket', test_id; + INSERT INTO bookmark_associations(bookmark_label_id, role) + VALUES (bookmark_alpha, 'bookmark_end') + RETURNING bracket_begin_id, hx_tuple + INTO STRICT r; + IF r.hx_tuple != hxtk_current_hx_tuple() THEN + RAISE 'Close bracket: Hx tuple % is not current', br_hx_tuple; + END IF; + IF r.bracket_begin_id != open_br_id THEN + RAISE 'Close bracket: Expected bracket_begin_id = %, got %', open_br_id, close_br_begin_id; + END IF; + + RAISE NOTICE 'TEST % - END', test_id; + -- END TEST 2 + + END; +$$; +ROLLBACK; diff --git a/pycds/orm/tables.py b/pycds/orm/tables.py index 531fc939..d36e915d 100644 --- a/pycds/orm/tables.py +++ b/pycds/orm/tables.py @@ -29,12 +29,14 @@ String, Date, Index, + Enum as EnumType, ) from sqlalchemy import DateTime, Boolean, ForeignKey, Numeric, Interval from sqlalchemy.orm import relationship, synonym, declarative_base from sqlalchemy.schema import UniqueConstraint from sqlalchemy.schema import CheckConstraint from geoalchemy2 import Geometry +from enum import Enum from sqlalchemy.dialects.postgresql import CITEXT as CIText @@ -607,3 +609,101 @@ class DerivedValue(Base): name="obs_derived_value_time_place_variable_unique", ), ) + + +class BookmarkLabel(Base): + """ + A bookmark label is a named object that can be associated to one or more history + tuples. + + Every bookmark label belongs to a network. Together with the uniqueness constraint + on (label, network_id), this enables likely common bookmark names to be reused across + networks but not collide with each other. We will want to insist that name is unique, + and partitioning by network seems like an easy sanity-maintaining measure. + TODO: Review this decision. + """ + + __tablename__ = "bookmark_labels" + + bookmark_label_id = Column(Integer, primary_key=True) + network_id = Column(Integer, ForeignKey("meta_network.network_id"), nullable=False) + label = Column(String, nullable=False) + comment = Column(String) + + # NB: The following values are enforced (overridden) by trigger functions. + # Defaults provided here are more documentation than anything. + mod_time = Column(DateTime, nullable=False, server_default=func.now()) + mod_user = Column( + String(64), nullable=False, server_default=literal_column("current_user") + ) + + UniqueConstraint("network_id", "label"), + + +class BookmarkAssociationRole(Enum): + """The SQL enumeration type for the `role` attribute of `BookmarkAssociation`. + For more on the meanings and use of association roles, see README documentation. + + Note that only the names of the class elements are persisted, not the values, + which are arbitrary. We've chosen here to use the same element names as values. + SQLAlchemy doc on enum: + https://docs.sqlalchemy.org/en/13/core/type_basics.html#sqlalchemy.types.Enum + See also this SO post for usage with Alembic: + https://stackoverflow.com/a/73922844 + """ + + singleton = "singleton" + bracket_begin = "bracket_begin" + bracket_end = "bracket_end" + + +class BookmarkAssociation(Base): + """ + A bookmark association associates a bookmark label to a tuple of history id's. + When we say "bookmark this point in history", we mean: create such an association + using a given bookmark label. + + An association includes a bookmark label and the role of the label in the association. + For more on the meanings and use of association roles, see README documentation. + """ + + __tablename__ = "bookmark_associations" + + bookmark_association_id = Column(Integer, primary_key=True) + bookmark_label_id = Column( + Integer, ForeignKey("bookmark_labels.bookmark_label_id"), nullable=False + ) + role = Column(EnumType(BookmarkAssociationRole), nullable=False) + # bracket_begin_id matches a bracket-end to a bracket-begin. It must be non-null + # if and only if role == bracket_end, and in that case it must be the id of a + # bracket_begin association that is not already matched. This condition is enforced + # by a trigger function, which also provides a value in the case that there is + # exactly one open bracket and a value is not explicitly specified. + bracket_begin_id = Column( + Integer, ForeignKey("bookmark_associations.bookmark_association_id") + ) + comment = Column(String) + + # History tuple. Every history table must be included here. + obs_raw_hx_id = Column( + Integer, ForeignKey("obs_raw_hx.obs_raw_hx_id"), nullable=False + ) + meta_network_hx_id = Column( + Integer, ForeignKey("meta_network_hx.meta_network_hx_id"), nullable=False + ) + meta_station_hx_id = Column( + Integer, ForeignKey("meta_station_hx.meta_station_hx_id"), nullable=False + ) + meta_history_hx_id = Column( + Integer, ForeignKey("meta_history_hx.meta_history_hx_id"), nullable=False + ) + meta_vars_hx_id = Column( + Integer, ForeignKey("meta_vars_hx.meta_vars_hx_id"), nullable=False + ) + + # NB: The following values are enforced (overridden) by trigger functions. + # Defaults provided here are more documentation than anything. + mod_time = Column(DateTime, nullable=False, server_default=func.now()) + mod_user = Column( + String(64), nullable=False, server_default=literal_column("current_user") + )