-
-
Notifications
You must be signed in to change notification settings - Fork 797
feat: allow pagination of message by date #890
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,11 @@ | ||
| package database | ||
|
|
||
| import ( | ||
| "time" | ||
|
|
||
| "github.com/gotify/server/v2/model" | ||
| "gorm.io/gorm" | ||
| "gorm.io/gorm/clause" | ||
| ) | ||
|
|
||
| // GetMessageByID returns the messages for the given id or nil. | ||
|
|
@@ -34,14 +37,33 @@ func (d *GormDatabase) GetMessagesByUser(userID uint) ([]*model.Message, error) | |
| return messages, err | ||
| } | ||
|
|
||
| // GetMessagesByUserSince returns limited messages from a user. | ||
| // GetMessagesByUserPaginated returns limited messages from a user. | ||
| // If since is 0 it will be ignored. | ||
| func (d *GormDatabase) GetMessagesByUserSince(userID uint, limit int, since uint) ([]*model.Message, error) { | ||
| func (d *GormDatabase) GetMessagesByUserPaginated(userID uint, limit int, since, after uint64, by string) ([]*model.Message, error) { | ||
| var messages []*model.Message | ||
| db := d.DB.Joins("JOIN applications ON applications.user_id = ?", userID). | ||
| Where("messages.application_id = applications.id").Order("messages.id desc").Limit(limit) | ||
| Where("messages.application_id = applications.id").Order(clause.OrderBy{Columns: []clause.OrderByColumn{ | ||
| { | ||
| Column: clause.Column{ | ||
| Table: "messages", | ||
| Name: by, | ||
| }, | ||
| Desc: since != 0 || after == 0, | ||
|
Comment on lines
+47
to
+51
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this mean, the messages are returned in another order based on if since or after is set. I feel like this is unexpected if both parameters are set. It should be another parameter to make it more explicit. Either way, it should always use sorting by id, as this will be more precise than ordering by date, as there could be messages with the same date.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree this is not ideal, the reason I used this scheme is mostly: The computational complexity of finding inequalities on two keys is O(N) time O(N) space (or you can build 4 indices, regardless the complexity is squared and as if no indices apply), this is made worse by the fact that the database will not be permitted to assume So I am trying to design the API to prevent making it possible to make queries like
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am open to using distinct keys and just enforce they are mutually exclusive to each other though, but the tradeoff is vocabulary bloat instead of this "switch pagination key" semantics.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry, I'm currently a little unresponsive, and probably will continue to be for the rest of the month. Ahh yeah, that's tough. From a general point of view, do we need the new "after" filter. If we want to query a specific date range e.g. 2025-01-01 to 2025-01-31, I don't think it makes much difference if you start fetching the messages from the newest (2025-01-31) or oldest (2025-01-01 messages. The user would be responsible to fetch as many pages until all messages in the range were fetched. The only thing to consider would be big date ranges, and wanting to lazy load and starting with the oldest message, but I'm not sure if we need to support this. I'm probably also okay with just not having an index for the features not used in the official clients. I don't think the difference it performance should be too visible, because the query is still pretty simple, and we're probably not talking about billions of messages in gotify. Another idea would be for an intermediate api to query the message id boundaries. Something like We'd only query messages with the actual ids, so it should make the pagination simpler. Maybe we could do this also internally in our pagination api, if a date is provided we translate it to a message. And if both is provided we do a min(translated date id, provided since message id) or similar. What do you think about any of that?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah honestly I don't really care about the perf degredation for my own use case - I am just worried about it becoming a hard to resolve issue once somebody opens a ticket saying they have DB timeout for what looks like a simple query why (had this problem in another project I am engaged in) This final proposal looks best to me because it inherits existing schemes , but we also need to implement the same thing again in the /application path, probably needs some refactoring, but the API presentations looks okay to me. @cxtal Do you have any input on this? It should work for your case and respect your preference for long timestamps. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Sounds cool, having a good one?
Imagine it like this. Let's say I have a server with 100 messages. The user asks me for messages between two dates. I write a client like Winify that queries the 1000 messages. Because there is no way to filter the responses, I have to pull all the 1000 messages from the server, put them in a collection or array, then I need to sort them by their date field and only then I can offer the result to the user by matching the date-range. This blows up quickly, let's say on an Android app. Imagine I am across the planet and I query my server back home with a date-range. Imagine the network bloat just to pull those messages and imagine the RAM or CPU my Android device would just waste on sorting useless messages that could have been pre-filled by the server. The actual way of retrieving these messages would not matter and the current way that you are doing things are just fine using IDs and pagination. When Gotify receives a query like: Gotiy could retrieve all those messages using
RFC3339 dates that @jmattheis suggested carry the timezone so it should then be: /message/2025-10-15T23:00Z where With RFC/ISO times, you would know the server timezone so you could even pass your local timezone and Gotify would know what that date represents.
You're right, pagination is great for UI-oriented frontends but they're not too great for data. Clearly, it would feel awkward to display on the same page 10000000s of results like on a Google search result page. Imagine the scrolling, lol. Instead pagination is awkward when accessed via programmer's hook because I have to send stuff like this: till I retrieve all the messages for application When I couple Gotify with, say Amazon Alexa, and I tell Amazon Alexa "Alexa, show all notifications last Tuesday", there is no UI, pages, calendar selector or anything of the sort, not even conceptually. IoT transforms "last Tuesday" into a date and then a program like Winify will query Gotify by performing calls similar to the above in order to retrieve all the messages from "last Tuesday" and then send a message back to IoT that will make Alexa talk back, perhaps, the notification titles of the messages matching the date "last Tuesday". There is no user-selection here, well, except the voice, and Gotify has to be queried programmatically by Winify to process the messages itself. (By the way, this scenario is real, Winify can do that because it connects with MQTT to brokers for automation.)
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, I get what you are talking about .. let me think about it (probably after I come back).. in short I think this debate cannot end unless we literally redesign away from REST principles (which , by construction, enforces some kind of hierarchical ordering of keys that isn't universally beneficial).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we want to be ambitious I mean we can have a graphQL endpoint for ultimate freedom on how to lookup messages .. I am open to it but depends on Jannis's appetite for complexity. The thing is, I totally get your case, if you put this ticket into my own fringe project I will make an API for you specifically. But this is a multi person project where I have to be responsible for a lot of user's API stability and long term scope bloat (like magic query parameters, application specific endpoints, etc) control. I am not quite convinced any solution proposed here isn't just "hey this is a way to modify API so my specific case of a structural problem is fixed" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @eternal-flame-AD Great idea, I mean, I see why you stay close to UI because Gotify indeed has an UI that is used to setup applications but in principle, not all REST endpoints have to be user-endpoints and they can be programmer endpoints and hooks. Then, retrieving "all" messages is something an end-user would not do but a programmer that wants to query Gotify would. It's a bit of a cheat because then the API becomes a wrapper around the database but that's the only way it would work because some backends like SQLite are not really concurrent safe so querying the database while Gotify uses it would not work reliably. Like if you use Sonarr, Radarr, etc, they all have an API token where you can perform various actions that are more data-oriented than what their respective UIs offers. I was about to write that SQLite has a pretty powerful FTS engine, so at least for SQLite it would even be easy to offer a "fitler" to search message text itself and it's "free" because this is already built-in to SQLite via plugin.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think speaking as a user I never treated Gotify as some kind of data source or sink .. If I lost my database I won't be losing a night's sleep on it. That's why I am as you said "stay close to UI" because I think that's the primary use case. Whether we should go for expose the entire database as long as it isn't a security vuln kind of thinking .. I can accept the validity but I would say would be a niche use case. If gotify were a paid product, the selling point would be a simple no frills API to get stuff on your phone notification , not how extensible the message query system is. About FTS .. yes we could have that but we need to have coverage on MySQL and Postgres as well. I'm sure it's a resolvable problem but let's keep it to a different issue. |
||
| }, | ||
| }}).Limit(limit) | ||
| if since != 0 { | ||
| db = db.Where("messages.id < ?", since) | ||
| sinceVal := any(since) | ||
| if by == "date" { | ||
| sinceVal = time.Unix(int64(since), 0) | ||
| } | ||
| db = db.Where(clause.Lt{Column: clause.Column{Table: "messages", Name: by}, Value: sinceVal}) | ||
| } | ||
| if after != 0 { | ||
| afterVal := any(after) | ||
| if by == "date" { | ||
| afterVal = time.Unix(int64(after), 0) | ||
| } | ||
| db = db.Where(clause.Gte{Column: clause.Column{Table: "messages", Name: by}, Value: afterVal}) | ||
| } | ||
| err := db.Find(&messages).Error | ||
| if err == gorm.ErrRecordNotFound { | ||
|
|
@@ -60,13 +82,32 @@ func (d *GormDatabase) GetMessagesByApplication(tokenID uint) ([]*model.Message, | |
| return messages, err | ||
| } | ||
|
|
||
| // GetMessagesByApplicationSince returns limited messages from an application. | ||
| // GetMessagesByApplicationPaginated returns limited messages from an application. | ||
| // If since is 0 it will be ignored. | ||
| func (d *GormDatabase) GetMessagesByApplicationSince(appID uint, limit int, since uint) ([]*model.Message, error) { | ||
| func (d *GormDatabase) GetMessagesByApplicationPaginated(appID uint, limit int, since, after uint64, by string) ([]*model.Message, error) { | ||
| var messages []*model.Message | ||
| db := d.DB.Where("application_id = ?", appID).Order("messages.id desc").Limit(limit) | ||
| db := d.DB.Where("application_id = ?", appID).Order(clause.OrderBy{Columns: []clause.OrderByColumn{ | ||
| { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe this can be unified between the two *Paginated apis, the only difference is the additional filtering for application id or?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes .. I don't think I wrote this part (?) so I am not sure the exact rationale but I assume it's to accommodate it being mounted on a different API path. It can be just one interface at the data model layer. |
||
| Column: clause.Column{ | ||
| Table: "messages", | ||
| Name: by, | ||
| }, | ||
| Desc: since != 0 || after == 0, | ||
| }, | ||
| }}).Limit(limit) | ||
| if since != 0 { | ||
| db = db.Where("messages.id < ?", since) | ||
| sinceVal := any(since) | ||
| if by == "date" { | ||
| sinceVal = time.Unix(int64(since), 0) | ||
| } | ||
| db = db.Where(clause.Lt{Column: clause.Column{Table: "messages", Name: by}, Value: sinceVal}) | ||
| } | ||
| if after != 0 { | ||
| afterVal := any(after) | ||
| if by == "date" { | ||
| afterVal = time.Unix(int64(after), 0) | ||
| } | ||
| db = db.Where(clause.Gte{Column: clause.Column{Table: "messages", Name: by}, Value: afterVal}) | ||
| } | ||
| err := db.Find(&messages).Error | ||
| if err == gorm.ErrRecordNotFound { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All other date-time related fields use RFC3339 format, this would break the consistency with it being a unix timestamp.
I think I'd prefer either something like an ISO interval e.g.
&interval=2025-10-10T23:00Z/2025-10-15T23:00Zand then allowing to use the after/before id offset, to page in this daterange to prevent the bugs described in #889 (comment)or have separate sinceTime and afterTime fields which accept an RFC3339 datetime.
What do you think? I don't dislike this current solution, so I'm open for any arguments (:.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 I prefer ISO dates to unix timestamps