Refactor subsequences 2025 #6511
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and with less tiptoeing around what's already there
previous work to `subsequences__old`
…2025 # Conflicts: # kobo/apps/subsequences__new/actions/manual_transcription.py
### 💭 Notes Rename "automated_google_\*" actions to "automatIC_google_\*". Also renames a few other methods for clarity. The unit tests that are failing are failing on the base branch as well.
action data within `_data` attribute for each version Two tests are failing as they were already on 0a92a24: FAILED test_models.py::SubmissionSupplementTestCase::test_retrieve_data_from_migrated_data - KeyError: '_version' FAILED test_models.py::SubmissionSupplementTestCase::test_retrieve_data_with_stale_questions - AssertionError: assert {'group_name/question_name': {'manual_translation': {'en': {'_versions': [{'_uuid': '22b04ce8-61c2-4383-836f-5d5f0ad73645', 'value': 'berserk',...
### Notes Unit tests only. Skips tests that we eventually want to implement but don't have implementations for yet. DRF failures are unrelated to PR. The only substantive difference is in how we add supplements to duplicated submissions. The old `update_submission_extras` method has been removed so instead we just create a SubmissionSupplement object with the correct data. --------- Co-authored-by: John N. Milner <[email protected]>
…DEV-1229 (#6492) ### 💭 Notes Add new QuestionAdvancedAction model and associated CRU endpoints. This PR does not involve actually using the models, though it does include audit logs for when users hit those endpoints. QuestionAdvancedAction logs cannot be deleted. Also includes the automatic migration of the `advanced_features` dict into corresponding QuestionAdvancedAction objects. For now it does not change the `advanced_features` dict since we are still using it, but eventually it will be updated to signal that the data in it has already been migrated and we should use the associated QuestionAdvancedAction models instead. The OpenAPI errors are pre-existing and will be dealt with at the branch level sometime before merging the full project branch. ### 👀 Preview steps 1. ℹ️ have an account and a project with an audio question 2. POST to `/api/v2/assets/<asset_uid>/advanced-features/` the following data: ``` { "action": "manual_transcription", "question_xpath": <audio question xpath>, "params": [{"language": "en"}] } ``` 4. Navigate to `/api/v2/assets/<asset-uid>/advanced-features` in a browser 5. 🟢 There should be one advanced feature in the list 6. Note the uuid of the action you just created 7. PATCH `/api/v2/assets/<asset_uid>/advanced-features/<action_uuid>/` with `{"params": ["language": "es"]}` 8. Reload `/api/v2/assets/<asset-uid>/advanced-features` 9. 🟢 [on PR] notice that the params for the action now include both English and Spanish
…6523) ### 📣 Summary Add supplemental NLP columns to data table. ### 📖 Description This is just for adding the columns to the data table. They may not be populated correctly. If an NLP action is enabled, there will be a column for it, even if there are presently no responses. ### 💭 Notes Using analysis_form_json to avoid having to make changes on the frontend even though it's not a very descriptive name. Does not add QA questions, those will come later. Removed some of the response fields from the old analysis_from_json since they don't seem to be used. It's possible in QA or when dealing with exports they will turn out to be used but we can always add them back. ### 👀 Preview steps 1. ℹ️ have an account and a project with an audio question with at least one response 2. In a python shell, enable all NLP actions by running ``` asset = Asset.objects.get(uid={uid}) for action in [ Action.MANUAL_TRANSLATION, Action.MANUAL_TRANSCRIPTION, Action.AUTOMATIC_GOOGLE_TRANSLATION, Action.AUTOMATIC_GOOGLE_TRANSCRIPTION, ]: language = 'en' if 'transcription' in action else 'es' QuestionAdvancedFeature.objects.create( question_xpath={xpath}, action=action, params=[{'language': language}], asset=asset, ) ``` Note: this will enable English manual/automatic transcripts and Spanish automatic/manual translations 3. Navigate to the data table 4. 🟢 [on PR] Notice there are columns for English transcript and Spanish translation for the relevant question
### 📣 Summary
Ensure transcriptions and translations are displayed in the data table.
### 📖 Description
Only accepted transcriptions/translations will be displayed.
### 💭 Notes
There were several issues preventing transcriptions and translations
from showing up in the data table:
1. retrieve_data was not being called with `for_output=True`,
2. The method signature and the implementations of
transform_data_for_output did not correctly reflect how the method was
being called by SubmissionSupplement.retrieve_data
3. Even when for_output was set, SubmissionSupplement.retrieve_data did
not output the data in the format expected by the frontend
This PR addresses all of these issues. It uses the `_advanced_features`
field to determine the activated features because
QuestionAdvancedFeatures are not fully implemented yet.
Deferred for later:
Fixing drf
Using QuestionAdvancedFeatures instead of `_advanced_features`
Handling Qual actions
### 👀 Preview steps
Going through the preview is a little annoying because columns are
determined the new way (using QuestionAdvancedFeatures) but the data in
the rows is determined the old way (using Asset._advanced_features).
Also data can only be added by PATCH and not through the UI.
1. ℹ️ have an account and NLP set up
2. Create a new project with an audio question
3. Add a submission with an audio response
4. In a django shell, enable NLP actions the new way by running
```
asset = Asset.objects.get(uid=<uid>)
for action in [
Action.MANUAL_TRANSLATION,
Action.MANUAL_TRANSCRIPTION,
Action.AUTOMATIC_GOOGLE_TRANSLATION,
Action.AUTOMATIC_GOOGLE_TRANSCRIPTION,
]:
language = 'en' if 'transcription' in action else 'es'
QuestionAdvancedFeature.objects.create(
question_xpath=<xpath>,
action=action,
params=[{'language': language}],
asset=asset,
)
```
5. Enable NLP actions the old way by running
```
asset = Asset.objects.get(uid=<uid>)
asset.advanced_features = {
'_version': '20250820',
'_actionConfigs': {
<xpath>: {
'manual_transcription': [{'language':'en'}],
'manual_translation': [{'language':'es'}],
'automatic_google_transcription': [{'language':'en'}],
'automatic_google_translation': [{'language':'es'}]
}
}
}
asset.save()
```
6. Using curl and your authorization token, PATCH the following JSONs to
`http://kf.kobo.local/api/v2/assets/<asset_uid>/data/<sub_uuid>/submission-supplement`.
In between each PATCH refresh the data table.
7. manual transcription: `'{"_version":"20250820", "<xpath>":
{"manual_transcription": {"language":"en", "value":"Hello"}}}'`
8. 🟢 [on PR] The transcription column should contain "Hello"
9. automatic transcription: ` '{"_version":"20250820", "<xpath>":
{"automatic_google_transcription": {"language":"en"}}}'`
10. 🟢 [on PR] The transcription column should contain "Hello"
11. accepting the automatic transcription: `'{"_version":"20250820",
"<xpath>": {"automatic_google_transcription": {"language":"en",
"accepted": true}}}'`
12. 🟢 [on PR] The transcription column should contain the automatically
generated transcription
13. automatic translation: `'{"_version":"20250820", "<xpath>":
{"automatic_google_translation": {"language":"es"}}}'`
14. 🟢 [on PR] The translation column should be empty
15. accepting the automatic translation: `'{"_version":"20250820",
"<xpath>": {"automatic_google_translation": {"language":"es",
"accepted": true}}}'`
16. 🟢 [on PR] The translation column should contain the automatic
translation
17. manual_translation: `'{"_version":"20250820", "<xpath>":
{"manual_translation": {"language":"es", "value":"Hola"}}}'`
18. 🟢 [on PR] The translation column should contain "Hola"
…aint` DEV-1432 (#6534) ### 💭 Notes Will need to re-apply the 0005_questionadvancedfeature migration with the new changes.
### 💭 Notes Use the new QuestionAdvancedFeature model for revising/retrieving data instead of the asset.advanced_features dict. ### 👀 Preview steps 1. ℹ️ have an account 2. Create a new project with an audio question 3. Add a submission 4. Enable transcriptions by running ``` curl -X POST -H 'Authorization: Token <your token>' http://kf.kobo.local/api/v2/assets/<asset_uid>/advanced-features/ --json '{"question_xpath":<audio_question_xpath>, "action": "manual_transcription", "params": [{"language": "en"}]}' ``` 5. Add an English transcription by running ``` curl -X PATCH -H 'Authorization: Token <your token>' http://kf.kobo.local/api/v2/assets/<asset_uid>/data/<submission_uuid>/supplement/ --json '{"_version":"20250820", "<audio_question_xpath>": {"manual_transcription": {"language":"en", "value": "hello"}}}' ``` 6. Navigate to the data table 7. 🟢 [on PR] The transcript for the submission should show up in the table
…_for_output` for `QualAction` (#6504) ### 📣 Summary Add implementation of `get_output_fields()` and `transform_data_for_output()` in `QualAction`. ### 📖 Description This update enables qualitative analysis results to appear correctly in exports or the table view. The new logic: - Defines the output fields for each qualitative question (including labels, types, and choices). - Converts stored qualitative results into export-ready values, including expanding choice UUIDs into readable label objects.
### 💭 Notes Migrate asset.advanced_features to asset.advanced_features_set when someone hits an advanced-features endpoint or saves an existing asset. Notable Decisions: We will use known_cols only to determine which questions had nlp actions performed. If any question had a transcript or a translation in any language, we will enable all 4 nlp actions (manual transcript, automatic transcript, manual translation, automatic translation) for it, using the languages in `advanced_features` as params Removed the `set_version` method because it was causing circular imports that were quite difficult to fix and it didn't seem worth it. Note the data table will not load properly because this PR only migrates `advanced_features` and not `SubmissionSupplements`. ### 👀 Preview steps 1. ℹ️ have an account 3. [on main] Create a new project with an audio question and at least one submission 4. [on main] Add at least one transcription, one translation, and a QA question 5. Switch to the PR branch 6. Navigate to `/api/v2/assets/<uid>/advanced-features` 8. 🟢 [on PR] notice there are configured advanced features for all nlp actions (manual/automatic transcription/translation) and qual for the relevant audio question
…ect params DEV-1441 (#6548) ### 📣 Summary Validate `params` before creating new advanced features. ### 👀 Preview steps 1. ℹ️ have an account and a project 2. `curl -X POST -H 'Authorization: Token <your token>' http://kf.kobo.local/api/v2/assets/<asset_uid>/advanced-features --json '{"question_xpath": <xpath>, "action": "manual_transcription", "params": [{"something":"bad"}]}' 3. 🔴 [on refactor-subsequences-2025] request 500s 4. 🟢 [on PR] request 400s
…ental DEV-934 (#6422) ### 💭 Notes Fill out the method for converting old SubmissionExtra content dicts to the new format expected by SubmissionSupplemental for translations and transcripts. This code makes numerous assumptions to fill in information that is not present in the old structure but required in the new: 1. If old[xpath]['transcript']['value'] == old[xpath]['googlets']['value']and the language codes are the same, we assume the most recent transcript was automatically generated 2. If, for any revision in old[xpath]['transcript']['revisions'], revision['value'] is the same as old[xpath]['googlets']['value'] and the language codes match, we assume that revision was automatically generated. If multiple match, we assume they were all automatically generated. This should be pretty rare but it's possible 3. 1-2 also apply to transcriptions 4. old[xpath]['transcript']['dateModified'] will be assumed to be the creation date of the most recent revision (ie whatever is in old[xpath]['transcript']['value']). The same goes for translations 5. All uuids are newly generated 6. All old transcriptions/translations have status=complete with a _dateAccepted of now() (whenever the code is running) 7. To determine the dependency of any old translation, whether automated or manual: * If we know the source language, look for the most recent transcript in that language that was created before the translation. * If there is none, take the most recent transcription in that language * If there are no transcriptions in the source language, take the most recent transcript * If we don't know the source language, take the most recent transcript that was created before the translation * If there is none, take the most recent transcript 8. We can ignore any badly formatted revisions/transcripts/translations 9. Most recent revisions will be first in the version array
…` structure DEV-1443 (#6549) ### 📣 Summary Restore background and NLP processing by reading values from the new `_data` field. ### 📖 Description This fix updates the background processing logic to support the new data structure where value, language, and status (when present) are now nested under a `_data` dictionary. Some automated NLP actions were broken because they were still looking for these fields at the top level, where they can no longer exist. ### 👀 Preview steps 1. ℹ️ have an account and a project with an audio question 2. submit an audio file (longer than 2 minutes) 3. use the shell to process an automatic action (look at Linear for snippet) 4. 🔴 [on main] notice that the background process never starts, and if user sends acceptance, external service is still called 5. 🟢 [on PR] notice that everything works as expected
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗒️ Checklist
#Support Docs Updates, if any<type>(<scope>)<!>: <title> DEV-1234Front endand/orBack endorworkflow📣 Summary
TODO
📖 Description
TODO
👷 Description for instance maintainers
TODO
💭 Notes
TODO
👀 Preview steps