Skip to content

Commit f15f09f

Browse files
authored
Mark streamfields as containing personal information (#4983)
When PII_FIELD_MARKING_ENABLED is set to True streamfields get a "Personal information" checkbox. Fields with this checked are only shown to the applicant and co-applicants and to staff.
1 parent 3e4362f commit f15f09f

20 files changed

Lines changed: 2074 additions & 43 deletions

File tree

‎docs/setup/administrators/configuration.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,14 @@ Should submission anonymization be enabled. Allows for both manual anonymization
320320

321321
----
322322

323+
Should form authors be able to mark application form fields as containing personally identifiable information (PII). Answers to marked fields are only shown to the applicant, their co-applicants and staff.
324+
325+
See [here](pii-fields.md) for more information.
326+
327+
PII_FIELD_MARKING_ENABLED = env.bool("PII_FIELD_MARKING_ENABLED", False)
328+
329+
----
330+
323331
## Slack settings
324332

325333
SLACK_TOKEN = env.str('SLACK_TOKEN', None)
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Marking form fields as personal information
2+
3+
Application forms often ask for details that identify the applicant personally —
4+
a phone number, a home address, a date of birth. Hypha lets you mark those
5+
questions so that only the people who genuinely need the answers can read them.
6+
7+
Set `PII_FIELD_MARKING_ENABLED=True` to turn the feature on.
8+
9+
## Marking a field
10+
11+
With the setting enabled, the question blocks in the form builder (Wagtail
12+
Admin → "Apply" → "Forms") gain a **"Personal information"** checkbox. Tick it
13+
on any question whose answer will contain personally identifiable information.
14+
A few blocks do not have the checkbox, see
15+
[Questions without the checkbox](#questions-without-the-checkbox) below.
16+
17+
The checkbox is empty by default, so no existing question changes behaviour when
18+
you turn the feature on.
19+
20+
## Who can see the answers
21+
22+
Answers to marked questions are shown to:
23+
24+
* the applicant who created the submission
25+
* the co-applicants invited to that submission
26+
* staff (and staff admins)
27+
28+
Everyone else — including reviewers, external reviewers and community reviewers
29+
— sees the question, but the answer is replaced with "Hidden — contains personal
30+
information." Keeping the question visible means reviewers can still tell what
31+
was asked and that an answer exists.
32+
33+
Staff also see a small "(PII)" marker after the label of each marked question,
34+
so they can tell at a glance which answers are restricted. Applicants and
35+
co-applicants do not see the marker.
36+
37+
## Questions without the checkbox
38+
39+
The built-in required fields (Title, Full name, E-mail, Address, Organization
40+
name, Requested funding and Duration) do not have the checkbox, because they are
41+
rendered separately from the rest of the answers and so are never redacted.
42+
Applicant identity in those fields is controlled separately by
43+
[`HIDE_IDENTITY_FROM_REVIEWERS`](configuration.md).
44+
45+
The **"Group fields"** block — the question that reveals or hides a group of
46+
other questions — does not have the checkbox either, so the choice made on it is
47+
always shown to reviewers. Marking it would achieve little on its own: the
48+
toggle decides whether the questions inside the group are answered at all, so an
49+
empty group tells the reader which way it was set. Mark the questions *inside*
50+
the group instead, which is where the personal information is given.
51+
52+
## Other forms
53+
54+
The checkbox appears on application forms only. Review, determination, project
55+
(PAF and SOW) and report forms do not have it, since their answers are not
56+
redacted.
57+
58+
## Turning the feature off again
59+
60+
Setting `PII_FIELD_MARKING_ENABLED=False` hides the checkbox in the form builder
61+
but does **not** un-hide fields that have already been marked. This is
62+
deliberate: turning the setting off should never silently expose information
63+
that was marked as personal. To make an answer visible again, turn the setting
64+
back on and untick the checkbox on that question.
65+
66+
Note that changing a form only affects submissions made from then on. Each
67+
submission stores its own copy of the form it was filled in with, so existing
68+
submissions keep the marks they were submitted with.
69+
70+
For the same reason, a form definition copied or imported from an instance
71+
where the feature was on arrives with its marks intact, and answers to those
72+
questions are redacted even though the checkbox is not on offer. Turn the
73+
setting on if you need to see or change which questions are marked.
74+
75+
## Limitations
76+
77+
Answers are indexed for search regardless of whether they are marked, so a
78+
search by a reviewer can still match on the text of a hidden answer. The search
79+
results themselves only show submission titles, never the matched answer.
80+
81+
Redaction applies to the submission page and everywhere the answers are shown on
82+
it, including the answers panel embedded in the review and determination forms.
83+
84+
Redacting a file or image answer removes the attachment, and with it the
85+
download link, from the page. It does not revoke access to the file itself: a
86+
link to it obtained some other way — pasted into a comment, or kept from before
87+
the question was marked — still works for anyone who can view the submission.
88+
The link contains the internal id of the question, which is never shown for a
89+
redacted answer, so it cannot be guessed.
90+
91+
The other places answers appear are restricted to staff already, so nothing is
92+
redacted there: the submission PDF download, the revision comparison view, and
93+
CSV exports (the last also governed by `SUBMISSIONS_EXPORT_ACCESS_STAFF` and
94+
`SUBMISSIONS_EXPORT_ACCESS_STAFF_ADMIN`).

‎hypha/apply/categories/blocks.py‎

Lines changed: 27 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -99,17 +99,35 @@ def prepare_data(self, value, data, serialize):
9999
data = category.options.filter(id__in=data).values_list("value", flat=True)
100100
return data
101101

102-
def render(self, value, context):
103-
# Overwriting field_label and help_text with default for empty values
102+
def get_display_value(self, value):
103+
# Fall back on the category's own name and help text for empty values.
104104
category_fields = {"field_label": "name", "help_text": "help_text"}
105105

106-
for field in category_fields.keys():
107-
if not value.get(field):
108-
category = value["category"]
109-
if isinstance(category, int) or isinstance(category, str):
110-
category = self.get_instance(id=category)
111-
value[field] = getattr(category, category_fields[field])
112-
return super().render(value, context)
106+
missing = [field for field in category_fields if not value.get(field)]
107+
if not missing:
108+
return value
109+
110+
# Look the category up once, however many fields fall back on it.
111+
category = value["category"]
112+
if isinstance(category, int) or isinstance(category, str):
113+
category = self.get_instance(id=category)
114+
115+
defaults = {
116+
field: getattr(category, category_fields[field]) for field in missing
117+
}
118+
119+
# Return a copy, so that the fallbacks are never written back to the
120+
# stored form definition.
121+
block = getattr(value, "block", None)
122+
if block is None:
123+
display_value = dict(value)
124+
else:
125+
display_value = value.__class__(block, value.items())
126+
display_value.update(defaults)
127+
return display_value
128+
129+
def render(self, value, context):
130+
return super().render(self.get_display_value(value), context)
113131

114132
def get_searchable_content(self, value, data):
115133
return None

‎hypha/apply/funds/blocks.py‎

Lines changed: 47 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,30 @@
88
from wagtail import blocks
99

1010
from hypha.addressfield.fields import ADDRESS_FIELDS_ORDER, AddressField
11-
from hypha.apply.categories.blocks import CategoryQuestionBlock
11+
from hypha.apply.funds.pii_blocks import (
12+
PIICategoryQuestionBlock,
13+
PIICharFieldBlock,
14+
PIICheckboxesFieldBlock,
15+
PIICheckboxFieldBlock,
16+
PIIDateFieldBlock,
17+
PIIDateTimeFieldBlock,
18+
PIIDropdownFieldBlock,
19+
PIIFileFieldBlock,
20+
PIIImageFieldBlock,
21+
PIIMarkdownTextFieldBlock,
22+
PIIMultiFileFieldBlock,
23+
PIIMultiInputCharFieldBlock,
24+
PIINumberFieldBlock,
25+
PIIRadioButtonsFieldBlock,
26+
PIIRichTextFieldBlock,
27+
PIITextFieldBlock,
28+
PIITimeFieldBlock,
29+
)
1230
from hypha.apply.stream_forms.blocks import FormFieldsBlock
1331
from hypha.apply.users.models import User
1432
from hypha.apply.utils.blocks import (
1533
CustomFormFieldsBlock,
1634
MustIncludeFieldBlock,
17-
RichTextFieldBlock,
1835
SingleIncludeBlock,
1936
)
2037
from hypha.apply.utils.templatetags.apply_tags import format_number_as_currency
@@ -272,8 +289,34 @@ class Meta:
272289

273290

274291
class ApplicationCustomFormFieldsBlock(CustomFormFieldsBlock, FormFieldsBlock):
275-
category = CategoryQuestionBlock(group=_("Custom"))
276-
rich_text = RichTextFieldBlock(group=_("Fields"))
292+
"""The form fields available when building an application form.
293+
294+
The fields that hold an answer are overridden with the variant that can be
295+
marked as personal information, see `hypha.apply.funds.pii_blocks`. Two
296+
kinds of field are left out. The built-in fields (title, email, ...) are
297+
rendered on their own rather than as part of the answers, so marking one
298+
would have no effect. The group toggle keeps the plain block because the
299+
questions it reveals give away the choice made on it either way; the answers
300+
worth marking are the ones inside the group.
301+
"""
302+
303+
char = PIICharFieldBlock(group=_("Fields"))
304+
multi_inputs_char = PIIMultiInputCharFieldBlock(group=_("Fields"))
305+
text = PIITextFieldBlock(group=_("Fields"))
306+
number = PIINumberFieldBlock(group=_("Fields"))
307+
checkbox = PIICheckboxFieldBlock(group=_("Fields"))
308+
radios = PIIRadioButtonsFieldBlock(group=_("Fields"))
309+
dropdown = PIIDropdownFieldBlock(group=_("Fields"))
310+
checkboxes = PIICheckboxesFieldBlock(group=_("Fields"))
311+
date = PIIDateFieldBlock(group=_("Fields"))
312+
time = PIITimeFieldBlock(group=_("Fields"))
313+
datetime = PIIDateTimeFieldBlock(group=_("Fields"))
314+
image = PIIImageFieldBlock(group=_("Fields"))
315+
file = PIIFileFieldBlock(group=_("Fields"))
316+
multi_file = PIIMultiFileFieldBlock(group=_("Fields"))
317+
rich_text = PIIRichTextFieldBlock(group=_("Fields"))
318+
markdown_text = PIIMarkdownTextFieldBlock(group=_("Fields"))
319+
category = PIICategoryQuestionBlock(group=_("Custom"))
277320
required_blocks = ApplicationMustIncludeFieldBlock.__subclasses__()
278321
single_blocks = ApplicationSingleIncludeFieldBlock.__subclasses__()
279322

‎hypha/apply/funds/management/commands/sanitize_database.py‎

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
ApplicationRevision,
1919
ApplicationSubmission,
2020
)
21+
from hypha.apply.funds.pii_blocks import unmarked_block_class
2122
from hypha.apply.review.blocks import (
2223
RecommendationBlock,
2324
RecommendationCommentsBlock,
@@ -86,13 +87,17 @@ def update_text_data():
8687
else:
8788
update_data(self.f.text(len(data))) # noqa: B023
8889

90+
# Application form fields are the PII-markable subclasses of the
91+
# blocks below, so compare against the block each one extends.
92+
block_class = unmarked_block_class(form_field.block)
93+
8994
if form_field.value["field_label"].lower() == "organization name":
9095
update_data(self.f.company())
9196
elif form_field.value["field_label"].lower() == "contact phone number":
9297
update_data(self.f.phone_number())
9398
elif (
9499
form_field.value["field_label"].lower() == "organization address"
95-
or type(form_field.block) is AddressFieldBlock
100+
or block_class is AddressFieldBlock
96101
):
97102
try:
98103
address = json.loads(data)
@@ -124,35 +129,35 @@ def update_text_data():
124129
update_data(self.f.ssn())
125130
elif form_field.value["field_label"].lower() == "additional contact email":
126131
update_data(self.f.email())
127-
elif type(form_field.block) is FullNameBlock:
132+
elif block_class is FullNameBlock:
128133
update_data(self.f.name())
129-
elif type(form_field.block) is EmailBlock:
134+
elif block_class is EmailBlock:
130135
update_data(self.f.email())
131-
elif type(form_field.block) is TitleBlock:
136+
elif block_class is TitleBlock:
132137
update_data(self.f.sentence(5))
133-
elif type(form_field.block) is MultiFileFieldBlock:
138+
elif block_class is MultiFileFieldBlock:
134139
update_data([])
135-
elif type(form_field.block) is RecommendationBlock:
140+
elif block_class is RecommendationBlock:
136141
update_data(random.choice(RECOMMENDATION_CHOICES)[0])
137-
elif type(form_field.block) is ScoreFieldBlock:
142+
elif block_class is ScoreFieldBlock:
138143
update_data(random.choice(RATE_CHOICES)[0])
139-
elif type(form_field.block) is ScoreFieldWithoutTextBlock:
144+
elif block_class is ScoreFieldWithoutTextBlock:
140145
update_data(
141146
random.randint(form_field.value["min"], form_field.value["max"])
142147
)
143-
elif type(form_field.block) is DropdownFieldBlock:
148+
elif block_class is DropdownFieldBlock:
144149
update_data(random.choice(form_field.value["choices"][1]))
145-
elif type(form_field.block) is ValueBlock:
150+
elif block_class is ValueBlock:
146151
update_data(random.randint(0, 10000000))
147-
elif type(form_field.block) in [
152+
elif block_class in [
148153
TextFieldBlock,
149154
CharFieldBlock,
150155
RichTextFieldBlock, # This may need to be updated sometime in the future
151156
RecommendationCommentsBlock,
152157
]:
153158
update_text_data()
154159
elif (
155-
type(form_field.block)
160+
block_class
156161
in [
157162
RadioButtonsFieldBlock, # If it's a radio button, then it doesn't really need to be randomized
158163
CheckboxFieldBlock, # Similar to above
@@ -166,9 +171,7 @@ def update_text_data():
166171
print(form_holder.id)
167172
print(form_field.value["field_label"])
168173
print(data)
169-
raise Exception(
170-
"Don't know how to handle " + str(type(form_field.block))
171-
)
174+
raise Exception("Don't know how to handle " + str(block_class))
172175
form_holder.save()
173176

174177
def handle(self, *args, **options):

0 commit comments

Comments
 (0)