Schema not applied correctly in library when using library.with_data_values().

[GrahamDumpleton]

What steps did you take and what happened:

Run:

#!/bin/sh

set -x

mkdir -p resources-1/_ytt_lib/mylib

cat > resources-1/_ytt_lib/mylib/00-schema.yaml << EOF
#@data/values-schema
---
apiVersion: ""
kind: ""
#@schema/type any=True
metadata:
  name: ""
  uid: ""
#@schema/type any=True
spec: null
#@schema/type any=True
status: null
EOF

cat > resources-1/_ytt_lib/mylib/99-output.yaml << EOF
#@ load("@ytt:data", "data")
--- #@ data.values
EOF

cat > resources-1/99-output.yaml << EOF
#@ load("@ytt:data", "data")
#@ load("@ytt:struct", "struct")
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ mylib = library.get("mylib")
#@ mylib_with_values = mylib.with_data_values(struct.decode(data.values))

--- #@ template.replace(mylib_with_values.eval())
EOF

cat > values-1.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-10-13T05:54:35Z"
  labels:
    kubernetes.io/metadata.name: default
  name: default
  resourceVersion: "201"
  uid: 591eeb77-5c81-49de-8698-28ac704d0467
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
EOF

ytt -f resources-1 --data-values-file values-1.yaml

This results in the error:

ytt: Error:
- library.eval: Evaluating library 'mylib': Overlaying data values (in following order: additional data values): Document on line ?: Map item (key 'metadata') on line ?: Map item (key 'creationTimestamp') on line ?: Expected number of matched nodes to be 1, but was 0
    in <toplevel>
      99-output.yaml:9 | --- #@ template.replace(mylib_with_values.eval())

With the files created from the test, if you were instead to run:

ytt -f resources-1/_ytt_lib/mylib --data-values-file values-1.yaml

In other words, directly process the files which are in the library, it works fine and produces:

apiVersion: v1
kind: Namespace
metadata:
  name: default
  uid: 591eeb77-5c81-49de-8698-28ac704d0467
  creationTimestamp: "2021-10-13T05:54:35Z"
  labels:
    kubernetes.io/metadata.name: default
  resourceVersion: "201"
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Note that there is no #@data/values file in the library.

Now run a second test where the only difference is that we add a #@data/values file to the library, but where it is only marked as such and has no actual values set:

#!/bin/sh

set -x

mkdir -p resources-2/_ytt_lib/mylib

cat > resources-2/_ytt_lib/mylib/00-schema.yaml << EOF
#@data/values-schema
---
apiVersion: ""
kind: ""
#@schema/type any=True
metadata:
  name: ""
  uid: ""
#@schema/type any=True
spec: null
#@schema/type any=True
status: null
EOF

cat > resources-2/_ytt_lib/mylib/00-values.yaml << EOF
#@data/values
---
EOF

cat > resources-2/_ytt_lib/mylib/99-output.yaml << EOF
#@ load("@ytt:data", "data")
--- #@ data.values
EOF

cat > resources-2/99-output.yaml << EOF
#@ load("@ytt:data", "data")
#@ load("@ytt:struct", "struct")
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ mylib = library.get("mylib")
#@ mylib_with_values = mylib.with_data_values(struct.decode(data.values))

--- #@ template.replace(mylib_with_values.eval())
EOF

cat > values-2.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-10-13T05:54:35Z"
  labels:
    kubernetes.io/metadata.name: default
  name: default
  resourceVersion: "201"
  uid: 591eeb77-5c81-49de-8698-28ac704d0467
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
EOF

ytt -f resources-2 --data-values-file values-2.yaml

This time the test works and we get same output as when had processed the files in the library directly.

So in order for library.with_data_values() to work, you need to have a #@data/values file present.

Be warned though, that the data values file cannot actually have any values set in it, or you get an error again as seen with test:

#!/bin/sh

set -x

mkdir -p resources-3/_ytt_lib/mylib

cat > resources-3/_ytt_lib/mylib/00-schema.yaml << EOF
#@data/values-schema
---
apiVersion: ""
kind: ""
#@schema/type any=True
metadata:
  name: ""
  uid: ""
#@schema/type any=True
spec: null
#@schema/type any=True
status: null
EOF

cat > resources-3/_ytt_lib/mylib/00-values.yaml << EOF
#@data/values
---
metadata:
  name: "xxx"
EOF

cat > resources-3/_ytt_lib/mylib/99-output.yaml << EOF
#@ load("@ytt:data", "data")
--- #@ data.values
EOF

cat > resources-3/99-output.yaml << EOF
#@ load("@ytt:data", "data")
#@ load("@ytt:struct", "struct")
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ mylib = library.get("mylib")
#@ mylib_with_values = mylib.with_data_values(struct.decode(data.values))

--- #@ template.replace(mylib_with_values.eval())
EOF

cat > values-3.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-10-13T05:54:35Z"
  labels:
    kubernetes.io/metadata.name: default
  name: default
  resourceVersion: "201"
  uid: 591eeb77-5c81-49de-8698-28ac704d0467
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
EOF

ytt -f resources-3 --data-values-file values-3.yaml

Which again generates the error:

ytt: Error:
- library.eval: Evaluating library 'mylib': Overlaying data values (in following order: _ytt_lib/mylib/00-values.yaml, additional data values): Document on line ?: Map item (key 'metadata') on line ?: Map item (key 'creationTimestamp') on line ?: Expected number of matched nodes to be 1, but was 0
    in <toplevel>
      99-output.yaml:9 | --- #@ template.replace(mylib_with_values.eval())

What did you expect:

Expect the following:

• It should not be required to have an empty #@data/values annotated file to get it to work.
• Where the #@data/values file sets values it should not error and the values passed to library.with_data_values() should me merged just like if --data-values-file was used on the command line and it was overriding a data values file included with a set of YAML files where a library wasn't involved.

Environment:

% ytt version
ytt version 0.38.0

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[pivotaljohn]

@GrahamDumpleton noted:

• It should not be required to have an empty #@data/values annotated file to get it to work.

Absolutely.

• Where the #@data/values file sets values it should not error and the values passed to library.with_data_values() should me merged just like if --data-values-file was used on the command line and it was overriding a data values file included with a set of YAML files where a library wasn't involved.

Yes: you case would be solved with #418.

This won't be possible to do this with @ytt:library.with_data_values(), as is.
That function accepts overlays as input, today; and it would be a breaking change for it to behave otherwise.

---

In the meantime, you've detailed a workaround (🙏🏻 ).

@gcheadle-vmware noted a interesting variant that works for cases where you don't have authorship of the library and you're okay with leaking the name of the internal library:

$ ytt -f resources-1 --data-values-file @mylib:values-1.yaml

ref: https://carvel.dev/ytt/docs/latest/ytt-data-values/#configuring-data-values-via-command-line-flags

[pivotaljohn]

We have (yet another) very compelling use-case for implementing #418 — this is, in effect, a request for that feature.

Within this flow, there is a bug — and I've pulled that aside in this issue: #566.

@GrahamDumpleton, I believe you've got a workaround (or two), and we've got a more systemic solution.

Is there anything beyond those two items you think we can do here to help?

[GrahamDumpleton]

So what I understand is that you are saying my workaround to make it work is actually the bug and when that bug is fixed I will not be able to do what I want unless you separately implement #418.

[pivotaljohn]

Yes. And that we'll ensure that we don't fix the bug until we provide the systemic solution. 👍🏻

And that we'll be doing both ASAP.

[pivotaljohn]

#418 was delivered as part of v0.39.0.