Hi GeoLift team,
First off, thank you for the fantastic work on this package - it's been incredibly useful, and it's clear a lot of thought went into both the design and the documentation.
While working with GeoLift, I noticed a potentially confusing behavior: sometimes the p-value is statistically significant, but the confidence interval still includes zero. In our case, we saw the warning:
Conformal method of Confidence Interval calculation unsuccessful. Changing Confidence Interval calculation method to jackknife+.
From reviewing the source code, it looks like when conformal inference fails for the CI, GeoLift falls back to jackknife+ for the CI, but still uses conformal inference for the p-value - hence the inconsistency.
While this behavior is expected, it can be confusing for users: which result should be trusted? Would it be possible to add an option to force both the p-value and CI to use the same inference method?
Also open to any advice on how to interpret results in this case.
Thanks again for building and maintaining such a great tool!
Is your feature request related to a problem? Please describe.
Inconsistent conclusions when CI falls back to jackknife while p-value uses conformal inference
Describe the solution you'd like
Add an option to force both the p-value and CI to use the same inference method
Additional context
Hi GeoLift team,
First off, thank you for the fantastic work on this package - it's been incredibly useful, and it's clear a lot of thought went into both the design and the documentation.
While working with GeoLift, I noticed a potentially confusing behavior: sometimes the p-value is statistically significant, but the confidence interval still includes zero. In our case, we saw the warning:
From reviewing the source code, it looks like when conformal inference fails for the CI, GeoLift falls back to jackknife+ for the CI, but still uses conformal inference for the p-value - hence the inconsistency.
While this behavior is expected, it can be confusing for users: which result should be trusted? Would it be possible to add an option to force both the p-value and CI to use the same inference method?
Also open to any advice on how to interpret results in this case.
Thanks again for building and maintaining such a great tool!
Is your feature request related to a problem? Please describe.
Inconsistent conclusions when CI falls back to jackknife while p-value uses conformal inference
Describe the solution you'd like
Add an option to force both the p-value and CI to use the same inference method
Additional context