Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
1504b35
Upgrade node; reset deps; switch to biome
cavis Sep 15, 2025
2905e71
Biome formatting/linting
cavis Sep 15, 2025
1fdcc98
Init blame ignore
cavis Sep 15, 2025
2a7a054
Run cjs-to-es6
cavis Sep 15, 2025
06dde45
Start on Jest testing
cavis Sep 15, 2025
1305d24
Pingurl updates + tests
cavis Sep 16, 2025
0ce3993
Aaaand the tests
cavis Sep 16, 2025
734901f
Tests for get records
cavis Sep 19, 2025
977e2ef
My first SDK v3 test!
cavis Sep 19, 2025
386b712
Unused stuffs
cavis Sep 19, 2025
f40ef28
Much updated BQ
cavis Sep 19, 2025
c271280
Move more functionality into decoder; rename
cavis Sep 25, 2025
bc8e147
Entrypoint for BQ inserts
cavis Sep 25, 2025
add1ddd
Move kinesis event builders to shared test helper
cavis Sep 25, 2025
a8ab24b
Remove unused support files
cavis Sep 25, 2025
cb4ef08
Remove already ported tests
cavis Sep 25, 2025
62c1941
Port bigquery tests
cavis Sep 25, 2025
a6130bc
Handle nulls
cavis Sep 26, 2025
25f8fdf
Start porting pingbacks
cavis Sep 26, 2025
42bdadd
Add in DTR increments
cavis Sep 26, 2025
03d7b95
More testing
cavis Sep 29, 2025
2ab8874
Remove ported tests
cavis Sep 29, 2025
b9ca1f7
Refactor DDB results into helper class
cavis Sep 30, 2025
b81efc9
Cobble dynamodb lambda back together
cavis Sep 30, 2025
1225f01
Fixup build
cavis Sep 30, 2025
ab40451
Update package manager
cavis Sep 30, 2025
d4028cd
Lower expectations
cavis Sep 30, 2025
07d540a
Port remaining ddb tests
cavis Oct 1, 2025
3280481
Abstract out dynamo parallel worker
cavis Oct 1, 2025
a5cd9cb
Unused
cavis Oct 1, 2025
97f5ff7
Helpers for frequency commands
cavis Oct 1, 2025
7d0c7d8
Port frequency caps
cavis Oct 1, 2025
63e9762
Remove remaining old code/tests
cavis Oct 1, 2025
402e893
Change to arm64
cavis Oct 2, 2025
ee96341
Use timestamp strings, so we can send milliseconds to BQ
cavis Oct 2, 2025
125f1fc
Start on integration tests
cavis Oct 2, 2025
153ce37
Fix utc-day deduping
cavis Oct 3, 2025
16f006b
Dynamo integration tests
cavis Oct 3, 2025
6b8287b
Integration test for frequency
cavis Oct 3, 2025
8972516
Update readme
cavis Oct 3, 2025
260d770
Move a bunch of stuff to the contributing file
cavis Oct 3, 2025
d81b99d
Fix link
cavis Oct 3, 2025
bef7e5f
Unused fixture
cavis Oct 3, 2025
a6c44b4
Publish this branch (for now)
cavis Oct 8, 2025
510085b
Apparently these need .js extensions
cavis Oct 8, 2025
0e7c2b7
Missed one
cavis Oct 8, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Editor configuration, see https://editorconfig.org
root = true

[*]
charset = utf-8
indent_style = space
indent_size = 2
insert_final_newline = true
trim_trailing_whitespace = true
3 changes: 3 additions & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# .git-blame-ignore-revs
# biome check --write --unsafe
2905e711bcd47376c1a474888eda4dc274bf2485
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,7 @@ jspm_packages
.env
build
*.zip

# sublimetext
*.sublime-project
*.sublime-workspace
7 changes: 6 additions & 1 deletion .lambdaignore
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
.dockerignore
.editorconfig
.env
.git
.git-blame-ignore-revs
.gitignore
.lambdaignore
.nvmrc
.tool-versions
biome.json
Dockerfile
docker-compose.yml
env-example
/build*
/test
/tmp
*.test.js
*.log
*.zip
1 change: 0 additions & 1 deletion .nvmrc

This file was deleted.

9 changes: 0 additions & 9 deletions .prettierrc.json

This file was deleted.

1 change: 1 addition & 0 deletions .tool-versions
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
nodejs 22.19.0
82 changes: 82 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# PRX Analytics Ingest Lambda

## Getting started

PRX uses [asdf](https://asdf-vm.com/) to manage NodeJS - make sure you've got the correct
version of node, and then just run `yarn` to install dependencies.

## Unit Tests

And hey, to just run the unit tests locally, you don't need anything! Just
`yarn test` to your heart's content.

There are some dynamodb tests that use an actual table, and will be skipped. To
also run these, set `TEST_DDB_TABLE` and `TEST_DDB_ROLE` to something in AWS you
have access to.

## Integration Tests

There are also some tests that have external dependencies, located in the
`test/` folder. To run them, use `yarn integration`.

These tests have external AWS/Google dependencies, so you'll need to copy
`env-example` to `.env` and fill in your info. Those dependencies are:

### BigQuery

To enable BigQuery inserts, you'll need to first [create a Google Cloud Platform Project](https://cloud.google.com/resource-manager/docs/creating-managing-projects),
create a BigQuery dataset, and create the `dt_downloads` and `dt_impressions`
tables required.

Then [create a Service Account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount) for this app. Make sure it has BigQuery Data Editor permissions.

Sorry -- no help on creating the correct table scheme yet! But they already
exist in the PRX GCP account, so just use those!

### DynamoDB

To enable DynamoDB gets/writes, you'll need to setup a [DynamoDB table](https://docs.aws.amazon.com/dynamodb/index.html#lang/en_us)
that your account has access to. You can use your local AWS cli credentials, or
setup AWS client/secret environment variables.

You can also optionally access a DynamoDB table in a different account by specifying
a `DDB_ROLE` that the lambda should assume while doing gets/writes.

You'll need 2 tables, for redirect/segment data and frequency caps. Sorry, no help
here for the schemas on those. But they already exist in the PRX AWS account, so
you should just be able to use them here.

## Linting

This project uses [Biome](https://biomejs.dev/) for code formatting and linting. It's a
dependency in `package.json`, so just make sure your editor is configured to run it.

Or run things manually with `yarn lint` and `yarn lint-fix`.

# Deployment

This lambda is built into a zip file using some simple scripting in `package.json`. You
can try it locally by running `yarn build`, and it'll make you a deployable zipfile.
However, it's probably better to let CI build it, as it uses the official AWS lambda
docker image (just in case there are native dependencies to build).

These lambda functions are deployed via a Cloudformation stack in the [Infrastructure repo](https://github.com/PRX/Infrastructure/blob/main/spire/templates/apps/dovetail-analytics.yml) as:

- `AnalyticsDynamoDbFunction`
- `AnalyticsBigqueryFunction`
- `AnalyticsFrequencyFunction`
- `AnalyticsPingbacksFunction`

See that repo for all the alarms/streams/etc goodness.

And you can easily-ish get the lambda zip built by the Dockerfile:

```
docker ps -a | grep analyticsingestlambda
docker cp {{container-id-here}}:/app/build.zip myzipfile.zip
unzip -l myzipfile.zip
```

# License

[AGPL License](https://www.gnu.org/licenses/agpl-3.0.html)
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM public.ecr.aws/lambda/nodejs:16
FROM public.ecr.aws/lambda/nodejs:22

MAINTAINER PRX <[email protected]>
LABEL org.prx.lambda="true"
Expand All @@ -9,7 +9,7 @@ WORKDIR /app
ENTRYPOINT [ "yarn", "run" ]
CMD [ "test" ]

RUN yum install -y rsync zip tar xz && yum clean all && rm -rf /var/cache/yum
RUN dnf install -y rsync zip && dnf clean all
ADD yarn.lock ./
ADD package.json ./
RUN npm install --quiet --global yarn && yarn install
Expand Down
Loading