Merge pull request #13 from jreyesr/feat/expose-status-code-and-headers

Read outputs from headers, status code and req time

.github/workflows/test.yml

@@ -10,9 +10,8 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-node@v3
         with:
-          node-version: 16
+          node-version: 20
       - run: npm ci
       - run: npm test --coverage
       - name: Coveralls
         uses: coverallsapp/github-action@v1
-

CHANGELOG.md

@@ -1,10 +1,18 @@
+# v1.4.0
+
+**Date:** 2024/05/22
+
+### New features
+
+- Add a way to read outputs (that will be written back to the CSV) from the response's headers, status code and request time, in addition to the response body as JSON ([#13](https://github.com/jreyesr/insomnia-plugin-batch-requests/pull/13))
+
 # v1.3.0
 
 **Date:** 2024/01/20
 
 ### New features
 
-- Add a configuration option to set default delay for all requests ([#11](https://github.com/jreyesr/insomnia-plugin-batch-requests/pull/11))
+- Add the ability to send requests in parallel, with configurable parallelism ([#11](https://github.com/jreyesr/insomnia-plugin-batch-requests/pull/11))
 
 # v1.2.0
 

README.md

@@ -5,11 +5,11 @@
 The Batch Requests plugin for [Insomnia](https://insomnia.rest) adds the ability to send a request repeatedly, changing parts of every request by variable data, taken from a CSV file. For every response, some data can be collected and added to the CSV file.
 
 - Repeatedly send a request by reading data from a CSV file
-- Extract data from JSON responses and write it back to the CSV file
+- Extract data from JSON responses (or other sources, such as the response headers, status code and time taken) and write it back to the CSV file
 - Works well if not using the plugin (when sending the request manually)
 - Add a delay between each request
 - Run multiple requests in parallel
-- Supports non-JSON responses too (but can't extract response data in such cases)
+- Supports non-JSON responses too (but can't extract response data in such cases. Can still use response headers and status code)
 
 ![A diagram displaying the flow of data in the plugin](images/flow.png)
 
@@ -46,9 +46,27 @@ On the plugin dialog, you should:
 
 1. Select a CSV file using the button. The file should have one column for each different placeholder/template tag that you have selected, plus one column for each result that you want to extract from the responses. The response/output columns can be empty, since they will be filled by the plugin.
 2. Review the loaded data in the table. It will show the first five rows of the CSV file. It is provided as a sanity check, so that you can verify that the CSV is being parsed correctly.
-3. (Optional) Configure the data that you want to output by adding `Outputs`. For each one, use the dropdown on the left to specify a CSV column, and write a JSONPath expression in the text field on the right. In the image below, the `$.total` field will be written to the `sales` column in the CSV file. This plugin uses [the `jsonpath-plus` syntax](https://www.npmjs.com/package/jsonpath-plus), which is [also used by Insomnia](https://docs.insomnia.rest/insomnia/responses#filter)
-4. Click the `Run!` button at the bottom of the dialog. It will only become active when you have chosen a file and (if any outputs exist) completely filled all Outputs.
-5. Click the `Save` button to write the extracted data back to the CSV file, if you need it. Wait until all requests have been performed (as indicated by the progress bar) before clicking this button.
+3. (Optional) Configure the data that you want to output by adding `Outputs`:
+   - Use the dropdowns on the left to specify a CSV column
+   - Use the dropdowns on the middle to specify from where the data will be collected. See [below](#sources-of-output-data) for the available sources
+   - If data is being read from the response body, write a JSONPath expression in the text fields on the right. In the image below, the `$.total` field will be written to the `sales` column in the CSV file. This plugin uses [the `jsonpath-plus` syntax](https://www.npmjs.com/package/jsonpath-plus), which is [also used by Insomnia](https://docs.insomnia.rest/insomnia/responses#filter)
+   - If data is being read from the response headers, write a header name in the text fields on the right. For example, write `content-length` to fetch the response's length. This field is case-insensitive (i.e. you don't need to match the exact casing returned by the server)
+4. If desired, specify a delay between requests, or a number of parallel requests. By default, no delay is applied, and requests are sent in sequence (one after the other, with no parallelization). See [below](#extra-settings) for more information.
+5. Click the `Run!` button at the bottom of the dialog. It will only become active when you have chosen a file and (if any outputs exist) completely filled all Outputs.
+6. Click the `Save` button to write the extracted data back to the CSV file, if you need it. Wait until all requests have been performed (as indicated by the progress bar) before clicking this button.
+
+### Sources of output data
+
+![a closeup of the UI, showing the Outputs section, with one output of each type (response body, response header, response status code, and request elapsed time)](images/output_datasources.png)
+
+Since `v1.4.0`, it's possible to extract data from several places in the response:
+
+- The response body (must be JSON). This is the default option and the only one available before `v1.4.0`. This option requires specifying [a JSONPath expression](https://www.npmjs.com/package/jsonpath-plus#syntax-through-examples) to extract a specific value from the JSON response, such as `$.data.id`
+- The response headers. This option requires specifying a header name,
+- The response status code (the numerical one, such as `200`). This option does _not_ require any further configuration
+- The time taken by the request (in milliseconds). This option does _not_ require any further configuration
+
+The source of data is chosen in the center dropdown of each Output. If required, the right-hand text field will appear and must contain something, otherwise it'll be hidden.
 
 ### Extra settings
 

__tests__/OutputFieldsChooser.test.js

@@ -31,7 +31,7 @@ it('notifies parent when field is added', async () => {
 
   await user.click(getByText("Add"));
 
-  expect(onChange).toBeCalledWith([{jsonPath: "", name: ""}]);
+  expect(onChange).toBeCalledWith([{jsonPath: "", name: "", "context": "body"}]);
 });
 
 it('notifies parent when field is deleted', async () => {
@@ -63,5 +63,20 @@ it('notifies parent when field is updated', async () => {
   await user.type(within(field).getByTestId("value"), "$.some.field");
   await user.selectOptions(within(field).getByTestId("fieldname"), "a");
 
-  expect(onChange).toHaveBeenLastCalledWith([{"jsonPath": "$.some.field", "name": "a"}]);
-});
+  expect(onChange).toHaveBeenLastCalledWith([{"jsonPath": "$.some.field", "name": "a", "context": "body"}]);
+});
+
+it("tracks the output's context", async () => {
+  const onChange = jest.fn();
+  const user = userEvent.setup();
+  const {getByText, getByTestId} = render(
+    <OutputFieldsChooser colNames={["a", "b"]} onChange={onChange} />,
+  );
+  await user.click(getByText("Add"));
+  onChange.mockClear();
+
+  const field = getByTestId("singlefield");
+  await user.selectOptions(within(field).getByTestId("context"), "Status code");
+
+  expect(onChange).toHaveBeenLastCalledWith([{"jsonPath": "", "name": "", "context": "statusCode"}]);
+})

__tests__/RequestSender.test.js

@@ -0,0 +1,56 @@
+import * as utils from '../utils';
+
+const mockSendRequest = jest.fn();
+
+const originalReadResponseFromFile = utils.readResponseFromFile
+utils.readResponseFromFile = jest.fn()
+
+beforeEach(() => {
+  utils.readResponseFromFile.mockImplementation(originalReadResponseFromFile)
+})
+
+afterEach(() => {
+  jest.clearAllMocks();
+})
+
+const mockContext = {
+  network: {
+    sendRequest: mockSendRequest,
+  },
+};
+
+it('parses outputs', async () => {
+  // Set up a mock response. We need to mock the actual response object,
+  // and also the code that reads the response's body from disk (on Insomnia,
+  // the response object only holds the filesystem path to the response body, plus resp. metadata)
+  mockSendRequest.mockReturnValue({
+    bytesRead: 42,
+    contentType: "application/json; charset=utf-8",
+    headers: [{name: "content-LENGTH", value: "123"}, {name: "X-Header", value: "X-Header value"}],
+    statusCode: 418, //  I'm a teapot!
+    elapsedTime: 999.999,
+  })
+  // this is the actual response body, we're intercepting code that reads data from disk
+  jest.spyOn(utils, "readResponseFromFile").mockReturnValue('{"foo": "Bar", "baz": "quux"}')
+
+  // CSV has two rows, first one will be changed
+  let csvData = [{row: 1}, {row: 2}]; 
+  const setCsvData = (cb) => {csvData = cb(csvData)};
+
+  await utils.makeRequest(mockContext, /*request*/null, 
+    /*i*/0, /*row*/csvData[0], 
+    /*delay*/0,
+    /*outputConfig*/[
+      {name: "d", context: "body", jsonPath: "$.foo"},
+      {name: "e", context: "headers", jsonPath: "CONTENT-length"},
+      {name: "f", context: "statusCode", jsonPath: ""},
+      {name: "g", context: "reqTime", jsonPath: ""},
+    ],
+    /*setSent*/() => {}, setCsvData
+  )
+
+  // Expected value: still two rows, second one must be untouched.
+  // First row has some columns added, one for each output field passed
+  // Values of output fields must be taken from the (mock) "response"
+  expect(csvData).toEqual([{row: 1, d: "Bar", e: "123", f: "418", g: "999.999"}, {row: 2}])
+});

components/BatchDialog.js

@@ -45,7 +45,10 @@ export default function BatchDialog({context, request}) {
     writeFile(csvPath, outString);
   }, [csvData, csvHeaders, csvPath]);
 
-  const canRun = csvData.length > 0 && outputConfig.every(x => x.name && x.jsonPath);
+  // Valid output configs:
+  // * Must contain a name, AND
+  // * Must contain a jsonPath, OR must be statusCode or reqTime (which don't need a jsonPath)
+  const canRun = csvData.length > 0 && outputConfig.every(x => x.name && (x.jsonPath || ["statusCode", "reqTime"].includes(x.context)));
   const onRun = async () => {
     setSent(0);
 
@@ -73,6 +76,8 @@ export default function BatchDialog({context, request}) {
     setOutputConfig(x)
   }
 
+  console.debug("canRun", outputConfig.map(x => `${x.name} - ${x.jsonPath} - ${x.context} - ${Boolean(x.name && (x.jsonPath || ["statusCode", "reqTime"].includes(x.context))) ? "T" : "F"}`))
+
   const onChangeDelay = ({target: {value}}) => {
     if(value < 0) return;
     setDelay(value)

components/OutputField.js

@@ -1,14 +1,24 @@
 import React, { useCallback } from 'react';
 import ActionButton from './ActionButton';
 
-export default function OutputField({options, name, jsonPath, onChange, onDelete}) {
+export default function OutputField({options, name, context, jsonPath, onChange, onDelete}) {
   const onChangeName = useCallback((e) => {
-    onChange(e.target.value, jsonPath)
-  }, [jsonPath, onChange]);
+    onChange(e.target.value, context, jsonPath)
+  }, [context, jsonPath, onChange]);
+
+  const onChangeContext = useCallback((e) => {
+    onChange(name, e.target.value, jsonPath)
+  }, [name, jsonPath, onChange])
 
   const onChangeJsonPath = useCallback((e) => {
-    onChange(name, e.target.value)
-  }, [name, onChange]);
+    onChange(name, context, e.target.value)
+  }, [name, context, onChange]);
+
+  const placeholder = {
+    body: "$.store.books[*].author",
+    headers: "X-Some-Header"
+  }[context];
+  const shouldShowValueField = ["body", "headers"].includes(context);
 
   return <div className="form-row" data-testid="singlefield">
     <select
@@ -19,7 +29,22 @@ export default function OutputField({options, name, jsonPath, onChange, onDelete
       <option value="">---Choose one---</option>
       {options.map(o => <option key={o} value={o}>{o}</option>)}
     </select>
-    <input type="text" value={jsonPath} onChange={onChangeJsonPath} placeholder='$.store.books[*].author' data-testid="value"/>
+
+    ⟸
+
+    <select value={context} onChange={onChangeContext} data-testid="context">
+      <option value="body">From body</option>
+      <option value="headers">From header</option>
+      <option value="statusCode">Status code</option>
+      <option value="reqTime">Request time (millis)</option>
+    </select>
+
+    <input
+      style={{visibility: shouldShowValueField ? 'visible' : 'hidden' }}
+      type="text" value={jsonPath} 
+      onChange={onChangeJsonPath} 
+      placeholder={placeholder} data-testid="value"/>
+
     <ActionButton title="" icon="fa-trash" onClick={onDelete} data-testid="deletebtn"/>
   </div>
 }

components/OutputFieldsChooser.js

@@ -8,15 +8,15 @@ export default function OutputFieldsChooser({colNames, onChange}) {
   const [outputs, setOutputs] = useState([]);
 
   const addNew = useCallback(() => {
-    const newVal = outputs.concat([{name: "", jsonPath: ""}]);
+    const newVal = outputs.concat([{name: "", context: "body", jsonPath: ""}]);
     setOutputs(newVal);
     onChange(newVal);
   }, [outputs, setOutputs, onChange]);
 
-  const updateField = useCallback((i) => (newName, newJsonPath) => {
+  const updateField = useCallback((i) => (newName, newContext, newJsonPath) => {
     // Poor man's deep copy, since I'm not sure if you should modify React state in place
     const cloned = JSON.parse(JSON.stringify(outputs))
-    cloned[i] = {name: newName, jsonPath: newJsonPath};
+    cloned[i] = {name: newName, context: newContext, jsonPath: newJsonPath};
     setOutputs(cloned);
     onChange(cloned);
   }, [outputs, setOutputs, onChange]);
@@ -30,7 +30,11 @@ export default function OutputFieldsChooser({colNames, onChange}) {
 
   return <FormRow label="Outputs">
     {outputs.map((o, i) => 
-      <OutputField key={i} options={colNames} name={o.name} jsonPath={o.jsonPath} onChange={updateField(i)} onDelete={deleteField(i)}/>
+      <OutputField key={i} 
+        options={colNames} 
+        name={o.name} context={o.context} jsonPath={o.jsonPath} 
+        onChange={updateField(i)} onDelete={deleteField(i)}
+      />
     )}
     <ActionButton title="Add" icon="fa-plus" onClick={addNew}/>
   </FormRow>

package.json

@@ -5,7 +5,7 @@
   "repository": {
     "url": "https://github.com/jreyesr/insomnia-plugin-batch-requests"
   },
-  "version": "1.3.0",
+  "version": "1.4.0",
   "author": {
     "name": "jreyesr",
     "url": "https://github.com/jreyesr"

utils.js

@@ -74,19 +74,44 @@ export async function makeRequest(context, request, i, row, delay, outputConfig,
     return
   }
 
-  // Check that the Content-Type header is sensible, otherwise error out
-  if(!response.contentType.includes("json")) {
-    context.app.alert("Error!", `The response has invalid Content-Type "${response.contentType}", needs "application/json"! Alternatively, delete all Outputs and try again.`)
-    return // There's no point in attempting to parse the response, just jump to the next request
+  let responseData = {};
+  // If any outputConfigs refer to the response body, we must parse it
+  if(outputConfig.some(x => x.context === "body")) {
+    // Check that the Content-Type header is sensible, otherwise error out
+    if(!response.contentType.includes("json")) {
+      context.app.alert("Error!", `The response has invalid Content-Type "${response.contentType}", needs "application/json"! Alternatively, delete all Outputs and try again.`)
+      return // There's no point in attempting to parse the response, just jump to the next request
+    }
+
+    console.debug("parsing response data")
+    // NOTE: The exports.XYZ is required so mocks can hook this
+    // See https://medium.com/welldone-software/jest-how-to-mock-a-function-call-inside-a-module-21c05c57a39f
+    responseData = JSON.parse(exports.readResponseFromFile(response.bodyPath))
   }
-
-  console.debug("parsing response data")
-  // Read the response data, then apply JSONPath expressions on it and update the CSV data
-  const responseData = JSON.parse(readResponseFromFile(response.bodyPath))
   console.debug(responseData)
-  for(const {name, jsonPath} of outputConfig) {
-    let out = applyJsonPath(jsonPath, responseData) ?? null
-    console.debug(name, "+", jsonPath, "=>", out)
+
+  // WEIRD: Labeled statement! https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/label
+  writerForLoop:
+  for(const {name, jsonPath, context: ctx} of outputConfig) {
+    let out;
+    switch(ctx) {
+      case "body":
+        out = applyJsonPath(jsonPath, responseData) ?? null
+        break
+      case "headers":
+        out = response.headers.find(h => h.name.toLowerCase() === jsonPath.toLowerCase()).value
+        break
+      case "statusCode":
+        out = response.statusCode.toString()
+        break
+      case "reqTime":
+        out = response.elapsedTime.toString()
+        break
+      default:
+        console.error("Unknown outputConfig context:", "name", name, "jsonPath", jsonPath, "context", ctx)
+        continue writerForLoop // Skip to next outputConfig
+    } 
+    console.debug(name, "+", jsonPath, "@", ctx, "=>", out)
 
     setCsvData(csvData => {
       let newData = [...csvData] // Make a copy of the old data so we can mutate it normally