Merge pull request #2636 from OwenK2/2634-scenario-outline-metadata

Add afterScenarioOutline & karate.scenarioOutline

README.md

@@ -2252,6 +2252,7 @@ You can adjust configuration settings for the HTTP client used by Karate using t
 `printEnabled` | boolean | Can be used to suppress the [`print`](#print) output when not in 'dev mode' by setting as `false` (default `true`)
 `report` | JSON / boolean | see [report verbosity](#report-verbosity)
 `afterScenario` | JS function | Will be called [after every `Scenario`](#hooks) (or `Example` within a `Scenario Outline`), refer to this example: [`hooks.feature`](karate-demo/src/test/java/demo/hooks/hooks.feature)
+`afterScenarioOutline` | JS function | Will be called [after every `Scenario Outline`](#hooks). Is called after the last `afterScenario` for the last scenario in the outline. Refer to this example: [`hooks.feature`](karate-demo/src/test/java/demo/hooks/hooks.feature)
 `afterFeature` | JS function | Will be called [after every `Feature`](#hooks), refer to this example: [`hooks.feature`](karate-demo/src/test/java/demo/hooks/hooks.feature)
 `ssl` | boolean | Enable HTTPS calls without needing to configure a trusted certificate or key-store.
 `ssl` | string | Like above, but force the SSL algorithm to one of [these values](http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#SSLContext). (The above form internally defaults to `TLS` if simply set to `true`).
@@ -3690,6 +3691,7 @@ Operation | Description
 <a name="karate-response"><code>karate.response</code></a> | returns the last HTTP response as a JS object that enables advanced use-cases such as getting a header ignoring case: `karate.response.header('some-header')`
 <a name="karate-request"><code>karate.request</code></a> | returns the last HTTP request as a JS object that enables advanced use-cases such as getting a header ignoring case: `karate.request.header('some-header')`, which works [even in mocks](https://github.com/karatelabs/karate/tree/master/karate-netty#requestheaders)
 <a name="karate-scenario"><code>karate.scenario</code></a> | get metadata about the currently executing `Scenario` (or `Outline` - `Example`) within a test 
+<a name="karate-scenarioOutline"><code>karate.scenarioOutline</code></a> | get metadata about the currently executing scenario outline within a test
 <a name="karate-set"><code>karate.set(name, value)</code></a> | sets the value of a variable (immediately), which may be needed in case any other routines (such as the [configured headers](#configure-headers)) depend on that variable
 <a name="karate-setall"><code>karate.set(object)</code></a> | where the single argument is expected to be a `Map` or JSON-like, and will perform the above `karate.set()` operation for all key-value pairs in one-shot
 <a name="karate-setpath"><code>karate.set(name, path, value)</code></a> | only needed when you need to conditionally build payload elements, especially XML. This is best explained via [an example](karate-core/src/test/java/com/intuit/karate/core/xml/xml.feature#L211), and it behaves the same way as the [`set`](#set) keyword. Also see [`eval`](#eval).
@@ -4440,6 +4442,7 @@ Before *everything* (or 'globally' once) | See [`karate.callSingle()`](#karateca
 Before every `Scenario` | Use the [`Background`](#script-structure). Note that [`karate-config.js`](#karate-configjs) is processed before *every* `Scenario` - so you can choose to put "global" config here, for example using [`karate.configure()`](#karate-configure).
 Once (or at the start of) every `Feature` | Use a [`callonce`](#callonce) in the [`Background`](#script-structure). The advantage is that you can set up variables (using [`def`](#def) if needed) which can be used in all `Scenario`-s within that `Feature`.
 After every `Scenario` | [`configure afterScenario`](#configure) (see [example](karate-demo/src/test/java/demo/hooks/hooks.feature))
+After every `Scenario Outline` | [`configure afterScenarioOutline`](#configure) (see [example](karate-demo/src/test/java/demo/hooks/hooks.feature))
 At the end of the `Feature` | [`configure afterFeature`](#configure) (see [example](karate-demo/src/test/java/demo/hooks/hooks.feature))
 
 > Note that for the `afterFeature` hook to work, you should be using the [`Runner` API](#parallel-execution) and not the JUnit runner.

karate-core/src/main/java/com/intuit/karate/RuntimeHook.java

@@ -46,6 +46,10 @@ default void afterScenario(ScenarioRuntime sr) {
 
     }
 
+    default void afterScenarioOutline(ScenarioRuntime sr) {
+
+    }
+
     default boolean beforeFeature(FeatureRuntime fr) {
         return true;
     }

karate-core/src/main/java/com/intuit/karate/core/AfterHookType.java

@@ -0,0 +1,45 @@
+/*
+ * The MIT License
+ *
+ * Copyright 2022 Karate Labs Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+package com.intuit.karate.core;
+
+/**
+ *
+ * @author OwenK2
+ */
+public enum AfterHookType {
+
+    AFTER_SCENARIO("afterScenario"),
+    AFTER_OUTLINE("afterScenarioOutline"),
+    AFTER_FEATURE("afterFeature");
+
+    private String prefix;
+
+    private AfterHookType(String prefix) {
+        this.prefix = prefix;
+    }
+
+    public String getPrefix() {
+        return prefix;
+    }
+}

karate-core/src/main/java/com/intuit/karate/core/Config.java

@@ -93,6 +93,7 @@ public class Config {
     private HttpLogModifier logModifier;
 
     private Variable afterScenario = Variable.NULL;
+    private Variable afterScenarioOutline = Variable.NULL;
     private Variable afterFeature = Variable.NULL;
     private Variable headers = Variable.NULL;
     private Variable cookies = Variable.NULL;
@@ -175,6 +176,9 @@ public boolean configure(String key, Variable value) { // TODO use enum
             case "afterScenario":
                 afterScenario = value;
                 return false;
+            case "afterScenarioOutline":
+                afterScenarioOutline = value;
+                return false;
             case "afterFeature":
                 afterFeature = value;
                 return false;
@@ -382,6 +386,7 @@ public Config(Config parent) {
         cookies = parent.cookies;
         responseHeaders = parent.responseHeaders;
         afterScenario = parent.afterScenario;
+        afterScenarioOutline = parent.afterScenarioOutline;
         afterFeature = parent.afterFeature;
         continueOnStepFailureMethods = parent.continueOnStepFailureMethods;
         continueAfterContinueOnStepFailure = parent.continueAfterContinueOnStepFailure;
@@ -538,6 +543,14 @@ public void setAfterScenario(Variable afterScenario) {
         this.afterScenario = afterScenario;
     }
 
+    public Variable getAfterScenarioOutline() {
+        return afterScenarioOutline;
+    }
+
+    public void setAfterScenarioOutline(Variable afterScenarioOutline) {
+        this.afterScenarioOutline = afterScenarioOutline;
+    }
+
     public Variable getAfterFeature() {
         return afterFeature;
     }

karate-core/src/main/java/com/intuit/karate/core/ExamplesTable.java

@@ -24,7 +24,9 @@
 package com.intuit.karate.core;
 
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 /**
  *
@@ -35,6 +37,7 @@ public class ExamplesTable {
     private final ScenarioOutline outline;
     private final Table table;
     private List<Tag> tags;
+
 
     public ExamplesTable(ScenarioOutline outline, Table table) {
         this.outline = outline;
@@ -58,4 +61,13 @@ public Table getTable() {
         return table;
     }
 
+    public Map<String, Object> toKarateJson() {
+        Map<String, Object> map = new HashMap();
+        List<String> tagStrings = new ArrayList();
+        tags.forEach(tag -> tagStrings.add(tag.toString()));
+        map.put("tags", tagStrings);
+        map.put("data", table.getRowsAsMapsConverted());
+        return map;
+    }
+
 }

karate-core/src/main/java/com/intuit/karate/core/FeatureRuntime.java

@@ -195,16 +195,38 @@ private void processScenario(ScenarioRuntime sr) {
             if (!sr.result.getStepResults().isEmpty()) {
                 synchronized (result) {
                     result.addResult(sr.result);
+
+                    // Execute afterScenarioOutline if applicable
+                    // NOTE: Needs to be run after adding result, since result count is used to deterime
+                    // if the scenario is the last in the outline
+                    if (!sr.dryRun && isLastScenarioInOutline(sr.scenario)) {
+                        sr.engine.invokeAfterHookIfConfigured(AfterHookType.AFTER_OUTLINE);
+                        suite.hooks.forEach(h -> h.afterScenarioOutline(sr));
+                    }
                 }
             }
         }
     }
 
+    private boolean isLastScenarioInOutline(Scenario scenario) {
+        // Check if scenario is part of an outline
+        if (!scenario.isOutlineExample()) return false;
+
+        // Count the number of completed scenarios with the same section ID (in same outline)
+        int completedScenarios = 0;
+        for (ScenarioResult result : result.getScenarioResults()) {
+            if (result.getScenario().getSection().getIndex() == scenario.getSection().getIndex()) {
+                completedScenarios++;
+            }
+        }
+        return completedScenarios == scenario.getSection().getScenarioOutline().getNumScenarios();
+    }
+
     // extracted for junit5
     public synchronized void afterFeature() {
         result.sortScenarioResults();
         if (lastExecutedScenario != null) {
-            lastExecutedScenario.engine.invokeAfterHookIfConfigured(true);
+            lastExecutedScenario.engine.invokeAfterHookIfConfigured(AfterHookType.AFTER_FEATURE);
             result.setVariables(lastExecutedScenario.engine.getAllVariablesAsMap());
             result.setConfig(lastExecutedScenario.engine.getConfig());
         }

karate-core/src/main/java/com/intuit/karate/core/ScenarioBridge.java

@@ -551,6 +551,10 @@ public Object getScenario() {
         return new JsMap(getEngine().runtime.result.toKarateJson());
     }
 
+    public Object getScenarioOutline() {
+        return new JsMap(getEngine().runtime.outlineResult.toKarateJson());
+    }
+
     public Object getTags() {
         return JsValue.fromJava(getEngine().runtime.tags.getTags());
     }

karate-core/src/main/java/com/intuit/karate/core/ScenarioEngine.java

@@ -235,20 +235,35 @@ public void print(String exp) {
         evalJs("karate.log('[print]'," + exp + ")");
     }
 
-    public void invokeAfterHookIfConfigured(boolean afterFeature) {
+    public void invokeAfterHookIfConfigured(AfterHookType hookType) {
+        // Do not call hooks on "called" scenarios/features
         if (runtime.caller.depth > 0) {
             return;
         }
-        Variable v = afterFeature ? config.getAfterFeature() : config.getAfterScenario();
+
+        // Get hook variable based on type
+        Variable v;
+        switch (hookType) {
+            case AFTER_SCENARIO: 
+                v = config.getAfterScenario();
+                break;
+            case AFTER_OUTLINE: 
+                v = config.getAfterScenarioOutline();
+                break;
+            case AFTER_FEATURE: 
+                v = config.getAfterFeature();
+                break;
+            default: return;
+        }
+
         if (v.isJsOrJavaFunction()) {
-            if (afterFeature) {
+            if (hookType == AfterHookType.AFTER_FEATURE) {
                 ScenarioEngine.set(this); // for any bridge / js to work
             }
             try {
                 executeFunction(v);
             } catch (Exception e) {
-                String prefix = afterFeature ? "afterFeature" : "afterScenario";
-                logger.warn("{} hook failed: {}", prefix, e + "");
+                logger.warn("{} hook failed: {}", hookType.getPrefix(), e + "");
             }
         }
     }

karate-core/src/main/java/com/intuit/karate/core/ScenarioOutline.java

@@ -25,6 +25,7 @@
 
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Map;
 
 /**
  * @author pthomas3
@@ -40,6 +41,7 @@ public class ScenarioOutline {
     private String description;
     private List<Step> steps;
     private List<ExamplesTable> examplesTables;
+    private int numScenarios = 0;
 
     public ScenarioOutline(Feature feature, FeatureSection section) {
         this.feature = feature;
@@ -75,6 +77,7 @@ public Scenario toScenario(String dynamicExpression, int exampleIndex, int updat
             step.setTable(original.getTable());
             step.setComments(original.getComments());
         }
+        numScenarios++;
         return s;
     }
 
@@ -167,9 +170,23 @@ public void setSteps(List<Step> steps) {
     public List<ExamplesTable> getExamplesTables() {
         return examplesTables;
     }
+
+    public int getNumExampleTables() {
+        return examplesTables.size();
+    }
+
+    public List<Map<String, Object>> getAllExampleData() {
+        List<Map<String, Object>> exampleData = new ArrayList();
+        examplesTables.forEach(table -> exampleData.add(table.toKarateJson()));
+        return exampleData;
+    }
 
     public void setExamplesTables(List<ExamplesTable> examplesTables) {
         this.examplesTables = examplesTables;
     }
+
+    public int getNumScenarios() {
+        return numScenarios;
+    }
 
 }

karate-core/src/main/java/com/intuit/karate/core/ScenarioOutlineResult.java

@@ -0,0 +1,82 @@
+/*
+ * The MIT License
+ *
+ * Copyright 2022 Karate Labs Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+package com.intuit.karate.core;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ *
+ * @author OwenK2
+ */
+public class ScenarioOutlineResult {
+
+    final private ScenarioOutline scenarioOutline;
+    final private ScenarioRuntime runtime;
+
+    public ScenarioOutlineResult(ScenarioOutline scenarioOutline, ScenarioRuntime runtime) {
+        // NOTE: this value can be null, in which case the scenario is not from an outline
+        this.scenarioOutline = scenarioOutline;
+        this.runtime = runtime;
+    }
+
+    public Map<String, Object> toKarateJson() {
+        if (scenarioOutline == null) return null;
+        Map<String, Object> map = new HashMap();
+        map.put("name", scenarioOutline.getName());
+        map.put("description", scenarioOutline.getDescription());
+        map.put("line", scenarioOutline.getLine());
+        map.put("sectionIndex", scenarioOutline.getSection().getIndex());
+        map.put("exampleTableCount", scenarioOutline.getNumExampleTables());
+        map.put("exampleTables", scenarioOutline.getAllExampleData());
+        map.put("numScenariosToExecute", scenarioOutline.getNumScenarios());
+
+        // Get results of other examples in this outline
+        List<Map<String, Object>> scenarioResults = new ArrayList();
+        if (runtime.featureRuntime != null && runtime.featureRuntime.result != null) {
+            // Add all past results
+            boolean needToAddRecent = runtime.result != null;
+            for(ScenarioResult result : runtime.featureRuntime.result.getScenarioResults()) {
+                if (result.getScenario().getSection().getIndex() == scenarioOutline.getSection().getIndex()) {
+                    scenarioResults.add(result.toInfoJson());
+                    if(result.equals(runtime.result)) {
+                        needToAddRecent = false;
+                    }
+                }
+            }
+
+            // Add most recent result if we haven't already (and it's not null)
+            if (needToAddRecent) {
+                scenarioResults.add(runtime.result.toInfoJson());
+            }
+        }
+        map.put("scenarioResults", scenarioResults);
+        map.put("numScenariosExecuted", scenarioResults.size());
+
+        return map;
+    }
+
+}