Merge pull request #110 from joernio/andrei/frontend-docs

Initial Frontend Arg Documentation

docs.joern.io/content/frontends/_index.md

@@ -0,0 +1,41 @@
+---
+id: frontends 
+title: Frontends
+bookCollapseSection: true
+weight: 135
+---
+
+## Frontends
+
+Joern contains multiple frontends, each responsible for parsing the source code and generating the AST for their respective language. `X2CPG` is the base on which all of the other frontends are built, and contains common structures by all of the languages. Each frontend contains the base arguments from `x2cpg`, as well as custom arguments that are frontend specific. See each respective languages' page for a detailed explanation on what arguments are available and what each of the arguments do.
+
+## Invoking a specific frontend
+To generate the AST for for some program with a specific frontend, invoke the desired frontend from the `joern-cli/frontends` directory using the generated script. If the script is not there, make sure that Joern has been staged with `sbt`. As an example, to parse a given Python directory and generate an AST:
+```bash
+cd joern-cli/frontends/pysrc2cpg/
+sbt scala stage
+./pysrc2cpg /some/input/path --output someOutput --venvDir /some/venv/dir
+```
+The arguments given after the `pysrc2cpg` command can be a mixture of the args found in `x2cpg`, as well as the language specific args for Python.
+
+If you would like to generate the full `CPG` for a given source directory, you can do so using the `joern-parse` command:
+```bash
+./joern-parse /some/input/path --language PYTHONSRC --frontend-args --venvDir /some/venv/dir
+```
+The language argument selects the frontend used to generate the AST. If this argument is omitted, `joern-parse` will select a frontend based on the supported file type with the largest number of files in the given input directory. The arguments passed after `--frontend-args` will be passed to the frontend that is being invoked.
+
+Below is a list of each of the different frontends and the languages that they support:
+| **Frontend** | **Language** | **Language Arg** |
+| - | - | - |
+| `c2cpg` | `C` | `--language C` |
+| `csharp2cpg` | `C#` | `--language CSHARPSRC` |
+| `ghidra2cpg` | `Ghidra` | `--language GHIDRA` |
+| `gosrc2cpg` | `Golang` | `--language GOLANG` |
+| `javasrc2cpg` | `Java` | `--language JAVASRC` |
+| `jimple2cpg` | `Java` | `--language JAVA` |
+| `jssrc2cpg` | `Javascript` | `--language JAVASCRIPT` |
+| `kotlin2cpg` | `Kotlin` | `--language KOTLIN` |
+| `php2cpg` | `PHP` | `--language PHP` |
+| `pysrc2cpg` | `Python` | `--language PYTHONSRC` |
+| `rubysrc2cpg` | `Ruby` | `--language RUBYSRC` | 
+| `swiftsrc2cpg` | `Swift` | `--language SWIFTSRC` |

docs.joern.io/content/frontends/java.md

@@ -0,0 +1,24 @@
+---
+id: java 
+title: Java 
+weight: 10
+---
+
+### Frontend CLI Args
+The following arguments are specific to the `javasrc2cpg` frontend.
+
+| **Arg** | **Description** | **Type** | **Example** | **Hidden** |
+| - | - | - | - | - |
+| `inference-jar-paths` | Paths to extra jars used only for type information | `List<String>` | `--inference-jar-paths "/path/ex1.jar, /path/ex2.jar"` | `false` |
+| `fetch-dependences` | Attempts to fetch dependencies jars for extra type information | - | `--fetch-dependencies` | `false` |
+| `delombok-java-home` | Optional override to set java home used to run Delombok. Recommended to use Java 17 for best results| `String` | `--delombok-java-home /path/to/home` | `false` |
+| `delombok-mode` | Specifies how Delombok should be executed | `no-delombok \| default \| types-only \| run-delombok` | `--delombok-mode no-delombok` | `false` |
+| `enable-type-recovery` | Enable generic type recovery. | - | `--enable-type-recovery` | `true` |
+| `jdk-path` | JDK path used for resolving builtin Java types. If not set, current classpath will be used | `String` | `--jdk-path /path/to/jdk` | `false` |
+| `show-env` | Print information about the environment variables used by `javasrc2cpg` and exit Joern | - | `--show-env` | `false` |
+| `skip-type-inf-pass` | Skip the type inference pass. Results will be much worse, so should only be used for development. | - | `--skip-type-inf-pass` | `true` |
+| `dump-javaparser-asts` | Dump the `javaparser` ASTs for the given input files and exit Joern. Used for debugging | - | `--dump-javaparser-asts` | `true` |
+| `cache-jdk-type-solver` | Re-use the JDK type solver between scans | - | `--cache-jdk-type-solver` | `true` |
+| `keep-type-arguments` | Type full names of variables keep their type arguments | - | `--keep-type-arguments` | `true` |
+| `no-dummyTypes` | Disables the generation of dummy types during type propagation | - | `--no-dummyTypes` | `true` |
+| `type-prop-iterations` | Maximum iterations of type propagation | `Integer` | `--type-prop-iterations 2` | `true` |

docs.joern.io/content/frontends/javascript.md

@@ -0,0 +1,14 @@
+---
+id: javascript
+title: JavaScript
+weight: 20
+---
+
+### Frontend Args
+The following arguments are specific to the `jssrc2cpg` frontend.
+
+| **Arg** | **Description** | **Type** | **Example** | **Hidden** |
+| - | - | - | - | - |
+| `no-tsTypes` | Disable generation of types via `TypeScript` | - | `--no-tsTypes` | `true` |
+| `no-dummyTypes` | Disables the generation of dummy types during type propagation | - | `--no-dummyTypes` | `true` |
+| `type-prop-iterations` | Maximum iterations of type propagation | `Integer` | `--type-prop-iterations 2` | `true` |

docs.joern.io/content/frontends/python.md

@@ -0,0 +1,18 @@
+---
+id: python 
+title: Python 
+weight: 30
+---
+
+### Frontend CLI Args
+The following arguments are specific to the `pysrc2cpg` frontend.
+
+| **Arg** | **Description** | **Type** | **Example** | **Hidden** |
+| - | - | - | - | - |
+| `venvDir` | Virtual environment directory. If not absolute it is interpreted relative to `input-dir` | `String` | `--venvDir "/some/path/venv/"` | `true` |
+| `venvDirs` | Virtual environment directories. If not absolute it is interpreted relative to `input-dir` | `List<String>` | `--venvDirs "/some/path/venv1/, /some/path/venv2"` | `false` |
+| `ignoreVenvDir` | Specifies whether `venv-dir` is ignored. Default to `true` | - | `--ignoreVenvDir` | `false` |
+| `ignore-paths` | Ignores specified paths from analysis. If not absolute it is interpreted relative to `input-dir` | `List<String>` | `--ignore-paths "/path/to/ignore1, /path/to/ignore2"` | `false` |
+| `ignore-dir-names` | Exclude all files where the relative path from `input-dir` contains at least one of the names specified | `List<String>` | `--ignore-dir-names "rel/path/ignore1, rel/path/ignore2"` | `true` |
+| `no-dummyTypes` | Disables the generation of dummy types during type propagation | - | `--no-dummyTypes` | `true` |
+| `type-prop-iterations` | Maximum iterations of type propagation | `Integer` | `--type-prop-iterations 2` | `true` |

docs.joern.io/content/frontends/x2cpg.md

@@ -0,0 +1,17 @@
+---
+id: x2cpg 
+title: X2CPG
+weight: 5
+---
+
+## Frontend Args
+The frontend arguments for `x2cpg` is shared across all frontends for `Joern`. See the individual Frontend pages for frontend-specific args.
+
+| **Arg** | **Description** | **Type** | **Example** |
+| - | - | - | - |
+| `input-dir` | Directory containing all source files to be used with Joern | `String` | `"/some/path/to/folder"` |
+| `output` | Name of the file where output is written to | `String` | `--output "output.txt"` |
+| `exclude` | List of input file names to be excluded during CPG generation (paths must be relative to `input-dir` or absolute paths) | `List<String>` | `--exclude "test1.java, test2.java, ..."` |
+| `exclude-regex` | A regex specifying files to be excluded during CPG generation (paths must be relative to `input-dir` or absolute paths) | `String` | `--exclude-regex ".*-test.java"` |
+| `enable-early-schema-checking` | Enables early schema validation during AST creation (disabled by default)| - | `--enable-early-schema-checking` |
+| `enable-file-content` | Add the raw source code to the content of the FILE nodes to allow for method source retrieval via offset fields (disabled by default) | - | `--enable-file-content` |