Merge branch 'master' of https://github.com/CODAIT/text-extensions-for-pandas into notebook_review

Author: frreiss

.readthedocs.yml

@@ -5,9 +5,9 @@
 # Required
 version: 2
 
-# Build documentation in the docs/ directory with Sphinx
+# Build documentation in the api_docs/ directory with Sphinx
 sphinx:
-  configuration: docs/conf.py
+  configuration: api_docs/conf.py
 
 # Build documentation with MkDocs
 #mkdocs:

.travis.yml

@@ -1,34 +1,34 @@
 language: python
-python:
-- "3.7"
 
 jobs:
   include:
+    - python: "3.6"
     - name: "Pandas 1.0.x"
+      python: "3.7"
       env: PANDAS_VERSION=1.0.*
     - name: "Pandas 1.1.x"
+      python: "3.7"
       env: PANDAS_VERSION=1.1.*
 
 install:
-#install conda
- - sudo apt-get update
- - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
- - bash miniconda.sh -b -p $HOME/miniconda
- - source "$HOME/miniconda/etc/profile.d/conda.sh"
- - hash -r
- - conda config --set always_yes yes --set changeps1 no
- - conda update -q conda
+  #install conda
+  - sudo apt-get update
+  - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
+  - bash miniconda.sh -b -p $HOME/miniconda
+  - source "$HOME/miniconda/etc/profile.d/conda.sh"
+  - hash -r
+  - conda config --set always_yes yes --set changeps1 no
+  - conda update -q conda
 
- - conda info -a
+  - conda info -a
 
- - CONDA_HOME="${HOME}/miniconda" ./env.sh
+  - CONDA_HOME="${HOME}/miniconda" ./env.sh
 
 
 script:
-
- #activate python virtual environment
- - conda activate pd
- #check that doc generation is possible
- - ./generate_docs.sh
- #run unit tests 
- - pytest -v text_extensions_for_pandas
+  #activate python virtual environment
+  - conda activate pd
+  #check that doc generation is possible
+  - ./generate_docs.sh
+  #run unit tests
+  - pytest -v text_extensions_for_pandas

LICENSE renamed to LICENSE.txt

@@ -1,114 +1,131 @@
+
 # Text Extensions for Pandas
+
+[![Documentation Status](https://readthedocs.org/projects/text-extensions-for-pandas/badge/?version=latest)](https://text-extensions-for-pandas.readthedocs.io/en/latest/?badge=latest)
+
 Natural language processing support for Pandas dataframes.
 
-**This project is under development.** Releases are not yet available.
+Text Extensions for Pandas turns Pandas DataFrames into a universal data
+structure for representing intermediate data in all phases of your NLP
+application development workflow.
 
-## Purpose of this Project
+## Features
 
-Natural language processing (NLP) applications tend to consist of multiple components tied together in a complex pipeline. These components can range from deep parsers and machine learning models to lookup tables and business rules. All of them work by creating and manipulating data structures that represent data about the target text --- things like tokens, entities, parse trees, and so on.
+### SpanArray: A Pandas extension type for *spans* of text
 
-Libraries for common NLP tasks tend to implement their own custom data structures. They also implement basic low-level operations like filtering and pattern matching over these data structures. For example, `nltk` represents named entities as a list of Python objects:
+* Connect features with regions of a document
+* Visualize the internal data of your NLP application
+* Analyze the accuracy of your models
+* Combine the results of multiple models
 
-```python
->>> entities = nltk.chunk.ne_chunk(tagged)
->>> entities
-Tree('S', [('At', 'IN'), ('eight', 'CD'), ("o'clock", 'JJ'),
-           ('on', 'IN'), ('Thursday', 'NNP'), ('morning', 'NN'),
-       Tree('PERSON', [('Arthur', 'NNP')]),
-           ('did', 'VBD'), ("n't", 'RB'), ('feel', 'VB'),
-           ('very', 'RB'), ('good', 'JJ'), ('.', '.')])
-```
+### TensorArray: A Pandas extension type for tensors
 
-...while SpaCy represents named entities with the an `Iterable` of `Span` objects:
+* Represent BERT embeddings in a Pandas series
+* Store logits and other feature vectors in a Pandas series
+* Store an entire time series in each cell of a Pandas series
 
-```python
->>> doc = nlp("At eight o'clock on Thursday morning, Arthur didn't feel very good.")
->>> ents = [(e.text, e.start_char, e.end_char, e.label_) for e in doc.ents]
->>> ents
-[("eight o'clock", 3, 16, 'TIME'), ('Thursday', 20, 28, 'DATE'), ('morning', 29, 36, 'TIME'), ('Arthur', 38, 44, 'PERSON')]
-```
+### Pandas front-ends for popular NLP toolkits
+
+* [SpaCy](https://spacy.io/)
+* [Transformers](https://github.com/huggingface/transformers)
+* [IBM Watson Natural Language Understanding](https://www.ibm.com/cloud/watson-natural-language-understanding)
+* [IBM Watson Discovry Table Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables)
+
+
+## Installation
 
-...or an `Iterable` of `Token` objects with tags:
+This library requires Python 3.7+, Pandas, and Numpy. 
 
-```python
->>> doc = nlp("At eight o'clock on Thursday morning, Arthur didn't feel very good.")
->>> token_info = [(t.text, t.ent_iob_, t.ent_type_) for t in doc]
->>> token_info
-[('At', 'O', ''), ('eight', 'B', 'TIME'), ("o'clock", 'I', 'TIME'), ('on', 'O', ''), ('Thursday', 'B', 'DATE'), ('morning', 'B', 'TIME'), (',', 'O', ''), ('Arthur', 'B', 'PERSON'), ('did', 'O', ''), ("n't", 'O', ''), ('feel', 'O', ''), ('very', 'O', ''), ('good', 'O', ''), ('.', 'O', '')]
+To install the latest release, just run:
 ```
+pip install text-extensions-for-pandas
+```
+
+Depending on your use case, you may also need the following additional
+packages:
+* `spacy` (for SpaCy support)
+* `transformers` (for 
+* `ibm_watson` (for IBM Watson support)
 
-...and IBM Watson Natural Language Understanding represents named entities as an array of JSON records:
-
-```JSON
-{
-  "entities": [
-    {
-      "type": "Person",
-      "text": "Arthur",
-      "count": 1,
-      "confidence": 0.986158
-    }
-  ]
-}
+## Installation from Source
+
+If you'd like to try out the very latest version of our code, 
+you can install directly from the head of the master branch:
+```
+pip install git+https://github.com/CODAIT/text-extensions-for-pandas
 ```
 
-This duplication leads to a great deal of redundant work when building NLP applications.  Developers need to understand and remember how every component represents every type of data. They need to write code to convert among different representations, and they and need to implement common operations like pattern matching multiple times for different, equivalent data structures.
+You can also directly import our package from your local copy of the 
+`text_extensions_for_pandas` source tree. Just add the root of your local copy
+of this repository to the front of `sys.path`.
 
-It is our belief that, with a few targeted improvements, we can make [Pandas](https://pandas.pydata.org/) dataframes into a universal representation for all the data that flows through NLP applications. Such a universal data structure would eliminate redundancy and make application code simpler, faster, and easier to debug.
+## Documentation
 
-This project aims to create the extensions that will turn Pandas into this universal data structure. In particular, we plan to add three categories of extension:
+For examples of how to use the library, take a look at the notebooks in 
+[this directory](https://github.com/CODAIT/text-extensions-for-pandas/tree/master/notebooks).
 
-* **New Pandas series types to cover spans and tensors.** These types of data are very important for NLP applications but are cumbersome to represent with "out-of-the-box" Pandas. The new [extensions API](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.ExtensionArray.html) that Pandas released in 2019 makes it possible to create performant extension types. We will use this API to add three new series types: CharSpan, TokenSpan (span with token offsets), and Tensor. 
-* **An implementation of spanner algebra over Pandas dataframes.** The core operations of the [Document Spanners](https://researcher.watson.ibm.com/researcher/files/us-fagin/jacm15.pdf)formalism represent tasks that occur repeatedly in NLP applications. Many of these core operations are already present in Pandas. We will create high-performance implementations of the remaining operations over Pandas dataframes. This work will build directly on our Pandas extension types for representing spans.
+API documentation can be found at [https://text-extensions-for-pandas.readthedocs.io/en/latest/](https://text-extensions-for-pandas.readthedocs.io/en/latest/)
 
-## Getting Started
 
-### Contents of this repository
+## Contents of this repository
 
 * **`text_extensions_for_pandas`**: Source code for the `text_extensions_for_pandas` module.
-* **notebooks**: demo notebooks
-* **resources**: various input files used by the demo notebooks 
-* **env.sh**: Script to create an conda environment `pd` capable of running the notebooks in this directory
-
-### Instructions to run a demo notebook
+* **env.sh**: Script to create a conda environment `pd` capable of running the notebooks and test cases in this project
+* **generate_docs.sh**: Script to build the [API documentation]((https://readthedocs.org/projects/text-extensions-for-pandas/)
+* **api_docs**: Configuration files for `generate_docs.sh`
+* **config**: Configuration files for `env.sh`.
+* **docs**: Project web site
+* **notebooks**: example notebooks
+* **resources**: various input files used by our example notebooks 
+* **test_data**: data files for regression tests. The tests themselves are
+  located adjacent to the library code files.
+* **tutorials**: Detailed tutorials on using Text Extensions for Pandas to
+  cover complex end-to-end NLP use cases (work in progress).
+
+
+## Instructions to run a demo notebook
 1. Check out a copy of this repository
 1. (optional) Use the script `env.sh` to set up an Anaconda environment for running the code in this repository.
 1. Type `jupyter lab` from the root of your local source tree to start a [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/) environment.
-1. Navigate to the example notebook `notebooks/Person.ipynb`
+1. Navigate to the `notebooks` directory and choose any of the notebooks there
+
+
+## Contributing
+
+This project is an IBM open source project. We are developing the code in the open under the [Apache License](https://github.com/CODAIT/text-extensions-for-pandas/blob/master/LICENSE), and we welcome contributions from both inside and outside IBM. 
 
-### Installation instructions
+To contribute, just open a Github issue or submit a pull request. Be sure to include a copy of the [Developer's Certificate of Origin 1.1](https://elinux.org/Developer_Certificate_Of_Origin) along with your pull request.
 
-We have not yet posted a release of this project, but you can install by
-building a `pip` package or by directly importing the contents of the
-`text_extensions_for_pandas` source tree.
 
-To build a pip package from your local copy:
-1. (optional) Activate the `pd` environment that `env.sh` creates
-1. `python3 setup.py sdist bdist_wheel`
-1. The package's `.whl` file will appear under the `dist` directory.
+## Building and Running Tests
 
-To build and install a pip package from the head of the master branch:
+Before building the code in this repository, we recommend that you use the 
+provided script `env.sh` to set up a consistent build environment:
 ```
-pip install git+https://github.com/CODAIT/text-extensions-for-pandas
+$ ./env.sh myenv
+$ conda activate myenv
 ```
+(replace `myenv` with your choice of environment name).
 
-To directly import the contents of the `text_extensions_for_pandas` source tree 
-as a Python package:
-1. Add the root directory of your local copy of this repository to the
-   front of 
-```python
-import text_extensions_for_pandas as tp
+To run tests, navigate to the root of your local copy and run:
+```
+pytest
 ```
 
-## Contributing
+To build pip and source code packages:
 
-This project is an IBM open source project. We are developing the code in the open under the [Apache License](https://github.com/CODAIT/text-extensions-for-pandas/blob/master/LICENSE), and we welcome contributions from both inside and outside IBM. 
+```
+python setup.py sdist bdist_wheel
+```
+
+(outputs go into `./dist`).
+
+To build API documentation, run:
+
+```
+./generate_docs.sh
+```
 
-To contribute, just open a Github issue or submit a pull request. Be sure to include a copy of the [Developer's Certificate of Origin 1.1](https://elinux.org/Developer_Certificate_Of_Origin) along with your pull request.
 
-## Running Tests
 
-To run regression tests:
-1. (optional) Use the script `env.sh` to set up an Anaconda environment
-1. Run `python -m unittest discover` from the root of your local copy
 

README.md

@@ -43,20 +43,33 @@
 # What Sphinx extensions to activate. If something is not on this list, it
 # won't run.
 extensions = [
-    "sphinxcontrib.apidoc",
-    "sphinx.ext.autodoc",  # Needed for the sphinxcontrib.apidoc extension 
-#    "sphinx.ext.coverage", 
-#    "sphinx.ext.napoleon",
-#    "sphinx.ext.autosummary", 
-#    "sphinx.ext.intersphinx",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.coverage",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.intersphinx",
+
+    # Uncomment the following line to enable full automatic generation of
+    # API documentation files from code (currently we hard-code an 
+    # entry point for each module and rely on autodoc)
+    # "sphinxcontrib.apidoc"
 ]
 
-# Configure the sphinxcontrib.apidoc extension
+# Configure the sphinx.ext.autodoc extension
+autodoc_default_options = {
+    # TODO: Re-enable this once readthedocs.org upgrades to a version of
+    #  Sphinx where True is an acceptable value for the "members" option.
+    #  Then remove the redundant :members: and :undoc-members: annotations
+    #  from index.rst.
+    #"members": True,
+    #"undoc-members": True,
+}
+
+# Configure the sphinxcontrib.apidoc extension (currently not used)
 apidoc_module_dir = "../text_extensions_for_pandas"
 apidoc_output_dir = "."
-apidoc_excluded_paths = ["*test_*"]
-apidoc_separate_modules = False
-
+apidoc_excluded_paths = ["test_*.py"]
+apidoc_separate_modules = True
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']