Skip to content

Commit

Permalink
Updates to the dev faq and contributor guidelines
Browse files Browse the repository at this point in the history
  • Loading branch information
@BjarniRunar
BjarniRunar committed Jan 2, 2017
1 parent 67efcdc commit a539f45
Show file tree
Hide file tree
Showing 2 changed files with 184 additions and 66 deletions.
34 changes: 16 additions & 18 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,22 @@

First of all: Thank you! :heart:

#### Code of Conduct

Second of all, please adhere to our Code of Conduct when you participate
in our community. Be kind, be respectful.

[The full Code of Conduct can be found here](CODE_OF_CONDUCT.md).
#### Further Reading

Second of all, please adhere to our Code of Conduct when you participate
in our community. Be kind, be respectful. [The full Code of Conduct can
be found here](CODE_OF_CONDUCT.md).

#### License

Thirdly, please be sure you are comfortable with [our license](COPYING.md)
Please also be sure you are comfortable with [our license](COPYING.md)
before contributing any code to Mailpile.


#### Developer FAQ

Does Mailpile seem a bit strange? Are you unsure of whether something is a
bug or a feature? [Our developer FAQ might have answers](DEV_FAQ.md).

Please read it before filing issues or starting major development work.
Note that Mailpile is a slightly unsual app; the design is both political
and opinionated. If you are confused or unsure whether something is a bug or a
feature, [our developer FAQ might have answers](DEV_FAQ.md)! Please read it
before filing issues about design choices or attempting to reorganize the code
in any major way. The FAQ also contains a quick introduction Mailpile internals
and debugging techniques.


#### Did you find a bug?
Expand All @@ -48,6 +44,8 @@ Please read it before filing issues or starting major development work.
be useful to file a report in case others are having the same issue, but
bugs that can be reproduced will in general get fixed much faster!

* The [Developer FAQ](DEV_FAQ.md) has a section on debugging techniques.


#### **Did you write a patch that fixes a bug?**

Expand All @@ -63,7 +61,7 @@ Please read it before filing issues or starting major development work.
your feature is compatible with our high-level goals and design decisions.

* If you are unsure or would like some guidance, join the #mailpile
channel on IRC and discuss your plans there.
channel on IRC (Freenode) and discuss your plans there.

* Open a new GitHub pull request with the patch.

Expand Down Expand Up @@ -94,8 +92,8 @@ Please feel free to join our translation community
* Please feel free to contribute documentation to
[our wiki](https://github.com/mailpile/Mailpile/wiki).

* If you would like some guidance, join the #mailpile channel on IRC or
file an issue with your questions.
* If you would like some guidance, join the #mailpile channel on IRC
(Freenode) or file an issue with your questions.

* If you added a new page or made major major changes to an existing page,
please file [a new issue requesting a review](https://github.com/mailpile/Mailpile/issues/new)
Expand Down
216 changes: 168 additions & 48 deletions DEV_FAQ.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,41 @@

This document contain a collection of frequently asked questions (with
answers) about Mailpile development. Please familiarize yourself with
the contents before attempting to hack on Mailpile.
the contents before attempting any deep hacking on Mailpile.

You don't have to agree with all of our priorities to take part or make
use of Mailpile, but we do feel it helps if most of the community is
rowing in roughly the same general direction!

Note: If you are just looking for debugging tips and tricks, you can skip
to the end.

#### Why Mailpile?

### Why Mailpile?

The long-term goal of Mailpile is to help *non-technical people* become
**more independent** and **more private** online, online, in particular
when it comes to e-mail.
*more independent* and *more private* online, in particular when it
comes to e-mail.

By **more independent**, we mean users should be in control of their
By **more independent**, we mean people should be in control of their
e-mail and the software used to manage it. This is why Mailpile is Free
Software, and this is why we *don't* promote Mailpile as a tool for
building "cloud services".

By **more private**, we mean users should have more control over who has
By **more private**, we mean people should have more control over who has
access to their e-mail, when and how. This is why Mailpile tries to make
e-mail encryption more accessible, and this is why Mailpile tries to
make it convenient *and secure* for users to store their e-mail on their
make it convenient *and secure* for people to store their e-mail on their
own devices.

Our focus on *non-technical users* implies, amongst other things, that
Our focus on **non-technical people** implies, amongst other things, that
we cannot exclude people who are using non-free platforms such as
Microsoft Windows or Mac OS X, and we cannot require our users learn a
radically new, unfamiliar user interface or understand highly technical
concepts such as public key cryptography.


#### Are All of These Decisions Awesome?
### Are All of These Decisions Awesome?

No. Some of these decisions were almost certainly mistakes. But that's
part of innovating, you try new things and see how they go!
Expand All @@ -43,7 +46,7 @@ from picking arguments with us over these decisions. Making forward
progress is much more fun than rehashing the past over and over again.


#### Why Do You Reinvent So Many Wheels?
### Why Do You Reinvent So Many Wheels?

Usually, the answer is one or more of the following:

Expand All @@ -61,15 +64,15 @@ cross-platform packaging efforts.

For this reason we also tend to prefer pure-Python (or Javascript, for
the UI) libraries over native code, which also avoids certain classes of
security holes which are simply not present in a memory-safe, managed
language.
bugs and security vulnerabilities which are simply not present in a
memory-safe, managed language.

We have broken this rule more often than we'd like in our user-interface
code, due to cultural differences between web developers and Python
folks. *We would like to gradually reduce our front-end dependencies.*


#### Why a Search Engine?
### Why a Search Engine?

Mailpile started as an experimental search engine, a hobby project.
Everything else came later.
Expand All @@ -89,34 +92,34 @@ The entire app is built around the search metaphor, much like Google's
GMail. This is a fundamentally different way to build an e-mail client,
from the traditional "messages and mailboxes" model.

The search engine also makes it easy for Mailpile to do some very
interesting things internally; some examples:
The search engine also makes it easy for Mailpile to do some pretty cool
things. For example:

* we can evaluate the trustworthiness of a message by asking the
* Mailpile can evaluate the trustworthiness of a message by asking the
search engine about the past behaviour of the sender
* we can postpone processing of things like attached PGP keys until
we actually intend to use the key: the search engine makes the keys
easy to find later on
* Mailpile can postpone processing of things like attached PGP keys
until it actually intends to use the key: the search engine makes the
keys easy to find later on


#### Why doesn't Mailpile have a Native User Interface?
### Why doesn't Mailpile have a Native User Interface?

For now, because the team is very small and we would like to reach users
on many different platforms, we have bet entirely on the web as our
primary user interface.
primary user interface:

This decision allows us to target any operating system for which Python
2.7 has been made available.
* This decision allows us to target any operating system for which
Python 2.7 has been made available.

It also allows us to get help from the enormous community of web
developers; this is a much larger talent pool than the pool of
developers that know how to write native apps.
* It also allows us to get help from the enormous community of web
developers; this is a much larger talent pool than the pool of
developers that know how to write native apps.

Last but not least, the web interface is key to our plan to allow users
to access their Mailpile remotely over the network. Remote access is
critical if we want to get people to store their e-mail on their own
devices, since most people read their mail on multiple devices (laptop,
tablet, mobile, work computer, etc).
* Last but not least, the web interface is key to our plan to allow
users to access their Mailpile remotely over the network. Remote
access is critical if we want to get people to store their e-mail
on their own devices, because most people read their mail on
multiple devices (laptop, tablet, mobile, work computer, etc).

That said: we do want to have a minimal native user interface, on all
the major desktop operating systems. [That is what the gui-o-matic
Expand All @@ -125,7 +128,7 @@ spin-off project is about](https://github.com/mailpile/gui-o-matic).
Native mobile apps on Android and/or iOS would also be nice to have!


#### What is Mailpile's Approach to Security?
### How Does Mailpile Handle Security?

This is a huge topic! Please consult our [Security
Roadmap](https://github.com/mailpile/Mailpile/wiki/Security-roadmap).
Expand All @@ -142,14 +145,17 @@ Thwarting other adversaries (criminals, jealous partners, etc.) is also
very much something we care about, but is probably less divisive.


#### Why GnuPG? Why not GPGME? Why not PGPy?
### Why GnuPG? Why not GPGME? Why not PGPy?

GnuPG is mature, stable and although the user interfaces may leave
GnuPG is mature and stable. Although the user interfaces may leave
something to be desired, it has a rich ecosystem of powerful tools built
around it. If we didn't use GnuPG, we would have to reinvent a lot of
wheels that aren't central to Mailpile's mission. Our issue tracker
contains [further discussions on the topic of why GnuPG and not something
else, such as PGPy](https://github.com/mailpile/Mailpile/issues/1743).
around it and a wealth of documentation and support to be found online.
If we didn't use GnuPG, we would have to reinvent a lot of wheels that
aren't central to Mailpile's mission.

Our issue tracker contains [further discussions on the topic of why
GnuPG and not something else, such as
PGPy](https://github.com/mailpile/Mailpile/issues/1743).

[Use of GPGME is also being
discussed](https://github.com/mailpile/Mailpile/issues/1742). Currently,
Expand All @@ -160,10 +166,30 @@ opposed to Python, which is memory-safe and less likely to contain
certain classes of security vulnerabilities).


#### Why Not Django? Or Flask?
### Why not Python 3?

We depend on some libraries - spambayes in particular - which were not
available for Python 3 when we started this project.

We don't think Python 2 is going away in the near future.

[There is a Github issue discussing
this.](https://github.com/mailpile/Mailpile/issues/160)


### Why Not Django? Or Flask?

Reasons!

Our way may be unsual, but it's kinda awesome once you get used to it
and it wasn't obvious to us how we could get this kind of behaviour from
one of the standard frameworks.

Please read the next section for details.


### How Does Mailpile's Web UI Work?

The text-based command-line interface is an important part of Mailpile's
user interface. Our home-brewed framework allows us to generate web API
end-points, text commands and command-line arguments *at the same time*.
Expand Down Expand Up @@ -192,17 +218,111 @@ You can try this yourself, simply by editing the URL in your browser:
The filename part of the URL is used to select output formats. All
endpoints support `as.json`, most support HTML and/or text.

Our way may be unsual, but it's kinda awesome once you get used to it
and it wasn't obvious to us how we could get this kind of behaviour from
one of the standard frameworks.
The HTML output of each command is generated using Jinja2 templates
that are found in `shared-data/default-theme/html/...`. The directory
structure generally matches the URL paths seen in the browser, with
the main template for each command named `index.html`.

Alternate templates for the same API endpoint can have other names, for
example a template named `.../html/search/social.html` would be
accessible using URLs like so:

### Why not Python 3?
http://localhost:33411/in/inbox/social.html
http://localhost:33411/search/social.html?q=from:[email protected]

We depend on some libraries - spambayes in particular - which were not
available for Python 3 when we started this project.

We don't think Python 2 is going away in the near future.
### Can I Develop Plugins For Mailpile?

[There is a Github issue discussing
this.](https://github.com/mailpile/Mailpile/issues/160)
Sort-of!

Internally, the app is quite modular and there are methods which allow
code to register classes or functions that perform various functions.

However, the plugin API is not considered stable, it is incomplete and
it is not very well documented. It may also not be a very nice API, and
we rather expect it to develop and change rapidly post-1.0.

If you are interested in Mailpile's plugin APIs, take a look in
`shared-data/contrib/` for some examples of "external plugins" and
`mailpile/plugins/` for "internal plugins".


### How Do I Debug Mailpile?

Developers should learn to use the Mailpile CLI. The `mailpile>` prompt is
where all of the low-level magic happens. Future versions of Mailpile will
expose this functionality to the web interface itself, but for now you will
need to use your shell.

Possibly the most important command for Mailpile hackers, is to know
how to enable debugging. An example:

# Enable verbose debugging of HTTP requests and GnuPG integration
# Note: HTTP debugging disables all sorts of internal caches!
mailpile> set sys.debug = log http gnupg
...

Many other subsystems can have debugging enabled. At the time of writing,
the `sys.debug` can include the following terms to make various parts of
the app more verbose:

log http compose cryptostate autotag rescan keywords cache connbroker
vcard pop3 gnupg imap sources jinja timing sendmail httpdata

There are also a few other ways to examine the app state:

# Watch logging and debug messages fly by
mailpile> eventlog/watch
...
[CTRL+C]
...

# Examine event log (piped through less)
mailpile> pipe less eventlog
...
mailpile> pipe less eventlog incomplete
...

# Get an overview of what threads are running and what they are doing
mailpile> ps
...

Low-level changes and exploration of the configuration are also best done
from the CLI:

# Explore the configuration; see also mailpile/config/defaults.py
mailpile> print -short sys
...
mailpile> print -flat sources
...
mailpile> print -secrets secrets
...

# Change things (dangerous)
mailpile> set sys.gpg_binary = /bin/false
...

# Reset something to its default setting
mailpile> unset sys.gpg_binary
...

There is also a help command, and you can use tab completion to try and
"guess" what commands exist.

mailpile> help
...
mailpile> help tags
...

Finally, the app ships with a `hacks` plugin which is disabled by default.
If you load it, that will add a few more low-level commands, including an
embedded Python interpretor:

mailpile> plugins/load hacks
...

mailpile> hacks/pycli
...

There's sure to be more; please feel free to file a pull request against
this document to add your favourite tricks or clarify these.

0 comments on commit a539f45

Please sign in to comment.