From a539f453001eaff58db80e2763e8ef7b210ffa6e Mon Sep 17 00:00:00 2001 From: "Bjarni R. Einarsson" Date: Sun, 1 Jan 2017 16:52:44 +0100 Subject: [PATCH] Updates to the dev faq and contributor guidelines --- CONTRIBUTING.md | 34 ++++---- DEV_FAQ.md | 216 +++++++++++++++++++++++++++++++++++++----------- 2 files changed, 184 insertions(+), 66 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d938903da..ae58918f7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,26 +2,22 @@ First of all: Thank you! :heart: -#### Code of Conduct -Second of all, please adhere to our Code of Conduct when you participate -in our community. Be kind, be respectful. - -[The full Code of Conduct can be found here](CODE_OF_CONDUCT.md). +#### Further Reading +Second of all, please adhere to our Code of Conduct when you participate +in our community. Be kind, be respectful. [The full Code of Conduct can +be found here](CODE_OF_CONDUCT.md). -#### License - -Thirdly, please be sure you are comfortable with [our license](COPYING.md) +Please also be sure you are comfortable with [our license](COPYING.md) before contributing any code to Mailpile. - -#### Developer FAQ - -Does Mailpile seem a bit strange? Are you unsure of whether something is a -bug or a feature? [Our developer FAQ might have answers](DEV_FAQ.md). - -Please read it before filing issues or starting major development work. +Note that Mailpile is a slightly unsual app; the design is both political +and opinionated. If you are confused or unsure whether something is a bug or a +feature, [our developer FAQ might have answers](DEV_FAQ.md)! Please read it +before filing issues about design choices or attempting to reorganize the code +in any major way. The FAQ also contains a quick introduction Mailpile internals +and debugging techniques. #### Did you find a bug? @@ -48,6 +44,8 @@ Please read it before filing issues or starting major development work. be useful to file a report in case others are having the same issue, but bugs that can be reproduced will in general get fixed much faster! +* The [Developer FAQ](DEV_FAQ.md) has a section on debugging techniques. + #### **Did you write a patch that fixes a bug?** @@ -63,7 +61,7 @@ Please read it before filing issues or starting major development work. your feature is compatible with our high-level goals and design decisions. * If you are unsure or would like some guidance, join the #mailpile - channel on IRC and discuss your plans there. + channel on IRC (Freenode) and discuss your plans there. * Open a new GitHub pull request with the patch. @@ -94,8 +92,8 @@ Please feel free to join our translation community * Please feel free to contribute documentation to [our wiki](https://github.com/mailpile/Mailpile/wiki). -* If you would like some guidance, join the #mailpile channel on IRC or - file an issue with your questions. +* If you would like some guidance, join the #mailpile channel on IRC + (Freenode) or file an issue with your questions. * If you added a new page or made major major changes to an existing page, please file [a new issue requesting a review](https://github.com/mailpile/Mailpile/issues/new) diff --git a/DEV_FAQ.md b/DEV_FAQ.md index 4a8a64af6..8af9d78f1 100644 --- a/DEV_FAQ.md +++ b/DEV_FAQ.md @@ -2,38 +2,41 @@ This document contain a collection of frequently asked questions (with answers) about Mailpile development. Please familiarize yourself with -the contents before attempting to hack on Mailpile. +the contents before attempting any deep hacking on Mailpile. You don't have to agree with all of our priorities to take part or make use of Mailpile, but we do feel it helps if most of the community is rowing in roughly the same general direction! +Note: If you are just looking for debugging tips and tricks, you can skip +to the end. -#### Why Mailpile? + +### Why Mailpile? The long-term goal of Mailpile is to help *non-technical people* become -**more independent** and **more private** online, online, in particular -when it comes to e-mail. +*more independent* and *more private* online, in particular when it +comes to e-mail. -By **more independent**, we mean users should be in control of their +By **more independent**, we mean people should be in control of their e-mail and the software used to manage it. This is why Mailpile is Free Software, and this is why we *don't* promote Mailpile as a tool for building "cloud services". -By **more private**, we mean users should have more control over who has +By **more private**, we mean people should have more control over who has access to their e-mail, when and how. This is why Mailpile tries to make e-mail encryption more accessible, and this is why Mailpile tries to -make it convenient *and secure* for users to store their e-mail on their +make it convenient *and secure* for people to store their e-mail on their own devices. -Our focus on *non-technical users* implies, amongst other things, that +Our focus on **non-technical people** implies, amongst other things, that we cannot exclude people who are using non-free platforms such as Microsoft Windows or Mac OS X, and we cannot require our users learn a radically new, unfamiliar user interface or understand highly technical concepts such as public key cryptography. -#### Are All of These Decisions Awesome? +### Are All of These Decisions Awesome? No. Some of these decisions were almost certainly mistakes. But that's part of innovating, you try new things and see how they go! @@ -43,7 +46,7 @@ from picking arguments with us over these decisions. Making forward progress is much more fun than rehashing the past over and over again. -#### Why Do You Reinvent So Many Wheels? +### Why Do You Reinvent So Many Wheels? Usually, the answer is one or more of the following: @@ -61,15 +64,15 @@ cross-platform packaging efforts. For this reason we also tend to prefer pure-Python (or Javascript, for the UI) libraries over native code, which also avoids certain classes of -security holes which are simply not present in a memory-safe, managed -language. +bugs and security vulnerabilities which are simply not present in a +memory-safe, managed language. We have broken this rule more often than we'd like in our user-interface code, due to cultural differences between web developers and Python folks. *We would like to gradually reduce our front-end dependencies.* -#### Why a Search Engine? +### Why a Search Engine? Mailpile started as an experimental search engine, a hobby project. Everything else came later. @@ -89,34 +92,34 @@ The entire app is built around the search metaphor, much like Google's GMail. This is a fundamentally different way to build an e-mail client, from the traditional "messages and mailboxes" model. -The search engine also makes it easy for Mailpile to do some very -interesting things internally; some examples: +The search engine also makes it easy for Mailpile to do some pretty cool +things. For example: - * we can evaluate the trustworthiness of a message by asking the + * Mailpile can evaluate the trustworthiness of a message by asking the search engine about the past behaviour of the sender - * we can postpone processing of things like attached PGP keys until - we actually intend to use the key: the search engine makes the keys - easy to find later on + * Mailpile can postpone processing of things like attached PGP keys + until it actually intends to use the key: the search engine makes the + keys easy to find later on -#### Why doesn't Mailpile have a Native User Interface? +### Why doesn't Mailpile have a Native User Interface? For now, because the team is very small and we would like to reach users on many different platforms, we have bet entirely on the web as our -primary user interface. +primary user interface: -This decision allows us to target any operating system for which Python -2.7 has been made available. + * This decision allows us to target any operating system for which + Python 2.7 has been made available. -It also allows us to get help from the enormous community of web -developers; this is a much larger talent pool than the pool of -developers that know how to write native apps. + * It also allows us to get help from the enormous community of web + developers; this is a much larger talent pool than the pool of + developers that know how to write native apps. -Last but not least, the web interface is key to our plan to allow users -to access their Mailpile remotely over the network. Remote access is -critical if we want to get people to store their e-mail on their own -devices, since most people read their mail on multiple devices (laptop, -tablet, mobile, work computer, etc). + * Last but not least, the web interface is key to our plan to allow + users to access their Mailpile remotely over the network. Remote + access is critical if we want to get people to store their e-mail + on their own devices, because most people read their mail on + multiple devices (laptop, tablet, mobile, work computer, etc). That said: we do want to have a minimal native user interface, on all the major desktop operating systems. [That is what the gui-o-matic @@ -125,7 +128,7 @@ spin-off project is about](https://github.com/mailpile/gui-o-matic). Native mobile apps on Android and/or iOS would also be nice to have! -#### What is Mailpile's Approach to Security? +### How Does Mailpile Handle Security? This is a huge topic! Please consult our [Security Roadmap](https://github.com/mailpile/Mailpile/wiki/Security-roadmap). @@ -142,14 +145,17 @@ Thwarting other adversaries (criminals, jealous partners, etc.) is also very much something we care about, but is probably less divisive. -#### Why GnuPG? Why not GPGME? Why not PGPy? +### Why GnuPG? Why not GPGME? Why not PGPy? -GnuPG is mature, stable and although the user interfaces may leave +GnuPG is mature and stable. Although the user interfaces may leave something to be desired, it has a rich ecosystem of powerful tools built -around it. If we didn't use GnuPG, we would have to reinvent a lot of -wheels that aren't central to Mailpile's mission. Our issue tracker -contains [further discussions on the topic of why GnuPG and not something -else, such as PGPy](https://github.com/mailpile/Mailpile/issues/1743). +around it and a wealth of documentation and support to be found online. +If we didn't use GnuPG, we would have to reinvent a lot of wheels that +aren't central to Mailpile's mission. + +Our issue tracker contains [further discussions on the topic of why +GnuPG and not something else, such as +PGPy](https://github.com/mailpile/Mailpile/issues/1743). [Use of GPGME is also being discussed](https://github.com/mailpile/Mailpile/issues/1742). Currently, @@ -160,10 +166,30 @@ opposed to Python, which is memory-safe and less likely to contain certain classes of security vulnerabilities). -#### Why Not Django? Or Flask? +### Why not Python 3? + +We depend on some libraries - spambayes in particular - which were not +available for Python 3 when we started this project. + +We don't think Python 2 is going away in the near future. + +[There is a Github issue discussing +this.](https://github.com/mailpile/Mailpile/issues/160) + + +### Why Not Django? Or Flask? Reasons! +Our way may be unsual, but it's kinda awesome once you get used to it +and it wasn't obvious to us how we could get this kind of behaviour from +one of the standard frameworks. + +Please read the next section for details. + + +### How Does Mailpile's Web UI Work? + The text-based command-line interface is an important part of Mailpile's user interface. Our home-brewed framework allows us to generate web API end-points, text commands and command-line arguments *at the same time*. @@ -192,17 +218,111 @@ You can try this yourself, simply by editing the URL in your browser: The filename part of the URL is used to select output formats. All endpoints support `as.json`, most support HTML and/or text. -Our way may be unsual, but it's kinda awesome once you get used to it -and it wasn't obvious to us how we could get this kind of behaviour from -one of the standard frameworks. +The HTML output of each command is generated using Jinja2 templates +that are found in `shared-data/default-theme/html/...`. The directory +structure generally matches the URL paths seen in the browser, with +the main template for each command named `index.html`. +Alternate templates for the same API endpoint can have other names, for +example a template named `.../html/search/social.html` would be +accessible using URLs like so: -### Why not Python 3? + http://localhost:33411/in/inbox/social.html + http://localhost:33411/search/social.html?q=from:person@foo.com -We depend on some libraries - spambayes in particular - which were not -available for Python 3 when we started this project. -We don't think Python 2 is going away in the near future. +### Can I Develop Plugins For Mailpile? -[There is a Github issue discussing -this.](https://github.com/mailpile/Mailpile/issues/160) +Sort-of! + +Internally, the app is quite modular and there are methods which allow +code to register classes or functions that perform various functions. + +However, the plugin API is not considered stable, it is incomplete and +it is not very well documented. It may also not be a very nice API, and +we rather expect it to develop and change rapidly post-1.0. + +If you are interested in Mailpile's plugin APIs, take a look in +`shared-data/contrib/` for some examples of "external plugins" and +`mailpile/plugins/` for "internal plugins". + + +### How Do I Debug Mailpile? + +Developers should learn to use the Mailpile CLI. The `mailpile>` prompt is +where all of the low-level magic happens. Future versions of Mailpile will +expose this functionality to the web interface itself, but for now you will +need to use your shell. + +Possibly the most important command for Mailpile hackers, is to know +how to enable debugging. An example: + + # Enable verbose debugging of HTTP requests and GnuPG integration + # Note: HTTP debugging disables all sorts of internal caches! + mailpile> set sys.debug = log http gnupg + ... + +Many other subsystems can have debugging enabled. At the time of writing, +the `sys.debug` can include the following terms to make various parts of +the app more verbose: + + log http compose cryptostate autotag rescan keywords cache connbroker + vcard pop3 gnupg imap sources jinja timing sendmail httpdata + +There are also a few other ways to examine the app state: + + # Watch logging and debug messages fly by + mailpile> eventlog/watch + ... + [CTRL+C] + ... + + # Examine event log (piped through less) + mailpile> pipe less eventlog + ... + mailpile> pipe less eventlog incomplete + ... + + # Get an overview of what threads are running and what they are doing + mailpile> ps + ... + +Low-level changes and exploration of the configuration are also best done +from the CLI: + + # Explore the configuration; see also mailpile/config/defaults.py + mailpile> print -short sys + ... + mailpile> print -flat sources + ... + mailpile> print -secrets secrets + ... + + # Change things (dangerous) + mailpile> set sys.gpg_binary = /bin/false + ... + + # Reset something to its default setting + mailpile> unset sys.gpg_binary + ... + +There is also a help command, and you can use tab completion to try and +"guess" what commands exist. + + mailpile> help + ... + mailpile> help tags + ... + +Finally, the app ships with a `hacks` plugin which is disabled by default. +If you load it, that will add a few more low-level commands, including an +embedded Python interpretor: + + mailpile> plugins/load hacks + ... + + mailpile> hacks/pycli + ... + +There's sure to be more; please feel free to file a pull request against +this document to add your favourite tricks or clarify these.