Move atoms implementation from stepA to step6

[dubek]

(following discussion in #126)

Moving atoms from stepA to step6. The @ reader macro is left as optional in step6. Since atoms are implemented only in types+core+reader, hopefully there's no need to change all the existing implementations. I tested C, D, ruby and python on my machine and they pass the tests.

This is not ready for merge, but I'm ready for comments.

Still missing:

• Describe the 5 atom functions (left as TODOs in guide.md)
• Some general explanation about the immutability of all Mal data types (and then, by contrast, the mutable data type atom)
• Short list of possible usages of atoms (?)
• Fix the diagrams (that will be last after we discuss the whole thing and decide it's ready)

[dubek]

The Travis build (https://travis-ci.org/kanaka/mal/builds/96677124) is still running, but there are already few implementations that fail on the new step6: cpp, forth, factor, guile, rpython. I'll need to look deeper.

[kanaka]

rpython is just a pre-existing compile failure that happens sporadically (I'm hoping a newer version of the rpython compiler will fix it). I'm not sure about factor and guile yet, but cpp and forth are because swap! is defined as a mal function in stepA rather than natively.

Apart from the failures, I think the only thing I would like to see is to clarify that with atoms, they are a reference type and so it's not the value in an atom that changes but rather that the atom switches from referring from one immutable piece of data to referring to another immutable piece of data (identity). It might also be worth referring to http://clojure.org/state which gives an overview of Clojure's approach to state which is basically the same approach in mal. Also, Rich Hickey's "Value of Values" talk is great although might be a bit much to include in the guide. I've been thinking about an adjunct document that has a been of links for each step that point to other resources where people could go for deeper learning about the concepts from that mal step. Just a thought.

[dubek]

I have a solution for guile (need to port a bit of function wrapping done in step8 earlier to step6 to support the swap! implementation).

The problem with forth and cpp is that swap! is defined (in mal) in terms of apply, which itself is introduced only in step9. So solving it will probably require implementing swap! natively (copy stuff from apply, or calling an underlying helper function).

[kanaka]

Implementing swap! in terms of apply and reset! is perfectly valid, but I think we should re-implement those as native for two reasons: consistency between all the implementations, and I find that implementing swap! natively is really useful in terms of learning because it manipulates and recombines parameters in a way that other functions don't. We could also move apply to step6 too, but it fits well with map (higher order functions) so moving apply and map (even just apply) to step6 feels like maybe a bit too much in step6. Anyways, I could probably be convinced otherwise, but for now I'm leaning towards just making cpp and forth have native swap! functions.

On a related note, for being a Scheme (which is known for being a simple well planned variant of Lisp), I keep being surprised by the number of messy little workarounds that are necessary in the guile implementation. Oh well. :-(

[dubek]

Regarding the atoms discussion, the problem in step6 is that we haven't yet defined any data structure modification functions (cons, conj, assoc), so it's a bit hard to explain the point (again, for people not already familiar with the notion of immutable data structures).

Maybe we can use str together with atom to show how we can create a reference to a string that can "change" (the atom can point to a new string which is a modification of a copy of the old one).

[kanaka]

Hmm, that's a really good point. The problem with strings is that in many (maybe most) languages, strings are already immutable so it might lose some of the impact.

I'm fairly under the weather today so deep thoughts and analysis is probably not going to happen from my end. Do you think this is a serious enough issue to reconsider the direction? We could always have the discussion of immutability in a later step and just allude to it during the implementation of atoms in step6, but that does feel a little bit unsatisfying. Hmmm...

[kanaka]

Maybe spreading the discussion of immutability over several steps wouldn't be such a bad thing. An intro when atoms are introduced, then go a bit more in depth in step8 or step9 as we introduce introduce functions that would be expected to mutate in other languages. Maybe ... still mulling it over.

[dubek]

(get well soon!)

Re native swap!: A native implementation of swap! might further enhance the fact that it should be an atomic operation. Of course Mal doesn't have threads, so the cpp/forth solution works, but it is not valid in the general sense to implement swap! simply as deref->apply->reset!, because ideally it requires optimistic locking or some other sync mechanism.

Re atoms in step6 - I think the direction is OK. Atoms are useful even for an integer counter (the *gensym-counter* that started this whole discussion). The tests in stepA test an atom which refers to a hash-map (imitating a mal Env object), so maybe I'll add a short paragraph there about how it all integrates.

---

Aside: I'm not sure what are your goals in this project (besides being fun, and the challenge like the make implementation). You're not trying to make Mal the most feature-rich language, but it's also not the smallest possible language. For example - why should gensym be part of the guide and the standard Mal implementation? Is it that important or can it be left as an example function?

[sdt]

I have a native C++ swap! function ready to go here.

sdt@19a6c20

I'll make a PR for that later today if you like.

[dubek]

Thanks @sdt that looks good.

@kanaka I'm not sure we have a test-case for swap! with a list as the last argument - I believe it should behave like apply:

Mal [clojure]
user> (apply str "abc" "def" ["ghi" "jkl"])
"abcdefghijkl"
user> (def! a (atom "abc"))
(atom "abc")
user> (swap! a str "def" ["ghi" "jkl"])
"abcdef[\"ghi\" \"jkl\"]"

Here from Ruby (notice the different string serialization - probably another issue):

Mal [ruby]
user> (apply str "abc" "def" ["ghi" "jkl"])
"abcdefghijkl"
user> (def! a (atom "abc"))
(atom "abc")
user> (swap! a str "def" ["ghi" "jkl"])
"abcdef[ghi jkl]"

[kanaka]

@sdt a PR would be great, thanks.

@dubek good point about swap! needing to be atomic.

Regarding the motivation for the project, you're right that it started out as a fun challenge to implement Lisp in some interesting languages, but it evolved into a learning tool and that's really what I consider the primary goal (I go over a bit of the history and current motivation in the presentation I did for midwest.io last month: https://www.youtube.com/watch?v=lgyOAiRtZGw). And that goal breaks down into two subgoals: learning about Lisp, learning an arbitrary other language (with the second one probably being the primary).

Mal as a learning tool manifests in several ways:

• Low barrier to entry with continual progress/feedback along the way.
• Small incremental chunks that build on previous learning. As part of this I try and keep each step to reasonable and similar-ish size. This get complicated by the fact that things that are easy in one language are difficult in another, but I try having done a bunch of languages, I think the steps are in a similar enough ball-park. Step5 is probably the exception. Since step5 tests are only run for about half the implementations, I've been thinking about a different way to mark those tests off for the implementation that do, and then moving a few items from step4 and maybe moving some from later steps (cons, first, rest, etc) into step5.
• An final accomplishment/goal for a learner to aim for that is large and satisfying but not insurmountable (self-hosting).
• Give the dev options for what to attack first. I.e. identify clearly things that can be deferred until later if desired.
• Consistency among implementations so that a dev can compare a language they know and one they don't.
• Teach modern Lisp concepts (part of the reason for modeling it on Clojure). Traditional Lisp things that are really more historical artifact of the first implementation aren't a priority (such as car, cdr, dotted list, etc).
• I tend to lean towards native implementation for concepts rather than Lisp implementations, because I weigh learning the target language slightly higher than learning Lisp (note the target may be Lisp itself) although it's kind of case-by-case depending on which I think provides a better learning opportunity (and my own whim on a given day of course). However, I think it's also great to show examples of how the same thing can be implemented natively in Lisp itself. I think the two steps of implementing something natively and later discovering that there is a Lisp way to do something can be doubly enlightening. Self-hosted mal is itself an example of that.
• Conciseness of implementation is also important but that also derives from wanting mal to be a learning tool and not just a code-golf game. Overly verbose code often means more to keep in your head, but really concise implementations can be equally hard to grasp because they are so dense. The most idiomatic implementations are often among the shortest but not always. And I freely admit that in some of the implementations I probably went for shorter implementation rather than most idiomatic. I'll happily take PRs that make an implementation more idiomatic.
• Correctness of implementation is also important but also derives from the learning tool goal. I want people to learn the right way (TM) to implement something. So in the gensym case, gensym is pretty important for being able to write macros that are correct, efficient and don't do accidental identifier capture.
• Completeness or production readiness is a non-goal. Part 2 of the guide (if such a thing existed) would include the things needed to move a mal implementation towards production such as making a compiler (as opposed to an interpreter), more complete error checking (e.g. an if form with more than three arguments should throw an error), full interop (which admittedly is highly target language dependent), concurrency, standard libraries, etc. There are a lot additions that I would definitely consider if the implementation is fairly clean and small (e.g. I think namespaces could be implemented without much fuss), but I do want to keep the footprint fairly small and easy to learn and reason about. Also, I've thought about adding a stepB where functionality not needed for self-hosting, but still interesting could live (e.g. conj and meta-data on other collection types would move from stepA to stepB)

And my secret "world domination" goal is to see an implementation of mal in every programming language so it becomes THE rosetta stone for language comparison and learning. Okay, every, will probably never happen unless mal becomes the standard/convention example code/program that new language developers also implement along with their language. But hey, a person can dream .... :-)

Some of the above should probably be made into a wiki page. But I'm too tired and fuzzy to do that right now.

[kanaka]

@dubek yeah, we should add a few more swap! tests to catch more cases (I'm sure some impls will break). Can you file a ticket on the string issue? I think there are a couple of other cases where there might be some lingering string issues too although I'm having trouble recalling them right now.

[sdt]

Should the final argument to swap! work the same as for apply ?

Also, in both cases, does the final argument need to be a list/vector?

The guide has:

(apply F A B [C D]) is equivalent to (F A B C D)

Should (apply F A B) be desugared to (apply F A B [ ]), or is that considered an error?

[dubek]

@kanaka Thanks for the thorough explanations. You should definitely extract this to some manifesto.

I'll open separate issues for the edge cases of str , apply , swap! - try to imitate Clojure behaviour for all of them - but not exactly - try (str (list)) in Clojure :-(

I'll enhance the atoms description and add references to Clojure's state handling rationale.

Then there's work on factor's and forth's swap! impl.

(I'm travelling in the next 2 days so will probably be mostly offline.)

[dubek]

@sdt - My intuition about swap! was wrong. Unlike apply, swap! doesn't force the last argument to be a sequence and doesn't "splice" it. Here's pure Clojure behaviour:

user=> (swap! (atom [1 2 3]) conj 4 5 [6 7] [8 9])
[1 2 3 4 5 [6 7] [8 9]]
user=> (apply conj [1 2 3] 4 5 [6 7] [8 9])
[1 2 3 4 5 [6 7] 8 9]

So I believe the current implementations are correct (I'll probably add some more tests as part of the current PR just to be sure).

[dubek]

I expanded the atoms description in step 6.

Still not fixed: forth, factor

Also the diagrams should be modified...

[kanaka]

The guide updates look good. I'll be traveling for the next couple of days so I won't have a chance to update the diagrams, but I may be able to get to it later this week.

[kanaka]

@dubek I made updated diagrams: a46e6d5

I'll merge them together with this PR (after forth and factor). Note I'll be AFK for the next week so it won't be this next week.

[dubek]

Native swap! in forth in 10fad47 .

[dubek]

Fix factor in 9eccf95 . I hope travis will be green on this one...

[dubek]

Rebased branch on top of current master to take the latest matlab/octave changes and assess our situation there.

[dubek]

Now the only failing impl is matlab in step6.

[kanaka]

Okay, I'll take a look at matlab. Stilll on vacation with limited Internet connectivity so it might be a couple of days before I'm able to merge and push everything.

[kanaka]

Merged with matlab and diagram fixes. Tests currently running: https://travis-ci.org/kanaka/mal/builds/101646135

[kanaka]

RPython failed for the normal reasons, but everything else passed so looks good.