Posts on Tuplet

Dev Diary 8

Tue, 23 Aug 2022 00:00:00 +0000

Curzon Raku, my beloved old friend

There's a larger post bubbling in the back of my head about this, but for now, lets say that digging into Raku feels a lot like rediscovering an old friend. At the same time, I'm realizing that I've changed.

Raku is… brilliant. I regret not having stayed in touch with it during it's rather prolonged development (21 years!!). You see, I loved Perl 5. I don't know if it was because it was my first "real" language (after Basic) and I imprinted on it, or if I was just exceptionally fortunate to start with a language that really supported my way of thinking. It's, coincidentally, the same reason I'm not a Python dev. Python's great, but it forces me to solve tasks in ways counter to how my brain wants to.

Anyway, I'm still just getting into it, but I'm loving what I'm discovering. At the same time, I'm learning that I'm not the same person I was. There was a tiny bit of fear that I'd be all "Never mind, screw tuplet. I'm just going to use Raku." But, I'm finding that I still want something different.

For example, I'm annoyed by the c-style curlies. They feel like unnecessary clutter. I dislike the inconsistency of sometimes needing parens around args and sometimes not. While it is definitely multi-paradigmatic I find myself thinking that it hasn't gone far enough towards supporting a lisp-like syntax. Also, I've spent so long in Ruby-land that the sigil-heavy syntax isn't really a feature so much as a readability issue.

So, tuplet, is still on track.

…

[Days later]

Raku is, brilliant, and… a lot. It's got a ton of documentation, but it's simultaneously too much and too little. It's not good for a new dev. For most languages I'm just like "ok how do you express this thing with your syntax". Same idea, different clothes. But Raku isn't like that. Raku's like…

"Well, you see… We don't do it like that 'round here. We express things differently. Oh, sure, you could do it in the old-n-busted way, but 'cmere." And then they lead you into this old broken down looking shed that's actualy a Tardis in disguise. "Lemme show you some of the tools I've built with this-here CNC machine. … Yup. I see you checkin' out the thingawhatzit. That puppy'll strip varnish of a flooblebobber to within 3 microns!"

Meanwhile you're wondering what a "flooblebobber" is, and why you'd want to strip varnish from one.

I am still optomistic about it, but it feels like one of those projects that isn't going to be easy, but'll definitely be worth it in the long run.

—- In case you don't get the reference: Curzon, my beloved old friend.

Dev Diary 5

Thu, 04 Aug 2022 00:00:00 +0000

Back to writing a Language target

Or… not.

It turns out that I dramatically overestimated the amount of documentation available for creating a new language target for Antlr4. Basically it boils down to "edit one of the existing targets" Except there are all sorts of template directives in there that aren't explained. I suspect the String Template documentation will probably address the syntax enough for me to understand the "what" of these templates but not the "why".

So, after staring at the Python language target file I tried just generating the syntax with it. Now I'm reverse engineering. I thought I'd start with the Ruby one since I'm so familiar with it, but in addition to not being supported it has a lot of files. Like… a lot of files:

I don't know if this is reasonable, or ridiculous. I'm inclined to go with "reasonable" because why would you write over 8,000 lines of code if you didn't have to?

So, now I'm wondering about maybe figuring out the grammar in Antlr using it's sweet plugin and visualization tools. Then, take the grammar, and reimplement it in Janet using its built in Parsing Expression Grammar (PEG). Every time I tweak the grammar, I do it in Antlr where I can visualize and tweak things until it works. Then re-implement the changes in Janet.

This is, of course, just a terrible idea.

However, my options are:

Spend a very long time reverse engineering how the eff to write an Antlr4 Language target
- frustrating, time consuming, not even remotely fun
- Probably will require writing thousands of lines of support code just to make it work, nevermind the code require to actually do the translation of the grammar into Janet.
Use one of the existing language targets (most likely Go)
- So my compilation pipeline would be Tuplet -> Go -> Janet -> C -> executable This, of course, feels even stupider than reimplementing the grammar in Janet.
Port the grammar to Janet

I honestly can't decide which is a worse idea. Passing through Go, or porting the Grammar to Janet and then porting every new language feature too.

The Go solution has the option of being something I could get up an running quickly, that would likely give me a fast running executable that could be replaced in some future date. It's ridiculous, but it may be the fastest way to get to a working language. Also, it would likely prove to be a faster iteration cycle for new language features. I don't have to reimplement a working, and somewhat tested, grammar while potentially introducing bugs in a much harder to test environment.

I could write unit tests for the Janet PEG I'd write, so they could be reliable, but once again I'm introducing more time into iteration loop of going from some new syntax idea, to compileable Tuplet code.

I guess that's the answer. Most languages are bootstrapped off of some other language, because it'd take forever and be more painful to skip that. I guess the same idea applies here.

Sacrifice idealized compilation tool chain for speed of development. As i'm not really passionate about the compilation tool chain, that's probably the right solution.

[Update]: I spent the night dreaming about writing a PEG in Janet. 🤦‍♀️

Dev Diary 7

Thu, 04 Aug 2022 00:00:00 +0000

Target Woes

Started writing a tiny grammar in Janet last night just to get a feel for it. Wrote one that matches my multi-line comments which I'll need for the initial pass of the parser.

Discovered some problems.

Janet's PEG doesn't support unicode out of the box. It's stuck in ASCII land. Gods forbid anyone wants to try and match a string that was written in another language. You know, something really esoteric like Spanish, or French. 🤦‍♀️Technically it does support unicode but there's no easy way to do basic things like defining something that matches letters / word characters unless you're willing to limit your input to ASCII.
It doesn't appear that grammars can use other grammars, which means each grammar must be self-contained. Not a big deal. Fairly standard actually, but it means you can't compose grammars.
There's no good way to see which rule matched a given portion of the text. You can "tag" captured text, but it's only a thing used for referencing past matches within the grammar.
Janet has no regular expressions. They explain this by saying that their PEG is better, and they've got some reasonable arguments, BUT we're back to unicode not being supported out of the box.

I use regular expressions a lot. I "cut my teeth" on Perl 5. I love them, and actually understand them at an expert level. The syntax is freaking horrible for newbs, but that's a separate discussion. Anyway…. no regexp is going to be a pain, and frankly I don't want to have to write unicode support into a regexp engine or a PEG.

This morning I decided to check in on the current state of Raku (a.k.a Perl 6), which has some amazing features, and I saw this piece of beauty.

ARG. I DON'T WANT TO BE SWITCHING LANGUAGES AGAIN.

I need this language to be done. Or, at least working. I need to get it out of my brain so that I can sleep again.

Dev Diary 6

Tue, 02 Aug 2022 00:00:00 +0000

Nanopass Compiler?

I realized that in order to really leverage my contract stuff I'm going to need a multi-pass compiler. For example, I want to have the compiler let you know that there's a problem when you feed the output of a function that returns an Integer to the input of a function that takes a String. You can't do that until you've ingested all the functions in your app, and processed their contract definitions. So, it's really an "ok, your syntax is fine, let's see if your usage is too" phase of compilation.

A multipass compiler is thus required. The Nanopass Compiler framework is the king of that hill. If you aren't familiar with it I recommend you watch this talk from 2013. It's about 40 minutes and moves fast but don't worry if some of the stuff goes over your head. It's the basic gist of how things work that's important, not the actual implementation details.

I'm having a really hard time finding a good guide for how to do much of anything with transpilers though. I figured that Racket has been a playground for language prototyping and design for years now, so surely there must be something. There really wasn't; at least, not that I could find. So, I asked on the Racket subreddit. I got some useful responses about Nanopass compilers, but nothing really about tutorials on transpiler creation.

Create Your Own Freaking Awesome Programming Language is still the best i've found but I don't believe it covers multipass compilation. It's not that I can't figure out how to do it from the resources available. The problem is that I don't know what the right way to approach the problem is. What's the right way to structure my passes? How do i decide how much to tackle in each pass or what order to do them in? The how isn't that hard. It's just learning syntax. It's the approach that's unclear.

I don't know how to think about compiler design, especially multipass ones.

So, last night was a rabbit hole of education. I watched the video linked above. I learned that nanopass compilation isn't just a general technique but a general technique coupled with a specific framework that has only been implemented in a couple languages (not Janet).

As an aside, I'm getting really sick of how much emacs hates inserting tabs, so i went on a little side-track as to what it'd take to define a new language in emacs and have tab indentation be the default there. The long and short of it is that indentation is complicated in emacs, and written by monsters who not only think that spaces are better because they "keep everything the same" when different people have different tab widths, but also think that it's fine to mix tabs and spaces for indentation so that it minimizes the number of characters used (why?!?!). If you care at all about accessibility for low vision users you will stop indenting with spaces.

Anyway. Lots of time. Little actual progress. I then spent far to much time awake in the middle of the night not-quite-dreaming about writing compiler code based on the little I'd learned.

Dev Diary 4

Tue, 26 Jul 2022 00:00:00 +0000

Lambda Lambda Lambda

I'd come up with the temporary syntax of ~>[args]{function body} for lambdas but it was very not in line with the rest of the language, and not at all self-evident, and I just didn't like it.

Then i saw how python handles it:

# lambda arguments : expression
x = lambda a : a + 10
print(x(5))

So what if we built off of that? What if lambda: is a special function (basically a macro)?

# lambda: []
#		do_something:

var: plus_ten lambda: [a]
	return: +: a 10

println plus_ten: 5
#=> outputs 15

under the covers it'd be converted to

(def x (fn [a]
           (+ a 10)))
(print  (x 5) )

Janet's fine with printing a non-string value so we don't have to wrap it in a (string/format "%s" (x 5)) or anything like that.

Here's what it'd look like using it in a map.

map: a_list lambda: [x]
				return: +: x 11

map: a_list <lambda: [x]
				return: +: x 11>

map: a_list plus_ten

The first 2 convert to

(map (fn [x]
         (+ x 11)) a_list)

and the last is

(map plus_ten a_list)

We learn 2 things from the above.

yup, looks legit. Not only that, looks like any other function call. YAAAY
it's legal to indent as many levels as you want in a single line. in this case going from 1 to 3. We already knew it was legal to dedent multiple levels. Now I'm explicitly declaring that as acceptable.

And yes, I've decided to make maps function as map: list function instead of the lisp standard of (map function list) becasue the "normal" order always felt weird to me.

And that colon after return: is still bugging me. Also, I'm just not a fan of the explicit return in a lambda. In that context it feels excessively verbose. To me a lambda is all "hey i just need a quickie function in here that isn't going to be referenced anywhere else." You rarely write a lambda for side-effects. They tend to be written explicitly for their return value.

Return

I'm explicitly requiring return to be called in a function for two reasons:

when functions have implicit returns it is frequently impossible to tell if the thing a function is returning is something you can rely on, or just the coincidental output of the last line of a function. That is to say, I don't know if i can trust that your function will always return a boolean as it is now, or if that was just an unintented side-effect of the last action you performed in you rfunction.
it's sometimes non-obvious what a function is or could be returning, and you have to spend a couple minutes working your way through its branches to figure it out.

So, that's why I'm calling it out in lambdas. However, as I write this, I'm realizing that contracts would solve this problem.

There could be a compile-time check to see if a function has a contract, and if so, doesn't require a return, but any function without a contract, would require one. I'm not sure I like the inconsistent result then. Some functions would have a return and some wouldn't.

… Just fixed return so that it no longer needs a colon after it. I needed to move the RETURN token down below the FUNCTION_NAME token that referenced it.

Line continuation

I'm running into instances where it's just not clear what function the following line is an argument to. For example:

        if: and: x true
            y
            z

It would be legitimate to assume that y is a 3rd parameter to and:. It's on a new line, and it's indented more than the prior line. At the same time, it's also reasonable to assume that it's a 2nd argument to if: which Is what I intended it as.

The current solution to this would be to say

        if: <and: x true>
            y
            z

And that's fine, but the code is waving a metaphorical red flag at me and letting me know that we're going to need the ability to say "this thing continues on the next line." Python has it, and those folks know what they're doing with indentation.

Yes, I really should brush up on my Python here rather than re-inventing their wheel badly, but until then, I've decided that my continuation string will be ... or an actual elipsis (…). Because an elipsis indicates that the sentence / line has literally trailed off without completing.

I'll figure out what the right rules around its usage are later. Correction, I'll shamelessly rip-off Python's rules for its usage later.

Python rules: "Line continuation is automatic when the split comes while a statement is inside parenthesis ( () ), brackets ( [] ) or braces ( {} )." - Rhino Developer

So, there's your answer. You need the line continuation indicator whenever the line is continuing but it's not in a grouping. That… doesn't actually work for me. I want this to be valid syntax:

        if: true
            do_this:
            do_that:

So, back to this

        if: and: x true
            y
            z

I think it's just too ambiguous. I can't describe a rule in english that clearly describes that such that y is unambiguously an argument to if: and not and:.

Also, I just realized that it's also not clear if x is an argument of if: or and:

So, that means we have to do this

        if: <and: x true>
            y
            z

Dev Diary 2

Sun, 24 Jul 2022 00:00:00 +0000

The Language So far

In the prior post I described how Tuplet came to be. This post is going to lay out where the language stands today.

What is actually written

As of today, it's just a grammar. Antlr can convert this into a lexer and parser for me, but before I do that I'll have to create a new language target for Antlr. Specifically, I need to create a target for Janet. I could use one of the existing language targets like Go, but then the tool-chain would be Tuplet->Go(executable)->Janet->C->executable which would require working in 3 languages (Tuplet, Go, and Janet) to expand the language, and that just seems ridiculous. Heck, Tuplet->Janet->C->executable seems ridiculous. At least the Janet->C portion of it is invisible.

Dev_diary_3

Sun, 24 Jul 2022 00:00:00 +0000

Building an Antlr4 Language Target

As mentioned in the last post I need to create an Antlr4 language target for Janet (the language Tuplet will compile to). There are 4 minor problems.

I've never created an Antlr language target
There are no existing Antlr language targets for a lisp to learn / copy from.
I haven't touched Java since like 2009 (13yrs).
I've never programmed anything more sophisticated than a "hello world" in Janet.

So that's "fun".

Step 1

I need to create a template for a Janet module. So, I need to learn how Janet handles modules.

Sidetrack 1: function names

This of course, leads me to think about functions, which … long story short, = just added the ability to have some more funky characters in function names:

fragment FUNCTION_ID_CONTINUE
 : ID_START
 | [/><=-] | DIGIT

Now you can have a x->string: function or x==y: function or x2d->3d: function. Still not allowing you to start a function with anything other than a letter. Sadly, still only ASCII letters because I haven't figured out the incantation for any letter of any language without including underscore which regexp seems to think is a "word character".

Sidetrack 2: private functions

Janet has a concept of private functions that aren't exported with the module. Also private constants.

I have wanted to implement this but haven't gotten around to it. Now it's relevant. In Janet a private function looks like this (defn- ...) but a public function is (defn ...). Constants are similar (def- ...) is a private constant and (def ...) is a public constant. Constants because they're not mutable. Variables, oddly take a 2nd parameter… (var \*private-var\* :private)

# public constant
(def my-public-constant "docs" :abc)
# private constant
(def- my-private-constant "docs" :abc)

# public function
(defn my-function "docs" ... )
# private function
(defn- my-private-function "docs" ... )

# public variable
(var *api-var* "docs" nil)
# private variable
(var *private-var* :private "docs" 123)
# ^^^ inconsistent stuff like this is why i'm writing a language
# why isn't it just var- ?
# i think you can use :private with def too???

It seems easier to just handle all the forms of function definition up front rather than going back and handling it later, so, now I need to decide how to handle privacy in my lang. Are there just 2 levels: Private and Public? Or maybe 3 like Java: Private, Protected, and Public. I think I can get pretty far with just 2. Realistically I haven't encountered any practical use of Ruby's protected in my day job in years. I seem to recall it being useful in Java ~20 years ago…

I know I said handle it all now, but "protected" is different. It opens up a lot of questions and complexity. So, we'll deal with that as we need it. If we need it.

So how to handle private? My initial instinct was to emulate ruby and add a private keyword and everything that came after that in the file was private. But, that leads to weird separation of methods between the public method and the private internal ones it relies on. Also, there's no way to visually distinguish between a public method name and a private method name and that violates two of the guiding principles.

So, new rule. Starting a function name with p- designates it as a private function. Thus, when you see p-multiply: you know that you're looking at a private function that, presumably, multiplies things. @P_THINGY would be a private global variable that can't be seen from outside the module. Non-global variables are always scoped to the function or namespace, and should never be directly accessible from outside.

This doesn't involve any grammar changes because p-foo: is already supported naming for a function. It would involve parser changes, but I haven't written that yet because I haven't written the language target which is what I'm ostensibly supposed to be doing right now.

Something something modules

By default, modules correspond one-to-one with source files in Janet, although you may override this and structure modules however you like. - Janet docs

Sidetrack 3:

Case statements, and if statements. For if statements the plan was to just go with standard scheme for now:

(if test true_response false_response)

So, for example

if: true
    println: "twas true"
    println: "twas false"

Simple right?

Then i realized i have the potential for variables of multiple types, which meant my contract grammar needed to support it. I thought i'd handled that case but went to check. Nope. Turns out i only handled it in a former incarnation of contracts which were just glorified type annotations.

So, I went to implement the grammar for that, and in the process decided to write an example implementation of the contract: function. Well, that involved the need for a case: function which I hadn't tried implementing, and that led to the following if statement

var: a_list [1 2 3]
if: >: size: a_list
		2
	println: "big list"
	println: "small list"
	return: true

That should convert to

(if (> (size a_list) 2)
	(println "big list")
	(println "small list")
	true)

but the tuplet version completely violates the guideline of removing ambiguity.

This isn't an option

if: >: size: a_list 2

because 2 becomes an argument to size: not >:

This works, but it's gross.

if:
    >: size: a_list
        2

this is clearer, but still gross and annoying to type

if:
     >:
         size: a_list
         2

Really, I need a grouping operator that isn't just another array.

Parentheses would be the most obvious but I want to reserve those for raw lisp. I'm thinking there are 2 obvious options:

grouping with angle brackets < ... > this works because calling less than or greater than functions always have a trailing colon: (>: and <:) so that you know it's a function call.
grouping with parentheses, which means doing something else for raw lisp…
- ~(raw lisp) or l(raw lisp) or lisp(raw lisp)

I'm leaning towards angle brackets because people tend to be familiar with their use in HTML as a grouping, and it leaves the lisp completely unadulterated.

We can also say that the first thing in a grouping is going to be a function that gets called with the remaining arguments, just like lisp.

So, that gives us:

if: >: <size: a_list> 2

but also this unfortunate case

if: <>: <size: a_list> 2>

I think that would actually work because <>: won't be interpreted as a function, because functions have to start with a letter unless they're one of the "special functions" and I don't like the idea of using <>: as a replacement for !=: becacuse you should say what you mean not use some esoteric shorthand for is greater than or less than. So, there is no special case for <>:.

So, it would work, but it'd be ugly, but that'd also be the developer's choice not a language requirement. There's no stopping people from writing ugly code.

So, time to add angle brackets to the grammar….

In the process of doing that I discovered that there's a problem with the grammer with regards to return. It only seems to want to lex correctly as a function: return: but I'm getting a headache and decided it wasn't worth fighting over. So, for now i return with return: not return. A fight for another day. I think it should be without the colon because it's not going to behave like a normal function in that it can break you out of a loop like break or break out of a function early.

Dev Diary 1

Sat, 23 Jul 2022 00:00:00 +0000

This first development diary is going to be long. There's a lot of thinking to write down before I can catch you up to where I today; July 23, 2022.

The origins of Tuplet

I write a lot of utility scripts. For years I've been writing them in Bash becasue I've wanted them to be something I could share with my peers. One day, I just got sick of having to fight with yet another piece of Bash's weird syntax. I've wanted to write my own language for years, something that was more tailored to the way my brain worked, and the ways I've come to believe a programming language should work.

I decided I wasn't going to fight with Bash any more. I was going to act upon my dreams and build a language for me, but it'd compile to Bash so that I could still create those utility scripts that I could share with my peers. So, that's what I set out to do.

One day I was working out how arrays would function, and I went down a little rabbit hole of what it takes to pass arrays into and out of functions in Bash. It wasn't pretty, and it required Bash 4.4 or higher if you wanted any hope of doing it without truly ugly hacks.

I continued on though, until I tried to write a string-split function as part of the standard library. I ended up on this amazing Stack Overflow answer, and spent the next couple hours trying to make it actually work. The. Next. Couple. Hours. It took me hours, with explicit – although imperfect – instructions to write a function that split a string on spaces. Hours. I've been coding professionally for nearly 30 years, and it took me hours to write a generic form of that function, and I had to compromise to do it too.

That was the "straw that broke the camel's back", but the decision to abandon Bash fully took a couple more days to sink in. It was the realization that even if I did succeed, I couldn't actually give my scripts to any of my peers, because I use macs, and they use macs, and Apple stopped distributing Bash at version 3.2.57. A version that couldn't run the newer functions I'd need anway. Yes, you can upgrade bash on the mac, but I think very few geeks actually do.

On top of that, was the acceptance of the fact that no-one cares about my scripts. I've got tons of things that'd save people time and effort, and when relevant, I let people know about them. They never ask for details. They never run the ones I share. Or if they do, they never tell me about it.

A few more days and the implications of that really sank in.

There is absolutely no point, in my trying to make a language with other people's needs, or convenience, in mind. No-one's going to use it, so why should I make something with compromises to satisfy non-existent users.

Tuplet is my langauge. It's a language for me. It is purely selfish. That's not because I don't want others to use it, but because it's highly unlikely anyone else ever will. Just like it's highly unlikely that anyone will ever read this.

Finding a target

With bash out of the picture I started hunting around for a target language. I was still holding out hopes of "maybe I can make this easy for folks", but one of the languages I was considering anyway was [Janet](https://janet-lang.org/). Janet, is a thoroughly modern Lisp. I don't know why it took so long for someone to write such a beast, but what's important is that it exists now.

At this point, I'd been iterating on the grammar for a while and had made a lot of progress. One night, I realized that I wasn't using parentheses for anything, and that that, combined with the fact that I was writing a transpiled language, meant that I could literally write raw lisp in my code, and it would take zero effort to compile it. My parser could literally just go "oh look, stuff in parentheses. Just spit it all out unchanged." This meant that I could trivially bootstrap the language with lisp until I had a standard library written in itself.

It's important to note that Tuplet is basically a lisp. It's heavily inspired by the thinking behind Wisp, otherwise known as SRFI-119. Wisp is basically just "I wish I could write Scheme but with Python style indentation instead of parentheses all over the place." Tuplet takes this idea to the next step. It says "That's cool, but there are a lot of places where languages like Python and Ruby are way, way easier to use."

To put it another way, there are a lot of things I've come to love about Ruby and other modern languages, but I also love writing in Scheme.

Anyway… the realization that there was no real value in catering to others, combined with a love of Scheme, plus a thoroughly modern lisp that I wanted to use, made me decide to use Janet.

Figuring out what I wanted

It's hard to know what you realy want in a language until you start writing one. So, I just started writing down my ideas in an org file, and then writing grammars to generate lexers and parsers. Long story short, much time was spent, and frustration was had until I decided to use Antlr and it's fantastic plug-in for the various JetBrains editors.

As I wrote, and edited, and saw the interplay of features I was designing, the Guiding Principles began to emerge. Now that those are starting to form decisions are becoming easier because I can use them not only to guide those decisions, but to help understand why I want what I want.

The actual language…

…will have to wait for another post. This one enough on it's own, and there's a lot of syntax already designed.