Introducing Joy: A Go-to-Javascript Compiler

Introduction

Today I'm excited to share the Joy compiler with the developer community. Joy brings Go's simple design and excellent tooling to the frontend. I believe Joy will become the most productive way to build large-scale, maintainable web applications. I hope you'll find Joy delightful as I do.

What is Joy?

• A fast 1:1 Go to Javascript compiler
• Uses Go's static analysis tools to eliminate dead code
• Outputs ES3 Javascript that works in every browser (coming soon)
• First-class support for React & Preact (coming soon)
• Contains a simple macro system for integrating with existing Javascript
• Uses a built-in headless Chrome for running scripts
• Provides zero-cost, typed DOM & virtual DOM packages (coming soon)
• Ships a minimal runtime only when it's needed
• Open source on GitHub

What does Joy look like?

The following examples were selected from the test suite and should give you a taste of Joy's capabilities. For more examples, you'll want to check out the full test suite on GitHub.

The following Go code:

package main

import (
  "github.com/matthewmueller/joy/dom/htmlbodyelement"
  "github.com/matthewmueller/joy/dom/window"
)

func main() {
  document := window.New().Document()

  a := document.CreateElement("a")
  println(a.NodeName())
  strong := document.CreateElement("strong")
  println(document.CreateElement("strong").OuterHTML())
  a.AppendChild(strong)

  strong.SetTextContent("hi world!")

  body := document.Body().(*htmlbodyelement.HTMLBodyElement)
  body.AppendChild(a)
  println(document.Body().OuterHTML())
}

Compiles into this Javascript:

;function() {
var pkg = {};
pkg["52-basic-dom"] = (function() {
function main () {
var document = window.document;
var a = document.createElement("a");
console.log(a.nodeName);
var strong = document.createElement("strong");
console.log(document.createElement("strong").outerHTML);
a.appendChild(strong);
strong.textContent = "hi world!";
var body = document.body;
body.appendChild(a);
console.log(document.body.outerHTML)
};
return {
  main: main
};
})();
return pkg["52-basic-dom"].main();
}()

Download Joy

The easiest way to get started is to run this curl command in your terminal, but you can also download Joy from the GitHub releases tab.

curl -sfL https://raw.githubusercontent.com/matthewmueller/joy/master/install.sh | sh

Joy's Guiding Principles

These principles greatly influenced the design and implementation of Joy:

• Code is written once, read many times
• Less syntax is better than prettier syntax
• Typed code is easier to update and maintain
• Avoid large runtimes and unused code
• Don't break browser support without a good reason
• Generate concise Javascript from idiomatic Go
• Embrace standard libraries and standard tools

Why did you work on Joy?

I want to let you in on a little secret - I love Javascript.

I started making websites when I was 13 and became obsessed with AJAX after playing with Google Maps and chatting on Meebo. I owe my career to the Javascript and Node.js community and still consider myself a frontend engineer. In the early days, Javascript didn't get much love. Consistency across browsers wasn't taken seriously, so you needed frameworks like jQuery to get anything done.

Now we have the opposite problem - Javascript gets too much love.

While browser consistency has greatly improved, the language itself has been rapidly changing. This has caused major breaking changes over the last couple years. These breaking changes have segmented the community, left behind older browsers and caused cascading issues downstream in tools and across libraries. There's no doubt that the language has been improving and critics would argue that these changes are just growing pains. While I do believe the language will settle down in the future, these issues make it very difficult to author long-lasting web applications today. For me, the breaking changes weren't enough motivation to work on Joy. I began looking for post-Javascript alternatives after I built Standup Jack.

Standup Jack is a medium-sized Node.js application that schedules and coordinates team standups on Slack. Standup bots are particularly tricky because if you take the service down for even a few minutes, there are going to be teams not receiving their standups on time.

I found that while it was easy to develop Standup Jack, it's been very hard to maintain and add features to the service, even with good test coverage. Over time, your understanding of your work fades and you need good tools to tell you what you should and shouldn't do. This issue is compounded when you didn't write the code yourself or you're under pressure to push a fix.

You need tools that can walk all the possible code paths and warn you, "Hey slow down, you're forgetting about this". Tools like this require a type system. Microsoft realized this and developed Typescript in 2012. Facebook realized this and developed Flowscript in 2014.

While I've used and prefer these type systems over vanilla Javascript, they come with their own additional complexities because they're adding types on top of an untyped, dynamic language. Go's type system is a lot simpler because it was developed alongside the design of the language.

Additionally, the browser APIs have been getting more powerful over the years. You used to need big Javascript frameworks to be able to get anything done. Nowadays, there are only a handful of indispensible Javascript libraries. The rest can be re-written or bound inside a Go application with minimal effort.

For these reasons, I believe that writing maintainable and long-lasting Javascript that works in every browser is more time-consuming and painful than starting from scratch and kickstarting a fresh ecosystem built on top of a more solid foundation.

That's why I got to work.

Why Go?

There are 5 primary reasons why I chose Go as the source language:

Go embraces write once, read many times

Go was built to be easy to read, understand and maintain. As one of the Go's original authors puts it:

Go was deliberately designed from the start to support large scale programs implemented by hundreds or thousands of different programmers.

Go's syntax is stable

The language has been stable and backwards compatible since 2012. This is very important and it allows us to be confident that the Go code we write today will work 5 years from now.

Go's type checking and compiler are blazingly fast

Coming from the frontend world, we expect to be able to see our changes show up in the browser in the blink of an eye. We've spent countless hours tweaking webpack configurations to get hot reloads working so we can reach a feedback loop that puts us in the zone.

You'll find that Go's type checker and compiler performance check this box and the cross-platform binary builds make it an excellent language for frontend tooling.

Go has an excellent standard library and tools

Go has a comprehensive and well-tested standard library and nearly everyone uses the same set of tools. While this may sound like a lack of diversity or an innovation killer, it actually makes life easier.

Less options mean easier decision-making, more expertise and more resources can be directed towards a smaller set of libraries and tools.

Implicitly asynchronous

With Go you can move asynchronous behaviors inside functions and libraries so from the caller's perspective, an asynchronous function looks synchronous. No more callback hell, promise chains or async function wrappers.

Critics may say that this hides expensive calls, but you can answer these kinds of questions easily with the "Jump to Definition" tool that's supported by all the major editors.

How does Joy Work?

The Joy compiler has 5 steps: parse, index, graph, translate and assemble.

1. Parse turns your Go code into an abstract syntax tree (AST). This is primarily done using the x/tools/loader library provided by the Go team.
2. Index processes these Go ASTs allowing us to answer deep questions about our program. Questions like: "what packages does this function depend on?" and "which structs implement this interface?". This step is very similar to what our editors do to give us features like autocomplete, "Find all References" and "Jump to Definition".
3. Graph uses the index to create a dependency graph. This graph is sorted topologically and pruned, resulting in a list of nodes containing the functions, variables, structs and interfaces that are used in our program.
4. Translate is the fun part. Translate is where we take the nodes in our graph and the knowledge in our index to translate the Go AST into a Javacript AST.
5. Assemble takes the resulting Javascript AST and generates Javascript code.

How does Joy work with existing Javascript?

There are multiple ways to bring existing Javascript into a Go codebase using the macro package bundled with Joy:

• macro.Rewrite(expr, variables...) is a powerful macro that makes it easy to turn Go code into custom Javascript without paying any size or performance penalties.
• macro.File(jsFile) makes it easy to import existing Javascript libraries inside a Go application.
• macro.Raw(jsString) is the easiest way to add Javascript but also the least recommended. This function will inline Javascript directly into your Go program.

There will be more documentation and examples on these macros in the coming weeks, but for now you can check out the examples above and the DOM's window API on GitHub to see the macros in action.

How does Joy work with existing Go?

Most existing Go code does not yet compile to Javascript. This is because most of the standard library still needs to be translated to Javascript. This will be a focus of the 2.0 release. Signup for the mailing list to follow along with the progress.

What's next for Joy?

1.0: Getting Joy Production Ready

Finish up the Go to Javascript translator and finalize the DOM and virtual DOM APIs.

Right now the translator is about 90% complete. There are still some missing statements (e.g. switch, select), syntax variations and a few scoping bugs that need to be addressed.

Additionally, we need to finish the production build. According to this issue, it's possible to bring generator support to ES3, which means channels can be supported. At the very least, we can support ES5 by removing async/await function calls and the bind functions.

Lastly, I'd like to add incremental compilation here too, so we can keep the compiler zippy for large projects.

After Milestone 1 is complete, Joy will be considered production-ready and stable. At this point, we will take breaking changes very seriously. If we do have to make breaking changes, they will only be done during major releases.

2.0: Growing the Ecosystem

Implement much more of Go's standard library. Standard libraries like time, url, fmt need to be translated.

We'll also be improving the toolchain, by adding support for go test (to run browser tests!) and adding more support for essential frontend libraries like D3.

3.0: Hygienic Macro System

At this point, I'd like to explore creating a hygienic macro system to allow Go to be translated into more variations of Javascript. This will allow the compiler to be extended to support more flavors of Javascript without needing the Joy authors' blessing. It's also more decentralized than a plugin system like the one in the Babel compiler because it allows library authors to implement their own plugins without needing you to set it all up manually.

4.0: Optimize performance and build size

Further reduce the Javascript build size, improve Javascript performance and improve the compiler's performance. This will be ongoing, but will be a focus for this phase since the compiler's internal APIs should be settled.

5.0: Target WebAssembly

Target WebAssembly for non-DOM libraries. This work may happen in parallel with the other milestones, but will be much more of the focus going forward since that's where the future is heading.

FAQ

How does Joy compare to GopherJS?

GopherJS and Joy are two different approaches to the same goal.

PaluMacil describes it best:

GopherJS started with Go as a priority and is working towards great JavaScript interoperability whereas Joy started with JavaScript as a priority and is working towards great Go interoperability.

GopherJS currently implements much more of the standard library and has a complete syntax translation. Joy is working to get to this point.

Joy uses Go's static analysis tools to eliminate dead code and uses a macro system to produce much smaller Javascript builds with better performance. Joy's goal is to produce Javascript that looks like it could have been handwritten. The compiler architecture was very much inspired by projects like Bucklescript and Rollup.

Joy's development started with me asking myself:

Could I create a more productive and fun frontend development environment in Go over using Typescript, Webpack, React and the Javascript module ecosystem? To that end, the initial focus is on supporting the DOM and essential frontend libraries. Later on, we'll be working on compatibility with existing Go libraries.

How much is Joy going to cost me?

I hope to keep the core compiler free forever and find other ways to pay for Joy's development and support.

As an independent developer with limited resources, I can only take this project so far on my own. The only way for this project to reach it's full potential is for Joy to become community-driven and financially self-sustainable.

The work I do to reach a break-even is even more important than the code I write and I will be experimenting with various ways to reach this goal.

If you can help me do that or you have ideas on how to get there, please reach out to me directly on Twitter or comment on this Github issue.

What is Joy's license?

Joy is licensed under GPLv3. In short, this means that Joy is free to use and modify, but any remixes to the Joy compiler must also be free and open source.

Of course, this only applies to the compiler code itself. You own the input and output source code.

If you'd like to use Joy in your organization but find this license too restrictive, please reach out to me directly.

Can Joy compile to WebAssembly?

Not yet, but it's part of the master plan (cue sinister laugh).

WebAssembly is very much on my mind and I think Joy's parsing, indexing and graphing phases can be re-used to translate Go into an extremely compact WebAssembly binary format.

I wanted to start with Javascript because WebAssembly is still experimental technology and available in about 68% of web browsers at the time of writing. Additionally, the current implementation of WebAssembly cannot access the DOM directly and is meant to be used alongside Javascript in the form of WASM modules.

Outside of Joy, there's some work being done on the Go compiler itself to bring WASM support natively. If that works out, I'm hoping we can use that and choose the output format in an intelligent way based on what type of code you're compiling and the target browser.

In either case, if we want to start using Go on the frontend, we'll need to start building frontend tools and bindings in Go, so why not get started on that work today!

Isn't Go too verbose?

When compared with Javascript, Go takes a lot more keystrokes to express the same idea. This was very frustrating to me at first, but I soon realized that Go will save you time in other ways:

• The type system saves you countless hours on syntax and type errors. It's like a spell checker for your program.
• Go's consistent packaging system and syntax saves you a bunch of time normally spent futzing around with NPM, Webpack and Babel.
• Autocomplete and Jump to Definition saves you time looking up function signatures and documentation online.
• Less syntax variations make it easier to come back to code later and understand it or read through other people's code.
• gofmt consistently formats everyone's Go code so you spend less time learning someone else's style
• goimports will automatically bring in the right dependencies as you need them allowing you to stay focused on the code at hand.
• Go's standard library means you have immediate access to high-quality code and you're spending less time searching around for the library that meets your needs.

More generally, this means that you spend more time in your editor coding and less time context-switching and getting distracted.

Does Joy work with React?

Absolutely! Joy has virtual DOM bindings for React today, but it's not well-documented yet.

In the coming weeks I'll be introducing a framework based on Next.js that will make programming with React even easier. Sign up for the mailing list to find out when that's ready.

I believe React is just the beginning here. Now that we have a Go to Javascript compiler that produces really great Javascript, we can create all sorts of tools for the front-end ecosystem.

Personally, I'd love to see the Vue.js compiler brought to Go where the <script> tags would export Go functions.

Is Joy right for my team?

It would be misleading for me to say that Joy is a right call for every team. Joy is still in it's infancy and your choice in tech stack will largely depend on the needs of your application and the size of your team.

I think Joy is a great choice if your web application is sufficiently complex, expected to be updated over multiple years or you have multiple teammates working on the same codebase.

While Joy and Go's frontend ecosystem are still nascient, I believe they will grow very quickly to cover a variety of needs.

My hope is that you'll give Joy a try for your next project and see if it's right for you and your team and if it's not, I'd love to hear why.

Why do you ask for our email address?

I'd like to be able to reach you when we add features, create tutorials and spotlight websites using Joy in production.

I hate spam as much as the next person and if the emails you receive aren't helpful, I'd love to hear how we can improve them. And if you want to stop receiving email altogether, you'll have an option to unsubscribe in each email sent.

What metrics do you collect?

Joy collects metrics on when events happen, number of lines of code compiled and compiler errors. Joy will never collect any information on the actual contents of the code.

These metrics will helps us move quickly to improve the performance of the compiler and address regressions.

How can I help?

The best way to help would be to provide financial support. If I can work on Joy full-time, the project will move a lot faster.

Where can I find more examples?

Right now the best place to find examples is in the test suite. I'll be adding many, many more examples in the near future.

How can I contact you?

The fastest way to reach me is on Twitter but you can also email me. In either case, please be patient and respectful.

Anything else?

The sky is the limit! Get after it.

More Links

• Run joy help to see what else Joy can do
• Chat with us in #joy-compiler on Slack at gophers.slack.com
• Star github.com/matthewmueller/joy
• Follow twitter.com/mattmueller for project updates

A Special Thanks

The Joy Compiler stands on the shoulders of some excellent engineers.

TJ Holowaychuk

The feedback I received from TJ was invaluable during the early stages of design and development.

The Go Team

They created a truly pragmatic, modern programming language. Their tooling is in a league of its own and most of the heavy lifting done by Joy is actually making use of the hard work put into the Go standard library and it's peripheral libraries like x/tools.

A special shoutout to Alan Donovan whose presentations on Youtube helped me to make sense of some of the lesser documented tools, Robert Griesemer for a just-in-time pull request adding support for Go AST rewrites and Michael Fraenkel for a much smarter Go imports command that greatly simplified the DOM and VDOM code generation.

The GopherJS authors

GopherJS was instrumental in showing me that creating a Go to JS compiler was even possible and whenever I had a question about what a Go statement should translate to in Javascript, I looked to GopherJS for inspiration.

The Typescript & Edge teams

Joy would not have seen the light of day if I knew I would have to write an entire DOM API by hand. Fortunately, the Typescript team with the help of the Edge team did most of that work already, allowing me to programmatically generate a DOM API.

The Chromium Team

From the beginning, I knew I wanted joy run to execute against a real DOM. Thankfully, the Chromium team came out with a headless version of Chrome that is performant, reliable and easy to use.