Implementing using in Swift with SwiftSyntax

Last year I wrote about Improving struct composition in Swift by levering the using keyword and feature taken from Jai. Recently I’ve been inspired by pointfree to implement this feature using code generation.

In this post I want to dive in on the development of this tool and analyse the shortcomings of it.

I already went into detail in my previous post but basically using is a feature from Jai the, still in development, language started by Jonathan Blow. It lets you expose properties from a type as if they were part of the enclosing type.

The nice thing about it is that it brings one of the main advantages that class inheritance have over struct composition. But it is even more powerful because by using composition you can basically “inherit from multiple types”.

SwiftSyntax

SwiftSyntax is an SPM package that lets you use libSyntax from Swift. It is a tool that allows you to work with Swift code, using an AST. The interesting thing is that the AST is made specifically to keep syntactic information, as opposed to a purely semantic AST that compilers usually use.

The way to interact with the library is by implementing classes following the Visitor pattern. You subclass SyntaxVisitor or SyntaxRewriter and overwrite the methods for the specific syntax node you care about.

If you just want to do local changes or analysis this approach seems really good. But in my case I had to adopt a multiple step process, and for that, the visitor API seems too restrictive. I have to admit that the lack of documentation is probably the cause of making this trickier than it may be, specially since there are a lot of syntax nodes in a simple peace of Swift code that you don’t really think about but that you need to deal with when using this tool.

It all went better when I found this AST explorer. I still missed a better visualisation of what each node has, but it helped me a lot when having to navigate the deep hierarchies of member declarations and pattern bindings. Even with that tool I was relying a lot on LLDB and printing the runtime types of the nodes.

I left in GitHub the ugly code I was using when implementing the prototype, the level of casting and nesting was too much for me to try to simplify at that point. That code was being used for a while until I was able to refactor it to something more sensible.

The main problem is that what I needed doesn’t match the level of abstraction that SwiftSyntax operates at. For example all that code is just trying to get the identifier and type of a member declaration. The problem is that with the AST being at syntactic level it needs to distinguish between let name: String vs. let name, surname: String or even let name = "". All of those cases are basically the same, a variable named name of the type String, but if you just want that information it takes a bunch of code to unify all the scenarios.

Even in the current version I’m not supporting all those scenarios. Now that I have a better understanding of the AST and the relation between the nodes it wouldn’t be too hard to include support for them.

As a word of advice, approaching the code generation with String literals and interpolation can seem nice in the beginning, that’s how I started. But I realised later that it made other things difficult. For example, of the features I had to implement was to find and update already generated code instead of duplicating it. With the generated code being just strings it was quite hard to find a match, and even harder to modify it. After losing some time with different String indices versions I decided to switch to code generation using the AST APIs.

SwiftSyntax provides different sets of APIs to create AST nodes. I was using the make API and with API via the SyntaxFactory. These APIs are not bad at all. Make it quite clear what’s the structure of the nodes and helped me get a better understand of Swift syntax. That said, I can see how writting string templates (Swift code) is way nicer. You can see an example of how to generate a getter and is not super clear. I ended up factoring out portions of code into extensions of the factory which improved the legibility by a lot. As Stephen pointed out to me in an email, maybe a middle ground could be found by making a small DSL for the syntax nodes, kind of what swift-html does for HTML nodes. I find myself thinking about Rust macros and how some DSL in Rust look much nicer than the code I wrote for this tool.

One of the trickier parts to get right is dealing with the trivia, the name the library gives to spaces, new lines, etc. As you expect from a tool focused on refactoring and formatting working with trivia is an integral part of the API. I tried to keep the unspoken rule (I heard it in some video a while ago) that the trivia should be part of the previous node, so I put as much trivia as I could into the trailing. To know exactly where it should go I used the AST explorer and tried to mimic that behaviour. Having to manually format code like this makes you appreciate all the nuances that simple code has.

SwiftUsing

The current version of this tool allows you to code generate extensions with computed properties for types found in the same file. You can see how this goes further than “local changes” and why the visitor pattern was quite limiting.

The first approach I tried was to gather the information (nodes) and then modify them manually. I wanted to do all in a single pass because doing more seemed wasteful. This got me far but when I tried to save the file with a text representation of those nodes, including the extra logic to update existing code, it got really complicated. As I mentioned before, working with strings is not the ideal situation.

Instead the current approach of doing multiple passes seems to work much better. First I gather the type information and the variables that want this feature; then I code generate the extensions; and finally I rewrite the original syntax tree.

The first and last steps are both integrated into a “pass” using a SyntaxVisitor and a SyntaxRewriter respectively. This defiantly seems to be a better approach that matches SwiftSyntax API design.

You can see an example of the tool in action on the example file.

The tool is capable of understanding mutability by checking if the properties are var or let. When it detects that a variable is immutable it skips the generation of the setters. This matches the programmer intend and makes the code compile 😊.

As mentioned before, there are other scenarios that are not handled, specially around different syntaxes. But one of the main shortcomings of this prototype is the inability to get type information from multiple files or libraries. I haven’t worked on that yet because the occasions I’ve been using this I haven’t really needed it, and, honestly, because seems quite tricky with SwiftSyntax. This is something that would be nice to improve in the future.

Deeper integration

Code generation with external tools is powerful but I always feel that is just poor man metaprogramming. My desire would be that this could be made as a plugin that runs as part of the compiler. This would imply that no source code needs to be generated, no files have to be touched and that the user doesn’t have to run the tool manually or deal with multiple compilation steps. It would be completely transparent for the user.

As the tool creator, a plugin system like this would let me fed the changes on the AST directly into the compiler pipeline at the right moment, that’s the dream. Hopefully this would make it easier to gather more information and to reduce the performance cost of this functionality.

But also keep in mind that this code generation only get us that far. The original using design in Jai allows you to use it in many more places, for example inside local scopes to expose type properties as if they were local variables! This would definitely need to have a proper feature design and integration directly into the language and the compiler.

Alternatives

Of course this flexible compiler is not gonna come any time soon, and I doubt this feature would pass trough Swift Evolution. But there are some other alternatives that we could consider to improve it.

First, from Swift Evolution, seems like soon we may get Custom Attributes. This won’t get us compiler integration but at least we don’t have to rely on ugly comments to annotate the properties that want to use the feature.

And of course, I’m pretty sure many of you are thinking about Sourcery. Being honest the only reason to use SwiftSyntax is because I wanted to learn and try what could be done with a library that is closer to the compiler and uses an AST. For AdHoc code generation Sourcery is the king no doubt, but I feel like for building a tool like this working at the AST level instead of string templates is much better. But honestly, I just wanted to play with SwiftSyntax 😋

Conclusion

SwiftUsing is available on GitHub and I’m personally using in a couple of projects. I’m really happy of how it turned out specially for something that started as a prototype just a few days ago.

It’s almost 100% unit tested, and this is a great reminder that the numbers alone don’t mean anything, there are many scenarios that the tool doesn’t cover 😉

I may keep working on it if I find that I need more from it or if Swift itself improves and makes some of this things easier.

Thanks for reading!