Open Design

Reverse engineering Sketch’s Smart Layouts algorithm

Reverse engineering Sketch’s Smart Layouts algorithm

Viktor ChlumskyViktor Chlumsky

October 13, 2021

Open Design allows you to crack open a design and see everything that's inside - from a rendering of an artboard all the way down to the properties of an individual layer. To get this level of detail without relying on the original design tool, we have to reimplement all of the functionality from every design tool that we support - that's Figma, Sketch, Adobe XD, Adobe Photoshop, and Adobe Illustrator.

Today, we render designs exactly as intended 99.6% of the time. We don't claim to support 100% of every design tool but we do care a lot about rendering precision and prioritize supporting features and fixing bugs that meaningfully affect this metric.

There's one design tool feature in particular that has been a thorn in our side and has cost us more engineering time than any other - Sketch Smart Layouts.

When re-implementing a feature from a design tool, the first step is to understand what it does. Typically, this is the easy part. Even with an advanced graphical effect, an experienced graphics engineer can generally get a rough idea right away. But Smart Layouts are different - their functionality is intertwined with the hierarchical structure of the entire design file. They were too complex to simply guess how they worked, so we had to come up with a new approach to reverse-engineer them.

What are Smart Layouts?

So, what are Smart Layouts anyway? When you design a web page or a mobile app, certain elements appear more than once and should have a consistent appearance. To retain the ability to globally modify all occurrences of these elements, Sketch introduced the concept of symbols. A repeating element can be defined as a “symbol,” and in place of mere copies, symbol instances are made, which are inherently linked to the common “symbol source,” formerly known as symbol master.

Even though the designer wants symbol instances to look alike, sometimes that doesn’t mean identical. For example, a UI button can be turned into a symbol. And while all buttons should have the same design, each one has a different label. That’s where symbol overrides come in. While all instances start out as exact copies, optionally, parts of them can be individually modified with the so-called “overrides.”

And finally, we arrive at the problem that Smart Layouts attempt to solve. When you construct a symbol that is more complex than just a simple button with a label, it often has its components laid out in a non-trivial way. But when you start changing parts of it with overrides, they may no longer fit this carefully crafted layout. What’s worse though, is that since we are still dealing with a symbol instance, the layout can’t really be fixed up without modifying the symbol source and messing up all other instances.

So Sketch came up with Smart Layouts, a system that attempts to deduce how the layout should change automatically, merely from one single hint - the layout direction. In the following example, crisis is averted with the use of several Smart Layouts:

The problem with Smart Layouts

Generally, components in a page are either positioned by hand (as in most graphics editors, including Sketch) or by a rigid layout system (e.g. HTML + CSS). With Smart Layouts, Sketch curiously tries to do both at the same time. This attempt to reconcile the two opposing paradigms gives designers a lot of freedom, but this in turn makes the layout logic very complicated and unpredictable.

This is the source of many issues that we faced during the reverse-engineering process. We found many inconsistencies in the implementation, some of which the Sketch team acknowledged and some they claimed were "features". In the latter case, we had to support those bugs in our implementation (since we wanted to be as close to 100% precision as possible).

In contrast, Figma introduced “Auto Layouts,” a feature that solves the exact same problem as Smart Layouts. Despite this, and in sharp contrast to Smart Layouts, I was able to understand how they work almost immediately. They constrain the designer to a more rigid layout system (they can't place contents arbitrarily), but this limitation is what ultimately makes the implementation predictable and robust.

So how do they work?

A key point is that Smart Layouts only take effect when overrides are present. Otherwise, they do not make any difference whatsoever. What happens in the basic case is that an element whose size has changed due to an override essentially warps the entire spatial axis (horizontal or vertical, depending on the direction of the Smart Layout) from the point where it ends onwards in such a way that elements following it stay the same distance from it. However, elements that intersect the “spatial warp boundary” do not fare so well. For example, their left side may be unaffected, while their right side may be swept away by the warping effect. In these cases, the transformation is only applied on the four corners of the layer’s bounding box, simply stretching the layer to a different size. But nothing is stopping the right side of the layer’s bounding box from being “warped” all the way to the left of what used to be its left boundary. These elementary mechanics, including the case mentioned at the end, are captured in the following animation:

Now you might be thinking that, while perhaps not immediately obvious, surely these rules weren’t that hard to figure out. And indeed, up until this point, they weren't. I believed that figuring out how to apply these rules to the general case would be just a formality before wrapping up the same day, but that couldn’t be further from the truth. It turned out that understanding this simple case had no bearing on solving almost any other, even one only slightly more complex.

For example, do you know what would happen if multiple overridden elements overlapped? Or what if there were nested smart symbols affected by the “warping”? How would this affect a parent instance, where the smart symbol is embedded? Did you know that ordinary layer groups can also have Smart Layout enabled but behave slightly differently? And how do you suppose this all works if some of the layers also have resizing constraints - a completely separate layout mechanism? Almost none of these examples behaved the way I expected and a complete documentation of my findings would be much longer than this article.

The Research

After initially making some good progress on reverse engineering, I hit a wall. Eventually, I was trying lots of different things without any clear direction and it became clear that I was not getting any closer to figuring out the complete algorithm governing Smart Layouts. A more systematic approach was necessary to crack this, so I turned to the scientific method.

For us, Sketch’s algorithm is a black box. We cannot look inside it to analyze how it works, but we can conduct experiments by sending inputs in and seeing what comes out. This way, we can form hypotheses in the form of our version of the algorithm or its parts and test if they hold up. I am not saying it’s on the same level as when physicists in CERN test which quantum model of the universe is correct by sending particles into the large hadron collider and seeing what comes out, but can you really claim the difference is more than semantic?

But before I could start my experiments, I had to develop a testing framework that prepared the inputs, supplied them to Sketch, and compared their output with my own. Sketchtool, the official command-line utility shipped with Sketch was instrumental in this endeavor, as without it I wouldn’t have been able to automate this process and test millions of design files.

Subscribe to get the latest
Open Design news by email

Step by step

That’s right. Several million design files with Smart Layouts have been tested during this endeavor, and although I haven’t managed to claim a 100% success rate, over 99% are a perfect match.

So where did we get that many design files? Don’t worry, we didn’t use those uploaded by Avocode users. I simply made a random design generator. Now, I still haven’t explained how this new approach is any more systematic. The key was that this custom design generator allowed me to fine-tune what kinds of designs were tested. I started with the simplest ones and once I achieved a (near) 100% match rate for a given complexity level, I would turn up the generator to come up with something harder. Rinse and repeat.

Before long, I was generating so many designs that even Sketchtool couldn’t keep up. I found that no matter how simple the design was, the Sketchtool utility would always take at least half a second to (presumably) initialize itself. I went so far as to implement a system to wrap several Sketch files into one, and then separate them again on the return journey. With this optimization, my testing throughput increased to 108 designs per second on a regular MacBook. And with a simple Python TCP server in under 20 lines, I didn’t even have to use the MacBook directly. (Note: Sketch and Sketchtool only run on macOS.)

Having a lot of examples for each scenario not only made it less likely that errors went under the radar, but I also had a larger pool of cases that failed the test, which I could closely analyze. In fact, I added in some extra features to streamline this process such as automatic lookup of the simplest case that failed, and even reducing some of the generated designs to just one dimension instead of two.

The End?

I wish I could conclude this by saying that the mystery of Smart Layouts has been solved once and for all, but to tell you the truth, there actually hasn’t been a eureka moment where everything suddenly fell into place and I knew that what I have is the one and only true Smart Layouts algorithm. Instead, it was a thorny road of incremental improvements, and although the success rate is now at over 99%, there are still known cases that just don’t match the original. What is important though, is that now we have a tried and tested procedure to iron out these remaining flaws in the future.

Subscribe to get the latest
Open Design news by email