Obvious features aren't obviously made

Written by Rico Kahler, Bjørge Næss, Mikolaj Dobrucki

References in Place lets you create references to new documents inline, without having to navigate away from the document you’re currently authoring—a must-have feature if you want connected content.

And starting with v2.23.0, you’ll see the feature as a “Create new” button, as well as a clickable preview that opens the referred document in a new pane alongside your current one. Sounds pretty straightforward to make, right?

It was not.

Believe it or not, this feature was planned to launch with the initial release of Sanity but was later deprioritized due to the challenges we’ll talk about in this post. Since then, References in Place has never really left our minds. It just took a few breakthroughs to make the feature possible.

Watch the demo from the Open House:

What’s a document?

Let’s start by defining a document. It’s not as straightforward as it may appear.

There are two possible definitions of a document:

1. In the context of Content Lake, a document is a JSON object with an _id and a _type.
2. But in the context of the Sanity Studio, a document can be comprised of two Content Lake documents. We call this a document pair. One document represents the “published” variant of the document while the other document contains the “draft” variant.

When you edit a document, you’re not editing the published variant. You’re actually editing a copy of that document that only initially differs by its ID. If you start editing a published document without a draft variant, we’ll copy the contents of the published document, create a new draft document, and attach a special prefix (drafts.) to its _id.

What does “publish” mean?

In the context of the Sanity Studio, publishing is the opposite of the process we outlined above. We copy the contents of the draft document into a “published” version, removing the drafts. prefix.

But that’s not all there is to publishing. In order for this publish operation to be allowed the document must pass your validation criteria. This is an important concept to understand for the next section.

So why was making References in Place so hard?

Prior to References in Place, the Studio didn’t let you reference a document unless it had a published version.

This is an important aspect of Sanity’s data integrity model because it ensured that the document being referenced was in a valid state and had gone through your organization’s approval process.

Implementing References in Place was challenging because it required us to let you reference draft documents anyway.

So we tried to do exactly that and the more we thought about it the more issues we found.

If we naively allowed referencing draft documents, then we would break the assumption that all referenced documents are in a valid state. Breaking this assumption causes two issues:

1. Applications that assumed all references were validated could break. For example, if you have a document with validation that ensures an array always has one item, it’d be safe to assume the first item of the array will always be present.
2. Authenticated clients could potentially leak draft content. For example, if you use a static site generator that fetches data with an authenticated client, it could potentially output draft content if you forgot to filter it out.

Then there were user experience implications as well. We knew we wanted References in Place to be quick and easy to use. And we found that if we allowed referencing draft documents, then you’d quickly run into an issue – draft documents couldn’t be deleted unless you removed all references to it.

Typically, you’d want this deletion protection because it ensures your apps don’t break due to missing references but when iterating on multiple drafts by multiple editors in real-time, it can add a lot of friction.

Though seemingly minor, this problem led to an important breakthrough that made everything come together.

What if we used weak references?

By default, Content Lake ensures that all references resolve to documents that exist within your dataset. We call these strong references because they are enforced at every level. For example, if you try to delete a document that’s actively being used in a reference (even via the API), you’ll get an error message saying the mutation failed.

Weak references on the other hand are references that don’t need the referenced document to exist. If you try to delete a document that is weakly referenced, Content Lake will no longer stop you from doing so. This would allow draft documents to be deleted even if there was another draft that still referenced it. You may end up with some dangling references but those can be cleaned up by requiring them to resolve to existing documents using validation errors.

Weak references were a key breakthrough for multiple reasons. They solved the deletion overprotection problem and it turns out they gave us a path to solve the rest.

What if we referenced a document that didn’t exist?

The main issue with referencing draft documents is that we break the assumption that all referenced documents are in a valid state. So, with weak references in mind, the question became: What if we didn’t actually reference the draft document?

That is, what if we utilized weak references to reference the _id of the published document (that does not include the drafts. prefix) that didn’t exist yet?

This was our breakthrough. By utilizing weak references to reference the _id of the published document, we can keep the assumption that all referenced documents are in a valid state, preventing applications from breaking and preventing draft content from accidentally leaking.

How we pulled it off and how it works

Using weak references helped us solve a lot of the problems with referencing a draft document, but that strong reference protection was nice, right?

Of course, referential integrity is an important aspect of Sanity’s data integrity model and we wouldn’t ship this feature if we couldn’t keep that.

In order to keep referential integrity, we “strengthen” the reference at publish time.

We added a new key to the references, _strengthenOnPublish that tells the studio to convert the weak reference to a strong reference when you click the publish button. This re-introduces all the data integrity safeguards we mentioned earlier.

And with all that, we had our path forward to implement References in Place.

We’re quite happy with how it turned out. This allowed us to finally ship a feature that seemed to go against everything we uphold while still maintaining the status quo.

Other challenges we encountered

All the issues we’ve talked about so far have just been about making the feature possible.

Even after our breakthrough, References in Place still posed challenges across interaction and user interface design, as well as engineering with a lot of edge cases in consideration. That’s why it took us over 3 years to ship it. We knew we wanted to ship this feature in the right way and it wasn’t easy getting here.

Interaction design and panes

Opening one reference without losing context is one thing, but opening multiple references (from a reference, from a reference, from a reference) is another.

We had to ask ourselves what it meant to open a reference, what it should feel like, and what it would be similar to in the current studio. We knew shipping this feature in the right way meant we had to find a way to open those references in a recursive way that felt natural in the Studio.

So we built a new way of opening documents in a new pane.

This meant taking a deep dive into the existing pane system, refactoring it to support the new requirements, and preparing for potential pane-related additions too!.

Updating the reference input for drafts

Prior to Reference in Place, we only allowed references to published documents. This meant the previous reference input could filter out all draft documents to simplify search results.

With the addition of References in Place, this is no longer the case. We had to update the reference input to consider both the draft documents and published documents which meant we had to update the reference input to de-duplicate the results.

We now show two different icons so you can know if a published and/or draft variant of the document exists.

Updating validation for draft references

Similar to the reference input updates, validation also needed to be updated to support a new use case only relevant to Reference in Place.

Prior to Reference in Place, validation only depended on the current document but this is no longer the case. In order to ensure we can always “strengthen on publish”, we mark the current document as invalid if it references a document that has not been published.

This meant that we had to update the validation library to re-run when another document updated. Along the way, we fixed some bugs and converted it to TypeScript.

Arrays of references

We wanted the experience of editing an array of references to look and feel like a document list, meaning inline instead of a dialog.

This wasn’t as straightforward as it seemed because of how the Form Builder composes input components together. When we tried to render the reference inputs inline, we got not-so-great results.

So, we built a workaround that is aware of arrays and renders a special case. Now, we detect if an array item is a reference and we render it inline instead of inside a dialog. It’s not an ideal solution in our eyes but it’s a tradeoff we’re willing to make in the name of delightful editor experiences.

Initial Value Templates and Roles

As you may know, the Studio is a highly customizable application with lots of surface area. That means when we create new features, we have to make sure it works with the rest of them.

Like Initial Value Templates and Roles.

Initial Value Templates allow you to specify multiple templates with different pre-populated values. Roles allow you to specify custom filters for granular control over documents (including creation).

Put those two together and you now have requirements to disable an initial value template depending on your role.

Now layer that on the ‘Create new’ button.

In order to get this one done, we lifted some code to shared places, made some new (internal) APIs, and refactored existing implementations.

Conclusion

Implementing References in Place was an interesting and unique engineering challenge that involved many parts of the team and touched many surfaces of Sanity Studio. If solving these kinds of challenges sounds interesting to you, we’d encourage you to apply to our open positions so you can help us solve the next round of fun challenges!