Baking Metal shaders with Vapor and SwiftUI

14 min readApr 18, 2024

A new article - a new closed gestalt about “I would like to make what I saw on my own”. Today we will take this Metal shader for consideration.

Original Metal shader from Robb’s tweet

💡 Overall, I highly recommend checking out this account, Robb does amazing things with UI and is very inspiring, some things feel just like a magic.

What caught my attention here is the fact that the shader is edited on the fly. In the comments Robb suggested that he compiles this shader and then applies it.

What’s the first thing that comes to mind? Probably some analogue of eval in Swift. But the reality turns out to be much more prosaic. Apple provides an opportunity to compile shaders through CLI and then load them in runtime. And they have a good example for this in their documentation (kudos to them).

Building a Shader Library by Precompiling Source Files | Apple Developer Documentation

Create a shader library that you can add to an Xcode project with the Metal compiler tools in a command-line…

developer.apple.com

So what are we going to cover and implement in this article?

build a lil (hah) code editor using SwiftUI
draw a distortion shader
create a local server on Vapor to compile shaders
set up a WebSocket connection for data exchange

Looks like we’re going to make a whole service here. Intrigued? Then let’s get started!

💡 Important note. The code is written in Xcode 15.3, minimal version of iOS 17. Error handling when calling throws methods is left outside the scope of this article.

An Unexpected Journey begins

Editor

The basic idea is to split the screen into two parts — a preview of the shader and an area for editing the source code. Let’s sketch a small display for one of the halves.

struct SocketShaderScreen: View {
  var body: some View {
    Color.gray.opacity(0.2)
      .aspectRatio(contentMode: .fit)
      .clipShape(RoundedRectangle(cornerRadius: 16.0))
      .overlay {
        VStack {
          Text("Preview")
            .font(.system(.footnote, design: .monospaced))
            .foregroundStyle(Color.gray)
          Spacer()
        }
        .padding(16.0)
        .frame(maxWidth: .infinity, alignment: .leading)
      }
      .padding(8.0)
  }
}

Running the code, we get a canvas like this. But we need two such views.

To avoid duplicating code in the view, we will add a factory method that builds this canvas. In the parameters we pass the title and the content as @ViewBuilder, which we place on top of the fill colour in its overlay.

private func makeContainer<V: View>(
  title: String,
  @ViewBuilder content: () -> V
) -> some View {
  Color.gray.opacity(0.2)
    .aspectRatio(contentMode: .fit)
    .overlay {
      content()
    }
    .clipShape(RoundedRectangle(cornerRadius: 16.0))
    .overlay {
      VStack {
        Text(title)
          .font(.system(.footnote, design: .monospaced))
          .foregroundStyle(Color.gray)
        Spacer()
      }
      .padding(16.0)
      .frame(maxWidth: .infinity, alignment: .leading)
    }
    .padding(8.0)
}

The only thing left to do in the body is to call this builder. We put the views themselves in VStack, so that they are placed along the height of the screen.

var body: some View {
  VStack {
    makeContainer(title: "Preview") {}
    makeContainer(title: "Editor") {}
  }
}

The code editor is a regular TextEditor. For it we define @State variable in SocketShaderScreen, where the text will be stored. We will disable scrolling and autocapitalisation for it beforehand, so that text editing will not cause issues.

struct SocketShaderScreen: View {
 @State private var rawCode = ""
 
 var body: some View {
   VStack {
     makeContainer(title: "Preview") {}
     makeContainer(title: "Editor") {
       TextEditor(text: $rawCode)
        .textInputAutocapitalization(.never)
         .scrollContentBackground(.hidden)
         .scrollDisabled(true)
         .tint(.black)
         .font(.system(size: 14.0, design: .monospaced))
         .padding(.top, 40.0)
         .padding(.leading, 12.0)
     }
   }
 }

 ...
}

Before we go too far, let’s build Apple’s rainbow. The composition logic is pretty simple: VStack and colours for filling. We call drawingGroup to render the VStack together with the content as a whole. This is done for optimisation purposes, to reduce the number of views in the rendering tree and speed up the rendering process.

makeContainer(title: "Preview") {
  VStack(spacing: .zero) {
    Color.green.frame(height: 20.0)
    Color.yellow.frame(height: 20.0)
    Color.orange.frame(height: 20.0)
    Color.red.frame(height: 20.0)
    Color.purple.frame(height: 20.0)
    Color.blue.frame(height: 20.0)
  }
  .drawingGroup()
  .frame(width: 200.0)
}

We have this result, already something, further - more!

Shader

A bit of theory. A shader is a function computed on the GPU. They can be used to create high performance UI effects. End of theory

SwiftUI 5 gives us the ability to apply shaders directly to a mapping. There are currently 3 methods out of the box that work with different display characteristics:

colourEffect
distortionEffect
layerEffect

We will now focus on the distortion effect.

Shaders through .metal files, which are written in a special C++ dialect - Metal Shading Language. Create a new file to the project called distortion.metal and put the code from the original tweet to it.

#include <metal_stdlib> 
using namespace metal;

[[ stitchable ]]
float2 distortionEffect(
    float2 pos,
    float4 bounds,
    float t
) {
  float2 uv = pos / bounds.zw;
  pos.y += 4 * sin((4 * uv.x + 2 * t));
  return pos;
}

Let’s see what’s going on here. First of all, the function signature of this shader. SwiftUI requires it to be written as follows:

[[ stitchable ]] float2 name(float2 position, args...)

The stitchable modifier allows the compiler to subsequently call this function from Swift.

💡 In fact, stitchable has a much broader application, but the definition above will suffice for the purposes of this article. To familiarise yourself with Metal principles, you can refer to the official specification.

position is a coordinate vector representing the position of the pixel to be processed. This parameter is passed by the system. args is a variadic list of custom parameters that we can use to concretise the processing of the pixel.

In our example we pass bounds, which represents the position and size of the entire view to which we will apply the shader (a.k.a. a value of type CGRect) and a parameter t, which indicates the progress of the animation.

uv represents a normalised coordinate within the passed bounds value, this allows us to make further work with the transformation independent of dimensions.

float2 uv = pos / bounds.zw;

Distortion effect from the example is based on the sine function, which is harmonic and allows us to achieve the effect of oscillation — repetition of values with the passage of time.

pos.y += 4 * sin((4 * uv.x + 2 * t));

💡 You can read more about sine function and harmonic motion on Wikipedia.

Sine wave - Wikipedia

A sine wave, sinusoidal wave, or sinusoid (symbol: ∿) is a periodic wave whose waveform (shape) is the trigonometric…

en.wikipedia.org

After a little bit of theory, it’s time to get back to the familiar world of SwiftUI — let’s try to apply this shader.

The distortionEffect modifier we add to the previously described VStack. Its first parameter is a shader call. To call it, we must get an instance of ShaderLibrary.

💡 For now, its default implementation is enough for us, as the shader itself lies in our project (that's where it lies, right?).

We call the shader with the same name as it is described in distortionEffect.metal file. It should be noted once again that we do not pass the first parameter - pixel position, as the system does it for us. We only pass bounds, which we set equal to the size of VStack itself, and t, which we will provide with the value 0.0 for now.

The maxSampleOffset parameter defines how far the modified pixels can move away from their original position. It's kind of like a mask, but for the shader. Let's set the offset to 200.0 for width and height, so that the transformations don't deny themselves anything.

makeContainer(title: "Preview") {
  VStack(spacing: .zero) {
    Color.green.frame(height: 20.0)
    Color.yellow.frame(height: 20.0)
    Color.orange.frame(height: 20.0)
    Color.red.frame(height: 20.0)
    Color.purple.frame(height: 20.0)
    Color.blue.frame(height: 20.0)
  }
  .drawingGroup()
  .frame(width: 200.0)
  .distortionEffect(
    ShaderLibrary.default.distortionEffect(
			.float4(0.0, 0.0, 200.0, 120.0),
      .float(0.0)
    ),
    maxSampleOffset: CGSize(width: 200.0, height: 200.0)
  )
}

We run it and see that now we have not just a rectangle, but its distorted representation. It’s a huge success!

It’s time to use the previously defined parameter t and make this flag move.

Let’s add a couple of new values to SocketShaderScreen: time will store the countdown for the animation, and timer will periodically call a closure to update the counter.

struct SocketShaderScreen: View {
  @State var time: Float = .zero
  let timer = Timer.publish(every: 0.05, on: .main, in: .common).autoconnect()
  
  ... 
}

On VStack we'll put a modifier onReceive to track events from the timer, and internally we'll update the time value.

var body: some View {
  VStack { ... }
  .onReceive(timer) { _ in
    time += 0.1
  }
}

Now all that remains is to pass the value of time instead of 0.0 to t.

.distortionEffect(
  ShaderLibrary.default.distortionEffect(
    .float4(0.0, 0.0, 200.0, 120.0),
    .float(time)
  ),
  maxSampleOffset: CGSize(width: 200.0, height: 200.0)
)

Done, now our shadered view moves too!

Animated flag with distortion effect applied

💡 You can also create the animation with TimelineView. But for the sake of brevity, we have opted for the Timer solution. An alternative solution is offered as an exercise.

Raising the stakes

Vapor

The original tweet mentions remote compilation of the shader. Apple documentation describes the use of terminal commands for this purpose. To make this all work, we need to write a small backend. As true iOS developers, we will do this in Swift using the Vapor framework.

💡 Before proceeding — go through the manual on how to install Vapor on your device, if you don’t already have it. At the time of writing, the actual version of Vapor is 4.92.6.

Vapor: Install → macOS

Vapors documentation (web framework for Swift).

docs.vapor.codes

We need the most basic project. Open the terminal and run the following command.

vapor new shader-backend -n

Open the project. We are interested in the routes.swift file, which describes the available endpoints for communicating with the server. This is where we will spend most of our time creating the logic for compiling the shader.

The communication between the client and the server will be established via WebSocket protocol. The principle is simple — we send to the server the text of the file and receive in response binary data for .metallib. For such purposes this protocol is ideal for us.

Let’s declare a new endpoint. In the routes method, specify that the WebSocket connection will be initiated when accessing the host without additional paths. In the closure of the webSocket call, the req parameter represents the request that initiates the connection, ws is the communication channel for the WebSocket connection.

func routes(_ app: Application) throws {
    app.webSocket() { req, ws in }
}

Add a closure handler for the text that the client will send to the server.

app.webSocket() { req, ws in
    ws.onText { ws, text in }
}

Put the code aside for a moment. We need to familiarise ourselves with the principles of shader compilation.

Run, Xcode, run!

Xcode provides a CLI utility called xcrun, which allows us to do many different things. For example, it can be used to build instances of .xcframework. We will focus on the Metal processing part of its API.

When the backend receives text from the client, it builds a .metal file from it. This will be the starting point. Let's take the name of the file as shader.metal and make its content the code we used in distortion.metal earlier.

The first step is to generate an intermediate representation. Open the terminal and type the following command.

xcrun -sdk iphonesimulator metal -c -o shader.ir shader.metal

Here the sdk parameter is responsible for what platform the shader will be built for, o is the name of the final file. Since we do all testing on the simulator, we specify iphonesimulator as value. If you are running on a device - use iphoneos.

💡 The value for the sdk parameter can be passed as one of the parameters to the payload from the client.

The intermediate representation further needs to be converted to .metallib, which is the basis for the ShaderLibrary.

xcrun -sdk iphonesimulator metallib -o shader.metallib shader.ir

To verify that the final shader.metallib file works and valid, we can add it to the client project and replace ShaderLibrary.default with the following construct.

ShaderLibrary(
  url: Bundle.main.url(forResource: "shader", withExtension: "metallib")!
)

💡 The use of force unwrap here is purely for brevity. Please try to write safe constructions for unwrapping an optional value when working with production code.

Rolling jobs

Before we get back to writing endpoints, let’s dive into one more topic.

To build the .metallib, we needed to interact with the terminal. To run these same commands, but from Swift code, we will use the Process type. This type represents a program that can run inside another program - exactly what happens when we invoke commands in the terminal.

First, we need to specify which program to use for the call. We create an instance of Process and assign the path to xcrun to its executableURL property within the system on which the backend is running (assuming it's running on the same Mac instance).

let task = Process()
task.executableURL = URL(fileURLWithPath: "/usr/bin/xcrun")

💡 If you are not sure about the location of the programme, run the command which xcrun in the terminal - it will display the path to xcrun.

The next step is to specify the working directory for the process. This is where we will perform all manipulations with the file. Specify here the directory you are comfortable with.

task.currentDirectoryPath = ...

Next, we specify the list of parameters to call. The parameters are listed as an array of strings. For example, here we list the parameters for creating an intermediate representation of a .metal file.

task.args = [
  "-sdk",
  "iphonesimulator",
  "metal",
  "-c",
  "-o",
  "shader.ir",
  "shader.metal"
]

The last step is to run the process and wait for the result. To do this, we combine the run and waitUntilExit calls respectively.

try task.run()
task.waitUntilExit()

After that the process can be considered started and executed. For convenience, we will put the described calls into a separate method, into which we will pass the parameters of the command call.

func shell(_ args: [String]) throws {
  let task = Process()
  task.executableURL = URL(fileURLWithPath: "/usr/bin/xcrun")
  task.currentDirectoryPath = ...
  task.arguments = args
  try task.run()
  task.waitUntilExit()
}

Let’s get back to the creation of the initial shader.metal file. For these purposes we use FileManager: in onText add logic for processing the sent text.

We convert the text into Data to then feed it to FileManager. The file path in this call must match the directory where we run Process instance. At the end of this path we just add the file name shader.metal.

ws.onText { ws, text in
  guard let data = text.data(using: .utf8) else { return }
  FileManager.default.createFile(
    atPath: "/path/to/shader.metal", 
    contents: data, 
    attributes: nil
  )
}

💡 For brevity, it is assumed that all intermediate folders within this path exist. If not, then FIleManager fails to create a new file.

The next step is to make successive calls to xcrun, which we did earlier in the terminal.

ws.onText { ws, text in
  ...
	
  try? shell([
    "-sdk",
    "iphonesimulator",
    "metal",
    "-c",
    "-o",
    "shader.ir",
    "shader.metal"
  ])

  try? shell([
    "-sdk",
    "iphonesimulator",
    "metallib",
    "-o",
    "shader.metallib",
    "shader.ir"
  ])
}

The last step is to read the final .metallib file. The path to the file is still the same through the directory where we run Process, but now shader.metallib is at the end of this path. The resulting Data instance is then sent over the ws connection by calling send.

ws.onText { ws, text in
  ...
  let url = URL(fileURLWithPath: "/path/to/shader.metallib")
  if let data = try? Data(contentsOf: url) {
     ws.send(data)
  }
}

💡 To be fair, it should be noted that error handling could be represented by sending corresponding data sets to the ws channel and parsing them on the client. The implementation of this logic is beyond the scope of this article.

All that remains is to start the server by running cmd + R. After startup, its local address will be displayed in the console - e.g. localhost:8080. We will use it later to create a connection from the client.

Client, are you still here?

We have already done so much work, the only thing left is to teach the client to communicate with the server.

Let’s go back to the client code and create a new type SocketShaderInteractor, which will manage requests to the backend and convert its responses into a shader library.

First of all, let’s define a shaderLibrary variable in it and assign a standard ShaderLibrary instance to it. Further we will assume that we have the original implementation of the shader.metal file in the project so that this default value will work.

import SwiftUI

@Observable
final class SocketShaderInteractor {
  var shaderLibrary: ShaderLibrary = .default
}

To create a WebSocket connection we will use URLSessionWebSocketTask, which can be created using URLSession. Let's define the necessary parameters and add the initiate method that will initialise the WebSocket connection. Initialise the url value with the address we got when we started the backend. Note that the connection scheme of URL is ws, which stands for WebSocket.

@Observable
final class SocketShaderInteractor {
  ...
  private let session = URLSession.shared
  private let url = URL(string: "ws://localhost:8080")!
  private var task: URLSessionWebSocketTask?
  
  func initiate() {
    task = session.webSocketTask(with: url)
    task?.resume()
  }
}

To listen to the events that the backend will send when creating a .metallib file, let's add a closure handler to the task. It passes an event as a parameter, it is of type Result<URLSessionWebSocketTask.Message, any Error>. By exposing its contents, we can extract an instance of Data, which according to our implementation is a .metallib file. We use it to build a ShaderLibrary, which will then be used on the screen.

task?.receive { [weak self] result in
  switch result {
    case .success(let value):
      if case let .data(data) = value {
        self?.shaderLibrary = ShaderLibrary(data: data)
      }
    default: break
  }
}

The peculiarity of working with URLSessionWebSocketTask is the following: after accepting the event the closure-handler receive gets nilled, so after accepting the event it is necessary to recreate the subscription. For convenience, we will put the calls into a separate method and call it after creating the shaderLibrary.

private func subscribe() {
  task?.receive { [weak self] result in
    switch result {
      case .success(let value):
        if case let .data(data) = value {
          self?.shaderLibrary = ShaderLibrary(data: data)
          self?.run()
        }
      default: break
    }
  }
}

We’ll also add a subscription call to initiate.

func initiate() {
  ...
  subscribe()
}

Before we get too far, let’s add a method to the interactor to send a message to the backend.

func send(text: String) async {
  try? await task?.send(.string(text))
}

Now let’s go back to SocketShaderScreen. First, let's replace the ShaderLibrary.default call with interactor.shaderLibrary so that the screen always has access to the actual library instance. In addition, add a task modifier to VStack, which starts connection initialisation in the interactor.

.task {
  interactor.initiate()
}

We also need to add tracking of changes to the input text and cause the content to be sent. It’s done by defining onChange modifier on VStack and calling interactor’s send method.

.onChange(of: rawCode) {
  Task {
    await interactor.send(text: rawCode)
  }
}

After that, all that’s left to do is run the backend, then the client, and experiment!

💡 This solution can be improved a bit by reducing the number of calls to the server through adding debounce to the changes tracking.

Conclusion

In this article we’ve done a lot of things: we created a shader and wrote a backend. We have made a whole service!

I love experiments like this, they allow one to try something new and look at familiar things from a whole new angle.

Thanks for reading and stay tuned 🙌