After a few Google searches and some failed attempts, I identified a couple of key tricks.

The first is about obtaining the avatar image for a given GitHub account. Although the URL is hard to predict, adding .png to the account page acts as redirection. For example, this url:

https://github.com/abey79.png

redirects to:

https://avatars.githubusercontent.com/u/49431240?v=4

Which corresponds to this image:

The second trick is about resizing, masking, and caching that image to generate the circular avatar. It turns out that the free and open-source service wsrv.nl does exactly this. You can check the documentation for details, but this URL does the trick:

https://wsrv.nl/?url=github.com/abey79.png?w=64&h=64&mask=circle&fit=cover&maxage=1w

Here is the corresponding image:

The parameters specify a scaling to 64x64 pixels with a tight fit, a circular mask, and a cache expiration of one week.

For the final markdown code, I further set the image size to 32x32 pixels for a nice HiDPI look:

[<img src="https://wsrv.nl/?url=github.com/abey79.png?w=64&h=64&mask=circle&fit=cover&maxage=1w" width="32" height="32" alt="abey79" />](https://github.com/abey79)

Here is how it looks (my blog is markdown-based too, so I literally copy/pasted the line above):

That’s it! You can see it in action in a changelog file or a GitHub release.

This article explores the topic of layers, their name, and how they can be used to control the AxiDraw plotting process.

SVG has layers?

Actually, no.

Instead, SVG has groups, which can be nested or omitted altogether, which is not exactly the same thing. For this reason, the Inkscape project has introduced an XML namespace named inkscape to support its layer feature. Amongst other thing, it defines the inkscape:groupmode="layer" group attribute, to mark a top-level group as layer, and the inskscape:label attribute, to assign a label—or name—to the layer:

<?xml version="1.0" encoding="utf-8" ?>
<svg xmlns="http://www.w3.org/2000/svg"
     xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
     height="10.0cm" width="10.0cm">
  <g inkscape:groupmode="layer" inkscape:label="my layer label" >
      <!-- first layer's content -->
  </g>
  <g inkscape:groupmode="layer" inkscape:label="my other layer" >
      <!-- second layer's content -->
  </g>
</svg>

The Inkscape extension has become the de facto standard for SVG layer definition and naming, and is used by both vpype and the AxiDraw software.

vpype has layers?

Yes.

Layers are first-class citizens in vpype, and, like any good citizen, they are primarily identified by a non-zero, positive integer identification number, in this case called “layer ID”. When loading an SVG, vpype try to construct its layer based on their Inkscape label, if any, their SVG group id (<g id="...">) if any, or sequentially otherwise. For example, when reading an Inkscape file with two layers named “layer 4” and “some layer 6”, vpype will assign layer IDs 4 and 6 to these layers¹. If the label and group ID lack numerical digits, then vpype will revert to using sequential IDs.

One of the easiest ways to visualise layers and their IDs is to use the show and the viewer’s layer menu:

Both the layer ID and the label, if any, is shown.

Many vpype commands operate on a layer-by-layer basis. They are known as layer processors. By default, they operate on all layers—for example translate 0 2cm translates the whole document by 2 cm vertically. For more control, the exact layer(s) on which the command should operate can be specified using the --layer LID option (or -l LID for short). For example, translate -l 3 0 2cm will translate only layer 3. Likewise, translate -l 3,4 0 2cm will translate both layer 3 and 4.

AxiDraw escape codes

The EMSL wiki contains a thorough description of the supported escape codes and their effect. Here is a summary of what can be achieved:

• By prefixing the layer name with a number in the range of 0 to 1000, layers can be grouped together and enabled/disabled by the AxiDraw software (this resembles, but is not to be confused with vpype’s layer IDs). For example, 10 my layer assigns the number 10 to the corresponding layer. With the “Layer” tab of the InkScape extension, it is possible to plot all layers prefixed with a given number.
• The pen down position can be adjusted by adding +H immediately followed by a number right after the layer number, e.g. 10+H55 my layer sets the pen down position to 55. This works without layer number as well, e.g. +H55 my layer.
• Likewise, the pen down speed can be adjusted with +S, e.g. 12+S30 my layer.
• Another possibility is to set a delay at the beginning of a layer using +D. This enables behaviours such as letting time for the ink to dry, or for a timelapse picture to be taken.
• A layer may be marked as “do not plot” by adding % at the beginning of the layer name. This could be used to add debug elements or decorations to an SVG that should not be physically plotted.
• Likewise, starting the layer name with ! will force the AxiDraw to pause, for example to allow for paper swapping.

Naming layers

Equipped with this understanding of how layers work, let’s use the name command to control the AxiDraw. For example, the drawing speed for layer 3 can be specified this way:

vpype  read multilayer.svg  name -l 3 "+S30 my layer label"  write output.svg 

vpype 1.13 introduced a new lid built-in expression variable which comes in handy in this context. It can be used to match the AxiDraw layer number with vpype’s layer ID:

vpype  read multilayer.svg  name -l 3 "%lid%+S30 my layer label"  write output.svg 

In this case, layer 3 is renamed “3+S30 my layer label”. The nice thing is that this also works when renaming multiple layers at once. For example:

vpype  read multilayer.svg  name -l 3,4 "%lid%+S30"  write output.svg 

Here, layer 3 and 4 are renamed “3+S30” and “4+S30”, respectively.

Empty layers

To achieve certain behaviours with the AxiDraw, one may need to insert any empty layer with a specific name. For example, a final empty layer named “+D1000” makes the AxiDraw pause for one second in pen-up position at the final drawing location, before moving the head back to the home position.

Interestingly, there is currently no explicit support in vpype to create empty layers. The layer manipulation commands (like lmove, lcopy, etc.) specifically remove layers they leave empty.

While preparing this article, I realised that one command could be used to create an empty layer: random. Its purpose is to generate some random lines for debugging purposes, but the number of lines it generate can luckily be controlled. So here is how to create an empty layer with ID 5:

vpype  [...]  random -l 5 -n 0  [...]

The -n 0 option tells random to generate 0 random lines, but it still creates the layer. This can be verified using the stat command:

vpype  random -l 5 -n 0  stat

This prints the following output:

========= Stats =========
Current page size: None
Layer 5
  Length: 0
  Pen-up length: 0.0
  Total length: 0.0
  Mean pen-up length: 0.0
  Median pen-up length: 0.0
  Path count: 0
  Segment count: 0
  Mean segment length: n/a
  Bounds: None
  Properties:
Totals
  Layer count: 1
  Length: 0.0
  Pen-up length: 0.0
  Total length: 0.0
  Path count: 0
  Segment count: 0
  Mean segment length: n/a
  Bounds: None
  Global properties:
=========================

Layer 5 exists, but is empty.

That about wraps up what can be said on vpype layers, their names, and how they can interact with the AxiDraw. I’d love to hear about your creative use of these features, in the comments or on the socials.

The main improvement is Python 3.11 support, but it does come with a few other bells and whistles, so let’s dive in.

Python 3.11 support

• Added support for Python 3.11 and dropped support for Python 3.8 (#581)

Known issue:

• As of PySide 6.4.2, a refresh issue arises on macOS when the viewer window is resized by a window manager (#603)

Python 3.11 was first released on October 22nd, 2022, and the current version is 3.11.2 as of this writing. So it was way overdue for vpype to support it.

As usual, adding support for a newer Python means dropping an older one—bye bye Python 3.8! 👋 That said, if you must use Python 3.8, vpype 1.12.1 remains available and has been a rather stable release.

In the process of adapting vpype for 3.11, I had to update PySide6 to the last release (6.4.2 as of this writing). Unfortunately, it introduces a refresh glitch on macOS when using a window manager:

I waited for a fix, but this is taking too long, so I opted to make a release with this known issue.

lid built-in expression variable

• Added the lid built-in expression variable for generator and layer processor commands (#605)

Many vpype commands operate on a layer-by-layer basis. They are known as layer processors. One example is the translate command. By default, it appears to run on all layers at once, but what actually happens is that it is run once for each layer. You can control on which layer(s) it should actually run with the --layer option.

Any expression passed to such a command is also evaluated once per layer. This enables per-layer customisation of the command behaviour. This feature introduces the lid built-in variable, which is set to the ID of the layer that’s being currently processed.

Here is an example for illustration:

$ vpype  read file_with_many_layers.svg  name "Layer %lid%"  write output.svg

Here, the name command renames all layers (--layer is not used). But since lid is set by vpype to the current layer ID, the output SVG has its layers labelled “Layer 1”, “Layer 2”, etc. (the exact numbers depend on the input SVG’s actual layer structure).

Layer ID determination

• Fixed a design issue with the read command where disjoint groups of digits in layer names could be used to determine layer IDs. Only the first contiguous group of digits is used now, so a layer named “01-layer1” has layer ID of 1 instead of 11 (#606)

Speaking of layer IDs, this fix slightly changes how vpype attributes layer IDs when reading SVGs using the read command. Previously, vpype would strip all non-digit characters from the layer name (or, lacking one, from the corresponding group ID), and use the resulting number as ID, if any. A layer named “01-blue3” would thus end up with ID 13, which is a rather unexpected result! The new behaviour consists of taking the first contiguous group of digit, if any, and use this as layer ID instead. Now, layer “01-blue3” has an ID of 1.

BTW, I’m working on a new article on layer names, and how they can be used to control the plotting process with the AxiDraw. So stay tuned for more on this! Done!

Wayland-related crash

• Fixed an issue on Wayland-based Linux distributions where using the viewer (e.g. with the show command) would crash (#607)

Qt, the GUI library behind PySide6, has an abstraction layer dedicated to the platform its running on. This is basically what enables Qt projects to seamlessly run on macOS, Linux, or Windows. There appears to be some technicalities involved with this when using OpenGL (as vpype does), which led the viewer to crash on Wayland-based Linux install. The workaround consisted of “forcing” Qt to use the xcb instead of wayland platform by setting an environment variable:

export QT_QPA_PLATFORM=xcb

With vpype 1.13, this is no longer needed as it automatically sets this environment variable when a Wayland-based Linux is detected.

I recently built an “Automatic #plotloop Machine”, named after the common social media hashtag for these animations made of individually plotted frames. This article is a thorough description of the project, covering both hardware and software aspects.

First, here is a video:

So, this is all very nice, but why does this article need to be more than six thousand five hundred word long?! Sure, the machine’s neat, but it’s not like everyone wants to make one, right?

Well, in the making of this project, I found myself using a number of tools and techniques which, I think, might be of more general interest—that is, also for someone without aspirations for maximally complex animation production methods.

Rather than the seasoned maker, who might more efficiently fill knowledge gaps with a few targeted Google queries, I wrote this article for the many people in the plotter space with an artistic rather than technical background. I’m hoping this article might provide some technical baggage to fuel their tinkering urge, which, being involved with plotters, they obviously have.

So, if any or all of the following topics sound like they might be useful, have a read!

• How to use a Raspberry Pi to automate things like taking pictures or controlling stepper motors?
• How to mate a stepper motor with LEGO Technic contraptions?
• How to efficiently control a RPi via ssh?
• How to efficiently control a RPi using a web framework?
• How to use vsketch to produce plot loops (automated or otherwise)?
• How to use doit to automate complex workflows?
• Etc.

This article is not a beginner tutorial either. A number of the topics covered here would, in tutorial form, warrant an even longer treatment by themselves. There should be enough information to understand how things work and understand the relevance of the tools I’ve used. Actually applying them in a project may, however, require some more focused reading. Likewise, the code is what I’d call “project quality”—it does the job, I’m not too ashamed of it (mostly), but shouldn’t be construed as top-notch, state-of-the-art, production-ready copy-paste material.

Overview

Here is how the setup looks like in on my mobile plotting station:

Here are the main components involved:

1. An AxiDraw SE/A3 from Evil Mad Scientist Laboratories, driven by a Raspberry Pi 4 not visible in the picture (it’s neatly installed in the lower section of the plotting station).
2. A Raspberry Pi 3 hooked to the High Quality Camera module and a EasyDriver stepper motor driver, held by a Manfrotto 241s Pump Cup and 244 Mini Friction Arm.
3. An 80-mm paper feeder contraption made of LEGO Technic and a stepper motor.
4. A Manfrotto ML840H On-Camera LED light, held by a couple of umbrella swivel adapters and a Phottix Multi Clamp.

Clearly, having a bunch of photography-related gear around is convenient for this kind of project! 😄

Here is a schematics view of the same setup:

The most salient aspect is the use of two Raspberry Pis. This is entirely unnecessary—a single RPi would be entirely sufficient. This arrangement happened to be more convenient for me because my plotting station already includes a RPi (with hostname axidraw.local) for my day-to-day use of the AxiDraw.

Missing from both pictures is my computer, which I use to generate the frames and controls the plotting process by sending orders to both RPis via Wi-Fi.

In the next sections, I will dig into the details of many hardware and software aspects of this setup, culminating with the doit script which orchestrates everything ranging from generating the frame data, creating a simulated animation, controlling both RPis for plotting and picture acquisition, and assembling the final GIF:

Hardware

Camera

The camera assembly includes a Raspberry Pi (with hostname campi.local), the camera module and its optics, as well as the stepper motor driver. The three boards are assembled together using custom-cut plexiglass plates and nylon spacers. The camera module is connected to the RPi’s camera interface using the provided 200mm ribbon cable.

I chose the High Quality Camera for the following reasons:

• It is made by the Raspberry Pi Foundation itself, so it has best-in-class software support out-of-the-box.
• With 12 megapixels and a large pixel size, it is one of the highest quality camera available for the Raspberry Pi.
• It uses interchangeable, C-mount lenses, which means that I can use a lens that’s best suited for this setup.

For the lens, I selected the 16mm telephoto. With its relatively narrow field of view, it minimises the distortions and can be placed high enough to leave space for the plotter to operate. With this lens, the positioning is mainly driven by the minimum focus distance, which is approximately 24cm, measured from the front-most part of the lens.

Here is a sample frame and the corresponding raw image as taken by the camera (high-res version):

Note that the image is rotated by 90° compared to the actual frame. It just happened to be easier to set up the camera this way. It will be automatically corrected before the final animation is assembled.

Stepper motor driver

The campi.local RPi is also in charge of driving the paper feeder’s stepper motor via an EasyDriver board. As their name imply, stepper motors divide their full rotation into a number of discreet, equal steps. This makes them very convenient for use cases where accuracy and reproducibility is important, like 3D printers, plotters, and… makeshift paper feeders. Stepper drivers generate the high power electrical signals needed to run the motor based on a simple GPIO inputs.

Here is a close-up of the wiring schematic:

The main inputs are STEP and DIR. The motor turns by one step for each STEP pulse (transition from 0 to 1), while the rotation direction is controlled by DIR.

The ENABLE input controls whether the motor outputs are powered or not. When they are, the driver actively keeps the motor in its position, which tends to heat both the driver and motor itself. As the feeder spends most its time waiting for the plotter to draw the frame, it is good idea to drive ENABLE only when actually running the feeder.

The MS1 and MS2 inputs control the so-called “micro-stepping” capability of the driver. This feature increases the accuracy of the motor by further dividing the motor’s physical steps into up to 16 micro-steps. For a low-precision paper feeder like mine, this is not useful and the wiring can be skipped¹.

One nice feature of the EasyDriver board is the ability to choose either 3.3V or 5V GPIO voltage. The default is 5V, but since the RPi uses 3.3V, I set the board to this voltage using a small solder bridge over the pads at the very bottom left of the board:

LEGO structure

My LEGO building skills aren’t anything to boast about—a skilled builder would likely do a much better job. Yet, my contraption turned out to work rather reliably thanks to a few key design decisions.

Most importantly, I wanted to leave the feeder “open” on the “top” side to minimise the chance of collision with the plotter or the pen during operations. The lower parallel structure is thus designed to maintain some rigidity between the paper roll part and the feed part. This actually worked much better that I anticipated!

These structures are really easy to build once your remember that 3² + 4² = 5²:

All my diagonals use this arrangement and provide rigidity to the structure (purely rectangular structures are prone to parallelogram deformation).

Using the stepper motor weight to put pressure on the feeding wheel was another successful design. This ensured practically no paper slippage.

Finally, remembering that the LEGO stud pitch is 8mm, a 10-stud distance is a good fit for 80mm paper rolls. These are easy to source in office supply shops.

Stepper-LEGO integration

One might wonder why I used a custom stepper motor instead of native LEGO motors to drive the feeder. The decision boils down to two issues with LEGO motors:

1. There are no stepper-type LEGO motors, so their accuracy is not great, nor is their power.
2. Interfacing LEGO motors to a RPi is more complicated than using a stepper motor (especially if you already have stepper drivers lying around).

I actually tried to use a LEGO Mindstorm motor initially. I used a BrickPi board to interface the motor to the RPi. It worked, but as noted above these motors are not ideal for the task and ended up not being precise and reliable enough.

An alternative to the BrickPi board would be to use the actual Mindstorm controller. This is not ideal though, as the controller is battery powered and must be tinkered with to accept external power. Also, the RPi must connect to it via Bluetooth, and a specific Python package must be used.

All in all, in my experience, using a stepper motor is just easier.

This begs the question of how to physically interface the stepper motor with LEGO Technic parts.

I used a 3D-printed adapter of my own design to interface the 5mm axle of my motor to Technic axles. There are many such designs available online for various shapes and sizes of motor axles.

It turns out that physically assembling the motor with LEGO Technic is easy enough when using the NEMA 17 form factor (one of the most commonly used by makers). The offset between their M3 attachments is 31mm. This is close enough to the 32mm offset between two 3-stud-apart Technic holes. Using conic screws easily takes up the remaining 1mm difference:

Crucially, this method keeps the motor axle properly aligned with the Technic grid.

Software

Setting up the Raspberry Pi for friction-less SSH

There is ample documentation available online on how to setup Raspberry Pis, which I won’t reproduce here. I’d like to focus instead on how to configure the RPi such as to minimise any friction when interacting with your RPi via SSH—either manually or automatically.

Here is what’s needed to achieve this:

1. Unless you are using wired Ethernet, your home Wi-Fi must be configured on the RPi (of course, duh).
2. SSH must be configured for key-based authentication. This removes the need to input a password when SSHing into your RPi, which is annoying with manual operations and precludes automation.
3. ZeroConf/Bonjour/Avahi is installed and active. This means that you can connect to the RPi using a URL in the form of hostname.local instead of using an explicit IP address, which is hard to remember and subject to change.
4. You computer must “know” which username to use when connecting to a given RPi.

For key-based authentication, you must first generate a public/private key pair on your computer (if you haven’t done so already). This is done using the following command:

ssh-keygen

You’ll need to answer a few questions, which can all be left as default. This will create two files in your home directory:

• The private key: ~/.ssh/id_rsa
• The public key: ~/.ssh/id_rsa.pub

(Actual naming may differ depending on the type of key generated.)

The idea is to provide your public key (this is important—not the private key!) to the RPi so that it can accept connections without requesting a password when the originator possesses the corresponding private key.

Most of the configuration, fortunately, is addressed by the official Raspberry Pi Imager, which I strongly suggest using. Everything can be set in the “Advanced Options” dialog, including the hostname, the SSH public key, and the Wi-Fi credentials:

Again, note that the content of your public key (~/.ssh/id_rsa.pub) should be pasted in the relevant field.

Finally, you need to tell your computer’s SSH that user pi (or whichever you chose) should be used by default when connecting to the given RPi. Create the ~/.ssh/config file if needed, and add the following content:

Host axidraw.local
    User pi

This tells your computer’s SSH to default to the username pi whenever it connects to axidraw.local.

If you already have a working RPi image, and you don’t want to recreate one from scratch, you can do the same configurations manually as follows:

• Check this article for Wi-Fi configuration.
• For key-based SSH authentication, create or edit the ~/.ssh/authorized_keys2 file on the RPi and add the content of your public key on a new line.
• Install ZeroConf/Bonjour/Avahi support with sudo apt-get install avahi-daemon.
• Change the hostname by running sudo raspi-config. The configuration is in the Network Options menu.

Here are the cool things that you can now do remotely without entering a password:

scp file.svg axidraw.local:          # copy a local file to your remote user directory
scp file.svg axidraw.local:files/    # copy a local file the remote ~/files/ directory
scp axidraw.local:files/file.svg ./  # copy a remote file locally

ssh axidraw.local     # log to your remote RPi
ssh axidraw.local ls  # list the files in your home directory

cat my_file.svg | ssh axidraw.local axicli -L 2 -d 37 -u 60 -N -m plot   # plot a file!

This last one is particularly nice. SSH allows you to pipe the output of a local command (here cat just outputs the contents of my_file.svg) into a remote command’s input. This is a very powerful combination that I use for this project.

Controlling the camera and the feeder

The campi.local Raspberry Pi has two missions: taking a picture of the frame after it is drawn, and run the motor to feed blank paper for the next frame. One of the easiest ways to make these functionalities remote controllable is to run a small HTTP server with two end-points, one for each task.

This is really easy to do with FastAPI (Flask would also work just as well). The corresponding code is available here, along with a requirements.txt file listing the required packages.

Taking pictures

For illustration, here is a shortened version with only the image acquisition part:

from fastapi import FastAPI
from fastapi.responses import FileResponse
from picamera2 import Picamera2

# create a FastAPI server
app = FastAPI()

# setup and start the Pi camera for still frame acquisition
picam2 = Picamera2()
still_config = picam2.create_still_configuration(controls={"ExposureValue": 0})
picam2.configure(still_config)
picam2.start()

# create a GET endpoint accepting a "ev" parameter and returning a JPG file
@app.get("/img/")
def get_picture(ev: int = 0):
    picam2.set_controls({"ExposureValue": ev})
    array = picam2.capture_file("/tmp/temp.jpg")
    return FileResponse("/tmp/temp.jpg", media_type="image/jpeg")

All it takes to run the server is to the following command:

uvicorn campi:app --host campi.local --port 8000

Uvicorn is a web server implementation fo Python uses FastAPI (in our case) to handle web requests. Here, campi:app tells Uvicorn to use the app object in the campi module (assuming our file is named campi.py).

Note in passing how having ZeroConf setup with the RPi means that, once again, we don’t need to mess with explicit IP addresses. (Here, specifying a --host other than the default localhost is required to allow another computer to access the server.)

With the server running, any other computer on the local network can acquire a photo using the following command:

curl -s -o /tmp/test.png campi.local:8000/img/?ev=1

A couple of learnings from the field:

• Like in the example above, I’m using an exposure value of +1 with the #plotloop machine. This is because the white paper tend to lead to underexposure. With the code above, this parameter doesn’t seem to have effect until after the second picture is taken. So, after starting the server, I always use the command above to have one picture taken with ev=1 so the camera is primed when the machine actually uses it.
• The server can run for a long time, taking multiple hundreds of pictures (the rotating earth loop is 280-frame long). So it’s pretty important that the /img/ end-point doesn’t leak any memory. Especially uncompressed-frame-sized buffers. I actually ran into this issue with the first implementation, which used a memory buffer instead of a temporary file. After messing with tracemalloc for a while to figure this out, it ended up being just easier to switch implementation. For my first loop with the machine, the leak would fill my RPi’s memory and crash every 20 images or so—it was painful process to go through the whole loop!

Moving the motor

I use another end-point to control the feeder motor. This time, it includes a mandatory parameter in the URL: the approximate number of centimetre of paper to feed. Here is how it looks in the code:

@app.get("/motor/{cm}")
def run_motor(cm: int):
	...

Using this end-point is as easy as:

curl -s campi.local:8000/motor/5

The implementation is really boring. The gist of controlling a stepper motor is to toggle a GPIO up and down as many times as needed—one per step. This could easily be done manually, but I’m using RpiMotorLib to reduce the code to a single line. The conversion from centimetre to step count depends on your motor (mine has 400 steps per full rotation, which is fairly standard) and the feeder wheel size. I have a magic number in the code that I tuned by trial-and-error.

It’s worth noting that the accuracy of the feeder mechanisms is not critical for this machine because the paper isn’t moved at all between the frame being drawn and the picture taken. The feeder has thus no impact on frame-to-frame alignment—only the plotter’s repeatability matters here (which is a non-issue with the AxiDraw ❤️).

Another couple of things I learned on the way:

• As mentioned earlier, the motor and the driver can generate quite a bit of heat and power consumption when active, even when not moving. That’s why I manually toggle the ENABLE pin in the motor end-point. That way the motor is kept unpowered most of the time, and only activated when it must be moved.
• With the Raspberry Pi, it’s important to properly shut down the GPIO sub-system when exiting your program. It avoids some errors and minimise the risk for the hardware². The proper way to do this with FastAPI is to implement a shutdown event handler:

  @app.on_event("shutdown")
  def shutdown_event():
      GPIO.cleanup()
  

Using vsketch for plot loops

Unsurprisingly, I’ve used vsketch to generate the animation frames. Before diving into the actual sketch I made for this article, I want to shortly digress on how to structure a sketch to produce SVGs suitable for plot loops.

Here is a simple example of a plot loop sketch:

import math
import vsketch

class PlotloopSketch(vsketch.SketchClass):
    frame_count = vsketch.Param(50)
    frame = vsketch.Param(0)

    def draw(self, vsk: vsketch.Vsketch) -> None:
        vsk.size("5x5cm", center=False)
        vsk.scale("cm")

        radius = 2
        vsk.circle(2.5, 2.5, radius=radius)
        angle = 360 / self.frame_count * self.frame
        vsk.circle(
            2.5 + radius * math.cos(math.radians(angle)),
            2.5 + radius * math.sin(math.radians(angle)),
            radius=0.1,
        )

    def finalize(self, vsk: vsketch.Vsketch) -> None:
        vsk.vpype("linemerge linesimplify reloop linesort")

This sketch (available in the vsketch repository) animates a small circle rotating around a larger circle³:

The first noteworthy aspect is its use of the two parameters: frame_count and frame. This makes it super easy to control the animation length and generate all the frames. For example, the frames for this GIF were generated with the following command:

vsk save plotloop -m -p frame_count 13 -p frame 0..12

(Note the use of -m to use all available CPU cores.)

Another key element is to use center=False in the initial vsk.size() call. Without this, vsketch would auto-centre every frame based on the geometry and this would result in the animation wobbling around. I actually made a similar mistake with one of my first plot loop⁴:

Last but not least, notice how the algorithm generates the exact same frame for frame = 0 and frame = frame_count⁵. This is obviously necessary to obtain a properly looping behaviour—but is easier said than done for all but trivial examples.

One way is to use periodic trigonometric functions such as sine and cosine like I did in the example above. This is also the approach I used for the rotating Earth loop.

Many generative artwork rely on Perlin noise. Some implementations may offer some kind of periodicity that could be used to generate a looping set of frames. Alternatively, periodicity can be achieved by sampling the noise field along cylindrical coordinates instead of on a cartesian grid.

An entire article could be written on this topic, and this one is long enough. Instead, I’ve added another example to the vsketch repository to illustrate this principle:

The rotating Earth sketch

The full sketch code is too long to be reproduced in this article—it’s available here in my sketches repository on GitHub. Instead, I will provide here an overview of how it’s implemented. You might want to open the code in another window to follow along.

Data preprocessing

First, the data. I used the World Country boundaries from ESRI. It contains polygons for all countries in the world. By merging them with Shapely’s unary_union(), one can obtain the land/water boundaries. This happens in the build_world() function.

Here is how the merged countries look after the union step:

There is one oddity I had to deal with, which explains the magic numbers and other ugliness in that function: Antarctica. This is the only body of land sitting over one of the poles, which are singularities in the latitude/longitude coordinate system.

The left is how the data looked like originally. On the right is the boundary once the artificial limit at ~80°S is removed. In lat/long coordinates, it becomes a self-intersecting polygon. This can be dealt with a simple Shapely trick: create a LinearRing with the (self-intersecting) boundary and apply unary_union() on it. This creates a MultiLineString containing a corresponding list of non-intersecting linear rings.

This is the glitch that this procedure solved:

Another important step is to filter land masses by area, to avoid the myriads of tiny isles that would clutter the result and massively increase plotting time. You can’t just use the .area attribute from Shapely with lat/lon coordinates as this isn’t an equal-area projection, so I shamelessly copy/pasted polygon_area() from StackOverflow.

The final preprocessing step consist of converting the lat/lon land boundaries (stored in a Shapely Polygon instance) into 3D points on the unit sphere (stored as a Nx3 NumPy array). This is done by the project_polygon() function. The projected boundaries are stored in the LINES global variable.

Rendering the Earth

The Earth rendering can be broken into the following steps:

1. Rotate the Earth data as needed.
2. Crop away the “far side” part of the data.
3. Project orthogonally the rest of the data, a.k.a. drop the coordinate along which the backside was cropped and use the other two for drawing.
4. Draw a circle :)

For rotation, I first compute 3 angles around the X, Y, and Z axes. These angles can be manually set for testing, or generated by trigonometric functions with different frequencies. Naturally, I make sure that these function are periodic with the frame count for a properly looping behaviour.

I then generate 3 rotation matrices:

rot_x = np.array(
    [
        (1, 0, 0),
        (0, math.cos(math.radians(rot_x_angle)), -math.sin(math.radians(rot_x_angle))),
        (0, math.sin(math.radians(rot_x_angle)), math.cos(math.radians(rot_x_angle))),
    ]
)
rot_y = np.array(
    [
        (math.cos(math.radians(rot_y_angle)), 0, math.sin(math.radians(rot_y_angle))),
        (0, 1, 0),
        (-math.sin(math.radians(rot_y_angle)), 0, math.cos(math.radians(rot_y_angle))),
    ]
)
rot_z = np.array(
    [
        (math.cos(math.radians(rot_z_angle)), -math.sin(math.radians(rot_z_angle)), 0),
        (math.sin(math.radians(rot_z_angle)), math.cos(math.radians(rot_z_angle)), 0),
        (0, 0, 1),
    ]
)

Finally, I combine them a single transformation matrix (did I mention I love NumPy?):

rot = rot_x @ rot_y @ rot_z

With that, rotating every single points of one of the land boundary line is just a matter of:

rotated_line = (rot @ line.T).T

Here, .T is used for transpose, and is needed for NumPy broadcasting rules to work. Remember that the actual calculation (3x3 matrix multiplication on every single points in line) happens in highly optimised C code, so this operation is fast.

Cropping is actually a bit trickier because you have to account for lines that may go from the front side to the far side and back, possibly multiple times. The cropping operation on a single line may thus result in multiple “sub-lines”.

Luckily, I had already sorted out most this for vpype’s crop command. vpype’s API includes the crop_half_plane() function, which crops a line at a give location along one of the X or Y axis⁶. Adapting it to the 3rd dimension was trivial. While copy/pasting the function, I took along _interpolate_crop(), which deals specifically with computing the intersection of a line segment with the cropping plane.

Once the land boundary lines are cropped along one dimension, it’s a matter of drawing them using the other two dimensions using vsk.polygons. Which dimension is used for what is not very important since we’re dealing with a sphere. I made it so that when rotation angles are set to 0, the (0°, 0°) lat/lon point (somewhere in the Atlantic ocean, off Ghana) is dead in the center of the rendered Earth.

In the sketch code, you’ll also find controls to enable pixelation using vpype-pixelart. I ran some trials with it but decided against it—to messy for this kind of line work.

Putting it all together with doit

At this point, all the #plotloop machine’s body parts are in place and just need a beating heart to set them in motion. doit is the perfect tool for this.

Although doit is rather easy to use, it still has a tiny bit of a learning curve. If this is your first ever encounter with it, you might want to check the introductory article I recently wrote. This project takes this to a whole new level.

Instead of looking at the dodo.py file line by line, I’ll first provide an overview of the workflows it implements (again, you might want to open the file in another window to follow along). Then, I’ll go deeper into a few, hand-picked topics to highlight interesting techniques.

The workflows

The bulk of the dodo.py file implements two workflows using a bunch of tasks: one is to create a simulated animation based on the frames' SVG, and another to plot, photograph, post-process, and assemble the frames into the final animation.

Here is a schematic of the workflows.

Let’s review the tasks involved in creating the simulated animation:

• The generate task generates all the frame SVGs with a single call to vsketch. It is basically running some version of this command:

  vsk save . -m -p frame_count 280 -p frame 1..280
  

  The outcome of this task is one SVG file per frame, numbered from 1 to 280.
• Each of the simframe:XXXX sub-task takes one frame SVG and convert it to a JPG with a _simulated prefix using librsvg’s rsvg-convert⁷. The sub-tasks are named after their corresponding frame number, e.g. simframe:0010 correspond to the frame number 10.
• Finally, the simulate task combines all the simulated frames into a single animated GIF, using ImageMagick’s convert command.

I’m calling this a workflow because each of these tasks have their file_dep and targets carefully defined. As a result, doit understands the dependency relationship between them. From a clean slate, calling doit simulate will first execute generate, then each of the simframe:XXXX sub-tasks, before finally running simulate to produce the animation.

The workflow for the actually plotted animation is similarly structured, but includes an additional post-processing step:

• The workflow starts with the same generate task as before.

• Each of the frame is then plotted and photographed by the corresponding plot:XXXX sub-task, which performs the following steps:

  • Plot the frame by sending the SVG via SSH to axicli running on the axidraw.local RPi (as described earlier).
  • Move the pen away by 3 inches to get it out of camera view (again, using SSH and axicli).
  • Take a picture of the frame and download the corresponding image using curl (as described earlier).
  • Move the pen back to its original position.
  • Feed fresh paper using curl (as described earlier).

  This is a good example of how a single doit task may execute multiple CLI commands.

• Each frame is then post-processed by the corresponding postprocess:XXXX sub-task using convert. It rotates the image in the correct orientation, crops it tightly around the earth, converts it to grayscale, and bumps its brightness and contrast a bit. This is the command used:

  convert frame_XXXX_plotted.jpg -rotate 270 -crop 1900x1900+605+785 \
    -colorspace Gray -brightness-contrast 5x15 frame_XXXX_postprocessed.jpg
  

• Finally, the animation task combines all post-processed frames into the final animated GIF using convert.

One may wonder, why is the postprocess:XXXX task separate from the plot:XXXX task? The convert command could just as well be added to the list of commands plot:XXXX executes. The answer is to be able to tweak the post-processing step without invalidating the plotting process. If both tasks were merged, any modification to the post-processing (e.g. adjusting the cropping parameters) would require re-plotting the entire frame—a lengthy process! This issue disappears with a stand-alone postprocess:XXXX task, which is very powerful when fine-tuning the workflow.

Basic doit syntax

Armed with this dodo.py file, we are now in complete control of our workflow.

Here are a few example commands (I won’t list all the output here, check my other article for a gentler introduction on how it behaves).

First, doit always provides a list of available tasks with doit list:

animation     Make the animation.
disable_xy    Disable X/Y motors
generate      Generates golden master SVGs from the list of input files.
plot          Plot the plotter-ready SVGs.
postprocess   Post-process the plotted images.
shutdown      Shutdown everything
simframe      Simulate a frame.
simulate      Make the simulated animation.
toggle        Toggle pen up/down

Generating the frame SVGs is a matter of:

doit generate

This is not needed though, as this task is automatically run when executing other tasks depending on it. For example, this executes generate (if needed) and all the simframe:XXXX sub-tasks to produce the simulated GIF:

doit simulate

A single simulated frame can be generated by specifying the frame number:

doit simframe:0118

This works because I chose to name sub-tasks after the corresponding (zero-padded) frame number.

All the frames can be generated at once by omitting the sub-task name:

doit simframe

Likewise, producing the final, plotted animation is just a matter of running the following command and waiting 9 hours 🕰:

doit animation

Executing a range of sub-tasks

It is often useful to execute a range of sub-tasks. For example, early testing requires plotting, say, the first 10 frames to verify that everything works correctly. (Spoiler alert: it doesn’t! The process must be repeated several times until all the glitches are worked out.)

Thankfully, this is made very easy thanks to bash’s brace expansion syntax (it works the same with zsh and, probably, other shells⁸).

Here is an illustration to demonstrate the idea:

echo {1..5}

The braces with the .. syntax are interpreted by bash as a range that needs expansion. Accordingly, the output of the above is:

1 2 3 4 5

The good news is that it understands zero-padding:

echo {0001..0015}

This produces:

0001 0002 0003 0004 0005 0006 0007 0008 0009 0010 0011 0012 0013 0014 0015

Knowing this, we can instruct doit to plot a specific frame range with the following command:

doit plot:{0005..0015}

This expends to doit plot:0005 plot:0006 plot:0007 ..., which doit interprets as a list of tasks to be executed.

This syntax is beautifully consistent with both the single task form (doit plot:0012) and vsketch’s -p,--param option (vsk save -p frame 1..280). It’s also yet another shining example of how powerful terminals can be for automation.

Note that, again, this is enabled by my choice of consistently naming sub-tasks after their zero-padded frame number.

Path management

This dodo.py file wrangles with a lot of different files. Each frame has up to 4 corresponding files (the original SVG, the simulated JPG, the plotted JPG, and the post-processed JPG), each with a specific suffix.

A small helper class is useful to manage this complexity. Here is how it looks:

import pathlib

PROJECT_DIR = pathlib.Path(__file__).parent
FRAME_COUNT = 280
PIXELIZE = False
PROJECT_NAME = "world"
BASENAME = f"{PROJECT_NAME}_frame_count_{FRAME_COUNT}_pixelize_{PIXELIZE}"

class FileSpec:
    def __init__(self, frame: int):
        self.frame = frame
        directory = PROJECT_DIR / "output"

        # vsketch doesn't add zero padding to frame number
        self.source = directory / (BASENAME + f"_frame_{self.frame}.svg")

        # for the other file we add the zero padding to keep the order with CLI tools
        base_frame = BASENAME + f"_frame_{self.frame:04d}"
        self.simulated = directory / (base_frame + "_simulated.jpg")
        self.plotted = directory / (base_frame + "_plotted.jpg")
        self.postprocessed = directory / (base_frame + "_postprocessed.jpg")


FILE_SPECS = {i: FileSpec(i) for i in range(1, FRAME_COUNT + 1)}

FileSpec instances are created based on a frame number, and contain all the paths related to the corresponding frame. This is easy to implement using the pathlib module from the standard library.

Note that vsketch doesn’t zero-pad the frame number in the SVGs it produces. This is a bit unfortunate as it messes up alphabetical ordering for most CLI tools (for example, this creates frame ordering issues when creating a GIF from a bunch of JPGs). All the other paths are constructed with zero-padding.

Also notice the PIXELIZE global variable. Its value is forwarded as a parameter to vsketch, which controls whether vpype-pixelize should be used. As noted above, I’ve decided against it for this project. In any case, I made sure to reflect this value in all the files' base name to avoid confusion by doit when changing the parameter’s value.

The FILE_SPECS global variable contains a dictionary which maps the FileSpec instance to the corresponding frame number. This simplifies a lot the task implementation.

Task dependencies

As I explained in my previous article, proper handling of task targets and dependencies is key for doit to understand the structure of the workflows and be smart about which task must be executed when.

Let’s consider the simframe:XXXX sub-tasks as an example:

def task_simframe():
    """Simulate a frame."""
    for frame, spec in FILE_SPECS.items():
        yield {
            "name": f"{frame:04d}",
            "actions": [
                f"rsvg-convert -b white -h 200 {spec.source} > {spec.simulated}",
            ],
            "file_dep": [spec.source],
            "targets": [spec.simulated],
            "clean": True,
        }

Each sub-task has the source SVG file set as file_dep, and the simulated JPG file as targets. Since this stage of the workflow is structured as sub-tasks—one sub-task per frame—each sub-task focuses on a single input and output file.

In contrast, the simulate task needs all the simulated frames to create a single animated GIF:

def task_simulate():
    """Make the simulated animation."""
    file_list = [spec.simulated for spec in FILE_SPECS.values()]
    paths = " ".join(str(file) for file in file_list)
    target = PROJECT_DIR / "output" / f"{BASENAME}.gif"
    return {
        "actions": [f"convert -delay 5 -loop 0 {paths} {target}"],
        "file_dep": file_list,
        "targets": [target],
        "clean": True,
    }

In this case, file_dep contains all the simulated frames, while targets has the animated GIF.

These snippets also illustrate how the FileSpec class and the FILE_SPECS global variable simplify the task implementation.

Depending on dodo.py

Adding the dodo.py file to the dependency list might sound like a smart idea. Indeed, a modification of the dodo.py may potentially invalidate all generated files.

In our case, this can be done in the generate task, on which all other tasks depend:

def task_generate():
    """Generates golden master SVGs from the list of input files."""
    return {
        "actions": [
            (
                f"vsk save -n {PROJECT_NAME} -p frame_count {FRAME_COUNT} "
                f"-p pixelize {PIXELIZE} -p frame 1..{FRAME_COUNT} -m ."
            )
        ],
        "file_dep": [PROJECT_DIR / f"sketch_{PROJECT_NAME}.py", __file__],  # <-- LOOK HERE
        "targets": list(spec.source for spec in FILE_SPECS.values()),
        "clean": True,
    }

The path of the dodo.py file is conveniently stored in the __file__ global variable by the Python interpreter, so it is a matter of adding it to the file_dep.

(Notice, in passing, that the sketch file is also listed as file_dep. This triggers a complete rebuild whenever the sketch file is modified.)

This technique is useful in the beginning to ensure that intermediate files potentially contaminated by bugs of an early, in-construction dodo.py file are properly rebuilt when the bug is fixed. It can, however, become an annoyance later on. Executing all of the plot:XXXX sub-tasks takes about 9 hours in total. You really don’t want to repeat the whole thing just because of a code formatting fix or added comment in dodo.py! As such, the dodo.py is not listed as dependency in the file on GitHub.

Helper tasks

In addition to the main workflows described so far, the dodo.py file includes a few helper tasks to toggle the pen up/down, disable the motor power (which is always better when the keeping the AxiDraw powered but unused for long periods of time), and shutdown everything (the RPis and the AxiDraw).

The code is really straightforward, but very handy to have around. It’s basically a thin wrapper over axicli functionalities:

def task_toggle():
    """Toggle pen up/down"""
    return {"actions": [f"{AXICLI} -m toggle"]}


def task_disable_xy():
    """Disable X/Y motors"""
    return {"actions": [f"{AXICLI} -m manual -M disable_xy"]}


def task_shutdown():
    """Shutdown everything"""
    return {
        "actions": [
            "ssh campi.local sudo poweroff",
            "ssh axidraw.local sudo poweroff",
        ],
        "task_dep": ["disable_xy"],
    }

Notice the use of task_dep in the shutdown task. It basically says that disable_xy should be executed whenever shutdown is called.

Final words

And that wraps up what I wanted to cover about this project. This was waaay longer than I anticipated! I hope you enjoyed it and, possibly, learned something.

I must once again credit Simon Willison for the inspiration and his exhortation to blog about projects. Here, I took the advice, and pushed it to the extreme! 😅 I don’t expect to be as thorough every time, but will certainly continue to produce some coverage my future endeavours.

Please hit me up in the comments or on social medias with any feedback you may have—or just to let me know you made it this far 🏆, it’ll make my day! ❤️

Most plotter workflows involve one or more repetitive steps which, when executed manually, take time, are boring, and possibly error-prone. Here are some examples that come to mind:

• Optimizing SVGs using vpype’s linemerge reloop linesort linesimplify commands.
• Converting SVGs into a format your plotter understands (e.g. HPGL, or G-code using vpype-gcode).
• Splitting multi-layer SVGs into individual layers (e.g. if this is a requirement of your plotter for multi-colour plots).
• Making a PNG version of SVGs for archival purposes.
• Running the axicli command to plot an SVG with an Axidraw.
• Uploading optimised files to the computer/server/Raspberry Pi in control of your plotter.
• Etc.

Not only your workflow may include one or more of these steps, but you may need to apply it on a single SVG at a time, or on a bunch of them at once. Even better, you might want to apply your workflow only on SVGs which were updated or created since the last execution.

You can do exactly that with doit—let’s see how.

Installing doit

Although its documentation sadly doesn’t mention it, pipx is the best way to install doit (as for vpype):

$ pipx install doit

You can check that the installation was successful by running this command:

$ doit --version
0.36.0
lib @ /Users/<username>/.local/pipx/venvs/doit/lib/python3.10/site-packages/doit

Basics

As a starting point, let’s assume you have a bunch of SVGs which need optimising before plotting, stored in a originals subdirectory. Save the optimisation commands in a VPY file named optimize.vpy, with the following content:

linemerge reloop linesort linesimplify

Then, create a subdirectory named processed, which will contain the optimised SVGs:

$ mkdir processed 

Here is how your file hierarchy should look like:

.
├── optimize.vpy
├── originals/
│   ├── dots.svg
│   ├── halftone.svg
│   └── hline.svg
└── processed/

Our goal is to have doit automate the optimisation of the source SVGs in originals, and store the result in processed.

doit operates by loading a description of the task(s) it must execute, typically in a file named dodo.py¹. As the name suggests, the content of this file is Python code.

Create a dodo.py file with the following content:

import pathlib                                                            # (1)

DIR = pathlib.Path(__file__).parent                                       # (2)
SOURCES = list((DIR / "originals").glob("*.svg"))                         # (3)
VPY = DIR / "optimize.vpy"                                                # (4)

def task_optimize():                                                      # (5)
    """optimize SVGs"""                                                   # (6)
    for source in SOURCES:                                                # (7)
        optimized = DIR / "processed" / (source.stem + "_optimized.svg")  # (8)
        yield {                                                           # (9)
            "name": source.stem,                                          # (10)
            "actions": [
                f"vpype read '{source}' -I '{VPY}' write '{optimized}'"   # (11)
            ],
        }

Let’s examine this code line-by-line.

1. The pathlib built-in module is great at file wrangling. Check this Real Python article for a gentle yet thorough introduction.
2. Here we use it to find our project directory, which is the parent of the present file, whose path is stored in the __file__ variable by the Python interpreter.
3. We list all the SVGs contained in the originals subdirectory, and store them in the SOURCES variable. Note that glob() returns a generator, which must be converted to a list if SOURCES is to be iterated multiple times.
4. We keep the path to the optimize.vpy file in the VPY variable.
5. Python functions with name starting with task_ are interpreted by doit as tasks. Here we have just one. Let’s call it “optimize”, thus the task_optimize() function name.
6. The function’s docstring is used by doit as help string for the task, so it is useful to include one.
7. Task functions must return one or more Python dictionaries describing the task. In our case, we want to create one sub-tasks per source SVG file.
8. For each source SVG, we derive the path for the corresponding optimised SVG. The optimised SVG are located in the processed subdirectory and have a _optimized.svg suffix to their name.
9. Using yield keyword (instead of return) makes our function a generator (gentle introduction available here). This is a convenient way to return (er… yield) multiple objects, which is supported by doit. Here, we yield one dictionary per sub-task.
10. Sub-tasks must be individually named so that they can be distinguished. Here we derive the sub-task name from the source SVG filename. For example, the sub-task corresponding to my_file.svg will be named my_file, and can be referred to with doit as optimize:my_file.
11. Last but not least, the "actions" entry of the sub-task dictionary lists the actions to be performed by the task. doit interprets strings as shell commands, so we build a vpype pipeline to optimise the source SVG using our VPY and saving the result in the desired location. For example, for my_file.svg, the action will be vpype read originals/my_file.svg -I optimize.vpy write processed/my_file_optimized.svg².

Let’s take a step back to properly understand what’s going on.

The function task_optimize() produces a task description—it does not actually run the task. When we run doit (using the doit command), it loads the dodo.py file, notices that it contains a task function, and calls it to learn about that task. It’s only then that it can decide which action(s) to actually execute, based on the task description. In this case, the actions are the vpype pipelines stored in the "actions" entries.

Although this dodo.py file is not overly complicated, it can still feel like quite some work compared to, you know, just calling vpype manually. I certainly felt so when first using doit. So let’s see what we gained by going through this effort.

First and foremost, we now have a potent batch processing system. We can optimise all of our source SVGs by telling doit to execute the optimize task:

$ doit optimize
.  optimize:dots
.  optimize:halftone
.  optimize:hline

Here is the result after running this command:

.
├── dodo.py
├── optimize.vpy
├── originals/
│   ├── dots.svg
│   ├── halftone.svg
│   └── hline.svg
└── processed/
    ├── dots_optimized.svg
    ├── halftone_optimized.svg
    └── hline_optimized.svg

doit indeed created properly-named, optimised versions of the source SVGs in the processed directory! 🎉

Since we only have just one task defined, we don’t even need to specify its name:

$ doit
.  optimize:dots
.  optimize:halftone
.  optimize:hline

You can also specify a specific sub-task to execute:

$ doit optimize:halftone
.  optimize:halftone

Pretty neat already—but there is a lot more to gain with a little more effort!

Handling targets and dependencies

Playing with the commands above, you may notice that each call of the optimize task triggers the processing of the corresponding SVGs—even if said SVGs were already processed before. The reason for this is that doit doesn’t yet know what the task inputs and outputs are, so it cannot check whether that output exists or is outdated. So, to be on the safe side, it always executes all specified tasks every time.

By letting doit know about tasks' inputs and outputs, doit can be much smarter about what it actually needs to do.

In doit parlance, the file(s) a task uses as input are called dependencies ("file_dep" entry). Likewise, the file(s) created as output are called targets ("targets" entry). By specifying what these are in the dodo.py file, doit can decide whether the target of a given task needs to be generated or not, saving a lot of time when repeating the workflow.

Update the dodo.py file as follows:

import pathlib

DIR = pathlib.Path(__file__).parent
SOURCES = list((DIR / "originals").glob("*.svg"))
VPY = DIR / "optimize.vpy"

def task_optimize():
    """optimize SVGs"""
    for source in SOURCES:
        optimized = DIR / "processed" / (source.stem + "_optimized.svg")
        yield {
            "name": source.stem,
            "actions": [
                f"vpype read '{source}' -I '{VPY}' write '{optimized}'"
            ],
            "targets": [optimized],         # (1)
            "file_dep": [source, VPY],      # (2)
        }

1. The "targets" entry is a list of all the files generated by the sub-task. In our case, there is only one, whose path is stored in the optimized variable.
2. The "file_dep" entry is a list of all the files the sub-task depends on. In our case, both the source SVG and the VPY file are involved to create an optimised SVG, so we list them both.

It would be easy to forget the VPY file in the "file_dep" entry. That would be a mistake. All the optimised SVGs should be regenerated when the VPY file is modified. For doit to realise this, we must list the VPY file as a dependency.

With the modification above, doit now knows when to run optimisation sub-tasks and when they can be skipped.

Let’s experiment with a clean slate by deleting all the processed files:

$ rm processed/*.svg

doit must now execute all sub-tasks:

$ doit
.  optimize:dots
.  optimize:halftone
.  optimize:hline

Notice the dot (.) prefixing each line and how the execution is relatively slow.

Now, this is what happens if we run doit again:

$ doit
-- optimize:dots
-- optimize:halftone
-- optimize:hline

Execution time is now much faster and each line is now prefixed with --, indicating that doit skipped the corresponding sub-task.

Let’s see what happens if one of the source file is modified.

$ echo " " >> originals/halftone.svg
$ doit
-- optimize:dots
.  optimize:halftone
-- optimize:hline

We first append a single space to the halftone.svg (which is harmless on a valid SVG) to simulate a change³. As expected, doit rebuilds the of halftone.svg without running the other tasks! 🎉

We now have a setup able to automatically process large batches of files and be smart about if/when any sub-task must be repeated. You have a thousand SVGs to process? It’s coffee time while the CPUs churn through them⁴. You add just one to the list? Instant results, thanks to doit!

Cleaning up

The files created by the optimize task can be considered “temporary”. When missing, they are automatically recreated by doit, and are overwritten by a new version when the input file (or the VPY file) change. In that sense, they matter much less than the source SVGs and the dodo.py file, which collectively form the “recipe” to build the optimised SVGs⁵.

The ability to delete these files may occasionally be useful. For example, to force a complete rebuild of the optimised files, to make an archive with only the true source files, or simply to free some disk space.

doit provides this feature with a single modification to the dodo.py file:

import pathlib

DIR = pathlib.Path(__file__).parent
SOURCES = list((DIR / "originals").glob("*.svg"))
VPY = DIR / "optimize.vpy"

def task_optimize():
    """optimize SVGs"""
    for source in SOURCES:
        optimized = DIR / "processed" / (source.stem + "_optimized.svg")
        yield {
            "name": source.stem,
            "actions": [
                f"vpype read '{source}' -I '{VPY}' write '{optimized}'"
            ],
            "targets": [optimized],
            "file_dep": [source, VPY],
            "clean": True,                  # (1)
        }

1. Tell doit that target files should be deleted when running doit clean.

Let’s see this in action:

$ doit clean
optimize:hline - removing file '.../processed/hline_optimized.svg'
optimize:halftone - removing file '.../processed/halftone_optimized.svg'
optimize:dots - removing file '.../processed/dots_optimized.svg'

Works as expected! 🎉

Multiple tasks

Although doit already shines dealing with a single task, it reveals its true power when multiple tasks are involved—even more so when they depend on each other.

For the illustration purposes, let’s imagine that we need to convert the optimised SVGs to HPGL, so that we may plot them on a shiny ‘83 HP 7475a. We’ll add a second task for this⁶.

First, let’s start by creating a new hpgl subdirectory to store the HPGL files:

$ mkdir hpgl

Since we cleaned the optimised SVGs in the previous steps, this how your project directory should look:

.
├── dodo.py
├── hpgl/
├── optimize.vpy
├── originals/
│   ├── dots.svg
│   ├── halftone.svg
│   └── hline.svg
└── processed/

Now, update the dodo.py file with the following content:

import pathlib

DIR = pathlib.Path(__file__).parent
SOURCES = list((DIR / "originals").glob("*.svg"))
VPY = DIR / "optimize.vpy"

def optimized_path(source: pathlib.Path):                              # (1)
    """derive optimized path from source path"""
    return DIR / "processed" / (source.stem + "_optimized.svg")

def hpgl_path(source: pathlib.Path):                                   # (2)
    """derive HPGL path from source path"""
    return DIR / "hpgl" / (source.stem + ".hpgl")

def task_optimize():
    """optimize SVGs"""
    for source in SOURCES:
        optimized = optimized_path(source)                             # (3)
        yield {
            "name": source.stem,
            "actions": [
                f"vpype read '{source}' -I '{VPY}' write '{optimized}'"
            ],
            "file_dep": [source, VPY],
            "targets": [optimized],
            "clean": True,
        }

def task_hpgl():
    """convert to HPGL"""
    for source in SOURCES:                                             # (4)
        optimized = optimized_path(source)                             # (5)
        hpgl = hpgl_path(source)
        yield {
            "name": source.stem,
            "actions": [
                f"vpype read '{optimized}' write -d hp7475a -p a4 -q -c '{hpgl}'"
            ],
            "file_dep": [optimized],                                   # (6)
            "targets": [hpgl],                                         # (7)
            "clean": True,
        }

Let’s examine the changes one-by-one.

1. To clean things up and avoid code duplication, we factored in optimized_path() the code to derive the path of an optimised SVG from a source SVG.
2. We do the same to derive the path of an HPGL output from a source SVG in the hpgl_path() function. Note that neither of these function names start with task_, so they aren’t interpreted as tasks by doit.
3. The only change to the optimize task is to use the optimized_path() helper function.
4. This part is interesting. The purpose of the hpgl task is to convert optimised SVG into HPGL files, yet we iterate over the source SVGs instead. The reason is, for our purposes, SOURCES is our master “TODO list”. Everything the hpgl task must do is indirectly due to the presence of source SVGs.
5. The source path is used only to derive the paths for the optimised SVG as well as the HPGL output. In particular, notice how source is not used anywhere in the return dictionaries.
6. The optimised SVGs is now a dependency (as opposed to a target in the optimize task).
7. Instead, the target is the HPGL file.

These two tasks collectively form a “pipeline”. The output (or target) of the first task corresponds to the input (or dependency) of the second. doit understands that thanks to the "file_dep" and "targets" entries being properly populated—and can now be smart about it!

Let’s take it for a spin by executing the hpgl task:

$ doit hpgl
.  optimize:dots
.  optimize:halftone
.  optimize:hline
.  hpgl:dots
.  hpgl:halftone
.  hpgl:hline

doit knows that it needs optimised SVGs to create HPGL file, so it automatically executes the optimize task.

Let’s remove a single HPGL file to test what happens. This can be done using the doit clean command:

$ doit clean hpgl:hline
hpgl:hline - removing file '.../hpgl/hline.hpgl'

This is what happens when we run the hpgl task again:

$ doit hpgl
-- optimize:dots
-- optimize:halftone
-- optimize:hline
-- hpgl:dots
-- hpgl:halftone
.  hpgl:hline

The optimised version of hline.svg is still present and up-to-date, so the corresponding task is skipped. Only the HPGL conversion is executed.

Now, let’s change one of the source files, like we did earlier:

$ echo " " >> originals/dots.svg  
$ doit hpgl
.  optimize:dots
-- optimize:halftone
-- optimize:hline
.  hpgl:dots
-- hpgl:halftone
-- hpgl:hline

doit correctly runs both the optimize and hpgl sub-tasks for the corresponding file! 🎉

Helper tasks

Tasks don’t have to be part of an intricate pipeline with carefully specified targets and dependencies. They can also be just a nice little helper that encapsulate a useful shell command.

Consider for example this task, which can readily be added to our dodo.py file:

def task_show():
    """display SVG"""
    for source in SOURCES:
        yield {
            "name": source.stem,
            "actions": [f"vpype read {source} show"],
        }

Its action consist of loading the source SVG and displaying it with vpype. This isn’t necessarily part of your workflow, but is convenient to have handy:

$ doit show:dots

The corresponding SVG is displayed by the vpype viewer:

This example is taken from vpype-perspective, where all the README’s figures are made from VPYs files stored in the repository’s examples/figures subdirectory. The conversion of these VPYs into SVGs is handled by doit using this dodo.py file. It’s a nice example of what can be done with doit.

Final words

If you made it that far, I hope you are convinced of how useful doit is for workflow automation.

In this article, I focused on vpype, but doit can be used for entirely different things. As a matter of fact, I used it to automate my #plotloop machine, which I’ve described in my next article.

One of doit drawbacks is the fact that its dodo.py file is written in Python. Creating one requires at least some Python basics—or willingness to acquire them. This might put off people uninterested by code.

But this is also its greatest strength. You wield the full power of Python when writing your dodo.py file, without any of the constraints of configuration languages such as YAML or TOML. This extends the possibilities much further than what was covered here, and makes learning doit a great investment! 🎯

Ready to take the plunge? I’m happy to help—just share details of your workflow in the comments 👇, on Twitter/Mastodon, or on the Drawingbots Discord.

Edit: TIL what “dolt” (lowercase L) means 😅, and changed DoIt (uppercase i) into doit, consistently with their documentation.

Edit: Added a link to the Automatic #plotloop Machine article and updated the video.

So, here is the thing. A Raspberry PI paired with a touch-screen can serve as a touch-based, human-machine interface (HMI) for things like DIY projects, robots, and whatnot. For example, this is how I control my Axidraw plotter. The current implementation of my HMI software – named taxi – uses Kivy. Here it is in action:

This requires a full X11 and desktop environment (e.g. LXDE, used by default by Raspberry Pi OS), which is heavy (the full Raspberry Pi OS doesn’t fit on a 4GB memory card) and harder to turn into kiosk mode.

Ever since I first learned of the modern TUI framework Textual, I’ve been thinking of using it as a lightweight, kiosk-mode, touch-based HMI with minimal requirements for the underlying OS. All it needs is a terminal!

My first idea was to simply use the Linux console, which is the terminal-y thing you see at boot, and can use to login if you don’t have a X11/desktop environment. Unfortunately, due to severe limitations in the number of glyphs it can handle (or of my comprehension of it), the results were… underwhelming:

This screenshot was made with a Debian VM running in Parallels Desktop, but the it’s the same for a Raspberry Pi and a physical screen.

Then I learned about framebuffer terminal emulators. The Linux framebuffer is a subsystem to display on-screen graphics over the system console without relying on a X11 server. Framebuffer terminal emulators basically emulate a terminal and “draw” it to the screen using the Linux framebuffer. As it turns out, they are extremely niche pieces of software, so it felt like a trip to the past to get them to run! 😅

For a bunch of reasons discussed below, this is most likely a dead-end in my quest. Still, I learned a few things and I surely won’t remember any of it unless it’s writen down.

The plan

I found a bunch of framebuffer terminal emulators, including fbcon, fbterm, bterm, yaft, and fbpad. I tried some of them (not all), and fbpad, by Ali Gholami Rudi, was the first to yield decent results, so it’ll be the focus of this article.

Here is an overview of the steps:

1. Download a suitable TTF font.
2. Download and build ft2tf, and use it to convert the font to fbpad’s custom format.
3. Download, configure, and build fbpad.
4. Download Textual and run it in fbpad.

I’ll assume a Debian type of OS, like Raspberry Pi OS, Ubuntu, or a Debian distro.

Preparing the font

First, let’s download a suitable monospace font. I chose to use Fira Code.

cd ~
wget -O fira.zip "https://fonts.google.com/download?family=Fira%20Code"
unzip fira.zip -d fira

Then, download and build ft2tf, a font conversion tool by the same author as fbpad:

cd ~
apt-get install libfreetype-dev
wget http://litcave.rudi.ir/ft2tf-0.9.tar.gz
tar xzf ft2tf-0.9.tar.gz
cd ft2tf-0.9
make

Now we’re ready to convert the font:

./ft2tf -h18 -w10 ~/fira/static/FiraCode-Regular.ttf:6 > ~/fira/fira.tf

The -h and -w options specify the final glyph size in the terminal, which in turn determines the column and row count based on your screen resolution. The :6 part after the font path specifies the size at which the TTF font is scaled before conversion. I found these values to work decently well for my Raspberry Pi’s screen (a Waveshare 7" HDMI LCD), but YMMV.

Configure and build fbpad

First, download fbpad:

cd ~
git clone https://github.com/aligrudi/fbpad
cd fbpad

Then, edit conf.h and change these two options:

// ...

#define TERM      "xterm-256color"
#define FR        "/home/USERNAME/fira/fira.tf"

// ...

Obviously, use your actual username. You can leave the rest unchanged.

Finally, compile fbpad:

make

Running Textual in fbpad

First, grab a copy of Textual (here from source, using Poetry – installing in a venv should work as well):

cd ~
git clone https://github.com/Textualize/textual.git
cd textual
poetry install

We’re finally ready to roll! Here is Textual color reference:

TERMCOLOR=truecolor ~/fbpad/fbpad poetry run textual colors

Note that TERMCOLOR=truecolor is required to have 256 colors, but it doesn’t actually enable 24bit true color mode, which fbpad does not support.

Here is how it looks like in the Debian VM:

The Textual demo can ben run with the following command:

TERMCOLOR=truecolor ~/fbpad/fbpad poetry run python -m textual

Finally, here is a video of how it runs on the actual Raspberry Pi:

Final words

As I mentioned before, this approach has many limitations:

• It’s obviously cumbersome to setup – compiled configuration file anyone??
• Hardly any of the framebuffer terminal emulators are active and well maintained projects.
• None of them have mouse support – let alone handle a touch-screen¹.
• The performance is bad, without an ounce of hardware acceleration.

For my purposes, I’ll likely move on to a bare-bone X11 setup without windows manager nor Desktop environment (basically startx + fullscreen xterm), since it addresses most of these limitations. Still, it was a fun niche to dig into! 🤓

No ground-breaking features, but an improved “quality-of-life”, especially for Apple-silicon Mac owners, and few other goodies.

Let’s dive in.

Migration to PySide6

• Migrated to PySide6 (from PySide2), which simplifies installation on Apple silicon Macs (#552, #559, #567)

PySide2 is the official Python wrapper for Qt 5, the GUI toolkit I use for the viewer. As Qt 5 doesn’t officially support Apple-silicon Macs, PySide2 – and thus vpype until now – were notoriously difficult to install on these computers. This is resolved with the transition to PySide6, which wraps Qt 6 and officially supports Apple-silicon Macs.

This took me waaaay too long. I actually feel bad for the struggle incurred to vpype users 😅 All things considered, the migration wasn’t that complicated, but there were still a few pitfalls to figure out due to Qt 6 breaking changes around the OpenGL-based widget.

Migrating to PySide6 is also a major step towards supporting Python 3.11, which brings a host of novelties as well as a significant performance bumps. I’m hoping this will happen by the next release, which means vpype 1.12 might well be the last to support Python 3.8.

Other fixes and improvements

• The layout command now properly handles the tight special case by fitting the page size around the existing geometries, accommodating for a margin if provided (#556)
• Fixed a viewer issue where page width/height of 0 would lead to errors and a blank display (#555)

Using layout tight would formerly set the page size to 0 by 0, which is useless in itself and caused a blank display. Not only the blanking issue has been resolved, but layout tight is now actually useful. It sets the page size to fit exactly the current geometries, accounting for a margin if --fit-to-margin MARGIN is provided.

• Added inch unit as a synonym to in, useful for expressions (in which in is a reserved keyword) (#541)

This addresses an oversight introduced with expressions in vpype 1.9. The units available for length CLI options are also available as scaling factor in expressions. For example, this creates a 10x15 cm rectangle:

vpype rect 0 0 10cm '%15*cm%' show

The expression works because the cm variable is made available by the interpreter, and set to the conversion factor between centimetres and pixels. This would however break for inches, because in is a reserved keyword in Python. The variable inch is now available instead. Either form can be used in CLI options, but inch must be used in expressions:

vpype circle 5in 5inch '%3*inch%' show

• Fixed a viewer issue where fitting the view to the document would not adjust when page size changes (vsketch only) (#564)

This change doesn’t directly benefits vpype as the page size cannot change while the viewer is active. In vsketch, however, the sketch code is free to set/change the page size based on GUI parameters, like in the included quick_draw example. In this case, when the view is fitted to the page size (i.e. as long as you don’t zoom or scroll), the view will adjust when the page size changes.

• Updated svgelements to 1.8.4, which fixes issue with some SVG constructs used by Matplotlib exports (#549)

Supporting all of the SVG standard subtleties is hard. Not only svgelements does a great job at it, but @tatarize’s reactivity when edge cases appear is unmatched. In this instance, Drawingbots' Discord user apur wanted to plot Matplotlib-generated SVGs of LaTeX equations. They included unusual <use> elements, which didn’t import properly. This is now fixed and I eagerly await the next niche corner case! 🤗

• Migrated to Plausible.io (from Google Analytics) for vpype.readthedocs.io (#546)

Plausible is a privacy-focused, GDPR-compliant web statistics service. As I did for this site, I migrated from Google Analytics with my projects' documentation web sites. This is a paid service, so that neither your or I are the product.

Mystery changes

• Added new units (yd, mi, and km) (#541)
• Added vpype.format_length() to convert pixel length into human-readable string with units (#541)

Yes, vpype supports kilometer-scale plots!¹

Seriously though, these changes are part of the WIP improvements of vpype’s terminal output. This will happen in future versions, but for some reason it was easier to integrate those change early.

Developer-related changes

• Poetry 1.2 or later is not required (developer only) (#541)
• A justfile is now provided for most common operations (install, build the documentation, etc.) (#541)

Shoutout to these two great dev tools: Poetry (Python dependency management) and just (make replacement for useful commands). I use them on a daily basis!

This article explains how to setup a Read the Docs-based documentation with Plausible.io such that metrics are enabled only on “production” builds — e.g. the “latest” builds from the main branch and the version-tagged builds. This minimises the contamination of traffic statistics by development-related activities.

To achieve this, the basic idea is to customise your Sphinx template such that the Plausible.io script is only included when a conf.py-defined flag is set to True. This flag is then set based on environment variables provided by Read the Docs.

Let’s dive in the details a step at a time.

Enabling templates

If you haven’t done so already (for example to customise your API documentation), create a _templates sub-directory and let Sphinx know that this is where custom templates are to be found:

# conf.py

templates_path = ["_templates"]

Customising the template

Then, a custom template can be created to include the Plausible.io script. Create a _templates/base.html file with the following content:

<!-- _templates/base.html -->
{% extends "!base.html" %}
{% block extrahead %}
    {% if enable_plausible %}
        <script defer data-domain="myproject.readthedocs.io"
                src="https://plausible.io/js/script.js"></script>
    {% endif %}
    {{ super() }}
{% endblock %}

In _templates/base.html, replace myproject.readthedocs.io by the actual domain name of your documentation. This domain name must also be enabled in your Plausible.io account.

Note that the <script> tag is added only if the template variable enable_plausible evaluates to True. This is how we can control whether or not metrics should be enabled for a given build.

Important: I’m using the Furo theme, which uses base.html as main HTML file. Other themes (including the default Sphinx theme) might be using layout.html instead, as indicated in Sphinx’s documentation on templating. This initially threw me off, so make sure to check which of your template’s file must be extended.

Enabling metrics on production build

The enable_plausible variable must be defined for our template above to function. This is done in conf.py file using the html_context variable as follows:

# conf.py
import os

# [...]

READTHEDOCS_VERSION_TYPE = os.environ.get("READTHEDOCS_VERSION_TYPE", None)

html_context = {
    "enable_plausible": READTHEDOCS_VERSION_TYPE in ["branch", "tag"],
}

I use the READTHEDOCS_VERSION_TYPE environment variable, which is set by Read the Docs. Its value is "branch" when the docs are built from the main branch, and "tag" when they are built from a tagged release. We want enable_plausible to be set to True in those instances. In any other case, including when READTHEDOCS_VERSION_TYPE is undefined (as is the case for local builds), enable_plausible is set to False.

And that’s about it – these few steps are all it takes more compliant and privacy-friendly metrics thanks to Plausible.io. For a real-world example, you can check my vpype project (relevant PR).

For instance, the Built-in symbols section lists the units available to expressions:

These units are related to the following piece of code:

# vpype/utils.py

UNITS = {
    "px": 1.0,
    "in": 96.0,
    "inch": 96.0,
    "ft": 12.0 * 96.0,
    "yd": 36.0 * 96.0,
    "mi": 1760.0 * 36.0 * 96.0,
    "mm": 96.0 / 25.4,
    "cm": 96.0 / 2.54,
    "m": 100.0 * 96.0 / 2.54,
    "km": 100_000.0 * 96.0 / 2.54,
    "pc": 16.0,
    "pt": 96.0 / 72.0,
}

I recently added support for more units and, of course, the documentation was at risk of running out of sync. Obviously, generating the list of units based on the code would be a better solution. After some Googling, here is how I did it.

The basic idea is to use substitutions. A substitution consists of assigning a text snippet to a keyword, and subsequently use said keyword (with the |keyword| syntax) in the documentation’s body. The second insight is to use the rst_prolog variable (within the conf.py file) for the definition. This being regular Python, the definition can easily be auto-generated based on the original code.

Here is how it looks for the case above:

# docs/conf.py

import vpype as vp

# [...]

UNIT_STRINGS = ", ".join(f"``{s}``" for s in sorted(vp.UNITS.keys()) if s != "in")

rst_prolog = f"""
.. |units| replace:: {UNIT_STRINGS}
"""

(Note that in is explicitly excluded from the list because it is a reserved Python keyword and cannot be used in the context of vpype expressions.)

And this is how the substitution is used in the actual documentation file:

..
  docs/fundamentals.rst

* Units constants (|units|).

  These variables may be used to convert values to CSS pixels unit, which *vpype* uses internally. For example, the expression ``%(3+4)*cm%`` evaluates to the pixel equivalent of 7 centimeters (e.g. ~264.6 pixels). (Note that expressions may overwrite these variables, e.g. to use the ``m`` variable for another purpose.)

Et voilà! Nice and easy. I certainly expect to use this technique often in the future.

Thanks a lot to Andee Collard for his useful feedback and providing this article’s banner!

Painting with a plotter

• Added the splitdist command to split layers by drawing distance (thanks to @LoicGoulefert) (#487, #501)

The new splitdist command, contributed by Loïc Goulefert (thanks a lot!), is the core of the paint plotting use-case. It splits each layer into newly created layers such that their respective drawing distance is each below the specified limit.

This command could readily be used with a clever vpype-gcode profile that implements the dipping mechanism at the beginning of each layer. Alternatively, it can be combined with the forlayer block command to insert dipping patterns into the line work. We’ll see an example of such a pipeline below.

• Added meters (m) and feet (ft) to the supported units (#498, #508)
• Fixed an issue with expressions where some variable names corresponding to units (e.g. m) could not be used (expressions may now reuse these names) (#506)

These are rather large units for typical plotting workflow, but come in useful for specifying the maximum drawing distance with splitdist.

As a reminder, units are available in two contexts:

1. Every time a command accepts a length-type argument or option (e.g. translate 5mm 3cm or linemerge --tolerance 0.05mm).
2. In expressions (e.g. forlayer translate "%_i*3*cm%" 0 end).

In the latter case, the existence of the unit constant precluded the use of variables with the same name. This issue worsened with the addition of m as this is a rather common variable name (e.g. this cookbook recipe uses it). To address this, they are no longer read-only and may now be overwritten. Of course, doing so renders their original value unavailable in the pipeline’s subsequent expressions.

• Fixed an issue with blocks where certain nested commands could lead totally unexpected results (#506)
• API: removed the faulty temp_document() context manager from vpype_cli.State() (#506)

The improved blocks introduced in vpype 1.9 had a major flaw which could, in some circumstances, result in erratic results. It turns out that the new splitdist command triggered this issue and brought it in the spotlight. This is now fixed, and the vpype_cli.State.temp_document() API is a casualty of this patch (luckily, it was introduced recently and I’m pretty sure no one used it yet besides me).

• Fixed an issue with the lmove command where order would not be respected in certain cases such as lmove all 2 (the content of layer 2 was placed before that of layer 1) (#506)

This is yet another issue highlighted by to the “plotting with paint” workflow. When the source layers included the destination layer (as is the case for lmove all 2), the order of the source layers would not be respected (e.g. for a 3-layer pipeline and thelmove all 2 command, layer 2 would end up with its original content, then layer 1, then layer 3). With this fix, the destination layer will now include the source layers' content in the correct order (e.g. in the previous example, layer 2 would end up with the content of layer 1, then layer 2, then layer 3).

Collectively, these changes enable the “plotting with paint” workflow using the following pipeline:

$ vpype \
      read input.svg \
      forlayer \
        lmove %_lid% 1 \
        splitdist 1m \
        forlayer \
          lmove %_lid% "%_lid*2%" \
          read -l "%_lid*2-1%" dip_%_name%.svg \
        end \
      lmove all %_lid% \
      name -l %_lid% %_name% \
      color -l %_lid% %_color% \
    end \
    write output.svg

For this to work, the layers in input.svg must be named after their respective color and, for each such color, a file named dip_COLORNAME.svg must exist. For example, if input.svg has two layers named “red” and “blue”, then the dip_red.svg and dip_blue.svg files must exist.

The following figure illustrates the results for synthetic data.

This pipeline is listed in a cookbook recipe and will be explained in details, along with the forlayer block command, in a future article.

Other changes

• Improved the linemerge algorithm by making it less dependent on line order (#496)

The linemerge command is implemented using a greedy algorithm which roughly works as follows:

1. Pick the first available line.
2. Look for another line that can be appended.
3. If found, merge both lines and look for further line to append (back to step 2). If not, save the current line, pick the next available one, and repeat (back to step 1).

By default, linemerge always considers both endings of each line, possibly reversing them if this enables a merge. This is not always desirable though, which is why the --no-flip option exists. In this case, the algorithm would only try to append to the current line, without trying to prepend as well. This oversight led to a greater dependence on line order and, occasionally, suboptimal results, as illustrated by the figure below.

With this fix, linemerge --no-flip now tries to both append and prepend, leading to more consistent results.

• Added --keep-page-size option to grid command (#506)

By default, the grid block command sets the page size to its geometry. For example, the block grid --offset 4cm 3cm 3 5 [...] end sets the page size to 12x15cm. This behaviour can now be disabled with the --keep-page-size option.

This change mainly helps for the testability of the blocks feature (in this release, I’ve added multiple tests to minimise the risk of future regression), but I figured it could have its occasional use out there.

• Added HPGL configurations for the Houston Instrument DMP-161, HP7550, Roland DXY 1xxxseries and sketchmate plotters (thanks to @jimmykl and @ithinkido) (#472, #474)

Thanks a lot, Jimmy Kirkus-Lamont and @ithinkido! ❤️

• Added equality operator to vpype.LineCollection and vpype.Document (#506)

I can now check if two layers or documents have the same content and metadata using the equality operator ==. This is immensely useful when writing tests. I have no idea why it took so long… 🤷

• Pinned Shapely to 1.8.2, which is the first release in a long time to have binaries for most platforms/Python release combination (including Apple-silicon Macs and Python 3.10) (#475)

It was quite the roller coaster ride for Shapely to be properly packaged for both Python 3.10 and Apple-silicon Macs, but now this is fully sorted out. That’s one less hassle when installing vpype.

• Removed deprecated API (#507)

With vpype 1.9, a number of APIs migrated from the vpype package to the vpype_cli package. The former APIs still worked but emitted deprecation warnings. They are now gone forever.

This article addresses the following objectives:

• Produce a beautiful API documentation based on the code docstrings that is both nice to look at and easy to navigate.
• Support for a proper table of content navigation down to each class/module’s member.
• Nice looking summary tables listing modules' and classes' contents.

This article targets an audience of FOSS maintainers who are unhappy with the state of their project’s API documentation, and are frustrated with the process of improving it. A basic understanding of Sphinx as well as an existing documentation setup is assumed. This article basically contains everything I wish I was told when I first started on my API reference improvement journey. For the beginners, I’ll provide pointers to help setting up a basic Sphinx.

Note that although this article is structured as a tutorial, it covers tips and techniques which are likely useful for other kinds of documentation customisation.

Basic setup

As stated above, the basic steps to setup a Sphinx-based documentation project are outside the scope of the present article. I suggest reviewing the following resources to get started:

• @Mariatta made a brilliant tutorial on how to kick-start a Sphinx documentation project.
• Thomas Cokelaer has a very nice reStructuredText cheat sheet.
• Simon Willison wrote another cheat sheet which covers “the subset of reStructuredText worth committing to memory”.
• Obviously, Sphinx’s documentation is an important resource. Although it is somewhat arid for the newcomer, I strongly suggest not giving up on it. I had multiple “oh there it is!” moments with it in the process of writing this article.
• Finally, Read the Docs is likely the best place to host your documentation. It’s very simple to setup and free for open source projects.

As for the theme, my preference goes for Pradyun Gedam’s Furo. I’m using it with the default configuration, so the only requirement is to enable it in your conf.py file:

html_theme = "furo"

Note that some of the CSS provided in this article may need adjustments should you opt for a different theme.

Autoapi setup

After trying both autodoc/autosummary and Sphinx AutoAPI, I opted to use the latter. Here are the main reasons behind this choice:

• Autosummary does not generate TOC entries for API elements such as classes/modules and their members. This is due to a long-standing Sphinx limitation. Autoapi works around this limitation (albeit imperfectly, as we’ll later note).
• Autosummary defaults to not generating anything and is in my experience frustrating to setup. In contrast, autoapi produces usable output out-of-the-box.
• Templates are easier to write thanks to the rich “mapper” objects AutoAPI provides after parsing your code (see the AutoAPI objects section below).

Note that there are two things called “autoapi” floating on the Internet: the Sphinx AutoAPI project (documentation) is the good one. The other one is unmaintained and barely documented. Make sure you don’t loose time dealing with the wrong one.

Basics

Setting up Sphinx AutoAPI is covered in their documentation. It boils down to the following steps.

1. Install the sphinx-autoapi package:

   $ pip install sphinx-autoapi
   

2. Add AutoAPI it to the Sphinx extension list:

   extensions = [
       ...,
       'autoapi.extension',
   ]
   

3. List your package directories (or the directory containing them) and set basic options:

   autoapi_dirs = ['../mypackage']
   autoapi_type = "python"
   

4. Add the generated documentation to your index.rst toctree:

   .. toctree::
      :maxdepth: 3
   
      ...
      autoapi/index
   

Setting up templates

We will customise Sphinx AutoAPI’s default templates. The easiest is to copy Sphinx AutoAPI’s default templates in your project to serve as a starting point.

First, run the following commands (adjusting for your Python version) from your documentation directory:

$ mkdir _templates
$ mkdir _template/autoapi
$ cp $VIRTUAL_ENV/lib/python3.10/site-packages/autoapi/templates/index.rst _templates/autoapi/
$ cp -r $VIRTUAL_ENV/lib/python3.10/site-packages/autoapi/templates/python _templates/autoapi/ 

Then, tell Sphinx AutoAPI of its template directory in your conf.py file:

autoapi_template_dir = "_templates/autoapi"

A useful tip is to make a Git commit just after copying the built-in templates, such that you can track (and revert) your modifications. I’ve used this extensively while working on my templates.

At this point, I suggest spending some time to become acquainted with the built-in templates and how they are organised and implemented. If you haven’t used it before, it is also useful to review the Jinja2 templating language documentation.

Other configuration options

autoapi_options

The autoapi_options controls various aspect of the generated documentation, including the type of class/module members that are listed. Its default value is sensible but I still felt like customising it:

autoapi_options = [
    "members",
    "undoc-members",
    "show-inheritance",
    "show-module-summary",
    "imported-members",
]

In particular, I want the summary at the top of the module’s documentation (show-module-summary), but we will heavily customise it. Check the documentation for a list of available options and their descriptions.

autoapi_keep_files

Another useful option is autoapi_keep_files. Sphinx-autoapi generates .rst pages for the documentation during the build process, but defaults to deleting them after completion. It’s often useful to keep them around for inspection and debugging purposes:

autoapi_keep_files = True

autodoc_typehints

This is technically an autodoc setting, but Sphinx AutoAPI honours it. It controls if/where type hints are included in the documentation. The possible values are the following:

• "signature": type hints are included in the function signature, which appears first in the member’s documentation
• "description": type hints are included within the function description, when the arguments are listed
• "both": type hints are included in both places
• "none": type hints are not included

My preference goes for "signature":

autodoc_typehints = "signature"

AutoAPI objects

Understanding the Sphinx AutoAPI objects is key to customising templates. They are one of the major difference with respect to autodoc/autosummary.

In order to generate the API documentation, the autodoc loads your actual code and uses Python’s introspection capabilities to extract the required information from your module and class objects. In contrast, Sphinx AutoAPI parses your Python code and builds a collection of so-called “mapper” objects which describe your code and its structure. These objects are then passed on as context to the Jinja2 templating engine. Oddly, the documentation doesn’t provide a reference about them, but their implementation is easy to read.

Here is a summary of some of the attributes that are useful when writing templates:

obj.name

Name of the mapped object, e.g. "MyClass" or "my_method".

obj.id

Fully qualified name of the object, used for cross-referencing, e.g. "my_module.MyClass" or "my_module.MyClass.my_method".

obj.summary

Summary of the object’s docstring (i.e. the first line).

obj.docstring

Full docstring of the object.

obj.display

Indicates whether or not this object should be displayed, based on the options set in conf.py and the result of the autoapi-skip-member event (discussed later).

obj.children

(Modules and classes only) List children functions, methods, attributes, etc.

obj.properties

(Functions and methods only) List of properties, such as "classmethod", "staticmethod"", "abstractmethod", "property", etc.

obj.obj.args

(Functions and methods only) List of 4-tuples describing the function’s arguments. The first item is the star operator if any ("*", "**", or None), the second is the argument name, the third is the argument type or None, and the fourth is the argument default value or None. This key piece of data will enable us to recreate the signatures according to our needs.

When working on your documentation, it is often useful to inspect the contents of these mapper objects using a debugger. This can be achieved by adding an handler for the autoapi-skip-member event and setting a conditional breakpoint:

def skip_member(app, what, name, obj, skip, options):
    # conditional breakpoint here
    return skip


def setup(sphinx):
    sphinx.connect("autoapi-skip-member", skip_member)

This event will be triggered for every single Python object parsed from your code. By breaking, for example, when obj.name == "my_module", the obj argument and its children can be fully inspected. I use the following run configuration in my IDE for this:

• Execute module: sphinx.cmd.build
• Parameters: -M html . _build
• Working directory: docs/

An autosummary-like macro

By default, Sphinx AutoAPI provides a summary list of classes, functions, and attributes at the top of a module’s documentation, which is very nice. Our objective is to add a similar table at the top of each class description, to facilitate navigation. However, Sphinx AutoAPI uses its own autoapisummary directive, which derives from autosummary’s autosummary directive. Both suffer from the following limitations:

• The way callables are rendered is hard-coded and cannot be customised via templates. In particular, if autodoc_typehints is set to "signature" or "both", autosummary will include type hints in the summary table as well. Unfortunately, this dramatically increases the length of the signature, which is detrimental to the table layout and usability. Alternatively, signatures can be entirely removed by using the :nosignatures: option. However, in this case, not even parenthesis are displayed, which hides the callable nature of the function. The best compromise is to have the full signature with their arguments, but without typing annotations.
• Properties are listed as functions, including their signature. This hides the fact that, API-wise, they behave as data members (though it would still be useful to indicate that they are in fact properties).
• There is not indication that a method is abstract, static, or class-based.

To address these shortcomings, we will create a Jinja2 template macro to replicate and improve on autosummary/autoapisummary functionality.¹

Our aim is to create tables where callable have their full – but unannotated – signature, where properties are indicated as such but rendered as attributes, and where static, class, and abstract methods are marked as such. Here is an example of this:

Basic macro setup

The basic insight is that a summary table can be implemented using Sphinx’s list-table:

.. list-table:: Title
   :header-rows: 0
   :widths: auto
   
   * - Item 1
     - This is the description of the first item.
   * - Name 2
     - This is also a description, but this time for the second item.
   * - ...
     - ...

Such a table can be generated with the following Jinja macro:

{% macro auto_summary(objs, title='') -%}

.. list-table:: {{ title }}
   :header-rows: 0
   :widths: auto

{% for obj in objs %}
    * - obj.name
      - obj.summary
{% endfor %}
{% endmacro %}

To test this, create a file named _templates/autoapi/macros.rst and add the code above. Then, make the following edits to the _templates/autoapi/python/module.rst file:

• At the top of the file, import macros.rst to make it available for use:

  {% import 'macros.rst' as macros %}
  

• Locate where the class summary is generated:

  .. autoapisummary::
  
  {% for klass in visible_classes %}
     {{ klass.id }}
  {% endfor %}
  

• Replace the code above by a call to our macro:

  {{ macros.auto_summary(visible_classes, title="Classes") }}
  

Here is the result I obtain with my project:

This is a good start, but we’re obviously far from the result we want. To start with, no cross-reference links are generated. And, had we passed functions or methods instead of classes to our macro, no signature would have been generated.

Custom labels

Before fixing our macro, we must discuss these nice looking “prop”, “static”, and “class” tag-like labels in the example tables above. These are implemented using a custom role with some CSS attached to it.

This StackOverflow answer explains how to create a custom role and make it globally available to your documentation. Basically, just add the following to your conf.py file:

rst_prolog = """
.. role:: summarylabel
"""

The role directive creates a new role which can then be used as follows:

:summarylabel:`My Label`

Sphinx generates the corresponding HTML code:

<span class="summarylabel">My label</span>

Since it sets an HTML class named after the role, it’s easy to adjust the label appearance using some custom CSS. Create a file named _static/css/custom.css in your documentation directory and add the following CSS:

span.summarylabel {
    background-color: var(--color-foreground-secondary);
    color: var(--color-background-secondary);
    font-size: 70%;
    padding-left: 2px;
    padding-right: 2px;
    border-radius: 3px;
    vertical-align: 15%;
    padding-bottom: 2px;
    filter: opacity(40%);
}

Note the use of CSS variables in order to support Furo’s dynamic night mode feature.

Finally, we must tell Sphinx about this CSS file in the conf.py file:

html_css_files = [
    "css/custom.css",
]

Customising the table appearance

A similar CSS approach can be used to customise the appearance of the summary table itself. By adding the :class: option to the list-table directive, we can tell Sphinx to attach an HTML class to the <table> element, which we can then customise with CSS:

.. list-table:: Title
   :header-rows: 0
   :widths: auto
   :class: summarytable
   
   * - ...
     - ...

For my project, the only change I made to the default appearance is to force the table to span the entire width regardless of its contents. This can be done by adding the following code to our custom.css file:

table.summarytable {
    width: 100%;
}

Putting it all together

We are now ready to put everything together and improve our auto_summary() macro to our liking. Here is the final code for macros.rst:

{% macro _render_item_name(obj, sig=False) -%}
:py:obj:`{{ obj.name }} <{{ obj.id }}>`
     {%- if sig -%}
       \ (
       {%- for arg in obj.obj.args -%}
          {%- if arg[0] %}{{ arg[0]|replace('*', '\*') }}{% endif -%}{{  arg[1] -}}
          {%- if not loop.last  %}, {% endif -%}
       {%- endfor -%}
       ){%- endif -%}
{%- endmacro %}

{% macro _item(obj, sig=False, label='') %}
   * - {{ _render_item_name(obj, sig) }}
     - {% if label %}:summarylabel:`{{ label }}` {% endif %}{% if obj.summary %}{{ obj.summary }}{% else %}\-{% endif +%}
{% endmacro %}

{% macro auto_summary(objs, title='') -%}
.. list-table:: {{ title }}
   :header-rows: 0
   :widths: auto
   :class: summarytable

  {% for obj in objs -%}
    {%- set sig = (obj.type in ['method', 'function'] and not 'property' in obj.properties) -%}

    {%- if 'property' in obj.properties -%}
      {%- set label = 'prop' -%}
    {%- elif 'classmethod' in obj.properties -%}
      {%- set label = 'class' -%}
    {%- elif 'abstractmethod' in obj.properties -%}
      {%- set label = 'abc' -%}
    {%- elif 'staticmethod' in obj.properties -%}
      {%- set label = 'static' -%}
    {%- else -%}
      {%- set label = '' -%}
    {%- endif -%}

    {{- _item(obj, sig=sig, label=label) -}}
  {%- endfor -%}

{% endmacro %}

The work is now split in three macros:

auto_summary()

This is the “public” macro. It generates a table based on a list of mapper objects, with an optional title. It iterates over the list of objects, and, for each of them, determines if the signature should be generated (functions and non-property methods) and if some label should be attached. It then uses _item() to generate each object’s code.

_item()

This helper macro generates the code for each object, prepending a label to the summary if requested.

_render_item_name()

This helper macro focuses on generating the properly-cross-referenced object name. If the signature is requested, it iterates over the obj.obj.args list to produce a full (but unannotated) list of arguments.

Improving the default templates

With our auto_summary() macro completed, we are now ready to customise our templates, but we still have one issue to resolve before we do so.

Categorising objects with a custom Jinja2 test

As we saw in the AutoAPI objects section, mapper objects representing modules or classes have a children attribute which lists the objects it contains. For example, a module’s children attribute lists all the classes, functions and attributes defined within it.

In order to categorise these children into separate sub-lists, the built-in templates heavily use the selectattr() and rejectattr() filters. For example, a list of classes in a module can be obtained as follows:

{% set visible_children =
    module_object.children|selectattr("display")|rejectattr("imported")|list %}
{% set visible_classes =
    visible_children|selectattr("type", "equalto", "class")|list %}

This code selects the visible (but not imported) children from module_object, and then further selects children which have their type set to "class". In the code above, "equalto" is known as a Jinja test. There are many such built-in tests in Jinja2.

As stated before, we aim to categorise properties as attributes instead of methods. To that end, we will have to filter methods whose properties attribute contains "property". Intuition dictates that the following code achieves this:

{% set property_methods =
    all_methods|selectattr("properties", "contains", "property")|list %}

The bad news is that no such "contains" test exists by default in Jinja2. The good news is that it is trivial to add one.

First, the actual test must be written. Add the following code to your conf.py file:

def contains(seq, item):
    return item in seq

Then, we just need to add this test to the Jinja environment. Sphinx AutoAPI provides a hook for that:

def prepare_jinja_env(jinja_env) -> None:
    jinja_env.tests["contains"] = contains

autoapi_prepare_jinja_env = prepare_jinja_env

With this in your conf.py file, the template code above will work as expected.

Updating templates

We previously replaced one of module.rst’s use of autoapisummary by our auto_summary() macro (see Basic macro setup). It is now time to generalise the use of our macro. At this stage, the details of how this is done is to a large extent up to reader’s taste. The templates of vsketch can serve as fully-functional example and can readily be used in another projects.

For the module.rst template, I have opted to simplify the overview’s structure by just generating tables (without headings) for classes, functions, and attributes:

{% if "show-module-summary" in autoapi_options and (visible_classes or visible_functions) %}
{% block classes scoped %}
{% if visible_classes %}
{{ macros.auto_summary(visible_classes, title="Classes") }}
{% endif %}
{% endblock %}

{% block functions scoped %}
{% if visible_functions %}
{{ macros.auto_summary(visible_functions, title="Functions") }}
{% endif %}
{% endblock %}

{% block attributes scoped %}
{% if visible_attributes %}
{{ macros.auto_summary(visible_attributes, title="Attributes") }}
{% endif %}
{% endblock %}
{% endif %}

For the class.rst template, I chose to rework the structure of the documentation into two rubrics:

   {% if visible_methods or visible_attributes %}
   .. rubric:: Overview

   {% set summary_methods = visible_methods|rejectattr("properties", "contains", "property")|list %}
   {% set summary_attributes = visible_attributes + visible_methods|selectattr("properties", "contains", "property")|list %}
   
   {% if summary_attributes %}
   {{ macros.auto_summary(summary_attributes, title="Attributes")|indent(3) }}
   {% endif %}

   {% if summary_methods %}
   {{ macros.auto_summary(summary_methods, title="Methods")|indent(3) }}
   {% endif %}

   .. rubric:: Members

   {% for attribute in visible_attributes %}
   {{ attribute.render()|indent(3) }}
   {% endfor %}
   {% for method in visible_methods %}
   {{ method.render()|indent(3) }}
   {% endfor %}
   {% endif %}

Note the use of the custom "contains" Jinja2 test we implemented earlier.

Hiding submodules

Sphinx AutoAPI include all subpackages and submodules recursively, unless those are marked as private by prefixing their name with an underscore. In my packages' __init__.py file, I carefully import from submodules the objects which are meant to be public, but haven’t necessarily marked the submodules as private. Sphinx AutoAPI has no option to control whether or not to add them (I suggested adding one), so I had to filter them out manually. This is done with the autoapi-skip-member event handler we mentioned earlier:

def skip_member(app, what, name, obj, skip, options):
    # skip submodules
    if what == "module":
        skip = True
    return skip

def setup(sphinx):
    sphinx.connect("autoapi-skip-member", skip_member)

Hiding members

Likewise, it may happen that you want to hide specific members from the documentation without marking them as private. Again, the autoapi-skip-member event handler can do that. The following example is based from actual code in vsketch:

def skip_member(app, what, name, obj, skip, options):
    if "vsketch.SketchClass" in name:
        if obj.name in [
            "vsk",
            "param_set",
            "execute_draw",
            "ensure_finalized",
            "execute",
            "get_params",
            "set_param_set",
        ]:
            skip = True
    return skip

Note that name is the fully qualified name of the object, so the vsk member has name set to vsketch.SketchClass.vsk. In contrast, obj.name is just the base name.

Also, for this to work as expected for modules, I had to change the following line in module.rst

{% set visible_children = obj.children|selectattr("short_name", "in", obj.all)|list %}

into

{% set visible_children = obj.children|selectattr("display")|selectattr("short_name", "in", obj.all)|list %}

Without this modification, objects marked as skipped would show up in the summary tables.

Ordering

By default, Sphinx AutoAPI generates the documentation in the same order as the code. This can be changed to alphabetical order, I like being in control from the code.

In vsketch, the top level content is determined by the imports in my package’s __init__.py file, so the import statements themself matter. Since I’m using isort, I had to short-circuit it in this particular case:

# vsketch/__init__.py

# isort: skip_file

# Ordered for the documentation
from .vsketch import Vsketch
from .shape import Shape
from .sketch_class import SketchClass, Param, ParamType

from .easing import EASING_FUNCTIONS
from .utils import working_directory

Conclusion

Well, this ended up being much longer than anticipated 😅 – if you made it this far, congratulations! 🎉

Here is what we covered:

• We built on a basic Sphinx project and added AutoAPI to generate an API reference documentation.
• We created custom templates based on the built-in templates provided by AutoAPI.
• We reviewed the mapper objects created by AutoAPI to described our code.
• We learned how to use a debugger to easily inspect these objects.
• We crafted an autosummary-like macro named auto_summary() to build beautiful summary tables.
• We customised these tables with custom CSS.
• We used a custom Sphinx role and CSS to create tag-like labels to be used in the summary tables.
• We learned of Jinja2’s tests and created a custom one.
• We controlled the visibility of submodules and members using a autoapi-skip-member event handler.
• We learned how to control ordering from our code.

Like any software project, improving the documentation is a never-ending endeavour. As it turns out, there is one remaining issue that has been bugging me and is yet unresolved. Due to a limitation in Sphinx, AutoAPI has a tendency to mangle the TOC ordering, especially when section headings are emitted from the templates. Check these two issues for more information. Hopefully they’ll get solved in the future.

I shall conclude by stating that, by all means, I do not consider myself a Sphinx expert — much to the contrary. I did spend a lot of time improving my API documentation, and figured it would be wasteful not to share my findings, especially given the relative scarcity of information on advanced Sphinx usage patterns. As a result, it is rather likely that I made mistakes and sub-optimal choices. If so, please do provide feedback, and I’ll update this article to improve it.

Edit: updated title and intro to clarify the nature of the API discussed, i.e. Python API (2022-05-11).

For example, this test pixel art plot would benefit from a slightly thinner pitch to avoid the visible overlap between neighbouring lines:

There is no way around experience to find the optimal pitch. I’ve created the fill_test sketch to create custom charts with test patterns precisely tuned to the pen of interest. There is indeed no point to testing a rotring isograph .35mm pen with a generic 0.1mm to 1.0mm chart.

Here is how the UI looks:

Beyond the general layout options (page size, grid size, etc.), the Smallest Width and Width Increment parameters enable a fine-grained exploration of pitches around the nominal pen size.

Here is an example with a rotring isograph .35mm in my notebook, which has a slight propensity for ink soaking. For this combo, fills with .4mm pitch yield the best results:

To use the sketch, download or clone my sketches repository and execute the sketch using your existing vsketch installation:

$ vsk run path/to/sketchs/fill_test

Several generative art algorithms, such as Truchet tiles, use a regular grid of square cells. For example, check this interactive demo from the Generative Design book or these few pieces of mine.

Now, let’s say you want to generate an iteration of your algorithm for printing or plotting such that all margins around the grid are the same for the given paper size. You can of course adjust the number of cell rows and columns, but how should you size the cell such as to achieve uniform margins?

The image above illustrates the problem. For a given page of size $W \times H$ and a regular grid of $N \times M$ cells, what should be the cell size $s$ to achieve uniform margins $m$ around the grid? What is then the value of $m$?

The solution

This is easy to solve with a bit of math. Here is the system of two equations that must be solved:

with all parameters ($N$, $M$, $W$, $H$) and the cell size $s$ being strictly positive.

Solving this for $m$ and $s$ is no rocket science but Wolfram Alpha can do the job for you if your high school math is rusty!

We have the following solutions:

The first solution corresponds to the special case of a square page size. In this case, the grid must be square ($N=M$) and the margins may have an arbitrary value, with the cell size varying accordingly. This is not very surprising.

The second solution is where things become interesting. As intuition dictates, it is valid only if the grid orientation (portrait or landscape) matches the paper orientation. If so, uniform margins is achieved by choosing a cell size of $s = \frac{H-W}{M-N}$.

Note that the resulting margin $m$ may, depending on the parameters, be negative. In this case, the grid overflows all around the page by a constant distance. This making plotting/printing your piece inconvenient, you will have to adjust $N$ and/or $M$ to reach a positive margin value.

The demo

I made a demonstration sketch made with vsketch. Here is how it looks:

This is a simplified version of the code which can be used as a starting point for your next grid-based design:

import itertools
import vsketch

class MySketch(vsketch.SketchClass):
    N = vsketch.Param(5, 1)
    M = vsketch.Param(7, 1)

    def draw(self, vsk: vsketch.Vsketch) -> None:
        vsk.size("a4", landscape=False, center=False)   # disable auto-centering

        cell_size = (vsk.height - vsk.width) / (self.M - self.N)
        margin = (self.M * vsk.width - self.N * vsk.height) / 2 / (self.M - self.N)
        
        if cell_size > 0:
            # account for the computed margin
            vsk.translate(margin, margin)
            
            # draw the grid
            for i, j in itertools.product(range(self.N + 1), range(self.M + 1)):
                vsk.point(i * cell_size, j * cell_size)
        else:
            # ERROR: N and M values must be adjusted!
            pass

New features and improvements

• Improved support for layer pen width and opacity in the viewer (#448)

  • The “Pen Width” and “Pen Opacity” menus are now named “Default Pen Width” and “Default Pen Opacity”.
  • The layer opacity is now used for display by default. It can be overridden by the default pen opacity by checking the “Override” item from the “Default Pen Opacity” menu.
  • The layer pen width is now used for display by default as well. Likewise, it can be overridden by checking the “Override” item from the “Default Pen Width” menu.

This alone is reason to upgrade, and, if we’re being honest, it should have been done in the previous release. The display logic of the viewer is now as follows:

• By default, honor the layer’s pen width and opacity if present.
• If pen width and/or opacity is not set, revert to the value set in the menu (0.3mm / 80% by default).
• Either or both of the displayed pen width and opacity can be forced to the value in the menu using the new “Override” menu item.

It is worth noting that opacity is not a standalone layer property, but is part of its RGBA color property (vp_color). Weirdly, vpype 1.9’s viewer would honor the base color (RGB), but not its alpha chanel. (See next feature, though.)

By the way, this article’s cover image is a screenshot of the viewer made with this command:

(Yes, this is a properly syntax-highlighted vpype command made with a custom script and Rich’s new SVG export. This is very preliminary, but, in time, something to improve on and deploy more widely in the doc and elsewhere.)

• Added the alpha command to set layer opacity without changing the base color (#447, #451)

While working on the viewer improvements, I realized how inconvenient it was to set an arbitrary opacity value. Using CSS color names (e.g. color red) always sets opacity to 100% and there is no way around the hex notation for a custom value (e.g. color #ff00007f or color #f007). The new alpha command fills that gap: color red alpha 0.5.

• Added HPGL configuration for the Calcomp Artisan plotter (thanks to Andee Collard and @ithinkido) (#418)

Good news for Calcomp Artisan’s owners! Let this be a reminder that I welcome this kind of contribution. Though I’d like to, I can’t own every single type of vintage plotter! 😅

• The read command now better handles SVGs with missing width or height attributes (#446)

  When the width or height attribute is missing or expressed as percent, the read command now attempts to use the viewBox attribute to set the page size, defaulting to 1000x1000px if missing. This behavior can be overridden with the --display-size and the --display-landscape parameters.

I recently came across a SVG with a viewBox defined but no width/height attributes:

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1191.26 1684.49">...</svg>

In such an instance, using the read command used to default to an A4 page size, while the vpype.read_svg() API (and friends) would default to a 1000x1000px page size. This is both inconsistent and missing the opportunity to fallback on the viewBox. vpype now fully delegates this fallback logic to svgelements, which does a good job at making the most of the available information. Also, if everything is missing (or width/height are expressed in percents), vpype consistently falls back to 1000x1000px.

• Added the --dont-set-date option to the write command (#442)

This one is a bit niche. vpype adds some metadata to the SVG, including the date and time at which it was generated (note the <dc:date> tag):

$ vpype line 1cm 1cm 5cm 3cm layout a6 write -f svg -
<?xml version="1.0" encoding="utf-8" ?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:cc="http://creativecommons.org/ns" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:xlink="http://www.w3.org/1999/xlink" baseProfile="tiny" height="14.8cm" version="1.2" viewBox="0 0 396.85039370078744 559.3700787401575" width="10.5cm">
  <metadata>
    <rdf:RDF>
      <cc:Work>
        <dc:format>image/svg+xml</dc:format>
        <dc:source>vpype line 1cm 1cm 5cm 3cm layout a6 write -f svg -
</dc:source>
        <dc:date>2022-04-07T10:15:00.532842</dc:date>
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <defs/>
  <g fill="none" id="layer1" inkscape:groupmode="layer" inkscape:label="1" stroke="#0000ff" style="display:inline">
    <line x1="122.8346" x2="274.0157" y1="241.8898" y2="317.4803"/>
  </g>
</svg>

This is all well and good until you automate the generation of SVGs under a version control system, which is what I did for vpype-perspective’s documentation figures. A PyDoIt dodo.py files converts any .vpy file it finds into corresponding SVGs – basically a Python-powered, overcharged Makefile. (This is a rather neat process which, by the way, should get its own article someday.) In this kind of setup, having a ever-changing date in the SVG yields many unwanted VCS diffs which can now be avoided using this option.

Bug fixes

• Fixed an issue with forlayer where the _n variable was improperly set (#443)

One word: inexcusable 🙄

• Fixed an issue with write where layer opacity was included in the stroke attribute instead of using stroke-opacity, which, although compliant, was not compatible with Inkscape (#429)

This one is Inkscape’s fault. Using stroke-opacity is “more compatible” anyways, so it’s a good move regardless.

• Fixed an issue with vpype --help where commands from plug-ins would not be listed (#444)

I ran into an issue with Click where a sub-command plug-in using APIs from the top-level command’s package (a scheme widely used by vpype and its plug-ins) would fail because of circular imports. The workaround I used in vpype 1.9 meant that plug-ins were no longer listed in vpype --help. This is fixed now, but this may not be the end of the story. I tried – and failed – to reproduce the original issue in a minimal demo project and I’ll have to further dig into this someday.

• Fixed a minor issue where plug-ins would be reloaded each time vpype_cli.execute() is called (#444)

By “minor”, I mean that this amounted to a tiny performance hit for Python scripts using vpype’s execute() API multiple times.

• Fixed a rendering inconsistency in the viewer where the ruler width could vary by one pixel depending on the OpenGL driver/GPU/OS combination (#448)

The ruler of vpype’s viewer is supposed to be 20px wide, but it turns out that either of the horizontal or the vertical one was 21px wide instead. Which one? It depends on the platform and my Intel/AMD- and M1-based laptops disagreed on the matter! 😲

It took me a while to discover that this is due to drawing lines at a 0.5px offset with respect to the pixel grid, leading to unpredictable rounding behavior. This is basically what happens when drawing horizontal or vertical lines with integer coordinates, and I wrote about it a few days ago.

You’d think that not one soul would care about this, but some of my tests are based on comparing newly rendered images of the viewer with previously-generated reference images, and those tests would fail on my new M1 Mac.

API changes

• Added vpype_cli.FloatType(), vpype_cli.IntRangeType(), vpype_cli.FloatRangeType(), and vpype_cli.ChoiceType() (#430, #447)

These Click types provide support for property and expression substitution. They were missing from vpype 1.9 because they aren’t needed internally. Plug-ins, however, wanted them, including flow imager and my new vpype-perspective.

• Changed vpype.Document.add_to_sources() to also modify the vp_source property (#431)

This will simplify the code needed to handle sources in plug-ins.

• Changed the parameter name of both vpype_viewer.Engine() and vpype_viewer.render_image() from pen_width and pen_opacity to default_pen_width and default_pen_opacity (breaking change) (#448)
• Added override_pen_width and override_pen_opacity boolean parameters to both vpype_viewer.Engine() and vpype_viewer.render_image() (#448)
• Added a set_date:bool = True argument to vpype.write_svg() (#442)
• Changed the default value of default_width and default_height arguments of vpype.read_svg() (and friends) to None to allow svgelement better handle missing width/height attributes (#446)

These are the API counterparts of some of the changes described before.

Other changes

• Added support for Python 3.10 and dropped support for Python 3.7 (#417)

Walruses have appeared already! 🦭

• Updated code base with modern typing syntax (using pyupgrade) (#427)

I was this year old when I learned that you can use modern typing syntax (such as list[int] | None) with older Python versions thanks to this statement:

from __future__ import annotations

I swiftly ran pyupgrade on the entire code base to bring it up to date.

• Updated the documentation template (#428)

It looks cleaner now IMO, though there is still a whole lot that could be improved.

• Updated installation instructions to use pipx (#428)

I have yet to get over how long it took me to realize this! 😳 I’m sorry for everyone who has struggled with virtual environments to install vpype!

In this case, the reference images were generated long ago on my previous, Intel/AMD-based MacBook Pro. This GIF highlights the discrepancy I’d get with images generated on my new computer (notice how the ruler’s thickness varies):

As I’m currently working on this viewer again, I finally spent two days tracking this issue – and finally found its cause.

Without giving it a thought, I first used integer coordinates for those ruler lines. However, coordinates refer to pixel boundaries – not pixel centres. This means than an horizontal line with integer coordinates (e.g. [(2, 2), (7, 2)]) sits halfway between two consecutive rows of pixel:

Which of the 2nd or 3rd row of pixel eventually gets drawn is up to a coin toss – or rather the rounding strategy of your particular OpenGL driver/GPU/OS combination.

By offsetting the coordinates by half a pixel (e.g. [(2, 2.5), (7, 2.5)]), one can force the line on a specific pixel row and avoid any rounding:

This makes the rendering more predictable across platforms.

Ultimately, the fix was very simple (I just changed the ruler thickness from 20 to 19.5), but figuring it out was tricky (relevant discussions on ModernGL’s Discord server). Hopefully I wont forget about it after writing this TIL.

vpype 1.9 is finally out! 🎉

I recently stumbled upon a post by Simon Willison where he promotes the idea of annotated release notes. As it turns out, this release is, by any metric I can think of, the biggest and most transformative so far. The associated change log is consequently rather unwieldy and calls, you guessed it 💡, for the present annotated release notes.

(Note: although the original release notes are extensively quoted in this article, I reshuffled and shortened the original material. Make sure to check the base material for an authoritative list of change.)

Properties

Basics

• Added support for global and per-layer properties (#359)

  This feature introduces metadata to the pipeline in the form of properties which may either be attached to specific layers (layer property) or all of them (global property). Properties are identified by a name and may be of arbitrary type (e.g. integer, floating point, color, etc.). A number of system properties with a specific name (prefixed with vp_) and type are introduced to support some of the new features.

Metadata is data which says something about other data, and vpype lacked such a thing. Until now, what was passed from one command to the next consisted exclusively of paths sorted into layers, without any context such as what the color of these paths might be. One command could “know” about something (e.g. read knows, from the SVG, the color of a layer), but it could not “tell” the next command(s) about it.

This is no more, thanks to properties. They offer a generic mechanism to attach data to pipeline and layers. They are the backbone of several features introduced today, and lay the ground for future features within vpype or in plug-ins.

Layer color, pen width, and name

• Layer color, pen width, and name are now customizable (#359, #376, #389)
  • The read commands now sets layer color, pen width, and name based on the input SVG if possible.
  • The new color, penwdith, and name commands can be used to modify layer color, pen width, and name.
  • The new pens command can apply a predefined or custom scheme on multiple layers at once. Two common schemes are built-in: rgb and cmyk. Custom schemes can be defined in the configuration file.
  • The show and write commands now take into account these layer properties.

Supporting arbitrary layer colors, pen widths, and names, has long been amongst the most requested features. Well, thanks to properties, here they are. It happens automagically when using read, and the new commands can further customise these values:

$ vpype \
    rect --layer 1 0 0 5cm 5cm \
    color --layer 1 purple \
    penwidth --layer 1 0.5mm \
    circle --layer 2 4cm 4cm 2cm \
    color --layer 2 orange \
    penwidth --layer 2 5mm \
    show

The new, high-level color, penwidth, and name commands are simple wrappers which change the value of specific system properties (i.e. vp_color, vp_pen_width, resp. vp_name):

$ vpype random name "hello" color purple penwidth 0.1mm proplist --layer 1
listing 3 properties for layer 1
  vp_color: (color) #800080
  vp_name: (str) hello
  vp_pen_width: (float) 0.37795275590551186

System properties differ from “regular” properties only in the sense that they have special meaning to vpype. By convention, their name is prefixed with vp_.

Special mention for the new pens command, which is short for here is the set of pens I intend to use for this plot. It sets in bulk layer colors, pen widths and/or names all at once, based on a built-in or custom configuration. For example, this produces a CMYK SVG using the flow imager plug-in:

$ vpype \
    flow_img [...] --cmyk input.jpg \
    pens cmyk \
    write output.svg

• Introduced new commands for low-level inspection and modification of properties (#359)

  • propget: gets the value of a given global or layer property
  • proplist: lists all global and/or layer properties and their value
  • propset: sets the value of a given global or layer property
  • propdel: deletes a given global or layer property
  • propclear: removes all global and/or layer properties

These are low-level commands to interact with properties. Although they have limited use in real-world workflows, they come in handy when learning about properties or crafting complex pipelines.

• Updated layer operation commands to handle properties (#359)

  • When a single source layer is specified and --prob is not used, the lcopy and lmove commands now copy the source layer’s properties to the destination layer (possibly overwriting existing properties).
  • When --prob is not used, the lswap command now swaps the layer properties as well.
  • These behaviors can be disabled with the --no-prop option.

With properties, some of the layer manipulation commands became somewhat ambiguous. For example, what happens with properties when using lmove all 1 (merges all layers into layer one) or move --prob 0.5 1 2 (picks geometries from layer 1 with a 50% probability and moves them to layer 2)?

I opted for a strategy where layer properties are affected only for unambiguous cases. This is basically when a single layer is moved/copied and when probabilistic behaviour is not used at all. In all other cases, the layer properties are left unchanged.

From SVG attributes to properties

• Added the --attr option to the read command to (optionally) sort geometries by attributes (e.g. stroke color, stroke width, etc.) instead of by SVG layer (#378, #389)

• The read and write commands now preserve a sub-set of SVG attributes (experimental) (#359, #389)

  The read command identifies SVG attributes (e.g. stroke-dasharray) which are common in all geometries within each layer. These attributes are saved as layer properties with their name prefixed with svg_ (e.g. svg_stroke-dasharray). The write command can optionally restore these attributes in the output SVG using the --restore-attribs option.

As noted, the read command now tries to extract SVG attributes and store them as layer properties. There are two motivations for that. First, it enables the write command to optionally restore these attributes in the output file, in order to achieve a higher degree of fidelity. (This feature is experimental and opt-in for the time being.) Second, it enables future features or plug-ins to do neat things such as generating hatch fills when fill is set to a color, or cutting paths in bits to emulate stroke-dasharray if defined.

Now, since properties are only available at the layer level (or globally), read discards SVG attributes that are not shared amongst every paths within a given layer. Let’s take an example:

<svg xmlns="http://www.w3.org/2000/svg" width="650" height="650">
    <circle cx="150" cy="150" r="100" stroke="red" stroke-width="0.5mm" fill="green" />
    <rect x="400" y="200" width="200" height="400" stroke="blue" stroke-width="0.5mm" fill="green" />
    <path d="M250,600 l-200,0 l0,-200 z" stroke="blue" stroke-width="0.1mm" fill="green" />
</svg>

This SVG only contains top-level elements, which vpype loads in layer 1 by default. The fill property is common to all paths and thus stored as a layer property, but the stroke and stroke-width attributes are heterogeneous and thus discarded. As a result, the show command uses the default color and pen width.

$ vpype read example.svg proplist --layer 1 show
listing 1 properties for layer 1
  svg_fill: (str) green

To address this issue, the read command has now the option to create layers based on SVG attributes instead of structure:

$ vpype read --attr stroke --attr stroke-width example.svg proplist --layer all show
listing 5 properties for layer 1
  svg_fill: (str) green
  svg_stroke: (str) red
  svg_stroke-width: (str) 0.5mm
  vp_color: (color) #ff0000
  vp_pen_width: (float) 1.8897648
listing 5 properties for layer 2
  svg_fill: (str) green
  svg_stroke: (str) blue
  svg_stroke-width: (str) 0.5mm
  vp_color: (color) #0000ff
  vp_pen_width: (float) 1.8897648
listing 5 properties for layer 3
  svg_fill: (str) green
  svg_stroke: (str) blue
  svg_stroke-width: (str) 0.1mm
  vp_color: (color) #0000ff
  vp_pen_width: (float) 0.37795296000000006

In this case, read creates one layer per unique combination of stroke and stroke-width attribute, resulting in a total of three layers, each assigned with the correct properties, and correctly displayed by show.

Source files

• The read command now records the source SVG paths in the vp_source and vp_sources system properties (see the documentation) (#397, #406, #408)

The idea of the vp_source and vp_sources properties is to keep track of the files from which the content of the pipeline originates from. The vp_source property is a single path, which is overwritten by the last command importing from a file. The vp_sources property is a set of all source files encountered so far. Both properties are pathlib.Path instances.

This is, for example, what happens when using read twice:

$ vpype read machine_typography_01_3.svg read machine_typography_02_3.svg proplist -g
listing 5 global properties
  svg_fill: (str) black
  svg_stroke: (str) none
  vp_page_size: (tuple) (396.850608, 559.3703808000001)
  vp_source: (PosixPath) /private/tmp/MT/machine_typography_02_3.svg
  vp_sources: (set) {PosixPath('/private/tmp/MT/machine_typography_01_3.svg'), PosixPath('/private/tmp/MT/machine_typography_02_3.svg')}

Here, vp_source points to the file read by the last read command, but vp_sources contains all two source files.

Currently, read is the only command which sets these variables, but the idea is that any command involved with reading a file (SVG or otherwise) should set these properties, including plug-ins such as hatched, flow imager, or vpype-embroidery.

One of the most common use case is to name the output file after the input file:

$ vpype flow_img [...] my_image.png write "{vp_name.stem}_converted.svg"

Note the use of a property substitution pattern, which brings us to the next topic.

Property substitution

• Added property substitution to CLI user input (#395)

  The input provided to most commands' arguments and options may now contain substitution patterns which will be replaced by the corresponding property value. Property substitution patterns are marked with curly braces (e.g. {property_name}) and support the same formatting capabilities as the Python’s format() function.

This is where things start becoming “meta”! 🤯

As shown in the previous example, the value of a property may now be used anywhere as input using property substitution patterns.

Here is another example where the full path of the input file is drawn and displayed:

$ vpype read example.svg text -p 0.5cm 0.5cm "{vp_source}" show

Again, vp_source is a pathlib.Path instance, so {vp_source.name} (file name) or {vp_source.stem} (file name without extension) could be used instead.

Multiple substitution patterns can be combined and mixed with regular text. For example, this creates an output file in the same directory as, and named after, the input file:

$ vpype read example.svg linesort \
    write "{vp_source.parent}/{vp_source.stem}_optimised.svg"

Of course, when using vpype interactively, it’s easier to simply spell out the output file name. Instead, this kind of mechanism makes it considerably easier to write generic, reusable shell scripts.

Note that, since property substitution internally relies on Python’s str.format() function, the number formatting mini-language is available as well (e.g. {vp_pen_width:.02f}). See the documentation for some examples.

Now, taking a step back, this feature is neat indeed, but its usefulness turns out to be limited in many non-trivial, real-world scenarios. I had hoped it would unlock several workflows I had in mind, but it just did not - or not elegantly enough. So much so that I even considered dropping the feature altogether.

This was a bit frustrating, to say the least. And ultimately lead to what is the next big chapter of this release.

Expressions

• Added expression substitution to CLI user input (#397)

  The input provided to most command’s arguments and options may now contain expression patterns which are evaluated before the command is executed. Expression patterns are marked with the percent symbol % (e.g. %3+4%) and support a large subset of the Python language. A lot of examples were added in the cookbook.

This is possibly the most transformative feature brought to vpype since its inception: anything between pairs of % characters is now evaluated as (a sub-set of) Python code, and the result is substituted in the input before it reaches the actual command. The documentation has been updated with a whole new section about expressions (which I’m not going to repeat here), and the cookbook has plenty of examples making use of them. Do check them out for a taste of what expressions are capable of!

This feature blurs the lines between a mere CLI tool and a programming language. This begs the question of why not using a programming language in the first place, a point raised by fellow Python dev and flow imager author Jonas Serych. vpype even offers a proper API for that!

Here are my thoughts about this:

• Users of vpype are often not Python developers – or developers at all. Expressions build on existing vpype knowledge and bring, at least through examples and recipes that can be copy/pasted/customized, tiny bits of programs which are readily useful, without the need to learn much of the Python syntax and ecosystem.
• For many real-world cases (see the examples linked in the release notes), the resulting one-liners are more compact than the equivalent in proper code - even Python, even with vpype API. (Arbitrarily complex pipelines can of course be conceivably crafted as counter-examples, but this is besides the point.)

• Added the eval command as placeholder for executing expressions (#397)

Though expressions can be used in any command’s input, some “space” dedicated to them in the pipeline can be useful. Typical cases include variable initialization or querying the user for some parameter with the input() function. Several examples shown or linked below make use of this.

Block processors

• Improved block processors (#395, #397)

  • Simplified and improved the infrastructure underlying block processors for better extensibility.
  • The begin marker is now optional and implied whenever a block processor command is encountered. Note: the end marker must always be used to mark the end of a block.
  • Commands inside the block now have access to the current layer structure and its metadata.

Block processors hardly got any love since the first release of vpype and, as far as I can tell, weren’t used much - if at all - due to their limitations. Properties and expressions completely reverse this situation and block processors are now where the magic happens. The changes above lay the ground work for this.

• Improved the grid block processor (#397)

  • The page size is now updated according to the grid size.
  • The command now sets expression variables for use in the nested pipeline.
  • Cells are now first iterated along rows instead of columns.

• The repeat block processor now sets expression variables for use in the nested pipeline (#397)

• Added forfile block processor to iterate over a list of file (#397)

• Added forlayer block processor to iterate over the existing layers (#397)

• The read command now will ignore a missing file if --no-fail parameter is used (#397)

• Changed the initial default target layer to 1 (#395)

  Previously, the first generator command of the pipeline would default to create a new layer if the --layer option was not provided. This could lead to unexpected behaviour in several situation. The target layer is now layer 1. For subsequent generators, the existing behaviour of using the previous generator target layer as default remains.

That’s two new block processor commands, and another two finally coming to life, plus a few changes to make them work better with real-world workflows.

One of the key changes is that block processors now set temporary expression variables (prefixed with _) that can be used in the nested pipeline. They are listed in each command’s help text:

$ vpype grid --help
Usage: vpype grid [OPTIONS] NX NY

  Creates a NX by NY grid of geometry

  [...]

  The following variables are set by `grid` and available for expressions:

      _nx: number of columns (NX)
      _ny: number of rows (NY)
      _n: total number of cells (NX*NY)
      _x: current column (0 to NX-1)
      _y: current row (0 to NY-1)
      _i: current cell (0 to _n-1)
  
  [...]

Another novelty is the introduction of two new block processor commands:

• The forfile command accepts a pathname pattern (e.g. *.svg) and executes the nested pipeline for each of the paths it expends into. It makes things like batch processing files, merging multiple SVGs into a multilayer file, or laying out multiple files on a grid a breeze.
• The forlayer command executes the nested pipeline for each of the exising layers, which is useful, e.g., to export one file per layer.

Checks the related documentation for more details.

This example, taken from the grid layout recipe, demonstrates best what vpype 1.9 is about:

$ vpype \
   eval "files=glob('*.svg')" \
   eval "cols=6; rows=ceil(len(files)/cols)" \
   eval "names={};n=100" \
   grid -o 10cm 15cm "%cols%" "%rows%" \
       read --no-fail "%files[_i] if _i < len(files) else ''%" \
       layout -m 0.5cm 10x15cm \
       forlayer \
           eval "%if _name not in names: names[_name] = n; n = n+1%" \
           lmove %_lid% "%names[_name]%" \
       end \
   end \
   write combined.svg

It creates a grid layout from multiple SVG files, combining layers using their name (e.g. all “yellow” layers in input files are merged in a single “yellow” layer in the output file). Check the recipe for a detailed explanation.

This pipeline has it all:

• extensive use of expressions,
• two nested blocks, using their expression variables (prefixed with _),
• use of properties (via the _name variable set by forlayer, which contains the current layer’s vp_name property).

Here is how it looks when run on my Machine Typography #ptpx project:

Other changes

Note: This is the last version of vpype to support Python 3.7.

It’s the year of the walrus for vpype! 🦭

• Added pagerotate command, to rotate the page layout (including geometries) by 90 degrees (#404)

This command is useful for plotters without native support for both portrait and landscape orientations. Your plotter support only, say, landscape orientation and you want to plot a portrait-oriented file? This pipeline does the trick:

$ vpype read portrait_input.svg pagerotate wrote landscape_output.svg

• Added --keep option to the ldelete command (to delete all layers but those specified) (#383)

There was formerly no way to delete all layers but one. The new --keep option fills this gap.

• Pinned poetry-core to 1.0.8 to enable editable installs (#410)

Poetry finally supports editable installs thanks to PEP 660 🎉

This change is relevant when developing jointly on vpype and another project that depends on it (e.g. vsketch, or some plug-in). In such cases, vpype can now be installed in editable mode from a local checkout. Modifications made to it will immediately be available to the dependent project:

$ cd my-plugin
$ source venv/bin/activate
$ git clone https://github.com/abey79/vpype ../vpype
$ pip install -e ../vpype

• Fixed an issue with the random command when using non-square area (#395)

That’s a two-year-old bug I can’t believe I hadn’t seen before 🙄 (I use random a lot when testing out stuff during development.)

• Renamed the bundled config file to vpype_config.toml (#359)

This is the config file bundled with vpype. It used to be called hpgl_devices.toml, but now it also contains the build-in configurations of the new pens command (cmyk and rgb). The old name didn’t make sense anymore.

API changes:

• Moved all CLI-related APIs from vpype to vpype_cli (#388)
• Updated the block processor API (breaking change) (#395)
• …

This release comes with scores of changes at the API level (to many to list here). Two of these changes deserve a note though.

First, a fair amount of infrastructure used by the vpype CLI (e.g. the @generator decorator and friends) used to reside in the vpype package instead of vpype_cli. This is not ideal for many reasons and I’m moving away from it. vpype should be a “pure” library, whereas vpype_cli should contain everything needed for its CLI (and for plug-ins). These are not yet breaking changes but will generate deprecation warnings with most plug-ins. I will ensure that they are fixed ASAP.

Secondly, as part of the block processor overhaul, the @block_processor decorator had breaking changes without backward-compatible deprecation. I am not aware of any third-party code actually using it, so this shouldn’t cause any issue.

What’s next?

Congrats if you got this far! 😲🏆

I hope you’ll enjoy vpype 1.9 as much as I sweated preparing it. 😅

Feedback is welcome, via discussions for support/suggestions or issues for bugs. As always, I hang out on the drawingbots.net’s Discord server and am available for a chat.

Contributions are most welcome too, and the documentation is one area where help is always beneficial. I’ve gathered a few ideas of what can be done here.

To conclude, here are my probable areas of focus for the coming weeks/months:

• Given the scope of this release, I’m expecting to deal with increased user support and a few kirks and bugs to address. I’ll be available for this in the short term.
• Property-related stuff must be ported to vsketch, including a nice API to set layer name, color, pen-width, etc.
• The next big topic for vpype is its UX. At least, I want to improve the look and usability of the integrated help (using the brand new rich-click project), and add some visual feedback during execution.
• With support for Python 3.7 dropped, compatibility with Python 3.10 is now on the menu.
• Whatever user feedback might steer my attention to 😉

Lately, I’ve been meaning to document some of my personal projects and endeavours, and realised that I lacked a suitable outlet. I figured this would be a great occasion to test Hugo and GitHub Pages, so here we are!

This is what might ramble about in the future:

• Stuff about vpype, vsketch, and my other plotter-related software projects.
• Logs for some of my past and future projects (plotter-related or otherwise).
• Posts related to Home Assistant or other home-automation topics.
• Who knows what else.

Follow me on Twitter for updates.

I recently joined Rerun.io as software engineer to learn Rust 🦀 while developing visualisation frameworks and products for computer vision and robotics practitioners.

Earlier, with a few friends, I co-founded senseFly (now named AgEagle), which soon became the leader of fixed-wing mapping drones for surveyors, farmers, humanitarians and many others. Until my departure in 2018, I lead as CTO a team of up to 45 engineers and developed multiple generations of end-to-end mapping drone systems.

I am also involved in advisory capacity with several Swiss start-ups.

On my spare time, I dabble in generative art and pen plotters, and maintain two open-source projects I created in that space:

• vpype: a Swiss-Army-knife CLI to generate, manage, process, and convert vector graphic files for plotters,
• vsketch: a Python plotter generative art framework and workflow automation tool.

When I am not at the computer, you may find me piloting a helicopter 🚁 or behind the camera 📷 on my travels.

About this site

This site is built using Hugo and the PaperMod theme. It is hosted on GitHub Pages and deployed using a simple GitHub Actions workflow. Privacy-friendly and GDPR-compliant analytics services are provided by Plausible (a paid service, so neither you or I are the product).