Development

Extending Ubuntu via GNOME Shell Extensions in 2024

by Liv

Happy 2024, everyone! December ended up being a bit of a blur for me, because I unexpectedly launched MemoryCache, made it to the top of the orange website, and have since been exploring new ways to evolve the project ideas that I have related to embedding personal machine learning projects directly into my operating system.

📚 Read more: Why I’m auto-saving browser documents to augment local AI models and building MemoryCache

https://liverickson.com/blog/?p=330

Before I get into the technical bits that I want to document, the start of a new year feels like a good time to recap my philosophy on artificial intelligence:

  1. Proceed with caution
  2. Local is better the future

Since May, I’ve been reflecting on the metaphors of user agents and operating systems, and the places that machine learning capabilities can be fit into these existing types of software. While I’ve been working in the open source community for almost the entirety of my career, the scope of my roles previously were largely at the application and platform layers. In 2023, I set a goal for myself to broaden that in service of increasing leverage that I have as a technologist. Three of my major 2023 accomplishments were:

  1. Learning how to build the Firefox web browser from source, modify an API, and create an extension to build MemoryCache
  2. Upskill in python to fork, run, and tinker with an on-device RAG agent (privateGPT) on my local machine
  3. Switching from Windows to run Ubuntu as my primary operating system, and learn how to write gnome-shell extensions to customize my local compute environment

I grew up in the age where the personal computer (“PC”) was relegated to Windows computers thanks to catchy commercials from Apple. In practice, the “personal computing” market has dramatically shifted from the original visions evolving out of Xerox PARC and the innovation labs of the past into a multi-level mediated… mess. Thankfully, open source operating systems are thriving, and the barrier of entry into an actually personal computing experience that is tailored to your own environment is lower than ever.

With that in mind, in December, I started looking at how I could start to extend my Ubuntu desktop.

Building a GNOME-shell extension

In the past, I’ve tried a few different methods of managing some simple tasks and shortcuts to personalize my desktop, including AutoKey and Firefox extensions, but I was feeling an itch to cut out the additional layers of abstraction and go closer to the source. Eventually, I settled on figuring out how to write extensions for the GNOME Shell, which provides the windowing for some Linux systems, including Ubuntu.

GNOME shell extensions can add menus to the status area of the Ubuntu operating system. Often, applications will use this area to add quick-launch items. This menu bar, at least for how it’s set up on my machine, is always visible, which means that it was a great place to add two-click shortcuts for tasks that I do frequently.

Personal IOT with the Elgato Keylight

While my ultimate goal is to build an agent interface against a privateGPT instance on my machine, I started off with a utility that would control my Elgato light. This felt like an easier, smaller-scoped project to learn the basics of an extension, and it would add a significant amount of utility to me, given that I rarely used my key light before building this interface. You can check out the project and source code here.

The anatomy of a GNOME shell extension can be broken down into:

  • extension.js – where the core behaviors of the extension lives
  • metadata.json – where the descriptive elements of the extension live
  • schemas – a directory to hold the human-writeable and compiled XML that act as storage for the extension
    • schema_file.xml – define the variables that can be stored and read from
    • gschemas.compiled – the compiled schema file

For the Elgato Keylight extension, extension.js does several tasks:

Adds a button to the GNOME taskbar that opens a sub-menu of tasks that can be done

      // Set up the indicator button
      indicatorButtonText = new St.Label({
        text: 'Elgato',
        y_align: Clutter.ActorAlign.CENTER,
      });

      this.add_child(indicatorButtonText);

Loads in the settings from the compiled schema to restore the light from its last known state

let settings, temperature, brightness;

function getLightSettingsFromSchema() {
  let GioSchemaSource = Gio.SettingsSchemaSource;
  let schemaSource = GioSchemaSource.new_from_directory(
    Me.dir.get_child('schemas').get_path(),
    GioSchemaSource.get_default(),
    false
  );
  let schemaObj = schemaSource.lookup(SCHEMA_STRING, true);
  if (!schemaObj) {
    log('Error finding saved settings schema');
  }
  return new Gio.Settings({ settings_schema: schemaObj });
}

settings = getLightSettingsFromSchema();

brightness = settings.get_int('bright');
temperature = settings.get_int('temperature');

Set up the sub-menu with a button to turn the light on and two sliders to adjust the temperature and brightness of the light

toggleButton = new PopupMenu.PopupMenuItem('Turn light:');
      toggleLabel = new St.Label({ text: lightOn ? 'off' : 'on' });
      toggleButton.add_child(toggleLabel);
      this.menu.addMenuItem(toggleButton);
      toggleButton.connect('activate', () => {
        log ('Clicked brightness item');
        toggleLight();
      });

      let brightnessPanel = new PopupMenu.PopupBaseMenuItem({activate: false});
      brightnessPanel.add_child(new St.Label({text: 'Brightness: '}));
      let brightnessSlider = new Slider.Slider(brightness/100);
      brightnessPanel.add_child(brightnessSlider);
      this.menu.addMenuItem(brightnessPanel);
      brightnessSlider.connect('drag-end', () => {
        updateBrightness(brightnessSlider.value * 100);
      })

      let temperaturePanel = new PopupMenu.PopupBaseMenuItem({activate: false});
      temperaturePanel.add_child(new St.Label({text: 'Temperature: '}));
      let temperatureSlider = new Slider.Slider((temperature - 2900)/4100);
      temperaturePanel.add_child(temperatureSlider);
      this.menu.addMenuItem(temperaturePanel);
      temperatureSlider.connect('drag-end', () => {
        updateTemperature(temperatureSlider.value);
      });

      this.menu.connect('open-state-changed', (menu, open) => {
        if(open) {
          log ('opened menu');
        } else {
          log ('closed menu');
        }
      });

When the items in the sub-menu are changed, functions call the corresponding command line tasks to update the key light

// Toggle the light on or off
function toggleLight() {

  log('Calling toggleLight...');
  if (lightOn) {
    lightOn = false;
    GLib.spawn_command_line_sync('keylight-control --bright 0');
    toggleLabel.set_text("on");
  } else {
    lightOn = true;
    GLib.spawn_command_line_sync('keylight-control --bright ' + settings.get_int('bright'));
    toggleLabel.set_text("off");
  }
}

// Update the brightness of the light 
function updateBrightness(level) {
  if (!lightOn) {
    lightOn = true;
    toggleLabel.set_text("off");
  }
  brightness = level.toFixed(0);
  GLib.spawn_command_line_sync('keylight-control  --bright ' + brightness);
  settings.set_int('bright', brightness);
}

// Update the temperature of the light 
function updateTemperature(level) {
  if (!lightOn) {
    lightOn = true;
    toggleLabel.set_text("off");
  }

  let levelToKelvin = 2900 + level*4100;
  temperature = levelToKelvin.toFixed(0);
  GLib.spawn_command_line_sync('keylight-control  --temp ' + temperature);
  settings.set_int('temperature', temperature);
}

This program is a wrapper around a CLI tool that I had already installed (keylight-control) but it could easily be updated to interface with the light directly via the Elgato API, something I plan to do in the next few weeks to get more familiar with making API requests through extensions.

Setting up a Custom Browser Environment with privateGPT

My second exploration into GNOME extension authoring was to streamline the process of getting my personal build of Firefox up and running without having to use the command line.

📚 Read more: Why build a custom version of Firefox? It lets you test out new functionality before it makes its way to the actual browser, and gives you more control over how you want your browsing experience to be.

https://liverickson.com/blog/?p=246

Before creating my new extension, I would have to manually run the commands: 

cd mozilla-central
./mach run 
cd privateGPT
./run_ingest.sh

Sure, not a huge number of tasks, but it was enough of a pain that I would find myself deeply unwilling to restart my computer or browser. To be fair, I also had been dragging my feet on signing my custom extension, so there was an additional setup step of having to manually load in the extension through about:debugging every time I re-launched the browser. I’ve since remedied that.

This extension was even simpler than the Elgato Keylight extension, because it just:

Adds a button to the taskbar 

// Set up the indicator button
indicatorButtonText = new St.Label({
  text: 'Firefox',
  y_align: Clutter.ActorAlign.CENTER,
});

this.add_child(indicatorButtonText);

Adds a single “Launch Browser” button to the menu that drops down

// Add button to launch application
launchButton = new PopupMenu.PopupMenuItem('Launch Browser');
this.menu.addMenuItem(launchButton);
launchButton.connect('activate', () => {
  launchCustomFirefoxBuild();
  initializeDownloadIngestionListener();
});

Launches the custom build from the mozilla-central directory

function launchCustomFirefoxBuild() {
    Util.spawn(['./mozilla-unified/mach', 'run']);
}

Launches the run_ingest script from the privateGPT directory

function initializeDownloadIngestionListener() {
    Util.spawn(['./GitHub/privateGPT/run_ingest.sh']);
}

Lessons Learned & Looking Ahead

This was a really fun project to take on, and I’m itching to start doing more of this as I think through what kind of ways that I want to expand and customize my personal computing environment. One thing that I’m considering is figuring out how to make a UI for MemoryCache that lives in this part of my desktop, so that I can do things like:

  • Quickly verify that items I’m saving from my browser are showing up in the document store
  • Get access to new insights that are materializing from the documents
  • Implement a fast-recall feature to help me find a document that I’ve recently read

One of my biggest takeaways from this was that it was both easier to implement a GNOME shell extension than I thought it was, and also that it required me to practice a lot more patience than I usually have. The first extension took me a few days to build, and just getting the shell of the code to work and get the item added to the taskbar was a challenge. The GNOME shell APIs have changed from version 42 (the version that I’m using) and 45 (the latest version of gjs), so finding something that worked with my version of Ubuntu took me a while. Once I had that in place, though, it was much easier to understand how the system worked to draw the menu button, the items in the menu itself, and how to hook up event listeners to the different UI elements.

📚 Read more: One of the most helpful resources for putting together a basic GNOME shell extension that I found was a tutorial on Code Project, by user JustPerfection. Thanks JustPerfection!

https://www.codeproject.com/Articles/5271677/How-to-Create-A-GNOME-Extension

The second takeaway from this project was that it really helped to have someone holding me accountable in a gentle way – in this particular instance, they didn’t even realize that they were influencing my motivation to work on the project more! When I had the first piece of the Elgato extension done, it was just a single toggle feature, but I was excited and shared it to Mozilla Social and someone asked for a screenshot. Just being asked to share a screenshot motivated me to learn how to add the whole menu and sliders, and that light accountability was delightful.

Overall, I’m very excited to have this as another way of experimenting with computing. I am quite delighted with the idea that we’re pushing the frontier of casual software development out further and further, and look forward to continuing to explore this in the future.