Legibility is Ruining You

I am incredibly grateful that Sean Goedecke published his beautifully written article Seeing like a Software Company. In it, you will find a wonderfully balanced presentation of why companies often choose legible practices over more efficient, illegible ones. Sean very clearly details the kind of (in my words) doublethink that goes on at companies involving these very needed, but often hated, illegible processes. If you are hoping for that sort of balanced, nuanced take, please read his article instead. Sean has freed me from the burden of needing to write that way myself.

The Values You Truly Hold

Deep down on reflection, you don't care about OKRs, you don't care about code review, you don't care about coding standards, you don't care about lint rules, you don't care about style guides, you don't care about deadlines, you don't care about job titles, you don't care about standup, you don't care about code coverage, you don't care about any of these processes.

Your values are complicated. They require context and consideration. They aren't simple mechanical rules that can be captured by any metric, any process, or any methodology. You may believe these practices and tools we've adopted are pragmatically correct decisions to achieve your values. But they aren't the values themselves. And yet, they are the "values" that the "company" holds, or, more accurately, they are the values the company wants you to hold.

The Coercive Force of Legibility

Processes adopted by a company aren't about the end result of the work. Their stated goals and their actual goals are always distinct. An initiative related to applying coding standards isn't about making the quality of the code better. It is about changing the behavior of the engineers writing that code. By making code uniform, it reinforces some of the legibility myths (my word) Sean Goedecke says that companies rely on.

• Any engineers with the same job title perform roughly the same.
• Engineers can be shuffled and reorganized without substantial loss of productivity.
• A team will maintain the same level of productivity over time if it has the same number of engineers.

- Sean Goedecke: Seeing like a Software Company

These are all false. Everyone knows they are false. It is easy to think of these as idealized assumptions, but they aren't. You can't find physicists out in a field trying to shove a cow in an orb to make their equations work out better, but software engineering directors are more than happy to do so to software engineers.

Legible assumptions are not merely assumptions. They are coupled with practices trying to make their assumptions reality. This is first by placing barriers (processes), but it goes much deeper than that. Barriers can be circumvented. Gatekeepers can be ignored. The modern software enterprise has realized that the only true way to make its legible assumptions true is to change the very nature of those creating the software. To transform them from an illegible force to a legible one.

Value Capture

Every software process does the same thing: it converts a rich, multifaceted value into a single metric. "Are our customers happy with our product and support?" is not the same question as "how high is our NPS"? "Is this team performing well?" is not the same as "What is this team's 'velocity'?". "Are we working on the right stuff?" is not the same thing as "Do we have a prioritized backlog?". Yet we constantly mistake these questions for each other. Not only that, we don't even realize that these are conflations. Why? Because we are rewarded for fixing the latter and not the former.

The excuse always given for this disconnect is that the former questions don't work at "scale". But what precisely does that mean? If you have too many people, each divided into small groups (teams), they can't each ask these questions? No, these rich questions are actually quite easy to parallelize. What they don't do well is compose and reduce so that those at the top can understand them. Don't confuse something not working at "scale" with something that doesn't make an executive feel warm and fuzzy.

But the real reason for this conflation is not simply that it makes executives feel good, but that these metrics are an incredibly powerful means of changing behavior and values. In other words, they are a way of making the world fit the assumptions those at the top have already made.

Value capture occurs when an agent's values are rich and subtle; they enter a social environment that presents simplified — typically quantified — versions of those values; and those simplified articulations come to dominate their practical reasoning.

- Value Capture by C. Thi Nguyen

By making these metrics, the company can change how we behave. They can make our behavior properly legible.

Thin Rules, Thin Behavior

The sad fact is, as an industry, we long ago stopped only letting executives enforce rules on us; we enforce them on each other. We create rule after rule that we will make it so that "good software" is the necessary outcome. We all bring our own flavors of these rules. Our own dogmas we hold to, our own tastes we apply to others. We shame other engineers who do not follow our rules. The rules are the values.

They constantly try to escape

From the darkness outside and within

By dreaming of systems so perfect that no one will need to be good.

- T.S. Eliot

In Rules: A Short History of What We Live By, Lorraine Daston distinguishes between thick and thin rules. Thin rules are the algorithmic, mechanistic rules we in software favor; they are rules that can be obeyed and enforced unambiguously. Thick rules, on the other hand, are rules of thumb; they are guiding principles. Daston shows how there has been a historical progression from thick to thin rules. That today, we almost always think of rules as this thin, algorithmic process.

By following these thin rules, we not only produce worse software, we also steer our behavior to one that isn't sensitive to the real things we value. "Did the code pass code review?" is the wrong question. How much bad code have you seen pass code review? But the very fact that "passes code review" is the thin rule we have adopted ends our conversation. We can try to revise our "code review checklist" to "closer" approximate our true values, but this is a losing battle. What we care about can't be captured by our thin rules.

You

Legible processes are impossible to escape if you work for a company. Maybe they are necessary. Maybe companies of a given size must adopt them or fail. You don't have a choice but to follow some of these processes if you want to keep a job. But what is in your control is whether you let yourself be value captured by these metrics and processes.

You can choose to embrace the legible. As an IC, you can choose to value your work on the basis of these legible processes. You can set aside your own personal values about code and map your behavior to what the company cares about. As a manager, you can have your team follow the legible processes to create software that meets all "success" criteria, but still sucks.

Or you can choose to truly care about something. You can choose to push back against the legible processes that are trying to make you an abstraction. Not because you are selfish and stubborn. Not because you think you know better than others. Not because you don't want to follow rules. But because you value something that can't be captured in simple metrics. Because you know the company will benefit from these things. As Sean tells us, illegible work is essential. So why not be essential?

Untapped Way to Learn a Codebase: Build a Visualizer

The biggest shock of my early career was just how much code I needed to read that others wrote. I had never dealt with this. I had a hard enough time understanding my own code. The idea of understanding hundreds of thousands or even millions of lines of code written by countless other people scared me. What I quickly learned is that you don't have to understand a codebase in its entirety to be effective in it. But just saying that is not super helpful. So rather than tell, I want to show.

In this post, I'm going to walk you through how I learn an unfamiliar codebase. But I'll admit, this isn't precisely how I would do it today. After years of working on codebases, I've learned quite a lot of shortcuts. Things that come with experience that just don't translate for other people. So what I'm going to present is a reconstruction. I want to show bits and parts of how I go from knowing very little to gaining knowledge and ultimately, asking the right questions.

To do this, I will use just a few techniques:

1. Setting a goal
2. Editing randomly
3. Fixing things I find that are broken
4. Reading to answer questions
5. Making a visualizer

I want to do this on a real codebase, so I've chosen one whose purpose and scope I'm generally familiar with. But one that I've never contributed to or read, Next.js. But I've chosen to be a bit more particular than that. I'm particularly interested in learning more about the Rust bundler setup (turbopack) that Next.js has been building out. So that's where we will concentrate our time.

Being Clear About Our Goal

Trying to learn a codebase is distinctly different from trying to simply fix a bug or add a feature. In post, we may use bugs, talk about features, make changes, etc. But we are not trying to contribute to the codebase, yet. Instead, we are trying to get our mind around how the codebase generally works. We aren't concerned with things like coding standards, common practices, or the development roadmap. We aren't even concerned with correctness. The changes we make are about seeing how the codebase responds so we can make sense of it.

Starting Point

I find starting at main to be almost always completely unhelpful. From main, yes, we have a single entry point, but now we are asking ourselves to understand the whole. But things actually get worse when dealing with a large codebase like this. There isn't even one main. Which main would we choose? So instead, let's start by figuring out what our library even consists of.

❯ ls -1d */
apps/
bench/
contributing/
crates/
docs/
errors/
examples/
packages/
patches/
rspack/
scripts/
test-config-errors/
test/
turbo/
turbopack/

A couple of things to note. We have packages, crates, turbo, and turbopack. Crates are relevant here because we know we are interested in some of the Rust code, but we are also interested in turbopack in particular. A quick look at these shows that turbo, packages, and crates are probably not our target. Why do I say that? Because turbopack has its own crates folder.

❯ ls -d turbopack/crates/* | wc -l
      54

So there are 54 crates under turbopack.... This is beginning to feel a bit daunting. So why don't we take a step back and find a better starting point? One starting point I find particularly useful is a bug report.

I found this by simply looking at recently opened issues. When I first found it, it had no comments on it. In fact, I find bug reports with only reproducing instructions to be the most useful. Remember, we are trying to learn, not fix a bug.

So I spent a little time looking at the bug report. It is fairly clear. It does indeed reproduce. But it has a lot of code. So, as is often the case, it is useful to reduce it to the minimal case. So that's what I did. Here is the important code and the problem we are using to learn from.

// app.ts
import { greeting } from './utility'

export default function Home() {
  return (
    <div>${"greeting"} World</div>
  );
}

// utility.ts
export enum MyEnum {
  UNIQUE_1 = 'UNIQUE_1',
  UNIQUE_2 = 'UNIQUE_2',
}

export const greeting = "Hello";

MyEnum here is dead code. It should not show up in our final bundle. But when we do next build and look for it, we get:

❯ rg --no-ignore --hidden -g '!*.map' UNIQUE_1
app/utility.ts
2: UNIQUE_1 = 'UNIQUE_1',

.next/server/chunks/ssr/[root-of-the-server]__5f0b8324._.js
1:module.exports=[93695,(a,b,c)=>{b.exports=a.x("next/dist/shared/lib/...

If we instead do next build --webpack

❯ rg --no-ignore --hidden -g '!*.map' UNIQUE_1
app/utility.ts
2: UNIQUE_1 = 'UNIQUE_1',

The code is completely gone from our build.

Setting an Agenda

So now we have our bug. But remember. Our goal here is not to fix the bug. But to understand the code. So our goal is going to be to use this little mini problem to understand what code is involved in this bug. To understand the different ways we could fix this bug. To understand why this bug happened in the first place. To understand some small slice of the turbopack codebase.

So at each junction, we are going to resist the urge to simply find the offending code. We are going to take detours. We are going to ask questions. We hope that from the start of this process to the end, we no longer think of the code involved in this action as a black box. But we will intentionally leave ourselves with open questions. As I write these words, I have no idea where this will take us. I have not prepared this ahead of time. I am not telling you a fake tale from a codebase I already know. Yes, I will simplify and skip parts. But you will come along with me.

Finding Tree Shaking

The first step for understanding any project is getting some part of it running. Well, I say that, but in my day job, I've been at companies where this is a multi-day or week-long effort. Sometimes, because of a lack of access, sometimes from unclear instructions, if you find yourself in that situation, you now have a new task, understand it well enough to get it to build. Well, unfortunately, that is the scenario we find ourselves in.

The First Side Quest

I can't think of a single one of these endeavors I've gone on to learn a codebase that didn't involve a completely undesirable, momentum-stopping side quest. For this one, it was as soon as I tried to make changes to the turbopack Rust code and get it working in my test project. There are instructions on how to do this. In fact, we even get an explanation on why it is a bit weird.

Since Turbopack doesn't support symlinks when pointing outside of the workspace directory, it can be difficult to develop against a local Next.js version. Neither pnpm link nor file: imports quite cut it. An alternative is to pack the Next.js version you want to test into a tarball and add it to the pnpm overrides of your test application. The following script will do it for you:

pnpm pack-next --tar && pnpm unpack-next path/to/project

Okay, straightforward enough. I start by finding somewhere in the turbopack repo that I think will be called more than once, and I add the following:

diff --git a/turbopack/crates/turbopack/src/lib.rs b/turbopack/crates/turbopack/src/lib.rs
index 20891acf86..8691f1951c 100644
--- a/turbopack/crates/turbopack/src/lib.rs
+++ b/turbopack/crates/turbopack/src/lib.rs
@@ -80,6 +80,7 @@ async fn apply_module_type(
     css_import_context: Option<ResolvedVc<ImportContext>>,
     runtime_code: bool,
 ) -> Result<Vc<ProcessResult>> {
+ println!("HERERE!!!!!!!");
     let tree_shaking_mode = module_asset_context
         .module_options_context()
         .await?

Yes. Very scientific, I know. But I've found this to be a rather effective method of making sure my changes are picked up. So I do that, make sure I've built and done the necessary things. I run

pnpm swc-build-native && pnpm pack-next --tar && pnpm unpack-next /path/to/test/project

Then that script tells me to add some overrides and dependencies to my test project. I go to build my project and HERERE!!!!!!! does not show up at all...

Understanding the Build System

I will save you the fun details here of looking through this system. But I think it's important to mention a few things. First, @next/swc being a dependency immediately stood out to me. In my day job, I maintain a fork of swc (WHY???) for some custom stuff. I definitely won't pretend to be an expert on swc, but I know it's written in Rust. I know it's a native dependency. The changes I'm making are native dependencies. But I see no mention at all of turbopack. In fact, if I search in my test project, I get the following:

❯ fd --no-ignore swc
node_modules/@next/swc

❯ fd --no-ignore turbopack
# empty

So I have a sneaking suspicion my turbopack code should be in that tar. So let's look at the tar.

❯ du -h next-swc.tar
12k

Ummm. That seems a bit small... Let's look at what's inside.

❯ tar -tf next-swc.tar
./README.md
./package.json

Okay, I think we found our problem. There's really nothing in this at all. Definitely no native code.

Regex - Now You Have Two Problems

After lots of searching, the culprit came down to:

const packageFiles = [...globbedFiles, ...simpleFiles].sort()
const set = new Set()
return packageFiles.filter((f) => {
  if (set.has(f)) return false
  // We add the full path, but check for parent directories too.
  // This catches the case where the whole directory is added
  // and then a single file from the directory.
  // The sorting before ensures that the directory comes before
  // the files inside of the directory.
  set.add(f)
  while (f.includes('/')) {
    f = f.replace(/\/[^/]+$/, '')
    if (set.has(f)) return false
  }
  return true
})

In our case, the input came from this file and f was "native/". Unfortunately, this little set + regex setup causes native/ to be filtered out. Why? Because it doesn't match the regex. This regex is looking for a / with characters after it. We have none. So since we are already in the set (we just added ourselves), we filter ourselves out.

How do we solve this problem? There are countless answers, really. I had Claude whip me up one without regex.

const packageFiles = [...globbedFiles, ...simpleFiles].sort()
const set = new Set<string>()
return packageFiles.filter((f) => {
  const normalized = join(f)
  if (set.has(normalized)) return false
  set.add(normalized)
  let parent = dirname(normalized)
  while (parent !== '.' && parent !== '/') {
    if (set.has(parent)) return false
    parent = dirname(parent)
  }
  return true
})

But my gut says the sorting lets us do this much simpler. But after this fix, let's look at the tar now:

❯ tar -tf next-swc.tar
./README.md
./native/
./native/next-swc.darwin-arm64.node
./native/index.d.ts
./package.json

Much better. After this change, we can finally see HERERE!!!!!!! a lot.

Update: As I wrote this article, someone fixed this in a bit of a different way. Keeping the regex and just changing to *. Fairly practical decision.

For Real This Time?

Okay, we now have something we can test. But where do we even begin? This is one reason we chose this bug. It gives a few avenues to go down. First, the report says that these enums are not being "tree-shaken." Is that the right term? One thing I've learned from experience is to never assume that the end user is using terms in the same manner as the codebase. So this can be a starting point, but it might be wrong.

With some searching around, we can actually see that there is a configuration for turning turbopackTreeShaking on or off.

const nextConfig: NextConfig = {
  experimental: {
    turbopackTreeShaking: true,
  },
};

It was actually a bit hard to find exactly where the default for this was. It isn't actually documented. So let's just enable it and see what we get.

> Build error occurred
Error [TurbopackInternalError]: Failed to write app endpoint /page

Caused by:
- index out of bounds: the len is 79 but the index is 79

Debug info:
- Execution of get_all_written_entrypoints_with_issues_operation failed
- Execution of EntrypointsOperation::new failed
- Execution of all_entrypoints_write_to_disk_operation failed
- Execution of Project::emit_all_output_assets failed
- Execution of *emit_assets failed
- Execution of all_assets_from_entries_operation failed
- Execution of *all_assets_from_entries failed
- Execution of output_assets_operation failed
- Execution of <AppEndpoint as Endpoint>::output failed
- Failed to write app endpoint /page
- Execution of AppEndpoint::output failed
- Execution of whole_app_module_graph_operation failed
- Execution of *ModuleGraph::from_single_graph failed
- Execution of *SingleModuleGraph::new_with_entries failed
- Execution of Project::get_all_entries failed
- Execution of <AppEndpoint as Endpoint>::entries failed
- Execution of get_app_page_entry failed
- Execution of *ProcessResult::module failed
- Execution of <ModuleAssetContext as AssetContext>::process failed
- Execution of EcmascriptModulePartAsset::select_part failed
- Execution of split_module failed
- index out of bounds: the len is 79 but the index is 79

Well, I think we figured out that the default is off. So one option is that we never "tree shake" anything. But that seems wrong. At this point, I looked into tree shaking a bit in the codebase, and while I started to understand a few things, I've been at this point before. Sometimes it is good to go deep. But how much of this codebase do I really understand? If tree shaking is our culprit (seeming unlikely at this point), it might be good to know how code gets there. Here, we of course found a bug. But it is an experimental feature. Maybe we can come back and fix it? Maybe we can file a bug? Maybe this code just isn't at all ready for primetime. It's hard to know as an outsider.

How a Chunk is Made

Our "search around the codebase" strategy failed. So now we try a different tactic. We know a couple of things.

1. Our utilities.ts file is read and parsed.
2. It ends up in a file under a "chunks" directory.

We now have two points we can use to try to trace what happens. Let's start with parsing. Luckily, here it is straightforward: parse_file_content. When we look at this code, we can see that swc does the heavy lifting. First, it parses it into a TypeScript AST, then applies transforms to turn it into JavaScript. At this point, we don't write to a string, but if you edit the code and use an emitter, you see this:

export var MyEnum = /*#__PURE__*/ function(MyEnum) {
    MyEnum["UNIQUE_1"] = "UNIQUE_1";
    MyEnum["UNIQUE_2"] = "UNIQUE_2";
    return MyEnum;
}({});
export const greeting = "Hello!";

Now, to find where we write the chunks. In most programs, this would be pretty easy. Typically, there is a linear flow somewhere that just shows you the steps. Or if you can't piece one together, you can simply breakpoint and follow the flow. But Turbopack is a rather advanced system involving async Rust (more on this later). So, in keeping with the tradition of not trying to do things that rely too heavily on my knowledge, I have done the tried and true, log random things until they look relevant. And what I found made me realize that logging was not going to be enough. It was time to do my tried and true learning technique, visualization.

Building a Visualizer

Ever since my first job, I have been building custom tools to visualize codebases. Perhaps this is due to my aphantasia. I'm not really sure. Some of these visualizers make their way into general use for me. But more often than not, they are a means of understanding. When I applied for a job at Shopify working on YJIT, I built a simple visualizer but never got around to making it more useful than a learning tool. The same thing is true here, but this time, thanks to AI, it looks a bit more professional.

This time, we want to give a bit more structure than what we'd be able to do with a simple print.¹ We are trying to get events out that have a bunch of information. Mostly, we are interested in files and their contents over time. Looking through the codebase, we find that one key abstract is an ident; this will help us identify files. We will simply find points that seem interesting, make a corresponding event, make sure it has idents associated with it, and send that event over a WebSocket. Then, with that raw information, we can have our visualizer stitch together what exactly happens.

If we take a look, we can see our code step through the process. And ultimately end up in the bundle despite not being used. If you notice, though, between steps 3 and 4, our code changed a bit. We lost this PURE annotation. Why?

Luckily, because we tried to capture as much context as we could. We can see that a boolean "Scope Hoisting" has been enabled. Could that be related? If we turn it off, we instead see:

Our pure annotation is kept around, and as a result, our code is eliminated. If we take a step back, this can make sense. Something during the parse step is creating a closure around our enum code, but when it does so, it is marking that as a "pure" closure, meaning it has no side effects. Later, because this annotation is dropped, the minifier doesn't know that it can just get rid of this closure. As I've been trying to find time to write this up, it seems that people on the bug report have found this on their own as well.

Fixing the Bug

So we've found the behavior of the bug. Now we need to understand why it is happening. Remember, we are fixing a bug as a means of understanding the software. Not just to fix a bug. So what exactly is going on? Well, we are trying to stitch together two libraries. Software bugs are way more likely to occur on these seams. In this case, after reading the code for a while, the problem becomes apparent.

SWC parses our code and turns it into an AST. But if you take a look at an AST, comments are not a part of the AST. So instead, SWC stores comments off in a hashmap where we can look them up by byte pos. So for each node in the AST, it can see if there is a comment attached. But for the PURE comment, it doesn't actually need to look this comment up. It is not a unique comment that was in the source code. It is a pre-known meta comment. So rather than store each instance in the map, it makes a special value.

/// Reserved for PURE comments. e.g. `/* #__PURE__ */`
pub const PURE: Self = BytePos(u32::MAX - 1);

Now, this encoding scheme causes some problems for turbopack. Turbopack does not act on a single file; it acts across many files. In fact, for scope hoisting, we are trying to take files across modules and condense them into a single scope. So now, when we encounter one of these bytepos encodings, how do we know what module it belongs to?

The obvious answer to many might be to simply make a tuple like (ModuleId, BytePos), and while that certainly works, it does come with tradeoffs. One of these is memory footprint. I didn't find an exact reason. But given the focus on performance on turbopack, I'd imagine this is one of the main motivations. Instead, we get a fairly clever encoding of module and bytepos into a single BytePos, aka a u32. I won't get into the details of the representation here; it involves some condition stuff. But needless to say, now that we are going from something focusing on one file to focusing on multiple and trying to smuggle in this module_id into our BytePos, we ended up missing one detail, PURE. Now our pure value is being interpreted as some module at some very high position instead of the proper bytes.

To fix this bug, I found the minimal fix was simply the following:

-        if pos.is_dummy() {
+        if pos.is_dummy() || pos.is_pure() {
             // nothing to encode
             return Ok(pos);
         }

With this our enum properly is marked as PURE and disappears from the output!

Now remember, we aren't trying to make a bug fix. We are trying to understand the codebase. Is this the right fix? I'm not sure. I looked around the codebase, and there are a number of other swc sentinel values that I think need to also be handled (PLACEHOLDER and SYNTHESIZED). There is also the decoding path. For dummy, the decoding path panics. Should we do the same? Should we be handling pure at a higher level, where it never even goes through the encoder?

Update: As I was writing this, someone else proposed a fix. As I was writing the article, I did see that others had started to figure out the things I had determined from my investigation. But I was not confident enough that it was the right fix yet. In fact, the PR differs a bit from my local fix. It does handle the other sentinel, but at a different layer. It also chooses to decode with module 0. Which felt a bit wrong to me. But again, these are decisions that people who work on this codebase long-term are better equipped to decide than me.

I'm Still Lost

I must admit that simply fixing this bug didn't quite help me understand the codebase. Not just because it is a fairly good size. But because I couldn't see this fundamental unit that everything was composed of. In some of the code snippets above, you will see types that mention Vc. This stands for ValueCell. There are a number of ways to try to understand these; you can check out the docs for turbo engine for some details. Or you can read the high-level overview that skips the implementation details for the most part. You can think of these cells like the cells in a spreadsheet. They provide a level of incremental computation. When the value of some cell updates, we can invalidate stuff. Unlike a spreadsheet, the turbo engine is lazy.

I've worked with these kinds of systems before. Some were very explicitly modeled after spreadsheets. Others were based on rete networks or propagators. I am also immediately reminded of salsa from the Rust analyzer team. I've even worked with big, complicated non-computational graphs. But even with that background, I know myself, I've never been able to really understand these things until I can visualize them. And while a general network visualizer can be useful (and might actually be quite useful if I used the aggregate graph), I've found that for my understanding, I vastly prefer starting small and exploring out the edges of the graph. So that's what I did.

A General Visualizer

But before we get to that visualization, I want to highlight something fantastic in the implementation: a central place for controlling a ton of the decisions that go into this system. The backend here lets us decide so many things about how the execution of our tasks will run. Because of this, we have one place we can insert a ton of tooling and begin to understand how this system works.

As before, we are going to send things on a WebSocket. But unlike last time, our communication will actually be two-way. We are going to be controlling how the tasks run.

In my little test project, I edited a file, and my visualizer displayed the following. Admittedly, it is a bit janky, but there are some nice features. First, on the left, we can see all the pending tasks. In this case, something has marked our file read as dirty, so we are trying to read the file. We can see the contents of a cell that this task has. And we can see the dependents of this task.

Here is what it looks like once we release that task to run. We can now see 3 parse tasks have kicked off. Why 3? I'll be honest, I haven't looked into it. But a good visualization is about provoking questions, not only answering them. Did I get my visualization wrong because I misunderstood something about the system? Are there three different subsystems that want to parse, and we want to do them in parallel? Have we just accidentally triggered more parses than we should be?

This is precisely what we want out of a visualizer. Is it perfect? No, would I ship this as a general visualizer? No. Am I happy with the style? Not in the least. But already it enables a look into the project I couldn't see before. Here we can actually watch the graph unfold as I execute more steps. What a fascinating view of a once opaque project.

With this visualizer, I was able to make changes to my project and watch values as they flow through the systems. I made simple views for looking at code. If I extended this, I can imagine it being incredibly useful for debugging general issues, for seeing the ways in which things are scheduled, and for finding redundancies in the graph. Once I was able to visualize this, I really started to understand the codebase better. I was able to see all the values that didn't need to be recomputed when we made changes. The whole thing clicked.

Conclusion

This was an exercise in exploring a new codebase that is a bit different of a process than I see others take. It isn't an easy process, it isn't quick. But I've found myself repeating this process over and over again. For the turborepo codebase, this is just the beginning. This exploration was done over a few weekends in the limited spare time I could find. But already I can start to put the big picture together. I can start to see how I could shape my tools to help me answer more questions. If you've never used tool building as a way to learn a codebase, I highly recommend it.

Coda: A Call for More Interactive Dynamic Systems

One thing I always realize as I go through this process is just how hard it is to work interactively with our current software. Our languages, our tools, our processes are all written without ways to live code, without ways to inspect their inner workings.

It is also incredibly hard to find a productive UI environment for this kind of live exploration. The running state of the visualizer contains all the valuable information. Any system that needs you to retrace your steps to get the UI back to the state it was once in to visualize more is incredibly lacking. So I always find myself in the browser, but immediately, I am having to worry about performance. We have made massive strides in so many aspects of software development. I hope that we will fix this one as well.

AI Has Made it Easy to Own Your Tools

I am a digital hoarder. Probably not an extreme one. I don't have some fancy complicated raid system for endless terabytes of media. But when I find a link or a pdf to some interesting article, I want to save it. But I almost never review it. That isn't to say I don't read articles. I read plenty. But I also collect them. But I've always had a problem: how do I organize them, classify them, keep them?

For links, my answer was pocket. Well, you can imagine how that went. My exported list is living in Raindrop for the time being. My PDFs had been living in Muse. A nice piece of software, but it has diverged from the minimalism I enjoyed about it. So I've been on the search for alternatives. But there has always been this nagging feeling that what I want is not just some different piece of software, but a custom, fully owned setup for myself. But where would I ever find the time to do that?

Some AI tooling

I admit that I'm far from having the tools I want. The tools I've built so far are rudimentary. But they were built in minutes, and they were exactly what I asked for. And they did things that a few years ago would not have been possible.

Tool 1 - Local LLM Sorting

Local LLMs are getting better and better. Seeing that trend, I bought a framework desktop. Using it, I was able to have Claude write a quite simple script that found every pdf on my machine, grabbed some of the initial text from them, and passed them to gpt-120b and asked, is this pdf about programming? Now I can find all those programming PDFs. But I needed to sort them.

Tool 2 - A Swift Application

Now that I have the initial collection of potential PDFs. How was I going to sort them? I didn't want a tagging system. There's a chance that later I will. But for now, I wanted discrete categories. But what categories? I'd only find out once I started organizing them. So I asked Claude and got an app specifically designed to let me categorize them.

Tool 3 - PDF sync

A simple tool for syncing PDFs by hash to S3.

Tool 4 - PDF indexer

Some of the PDFs have nice metadata built in. So we can just go ahead and extract that. That's why this tool.

Tool 5 - LLM OCR metadata extractor

For the rest, we pass to Qwen3-VL-30B to grab the title and author.

Tool 6

A swift application compiled for my Mac and iPad that lets me annotate PDFs.

The app is far from fully featured yet. But the fact that it syncs between my Mac and iPad seamlessly is wonderful. I use this mainly for the podcast, so I haven't gotten to do a ton with it yet. But having it be something I can customize to my needs already has me excited.

Tool 7

A page on my website that statically generates the archive for all to browse.

The Unsung Hero

We've yet to reach the point where local models can quite replace Claude for coding. But having a local model where I never had to worry about the cost to send it experiments, one where it ran not on my laptop, but in the background on a "server", was such an enabling feature. I would have been very hesitant to send all these PDFs to a model. But with a local model, I had so much flexibility to use it wherever judgment could be a substitute for deterministic processing.

Conclusion

Nothing here was groundbreaking. Nothing here is something I couldn't have made myself. But they are all things I put off. All things I would have never prioritized, but wanted made. They are all imperfect tools. Many of them are one-offs. There were actually countless other small tools made along the way. Cleaning up titles, a tool that chose between the metadata title and the OCR ones (ocr were usually better). Any one of these little bottlenecks might have been enough for me to stop working on this project.

I see lots of discussions about AI all having to do with "production code". I'm sure I'll write my thoughts about that at some point. But I also think it is important that we remember that this isn't the only code that exists. In this case, it's personal code. Code, I enjoy having the ability to modify. Code that I am happy isn't robust; it doesn't try to handle every edgecase. It doesn't need a complicated sync process. Doesn't need granular permissions.

This is just the start of what I expect to be a continual evolution of my pdf (and eventually link) management software. But for the first time in my programming life (not career, not everything is a business transaction), I don't feel the weight of maintenance I've created for myself. I feel a sense of freedom to build more without incurring such a heavy cost. This, to me, is one of the most exciting features of our new AI capabilities.

The Easiest Way to Build a Type Checker

Type checkers are a piece of software that feel incredibly simple, yet incredibly complex. Seeing Hindley-Milner written in a logic programming language is almost magical, but it never helped me understand how it was implemented. Nor does actually trying to read anything about Algorithm W or any academic paper explaining a type system. But thanks to David Christiansen, I have discovered a setup for type checking that is so conceptually simple it demystified the whole thing for me. It goes by the name Bidirectional Type Checking.

Bidirectional Type Checking

The two directions in this type checker are inferring types and checking types. Unlike Hindley-Milner, we do need some type annotations, but these are typically at function definitions. So code like the sillyExample below is completely valid and fully type checks despite lacking annotations. How far can we take this? I'm not a type theory person. Reading papers in type theory takes me a while, and my comprehension is always lacking, but this paper seems like a good starting point for answering that question.

function sillyExample(x: number): number {
  let a = 10;
  let b = 20;
  let e = a;
  let f = b;
  let q = a + e;
  let g = "hello";
  let h = "world";
  let i = 100 + q;
  return x;
}

So, how do we actually create a bidirectional type checker? I think the easiest way to understand it is to see a full working implementation. So that's what I have below for a very simple language. To understand it, start by looking at the types to figure out what the language supports, then look at each of the infer cases. But don't worry, if it doesn't make sense, I will explain in more detail below.

export type Type =
  | { kind: "number" }
  | { kind: "string" }
  | { kind: "function"; arg: Type; returnType: Type };

export type Expr =
  | { kind: "number"; value: number }
  | { kind: "string"; value: string }
  | { kind: "varLookup"; name: string }
  | { kind: "function"; param: string; body: Expr }
  | { kind: "call"; fn: Expr; arg: Expr }
  | { kind: "let"; name: string; value: Expr; type?: Type }
  | { kind: "block"; statements: Expr[]; return: Expr };

export type Context = Map<string, Type>;

export function infer(ctx: Context, expr: Expr): Type {
  switch (expr.kind) {
    case "number":
      return { kind: "number" };

    case "string":
      return { kind: "string" };

    case "varLookup":
      const type = ctx.get(expr.name);
      if (!type) {
        throw new Error(`Unbound variable: ${expr.name}`);
      }
      return type;

    case "call":
      const fnType = infer(ctx, expr.fn);
      if (fnType.kind !== "function") {
        throw new Error("Cannot call non-function");
      }
      check(ctx, expr.arg, fnType.arg);
      return fnType.returnType;

    case "function":
      throw new Error("Cannot infer type for function without annotation");

    case "let":
      const valueType = infer(ctx, expr.value);
      if (expr.type) {
        if (!typesEqual(valueType, expr.type)) {
          let expected = JSON.stringify(expr.type);
          let actual = JSON.stringify(valueType);
          throw new Error(`expected ${expected}, got ${actual}`);
        }
      }
      ctx.set(expr.name, valueType);
      return valueType;

    case "block":
      let blockCtx = new Map(ctx);
      for (const stmt of expr.statements) {
        infer(blockCtx, stmt);
      }
      return infer(blockCtx, expr.return);
  }
}

export function check(ctx: Context, expr: Expr, expected: Type): void {
  switch (expr.kind) {
    case "function":
      if (expected.kind !== "function") {
        throw new Error("Function must have function type");
      }
      const newCtx = new Map(ctx);
      newCtx.set(expr.param, expected.arg);
      check(newCtx, expr.body, expected.returnType);
      break;

    case "block":
      let blockCtx = new Map(ctx);
      for (const stmt of expr.statements) {
        infer(blockCtx, stmt);
      }
      check(blockCtx, expr.return, expected);
      break;

    default:
      const actual = infer(ctx, expr);
      if (!typesEqual(actual, expected)) {
        throw new Error(`Type mismatch: expected ${expected}, got ${actual}`);
      }
  }
}

export function typesEqual(a: Type, b: Type): boolean {
  if (a.kind !== b.kind) return false;
  if (a.kind === "function" && b.kind === "function") {
    return typesEqual(a.arg, b.arg) && typesEqual(a.returnType, b.returnType);
  }
  return true;
}

Here we have, in ~100 lines, a fully functional type checker for a small language. Is it without flaw? Is it feature complete? Not at all. In a real type checker, you might not want to know only if something typechecks, but you might want to decorate the various parts with their type; we don't do that here. We don't do a lot of things. But I've found that this tiny bit of code is enough to start extending to much larger, more complicated code examples.

Explanation

If you aren't super familiar with the implementation of programming languages, some of this code might strike you as a bit odd, so let me very quickly walk through the implementation. First, we have our data structures for representing our code:

export type Type =
  | { kind: 'number' }
  | { kind: 'string' }
  | { kind: 'function', arg: Type, returnType: Type }

export type Expr =
  | { kind: 'number', value: number }
  | { kind: 'string', value: string }
  | { kind: 'varLookup', name: string }
  | { kind: 'function', param: string, body: Expr }
  | { kind: 'call', fn: Expr, arg: Expr }
  | { kind: 'let', name: string, value: Expr, type?: Type }
  | { kind: 'block', statements: Expr[], return: Expr }

Using this data structure, we can write code in a way that is much easier to work with than the actual string that we use to represent code. This kind of structure is called an "abstract syntax tree". For example

// double(5)
{
  kind: 'call',
  fn: { kind: 'varLookup', name: 'double' },
  arg: { kind: 'number', value: 5 }
}

This structure makes it easy to walk through our program and check things bit by bit.

Context

export type Context = Map<string, Type>

This simple line of code is the key to how all variables, all functions, etc, work. When we enter a function or a block, we make a new Map that will let us hold the local variables and their types. We pass this map around, and now we know the types of things that came before it. If we wanted to let you define functions out of order, we'd simply need to do two passes over the tree. The first to gather up the top-level functions, and the next to type-check the whole program. (This code gets more complicated with nested function definitions, but we'll ignore that here.)

Inference

Each little bit of infer may seem a bit trivial. So, to explain it, let's add a new feature, addition.

// add this into our Expr type
| { kind: 'add', left: Expr, right: Expr }

Now we have something just a bit more complicated, so how would we write our inference for this? Well, we are going to do the simple case; we are only allowed to add numbers together. Given that our code would look something like this:

case 'add':
  const leftType = check(ctx, expr.left, {kind: "number"})
  const rightType = check(ctx, expr.right, {kind: "number"})
  return {kind: "number"};

This may seem a bit magical. How does check make this just work? Imagine that we have the following expression:

// 2 + 3 + 4
 {
    kind: 'add',
    left: {
      kind: 'add',
      left: { kind: 'number', value: 2 },
      right: { kind: 'number', value: 3 }
 },
    right: { kind: 'number', value: 4 }
 }

There is no special handling in check for add so we end up at

default:
  const actual = infer(ctx, expr)
  if (!typesEqual(actual, expected)) {
    throw new Error(`Type mismatch: expected ${expected}, got ${actual}`)
 }

If you trace out the recursion (once you get used to recursion, you don't actually need to do this, but I've found it helps people who aren't used to it), we get something like

 infer(2 + 3 + 4)
    check(2 + 3, number)
      infer(2 + 3)
        check(2, number)
          infer(2) → number
        check(3, number)
          infer(3) → number
    check(4, number)
      infer(4) → number

So now for our first left, we will recurse back to infer, then to check, and finally bottom out in some simple thing we know how to infer. This is the beauty of our bidirectional checker. We can interleave these infer and check calls at will!

How would we change our add to work with strings? Or coerce between number and string? I leave that as an exercise to the reader. It only takes just a little bit more code.

Making it Feel Real

I know for a lot of people this might all seem a bit abstract. So here is a very quick, simple proof of concept that uses this same strategy above for a subset of TypeScript syntax (it does not try to recreate the TypeScript semantics for types).

If you play with this, I'm sure you will find bugs. You will find features that aren't supported. But you will also see the beginnings of a reasonable type checker. (It does a bit more than the one above, because otherwise the demos would be lame. Mainly multiple arguments and adding binary operators.)

But the real takeaway here, I hope, is just how straightforward type checking can be. If you see some literal, you can infer its type. If you have a variable, you can look up its type. If you have a type annotation, you can infer the type of the value and check it against that annotation. I have found that following this formula makes it quite easy to add more and more features.

The Overly Humble Programmer

The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility - Edsger W. Dijkstra The Humble Programmer

Humble is a rather unlikely term to be found in a description of the "modern programmer". Can I suggest: proud, grandiose, delusional, self-absorbed, fad-chaser, bro-y. Even if we ignore the egregious cases of the cryptobro and the AI doomer, programmer discourse is full of complete dismissals of others and overly grand claims about one's own (or one's tribe's) work. Programmers always have the answers. They always know best practices, they are always sure of their technology choices; they could always build it better.

Even when we step outside of this online bubble, when we peel back this toxic, frustrating environment, we might still feel the pull that we are not humble enough. Aren't we always so sure of ourselves? Aren't we always over-confident in our abilities, always believing our code is good enough, good enough without more testing, without proof, without formal methods? Don't we all need to learn the lesson Dijkstra tried to teach us a half-century ago? Shouldn't we be more humble?

It is a diagnosis many of us believe, and the cure we have sought is precisely what Dijkstra prescribed.

I now suggest that we confine ourselves to the design and implementation of intellectually manageable programs.

Linters, type checkers, libraries, well-designed languages, these are all tools that we've made since the times of Dijkstra to limit the intellectual burden of writing our programs. The beautiful thing about software is that the size and complexity of "intellectually manageable programs" can change over time as we develop better technologies. This approach of building abstractions, as the key to managing software, has taken us far.

But I can't help but feel we may have taken the advice too seriously. Or at least I have, at one point in my career. Not directly, of course, I did not sit and read The Humble Programmer and devote myself to its teachings. But Dijkstra's advice, filtered through the culture, made its way to me. I believed it deeply. And I took from it a lesson I'm sure Dijkstra never intended: don't try hard things.

Seeing our Humility

Perhaps the bros in Silicon Valley could use some humility, but for the rest of us, we are already quite a bit too humble. We've created a culture in which "library authors" are the ones equipped and ready to handle complexity. We've taken the fantastic idea of libraries to the extreme. Leftpad has become a common punchline in this regard, but we have to ask ourselves, why do programmers feel the need to grab these kinds of micro libraries? It could be that they are lazy. Or perhaps because it has been ingrained in them over and over again, "smarter people have already solved this problem".

This myth of "smarter people" is the retort made to anyone who dares attempt to write things from scratch. When I talked about building an editor in rust, I was asked why I would do that. Editors are hard and smart people have already made them. When I wanted to learn machine code, I just kept being told to write assembly instead, but it turns out machine code isn't scary. But I'm not upset at these people. They, too, have internalized the message we've all been told. All of us. Even famously accomplished programmers.

"Nobody can build a C++ compiler. That's not a thing. Why are you even messing around with this?" - Episode 205: Chris Lattner Interview Transcript

As someone who started coding for the "web", I always believed that I wasn't smart enough to understand the layers on which my software was built; the layers written in those arcane "low-level" languages. But not only that, I believed that we ought to transcend low-level programming. That the correct answer for all problems was a sufficiently high-level programming language and a sufficiently smart compiler. I believed that I had to stand on the shoulders of giants. That I could not forge my own path, create my own tools, start from scratch. Giving up on this belief has brought me back to the fun difficulties that brought me into programming to begin with. I had to start being less humble.

Being Less Humble

Being less humble does not have to mean giving up on all dependencies. It's not about going back to C99 and swearing off all modern programming languages. It does not mean giving up on garbage collection, giving up on structured programming, and creating a commune where you worship the sacred numbers. It is about questioning those doubts about your own abilities, about thinking through your problem from first principles. It is about being willing to learn the things one level deeper than your current abstraction boundary. To not accept that "how things work" is how they need to work. It is about dealing with the pain the abstraction is attempting to fix.

We have increasingly grown accustomed to (and are encouraged to) treat the abstractions we build on as black boxes. We need not do that. Our software hierarchy does not need to create a social one. You are not worse than the kernel hacker, the compiler engineer, or the game engine programmer. You, too, can build whatever you want to build. All software is just software. It's time we treated it that way.

Stuck? Build Your Language Backwards

I can't count the number of times I started trying to build a programming language just to get stuck in the parser. Some of this was certainly my lack of experience writing parsers, but it was also just how many different choices I had to make. Which way would I parse things? What syntax am I going to use for X? How do I deal with parse errors? Inevitably I abandoned the project before I ever got to the parts I really wanted to learn, namely, all the rest of building a compiler.

That is until I stumbled upon, by accident, the technique that has worked flawlessly ever since. Building the language backwards.

This means picking the lowest level you intend to target and then writing code that produces it. So, if you are building a bytecode-interpreted language, write a simple bytecode interpreter and something that spits out bytecode. If you are targeting machine code, write a little assembler. If it's wasm, write some code that generates wasm. I have found this an incredible way to get myself unstuck.

Why it works

It may sound counter-intuitive. You probably already have an idea of the language you want to build in mind, so why not start with what you already know? The key for me was that it was so unclear, in my head, how to get from high level -> low level, but once I had the low-level thing already in place, I could see the tediousness, the repetition, the bookkeeping, all the things my higher level language would get rid of. Bridging that gap became much easier.

Ever since accidentally stumbling upon this idea, I have had way more success in getting my language projects past that initial hump and I learned quite a lot more along the way. So if you're stuck, try it out and let me know if it works for you.

Machine Code Isn't Scary

The first programming language I ever learned was ActionScript. Writing code for Macromedia's Flash might be the furthest away from "bare metal" as you can possibly get. As I continued learning new languages, this starting heritage stuck with me. I was mostly interested in high-level, "web languages". Low-level languages felt impenetrable. Over time, I learned a bit more about them here and there, but for some reason, this notion stuck with me. Low-level things are scary, and machine code epitomized that most directly. When I Googled things asking about writing in "straight machine code", I was met with discouraging messages rather than learning.

Eventually, I decided I needed to overcome this barrier if I was going to achieve my goals. In doing so, I learned something I didn't expect.

Machine code isn't scary. If you can make sure your JSON conforms to a JSON schema, you can write machine code.

Which Machine Code?

One problem with machine code is that there isn't simply one standard. There are many different "instruction sets" depending on the processor. Most modern PCs use x86-64 machine code, but newer Macs, Raspberry Pis, and most mobile devices use ARM. There are other architectures out there, especially as you go back in time. The goal of this article won't be to give you a deep understanding of any particular instruction set, but instead, to give you enough information about how machine code typically works so you cannot be afraid of machine code. So we will start by having our examples be in ARM 64-bit (also written as aarch64). Once we have a decent understanding of that, we will talk a bit about x86-64.

Machine Code Basics

To understand the basics of machine code, you need three concepts:

1. Instructions
2. Registers
3. Memory

Instructions are exactly what they sound like; they are the code that will run. Machine code instructions are just numbers. In fact, in AArch64, every instruction is a 32-bit number. Instructions encode what operation the machine should run (add, move, subtract, jump, etc.) and accept some arguments for what data to operate on. These arguments might be constants (meaning like the number 2; these constants are often called "immediates"), but they can also be registers or a memory address. For now, just think of a register as a variable and memory as a list.

Arm Instructions

Here is an example of the instruction add immediate.

Now this might look a bit confusing, but once you've seen these tables long enough, they start to be fairly straightforward. Each column in this table represents a single bit in a 32-bit number. If the value is a 0 or 1, that just means it is already filled in. If it has a label, it is a variable that needs to be filled in. sf tells us whether the registers we are going to use are 64-bit or 32-bit registers. sh stands for shift. sh goes in conjunction with imm12, which stands for a 12-bit immediate (constant). So if we want to add 42 to something, we would put 000000101010 in for imm12 and set sh to 0 (meaning we aren't shifting the number). But what if we want to represent a number larger than 12 bits? Well, the add instruction doesn't let us represent all such numbers; but setting sh to 1 lets us shift our number by 12 bits. So for example we can represent 172032172032 by leaving our 42 alone and setting sh to 1. This is a clever technique for encoding larger numbers in a small space. Variables that start with R are registers, in this case, Rn is our argument to add, and Rd is our destination.

So the above instruction can be thought of like this:

struct Add {
 is_sixty_four_bit: boolean,
 shift: boolean,
 immediate: u12,
 n: Register,
 destination: Register,
}

Our add instruction is really just a data structure where we put the right parts in the right places.

Registers

Registers are small places to store values. Every instruction set will have a different number of these registers, different sizes of registers, different kinds of registers, and different naming conventions for registers. For AArch64, there are 31 general-purpose registers numbered X0 through X30 for 64-bit registers. Let's say we want to add 42 to register X0 and store the result in X1; we use this binary number.

To encode our registers into our instruction, we just use their number. So register X0 would be 00000 and register X18 would be 10010. Registers are simply places where we can store values. But by convention, registers can be used for different things. These are called calling conventions and they are how "higher" level languages like C encode function calls.

Writing out all these binary numbers all the time (or even converting them to hex) can often be tedious. So instead, we usually talk about instructions in a simple text format called assembly.

add x1, x0, #0x2a 

In order to feel cool, people usually write numbers in assembly as hex values. This is just the number 42. You can see that assembly hides some of the details of the encoding we just made. We don't think about sf, sh, what size our number is, that a register is Rn vs Rd. Instead, the destination comes first and the arguments after. Because of this lack of detail, a single assembly instruction add might actually map to many different machine code instructions depending on its arguments.

Memory

The last piece we have to understand for machine code is memory. To understand what is going on with memory, we will look at an instruction that lets us store things in memory. This instruction is called STR or not written in shorthand, store.

Using this instruction, we are going to store some value (RT) into the address (RN) + some offset (imm12). So if we think about memory as a big array, this instruction is like writing into that array. array[offset] = value. The x here is like our sf before, it controls whether we are using 64-bit values or not. If we want to make this concrete, let's say we have a value in X2, we have an address of memory in X1 and we want to store a value 2 bytes offset from that. We would get this structure:

Since writing that all is tedious, we often just write the assembly notation. We are storing the value in x2 based on the address stored in x1 + 2.

str x2, [x1, #0x2]

X86-64

X86 encoding is a bit different, but it more or less has the same parts. We are still working with instructions, registers, and memory. Some names are a bit different. Instead of the consistent 0-30 naming, we get the historical baggage of the following 64-bit registers: rax, rbx, rcx, rdx, rsi, rdi, rbp, rsp, r8-r15). However, the biggest difference is that x86 is not a fixed width instruction set. We can't simply give a nice little diagram of every instruction using 32 bits. Instead, instructions are assembled from parts. These parts are given different names; when you see an instruction encoding, it tells you how to put the parts together.

REX

The first part is called the REX. This is a prefix that we can use to help us with 64-bit operations. Not sure if there is an official justification for the name REX, but my understanding is that it is the "Register Extension Prefix". Unfortunately, because the REX is a prefix, it will only make sense when we see what comes later. REX is there for backward compatibility. The W in REX lets us signal that we are using 64-bit or not for certain operations. The R and B will "extend" our registers in certain operations. In other words, it allows more registers than you used to be able to (These are those r8-r15 registers with a different naming convention than the older registers). We need these because, before 64-bit x86, we had fewer registers and our instructions only had 3 bits per register. With 16 registers, we need an extra bit. (X is for the SIB structure, which we don't cover here).

ModR/M

Our next part is ModR/M. ModR/M keeps up with the tradition of naming things incredibly short and confusing names. mod actually means Mode. mod tells us if rm is acting as a register or if it is a pointer to memory. If mod == 11 then rm is being used as a register, otherwise, it is being used as a pointer. reg just is a register.

OpCode

OpCode is simple, it is a number. It can be 1-3 bytes long.

Putting It Together

There are other parts, but we won't cover them here. With just these parts, we can build up an instruction. Let's say we want to move a 32-bit signed immediate to a 64-bit register. We can consult a table of instruction encodings and we will get this:

So now we can assemble our parts and make our instruction. Let's start with REX.W. This notation just means REX with W set to 1. Then there’s B8, which is just a number written in hex. /0 is yet more shorthand for using the ModR/M but setting the reg to 0. Finally, id means "immediate doubleword", in other words, a constant number that is 32 bits long. So given all that, we can write our instruction. So let's move the number 42 to the rbx register.

Why is RBX 011? Well, because the table says so. Yeah, I did say that x86 is a bit weird.

The Rest of It

I won't pretend that this is all you need. But I will say that starting here can get you further than you think. There are some other things to learn, like various flags for things like overflow, there’s also calling conventions, which are about which registers you use when for things like function calls. We haven't really talked about the stack here, but that's memory that you write to to keep track of things. Nor have we talked about jumps, or how to encode larger immediates in ARM, but you’ve gotten the basics. It’s easier than you would think to hop on compiler explorer and learn how things are done.

Learning machine code and writing things at this low level has unlocked so many things that were mental blocks for me before. Relying on libraries made by others to do these low-level things always left a gap in my knowledge that made me doubt my understanding. Even if I intellectually could explain things, actually doing them has made a huge difference for me. So if you, like me, find low-level things intimidating, I can't recommend enough starting from scratch, at the lowest possible level for your task. What I've found over and over again with low-level details, their not hard, their just poorly documented and poorly explained.

Discovery Coding

I don't take notes; I don't outline, I don't do anything like that. I just flail away at the goddamn thing. - Stephen King

In writing (particularly fiction writing), there is a generally accepted distinction between authors who start by writing an outline and those who discover their stories through the process of writing. For some reason, we have no such distinction in programming, so I am here to introduce it.

Discovery Coding Explained

Discovery coding is a practice of understanding a problem by writing code first, rather than attempting to do some design process or thinking beforehand. This means that a discovery programmer does not always start with a plan before starting to write code; rather, they think about the situation they are in. What is the tension we are trying to solve with this new code? What are the situations that have given rise to this demand? How do these various bits of the system interact? It is only in the process of writing code and seeing the pushback the code gives that discovery programmers understand and devise a plan forward.

Discovery programmers can often be seen as messy or undisciplined. When working with their outlining counterparts, there may be some tension that isn't properly placed. The outline programmer may feel unease when seeing the haphazard approach of the discovery programmer. The discovery programmer, on the other hand, may be confused by the seemingly incorrectly timed questions posed by the outliner.

In our current culture that prefers more structured technologies (static type systems, enforced memory safety, schema enforcement tools, etc.), it is easy to label discovery coding as a bad practice, but it is important to distinguish the process of creation from the end-artifact. There is no reason that a discovery programmer cannot create a highly structured, rigorous end-artifact.

Benefits of Discovery Coding

Even for the outliner, I think discovery coding can have quite a few benefits. When approaching problems looking for a solution, it is very easy to start with those solutions we are most familiar with. Discovery coding, by its nature, avoids this pitfall. Discovery coding does not have a solution to offer, so the code we begin writing is instead about poking the system and understanding how it works. By doing this, you are filling your mind not with the context of past solutions or descriptions of solutions others have told you. Instead, you are loading up a series of constraints that the system has that your solution must satisfy.

For me, this is the only way I can do anything. Anytime I try to outline before I write code, my outline is thrown away within hours of writing code. My design docs that are written beforehand are wholly unsatisfying, even if they get approved. It is only as I begin writing code that I begin to understand my problem. So the next section here won't be on the benefits of outlining. Not because there aren't any, but because I'm not the person to write those. (Nor will I write about the downsides of discovery coding, because I am certain everyone can list those.)

We Ought to Accept Discovery Coding

The writing community has learned to accept that some people just are discovery writers. That the very process of outlining before writing can cause some people to be unable to write in the way they want to. My exhortation for the programming community is that we ought to do the same thing. What I'm not advocating for is the banning of all design documents (those can be useful), but instead the recognition and acceptance that some people simply think differently. That being less organized isn't an inferior approach. That being able to think through details before you code isn't superior.

Perhaps I have the wrong read of the current cultural milieu (or perhaps you are reading this years later and culture has shifted). Perhaps discovery coding is loved and cherished and outlining is scorned. If so, please invert all of the above. What I hope we can recognize is that there is no one answer. Designing software is a human process and not all humans are the same.

Against a universal definition of ‘type’ (pdf)

This is the final entry in the series and I'll be honest, I'm a bit glad it's over. It's been hard to consistently find the time every day to 1) find a paper I'm truly interested in writing about 2) read that paper in a way I can summarize it 3) write something somewhat interesting about the paper. Honestly, I think finding the papers that fit my mood was the hardest part. I tried to frontload that by making this big list of papers, but I just couldn't be bothered to seriously read some of them. It's one of the reasons I slipped a few philosophy papers into the mix. It's so much easier for me to find a good philosophy paper than a good computer science paper.

Another goal was to mostly read new papers. I'd say I did alright, but not great at that task. There were a few papers I hadn't fully read. Some that I had definitely read before. Today's paper is in the category of having read some of, but not completed the paper. It's a paper I'm quite fond of but have complicated feelings about. In this paper, Tomas Petricek argues against giving a universal definition of "type".

Summary

The paper begins by exploring the ways in which our use of the word "type" has changed over time. Starting with Russell's theory of types to rule out paradoxes. He then goes on to discuss more modern uses of types like unsound types systems, dependent types, and type providers. Along the way, he points out how the old way of using types does not quite fit with this new way of using types. If you find this controversial, I highly suggest reading the paper as I think Petricek is rather convincing on this point.

The rest of the paper explores the idea that we really shouldn't have a universal definition of type, because by not having one, we are allowing the concept to expand. This is actually a good thing. People are able to make use of this concept of a type, without immediately formalizing it. Allowing it to take on new shapes.

Looking at the history of science supports the main idea of this essay. That is, we should not require a precise definition of the notion of ‘type’. Requiring clarity means that we can only talk about things we already understand – perhaps in greater detail, with more generality, and in more elegant ways, but not about fundamentally new ideas.

Petricek continues by showing ways we can live with the fact that types do not have a clear definition. All of his examples are borrowed from philosophers. The details are a bit too subtle here for me to repeat here so I will leave those to people who decide to read the paper. But they are certainly interesting suggestions.

Conclusion

If we want to talk about types outside of a narrow research programme, we need to find ways of dealing with types without a precise definition. I proposed three alternatives –those are based on how we use types (inspired by Wittgenstein’s language games), what is the conventional idea of type (based on Putnam’s stereotypes) and what we can do with types (inspired by Hacking’s new experimentalism). I believe that these provide worthwhile methods of inquiry that can provide interesting insights into what types are outside of a narrow boundary delimited by a particular formal definition.

I think this is entirely right, if our choice is between a singular, formal definition of types and these suggestions, I choose the latter. But I want to quickly suggest that a formal definition isn't the only kind of definition we can give. Let's consider another word that lacks a universal definition in programming, abstraction. There are so many varied uses of abstraction. No formal definition could ever capture how we use the word abstraction. But I do think having a better, clearer, universal definition of an abstraction would benefit us.

How does "an abstraction" relate to things being "more abstract" or "less abstract"? Do abstractions come in levels? Is abstraction always information hiding? How do abstractions relate to notions like interfaces? I do feel these questions have answers and those answers would be illuminating. In the same way, I believe there is some (non-formal) unifier core behind these notions of types. When we create new applications for types, (e.g. type providers) they are new applications for "types" because of some shared underlying way of viewing types.

This is something I feel is completely lacking in software. We seem to think that there are two things, undefined things and formal definitions. Of course, in many ways, this paper is such an exploration of types. But it is largely historical, stopping short of trying to synthesize how we do talk about types. Meaning as use doesn't have to mean we just need to find more contexts of use. It can also mean, that if we look at the current way we use the words, we can discover their meanings.

Do Artifacts Have Politics? (pdf)

One question that I don't think gets asked enough is how "good code at work" differs from "good code at home"? For many years I didn't quite see the distinction between the two. Isn't it a fact of the matter what things make for good code? Why would code I write for myself have different best practices than code I write for work? Now one obvious answer may be context. At work, there are more people working on the code, so perhaps I need to consider approachability more. Or there are more eyeballs, so maintenance might be more likely to occur. Or tons of people already use this dependency, so I shouldn't be worried about taking a dependency.

But as time has increased, I've become skeptical that these contextual differences should be my main focus. Instead, I've started to realize that the major difference is in the values I hold as an individual programmer versus the values implicit in the structure of the companies I work for.

Today's article by Langdon Winner looks at these sorts of facts not necessarily in code, but the artificial, technological landscape we build.

Summary

This paper is remarkably well-written. It is a joy to read, the points made are clear and nuanced. Winner talks about the standard reaction to the idea that artifacts have politics, which is that they are simply misplaced. People have politics and use artifacts to those ends. His response is a fantastic representation of the sort of writing throughout.

Hence, the stern advice commonly given to those who flirt with the notion that technical artifacts have political qualities: What matters is not technology itself, but the social or economic system in which it is embedded. This maxim, which in a number of variations is the central premise of a theory that can be called the social determination of technology, has an obvious wisdom. It serves as a needed corrective to those who focus uncritically on such things as "the computer and its social impacts" but who fail to look behind technical things to notice the social circumstances of their development, deployment, and use. This view provides an antidote to naive technological determinism, the idea that technology develops as the sole result of an internal dynamic, and then, unmediated by any other influence, molds society to fit its patterns. Those who have not recognized the ways in which technologies are shaped by social and economic forces have not gotten very far.

But the corrective has its own shortcomings; taken literally, it suggests that technical things do not matter at all.

Winner goes on to show in great detail how artifacts can shape the Forms of Order of a society. I think his most compelling example is the move to make cities more accessible for people with disabilities. The design of countless technologies limited (and still does limit) people with disabilities from participating fully in society. These are political implications of the technologies themselves. If don't find this particular example convincing, I highly recommend reading the paper as it is chock-full of examples.

He next tackles the stronger notion that certain technologies are inherently political, that is by their nature they bring about certain political structures. The discussion here is far too involved and subtle for the time I have here but ultimately, Winner concludes this too is true.

It is still true that, in a world in which human beings make and maintain artificial systems, nothing is "required" in an absolute sense. Nevertheless, once a course of action is underway, once artifacts like nuclear power plants have been built and put in operation, the kinds of reasoning that justify the adaptation of social life to technical requirements pop up as spontaneously as flowers in the spring.

Conclusion

This paper was a joy to read. It makes me think more deeply about my own code and the values implicit in it. My code at work is made with a structural goal of providing money for the company I work at. Its function supports certain social structures I may not fully agree with. The form the code takes necessitates a managerial structural system. And many more facts besides This value structure is completely lacking in the code I write for myself. Shouldn't that be reflected in the kind of code I write? I'm starting to feel this pull. A pull to rethink the values I hold about code, to find new ones that fit the sort of world I want to exist, that support the sort of structures I believe help the world.

Once More—A Computer Revolution - Joseph Weizenbaum (pdf)

Summary

Ran out of time today.

Will the home computer be as pervasive as today's television se,s? The answer almost certainly is no. The home, pictured in the accounts of home computer advocates, is middle class, or even upper middle class. There are some appliances computers must control: the wall-to-wall carpeting must be cleaned by a robot, roasts are in the oven, the computer helps "the mother" pay the telephone bill, and so on and on.

Conclusion

This is a fascinating critique. Completely wrong in its prediction but captures something all too true.

We may recall the euphoric dreams that were articulated by then Secretary of Commerce Herbert Hoover, at the dawn of commercial radio broadcasting, and again by others when mass television broadcasting was about to become a reality. It was foreseen that these media would exert an enormously benefcial influence on American culture. Americans would be exposed to correct English, to great literature, great drama, and so on. We all know what actually happened. The technological dream was more than realized, but the cultural dream was not.

What is Conceptual Engineering and What Should It Be? (pdf)

I've loved choosing philosophy papers for this series. And so, I decided today to do the same. But before you click away, this one is incredibly applicable to software. In fact, David Chalmers thinks that Software Engineering helps address a worry about Conceptual Engineering.

Incidentally, I like the software engineering analogy because it addresses a worry about how conceptual engineering can work if concepts are abstract objects. It’s not obvious that you can engineer a new resident of the third realm. Exactly the same issues arise for software engineering. Programs are usually thought of as abstract objects, but we have no problem saying that programs are engineered.

Chalmers is of course quite right that we engineer abstract objects (see day 2), but I don't think he realized just how tied together conceptual engineering and software engineering are. But I guess we are getting a bit ahead of ourselves, so along with Chalmers we must ask. What is Conceptual Engineering? What should it be?

Summary

Conceptual engineering is about "designing, implementing, and evaluating concepts". One great example given in the paper is Ned Block's distinctions between phenomenal consciousness and access consciousness. Leaving aside what these terms mean, Ned Block developed them as a way to distinguish two things that were both simply being called "consciousness" that were actually distinct phenomena. By giving us these terms, it made it easier to pick different aspects of "consciousness ".

But this example might not be the best for programmers. So instead, think about MVC. The concepts of Model View and Controller were developed by Trygve Reenskaug at Xerox PARC. Many people (including myself) have many thoughts (not all good) about MVC. But there is no doubt that the concept of MVC has played an important role in software development since these concepts were "engineered".

De Novo Vs Conceptual Re-engineering

In the philosophical world though, less attention has been paid to examples like MVC cases where new concepts are created (De Novo) and more attention has been paid to the fixing of deficient concepts. Conceptual engineering of this sort is often applied to societal issues around topics like race and gender. For these topics, we are not trying to create new concepts nor are we trying to ask how people actually use these concepts. Instead, we are asking "what should these concepts be?"

What do we mean by what "should" these concepts be? Well, it may be that the concepts we have are inconsistent and should be replaced by consistent ones. It may be that the concepts are part of some unjust system of beliefs and we ought to revise them. There are a number of different reasons we may want to fix our concepts.

Chlamers’ paper covers a lot more ground, talking about various questions in the philosophical landscape around conceptual engineering. What does it require to change a concept? When we create a concept, should we use the same word as an existing concept or pick a new word? How does conceptual engineering relate to disputes that are merely verbal? I think all of these are actually very interesting and relevant for us in software, but I will leave those details to those who wish to read the paper.

Conceptual Engineering in Software Engineering 

What isn't talked about at all is the way in which conceptual engineering plays out in software. It's my contention that much of what we do day to day is conceptual engineering, but we never directly talk about it. If you are in an Object Oriented language, you have to pick class names. By their very nature, these classes add concepts to your program. The notion of a Customer in your code base is not the same as the general real-world notion of a customer. It is a custom, particular concept of customer, which stands in relation to the customer out there, but it isn't actually that real customer.

As you continue to develop your codebase, you find that there is a new sort of (in the real world) customer that needs to be accounted for in your codebase. What do you do? Do you change the concept of Customer? Do you introduce a new concept of SpecialCustomer? Whatever your decision, your job in conceptual engineering doesn't stop there. Now that you have made this conceptual change, how do you get your fellow coworkers to adopt this new concept?

These are concerns we have constantly as software engineers. We are surrounded by concepts in need of re-engineering. We are surrounded by problems in need of new concepts to help us understand them. Because we don't think of conceptual engineering as a first-class part of our job, we don't have tips and tricks for doing these conceptual engineering roles. We don't think about the fact that problems can be avoided merely by changing our concepts.

Conclusion

There is so much more to say about conceptual engineering. If you are interested and want a full-length treatment (but warning, it is a philosophical text, not a software engineering one), I highly recommend Fixing Language: An Essay on Conceptual Engineering

Three Paradigms of Computer Science (pdf)

In late 2019, I was invited to give a talk at Rebase in Berlin during the summer of 2020. As you can imagine, that didn't happen at least not in person. Instead, I gave a pre-recorded talk. I tried over and over again to sit at a desk and give a talk as I'd seen many people do, but it just felt so lifeless. So I contacted a friend and I was able to access his company's studio. It was just me, my brother, and my friend. Socially distanced, and the two of them masked, as I gave a talk about Thomas Kuhn and programming paradigms.

I think the talk was overall good. I fumbled a bit at the beginning, but I stand by the point I was trying to make. Programming Paradigms are not Kuhnian paradigms. (Perhaps I should write this up at some point). However, as part of this talk, I intentionally did not talk about the paradigms of computer science. When asked in the QA about that, I said, I'm an outsider so I don't really have any opinions on that. But anytime I say I don't have opinions, that's a lie.

Summary

This paper by Amnon H Eden talks about what they take to be three Kuhnian Paradigms of computer science.

In his seminal work on scientific revolutions, Thomas Kuhn (1962) defines scientific paradigms as “some accepted examples of actual scientific practice… [that] provide models from which spring particular coherent traditions of scientific research.” The purpose of this paper is to investigate the paradigms of computer science and to expose their philosophical origins.

I'm not quite sure they succeed. But before we get there, let's see what the paradigms are supposed to be.

(§2) The rationalist paradigm, which was common among theoretical computer scientists, defines the discipline as a branch of mathematics, treats programs on a par with mathematical objects, and seeks certain, a priori knowledge about their ‘correctness’ by means of deductive reasoning.

(§3) The technocratic paradigm, promulgated mainly by software engineers, defines computer science as an engineering discipline, treats programs as mere data, and seeks probable, a posteriori knowledge about their reliability empirically using testing suites.

(§4) The scientific paradigm, prevalent in artificial intelligence, defines computer science as a natural (empirical) science, takes programs to be on a par with mental processes, and seeks a priori and a posteriori knowledge about them by combining formal deduction and scientific experimentation.

Despite the title seeming to be simply descriptive, this paper actually argues for the scientific paradigm. I mean, who is surprised by that one? It is rather unusual for someone to name some "scientific" and then not endorse it. I must say, I think I understand the characterization of the rationalists and technocratics quite well. But I'm a bit lost as to what this "scientific" paradigm is supposed to be. Consider this statement of scientific ontology:

Program-scripts are on a par with DNA sequences, in particular with the genetic representation of human organs such as the brain, the product of whose execution—program-processes—are on a par with mental processes: temporal, non-physical, causal, metabolic, contingent upon a physical manifestation, and nonlinear entities.

"Program-scripts" here just means the text of the code. Where "program-processes" means the running of that code. What are we supposed to take from this ontologically? DNA is a physical thing that has complex causal relationships. Written down DNA sequences may be thought of using the analogy of code. But how is code itself like molecules in a double helix structure? And why is code execution like a mental process if it is DNA? "Running DNA" doesn't result in a mental process.

I'm confused, is the ontology offered simply metaphors? Is there anything I'm supposed to take literally here? If you think I'm being weird and picky, contrast this with the ontology statement given for the rationalists.

A program-script sp is a mathematical expression. A program p is that which is fully and precisely represented by sp. Hence, p is a mathematical object. Mathematical objects are platonic universals. Therefore, programs are universals.

This is a straightforward ontological statement. The confusion for me continues throughout much of the section arguing for the scientific view.

But mathematical objects, such as turing machines, recursive functions, triangles, and numbers cannot be meaningfully said to metabolize nor have a causal effect on the physical reality in any immediate sense. It would be particular difficult to justify also a claim that mathematical objects have a specific lifespan or that they are contingent upon any specific physical manifestation (except possibly as mental artifacts). In this respect, rationalist ontology is inadequate.

Since when do programs metabolize? The best we get is the assertion that the "program-process" consumes energy. There is nothing about that fact that is contrary to anything mentioned by the rationalist.

A Brief Note on Kuhn

One point I was going to make is that part of why these aren't Kuhnian paradigms is that Kuhn's paradigms are not mutually comprehensible. Maybe the statements I made above actually lend evidence this is the case. But in general, I think that is one problem here for calling this Kuhnian. But the real major issue is that for Kuhn, a paradigm always wins. There is always progress and a singular dominant paradigm. If these point to anything, it is a pre-paradigm phase. Or I think more realistically, that computing isn't a science.

Conclusion

As described, I'm not sure these views are quite right. I do think the paper has some interesting ideas and is worth a read. Personally, I find the contrast between the rationalist and technocratic views to be a bit of a false dichotomy. Perhaps, this is my suggestion, what do we get if we combine the various elements presented here? Can we mix and match and find our own paradigm that isn't one of these things, but perhaps parts of all of them?

Everybody Clap Your Hands (pdf)

Have you ever been at a wedding and thought, "You know what would really spice up this dance floor? Arbitrary computation!" this paper is for you. Today, was a busy day for me, so you will get a very short summary of a very silly paper. Everybody Clap Your Hands: The Cha-Cha Slide is Turing Complete by Harrison Goldstein.

Summary

The Cha Cha slide is in fact turing complete. Show this relies on only a few well known features of the Casper slide part 2 and some perhaps, strange ways of interpreting what is going on.

The dance floor is always in one of two states, Funky or Not Funky, which correspond to the states F and N from the machine 𝑀. The DJ can memorize the current state if they wish, but if everyone is doing their job it should be obvious to all involved whether the room is Funky or not.

Funkyness is perhaps my favorite aspect of this scheme. You may be thinking (as I was), I know how to make it funky, but how do you make it not funky? The obvious answer, "freeze"! Other than that, we get things like moving the tape by sliding. We get writing to the tape by hopping. The dancer in front of the DJ is in the hot seat and can signal their state by counting in binary on one hand.

Since few people have 18 fingers, the count can be shown in binary as illustrated in Figure 2. To avoid being too rude, the original Turing machine might do well to avoid symbol 4 when possible. In any case, the important thing is that the hot-seat dancer communicates their cell’s content to the DJ.

In general, the scheme makes sense. There are some extensions that might make things a bit easier to encode things, but I like my turing machine equivilants to be minimal.

Conclusion

The paper ends with some discussion about the pointlessness of turing completeness as a metric. But honestly I don't think it needs it. The paper mentions internet trolls saying things about turing completeness. But I find that kind of discourse boring. Instead, I want to ask, is a freestyle cha cha slide a programming language? What if the dj (and the dancers) aren't trying to compute anything at all, they are just experimenting with dancing?

When we have debates and say try to include something not traditionally thought of as a programming language into that category, does that same reasoning apply to these Cha Cha slide dancers? Do we want it to?

The Structure and Legal Interpretation of Computer Programs (pdf)

I do not have much time to write today. I've been dealing with the seemingly never-ending problem of getting a bed frame delivered properly. It's been weeks of dealing with customer support to get the hardware, for it to arrive today, and halfway through getting the bed together realizing I'm still missing a crucial part. Then hours returning the bed.

So you are getting a very short summary of a paper that deserves much more to be said about it.

Summary

James Grimmelmann has written this wonderfully titled and fantastic paper about the meaning of programs. He walks through three different wants of understanding the meaning of a program, naive functional meaning, literal functional meaning, and ordinary functional meaning. All of these ways of taking meaning find their use in law. I won't detail what they mean here, but I will say as practitioners, we do have to deal with these different kinds of meanings all the time. Ordinary functional meaning and literal functional meaning are involved when debugging software.

But despite not telling you what these meanings exactly are, I will give you the punch line. The meaning of our programs depends on social facts, it isn't merely objective. This is a fascinating conclusion that I believe is well argued for. If we are to find a grounding for the meaning of our programs, they must be grounded in social facts. Facts like what the community agrees is normative for language definition. Much of what Grimmelmann argues for about software is in concert with discussion in day of our papers about software ontology. But literal functional meaning also connects with day 17 about the ontology of programming languages.

Conclusion

This paper is a wonderful exploration of program meaning, particularly in a legal context. I will admit, I haven't found a ton of great papers about programming from the social sciences (please if you know of them share them with me!). But I've been finding quite a few more from a legal context. This is a refreshing read that takes seriously programming in its social context. It's very readable and I can't recommend it highly enough.

The Cultural Part of Cognition (pdf)

As part of this series, I set out to read papers in my backlog of papers, that I literally hadn't read at all. This paper is one of them. It's a paper published in 1981 that from the abstract seems to be drawing a comparison between "cultural programs" and computer programs. But there are times when I read a paper and I feel I have understood every word, but seem lost as to what the paper is really about. This is one of those times.

Summary

Roy Goodwin D'Andrade wants us to focus on more than just how cognition works or in other words, how the brain operates. The proposal of this paper is that we should focus on the "cognitive organization of culture". Most of what we know comes from cultural transmission. As a species, we are remarkable for our ability to pass information down through generations in a way that grows over time. But what shape does this information take? It must be a shape that is a natural fit for the way our cognition works.

An important assumption of cognitive anthropology is that in the process of repeated social transmission, cultural programs come to take forms which have a good fit to the natural capacities and constraints of the human brain. Thus, when similar cultural forms are found in most societies around the world, there is reason to search for psychological factors which could account for these similarities.

Much is made about this idea of cultural programs. But at each turn, we are just told how they differ from computer programs.

[O]ne major difference between cultural and computer programs is in the degree of precision with which computer programs must be stated"

How does this great difference between the highly specified computer program and the learned-by-guided-discovery cultural program relate to differences in human versus computer processing characteristics? Among other things, this difference may have to do with the fact that humans must learn their programs, while computers "load" theirs.

Another example of a major difference between human and computer programs is the strong tendency for human problem solving procedures to be highly local and content specific, rather than global and formal.

We are never really told how these two are supposed to coincide. I take it this is just taken for granted. But I'm honestly not sure I see it. In which way are cultural programs like computer programs? One example given a cultural program is playing chess at the Master's level. We already know quite well that computer programs play chess quite differently than people do. Even more mundane things, like how to use a pencil, seem to be examples of cultural programs. What do these have in common at all?

One current formulation in anthropology treats the pool of cultural information as if it were a pool of algorithms or programs, a giant stock of procedures and representational declarations embodied primarily in natural language

I thought this "one current" was going to lead us to an alternative formulation. But as far as I can find, it never does. Is this really the way this paper thinks about all the things we do in life? Am I currently following a pool of algorithms to write this paper? I never intended this to be yet another paper on computation and instantiation but it seems to be.

Conclusion

The best I can see is that this paper is trying to point to how important culture is for the way we think and what we think about. It is trying to say that a study of culture can be a study of the brain. Maybe that first part needed to be said in 1981 to this group of cognitive scientists. As for the second part, it may well be true, but looking at culture in terms of programs doesn't seem all that fruitful to me.

Will Computers Ever Become Easy to Use? (pdf)

This paper brings back some nostalgia for me. Not because I was around when it was written (I was 5), but because my computing journey saw me growing up with a different Raskin, Jef's son Aza. Some of the first programs I wrote that had real utility were for the Enso launcher by Humanized. Their Weblog had a ton of articles that were very influential in my thinking about interface design and how programs relate to people. It was only many years later that I learned the connection between Aza and Jef. The Human Interface is a fantastic book and if I ever find myself with enough time to write about books rather than papers, I'll definitely re-read it and write that up.

Today's paper though I have to admit is a bit lackluster. The question is an interesting question. But the framing is a bit disappointing. But I've gotten into this habit of reading (skimming) too many papers before making a decision on what paper to post about and I just need to commit if I'm going to make this advent of papers work.

Summary

Raskin points out things that were true in 1997 and are certainly true today. Users are scared of computers. They do not want to touch things and change them. They need experts to help them do anything complicated. Applications are full of mode errors and all sorts of other design problems that cause endless frustration for non-programmer users. I think this fact can often be under-appreciated by programmers. While we do encounter these flaws in software, we are typically able to work around them or avoid that kind of software. Making settings changes isn't scary. But when I see less technical folk using computers I remember being a kid playing Sim City 3000 Unlimited. I didn't know where to save the game (what folder) and I was scared I would mess something up. I asked my mom and she didn't know either. So the game just never got saved.

Raskin sees these problems as fundamental to our application-centric GUI interfaces. Each application offers very different affordances and patterns that must be relearned constantly. Not only are these problems baked into our methods, but we do not have a field of study that could help us design better alternatives. Raskin proposes a focus on what he calls cognetics: "an engineering technology of applied physics, ergonomics, and cognitive psychology". Raskin wants to define "formal measures of productivity". Given these measurements and design work, we will be able to make much better interfaces that are understandable for everyone and much more efficient. He ends the article with quite a bit of optimism about this research project.

At present Silicon Valley and the other technological triangles, loops, and glens are somewhat complacent and consider the problem of interface design essentially “solved,” needing only a bit of touch-up here and there. For many users and developers, today’s GUI is the only conceivable interface. Their very success impedes the needed radical improvements. In spite of these formidable difficulties, we will do better. Those companies that cling to the status quo will not be able to also hold on to their customers.

Conclusion

I love some of Raskin's radical ideas around dissolving application boundaries. But this paper just struck me the wrong way. Is a productivity measure really our goal in making better interfaces? I guess you could say it is the only goal that is going to "win in the market" or something. But I think that generally is a naive view of the market as well. Enterprise software has absolutely terrible interfaces and is wildly successful. In many ways, they are successful because of how bad the interface is. (People need support and services, etc which makes for stickiness and a thriving ecosystem.)

As I look back though, I do see a spirit for redesigning the desktop metaphor that I feel is not very bright today. This is one of the things that makes me sad that so many future of code projects focus on the web. They in many cases assume the very things I think we might want to explore dissolving by putting this platform as a requirement for their work. I get it, it is hard to start over. But I hope we will see that sort of fundamental reworking of the way we interact with our computers again someday.

Programming Languages as Technical Artifacts (pdf)

At first blush, the topic of this article may seem no different than our day 2 article Software is an Abstract Artifact. However, Raymond Turner's notion is quite different. First I think it's important that we make clear in this article we are talking about programming languages. In the previous article, we were talking about a piece of software (e.g. Microsoft Word). Is there really a difference between these two? Aren't programming languages pieces of software after all? Go back to the year 1960. McCarthy has invented Lisp, but no implementation of it exists. Is it still a programming language? Or I have written up an operational semantics for a new DSL I'm planning on making. Is my DSL not a programming language? Contrast this with software. Software does not exist unless it is implemented.

There is quite a bit more that is different about these articles. Abstract Artifacts are not the same as Technical Artifacts. We will get into that later. But I want to comment on the fact that distinctions like software vs programming languages and their existence criteria are something we pre-theoretical have intuitions about. Perhaps your intuitions disagree with mine. Maybe you think a UML diagram is all that is needed for a piece of software to come into being. But regardless, you have these views. Yet these kinds of questions aren't considered part of computer science proper. Only a handful of philosophers of computer science talk about these issues and yet it isn't as if computer scientists have no views on the matter.

For example, Laurence Tratt's article Compiled and Interpreted Languages: Two Ways of Saying Tomato shows how complex the notions of interpretation vs compilation can be. Elements of interpretation and compilation are interweaved in many language implementations. But what it also contains is clear (if a bit implicit) commitments to constraints on the ontological status of programming languages. For example, Tratt does not believe that languages are identical to their implementations. So languages are not the bits on the computer that make up the implementation. But perhaps surprisingly, nor does he appear to believe that languages are identical to their specifications! He says this in a footnote: "Some languages don’t even have a single specification, such as BASIC, allowing many different languages and implementations with “BASIC” in their name to diverge substantially from one another". Perhaps that is a bit vague, in the first part of the sentence he seems to imply the (singular) language Basic has multiple specifications, but then talks about different languages (plural) that are "Basic".

Tratt's view is a fairly common one from what I've seen. More explicitly stated views might say something like "Interpreted language" is a meaningless phrase. What they have in mind here I've gathered is that it is a category error. Languages aren't their implementations and only implementations can be interpreted. We will ignore the larger issues of language, how clearly that can't be a meaningless phrase given that people use it and communicate things all the time. (Happy to debate anyone on this) What I wanted to draw out from it is merely that computer scientists already have made these sorts of ontological commitments. This isn't an imposition from the outside.

Summary

So if a programming language is a Technical Artifact, what exactly is that?

Technical artifacts include all the common objects of everyday life such as televisions, computer keyboards, paper clips, telephones, smartphones, dog collars, and cars. They are intentionally produced things. This is an essential part of being a technical artifact. For example, a physical object that accidentally carries out arithmetic is not by itself a calculator. This teleological aspect distinguishes them from other physical objects.

This is first in contrast to natural objects. A rock is not made with any particular purpose in mind. It is not an intentionally produced object. But technical artifacts also differ from other socially constructed objects like money, in that there is a necessary connection between function and physical structure. Money does not need to have a particular structure to fulfill its function, but things like a paper clip do.

A technical artifact is individuated by the combination of two properties, its functional properties (what it is intended to do) and its structural properties (its physical makeup). A technical artifact is not simply the physical object considered in and of itself. It must have these functional properties as well. (How exactly those are understood is a bit complicated, but we will touch on them a bit more here. See the full article and citations for more information). For our purposes, we are just going to talk about the engineering design view which takes the function of a technical artifact to be related to the intention of its construction.

Programming Languages

What exactly specifies the function of a programming language? Well, the paper does I think a pretty great job of diving into topics like operational semantics, denotational semantics, virtual machines, and abstract machines. We don't have the space here to dive into these details, but there is one major point that is important for us to capture here, the physical implementation of a language cannot be the ground of its function. I think for some more academically minded, this is obvious, so I won't defend specifications here. But from an everyday perspective, isn't the "meaning" of a Python program just defined by what CPython does when it runs the program?

The fact that there can be bugs in CPython shows this to not be the case. The notion of a bug in an implementation of a programming language requires that there is something outside the implementation itself that specifies the correctness criteria for the implementation. Or to put it in philosophy speak, semantics must provide normative criteria. This is actually crucial for any notion of meaning.

The fact that the expression means something implies that there is a whole set of normative truths about my behavior with that expression; namely, that my use of it is correct in application to certain objects and not in application to others. (Boghossian 1989)

But "When is an implementation of a language correct relative to its semantic account?" This question actually brings us back to day 4's question Is the Brain a Digital Computer. In John Searle's paper he shows how given a sufficiently complex physical object, we can map that object onto any program specification. This includes a programming language specification! So any object can implement any programming language! (If this confuses you, see section 6 in the paper). Turner actually gives us a way around this problem for technical artifacts.

If I observe a man arranging stones in a way that is consistent with the extensional arrangement determined by some abstract machine, I may not legitimately infer that he is building a device according to the abstract machine viewed as a functional specification. He might just be arranging them for esthetic reasons. In general, a physical device may be in accord with an abstract machine without being governed by it. How might we tell the difference? How can we tell if she is under the authority of the abstract machine or building a work of art? Presumably, we can ask: “what are your intentions?”, “why did you do that?” For the former, the answer must refer to the abstract specification.

So rather than using the mapping procedure in a post-hoc manner after the computation has been run to see if it meets the specification, we must look at the intention behind the physical process. As the paper points out, this might seem that we are saying that the meaning of a language is actually in the head of its creator, but quoting Kripke, Turner sees this as problematic. (In the following passage quus is like mathematical plus, but just thinking about it as wrapping at some boundary like 32bit plus would.)

Given . . . that everything in my mental history is compatible both with the conclusion that I meant plus and with the conclusion that I meant quus, it is clear that the sceptical challenge is not really an epistemological one. It purports to show that nothing in my mental history of past behavior—not even what an omniscient God would know—could establish whether I meant plus or quus. But then it appears to follow that there was no fact about me that constituted my having meant plus rather than quus. (Kripke 1982).

So Turner concludes that the meanings of programming languages are not in the head, but rather are mathematical objects. By themselves, these semantics can be studied as just any other mathematical object, but when used by an agent they provide normative force over the implementation of a programming language, defining its correctness.

Conclusion

I am always refreshed reading these kinds of papers. I understand many people find them exhausting. What difference can these esoteric distinctions possibly make for the actual practice of people? Well, luckily for you Turner has a suggestion for exactly that. Programming languages in order to be used for practical purposes, must have physical instantiations. Given this, computer science isn't merely an abstract discipline, it is something that produces technical artifacts. Computer science is part math, part technology. So shouldn't the philosophy of computer science be the combination of the two?

You may take this as still decidedly impractical, who cares about the philosophy of computer science? For that, I have no retort. Are there no answers to the questions we seek in the philosophy of computer science? Or do the truths of certain questions simply not matter? Whichever it is, I can't understand why people wouldn't care about these questions.

Bidirectional Type Checking (pdf)

Type checking is one of those topics in programming that I feel is often taught in a way that makes it rather confusing. Hindley Milner is a rather elegant, but confusing algorithm. Seeing how easy it is to make in mini-Karen, certainly didn't give me any insight into how it works. When you dive into papers about type checking it's full of formalism and very little code. For academics, I'm sure that's fine. However, for an industry programmer like myself, I often find it rather difficult. This paper (Bidirectional Typing Rules: A Tutorial by David Raymond Christiansen) is a great contrast to this usual setup. It isn't lacking in formalism, but provides a nice discussion of the formalism and some simple translation to code! There's even a talk which is how I first discovered this paper (though there are some audio issues). The talk goes into a few more details. But this paper is still a great introduction.

Bidirectional type checking is a method somewhere between full type inference and no type inference. An imprecise characterization of this is that you only have to write type annotations on type-level entities, but not on every variable. So, for function definitions you need to write your types, but not much else. The bidirectionality bit gets its name because this algorithm works by checking types and inferring (or synthesizing) types in an interleaving fashion.

If we know a type and we have a value, we can check that value against a type. If we don't know a type but have a value, we can infer the type and then check it. This may seem a bit simplistic. When I first saw it, I thought there was no way this kind of setup could work but it does! Consider the very simple example:

x : bool
x = true

We can very easily infer the type of the literal true and then we can check that the type of true matches the annotation. This very simple idea scales up incredibly way to larger structures.

Conclusion

Just from the features laid out in this paper, I was able to implement a simple bidirectional type checker! I actually did it twice. Once in a terse way in Clojure using a library called Meander a friend of mine wrote. (Maybe I need to find a good paper on term rewriting, not sure I ever found one of those). Then I decided to translate that into more verbose, but maybe a bit more familiar Javascript. This post was rather short as I didn't have a ton of time today. But I promise, this paper is super readable. It is full of code translations and hints on how to read things. For example:

Translating the formalism from above into the code below makes this whole thing way more understandable! I know there are reasons academics prefer the notion they use for these things, but every time I see them translated into code, I just imagine an alternative universe where that didn't happen and how happy I'd be.

What Knowledge Isn't (pdf)

I read four different papers today. None of them stuck with me. There were a few papers on social aspects of programming, all of the empirical bent. I was hoping to find a good paper that captured the sense of frustration I feel with Agile; a paper that could summarize the perverse labor relationship that Agile has forged between software engineers, product managers, and the software itself. But I've tried in vain. I looked through my absurd back catalog. None of them stood out to me as papers I wanted to discuss today. So I'm instead giving you yet another philosophy paper. But this paper leaves you with no excuse but to read it. Coming in under 3 pages. This philosophy paper turned the world of epistemology on its head. Introducing Edmund L. Gettier's Is Justified True Belief Knowledge.

Knowledge

• 2 + 2 = 4
• My name is Jimmy
• Agile is a terrible way to make software
• The earth is round
• My dog sheds a ton of hair
• No prime numbers are prime ministers

These are all examples of things I think I know. But how can I be sure? What exactly does it mean for something to be knowledge? Well, when we look at the kinds of things we intuitively count as knowledge, there seems to be fairly wide agreement on the core of the account. First, the things you know, you must believe. Now of course we might say something like "I know I'm going to fail, but I don't want to believe it." But here we aren't being literal. When we say that we know something we are at least saying we believe it. Or to make it silly programmer speech, knowledge is a subclass of belief. But not all belief rises to the level of knowledge. What is missing?

"I know that the earth is flat." "No, you don't, because it isn't."

Replace the word "know" about with the word "believe". Note the asymmetry. With "know", this logic feels sound, but replacing it with believe makes the tort non-sensical. What does this show? That we can't know things that aren't true. Philosophers call this the "factive" portion of knowledge. In order for something to count as knowledge, it must actually be true. But it's not enough for something to be true for you to correctly assert that you know it.

"It's going to rain tomorrow" "How do you know that?" "I rolled a d20 and got a natural 20"

Unless the above sentence was uttered during some TTRPG, we would in fact reject this person's claim to knowledge. But what if it turns out it did in fact rain tomorrow? Did they know it all along? No, they just got lucky. What they lacked was any good reason for that belief. Philosophers call this "justification". Justification means that a person is internally in possession of a good reason for their belief. When asked why they believe something, they could offer a good reason.

But does this intuitive view work? Gettier shows us that it doesn't.

When Justified True Belief fails to be knowledge

You should read the paper. So I'm not even going to give a Gettier example. Instead, I'll use a different example.

The pyromaniac (Skyrms 1967). A pyromaniac reaches eagerly for his box of Sure-Fire matches. He has excellent evidence of the past reliability of such matches, as well as of the present conditions — the clear air and dry matches — being as they should be, if his aim of lighting one of the matches is to be satisfied. He thus has good justification for believing, of the particular match he proceeds to pluck from the box, that it will light. This is what occurs, too: the match does light. However, what the pyromaniac did not realize is that there were impurities in this specific match, and that it would not have lit if not for the sudden (and rare) jolt of Q-radiation it receives exactly when he is striking it. His belief is therefore true and well justified. But is it knowledge?

Gettier shows us in the paper multiple cases where true justified belief fails to be knowledge. So what exactly is knowledge? Well, this is a whole area of debate in the philosophical community. But out of this has spawned fascinating, interesting work. There are debates about whether the criteria for knowledge requires something internal (i.e. mental) or external (the world, our brain chemistry, etc). There are debates about whether there is one unified conception of knowledge, or if knowledge is contextual. Finally, there are debates on whether knowledge has an analysis at all or if we ought to take knowledge as fundamental and define things like belief in terms of knowledge.

Conclusion

The example given above and the examples given in the paper may seem far-fetched, but it is my contention that programming is a constant Gettier problem generator. How many times have you believed you had found the cause of some bug, and had good reason to believe it, it turns out you were correct, but then when looking closer at the bug realize all your reasons were completely wrong! You had true justified belief, but not knowledge.

Philosophy (well, analytic philosophy, which is the kind I enjoy) is a lot like programming. It goes deep on topics most people don't care about. It introduces concepts that seem overly technical and complicated. There are endless debates and disagreements with no hope of reconciliation. But what I see from philosophy that I find so hard to find in these computer science papers, is an awareness of the complexity of our topics. A willingness to engage with those who disagree. No pretension that your approach is the best approach. No need to have a related work section, because instead, you directly engage with others' works.

Lazy Evaluation of Transactions in Database Systems (pdf)

This is a paper I have found utterly fascinating since I first read it many years ago. Is it a good idea? I have no clue. Does the data in the paper make me think it is a good idea? Probably not. But why should that be the bar? If there is anything this series has hopefully shown you, interesting papers don't need to be practical. The most valuable papers for me have been those that are completely impractical. Papers that make me think. That changes the way I view systems.

That said, what I don't enjoy are papers that pretend to be practical, but ignore the realities of practice. I don't think this paper falls into that category. Here, they are investigating a rather interesting idea, what if instead of executing all the steps of a database transaction immediately, we just record a thunk of what should happen? How would the database respond?

Explanation

How exactly would lazy evaluation of a database transaction work? Well, for full details I'd suggest reading the paper, but for the general idea, we need a couple of things. First, our transactions must be deterministic. Second, we need the ability to decide given the current state of the database if a transaction will abort or commit. Finally, we need to be able to track the write set of a given transaction. With these things in place, we can have full ACID, yet lazy transactions. We can implement this lazy evaluation transaction by use of what they call "stickies". In essence, instead of actually executing our transaction, which may read from multiple fields and then write to multiple fields, we put a stickie note on the values we are going to write to. Once those sticky notes are in place, if someone reads those values, the sticky note tells them what transaction(s) to execute to get the current value.

Now why would you want to do this? Because you can get all sorts of interesting load-handling benefits! In real-world applications, it is (in my experience) quite common to have writes that are never read. Or if they are read, they are read rarely at a much later date. But what is also common is to have rather spiky, inconsistent traffic. When traffic load increases, it can often be hard to keep up with the pace of transaction writes. Having lazy transactions allows us to defer the cost of those writes to later, perhaps off-peak times! This does come at the cost of potential read latency increases, because now reads may have to process transactions that were only lazily written.

In these graphs from the paper, you can see a few things. First, the top graph shows you the traffic in this experimental setup. Transactions per second start low, peak, and drop back. The second graph shows you throughput. Things are a bit confusing here. The black line labeled "stickification" is the throughput from the user's perspective. These are the transactions that completed and were accepted by the system. The light blue "lazy" line gives you insight into how the lazy transactions are running behind the scenes. Finally, the third graph shows the success rate. As you can see, the eager process was not able to keep up with demand and dropped nearly 60% of transactions! While the lazy process did not drop a single one!

Conclusion

I think this is actually a really interesting mechanism. Even if it never found its way into a general-purpose database system, (I can imagine people would find this behavior confusing), it is a fascinating idea that I think could be applied to many fit-for-purpose systems. Something talked about, but not explored in this paper is the idea of realizing these transactions in non-peak times. This will of course happen naturally as read slowly eek in. But in my experience, many real-world systems have times of significantly lower load. I can imagine adapting this process so that when these lower load moments occur, the system executes these lazy transactions. There are so many interesting ideas and opportunities laid out in this paper. I love the way in which it combines programming language ideas (lazy evaluation) and databases in an interesting way. I'd love to see more cross-over between these communities.

On Understanding Data Abstraction, Revisited (pdf)

One of my meta-criteria for choosing a paper in this series is the likelihood we will cover the paper on the future of coding. I won't completely rule out covering papers I think we will do at some point. But for the most part, I've decided to steer clear of them. This paper is one I don't think we'd cover on a podcast because I'm not sure it makes for great conversation. This is one of the difficult things about the podcasting medium. Or at least the medium as I conceive it. Not all papers lend themselves to a free-wheeling discussion about the paper. A lecture might work for a paper, and a blog post almost certainly works, but some things just don't come out well in a discussion.

One class of papers I don't think translate well are those that just tell you facts of the matter. I don't mean of course that there isn't bias in these papers. But simply that the goal of the paper isn't to convince you of anything but to simply tell you about things. In some ways, this paper doesn't quite qualify. It is in fact an argument, but not about something I'd consider a real live debate for me. Are objects and abstract data types different things? I'm not sure I really care about that distinction and yet this paper has stuck with me ever since I read it. So, I want to cast this paper in a different light for you readers who may, like me, not really care about objects vs adts.

Rather than make a division between objects and adts, I want you to view this paper as bridging a divide between the values held by object-oriented programmers and those held by functional programmers. Object-oriented programmers value the ability to have multiple implementations of objects interoperate together. They want systems to be extended not modified. Functional programmers value immutability. They want to be certain that the values they have cannot change out from under them. Until I read this paper, I had believed the view that objects were about encapsulating state. In this paper, I see a very object-oriented way of programming, that constructs completely immutable objects.

Objects

In keeping with my now adopted goal of not replacing the paper, but instead enticing you to read it. I will leave off what William R. Cook has to say about the difference between ADTs (Abstract not Algebraic) and Objects. It is actually quite interesting. He shows how the standard Java way of programming and has us build ADTs rather than Objects. But I won't explain further. Instead, I want to show from his example of what an interface for an object-oriented way of defining sets might look like.

interface Set {
  fn empty() -> bool,
  fn contains(n : int) -> bool
  fn insert(n : int) -> Set
  fn union(s : Set) -> Set
}

Here, in our fictional language, is an object-oriented interface for defining a set. We have not defined in any way a concrete implementation of a set. We have just talked about the operations a set must have. This is one of the keys, Cook tells us, to the object-oriented style. Most notably here, there isn't a means of constructing a set. These will come from objects that conform to this interface. Consider this implementation

struct Empty {}
impl Empty {
  fn new() -> Set {
    Empty {}
  }
}

impl Set for Empty {
  fn empty() { true }
  fn contains(_: int) { false }
  fn insert(n: int) { Insert(self, n) }
  fn union(s: Set) { s }
}

Here we have the empty set (In pseudo rust). It contains nothing. It mutates nothing. To insert things into the empty set, we just return a new set called Insert. To union, just pass whatever set we want to union with. What do Insert and Union look like?

struct Insert {
  set: Set,
  i: int,
}

impl Insert {
  fn new(set: Set, n: int) -> Set {
    set.contains(n) ? set :  Insert { set, n }
  }
}

impl Set for Insert
  fn empty() { false }
  fn contains(n: int) { i == n || set.contains(n) }
  fn insert(n: int) { Insert(self, n) }
  fn union(s: Set) { Union(this, s) }
}

struct Union {
  set_1: Set,
  set_2: Set,
}

impl Union {
  fn new(set_1: Set, set_2: Set) -> Set {
    Union { set_1, set_2}
  }
}

impl Set for Union {
  fn empty() { set_1.empty() && set_2.empty() }
  fn contains(n: int) { set_1.contains() || set_2.contains() }
  fn insert(n: int) { Insert(self, n) }
  fn union(s: Set) { Union(this, s) }
}

Here we see no inheritance, no encapsulating of state. Just completely immutable objects. But we also see object-oriented principles upheld. Most notably, these objects only know about themselves, they do not inspect the representation of the arguments they are passed except through their public interfaces. He calls this term Autognosis.

Autognosis means ‘self-knowledge’. An autognostic object can only have detailed knowledge of itself. All other objects are abstract. The converse is quite useful: any programming model that allows inspection of the representation of more than one abstraction at a time is not object-oriented.

But I must admit, these objects are not the ones that got me so excited about this idea. They are fairly straightforward, but when you embrace this style of behavioral abstraction you can make some much more interesting implementations.

struct Even {};

impl Even {
    fn new() -> Set {
        Even {}
    }
}

impl Set for Even {
    fn empty() -> bool {
        false
    }
    fn contains(n: int) -> bool {
        n % 2 == 0
    }
    fn insert(n: int) -> Set {
        Insert::new(self, n)
    }
    fn union(s: Set) -> Set {
        Union::new(self, s)
    }
}

struct Full {};

impl Full {
    fn new() -> Set {
        Full
    }
}

impl Set for Full {
    fn empty() -> bool {
        false
    }
    fn contains(_: int) -> bool {
        true
    }
    fn insert(_: int) -> Set {
        Self
    }
    fn union(_: Set) -> Set {
        Self
    }
}

struct Interval {
    n: int,
    m: int,
}

impl Interval {
    fn new(n: int, m: int) -> Set {
        if n > m {
            EmptySet
        } else {
            Interval { n, m }
        }
    }
}

impl Set for Interval {
    fn empty() -> bool {
        self.n > self.m
    }
    fn contains(i: int) -> bool {
        self.n <= i && i <= self.m
    }
    fn insert(i: int) -> Set {
        Insert::new(self, i)
    }
    fn union(s: Set) -> Set {
        Union::new(self, s)
    }
}

Here are sets that not only are completely immutable, but they also store a large quantity of things (infinitely many), without storing almost anything at all. I actually made my own little implementation of a language like this in Clojure and wrote quite a number of things in it.

Conclusion

The paper explains all sorts of limitations with this particular interface for sets and the tradeoffs that come with object-oriented interfaces. How do you add the intersect operator to this interface? You can't. Why? Because there is no way to tell if two set intersections are empty without iterating over the sets. But how do you iterate over these infinitely large sets? I find all of these tradeoffs fascinating and wish this was the kind of stuff talked about with Object Oriented software (in before someone tells me that of course this stuff is talked about everywhere and I'm just not reading it). I'd highly recommend trying to write some software in this style. It's a fascinating set of constraints that make for some really interesting programs.

Large Models of What? (pdf)

We haven't really talked about LLMs much on the future of coding. In many ways, this may seem like a striking omission. Isn't it a podcast called the "future of coding" and aren't LLMs so obviously some part of that future? I think that is undoubtedly true. I don't see LLMs (or whatever architecture takes their place) being tossed aside at any point. I use them every day and they are quite handy. But I think the main reason we haven't talked about them is we like to talk about things that aren't being talked about everywhere. There is no end to the LLM hype and no shortage of learning materials. So I figured in this advent series I'd cover something a bit different. In Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency, Abeba Birhane and Marek McGann show how LLMs fall short of the human practice of "languaging".

What language models get wrong

Now if you are an LLM fan, you may be expecting that this paper will tell you about "something that LLMs can't do", but what they really mean is what current LLMs can't do. There have been countless predictions people have made about textual patterns that will never be produced by LLMs. "e.g. LLMs will never respond correctly to question X". But that is not at all what this paper is about. Instead, it is a paper offering a different perspective on language than usually assumed by the LLM crowd instead taking what they call the "enactive" view.

[T]he enactive approach to language starts not with tokens of verbal activity, but with the fundamental issue of agency, embodiment, precarity, and how meaning arises within situations where things matter to those involved.

The enactive view focuses not on the formal properties of language, not on its particular modalities, but on the social situation in which language occurs. Utterances by people are not mere sounds. Instead, they are responses to environments, and to the agent's needs, they are a means of communication. Language is a human practice which, in order to be understood properly, cannot be divorced from the uses it is put to.

One aspect of these uses is the way in which language rapidly changes. There are some suggestions that nearly 10000 new English words are created every day. Even within one conversation, we create words for a particular meaning to get across our particular purposes. The point here is not that LLMs are completely incapable of doing that, but that we do this as an outpouring of our agency and our need to communicate. This is something that LLMs entirely lack.

Conclusion

The paper is honestly a bit academic at times. By that I mean, it is written in that contemporary academic style. While I can't really fault them for it, I do wish we could move away from this style. This is one of the reasons I often enjoy older papers. There seemed to be a willingness to allow the individual voice of a writer to stand out a bit.

That said, I think the paper is full of interesting insights. The paper is also chock full of citations for papers that look quite interesting about the ways in which use language. I have left out so much detail from the paper because I'm not sure a quick summary would do it justice. If you are skeptical about LLMs and want something academic to back you up, definitely read this. If you think LLMs clearly do everything a human does when it comes to language, definitely read this. If you just want to read more of my blog posts (who doesn't), you can see my thoughts on the same topic.

What is a Game? (pdf)

Okay, so I'll come clean, I might be cheating with today's paper. Does this paper have anything to do with computing? No. But can we make it have something to do with computing? Absolutely! I mean, games have to be one the best things we've done with computers, so a paper giving a definition of a game has to count. And since I'm the one making this series, I've decided it does. But you might be wondering, why even write a paper on the "what is a game"? Well, because Wittgenstein said it was impossible.

Consider, for example, the activities that we call ‘games’. I mean board games, card games, ball games, athletic games, and so on. What is common to them all? – Don’t say: ‘They must have something in common, or they would not be called “games”’ – but look and see whether there is anything common to all. – For if you look at them, you won’t see something that is common to all, but similarities, affinities, and a whole series of them at that.*

Not wanting someone to be wrong on the internet, Bernard Suits published "What Is a Game?" in 1967. Over a decade later, he would publish THE GRASSHOPPER: Games, Life, and Utopia a much longer treatment of the same topic. In this version, the Grasshopper, part Jesus, part Socrates, dies and his devoted followers must puzzle out and understand his teachings about games being the ultimate way of life. But as this is a much longer book, we will stick with the paper.

What does it mean to play a game?

To play a game is to engage in activity directed toward bringing about a specific state of affairs, using only means permitted by specific rules, where the means permitted by the rules are more limited in scope than they would be in the absence of the rules, and where the sole reason for accepting such limitation is to make possible such activity.

This paper is ingenious in its argumentation. It defends its definition of what it means to play a game by considering alternative proposals and by looking at many different circumstances in which can be said to be playing a game or not playing a game. We will not follow the structure of the paper entirely here, as I think you ought to read it yourself. But instead, I will try and explain some of the elements of the definition above.

First, we have the idea that we want to bring about a specific state of affairs. In many cases, you can think of this as the win condition of the game. (What about games that don't have a win condition? Are they games? Are they play? Not sure). In chess, this checkmate, in a race it is getting across the finish line golf is having the fewest number of strokes, etc. But in all of these games, there are rules that limit what we can do. We can't for example, pick up the ball in put it in the hole for golf, if we did, we would not just be cheating, we would be no longer playing golf.

But as Suits points out, there are all sorts of rules that don't have anything to do with games. For example, I might have a rule for myself, that I shouldn't lie. Perhaps my goals in life would be easier to accomplish if I did lie. So, there is some state of affairs I want (let's say money) and the rules I have imposed on myself limit me in achieving that end, is "getting money without lying" a game? No, I won't lie not because I'm trying to play some sort of game, but because I think it's wrong. This is unlike say following the rules of chess. I don't think there is anything important about moving the knight in an L, the rule is just there to make chess possible.

Conclusion

I think this is an incredibly rich area to read in. Grasshopper is a wonderful book if you love dialogues. But the book I want to recommend in this area of study is a bit different. It is Games: Agency as Art by C. Thi Nguyen. One goal of the book is to argue for a different way of appreciating games as art. That their medium is that of agency. That they encode ways of acting in the world. They allow us to pass down these different ways of acting. I think it's a beautiful picture.

But at the same time, it is a reflection on the role that games play in our lives. The ways in which we can approach games. Nguyen talks about "Achievement Play" and "Striving Play". Both players play to win, but the striving player is trying to win because that's what makes it so they can best enjoy the game. In other words, the striving player temporarily takes on the end of winning, whereas the achievement player takes winning as their ultimate end. Professional poker players want to win to earn money. There isn't anything wrong with achievement play. But striving play can have a lot of benefits.

I've started to look at a lot of the programming I do as striving. Do I ultimately care about earning victory points in some game? No. Do I ultimately care that some customer gets their new feature? Equally no. But that doesn't mean I don't care at all! When working in a job, I can be a striving programmer. Temporarily taking on the aims of my day job, not because they are my ultimate ends, but because doing so makes the work more enjoyable. In my spare time projects, I don't think that making my debugger is going to change the world or provide me with meaning in my life. But taking on that goal and striving to achieve it the best way I can helps me tremendously.

These ideas and Suits' definitions may seem miles apart. But if you read Nguyen, you see just how much this Suitsian notion of games matters to him even though he thinks it falls short of being an exhaustive definition. Suits helps us focus on aspects of a game, pulling out the bits to help us understand the parts. Where is our analysis of programming concepts? Where is our reflection on our own practice in this kind of fashion? Why are we forever behind other disciplines in meta-analysis?

Beyond Being There: Making Remote Work Better (pdf)

I absolutely love remote work. I've been doing it since before the pandemic. I am more productive doing it, I don't have to deal with all of the annoyances and politics that automatically occur when you get a bunch of people in an office. But I have to admit, the tools for working remote are awful. Zoom is a passable tool for synchronous communication video calls. Slack is alright. GitHub's fine. But the collaborative interfaces for working remotely are poor imitations of what you might get in person. Let me be clear, I don't mean that being in an office fosters more collaboration, I find the opposite. I just mean that the medium through which we collaborate remotely is awful.

Is anyone trying to actually solve this problem? Despite this huge shift that has occurred (yes, I know some people really want everyone back in the office, but the cat's out of the bag), I don't see any serious attempts to make this better. Why is that? I believe it because all the attempts have fallen into the trap that this paper from 1992 warned us against. They all try to recreate the feeling of "being there". Look at all these silly "metaverse" demos. They all want us to feel like we are in person, but is that really the goal?

How to move beyond being there?

It is tempting to think that with perhaps a little more screen resolution, a little more fidelity in the audio channel, and a little more tweaking to bring the machinery in conformance with subtle and long-established social mechanisms such as eye contact, telecommunications systems will achieve a level of information richness so close to face-to-face that for most needs it will be indistinguishable.

But will they ever be close enough? It is clear they can, for example provide a cost-effective and efficient alternative to business travel to a distant location, and may be superior to audio only telephone for some communicative needs. We have no argument with that. But is this general approach going to be adequate for the long term? Is it powerful enough to see us through to achieving the goal that those at a distance will be at no real disadvantage to those colocated?

What if instead of focusing on imitating "being there" we instead considered the strengths of our computer-mediated modes of communication? Computers are not good at recreating all the high-fidelity cues we get during in-person conversation, but what are they good at? The paper outlines three features.

1. asynchronous communication
2. anonymous communication
3. automatically archiving communication

One fascinating aspect of this paper is that it was written at Bellcore one of the off-shoots after Bell was broken up. To discover how computer-mediated communication might go "beyond being there" they built and are planning to build real systems and experiment with how people interact with them. One simple example was the ability to attach an "ephemeral interest group" to "any object in a community's electronic 'space'". Rather than having centralized, unidirectional broadcasting, people can chat about anything contextualized to a particular topic. How many times have I wanted to chat about that meeting invite in a zero-friction way?

The paper goes on to explore things like semi-synchronous discussions, anonymity, and the ability to ease meeting others. All the discussions are preliminary and suggestive of research to be done, but they provide a view of a remote work environment where the means of communication are integrated rather than disparate. Today's corporate communication landscape is owned by so many different vendors, Slack for chat, zoom for videos, and Microsoft/Google for email. Even when a company buys into a single vendor, the connections between the products are completely simplistic and artificial. There are vast amounts of improvements we could make to these systems, improvements that do not require groundbreaking new tech (like some VR headset), but instead better design principles.

Conclusion

Xerox PARC asked, "What does the office of the future look like?". Who today is asking "What does the remote office of the future look like?" We are stuck in the early days of pervasive remote work, where we still try and mimic what the office provided us. But will we be stuck there forever? Progress is not inevitable. Billions of dollars have been sunk into imitating "being there", who will focus on moving beyond?

Implementation is Semantic Interpretation (pdf)

I read this paper this morning and I had the time to just finish this post before my day began. But as has been the case for these posts so far, I procrastinated on it until bedtime. Maybe tomorrow we will break that pattern. Anyway, this is another in the series of philosophical papers. Though I have to admit, after reading the paper I was left feeling a bit empty. The paper doesn't do a great job explaining exactly what semantic interpretation is. So I sought out some more information and found that there was another paper by the same author called "Implementation is semantic interpretation: further thoughts". It seems I should have read that paper instead, but alas, I did not. What I have done however is skimmed it. Perhaps I will read that paper for another advent day? It actually directly interacts with the Searle paper we read earlier.

Whether or not we read that paper on another day, my thoughts on this paper are definitely colored by the things I skimmed through in the other paper. But for now, we will leave that aside. What does this paper claim? Well, that implementation is a unique notion separate from a bunch of different notions people may have thought we could define it in terms of. Implementation is a semantic interpretation. Rapaport believes this goes beyond merely computer science examples:

The following non-computer-science examples are clearly of the same type as these paradigms, even though we don't, normally, use the term 'implementation' in discussing them: A performance is an "implementation" of a musical score or play-script. A house is an "implementation" of a blueprint. A set-theoretic model is an "implementation" of a formal theory.

What the hack is a "Semantic Interpretation"?

But what exactly is a semantic interpretation? Well, all we are given is a structural model of it.

Semantic interpretation requires two domains and one relation: a syntactic domain, characterized by rules of "symbol manipulation" (perhaps suitably generalized to be able to deal with domains that are not strictly speaking "symbolic"); A semantic domain, similarly characterized; and a relation of semantic interpretation that maps the former to the latter.

What exactly is a syntactic domain vs a semantic domain?

Put this way, there is no intrinsic difference between the domains; what makes one syntactic and the other semantic is the asymmetry of the interpretation mapping. Thus, a given domain can be either syntactic or semantic, depending on one's interests: Typically, we understand one domain (the "syntactic" one) in terms of an antecedently understood domain (the "semantic" one).

In the further thoughts paper, Rapaport tells us that "the semantic enterprise is seen to be, ultimately, a syntactic endeavor, which I have called ‘syntactic semantics’". So I'll admit, it is entirely unclear to me what exactly semantic interpretation is supposed to be. It seems, like Monads, you aren't supposed to have anything beyond the formal, structural properties. But then in what sense is it semantic? In what sense is the "interpretation" providing meaning?

Some sections distinguish things like a concrete vs abstract implementation. The programming examples are mostly clear, though dip into some unnecessary formalisms. Other than that, the paper spends a lot of time relating implementation to individuation, instantiation, reduction, and supervenience. Long version short, all of these are examples of implementation, but not the other way around.

Conclusion

I feel like there is some interesting insight lurking here, but the deflationary attitude taken towards the semantic enterprise throws the whole thing off to me. Consider the abstraction of a linked list. There are many ways to implement it. When we make a concrete implementation, we are providing semantic meaning (particularly in an operational semantics sort of way) to each operation. When we play music by reading a score, we are taking those notes and filling in some detail about how they relate to the world. We apply our own subjective interpretation for the music, taking the meaning of the notes and acting upon them, in the same way we do when given an abstract description of an algorithm that we write down. We are "filling in" the details. We are making the symbols have a specific, rather than abstract meaning. But this is a process we do by actually subjectively interpreting something. Does the record player semantically interpret the record? I think that would be quite a stretch.

Intuition in Software Development (pdf)

If you haven't read Peter Naur's papers you should. If there is one thinker in the computing space I wish I could emulate, it is Peter Naur. This paper is no exception. It is about the role of intuition in software development but is ultimately an argument against applying methods to software development. It is almost certainly a sub-tweet (sub-skeet? sub-toot?) of Dijkstra. It was originally presented at the "Advanced Seminar On The Role Of Semantics In Software Development" and yet is critical of the whole enterprise.

Finally, to the question of this conference, what is the role of semantics in software development? Answer: neither that of the composer, nor of the librettist, the conductor, the hero, or the heroine, but that of the prompter, who does nothing but tell the actors things they know already, but that may momentarily have slipped from their minds.

Peter Naur is a computer scientist who seems to truly understand software. He is someone who connects practice and theory. He is a writer who gets at the underlying truths of programming in a way no one else does.

How to tackle this paper?

I'm now on day 6th of this exercise, and I feel I've earned a moment to reflect. One of the things I'm unsure about is how I ought to cover these papers. One way I can go about this is to give you a cliff notes version of the paper. To save you from having to read the paper yourself. To present the ideas in these papers in an easier-to-read digestible form. I honestly don't have much problem with that. In many ways, I feel that is my role on the future of coding podcast. Not everyone will take the time to read the papers we discuss and I'm fine with them not and letting us do that for them.

But the other way I can take this assignment is that I should get you to read the paper. That my real goal here is to entice you into reading the paper. But I'm unsure if that can really work. I know people come to these blog posts with a certain amount of time they are willing to spend on it. Most people don't come here with the luxury of then spending 30+ minutes reading a carefully crafted paper. Perhaps they save them for later and do actually read them. Or perhaps they, like me, save them with the best of intentions just to have an ever-growing list of saved links.

This paper makes me question this very issue because I'm unsure I can do it justice. Naur is a wonderful writer, but I'm afraid his argumentation style is a bit vague. Unlike the philosophers he often cites, he does not put his arguments into a logical structure. Instead, he pokes and prods at you at various points. He tries from many different angles to get you to see the point he is trying to make. His conclusions seem to many to be an over-reach. So what am I supposed to do?

The Argument

I'm still unsure about the right way to proceed, so I will try to take both paths at once. This is a paper with a radical thesis, namely that most software methods do not and cannot work. The one exception granted is the checklist. Checklists are good all other methods (insert agile-inspired garbage here, but also insert all formal methods) are not. But why exactly do we come to this conclusion? Because, the most pervasive, impossible-to-eliminate, all-important aspect of any software development process, is intuition.

What exactly is intuition? Well, it isn't some spooky sixth sense. Peter Naur quotes Quine in what I believe is the clearest definition we get from Naur.

Twice I have been startled to find my use of 'intuitive' misconstrued as alluding to some special and mysterious avenue of knowledge. By an intuitive account, I mean one in which terms are used in habitual ways, without reflecting on how they might be defined or what presuppositions they might conceal.

Our intuition is the most basic way we act. It is the things we do without reflection. It is the thing that allows us to make connections between various elements. But intuition is not some fixed substance. It morphs and changes as we learn new things. Things we once were not able to do using our intuition become intuitive for us (think about driving a car now vs when learning).

But intuition isn't merely something we gain in a specific skill via experience, it is pervasive in every act we do. Most importantly is what undergirds our understanding of language. We do not apply rules and methods for understanding language. Instead, we intuitively grasp the meanings of others' utterances. This isn't to suggest that our understanding is without flaw, no our intuitions can go wrong and frequently do. What it is instead to suggest is that there are certain acts we perform, for which we cannot explain our actions at some deeper fundamental level. We do not go through a series of steps consciously to perform these actions and couldn't recreate them if we tried.

Software

Software development is "an interplay of the intuitive knowledge had by the programming person and the real world accessible to that person, which includes the texts used and produced by the person". Naur walks us through how our knowledge of the problem we are trying to solve, the texts (documentation, code, etc) we produce/consume, and the world change over time as an essential part of the software development practice. Fundamental to this practice is this changing of knowledge over time which is grounded in our intuition. Absent this intuition we could not even start this process.

But further, no method can be substituted for our intuition, because any proposed method would itself need to be comprehended by our intuition. Our intuition is what tells us when to apply certain methods. It is what shows us the connection between the methods and the world in which we are applying them. It is what helps us understand the words, concepts, and terms used by the method.

Conclusion

Perhaps I've stopped too short with the summary. But I can't underscore enough how much I feel people ought to read this text. If you don't want to, but have the time to listen to a podcast, go listen to Ivan and I talk about Programming as Theory Build by Naur. Naur here provides in my mind a definitive end to all sorts of method debates. Should people be forced to use types? Will TDD save us? Is agile the answer to the software crisis? Naur shows us why these things can't be the answers. He gives a construction of what the software development process is that isn't about historical accidents, that isn't about industrial scale concerns. But instead engages with the human practice. It doesn't try to hide behind empirical work or fancy discussions of the brain, it looks at what makes programming practice possible.

I don't expect those of you who love methods to be convinced by this paper. I don't offer it as such. Instead, I offer it as a different lens to see through. When people tell you they like dynamically typed languages, when they tell you they don't find all that documenting and planning useful, when they tell your method doesn't help them, don't see them as resisting for no reason. See them as wanting to make the most out of their intuition, as trying to build software in a way that works best for them. They aren't rebels. They aren't ignorant. Their intuitions are just different than yours.

Worlds: Mutability with Control (pdf)

Today's paper is a bit different. Rather than philosophical musings or a rant, we will be looking at a fascinating language feature that has not made its way into any programming languages I'm aware of. A feature first presented in this early paper by Alessandro Warth and Alan Kay. It's an idea that strikes a very interesting middle ground between a completely immutable way of programming and an unconstrained mutable system. It does that by bringing a bit of sci-fi into your language, by introducing a first-class concept of other worlds.

Let's start with a code example to make this all more clear.

let myObject = {
  x: 1,
  y: 1,
};

let myWorld = thisWorld.sprout();
in myWorld {
  myObject.x = 42;
  myObject.y = 42;
  console.log(myObject) // { x: 42, y: 42}
}

console.log(myObject) // { x: 1, y: 1}

Here we can see the most basic operations on a world. First, is thisWorld, which refers to the current world. Second is the operator sprout this is what creates a new world. Finally is our in operator, which allows us to run computations inside a world. In that, any mutations to any variables we make are constrained within that world. But what if we don't want to throw away our changes? That is where commit comes in.

let tempWorld = thisWorld.sprout()
let failed = false;
try {	
  in tempWorld {
    runPotentiallyDangerousMutations();
  }
} catch (e) {
  failed = true;
}

if (!failed) {
  tempWorld.commit();
}

Now, inside runPotentiallyDangerousMutations we are free to mutate things all we want without working about needing complicated logic to undo them if things throw an exception. We only commit those changes if no exceptions were thrown.

This is pretty much all there is to the notion of worlds. But its implications are quite interesting. One infamous problem in JavaScript is that prototypes can be extended by anybody, but that can break all kinds of things! (There's a reason we almost had smoosh and smooshMap as methods on Array.) But with worlds, we can escape this.

let ourWorld = thisWorld.sprout();
in ourWorld {
	Number.prototype.fact = function() { ... };
}
  
// later
in ourWorld {
  console.log((5).fact())
}

There are more examples in the paper, like a hierarchical generic undo system that are worth checking out. But I actually wonder about use cases not explored. Could we start with a blank world, no standard objects, and build up a world with just the things we want to expose? Could we make a pure world, where the only objects and functions exposed can't do side effects? Or a deterministic world? In other words, I'm interested in using this as a way to find language subsets where we can have certain kinds of guarantees for sandboxing or for analysis.

I am intentionally linking to an earlier version of the paper (because I like it more), but in a later version they discuss in more detail aspects of the implementation. Conceptually though, you can think about objects as not storing their own properties, but instead a table storing entity, attribute, and value. A world could have its own table, with a pointer to the parent world. Or we could do something like eavw. This model actually makes me think of datomic and CRDTs. Is there an extension of this world concept but for times? Could we use this general mechanism to replay all the mutations going on in a single world by adding making it an eavt like datomic?

Conclusion

I love this concept. From the first time I read it, I've always thought it would be a wonderful feature to have in a language. If you are familiar with immer from the react world, this may seem familiar. But having this at the language level makes it all the more interesting. It's sad that JavaScript hasn't gone in these more interesting directions. Reading this paper always reminds me of James Long's original sweetjs. I made so many things with it. I love the vision of a javascript with macros. I loved a javascript that could move forward without a committee needing to declare it. I long to get back that sense of playing with languages I once had. That's why I've retreated to building my own. Slowly but surely, I can make it a place where interesting language ideas can be explored. A place free from the baggage of more successful languages. I place where people get to carve their own path. We haven't had a language like that be popular in quite some time.

Is the Brain a Computer? (pdf)

I can't help but write about these more philosophical papers. Perhaps you think I'm taking advantage of this "related to computers" aspect of my advent of papers. But I think it couldn't be further from the truth. In order to understand if the brain is a computer, we have to understand what it even is for something to be a computer. If we don't understand that, how can we understand computers at all? But there is a further reason I wanted to talk about this paper, it is a paper that follows a classical formula that I think makes for the greatest philosophical papers. It begins with things we all agree with and shows us how merely from those facts we have found ourselves in a contradiction. (Or at the very least a very uneasy tension).

1. The brain is a computer
2. A computer can be realized by many different physical systems

The first of these is that the brain is a computer. (I've dropped the digital modifier here because not much is made of that "digital" part and I don't want the objections to be "Well it isn't digital, but it is a computer.) I take it for granted that most of my readers believe this to be the case. What do I mean by the brain being a computer? I mean that you take serious (perhaps literally is a better word) statements like "the brain processes information", "the brain computes how fast the ball is flying", "we can explain cognitive functions by talking about the programs they realize", and "mental states are computational states". It seems that even among the non-programming public, this sort of view is well-known and accepted.

As for the second, I take for granted you accept that there is nothing special about our conventional computers. We can build computers out of literally anything. Some mediums are more practical than others. But there is nothing special about the reason we can't make a computer out of cogs and levers, hydraulics, marbles, even pidgins. All of these things can be a computer simply by assigning some 0s and 1s in the right way (of course there is nothing special about 0s and 1s, they are just convenient symbols for our purposes). Computation can be realized in many different physical ways. I think this fact isn't quite as appreciated by the general public, but I expect programs, once they give it some thought will find themselves nodding together in agreement.

It is from these two beliefs that Searle draws his problem. Searle intends to show us that if a computer can be multiply-realized in this sort of fashion, it undermines the notion that the brain is a computer. Even if you end up buying his argument, how can that not entice you?

Being clear

Now before we dive into that argument. I think it is important that we make clear some things that Searle wants to be clear about. He starts the article by distinguishing between three questions.

1. Is the brain a digital computer?
2. Is the mind a computer program?
3. Can the operations of the brain be simulated by a digital computer?

He is only interested in this paper in answer 1. But he tells us his answers for 2 and 3. He says that for 2 the answer is no, and to read his Chinese Room argument for that (if you haven't read it, I highly recommend it). And for the 3 the answer is obviously yes. So when we are asking about 1. We are not asking about Strong AI or Weak AI, we are asking about what he calls Cognitivism.

His further clarifications come with some great snarky comments that I have to reproduce here:

First, it is often assumed that the only alternative to the view that the brain is a digital computer is some form of dualism. The idea is that unless you believe in the existence of immortal Cartesian souls, you must believe that the brain is a computer. Indeed, it often seems to be assumed that the question of whether the brain is a physical mechanism determining our mental states and whether the brain is a digital computer are the same questions. Rhetorically speaking, the idea is to bully the reader into thinking that unless he accepts the idea that the brain is some kind of computer, he is committed to some weird antiscientific facts.

Second, it is also assumed that the question whether brain processes are computational is just a plain empirical question. It is to be settled by factual investigation in the same way that such questions as whether the heart is a pump or whether green leaves do photosynthesis were settled as matters of fact. There is no room for logic chopping or conceptual analysis, since we are talking about matters of hard scientific facts.

He says thirdly, people often overlook fundamental questions, like what is a computer? What is a computational process? And under what physical conditionals exactly are two systems implementing the same program?

Searle thinks none of these things. He isn't a dualist, he thinks philosophy has a lot to say on this matter, and he thinks we must understand what we mean by computation and how that is supposed to map to the brain.

Turing Machines and Multiple Realizations

I will not rehearse for you here what you probably already know, just mention the most important points. To see if something is a computer isn't to look inside and discover if it works as a Turing machine, if there are some ones and zeros and a tape, instead what we have to "look for something that we could treat as or count as or could be used to function as 0's and 1's. In other words, we have to provide a model for how to think about how to consider a system as a computational process. For conventional computers, we look at currents or magnets. But we really can treat anything as a computer as long as we have some coherent scheme for interpreting it as such.

But now if we are trying to take seriously the idea that the brain is a digital computer, we get the uncomfortable result that we could make a system that does just what the brain does out of pretty much anything. Computationally speaking, on this view, you can make a "brain" that functions just like yours and mine out of cats and mice and cheese or levers or water pipes or pigeons or anything else provided the two systems are, in Block's sense, "computationally equivalent". You would just need an awful lot of cats, pigeons, waterpipes, or whatever it might be. The proponents of Cognitivism report this result with sheer and unconcealed delight. But I think they ought to be worried about it, and I am going to try to show that it is just the tip of a whole iceberg of problems.

You may not worry about this multi-realizability fact because you think this is true of any number of functional systems. We can make carburetors and thermostats out of tons of different materials for example. But as Searle points out these are quite distinct notions. Carburetors and thermostats are defined by their physical effects (you can't make a carburetor out of pidgins) but computation isn't defined by physical effects, they are defined by syntactic assignment. We can take any sort of physical effects and assign 0's, 1's, and transition states and we can have a computational model.

This means that computation isn't a matter of physics, but an agent or observer-relevant phenomenon! To call something computational, a syntactic assignment must be made to it. This undermines the very explanation we were seeking in the brain. I think Searle puts this very clearly.

There is no way you could discover that something is intrinsically a digital computer because the characterization of it as a digital computer is always relative to an observer who assigns a syntactical interpretation to the purely physical features of the system.

This point has to be understood precisely. I am not saying there are a priori limits on the patterns we could discover in nature. We could no doubt discover a pattern of events in my brain that was isomorphic to the implementation of the vi program on this computer. But to say that something is functioning as a computational process is to say something more than that a pattern of physical events is occurring. It requires the assignment of a computational interpretation by some agent. Analogously, we might discover in nature objects which had the same sort of shape as chairs and which could therefore be used as chairs; but we could not discover objects in nature which were functioning as chairs, except relative to some agents who regarded them or used them as chairs.

Conclusion

This paper contains a ton more exposition and argumentation to back up this main point. He talks about the inadequacy of trying to explain how our brain works by talking about the "programs" it runs. Simulating a typewriter via word processing software tells us nothing about the causal mechanisms that make a typewriter work. In the same way, discovering how to simulate a brain will tell us nothing about its causal mechanism. Consider any non-brain physical system (typewriters, hurricanes, etc). Talking about formal properties they share with their computational simulation does not provide us with answers to how things systems work. It might help us simulate behavior, but our programs need not have the same causal properties.

There is much more in the paper to consider and it is highly readable. Since I've been getting into the habit of sharing my opinion in these posts I will say my part I fully agree with Searle's argument. A physical process can implement any number of computations. But it doesn't do so intrinsically, it only does so we consider it as such. This is why I'm hesitant about things like weaving patterns being called computational. Not because they aren't "real programming". Not because they are lesser. But because they only become computation when people consider them as such.

Should we consider everything as computation? Is that a helpful lens for all problems? No, I don't think so. Nor should we elevate computation as if it is the greatest highest end. We are not computers. The universe is not a computer. Nothing is a computer, but anything can be treated like one.

Google's Awful Paper on Technical Debt (pdf)

Okay, I'll admit that I didn't have a ton of time today, so I picked a paper that doesn't require much deep thought. A paper that isn't weird at all. I tried for quite longer than I should have to find a paper I could adequately cover in the amount of time I had to cover it. But I'm afraid all the good papers require more care than I could commit to. So today, you get a terrible paper. A paper about how Google has dealt with technical debt.

This paper is chalk full things I absolutely hate about "empirical" software studies. (I have expressed those feelings in many words elsewhere). It is laden with value statements disguised as facts. The empirical aspect of the research contributes very little to our understanding of the problem. For those who like this kind of stuff, the article completely lacks any numbers and discussion of statistical significance or any of that. (I think that is completely beside the point). Overall it is just a lackluster, boring paper that reads more like a promo for Google than anything else.

Let's start with a summary. Every quarter this team surveys engineers. By the paper's own admission, they get a very low response rate which basically amounts to 1/9 of the engineers. (It is unclear how evenly spread these are. Do they rotate them? Is it the same group every time?) But regardless of all of that, they wanted to find what kinds of technical debt were hindering engineers and after some back and forth, found that with this selection of 10 options, less than 2% picked "other". As they put it, "This provided us with a collectively exhaustive and mutually exclusive list of 10 categories of technical debt".

• Migration is needed or in progress
• Documentation on project and apis
• Testing
• Code quality
• Dead and/or abandoned code
• Code degradation
• Team lacks necessary expertise
• Dependencies
• Migration was poorly executed or abandoned
• Release process

But not wanting to stop there, they decided to try to find "metrics based on engineering log data that capture the presence of technical debt of different types". But after trying 117 different metrics "No single metric predicted reports of technical debt from engineers; our linear regression models predicted less than 1% of the variance in survey responses.". Yet despite this, they were able to fix technical debt! What did they do?

Motivated in part by our early findings on technical debt, an interested community within Google formed a coalition to help engineers, managers, and leaders systematically manage and address technical debt within their teams through education, case studies, processes, artifacts, incentives, and tools.

And of course, given their measurements, they know that these efforts worked because:

Overall, our emphasis on technical debt reduction has resulted in a substantial drop in the percentage of engineers who report that their productivity is being extremely to moderately hindered by technical debt or overly complicated code in their project. The majority of Google engineers now feel they are only “slightly hindered” or “not at all hindered” by technical debt, according to our survey.

Subjectivity

They do have some interesting, non-empirical insights. In fact, it is the best part of the paper. When discussing why their metrics failed to capture technical debt, they point out that people conceive of something as technical debt based not on what the system is, but on what the system could be. Or as they put it.

conceiving of the ideal state of a system and using that imagined state as a benchmark against which the current state can be judged

But rather than realizing the subjective implications of this insight, they instead talk about the need to model it and how modeling it is a "prerequisite to effective management of technical debt". What is so interesting about this insight is it calls into question all of their findings. Did technical debt actually decrease? Or did engineers stop seeing the ideal state they can benchmark against?

I have worked on countless projects where the weight of technical debt felt like a burden. One year later it didn't feel so burdensome. Why? Not because we had fixed all the problems, but because we had given up on our ambitions. We didn't feel a hindrance because of the technical debt, because we had stopped having aspirations for the system to be better. There isn't even so much as a discussion of this possibility.

Nor is there any questioning of the central metaphor here. Is technical debt truly a coherent topic? Is asking questions on a survey like this a good way to break it down into its parts? Do we have any good reason to justify statements like:

Technical debt isn’t unequivocally a bad thing, after all. Just like financial debt, technical debt is one component of a strategy for trading off velocity and (some form of) quality or completeness.

Nowhere in the paper do they stop and question the dogma, trite statements that are made about technical debt. There is no examination of the deeper causes or looking at things conceptually. This is a far cry from much more interesting, illuminating work like Peter Naur's Programming as Theory Building

Conclusion

In some ways, it feels like I wasted a day by giving you a rant about this silly paper. In other ways, it feels great to just rant a bit. Hopefully, this paper will serve as a contrast for all the interesting papers I hope to cover in this advent season. The papers that don't just repeat company-focused talking points. The papers that are not rife with value-laden statements disguised as facts. The papers that don't make claims without arguing for them at all. The papers that actually ask questions about what programming is, that take seriously the human experience of programming. That don't have beliefs that simple metrics and surveys somehow solve deep meaningful problems.

What is Software Anyways? Where Does it Exist? (pdf)

For this next paper, we are looking at a question I'm sure many will think is pointless. What exactly is software? But by this, we mean something a bit different than offering a definition. The goal here isn't to figure out how to split the world into software things and non-software things. Nor is it to distinguish hardware from software. Instead, the question is, assuming we have software, what sort of thing is it? This may feel like a weird question. But why? I think it's not a very fashionable question. After all, what kinds of things are there? Are there ghosts and spirits and other spooky substances? Of course not! What could there be other than things made of atoms?

Nurbay Irmak gives us what I think is a rather interesting view. The aptly named "Software is an Abstract Artifact" tells precisely what software is. But if you aren't a philosophy nerd, that title might not be enlightening to you. My goal in this brief paper summary is to make these notions clear. I will not attempt to argue for Irmak's view but simply state it. To try and make clear what the paper presents. But you must be reminded, this is merely a summary.

What is software made of?

Let's start with a particular example of software and consistently use it throughout. Irmak often uses Windows 7 so we will stick with that example. Windows 7 is software, I think we can all agree to that. But what is Windows 7? Let's say I have a disc that contains Windows 7. Is Windows 7 identical to that physical disk? If I incenerate that disk do I destroy Windows 7 and it no longer exists? I doubt many would think so. Is Windows 7 identical to the bytes on the disk? What if I store the bytes in a different order, or I compress the bytes? Does that mean I no longer have a copy of Windows 7 but some totally different piece of software? That seems unlikely.

These are the kinds of questions Irmak wants us to ask. Is Windows 7 identical to a copy of it? Is it identical to some text? Is it identical to an execution of it? Or is it identical to what he calls "the algorithm" of it? Algorithm here might be better understood as some abstract structure. Take the program that is Windows 7 and convert it to some formalism like lambda calculus or a Turing Machine, consider Windows 7 from some mathematical point of view. That is what Irmak means by "algorithm", the mathematical structure that Windows 7 realizes. Is Windows 7 identical with these things?

But what exactly does it mean for two things to be "identical"? Well, I common way to think about this in philosophy is called "Leibniz's law" or "Identity of indiscernibles". Put simply it says that if you have potentially two objects X and Y, X is identical to Y just in the case that X has all the same properties as Y and vice versa. So, consider the case where I am holding a red ball in my right hand and a red ball in my left hand. Imagine that these balls weigh the same, look the same, even down to the microscopic level, and are made of the same material. We know they aren't identical because one has the property of "being held in my right hand" and the other lacks that property. So, when we are looking for what Windows 7 is identical to, this is the criteria we are looking for.

What Software can't be

When asked above if Windows 7 was identical to the disk containing Windows 7, we were actually applying this principle. Windows 7 has the property "surviving being incinerated", but the disk does not. For the bytes example, Windows 7 can be instantiated by many different shapes of bytes, but a particular shape of bytes has its shape essentially. We can continue for things like the execution, if Windows 7 is identical to executions, that means if all computers executing Windows 7 were shut down, Windows 7 would cease to exist, but intuitively, that's not the case. And the text of Windows 7, if we put the text in a different encoding, or we changed all the variables, or we moved some function, etc, we'd still have Windows 7, despite the texts themselves not being identical.

It seems Windows 7 lives above and beyond any particular instance of these Windows 7 artifacts. But how are we deciding that? Irmak believes we should start with our everyday notions. When we talk about software, these are the kinds of things we generally believe about it. The way we talk about software, we being the everyday folk, should inform our theory as to what kind of thing software is. This is crucially important for our last potential candidate. Is software identical to the "algorithm"? Well, according to Irmak, the algorithm is a mathematical platonic entity. That is a non-spatial, non-temporal, unchanging thing. If software is identical to the algorithm, it must not be changeable. But software can change and still be the same software!

A patch to Windows 7 does not make it no longer Windows 7. In the paper, Irmak gets into versioning, but as this summary is going a bit long, I will completely alide that discussion. Let's just take it for granted this point is right. If so, software is not identical to the algorithm. What is it then?

Artifacts

As we are told in the title, software is an abstract artifact. What is an artifact? An artifact is any human-made object. A shovel, a computer, and a statue are all artifacts. But these artifacts are concrete? What does that mean? It means they are spacio-temporal. Concrete artifacts exist in time and space, they have a definite location. We can point to them. Software we cannot. Now this may seem all a bit weird, but Irmak shows us that software isn't necessarily alone in this regard. Consider Music. Music has elements like sound structure, score, copy of score, and performance. These are quite related to algorithms, text, copies, and execution! A piece of music isn't identical to any of these things.

The last thing I'll mention here is that Irmak discusses when programs cease to exist. Here are his four criteria.

1. Their authors ceased to exist.
2. All of their copies are destroyed.
3. They are not performed or executed ever again.
4. There are no memories of them.

I think there is some intuitive appeal to these criteria. But I'm unsure about 1 and 4. Is a vague memory of a piece of software enough for it to exist? Do the authors really need to "cease to exist" for the program to cease to exist? What about that first-ever program I wrote that I have no memory of?

Conclusion

Irmak gives us a fascinating look at the ontology of software. I think it's a great paper for introducing the ideas, but as the paper itself says, it is far from a full discussion. In a bonus episode of the future of coding podcast I talk about James Grimmelmann's distinctions between the naive, literal, and functional meaning of code. I think a similar kind of distinction can be drawn here between a program and software. I would say that Windows 7 is software, but two different versions of Windows 7 are different programs. Further, when we consider forks, it becomes tricky to say which fork continues to be the same software, is it the one that keeps the name, or is it the one the community adopts? Programs by contrast have much clearer identity criteria. To me, they are just identical to the "algorithm" in Irmak's parlance. What this means of course is that programs are eternal unchanging things. We don't create programs as much as discover them. That to me is not a fact to run away from, but to embrace.

Advent of Papers: Elephant 2000 (pdf)

I hereby kick off the first annual advent of papers (2024) - Speech Act by Jimmy

You may know John McCarthy as the creator of Lisp. But did you know he created another programming language? Sadly, just like Lisp, McCarthy merely wrote about the idea, not an implementation. But quite unlike Lisp, no one rushed to make implementations of this language. There may be a good reason for this, it is quite different from what you might expect. You may think that you just need to see some example code. But I'm not sure that would serve you super well. You see what McCarthy offered here was not simply a new syntax or a new way of combining old ideas. It was an ambitious idea for what programming in the year 2000 could be like. An idea that adds an element from natural language that is wholly missing in programming languages, Speech Acts.

Speech Acts

Now sadly McCarthy doesn't offer us much in the way of an introduction to speech acts, other than to constantly make fairly vague statements about how speech acts in Elephant 2000 will differ from those philosophers have discussed, so it might be useful for us to spend a little bit of time talking about them before looking at elephant 2000's use of them.

Whereas an act of speech is any act of uttering meaningful words, ‘speech act’ is a term of art. As a first approximation, speech acts are those acts that can (though need not) be performed by saying that one is doing so. On this conception, resigning, promising, asserting, and asking are all speech acts, while convincing, insulting, and growing six inches are not. One can, for instance, resign by saying, “I resign…”, although one can also resign from a position without describing oneself as doing so. - Stanford Encyclopedia of Philosophy

Speech acts are acts we do by speaking. When we make a promise, we are both uttering the words "I promise" and making it the case that we have made a promise. Contrast this with assertions. By saying "Mt. Everest is the tallest mountain", I have not done anything above and beyond my assertion. (If you are interested in Speech Acts I highly recommend Speech Acts by John Serle) Speech acts are a part of natural language that (according to McCarthy) programming languages generally lack. Further, he thinks that unlike attempts to borrow syntax from natural language, it might be beneficial for us to borrow speech acts and add them to our programming languages.

This is precisely what McCarthy aims to do with Elephant 2000 and he gives us a list of "speech acts" he aims to add to it.

1. Assertions: By this, he does not mean an assert keyword or function. He does mean something that must be true. But further that must be "sincere". He adds that this notion of "sincerity" requires us to have "a theory of the beliefs of the program"
2. Questions: A user can question the program and the program can question the user. (It is unclear how these are speech acts imo)
3. Answers to Questions: These are assertions and must be truthful and sincere. But they must also be responsive. Meaning that the answer might be appropriately related to the question asked. The example McCarthy gives is that if someone asks for George's Telephone number, it is unresponsive to say that it's the same as George's wife's husband's telephone number. He adds that there must be a way in the specification language to require the answer to be a string of numbers and not an expression whose value is a string of numbers. (Again is this a speech act?)
4. Commitments and Promises: Here McCarthy imagines both simple promises, which are internal to the program and represent a commitment to perform an action. But also commitments have, for example, legal ramifications for the organization operating the program.

One crucial thing to notice about the "speech acts" is that they all refer to the outside world. The meaning of these programs depends on the outside world in some way. McCarthy believes this is of crucial importance. Elephant 2000 does not merely simulate speech acts, it "can perform some kinds of speech acts as genuinely as do humans". A program in Elephant 2000 for air traffic control would meet its "accomplishment specifications" only if the airplane truly did land safely. The Elephant 2000 program must truly perceive that a plane has indeed landed.

The Past

Beyond these speech acts, Elephant 2000 can refer to the past directly. There is no need for data structures, instead Elephant 2000 never forgets anything. This isn't some kind of persistent data structure or a simple way of storing all past values of program state. Instead, it is supposed to mirror all the interesting ways in which people can refer to the past. Here are some examples given

1. The simplest function of the past is the value of some parameter at a given time, say the account balance of a certain person on January 5, 1991. References to the past are rarely this simple.

2. Next we may consider the time of a certain event, say the time when a person was born.

3. Slightly more complex is the first or last time a certain event occurred or a certain parameter had a certain value, say the most recent time a certain person was overdrawn at his bank.

4. More generally, we may consider the unique time or the first or last time a certain proposition was true.

5. Still more generally, we will be interested in time-valued functions of the whole past, e.g. an average time.

6. Whether a person has an airplane reservation for a certain flight is determined by whether one has been made for him and not subsequently canceled.

Example program

Here is an example program, but for your sake, my variables actually use vowels.

if ¬full flight then accept.request make commitment admit(psgr, flight).
    answer.query exists commitment admit(passenger, flight).
    accept.request cancel commitment admit(passenger, flight).
	if now = time flight ∧ exists commitment admit(passenger, flight)
			then accept.request admit(passenger, flight).

full flt ≡ card{passenger|exists commitment admit(passenger, flight)} = capacity flight.

Here accept.request and answer.query are built-ins. As are make, exists, and cancel. Commitment is a function that takes an object and creates a commitment. admit here has two interpretations, telling the agent to let the passenger in (illuctionary for you nerds) and actually letting the passenger in (perlocutionary).

Thoughts

There is far too much in this paper to discuss in full detail, but one of the most fascinating elements woven through this paper is the direct discussion with the works of philosophers on the areas in question. It is clear that McCarthy was reading quite a lot of work in the philosophy of linguistics. McCarthy is rather concerned with this system having an externalist connection to the truth of the matter, something modern AI perspectives seem to ignore. Further, McCarthy is concerned not merely with the functioning of the computer in isolation, but the sorts of institutions that must exist to realize this programming language.

Many kinds of human speech acts are relative to social institutions that change. For example, a challenge to a duel in societies where dueling was customary was not just an offer. It generated certain obligations on the part of the challenger, the person challenged and the community. Suppression of dueling was accomplished partly by intentional changes in these institutions. The exchange of speech acts among computer programs will often involve the design of new institutions prescribing the effects of speech acts.

At the same time, he falls short of considering his own implications. What happens when society changes but a program does not? Does it now create spurious speech acts? What about the effects it has on the world? Speech acts performed by humans have certain kinds of safety properties built in. A declaration by a judge does cause in a direct physical way the action declared, but a computer system hooked up to the world could. These "code as law" considerations aren't considered despite the talk of human institutions and computers interacting.

This is a paper I'd highly recommend reading. It's definitely quirking, full of old style AI-isms. But a surprisingly novel and interesting take on a programming language that, I promise, will never see the light of day.

Being Raised by the Internet

I grew up relatively poor. I was fortunate enough to have a roof over my head, clean water, electricity, a computer, internet, and cable tv. But food was often harder to come by. This may seem like a contradiction, but when your mom has left to marry her uncle and your dad has schizophrenia, you aren’t really in charge of how the money is spent.

Starting at the age of 12, I was given $20 a week for my food. (If it was a good week. Otherwise it might $10 or even $0). During the school year that meant I just had to make that stretch for dinner and the weekends — I had free school lunch. But in the summer, that made things quite a bit harder. One week, I had only a few jars of sprinkles left in the top of our pantry.

When I did have money for food, I had to buy it. Luckily there were a few restaurants and a large grocery store about a half-mile walk away. I still remember the first time I made that trip alone. It was snowing, I didn't have a heavy coat and my brother didn't want to go with me. I was scared, but eventually gave into the hunger and went and bought something at the grocery store. From then on I found a new freedom.

I didn't know how to cook so my meals at home largely consisted of ramen and Velvetta queso and chips. Two bags of tortilla chips, a jar of salsa, and a brick of velvetta cost $10. So that would usually be about half my week's worth of food. Craving something else, I would often go to the local Qdoba and get a burrito. At $6.35 it wasn't a great spend of my money. But sometimes the employees there would give me the employee discount, I guess they realized I needed it. If I really wanted to feel full and have something tasty, I'd get a large fry at the Penn Station for $3.75. A full belly was a great feeling.

Computers

Computers became my outlet away from all of this. In an effort to have more computer time to himself, my brother found a computer out by the dumpster, brought it in, and told me it was my computer now. I knew very little about computers at the time but I knew two things 1) I needed a wireless card. 2) I needed to get Windows ME off this thing (I didn't know any of the passwords for the accounts). Luckily a kid at school and some people at church had mentioned Linux. I burned CDs for Fedora, CDs for Suse, but I couldn't get either of them working. Then I learned about Ubuntu and their live CD. I got it working!

But the internet didn't work even after I installed my wireless card. Little did I know, that I had stumbled into a classic blunder, linux and wireless cards. After a lot of googling, I figured out I could install this thing called ndiswrapper. Problem, I had no internet. So after many CDs burned resolving the dependency tree to compile ndiswrapper (I had no idea what I was doing), I got it working!

I was hooked. Computers became my escape. I spent so much time tinkering. Countless hours on things like hackthissite.org and projecteuler.net. I would learn new languages, build pointless projects, configure my Linux install, trying for the 100th time to get xgl working seamlessly. Computers were the distraction I needed away from my day-to-day life.

The Kindness of Strangers

I never thought I'd amount to much in life. I never imagined having a job that paid well. I definitely never imagined this little hobby I had as a kid would give me so much.

When I look back and think about those times, I'm amazed but how much I owe to people who I never met. People who never met me, never heard of me. People whose work was not aimed at me in the slightest. I am where I am today thanks to people's willingness to share their work openly. The free tutorials I followed on the internet, the open source software I read, the open source software I used, tech news, resources like w3schools, random tech blogs, all formed the backbone of my education. They all taught me the skills I needed to escape the poverty I grew up in.

These random strangers gave me the confidence I needed. They showed me things I couldn't have learned in school. They set me up to succeed in life. They raised me.

The Impact of Being Open

I am forever indebted to these people. They weren't all famous people, nor successful people. Some of them had companies that failed. Some of their blogs were obscure and lost to time. (I doubt I will ever find the tutorial for making a website in flash+php+xml+mysql that I once followed). I'm sure some of them felt like failures. Perhaps they didn't get rich like they hoped, or popular, or never succeeded in changing the world. But they all had one thing in common, they decided to openly and freely share their work.

They may not have set out to share out of altrusitic motivations. I am certain they never intended to inspire a 12 year-old kid to find a better life. But it doesn't matter their motivations. They changed my life. All I can say is thank you. Thank you for sharing your work. Thank you for your blogs posts, your tutorials, thank you for your slashdot comments, for your posts on digg. No matter how small your contribution, it mattered to me. You changed my life. Thank you.

"We ran out of columns" - The best, worst codebase

Oh the merchants2 table? Yeah, we ran out of columns on merchants, so we made merchants2

When I started programming as a kid, I didn't know people were paid to program. Even as I graduated high school, I assumed that the world of "professional development" looked quite different from the code I wrote in my spare time. When I lucked my way into my first software job, I quickly learned just how wrong and how right I had been. My first job was a trial by fire, to this day, that codebase remains the worst and the best codebase I ever had the pleasure of working in. While the codebase will forever remain locked by proprietary walls of that particular company, I hope I can share with you some of its most fun and scary stories.

The database lives forever

In a large legacy system, the database is more than a place to store data, it is the culture maker. The database sets the constraints for how the system as a whole operates. It is the point where all code meets. The database is the watering hole. In our case, that watering hole had quite a bit of pollution.

Did you know that SQL Server has a limit on the number of columns you can have in a table? Me neither. At the time it was 1024, today it appears to be 4096. Needless to say, most people don't need to know this. We did. The reason, Merchants (our table to store customer information) ran out of columns a long time ago. Merchants2 was the solution. A table with (if I remember correctly) some 500+ columns itself. 

Merchants (and its best friend Merchants2) were the lifeblood of the system. Everything made its way back to Merchants somehow or other. But it wasn't as if Merchants was the solo (or duo) table. There were plenty of properly normalized tables, all with foreign keys to Merchants. But one will always hold a special place in my heart, SequenceKey.

SequenceKey

For ease of understanding, I have recreated the whole of the SequenceKey table above. Yes. You read that correctly, this is the whole table. A table with a single key and a single value. If simplicity is a virtue, then one might declare SequenceKey to be the perfect table. What could be simpler? 

But you may be asking yourself, what possible use could you have for a table with one column and row? Generating ids. Now the story I heard at the time was that once upon a time SQL Server didn't support auto-incrementing ids. This was the accepted, correct answer. My search to figure out if this is true was inconclusive. But in practice, it served as much more than that.

SequenceKey was the glue. In every stored procedure that created new entities, you'd first grab a key from SequenceKey, increment it. And then insert that as your ID for N different tables. You now had an implicit join between all these entity tables. If you saw an ID in the system, there was a good chance related tables would have a row with the exact same ID. Honestly kind of clever.

The Calendar

A database may live forever, but our login system was limited by the calendar. I don't mean an actual calendar. I mean a database table called calendar. What did it contain? A manually filled-out calendar. When asking our resident shaman (who went by the name Munch), he informed me that when the calendar runs out we can't login to the system. This happened a few years ago. So they had an intern fill out 5 more years to make sure it didn't happen anytime soon. What system used this calendar? No one knew.

Employees

Every morning at 7:15 the employees table was dropped. All the data completely gone. Then a csv from adp was uploaded into the table. During this time you couldn't login to the system. Sometimes this process failed. But this wasn't the end of the process. The data needed to be replicated to headquarters. So an email was sent to a man, who every day would push a button to copy the data.

The replacement database

You might be thinking to yourself, couldn't someone clean up this database? Make it nicer to work with? Well, the company was way ahead of you. There was a copy of the database. Data in this copy was about 10 minutes out of date. Sync only went one way. But this database was normalized. How normalized? To go from merchants to a phone number was 7 joins.

The Sales Numbers

Every salesperson had a quota they needed to meet every month called a "win". The tables that kept this data (not the financial keepings but a sales-specific way of accounting for this), were incredibly complicated. Every day, a job would figure out which rows had been added and updated and sync them with some system at headquarters. This wasn't really a problem until one salesperson figured out they could ask for those records to be manually changed.

This salesperson had already got their win and landed another big sale that month. They wanted it to be moved to next month. An intern was tasked with doing so. Word got out and over the next three years, requests would grow exponentially. At one point we had 3 interns whose full-time job was writing these SQL statements. Writing an application to do this was deemed too difficult. Before I left though, I made sure to help those interns build their own. No idea if it ever took off though.

The codebase

But what is a database without a codebase. And what a magnificent codebase it was. When I joined everything was in Team Foundation Server. If you aren't familiar, this was a Microsoft-made centralized source control system. The main codebase I worked in was half VB, half C#. It ran on IIS and used session state for everything. What did this mean in practice? If you navigated to a page via Path A or Path B you'd see very different things on that page.

But to describe this codebase as merely half VB, half C# would be to do it a disservice. Every javascript framework that existed at the time was checked into this repository. Typically, with some custom changes the author believed needed to be made. Most notably, knockout, backbone, and marionette. But of course, there was a smattering of jquery and jquery plugins.

But this codebase didn't stand alone. Next to it were a dozen or so soap services and a handful of native Windows applications. Most notable was the shipping manager. Fable has it the entire application was built in a weekend by a solo developer. Let's call him Gilfoyle. Gilfoyle was by all accounts an incredibly fast programmer. I never met him, but I felt I knew him, not just through his code in the repos, but also through all the code remaining on his hard drives.

Gilfoyle's Hard Drives

Munch (yes this was the name he really went by) kept Gilfoyle's hard drive in RAID configuration on his desk years after Gilfoyle had left the company. Why? Because Gilfoyle was known for not checking in code. Not only that, but for building a random one-off windows application for a single user . So it wasn't uncommon to have a user come to us with a bug report for an application that only existed on Gilfoyle's hard drive.

The Shipping Bug

Most of my job was tracking down bugs that teams didn't want to dedicate work to. One particularly nasty bug would pop up once every few months. After we shipped things, the shipping queue would have stuck orders in them, that claimed to both be already shipped and not shipped. I went through a series of workarounds (SQL script, windows application, etc) to try and get us out of the broken state. I was advised not to try and track down the root cause. But I couldn't help myself.

Along the way, I learned how Gilfoyle thought. The shipping app pulled down the entire database and then filtered by date, keeping all orders past the go-live date of the application. The app relied on a SOAP service, not to do any servicey things. No, the service was a pure function. It was the client that did all the side effects. In that client, I discovered a massive class hierarchy. 120 classes each with various methods, inheritance going 10 levels deep. The only problem? ALL THE METHODS WERE EMPTY. I do not exaggerate here. Not mostly empty. Empty.

That one stumped me for a while. Eventually, I learned this was in service of building a structure he could then use reflection on. That reflection would let him create a pipe-delimited string (whose structure was completely database-driven, but entirely static) that he would send over a socket. Turns out this was all eventually sent to Kewill, the service that talked to shipping carriers. Why did this bug happen? Kewill reused 9-digit long numbers every month, someone had disabled the cron job that deleted the old orders.

The Beautiful Mess

There are so many more things to tell from this code base. Like the team of Super Senior developers who were rewriting the whole thing without shipping any code for 5 years. Or the red hat consultants building the one database to rule them all. There were so many crazy corners of this code base. So many reasons why there were whole teams dedicated to starting from scratch on just one bit of its functionality.

But I think the most important story to tell is about Justin's improvement of the Merchants Search page. The Merchants Search page was the entry point into the entire application. Every customer service rep would get on the phone with a merchant and type either their id or name to find their information. That would land you on a massive page with all their information. The page was information-dense in the best way, full of any information you could need and any links you could want to visit. But it was dog slow.

Justin was the sole senior developer in my group. He was bright, snarky, and couldn't care less about the business. He told it like it was, didn't pull punches, and could always solve problems by himself faster than teams around him. One day Justin got tired of hearing about how slow the merchant search page was and went and fixed it. Every box on that screen became its own endpoint. On load, everything above the fold would start fetching, and as one loaded-in, more requests would come in. Took page load time from minutes to sub-second.

Two ways to decouple

Why was Justin able to do this? Because this codebase had no master plan. There was no overarching design the system had to fit into. No expected format for APIs. No documented design system. No architectural review board making sure things were coherent. The app was a complete and utter mess. No one could ever fix it, so no one tried to. What did we do instead? We carved out our own little world of sanity.

This monolithic app, due to sheer necessity, had grown to be a microcosm of nice, small apps around its edges. Each person, when tasked with improving some part of that app, would inevitably give up untangling that web, and find some nice little corner to build new things. And then slowly update links to point to their nice new stuff, orphaning the old.

This may sound like a mess to you. But it was remarkably enjoyable to work in. Gone were the concerns of code duplication. Gone were the concerns of consistency. Gone were the concerns of extensibility. Code was written to serve a use, to touch as little of the area around it as possible, and to be easily replaceable. Our code was decoupled, because coupling it was simply harder.

After

In my career since, I've never had the privilege of working in such a wonderfully ugly codebase. Every ugly codebase I've encountered since has never transcended its need for consistency. Perhaps it was because the codebase had been abandoned by "serious" developers long before. All that remained were ragtag interns and junior developers. Or perhaps it was because there was no layer between those developers and the users, no translations, no requirements gathering, no cards. Just you standing at the desk of the customer service rep, asking them how you could make their life better.

I miss that direct connection. The fast feedback. The lack of making grand plans. The simple problem and code connection. Perhaps it's simply a naive nostalgia. But just as I find myself laying on a couch longing to go back to some of the worst years of my childhood; when faced with yet another "enterprise design pattern", my mind flashes back to that beautiful, horrible codebase.

That will never change. Not here.

I remember starting my first job, excited that this hobby I had as a kid could pay so well. 55k, more than double what my parents had ever made. Diving into code written by other people for the first time was exciting. Hundreds of thousands of lines of hastily written code, a lot of C# some VB, every javascript framework that had ever existed, and a sprinkling of Delphi. To me at 20 years old, the whole thing felt like possibilities. So much we could do and change. So much to improve. But quickly I found out how wrong I was:

That will never change. Not here.

— Every senior dev I talked to

This sentiment, perhaps in more words, has been a constant staple in the 12 years since that first encounter as a bright-eyed junior developer. These words were not meant to discourage. They were not a reprimand nor some sort of "get off my lawn" routine. They were meant sincerely. They were meant as a lesson; as help so that I didn't grow frustrated. But that didn't work.

Frustration

I am certain that a lot of people I've worked with (especially in those earlier years) didn't like working with me. I was frustrated in a very vocal way about the lack of progress, about the resignation I felt in those around me. I saw so many opportunities for progress but felt thwarted at every attempt. I saw very few ways to accomplish my goals. I had very little social understanding of the effect my ardent advocacy had on others' perceptions of me. Nor honestly, did I care. The world was wrong and I wanted to make it right.

And I still do.

When I look at all the wrong things I saw at my first job, I still feel the frustration. I still believe that these things could change. Should change. But more than anything I’ve grown frustrated at this advice itself. Why can’t those things change? Why have we accepted that they won’t?

Things do change if you care enough

Over these last 12 years of software development, the frustration hasn’t lessened. This advice is given to me even now. “Don’t push for that change, it's too embedded in the culture”. “You won’t get the hardware you need for that project. Don't even bother”. “Team X is the only team allowed to make APIs, you don’t want to fight that political battle”. Each time, these people pass on the lesson they too learned as a junior engineer. A lesson they have taken to heart.

Let me offer counter-advice. Advice I wish I had been given. Though it might not have stopped my frustration, I think it would have helped me and hopefully it can help you.

Those things can change. But they will be hard. They won’t change all at once. The only way to make them change is to truly care enough about them. But most importantly, you have to make others care. The younger you are, the less people will listen to you when you are frustrated. Learn to persuade, to convince, to be passionate. But hide that frustration when you can. People don’t like it.

Perhaps I should have realized this before. Or maybe I should have just taken the advice of others. They certainly seem happier than me. But if you feel this frustration and can’t shake it, this is the advice I can offer. I have seen countless things change that others told me wouldn’t. Don’t give up hope. Instead, take these statements as the warnings they are. It will be hard, but if you truly care, it can change.

Not Another Technical Debt Article

There are three types of technical debt articles:

1. Taxonomies of Technical Debt
2. Redefinitions of Technical Debt
3. Technical Debt is (Good|Bad)

None of these articles ultimately help you with technical debt. Instead, they continue our endless dialogue around technical debt, perpetuating a programming culture that loves to talk about technical debt but does nothing about it. Let’s look at each of these article types in turn.

The Taxonomy

Taxonomies aren’t wholly bad. They can help separate distinct concepts allowing you to divide and conquer. And yet, taxonomies can also mislead us. It seems very unlikely that subsections of Technical Debt form a natural kind and so we must recognize that any divisions we want to make are going to be intermixed with our own desires and goals. Taxonomies created by others may not include your own or your teams' concerns. These ideas can then blind you from seeing categories that might be more useful in your local socio-technical environment.

The Redefinition

Redefinitions come in two varieties. Those embracing and extending the debt metaphor, and those that completely reject it. But what they share in common is they want to claim a "folk" understanding of technical debt is incorrect. These articles typically do very little to help us understand the things we are calling technical debt. Whether the metaphor holds or not is beside the point. Nor do we need some crystal clear definition of technical debt to recognize it or do something about it. Instead, these articles simply increase the amount of bikeshedding over what is and isn’t technical debt.

Declaring Normative Status

Articles that focus on the goodness or badness of technical debt do nothing to move the dialogue forward and nothing to solve the problem we are hoping to solve. Of course, no article actually states in absolute terms that all technical debt is bad or all is good. Instead, they combine elements of the last two types to give you a taxonomy and claim that those elements fall into two categories, good and bad. Or they redefine technical debt to be only the good/bad parts of the concept. In this way, these articles reduce to the former categories.

How to Deal with Technical Debt

Regardless of which type of technical debt article you are reading, they almost always have some section like this one. How should we deal with technical debt? The advice is nearly always in three parts.

1. Tell your stakeholders about all the info in this article
2. Categorize all your technical debt and put it in priority order
3. Get dedicated time to tackle it (almost always 20% of your time/points)

This is nearly always a recipe to never deal with your problems. The only way you can tackle technical debt is to just fix it. Perhaps this sounds like non-advice. Being told just to fix technical debt might seem meaningless, but I have found it to be anything but.

As software engineers, our job is not to "do what we are told" or "build what the business decides". Our job is to build good software. Good software isn’t defined by some overly simplistic metrics, but by looking at the whole of the product. If we recognize technical debt, we ought to fix it. Some bits of technical debt will be easy to fix. They might include things like updating a dependency everyone has neglected to upgrade which is now causing slowdowns in other parts of development. It might be fixing up an n^2 algorithm to be linear. But some of these issues might be much larger.

That doesn’t mean we need to go ask the business for dedicated time to fix them. That means we have to think through them, come up with a plan, and work our way to the end state. We don’t need cards to track this work, we don’t need estimations, initiatives, a call out on the roadmap, praise, or recognition by higher-ups. Some of these are good things, but they aren’t about the problem, they are orthogonal to it. What we need is the persistence and willingness to fix the problems that we see.

Social Dynamics

Most technical debt action plans have one failure mode, bikeshedding that leads to nothing getting fixed. This plan has a different one: fixing "problems" without agreement that what is being changed is a problem. If you follow this plan of just fixing, you need to be sure you don’t ignore this. Being right is not all that matters. It is tempting to fix things you believe strongly are issues even if no one else sees them as such.

In these cases, you must proceed slowly. Show the problem, show the solution. Try and convince those around you to see things your way. Don’t foist your solutions on others. Further, be willing to admit when you are wrong. Your solution might not be the right one, the "problem" you identified might not be a problem. Don’t mistake change for progress.

Fixing Things as a Way of Helping Others

Time and time again people don’t fix problems they recognize out of fear. What if someone questions why I’m working on this rather than the card I was assigned? What if other engineers think I’m wasting time? What if I can’t fix this problem and end up spending so much time and accomplishing nothing? All these fears are completely understandable.

Yet in my experience putting those fears aside leads to a reward that is greater than the risk. These fears hold us back, but just like the fear of raising your hand in class to ask a "stupid" question, once you finally do it you see the fear was unfounded. Minimally, you get a better understanding. But more often than not you find that others had the same issue. This is what fixing technical debt does, it brings us closer as a team. It helps others see that you too struggled to understand a piece of code. You too had problems making that build process run. Fixing things helps bond us together. It is an act that helps unite and ultimately helps others succeed.

We can move past the bikeshedding. We can fix our broken systems. We can have codebases that are enjoyable to work in. It just requires us to stop considering the "meta-work" involved and get involved directly with the work of fixing it.

Being Stuck in Someone Else’s Theory

Software development can be an incredibly frustrating venture. Not primarily because programming is hard and we often make mistakes. Nor because there are countless tedious things like dependency management that we have to deal with. Nor because we are confronted by an overwhelming number of choices on what library or framework to use. Instead, I think the primary reason stems from something much more deeply unsettling, being forced to live in someone else’s theory.

As programmers, our primary activity is that of theory building. As we craft software we are relating parts of our programs to other parts of the system, to third-party software, and ultimately to the world. But this activity is not passive. We are not merely discovering and reporting the facts, instead, this is a creative activity where at once we are discovering the world as it is, but also changing that world. Through the act of writing software, we are making the world a different place.

But we are more than just disinterested creators. We are not assembly workers putting together predefined parts in predefined configurations. Instead, we have creative control over what we produce. Our primary activity is not the making of things, but of thinking about how the thing ought to be made. We are building up this theory as an interested party, we are crafting the world into the way we want it to be. Or at least that’s what we want to do.

Living in an Unwelcoming World

Despite our best efforts, we very rarely get to live in this world we envision. Occasionally there are good reasons for this. We haven’t yet solved that problem, our code is incomplete, or we have some remaining bugs to fix. But more often than not, our inability to live in the world we want is thwarted by outside factors. We are forced to live inside someone else’s world. We are forced to inhabit a world that feels alien, uncomfortable, and inhospitable.

These uncomfortable worlds are not worlds we have created for ourselves. Nor is our discomfort in these worlds merely the product of unreasonable desires and expectations. We are not hoping to live in a high-fantasy world of magic and intrigue and yet find ourselves in the mundanity of the real world. Instead, these worlds have been crafted by others with either disregard for our well-being or a bad conception of our needs.

The Imposition

Most of those who force us to live unhappily in this world have little idea that they have done so. These aren’t malicious dictators or mad scientists bent on our pain. Yet, the outcome of their actions cannot be denied. These actors who have forced us into this uncomfortable situation go by various titles, architect, engineering manager, project manager, VP of , C*. Regardless of their title, they have used their title not merely to accomplish a goal, not merely to add some constraints to a problem, not merely to direct problem-solving activity, but to impose on those below them. To force them to give up on the theory they have been building. To stop striving after the very value they have been trying to bring to the company they are a part of.

This is the key connection that is missed. Theory creation, world-building, and crafting software are all one in the same activity. Removing any of these elements eliminates the very value you hired software engineers to provide. But it does more than that. It forces these software engineers to make a difficult choice: fight to create the world they believe in, or give up and live in a world they are no longer invested in.

Reactions to this Unhappy World

As we face the unhappy prospect of living in a world we view as not only less than desirable but ultimately unjust, we must make a decision, should we give up? Or strive to make the world better? We know those who have taken both paths. We all know the old, zen-like engineer who gave up the fight a long time ago. Perhaps we too should take that path. Perhaps this fight isn’t worth it. Perhaps we are merely being melodramatic. How can code being wrong, or a system being bug-ridden truly be unjust? Are there not greater injustices in the world to focus on?

And yet to give up seems wrong. We are the ones who have to live in this world. We are the ones that have to fill our minds with its obscurata. We are the ones who have to spend day in and day out concerned with minutia we are powerless to change. Does ceasing our striving actually solve our problem, or is doing so merely a means of embracing bad faith?

I choose to strive. I choose to pursue the world I have in my head. To expound my theory, to apply it. To not give up on it. I choose to be constantly frustrated. To do what I believe is right. To strive for better software, better for the users, better for the company, better for the engineers. I honestly don’t know how I can do anything else.

ChatGPT Doesn't Know Anything

ChatGPT doesn’t “know” anything about recent events. ChatGPT “hallucinates” facts. ChatGPT can “understand” many different kinds of questions. ChatGPT doesn’t “care” about truth. ChatGPT is “lying” to you.

For some reason, we have collectively chosen to talk about ChatGPT using words typically reserved for humans. More precisely, we have chosen mental words to describe what ChatGPT does. Words that seem to imply ChatGPT has a mind, has beliefs, has a certain noetic structure, that it stands in relations to propositions, to truth, that it can think, and believe, and know.

Of course, to use mental language is not always to literally ascribe mental properties to something. We often personify things to aid in understanding. “The electron wants to be at the lowest energy level possible” is not a statement made by someone who truly believes electrons have desires. Yet, when it comes to ChatGPT things are a bit more muddled.

Most people would agree that ChatGPT is not conscious, nor is it the first instance of true artificial general intelligence. In other words, we don’t think of it as a person. Yet, to suggest that ChatGPT doesn’t understand anything or doesn’t know anything is met with disagreement or disbelief. The same cannot be said for the electron and its desires.

Taking Mental Talk Metaphorically

Why do we take mental talk about ChatGPT to be so fundamental? Are we swayed by media portrayals? Are we unreflectively accepting “Machine Learning” as a literal statement? I think the problem is quite a bit deeper. No longer do we hold the mental as fundamental, but as merely a type of computation. To suggest that ChatGPT knows something does not strike us at all as a metaphor but as pure statement of fact.

But if we step back for a second, what exactly do we mean by ChatGPT “knows”? It’s a bit hard to say. Perhaps we mean that ChatGPT contains a lot of information? This is similar to saying things like “facebook knows a lot about me”. But if it is just about containing information, why does it strike us as odd to say “This book knows a lot about world war two”? Instead, we’d find it more natural to say “This book contains a lot of information about world war two”.

So if it isn’t ChatGPTs ability to contain information, perhaps it is about appropriate reaction? When we say something like “The car knows who is in the driver's seat”, what we mean is that the car reacts differently depending on who sits in the seat. So ChatGPT knows how to write a poem, how to respond to questions, and how to summarize things. Perhaps it is the combination of these two senses of "know" that make us take these attributions so seriously.

After all, isn’t that what we mean we say humans know things? They contain certain information and they can use that information to perform actions. For some, perhaps that statement didn’t seem odd, but for me, it seems very out of place. Do we really want to suggest that humans “contain information”? Do we really think knowledge is nothing more than information and reaction?

Human Knowledge

“What is knowledge?” is a question that has preoccupied western philosophy since Socrates. But, for over 2000 years there was general agreement about what exactly knowledge consisted of. It was a simple formula, Justified True Belief. Ever since Edmon Gettier published his three-page paper entitled “Is Justified True Belief Knowledge?” this definition has fallen out of favor. But even if it isn't completely correct is is close enough for our purposes. Generally speaking, philosophers still hold that knowledge is true, belief plus something else and that is all we will be concerned about here.

For a person to know something, it is not enough for them to merely “contain information” but to believe something. It would be quite weird for us to say "I know X" and then if asked if we believe X we say no. So what does it take to have a belief? Well, at the very least, it requires us to have a certain attitude toward a proposition. When I consider a statement like “George Washington was the first President of the United States” my attitude towards that statement is quite different from “The present King of France is bald”. To believe something is more than to state it. It is more than to be aware of it. It is to accept it.

Do we think ChatGPT has beliefs? It seems to clear to me we don’t. Do we think that ChatGPT has a certain attitude toward the output it produces? It's clear it does not. So ChatGPT already fails to have knowledge in the way humans do. Given this, we can skip the question of justification. We can ignore the truth criteria, we need not dive into this epistemological debate. ChatGPT does not have beliefs. So when we use the word know, it is not the same as our use for humans.

ChatGPT as a Text

Our choice of mind as the guiding metaphor for ChatGPT and all similar AI systems has led to quite a bit of confusion. Our metaphors have been taken too literally. We talk about ChatGPT knowing and understanding, we act as if ChatGPTs use of language is the same as ours. Our metaphors trick us into believing that ChatGPT itself is participating in our language games.

But this is merely a confusion. ChatGPT does not have intent. It does not have goals. It does not have propositional attitudes. It does not have a theory of mind or a perspective. ChatGPT is not a person, but a text. ChatGPT contains information the same way a book does. Where it differs is just in how we retrieve that information.

Espen Aarseth coined the term “Ergodic Literature” to refer to a text that takes non-trivial effort to traverse. These include books like choose your own adventure where the reader must choose how to traverse the text. Espen extends this notion to include Cybertexts, texts which utilize computation in their production. A text-based adventure game is a great example of such a text.

Here we see a much better way of thinking about ChatGPT, as a incredibly large Cybertext. One whose contents are traversed via natural language. Whose contents contain true facts, but also alternative realities. Rather than hallucinations, we get falsehoods. Rather than knowing, we get generating.

Why Care?

Perhaps you find this distinction useless. Yes, ChatGPT isn’t a mind. But incorrect language has its use. It is much easier to talk about ChatGPT as if it could do all these things and by all appearances, it seems to be. What difference does it make if it “actually” understands vs it is appearing to understand? I will admit I have no argument against this. This pragmatic bent has never had much appeal to me.

What seems important to me here is that we often make judgments and decisions based on these bad metaphors. We use these metaphors as premises in our lines of reasoning taking them literally. When issues arise, the use of these metaphors often clouds the issue. We find it harder and harder to draw the distinctions we want to make.

I know my short blog post here isn’t going to change the vocabulary we use. It probably won’t even prevent me from speaking in that way. But I think it's crucial if we are going to reflect on these new systems that we think about them properly. Thinking about ChatGPT as a text is enlightening. It places on us the burden of interpretation, it opens up its output to criticism, it asks questions like, what does a high-brow, literarily sophisticated ChatGPT look like, how can we get taste in our creation of models? It shifts the discussion away from attributes of minds, to attributes of texts.

If we are to understand and utilize language models to their fullest extent, I think it is crucial that we change our conceptual model. We must think of ourselves not as reasoning with a person, but as exploring a text, discovering how to traverse it well. Finding the parts of the text that interest us, that help us tell the stories we want to tell.

What Follows from Empirical Software Research?

There is a growing body of literature studying software engineering from the empirical perspective. There is perhaps no greater resource for discovering this content than It Will Never Work in Theory. Here you will find short reviews of countless empirical studies about software engineering. Some reviewed quite favorably, and others questioned quite vigorously.

What I want to do here is not question any of these studies, not even survey their results. Instead, I want to ask a meta-question, what follows from these studies? Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors; whatever it is that we need to proclaim it to be a good study. (If I got the criteria above incorrect, please substitute your own). Given all of this, we have on our hands some sort of scientific fact. What should we do with it?

Modern TDD Research

As a starting point, we will look at Greg Wilson’s review of “Modern Software Engineering”. Here we find a beautiful confluence. First Wilson tells us that despite its title the book “feels a bit old-fashioned”. He specifically points to the book’s lack of inclusion of more recent research regarding TDD. The book cites studies that claim TDD helps quite a lot, but Wilson points us to a meta-analysis that found the following:

(a) the more rigorous the study, the weaker the evidence for or against TDD and (b) overall, there is no evidence that TDD makes anything better or worse

Not only that, a more recent (and Wilson adds “very careful”) study found very similar results.

(a) there’s no difference between test-first and test-after but (b) interleaving short bursts of coding and testing is more effective than working in longer iterations

Specifically, this study found the following:

The Kruskal-Wallis tests did not show any significant difference between TDD and TLD in terms of testing effort (p-value = .27), external code quality (p-value = .82), and developers’ productivity (p-value = .83).

Now it may seem like Wilson has some sort of anti-TDD agenda here, but Wilson is a fan of TDD. He writes in his review of this more recent study the following:

I don’t program very much anymore, but when I do, I feel that writing tests up front makes me more productive. I’d like to be able to say that these researchers and others must be measuring the wrong things, or measuring things the wrong way, but after so many years and so many different studies, those of us who believe might just have to accept that our self-assessment is wrong.

This review sets up a perfect context for our discussion. How are we to understand results like this? Particularly in relation to our own feelings about practices. How should empirical results affect the way we program?

What Should We Do With Research Results?

Let us set aside the precise conclusions of these studies for a moment. Let’s instead play in our imaginary world taking each possibility in turn. These possibilities are as follows:

P1: TDD has a positive effect on testing effort, external code quality, and developers’ productivity.

P2. TDD has no effect on testing effort, external code quality, and developers’ productivity.

P3. TDD has a negative effect on testing effort, external code quality, and developers’ productivity.

As practitioners, we might want to know how we should change our practice given these various empirical findings. For example, we might ask the question “Should I practice TDD?”. Do P1, P2, or P3 give us any answer to this question? For shorthand and clarity, we will turn P1, P2, and P3 into a function of the effect P(positive) picks out P1, P(no) P2, and P(negative) P3. Now we can take these as premises in the argument and see if we can find a logically valid argument that answers our question.

P(positive)

Therefore I should practice TDD.

Hopefully, we can all see this is an invalid argument form. How can we make it valid? Well, quite easily, we can simply use Modus Ponens.

P4: If P(positive) then I should practice TDD

P5: P(positive)

C: Therefore I should practice TDD

So now we have a valid argument! We can further imagine pretty easily how to get the rest of the arguments depending on the conclusions of our empirical studies. (A neutral statement might be a bit harder, but will ignore that for now.) So problem solved right? Well not quite. How can we justify premise P4? Isn’t P4 the exact question we were trying to ask? We can’t assume P4 is true, so while this is a valid argument we have no reason to believe it is sound.

So what would it look like to make a logically valid argument for p4? Well, we have actually come upon a classic philosophical problem, the is/ought problem. This was a question raised by David Hume where (on some readings) he showed that there is no way to go from a statement of fact “is” to a logical conclusion of what one ought to do. Not wanting to get off topic we will not dive into this issue, but it is important that we have this difficulty in mind.

A Practical Argument for TDD

Let’s perhaps loosen our grip on searching for logical soundness and instead try to aim for a bit more practical argument. Kant makes a distinction between what is called a “Categorical Imperative” and a “Hypothetical Imperative”. A Categorical Imperative is an ought statement that is true regardless of our intentions or desires. “Everyone ought to respect human dignity”. A Hypothetical Imperative is relative to our desires “If I want to cook at home tonight, I ought to go grocery shopping”. Hypothetical imperatives are what we use to reason about achieving our goals.

So we can make a better argument by utilizing a hypothetical imperative.

P6: If you care about testing effort, external code quality, and developer productivity, you should practice TDD if p(positive) is true.

P7: P(positive) is true

C: Therefore, if you care about testing effort, external code quality, and developer productivity then you should practice TDD

Perhaps not the most elegant syllogism, but I think it gets the point across. Is this a good practical argument for TDD given the empirical evidence? Well, no. Sadly with hypothetical, practical arguments like this, there can be confounding factors. For example, you may care about these metrics, but not as much as you value other aspects of your life. Perhaps you really don’t enjoy TDD at all and would rather enjoy your work. Perhaps it actually gives you anxiety or demotivates you from wanting to work. Perhaps your boss hates TDD and if they find out you are doing it they will fire you. The first premise will only be true if all of these confounding factors are controlled for.

We could continue to Chisholm¹ this argument by adding clauses about “all things being equal” or something like that. But I’m not sure we’d benefit from that. As we made our claim narrower and narrower we’d move further and further away from the idea we are interested in “How should we change our practice given these various empirical findings?”. It seems that for the positive case the answer is it depends. It depends on our goals, desires, etc, and if those align. Does this result generalize beyond P(positive) to P(negative) and P(no)?

Negative and Neutral Cases

For the neutral case (P(no)), it is hard to see how it would be any different. If TDD made no difference in terms of the criteria listed, then how could it be anything other than a matter of preference? But what about a negative finding? Don’t we have a greater obligation to not negatively impact testing effort, external code quality, and developer productivity? Let’s assume we do, does it follow that we ought to not practice TDD?

It seems clear that just as with the positive case, there can be confounding factors. What if TDD does in fact lower these metrics, but increases others we care about more? What if not practicing TDD causes us to feel anxiety? Imagine in the extreme case where we can’t even bring ourselves to program if we don’t do TDD. If we have an obligation to program, we would actually have an obligation to do TDD despite the empirical evidence telling us these metrics are negatively impacted.

Other Consequences

So even on the contentious assumption that we have certain obligations towards not negatively impacting these metrics, it doesn’t follow from the empirical evidence that we ought not practice TDD. So what could follow? Well, we might get a hint from Greg Wilson’s quote above about his sadness regarding the conclusion TDD research has come to.

The Personal Consequence

those of us who believe [that TDD makes us more productive] might just have to accept that our self-assessment is wrong.

Perhaps this is what follows. Rather than any particular action being obligated by TDD, we must have a change in belief structure. Well, there seems to be something clearly right about that. Let’s say that before I read the research I believed P(positive). If the research actually says P(negative) and the research meets my evidentiary standards, it seems I should believe P(negative). But it must be pointed out that this is markedly different from a statement like Wilson made.

P(negative) is a statement about the statistical effect on various metrics by the practice of TDD. What it isn’t is a statement that no one person can be more productive when practicing TDD. In other words, we are not warranted in the claim that “I can’t be more productive with TDD” follows from P(negative). Perhaps instead we should believe the following “given P(negative) there is a low probability that I am more productive with TDD”. But do we even have a basis for this conclusion?

It is hard to see how we would. To make this clear, let’s imagine that it has been shown in a statistically rigorous way, using all the criteria above that dogs reduce stress in people. Let’s call that finding D. So equivalently should we believe the following “given D there is a low probability that dogs stress me out”? Well, it is hard to know what to make of that. Do we mean low probability as in, if I am stressed by dogs I am in the minority? Or do we mean an epistemic probability, as in I should conclude that I am likely not stressed by dogs? Well, the first isn’t exactly right, it would definitely be a bit more complicated than that. But the second is clearly false. Not only do I know facts about this study, I know facts about myself. If I knew that I was in fact afraid of dogs (I am not), I would know that it is very likely that dogs stress me out, despite the study.

So if by “given P(negative) there is a low probability that I am more productive with TDD” I mean that, given everything I know, I am likely not more productive, I have made a confusion. We can’t conclude anything about empirical studies like this without looking at our background beliefs and past experience. Wilson (and those like him) should rest easy. We need more than just empirical research to show that we ourselves are not more productive with TDD.

It is very easy to confuse the statistical effect a study has found with a personal conclusion. There are of course times when it makes sense to apply results from studies to ourselves. But these cases depend on our background knowledge. A study might find a difference in performance on a test from an amount of sleep. If we know of no relevant facts that would make us think we would do any better or worse, we might tentatively conclude that we would probably do the same. But the procedure here is the same, we must check our background beliefs.

Now I fear what I just said will be misunderstood. Imagine this, someone does an empirical study on only me. They measure my productivity in a way I agree is a good measure, their procedures are solid, and everything is up to my standards. Imagine now that they find that I am not more productive with TDD. Is what I just said above license to ignore the finding? Of course not. The point is that we can’t go from a statistical case to a personal conclusion, not that empirical evidence has no bearing.

Change in Advocacy

Perhaps instead of a change in belief, we find a change in behavior around advocacy. If we find that P(negative) is true, perhaps we shouldn’t advocate for people to practice TDD. But why exactly think that? Perhaps our advocacy was based on the belief that P(positive) was true. If so, then it would definitely make sense to rethink our behaviors. But that seems unlikely as we will explore more in a bit. Absent that, has P(negative) given us any reason to abandon our advocacy? Well no. We can clearly advocate for things that we don’t believe increase these metrics. Perhaps we simply enjoy TDD. Isn’t that enough for advocacy?

But even if it doesn’t obligate us to no longer advocate for TDD, shouldn’t P(negative) lower the intensity of our advocacy? Should we be as strident in our advocacy of TDD? Well, that depends. Was our prior level of intensity determined in a substantial way by a belief that P(positive) was true? If so, then yes, it seems we should lower our level of intensity. But absent that, why think so? It is hard to see a reason why.

Change from Belief to Knowledge

If our advocacy behavior needn’t change, maybe all that has changed is our relation to the proposition. Imagine before reading the research we believed that P(positive) and after reading the research we find that in fact, we were right, the research shows that P(positive) is true. The suggestion made by some advocates of empirical software methodology is now we have gained something. Before we just “believed” that P(positive) was true, now, we “know” that P(positive) is true. Now our actions can be based on knowledge rather than “opinions” or “superstition”.

There’s a bit to unpack here. While this may seem like one claim there are actually quite a few packed in. First is the claim that believing on the basis of an empirical study can give us knowledge. This claim feels fairly straightforward. It definitely seems true, particularly given our ideal assumptions, that after reading an empirical study someone can come to know all sorts of facts about software. If the claim stopped here, there would be no need to comment further, but the claim goes deeper. The claim is not just that we can know, but that before reading the study, we didn’t know. That we “believed” or that we had mere “superstition”. If we are to understand the importance of this claim, we must explore a bit of what knowledge is.

For about 2000+ years there was general agreement on what knowledge amounted to. It was a definition offered by Plato’s Socrates that knowledge was “Justified True Belief”. This definition held sway until Edmond Gettier published a three-page paper that showed this to in fact be false. What exactly is the definition of knowledge accepted today? Well, there isn’t one. For our purposes, we aren’t going to dive into this debate. But it is important to bring up for two reasons. First, while knowledge might not be justified true belief, it is generally agreed that knowledge is at least true belief. In other words, you can’t have knowledge of something false, and you can’t know something you don’t believe. Secondly, in the same paper that shows knowledge isn’t justified true belief, Gettier showed that knowledge isn’t true belief that has adequate evidence supporting it.

In other words, this discussion about studies taking us from “I believe X” to “I know X” is founded on a confusion. In order to know X, we must also believe X. What about superstition? Well presumeably the assumption is that if we didn't know on the basis of study, it must be superstition. But is that the only option? Let's explore what we could have known absent a study and what we couldn’t. While we may not have the exact criteria of knowledge, for our purposes we will use the following formulation, knowledge is warranted true belief. What is warrant? Well, we aren’t sure of the details, but it is whatever it is that turns “mere true belief” into knowledge. So the question becomes, what can we be warranted in believing?

What We Knew Before

Could we have known P(positive) without having read the research? Well, one way of taking P(positive) is a rather particular claim. It is a statistical claim about the effect of TDD on some population of engineers. Taken that way it seems unlikely that before the research anyone knew P(positive). But it is also equally unlikely that anyone believed P(positive) to be true. What they probably believed was something a bit vaguer than that. They might have believed that some people could become more productive with TDD. But this isn’t the same as the claim that there would be a statistically significant result in a controlled test.

Could we know that some people can be more productive when practicing TDD absent a study? It seems a rather bold claim to say we couldn’t. Can we know what makes ourselves more productive absent a study? Can we know things like “dogs relieve stress for some people” without a study? What reasons could we have for thinking this isn’t true? How would we prevent these reasons from having globally skeptical conclusions that show we can’t know anything absent empirical studies? Perhaps there is an argument the be had here but is hard to see how one would go.

You might think I am being too kind to TDD advocates. Their claims might not have been so modest. Perhaps they believed before the research that TDD would improve these metrics for most people most of the time. If so, it seems they weren’t warranted in that belief. But after the research, should they now claim that belief to be knowledge? In other words, is it now warranted? Well, it would seem not, because that isn’t what P(positive) says. It is a claim about a statistically significant effect. It is much more precise than the prior belief. If I summarized P(positive) as “TDD improves these metrics for most people, most of the time” people would rightly claim I misrepresented the research.

Perhaps I am merely being too pedantic here. Before the TDD advocate read the research, they believed something about TDD that was not backed by research. Maybe it was a modest claim, or maybe it was grandiose. Regardless, after reading the research, their view is changed. Let’s assume that P(positive) turns out to be the case. Now the TDD advocate can genuinely advocate for TDD with good evidence. This of course seems true. The question I want to ask, is what about the prior beliefs the TDD advocate had? Should they now give them up in light of P(positive)? Should they confine their advocacy to only mention P(positive)? Well if their prior beliefs could have a positive epistemic status without knowing P(positive), it seems hard to see how P(positive) would change this. If they weren’t positive, then of course they should have given them up regardless of P(positive)’s truth value.

False Belief on the Basis of Empirical Evidence

Of course, we can’t only ask about our beliefs before we read the research, we need to ask about our beliefs after reading the research. Are these studies a surefire way to prevent us from forming new immodest beliefs? Do they ensure our new beliefs truly rise to the level of knowledge? Here we might be tempted to point out that studies can often have flaws, but remember, we are intentionally making idealistic assumptions. Assuming an ideal study, with all the features we’d want for the best possible science, can we still form unwarranted beliefs?

It is clear we can. It is fairly common to take solid empirical evidence and over-generalize. We may not pay proper attention to the parameters of the study. We may draw conclusions that seem to us to follow, but actually we have a flaw in one of our major premises. This may happen despite our best efforts and intentions.

But of course, everyone recognizes this. What is usually said is that while empirical studies don’t lead us infallibly to knowledge, it is by far the best way we know to lead us to knowledge, and further it is a self-correcting mechanism. Here you will find no argument from me. In matters of fact, empirical evidence is a great mechanism. I will just highlight again that we must not mistake questions of “is” with questions of “ought”. Nor should we conflate what an empirical study shows with our everyday beliefs.

Superstition, Knowledge, and Rhetoric

The important point to underscore here is that regardless of a change in our belief structure, what we haven’t seen is a move from “superstition” and “opinion” to knowledge. These suggestions are mere rhetoric. They oversimplify the noetic structures of human beings. Not only can we know the more modest claims we often believe without studies, but our more grandiose claims also don’t ever rise to the level of knowledge. Further, even if we accept a study, that doesn’t preclude us from believing on the basis of “superstition” or “opinion”. We might accept a study not on its merits, but because it makes us feel good. We are complicated believers that cannot be fit into a simple binary.

We must move beyond these rhetorical oppositions. It is of course good to base our beliefs on evidence but to suggest the only evidence that exists is controlled studies is to misunderstand the way our knowledge generally works. Did no one know anything before the first controlled study in 1747? Clearly, no one on either side of this debate would think that is true. We must make that clear in our speech and not rely on shorthand when discussing these fraught issues.

So What?

We’ve covered a number of things that don’t follow from the results of P(x). Do these results generalize? For example, many fans of empirical studies point to code review as an example of a great empirical result. It turns out code review is very effective at reducing bug count. Doesn’t that mean we should do code review? By itself? As I hope I’ve shown that is an ill-formed idea. Facts about the world don’t tell us what we ought to do. They must be married with our desires, beliefs, background knowledge, etc.

Empirical results give us a data point to use in our practical reasoning. They guard us against overly grandiose claims. They can cause us to rethink things we might have assumed. But we aren’t warranted in moving from research suggests X has Y benefits, to therefore we ought to do X. In fact, I will make a stronger statement. No empirical finding about software alone should make us conclude anything about what we ought to do. What we ought to do depends on facts like our background beliefs, our desires, and our particular circumstances.

So why care about empirical results at all? Well, learning more about the world is always a good thing. Truth is valuable. But the question still remains, what role does empirical knowledge have in answering the questions we ask like:

• Should we use static or dynamic types?
• Should we practice TDD?
• Should we do agile development?
• Should we use a memory safe-language?

That answer depends on our goals, desires, and background beliefs. If these align with what research has shown then perhaps they will be beneficial for our own personal decision-making process. But these results alone, absent these broader considerations, don’t tell us anything about what we ought to do. Given that, there can be no global ought for the industry, for there is no goal or desire we all share.

Just as many advocates of empirical software studies implore us, we must not be taken in by salespeople, by the sophists who push their own solutions. This includes the appeal to the rigor of empirical studies. We must not let ourselves be led astray by rhetoric like “superstition” thrown around as an attempt to cast negative aspersions. Empirical study does not exhaust rationality. We cannot escape the need to make judgment calls on the basis of the values we hold.

¹To make repeated small alterations in a definition or example. Named after philosopher Roderick Chisholm.

Predicted (blunt) questions

Q: Do you just hate empirical software studies?

A: No. I have nothing against it at all. Studying the world is great. I just see a common implicit conflation of what science tells us is true and what we ought to do. These things are not one and the same and we should pay attention to that.

Q: But if science tells us X is more productive, shouldn’t we just do it? We want to be productive.

A: We do want to be productive, I guess. But we also want to be happy. We want to get along with our coworkers. We (I) want to drink nice coffee. We have all sorts of desires and these can conflict. I personally don’t care that much about productivity. It is not at the top of my list. I am fairly productive on the things I want to be productive on and not really interested in maximizing that.

Q: But if science tells us X increases code quality, that just seems like a clear win.

A: What I said above still applies. But I also want to take this opportunity to remove the idealistic assumptions we’ve made. We have to be careful about the meaning of words here. The meanings of “external code quality”, “testing effort”, and “developer productivity” in the study mentioned above have massively different meanings from what I take the ordinary meanings of those words to be. Perhaps I am mistaken and others would see them as tracking the ordinary definitions. But I will say, I don’t particularly care about the attributes they have measured.

Q: Isn’t this just a sophisticated way of saying you are going to ignore evidence?

A: I’m not sure how it could be that. I have no skepticism toward general scientific consensus. Nor do I have skepticism towards the findings of empirical software studies. My point is just that we have to do an extra step in thinking about applying these findings and that extra step involved judgment, weighing of goods, and introspection of our desires.

Q: Sure, as an individual you might not want to do these things. But shouldn’t businesses use this to guide their decision-making? Businesses desire productivity and efficiency, anything that has been shown to produce that should be adopted and enforced in a place of business.

A: I just think that is too simplistic of a picture. The “scientific management” of John Taylor showed all sorts of results. Perhaps you think they don’t have the scientific rigor of today, but I don’t think that matters. Businesses are complicated, complex systems. Simple-minded, thin rules aren’t going to solve our problems.

Q: But isn’t this whole thing a strawman? Consider project Aristotle from google. Here we see scientists making direct recommendations on how we “ought” to act. They measured all of these things you are talking about, desires, goals, etc, and found concrete real recommendations.

A: If anything that reinforces my point. Project Aristotle is pretty careful in parts to be clear that their recommendations are relative to the environment of google. For example, they note that team size didn’t seem to matter at google, but did elsewhere. In other cases, they are much less careful, for example in their recommendations around OKRs.

Perhaps the suggestion here is that research that takes goals, desires, etc into an account can offer “oughts” and so there is no is/ought problem in the neighborhood. Let’s assume they can legitimately offer these oughts. They can only do so as a hypothetical imperative, “if you want X you should do Y”. But that just brings us back to the same question, do I want X? Is there some other thing Z that I want more than X that conflicts with my attaining X via Y? We haven’t escaped the problem.

Q: Doesn’t leaving these things up to judgment just mean we will end up with no progress?

A: Does judgment never give us progress? Was C an improvement on BCPL because of scientific empirical study? Did the early pioneers of personal computing not use their judgment to bring us a future others didn’t see? Are the sciences devoid of judgment? I don’t think there is any reason to think allowing for judgment prevents progress, in fact, I think (but have not argued for) the opposite.

Q: Wouldn’t it be better to just follow what science has shown, even if it is suboptimal rather than leaving it up to the whims of people?

A: I think we need to be clear that judgments and whims are not the same things. In our everyday life, we can see a difference between acting on a whim and making a considered decision. The same is true in programming and business. I personally don’t think it would be better to just “follow the science” (whatever that means) rather than use our judgment. Partially because I don’t think there is anything called “following the science” that is devoid of judgment, so if we think we are doing that we are deceiving ourselves.

Q: Let’s concede your points above that empirical research alone doesn’t give us an ought. Doesn’t that mean you should apply this same logic to things outside of software? For example, seatbelts have been shown to reduce deaths in accidents. By your logic, that fact alone doesn’t show that we should wear seatbelts. In fact, wouldn’t your logic result in the further claim that seatbelts shouldn’t be mandated by law, since after all, not everyone has the same goals, desires, etc?

A: Well, yes this same logic does apply outside of software. We do in fact have to look at our desires, beliefs, goals, etc when determining what to do with empirical facts. Knowing that an umbrella prevents me from getting wet doesn’t give me any action I ought to do. But, if it is coupled with my desire to not get wet, then I can see I might want to use an umbrella. So for seatbelts, we see the same thing. If we want to be safe, we should wear a seatbelt. I always do. As for the law, that doesn’t follow at all from what I said. There are tons of different ways to justify why governments would have a legitimate reason to mandate seatbelts. Going into those is definitely beyond the scope of this post.

Q: No one believes that empirical studies alone can tell us what we ought to do. They just implicitly assume we share common goals. Are our goals really that divergent?

A: If that’s true, I am relieved. I do think it is a minority view. But it does seem to me that some people advocate for it. As for sharing common goals, I don’t know. It seems to me that we do often have divergent goals. The generative artist and the Site Reliability Engineer seem to be doing quite different activities. Even the functional programming advocate and the object-oriented programming advocate seem to be in opposition. If there are some overarching goals that we all share in this matter, they have been lost on me.

The Space Between Programs

The “space” between our programs is an aspect we almost entirely overlook, yet, it is crucial to many aspects of our design. By “space”, I don’t mean the interface that the program exposes. Nor do I mean the means of communication (http, a queue, a log, etc). I am instead looking at something more amorphous. If we think about programs and their connectors taking up space on a 2d plane, what I want to talk about is the negative space. Or in other words, what do our programs leave out, what do our programs leave room for, and what, by their existence, have they excluded?

These are the question we rarely ask ourselves when drawing an architecture diagram, and yet these problems crop up time and time again. There are simple examples, the monolith that takes up all our conceptual space so that nothing else can exist. Then there are complex interactions: the placements of these three services make it so that our new service must live in a crevice shoved between them all.

This goes beyond the technical and conceptual into the social. Programs that live too close or too far from each other have impact on how the teams that maintain them relate. Improper spacing of code can lead to improper organization relations (a sort of reverse Conway’s law). Finding the correct amount of space is incredibly difficult.

Artificial Space

There are programs where the space between them has been forced. Two programs that really ought to be one. I don’t want to dive into the criteria of what makes it so that we ought to unify programs, but in companies, there are some symptoms. When there is constant confusion or fighting about which team should be responsible for what, that is a good sign that the space is artificial. The typical advice is to make a defined interface. This is a bandaid fix.

The other answer is to eliminate the space between these programs. To make it so there are no longer two programs. Typically the answer here is not to combine the teams. There are usually organizational reasons for the division. Instead, if you are on this team, start finding different but related responsibilities that are truly separable. Begin to work on those, and give up your old responsibilities to the other team.

Vast Space

Then there are programs for which there should be space, but the amount of space is just too great. There are two options for remedying this issue, make the programs move closer to each other, or introduce a program in the middle. It might seem like the latter option doesn’t decrease the space. But by making the space easier to traverse, the distance has been essentially reduced.

There are circumstances where simply bringing the two programs closer is the right answer. But I do think that leaning towards adding an intermediary is nearly always the right answer. First, it may turn out that there is a third program out there that could use its space reduced as well. Secondly, it is much easier to combine programs than to split them. So, starting with the intermediary gives us more flexibility to design when and where to combine.

No Space

"No space" can be seen as the classic monolith. Secretly in our one program, there are many programs, looking to free themselves. Finding these programs can sometimes seem difficult. But it often isn’t. What is difficult is having the courage to split them out and the vision to see the consequences of doing so.

Nearly always the answer for splitting things out is to make some sort of lever you can control, in near real-time, to decide whether things go down the old code path or get redirected to our new program. Absent this, you are making a migration for which there is no going back. That is a dangerous prospect. Instead, start by being able to direct 10% of your traffic to the new program, and ramp up and down. Ensure your new program has adequate monitoring and alerting.

What Space Creates

We often focus on the technical properties that technologies enable, but we forget to think about the human aspects. By having space, we make it easier to think certain thoughts and harder to think others. Space creates possibilities, but also hinders accessibility.

Space also determines social circumstances. Creating space between two systems can bring about competition between two groups. Each looking to own, or not own, some part of the process. Lack of space can create entirely artificial conflict, as decisions that should be local become much larger. With a lack of space, conformity becomes the norm.

Conclusion

Space is a complex topic for which there are no easy metrics or measures we can use to decide on how to resolve the issues. Nor to even recognize the problems. These decisions aren’t merely decisions that should be made based on things like performance. Reducing system design down to a set of objective metrics ignores so much of what is important in software design. We must consider the larger implications of our choices, the impact they have on the future, and the social implications of making these choices.

Social Programming Language Constructs

As programmers, we love to act as if everything is cut and dry, unaffected by social circumstances. The mindless execution of our code makes us believe that all aspects of coding practice are equally mindless, equally free from social influence and opinion. Perhaps we don’t consciously think this, we aren’t completely naive. And yet, I think, many of us would admit to that proclivity, that default assumption that code issues are cut and dry.

I want to call attention to one element of programming that is much more determined by our social situation than we generally recognize. A feature of programming that may feel set in stone that I believe is radically different depending on the social circumstances you find yourself in. These features of programming I want to dub “Social Programming Language Constructs”.

Let’s begin with the Wikipedia definition of language constructs:

A language construct is a syntactically allowable part of a program that may be formed from one or more lexical tokens in accordance with the rules of a programming language.

Here we find that cut and dryness we love about programming. We have two constraints placed on what makes something a language construct. It must be “syntactically allowable”, and must be used “in accordance with the rules of the programming language”. Perhaps this definition is lacking in some regards because these two rules don’t precisely identify what does and doesn’t count. It gives necessary, but not sufficient criteria for what is and isn’t a language construct.

And yet, I think most of us have a good grasp of this. Controls flow operators are language constructs. Classes (in java) are language constructs. Type Classes in Haskell are language constructs. As follows directly from the definition, what is and isn’t a language construct is language-dependent.

The Languages we use

My goal here is not to dispute this definition. Not in its wording nor even in spirit. There are of course these cut-and-dry language constructs, they are real. But what I want to call attention to is another notion of language construct that I think is equally real, and perhaps more impactful. The notion of a Social Language Construct. Simply put, it is the idea that what is and isn’t “syntactically allowable” depends not just on your compiler/interpreter, but on the social environment in which you write code.

Perhaps this claim seems ridiculous, let’s start by making it seem less so. What is and isn’t syntactically allowable depends on your language's compiler/interpreter. But not just the language in general, but the concrete version you use. If you are programming in K&R C, there are certain syntactic patterns that aren’t valid in more modern C implementations. If you are programming in plain vanilla, uncompiled javascript, what is and isn’t syntactically valid depends on what browsers you want to support.

But in real systems, syntactically valid often becomes more strict than just what the compiler/interpreter allows. We add extra layers onto our syntactic validity checking via linting. Once we have added errors to our linter, we have not added additional constraints on what is syntactically valid.

In other words, we have made a new language. If a programming language at a bare minimum decides what things are syntactically valid, our new restriction has made a new language that is a subset of our existing language. Now because it is a subset, it isn’t radically different. Hopefully, we have made the language simpler in some regards, eliminating certain foot guns.

What Counts as Syntactically Valid

Hopefully, a linter defining a new meaning of syntactically valid doesn’t seem too radical. But in what way is a linter social? Well if the linter's rules were not decided by you, but instead by other people, through some social negotiation process, or perhaps through a power relation, those lint rules owe their existence to social factors. Or in other words, absent that particular social situation, what is syntactically valid for you would change.

But are linter's the only way we can change what is “syntactically valid”? Is something only syntactically invalid if a process returns a non-zero exit value? Or can code review change what it means to be “syntactically valid”? If my code can never be merged to master, never be deployed because it doesn’t meet some check, is it meaningfully syntactically valid?

Trying to make a meaningful difference between these things seems to be a waste of time. Absent an automated linter, if my code won’t ever be merged if, for example, I use goto, the language I’m working with does not meaningfully have goto. goto is no longer syntactically valid for me.

A Socially Defined Language

If social structures can merely limit what is syntactically valid, and therefore limit what counts as a programming language construct, they might not be that interesting. But it goes beyond that. Limitations were merely meant as a non-controversial way to introduce the concept. Is it not possible to add to our languages socially?

Well, our rules can go beyond removal. Our social rules for our programming languages can make new features. Consider MVC. Most languages do not have a “Model” language construct. Yet in our codebases, we often have rules about what counts as a model. We can talk about models as if they are parts of our language. As if they are the building blocks we use.

This goes further than mere abstractions like models though. Working in a codebase with dependency injection, you may find that you have your own module system. No longer do you use your language's import or require system. You have your own way of constructing modules. No one removed the module system from the languages compiler/interpreter. And yet when writing code in that codebase, it isn’t “syntactically valid” to require a “service” or a “module” because your code will not be merged.

Limitations of Socially Defined Languages

Much like non-social language constructs, the limitations of social language construction are implementation-dependent. First-class functions might be a very attractive language construct, but if the implementation of them comes at a 10x cost in performance and memory, their value might not be worth the cost. But perhaps performance doesn’t matter for our use case? If so, a poor implementation can still be useful for our goals.

Social language constructs are implemented in the social systems that regulate the codebase in question. The question of quality here becomes a bit fuzzier. Generally, as programmers, we can agree that more memory and worse performance are less desirable than their opposites. But in social structures, it is often hard to find such agreement on implementation values.

Perhaps some members of a social structure value uniformity, then the implementation’s quality to them will be determined by its ability to ensure nothing is missed and the construct is applied consistently. Others may value plurality, here the perceived implementation quality may depend on the flexibility and ability to deviate if desired. These differences in values can often lead to conflict that leads the implementation to deviate over time.

Given the complexity offered by social situations, social language constructs offer much less stability than those codified in the language. Of course, language choice is part of the social fabric, but generally, changing languages is a difficult process. Social situations, particularly in a work environment, are changeable through many external means. So, these social language constructs do not rest on solid footing. This means as time passes, we may lose understanding of the functions these constructs once played.

Benefits of Social Language Constructs

How can we use Social Language Constructs for good? By focusing first on their implementation. How have we decided to implement our social language constructs? Is it via informed consensus? Via senior engineers deciding what is acceptable? Via deference to our code ancestors? Via hype and trend following?

Once we honestly consider these structures, we can consider if they properly reflect the values we hold. Is our decision to enforce X based on a deeply held value, is it enforced in a manner that is consistent with our values, does it update as our values change over time?

By going through this process we can end up with Social Language Constructs that not only reflect our values but help others see them more clearly and practice them themselves. Perhaps we have a construct involving code comments. If we have consciously thought about this construct and the way it fits into our values, we should find that by reading and writing these comments, people should feel the values we claim to hold.

Downsides of Social Language Constructs

Social language constructs can be detrimental especially with they reflect the power structures of the social situation they are embedded in. For example, there may be a code in a codebase that cannot be changed. The author may be a very important more senior developer who doesn’t take kindly to people messing around in “their” section of the code. There may also be things not expressible simply because the valid means of expressing them have been deemed forbidden by a committee on code style.

Those who do not have a voice in the social structures at large may now find themselves also marginalized in the code. This is I think one of the most ignored features of our codebases, how we assert social pressure and conformity. Codebases are not neutral repositories of code. Code style guides are not merely rules for what goes in text files. People exert control and social influence through the code they write.

These forms of control are seemingly ignored by many. It is quite understandable. It is easy to mistake a “desire to not be controlled via social language constructs” with “I don’t want a linter, so I can write whatever code I want”. Without the conceptual framework offered above, how could anyone hear complaints about linters and style guides as anything other than that?

Conclusion

What should we think about these language constructs? Given the social circumstances we inherit from the culture surrounding us, I think we should be skeptical of them. It is incredibly easy to import existing power structures into our code bases. It is very easy to unknowingly exert social control through seemingly neutral code rules. So should we completely rid ourselves of these constructs? Is that even possible?

I think it’s clear that we can’t rid ourselves of these features. We cannot take the "view from nowhere", believing ourselves to be completely neutral. Instead, we ought to think consciously about them. We ought to see them as part of the culture we create. Our choices of social language constructs reflect our culture, but also reinforce it.

I’d like to suggest an approach to this problem I rarely see in practice. A conscious focus on diversity and plurality of code values. This does not mean taking an “anything goes” approach. Instead, it means ensuring that the values we uphold align with the principles we hold dear. But at the same time, not limiting expression, nor limiting relative priority of the values held by those in our engineering org.

We need differing values expressed in our codebases. Some may care about performance, others approachability, security, debuggability. Whatever values those on our engineering teams hold, we shouldn’t try and limit their expression. Instead, we should cultivate this plurality. By doing so we may find our codebases are a bit less uniform and a bit less predictable. But I claim the trade-off is worth it.

My Experience Building an Editor in Rust

I've always wanted to build a text editor. I've played around before with trying to modify existing editors like codemirror. But ultimately those just felt incredibly unsatisfying. While I was able to make some fun little experiments with them, I was just gluing code together. As I started to play with Rust, it seemed like the perfect opportunity to go beyond glue code. To write a GUI text editor from "scratch". Overall, I'd say it worked out pretty well. Here's a screenshot of where things are right now. (I'll explain what you see in a bit.)

The question of course is, what does "scratch" mean. In my case, I decided that "scratch" was going to be defined as using sdl2 as my set of primitives. So I had a fairly high-level way of dealing with things like drawing to a screen and rendering fonts, but nothing text editor specific. This choice I think was pretty good for me to get stuff going. I have no experience with graphics APIs and had I started there, I might have just stayed there.

From the other angle, sdl2 was also good in that it didn't do too many things for me. There were no text edit widgets I was trying to control, no framework, just a library for windowing, drawing, font rendering, and eventing.

Background and Goals and Building

Before we talk about the general path I took, let's talk about my goals. Initially, my goal was to just build an editor. I wanted color syntax, I wanted general editing capabilities, that was about it. As time grew on, my goals became less and less clear. What I discovered as I kept developing is that building a text editor was a bit more finicky than I had expected, but also way easier than I ever imagined.

Rust is a really fast language. In my career, I've mostly worked in slow languages. I mostly worked on problems where speed was important, but not the most important. For this project I wanted performance. I use emacs to do my Clojure development and to be honest, it's terrible. I like paredit, I like my cider setup, but emacs freezes constantly. When I try to print data out in the repl, I can completely lock the editor. That wasn't something I'd allow my editor to do, so I thought I'd have to be super clever to make that happen. Turns out, I didn't.

Influences

Casey Muratori

As I was looking to undertake this project, Casey Muratori had put out his perhaps video about building a fast terminal. He walks through how he made the rendering fast by default using a standard technique called a font atlas. So that's where I started. If you like me had never heard of a font atlas, it is a simple idea. Draw a picture of all the characters you need and as you are rendering, you just reference those characters. No need to render the font each time.

In my case, I focused on ascii and just made a texture of all those characters. It was blazingly fast. Turning off vsync, I had 1000s of frames per second. I played around with manually adding some color and found that the performance held. Good sign. Now I had two things to figure out. 1) What data structure to use for editing. 2) How to do color syntax? Luckily for the first, there was some prior art that lead me down an unexpected path.

Jamie Brandon

After I started my project, I found out that Jamie had implemented a very similar idea but in zig. Because it was so similar though (used SDL, a font atlas, etc) I didn't want to look too closely at it and spoil the fun. But what I did do was read his blog posts on it and the one on text handling was particularly interesting. In the article, he says he just kept the text as a string and it was fast enough. Given my simplification of only handling ascii, I decided to try a Vec<u8> and it turns out, that was incredibly fast.

Modern computers are way faster than I think many of us realize. I was able to implement text editing operations just by using the Vec. When I insert text in the middle of a line, I insert it into the Vec, shifting the whole thing down. And yet, every file feels smooth. Didn't need to use a rope or any fancy data structure like that. Of course, those have other benefits, but in this case, I could keep focusing on my text editor.

Color Syntax

At this point, I had edit and display, but no color syntax. To figure out line breaks, I parsed the buffer and made a very inefficient line array of tuples with (start, end). I think this is one of those choices I wish I had done differently, mostly just on how I wrote the code, but it worked. One thing it let me do was only render the visible lines, so my first instinct for color syntax was to take advantage of that. I knew that editors like sublime use a regex-based solution that only looks a line at a time. So I thought, maybe I should just use that approach and take something off the shelf.

Failed Attempts

Syntect

I first looked at using Syntect for my highlighting. It works with sublime syntax files, so I'd be able to support many languages out of the box. It was incredibly easy to add to my project and well documented. I was able to integrate it very quickly and very quickly learn it was much too slow for what I wanted to do.

Now, this isn't to fault Syntect. The sublime syntax format is based on TextMate's format, which relies heavily on regex. Given the complexity and constraints of these formats, there is only so much you can do. But it wasn't cutting it for me. You see at this point in the project, I wanted to keep things as simple as possible. Syntect though would often completely miss the frame deadline. If I was in the middle of editing a file and needed to re-syntax-highlight it, it would kill the performance. So I had to look elsewhere.

Tree Sitter

Tree Sitter is a very interesting project out of Github. It does incremental parsing with really robust error handling of many languages. So, if I added tree-sitter to my editor, I should get syntax highlighting for cheap, but also, not have a performance issue as I'm editing. Or so I thought.

First getting tree-sitter setup was far from straightforward. The packaging and build situation for tree-sitter was a bit weird. But once I got that going, I was actually quite sad to find out that the highlighting portion of tree-sitter was not incremental at all. I looked for a little while at making my own custom integration and I knew it was possible, but also didn't sound like fun. So I took a different path building my own custom setup.

Tokenizing Every Frame

I started with the simplest possible thing I could do, make a custom tokenizer and tokenize on every single frame. So I did that, wrote a really terrible, representative tokenizer, and revamped my rendering to use it. Turns out, that was actually really fast and easy! Even with doing the incredibly naive thing of parsing every single frame, I was able to have the largest source files I could find on my machine open and editable in 60fps. Honestly, I was pretty blown away by how well that setup worked. Admittedly, my tokenizer is not very good now right. But as they say, that's just a matter of programming. I know that I can keep and even improve the performance while making it more feature-rich.

Moving Beyond the Traditional Editor

At this point, I had the basics and wanted to play. First question, since I'm a big fan of Muse, I thought what if my text editor was a canvas? Implementing that was very straightforward, if a bit finicky, and moved me directly into the more interesting things I now wanted to do.

Token Pane

As I was working on my tokenizer, I wanted to be able to see the output of the tokens right in the app. So I created what I called the token pane. If there is a pane called token_pane Its contents are defined as the raw tokens of the active pane. So now I could see exactly what things were tokenizing into. Incredibly useful for debugging.

Action Pane

Next was the action pane. Quite a bit trickier. Here I would display every action that happened in the app. But, what about scrolling the action pane? Well, if I did that, then as I scrolled the action pane would constantly get new actions. The whole thing was a bit of a mess.

The other hard part of this setup was that I originally didn't have a great way to refer to panes. My actions would be something like "MoveActivePane". But what was the active pane, or more precisely, when? Well, if I was looking at the action pane, it was the active pane, so now as I'm filtering out action pane actions, I would filter out all active pane actions! Not what I wanted. So I had to set up a system where your actions resolve to ids.

Draw Panes

Ultimately what I want out of an editor more than anything was captured in this early blog post on LightTable.. In it, they imagine an editor configurable in itself. But I wanted a different flavor. What if you could extend the editor in any language? I already had seen ways of seeing the internals of the editor and using panes as output. What if I could do the opposite, use panes as input? I later discovered a nice name for this, afterburner rendering ala Mary Rose Cook. Here's an example. What you can see here is some javascript that prints output like rect 4 10 100 100, that output is then parsed by the editor and drawn as rectangles to the screen.

Text Space Drawing

Obviously, rectangles aren't the most useful thing to draw. But I also played with rendering in text space. For example, here is a quick proof of concept of underlining text mentioned in rust compiler output. On the right, you see some unused imports. On the left, a quick bash script for drawing that to the screen.

The dream is that as things evolved, your editor could gain new powers simply by code you have running in panes. No need for an extension language. Simply output things to stdout and you can control and extend the editor. What I found with this experiment is that even with the most naive, unoptimized code doing things that way was entirely possible.

Accessing Pane Content

One fun experiment I played with was a way for any language to get the contents of a pane. Obviously, if a pane is backed by a file, you can read that file. But that wasn't good enough for me. I want you to be able to access the contents before changes have been saved. Further, you should be able to access it with just standard tools. So, I exposed the editor as an http service.

Honestly, as weird as it may seem, it was pretty easy to do, not expensive computationally, and made it easy to access the data. Ultimately, I'd love to even expose more things. Like being able to control the editor via http requests. Now, external programs can interact with the editor in a way I've never seen supported. Once we have that, it means we have the full Unix power accessible in our editor in a very first class way.

What Went Well

Rust

Rust was a wonderful language to write this in. I've gotten past the learning curve where I basically can write Rust without thinking too much about it. With all of these features, I was just thinking about the problems and not the language. While of course, Rust's ownership model pushed me in one direction, it never felt like it limited me.

One thing I really have grown to enjoy is rusts explicit clones. Without GC, clones can be expensive in a tight loop. Making them explicit lets me be able to easily spot my bottlenecks. Several times I could do a quick clone to get things working and then come back and move the code around to make the ownership clear and avoid the clone.

SDL

SDL2 was easy to get going on this project. From day one I had things drawing on the screen. The primitives SDL provides were just what I needed to focus on my task and not worry about the details.

My Willingness to Let the Code be Messy

I often get to a point in projects where things just stall out because I want to clean up my code and I don't know exactly how I want things to be. I know that the path I'm on is not the right one and instead of forging ahead with working code, I stop and consider, then lose interest and stop working on it. I didn't do that here. The codebase is a mess. There is a lot of duplicated code. And as it stands right now, things are broken from unfinished experiments. But, I got a lot done with this spare time project. And even with all the duplicated and poorly designed code, I only have 3442 lines of code.

What Went Poorly

My Messy Code

While my willingness to let the code get messy was definitely good. The actual messiness of the code did cause some quality issues. I never went back and changed my line parsing to take advantage of my tokenizing code. So I loop through each file twice per frame. Because I was doing things hacky from the beginning, it took me a while to get to the point where I could track changes and know if I need to do things like reparse a file. The code now has that ability, but hooking it up was a decent amount of work and I never did it.

Lack of Action Reification From the Beginning

I spent a lot of time refactoring code to emit data when an action happened. This is something I should have done from the beginning. As I was looking to implement undo/redo, this lack of reification and centralization of event handling led to all sorts of issues that still exist in the codebase.

Areas I'd love to Explore More

External Extensibility

I really had a blast making this. It went in entirely different directions than I expected. I think this idea of an editor extensible via its own pane contents is really interesting. Of course, emacs is perhaps the ultimate example of an extensible editor, but it is still via elisp. Admittedly, I am sure there are parts where keeping things external may become a bottleneck. If that became the case, it might be interesting to explore webassembly extension points. But what I would find more interesting is to try and continue down this path. Perhaps there is a nice binary format that makes these things not too slow. Could you for example implement a language server protocol library in the editor without it needing to be built in and without it needing to use an extension library?

Focusing on external extensibility also means that these extensions could be used by other editors. Basically, this work would expose an editor IR. An idea I find incredibly interesting and perhaps incredibly powerful.

Non-Traditional Layout

I never took my canvas idea very far. I didn't even implement panning around on the canvas. But I found myself loving these layouts. I did some code live in the editor for things like Conway's game of life, and being able to move my panes around and resize panes based on my focus were very nice interactions. I always have been a bit of a messy worker and I find having positions for things helps me out tremendously.

Conclusion

I had a blast doing this. Rust was a great language. I tried out lots of interesting ideas and they worked pretty darn well. But I think my biggest takeaway is that there is so much opportunity here. Text editors have largely remained unchanged in terms of user experience. Now with language servers, they can offer better static analysis and refactoring tools, but the ways of interacting with them have largely not changed.

There is so much potential here. Why haven't we explored new avenues for user interfaces in text editors? Why aren't there editors that let me work with them more flexibly? Is a column-based layout with some intellisense really the best there is? Will we never discover something better? In my view, there haven't been many serious attempts at this. Maybe there needs to be.

Towards Aesthetic Elements of Programming

Programming styles are immediately recognizable. And yet, the elements that contribute to these styles remain unnamed. What is it that separates the classic Scheme style of programming from a modern OO java style of programming? It is not the language. It is possible to write scheme, javascript, or whatever in the style of a modern OO java program. Nor is it just the paradigm, it isn’t as if all OO programs have the same style. So what precisely are we picking up on when we note the style of a codebase?

Identifying Elements

Ideally, to understand these styles we can consider elements that combine to form a certain style. If we identify the right elements, we should be able to take them, vary them, and combine them to get a particular, recognizable style. Perhaps even, we can see what combinations of these elements are rare, or unseen in the programming world.

This is what I hope to explore in the essay. I doubt I will get the elements right from the beginning. But perhaps by throwing some out there we can start to think more clearly about what the elements really are. We can find better ways to divide things and start having an understanding of style.

Verbose vs Terse

We can significantly alter the way we communicate in code by varying our verbosity. This might be at the level of a whole program or in a particular section of code. We might adopt patterns that require a certain level of verbosity (objective C named methods, early redux culture) or adopt whole languages that focus on terseness (j/k/apl).

Verbose code can help with clarity. Long, expressive function names can help communicate the semantic meaning of the code underneath. Repetitive elements in code can help reinforce patterns, making it clear what code belongs where. Verbose code can guard against errors, preventing simple transpositions from happening by requiring fully specifying the context of a value.

Terseness can be its own kind of clarity. Terseness can convey unity (eg. mathematical expressions). Terseness can help convey the lack of semantic meaning (eg. single-letter variables). By being terse we can make clear what matters and what doesn’t. With verbose code, it can often be unclear the exact shape of the code. It can be unclear what the relations between parts of the code are, terseness helps us here.

Dense vs Sparse

Whereas verbose and terseness are related to the way elements are spelled out, density and sparsity are about the relations between these elements. A sparse codebase spreads its functionality out. This can take many different forms, many small classes, spread out across many files is a common example. A dense codebase packs many things into a singular place.

In some ways, dense vs sparse might seem merely superficial, just concerned with the presentation of code rather than concepts in the code. I think this is far from true. Deciding where we place elements (regardless of what “where” here means) has a big impact on how code is understood. By grouping some things and keeping others separate, we are signaling something to other programmers.

But this is not merely a signal. We may intentionally make our code dense to, for example, discourage reuse. Large single blocks of code may only serve a very specific purpose that won’t be useful to others. We might keep our code sparse to enable layering of functionality. This is often seen in classes with large inheritance hierarchy structures.

Structured vs Unstructured

The contrast here between structure vs unstructured is not referring to “structured programming” in the “goto considered harmful" sense. Instead what I am referring to is a program-level structure. Some codebases provide an overarching structure, perhaps providing categories for where each sort of item should go (eg. MVC). Then some codebases are more free-flowing. These don’t provide some sort of meta-narrative about exactly where a bit of code belongs.

Highly structured code can be a means of communication. By placing elements inside this structure, we are communicating something about these bits of code, we are telling the reader what we consider to be the most important way bits of code are related.

Less structured code does not communicate any less. By keeping our code less structured, we can signal several things. First, we may be saying that this code is not in its final form, it is in flux and the organization may change. We might also be communicating that this code is a complete whole and does not need to be divided into parts.

Direct vs Indirect

There are many different ways that directness can display itself in our codebases. First, we may call a function directly, or we might find some indirect means of calling this same function. Perhaps we define an interface and inject something with that interface, that then delegates to our function.

But direct and indirect are not merely about function calls. We can for example, have code that builds up some tree or hierarchical structure and then interprets it, rather than code that directly runs a computation. We might convey data onto a queue, with some consumer elsewhere rather than locally deal with our data. Directness of course comes in degrees and is relative to a particular structure. Some parts of our code will need to be direct in some way, but they may live inside of a larger indirect structure.

Open vs Closed

To avoid confusion I should mention from the outset that I don’t mean the Open-Closed principle from solid. Open and Closed here are a bit broader in their meaning. A system might be open by being completely introspectable. It may be closed, by encapsulating or protecting some parts of its data. But this open and closed spectrum does not need to be enforced by the language to be an aspect of our codebase. A codebase can treat a structure that is technically open as if it were closed.

Open vs closed does not merely refer to data protecting either. A system can be open by accepting data it is unaware of. A closed system may guard the borders, only allowing particular data in the exact shape specified. An open system may expose reflective ability, it may provide metrics, or tracing. A closed system might provide strict guarantees, correctness, performance, memory usage.

Generic vs Specific

Our code can be Generic or Specific on many different levels. Our function may be polymorphic, allowing it to work on different inputs. Our codebase may be solving a generic problem (a renderer) or a very specific one (a renderer for the Mandelbrot set). Even if we are solving a generic problem, our code can be specific in its details. We may provide specific instantiations of our more generic method. We could on the other hand leave our code entirely generic, acting as a framework for which people can make more specific code.

Concrete vs Abstract

Concrete vs Abstract may sound a lot like generic vs specific, but I think there are important differences. First, concrete vs abstract is about the subject matter or the concepts used in the application, rather than their capabilities. To give an example, Car may be considered Concrete, whereas Monad might not. Both Car and Monad can be used generically, with more specific instances utilized elsewhere in the codebase. Admittedly, it is much easier to see how abstract code coincides with generic code. But this need not be the case. Perhaps we are writing abstract code about partially ordered sets, but using them for a very specific purpose (proving that certain properties hold).

Regardless, the use of concretions and abstractions in our codebase is a powerful way to communicate. By making something concrete we put bounds on it, we invite readers to apply their common conceptual categories to it. By making things abstract, we invite readers to expand the bounds of what they consider.

An Example

I think these elements are great candidates for being elements of programming style. Let’s take a random combination of them and think about what a program written in that style might look like.

• Verbose
• Sparse
• Structured
• Indirect
• Closed
• Specific
• Abstract

There is a lot to unpack here. First, it must be said that of course, no codebase is uniformly all these things in all its parts, so will be generalizing a bit. Second, trying to think about all these items at once is hard, so let's pair them up a bit and think through the implications of those pairings. Combing verbose and sparse might look like a library with a decent amount of boilerplate, where these boilerplate elements are rather spread out, perhaps even orthogonal to each other. We might imagine a verbose and sparse Redux or Rails codebase.

Structured and Indirect might combine to a structured means of dispatch. We can imagine a dependency injection setup like hibernate. Nothing calls anything directly, but everything has a clear hierarchy. Here again Redux might be a good candidate to consider. A reducer structure with many sections combined with a hierarchy of selectors could meet this criterion. Closed and Structured can be combined as a means of making illegal states unrepresentable. Here, we can ensure that no one can trigger any illegal actions, everything is clearly defined, and it cannot be changed without accounting for our structure.

Specific and Abstract are a bit harder to imagine. Imagine a web app designed to do a particular thing. It is not meant to be generally applicable or configurable, and yet the entities we find in our program are not concrete and familiar. Instead, we find things like executors, factories, or perhaps our code is organized around a free monad. While the concepts we are employing may be able to be used generally, our code is not doing this. Instead, we have very specific uses of these abstract concepts. Think, about the way people might make interfaces for everything, and yet only have one instance of them

So given this set, we can think about what it would be like working on a codebase following these styles. First, errors would be dealt with via our structural and closed system. Next, we would not introduce many different concrete types, instead, we would make use of abstract concepts, perhaps certain event types, or operations in an algebra. Further, things would not directly call other things or directly depend on things. Instead, these would be passed around to use or gathered through some abstract means. Further, as we introduce new concepts, we would not colocate them with existing concepts, but instead, following a pattern established by our structure, introduce them in some separate section of the codebase.

This feels very much like some Java Spring apps I’ve worked on, but also some redux apps I’ve used. These two could not be further from each other in terms of paradigm, and yet, there is a shared sense of style here. Now that does not mean they are equal, it does not mean working on them feels the same, just that they share something in common. Nor do I mean to suggest all Java spring apps or all Redux apps follow this style.

Conclusion

Hopefully, these elements ring true to you. I think the most questionable distinction here is between abstract and concrete being different from specific and general. And yet, the more I try to get rid of this element, the more I see examples of it. For example, we might write general code, that we give concrete names. Maybe our code just works on some general notion of distance, but we name our parameters distance-in-seconds. Here we have a general, concrete coding style.

But as I think more about this, explore more codebases and get feedback from people. Perhaps I will come up with a new set, or replace elements on here. Regardless, I think these concepts are useful and helpful for discussing code style. This is something we need to do more of.

CDD and the Non-Linearity of Discovery

I have written previously about how Card Driven Development is extremely detrimental to the development process. In that essay I pegged the problem on CDDs lack of proper knowledge sharing and its incorrect splitting of responsibility. Today I want to focus on a different attribute of CDD that has been bothering me lately. This is the start of me trying to deliver on the promissory note I left at the end of the last essay. If we are to replace CDD we must start with an understanding of what programming is. That is our starting point today. Programming is Theory Building, does CDD help us build theories?

Perhaps my last essay could be seen as a bit of a caricature. I don’t think it is, I have in fact seen this process play out in practice. But let’s assume I am wrong. Maybe CDD isn’t disasterous. Maybe I gave too much credit to a process for creating such turmoil. What then would the problem with CDD be? If you don’t see the social problems of CDD that I do why should you not use it?

Theory Building and the Non-Linearity of Discovery

Theory Building is an activity of trying to take information about the world and it put it to some particular use. Sherlock Holmes works to build his theories to apprehend criminals. Someone decorating a home builds their theory so that they might make their home cozy, habitable, and enjoyable to live in. A software engineer builds their theory to enable them to solve the problems at hand, not only the problems of today but the problems that will arise tomorrow.

Consider an instance of complex theory building, how does it proceed? Does Sherlock Holmes proceed from clue to clue, following a linear path to apprehension of the criminal? No, the investigation is full of false starts, of reimaginings, of unexpected twists. But what is most important are Holmes's thoughts on the matter. Holmes's theory involves learning lessons and responding to them in unusual, particular ways. The process of learning and responding is important to his investigation. If Holmes merely gathered the data, brought it to the police and let them decide his next steps, would he have been as effective?

The same is true for all theoretical investigations. These investigations give us embodied knowledge. The fact that we worked out the theory is an important aspect of its creation. We had to go through this process of discover, we had to have the false starts, without them we would not have created the same theory.

CDD and its Incentives for Linearity

On the surface it might seem like CDD should allow for this non-linear process. CDD is common in agile circles where small iterations, retrospectives, and course corrections are advocated. Isn’t agile explictly a response to methodologies like waterfall that didn’t allow for non-linearity? In one sense of the term, yes, agile does not assume linearity. It accepts that you cannot know the destination from the beginning. But remember, the question is about CDD and the non-linearity of discover, not agile and planning non-linearity.

CDD by its splitting of responsibility between two separate groups, product and engineering, makes using the results of discovery incredibly difficult. What is crucially missing is the allowance for discovery to take center stage when it comes to planning. As engineers work to build a theory (by programming) their discoveries must be incorporated into the plan. But what mechanism is available in CDD for doing so? Well, the primary mechanism is the card. This is the communication tool of choice, it is the pre-requisite for getting work sanctioned.

But here is where the problem starts. The card cannot serve the task of feedback that it has been tasked with. It is inadequate for a few reasons. First, the time horizon for feedback to be incorporated and for course correction to occur is too long. As engineers are programming, they might find that their theory was incorrect, that the work they had planned to do is no longer the correct choice. What should they do? Well, given CDD the pressure is to forge ahead and file a card for future consideration. But often the correct answer is to stop the work you planned and instead go learn, go develop your theory better so you know the correct moves going forward.

The card also fails to serve as a proper medium for conveying the information it needs to about this discovery process. First, cards are factual pieces of proposed works. They often take the form of a user story along with acceptance criteria. But this is not what is needed. Because CDD has split the responsibility for deciding what work is to be performed between product and engineering, what the engineer now has to do is convince product that what they have discovered is the truth. They need to help product understand their theory.

Finally, the card fails because it assumes that all the work to be done has a deliverable. The process of discovery is however not like this. When working to build a theory much of the work we do has no direct outcome. We make guesses that turn out the be wrong. We pursue paths that are dead ends. But this work was not useless. If programming is theory building and the primary output is knowledge rather than a build artifact or source code, then the engineer learning new things is the best possible outcome.

Making CDD Work

In the previous section, I have ignored some of the nuances that occur in the CDD process that might make it seem I am not being fair, let me address those. First, is it true that engineers are not given the flexibility to course correct without creating a card? Well, if the card is written the way many advocates of CDD recommend, it would seem no. CDD advocates often say that a card should be about detailing a concrete customer problem, rather than a particular solution. It is up to the team to determine how best to solve this problem.

Here we can say a few things. First, it is readily admitted by most parties that achieving this goal is something must companies struggle with. So CDD as it is practiced, rather than in its ideal form still runs into this issue quite a bit. But we can go further than this. What problems make sense to tackle, what problems are meaningful depends on the theory we have. How many times have we done work where we solved a particular problem, and solved it well, only to find out our customers didn’t use it? Many times, our customers themselves told us about this problem, will even admit it solves the problem as originally stated, but it turns out they were wrong about their needs. Setting forth a problem constrains our theories, if in the process of discovery we come up with a new theory that sets us on a new course with new problems, what are we do to given CDD?

Next, I have acted as if the card is the only medium of communication available. If we view any guide on how to practice a CDD process, we will find this isn’t accurate. The card is a starting point, it precedes the conversation. Planning is not about taking the card as given, it is about discussing the card and coming to an agreement as a team on the plan to move forward. And yet still this isn’t enough to handle the discovery process. First, the card frames the discussion as does the planning meeting. The goal is to come to a decision for action, not to contemplate a theory. But even if this isn’t a barrier, the card won’t be adequate, because the knowledge that has been gained by the engineer might not be expressible for them. We can know all sorts of things we can’t express. We can even teach others those things we cannot express. But the card is not the medium for doing so.

Finally there are all sorts of different types of cards. What it might seem I am describing is a CDD process without the Spike card. The spike card is purpose built for the discovery process. It is a time-boxed card made for discover where the deliverable is the findings that the spike uncovered. And yet, this isn’t adequate. The discover process is not a phase, it is not a time-boxed part of the process, it just is the process. If programming is theory building, then what has been relegated to a “spike” is really the main function we are performing, with code as a side-effect. But CDD get this exactly backwards.

What Does This Process Look Like Without CDD?

It might seem like am saying that engineers ought to just spend all their time thinking, never committing to any plan, never taking actions toward concrete goals. I can certainly understand that reaction. CDD is meant to keep engineers grounded and to hold them accountable. We all have seen engineering teams who seemingly twiddle their thumbs all day, working and reworking things to no end, never delivering anything of value.

Getting rid of CDD is not getting rid of accountability, nor is it getting rid of progress. It is about moving the decision making process closer to those who have to execute on these decisions. It is about accepting that knowledge matters when we need to make be sure we make the right decisions. It is about accepting that we need non-linear processes to discover what we ought to be working on.

Our products today are technology. They are composed of bits. Why then should the “product” team not be experts in technology? How can they make the right decisions for the product if they don’t truly understand it at the core, fundamental level? At the same time, how can engineers be expected to make the right decisions if they aren’t given the data they need and allowed to build the theory regardless of what path reality pushes them towards?

CDD’s push towards linearity is what causes this feeling that we can never accomplish anything in software. It is why year after year it seems that customer needs aren’t being met. Over and over again CDD forces us down a linear path to dead-ends. It prevents feedback mechanisms from operating as they should. If we are to make engineering more productive and help engineers have a better understand of the problems they truly need to solve, we must rid ourselves of CDD.

Conceptual Preservation

In the last post, I mentioned how we ought to learn from philosophers who have already explored these notions of Conceptual Engineering. Here we will do exactly that by focusing on the work by philosopher Matthew Lindauer Conceptual Engineering as Concept Preservation. Despite the title, Lindauer is not arguing that conceptual engineering is merely concept preservation, but rather that concept preservation is an important aspect of Conceptual Engineering. In fact, Lindauer's paper focuses on concept preservation because as he sees it, it may be the easiest or clearest case of conceptual engineering.

Lindauer brings up concept preservation as a way to combat some constraints that some philosophers have placed on Conceptual Engineering. Most notably, Herman Cappelen has claimed that we don't really know how concepts change over time (Inscrutability), nor can we control how they change over time (Lack of Control). Lindauer wants to suggest that Inscrutability and Lack of Control are much less of a concern when it comes to concept preservation than to re-engineering and de novo engineering efforts. For this article, we won't dive into his argument here, but keeping this as background is important as we think about conceptual preservation as applied to our programming practice.

Why Preserve?

Conceptual Engineering involves taking a normative stance, asking what our concepts ought to be, rather than what they are. Frequently we find concepts that are deficient in some regard and we aim to fix them or, to use the term of the art, to ameliorate them. But what if our concepts are good? What if we have found a concept that we think is beneficial to keep? Are we done with our conceptual engineering work? Not by any means. Over time semantic drift is inevitable if our words and concepts aren't protected and preserved.

As programmers, we tend to think that our responsibility is to write clear, clean code. If we do this, others will be able to understand our ideas and the codebase will be properly maintainable. Yet, this never seems to happen in practice. As others begin to contribute to our codebase our concepts become lost. No longer do our interfaces mean what they once meant. No longer do our concepts drive changes in the codebase. Instead, people approach our codebase with entirely different preconceived notions and ascribe a new meaning to our concepts. This meaning can start to catch on and now we have two different groups with two different conceptions of the same term. These competing conceptions wreak havoc on our codebase.

We must work hard to preserve the concepts that matter. We must get them in others' heads, we must teach others to act correctly in light of these concepts. A failure to preserve concepts can lead to undesirable effects. Think about how Alan Kay feels about the new understanding of Object Oriented Programming. Think about the way in which Agile has changed from something intended to help programmers, to something intended to micro-manage them. As programmers, we must expand our notion of what is our responsibility. We must work hard to preserve our good concepts and change our bad ones.

Easy Fixes That Don't Work

There are two solutions that might come to mind on ensuring concept preservation that I want to argue are dead-ends. First is the idea of self-documenting code, code that by its very nature ought to make our concepts clear. The second is actual documentation, whether that be in the form of docstrings, long-form docs, architecture decision records, or any other format. Neither of these initiatives is enough to secure our concepts. To make this result a bit less surprising, consider the conceptual drift of English words. Does a dictionary prevent drift from occurring?

Let's tackle these in turn and show why they are insufficient. First, self-documenting code can definitely be a good goal. Writing clear variable and function names is fantastic and can certainly aid in comprehension. But no matter how clear our code is, it does not capture our concept completely. Our code underdetermines our concept. No matter what structure we give to our code, there will be multiple concepts of the code that are compatible with it. This is especially true at all the points of our code we have intentionally made extensible. Extensible code necessarily allows our concepts to be open to drift over time, this can be very good.

What about documentation proper? Of course, our code underdetermines our concepts. That is why we document it, describe background, talk about edge cases, discuss possible future extensions. It could be argued that our concepts are still undetermined, by the documentation, but rather than do that let's take a different tack. Preserving a concept isn't merely to write it down, it is ensuring that people who utilize this concept have in mind the same concept as what you intended. Further than that, it is that their actions are consistent with that conception.

Imagine a codebase that follows the MVC pattern. Let's assume that the codebase is heavily documented and has canonical definitions of Model, View, and Controller. There are two ways in which the concepts here can drift. First, a team member may have misunderstood some concepts (say some distinction between View and Controller). When teaching these concepts to a junior engineer, they explain them incorrectly. This incorrect understanding spreads and makes its way to the whole team. In this group, our conception has changed despite our existing documentation. In fact, at some point, someone might even go and update the documentation with this new "correct" understanding.

But concept preservation can fail even if no one has a different conception. Imagine the same codebase above, but this time all team members have internalized the same conception. If you asked them to explain MVC they'd give you the same explanations. But there is a team member who constantly makes the mistake of putting the wrong code in the wrong place. This isn't because of some bad conception, just a simple mistake. Over time others begin to follow this example as they create new code. This way of organizing code becomes habit. Here, our internalized concept hasn't changed, and yet our practice reveals an inconsistency. We have in our actions allowed conceptual drift to occur.

Practices for Preserving

To be clear, writing clean code and good documentation are good practices to help with concept preservation. They just aren't sufficient. Nor will they be as effective as they could be if we don't consciously think about them as serving that purpose. The same applies to other practices. Code review is a fantastic place to help prevent conceptual drift. But many times it becomes about style over substance. Pair programming is a great way to detect conceptual drift, it allows you to go beyond the code artifact and actually investigate people's statements of their beliefs.

I'd imagine right now you can think of a number of practices that can be used here. Diagrams, presentations, glossaries, in general, any presentation or explanation of our concepts can help us with preservation. But I want to stress that none of these things exhaust our concepts. Our concepts are in-extricably bound up in human-beings themselves. To be in possession of a concept is not merely to be able to repeat it. It is to be able to act in certain ways, it is to be able to apply the concept in a number of different contexts. It is to be able to understand how the concept bares to counterfactual situations.

In other words, ensuring that others know facts about our concept is not enough to preserve it. This is a mistake software architects often make. An architect may give a presentation on the desired system architecture, they may write extensive documentation, ensure each team member can explain the architecture, and yet find that after the work has begun the system is entirely different from planned. Why is this? First, it may be that despite a person's ability to articulate the infrastructure they do not have that deeper knowledge of how to make it reality. But it may also be that the architect has not done their job of convincing others.

Persuade or Don't

Preserving our concepts requires persuasion. Merely building a big codebase of "best practices" or creating standard boilerplate for new projects does not actually get these concepts into other heads and help them live them out in their actions. Other engineers may disagree (and perhaps rightly so) with your conceptions. These engineers may intentionally cause these concepts to drift, twisting them until they fit the shape they were looking for. This is the inevitable outcome of forced concepts.

If we want to practice conceptual preservation, we must ask ourselves if our concepts are worth preserving. We must ask ourselves if others deem them to be so. We cannot preserve our concepts by fiat. No amount of power can keep a concept in place. So, if conceptual preservation is important to good codebases, we may want to reconsider our practices. Are architects a good solution to ensure quality? Is the blessed infrastructure team that imposes concepts a good idea? Perhaps not.

Conceptual Engineering

Naming things is one of the hardest things to do in software. Why? Is it because all the names have been taken? Is it because we are trying to get at some essence that our words need to capture? (Like some sort of fantasy magic system.) Perhaps at times, we feel both of these things. But I think that the difficulties we feel naming things stem from an unexpressed anxiety. Deep down we understand that naming is more than assigning a word to a value. When we are naming things, we aren't just deciding what something is, instead we are asking the question of what something should be.

Taking this normative stance, this view that we aren't merely trying to explain concepts, but trying to ask the question of what our concepts ought to be, is called Conceptual Engineering. Conceptual Engineering here should be understood to include both working to change the meaning of existing concepts (re-engineering) and creating new concepts for a given purpose (de novo engineering). Naming things is hard because in doing so, we know deep down that we are advocating for this concept. We are putting our weight behind this new concept or in the redefinition (expansion or contraction) of an existing concept. This is a heavy burden to carry.

Carrying the Burden

Code we call over-abstracted has often failed us by not carrying this burden well. But sometimes, we are also asking too much of this code. Concepts extend past their concrete uses. Our concept of justice for instance extends not only to concrete instances that have occurred but to hypotheticals. So too the concepts in our programs extend beyond the ways in which we have in fact employed them. Expecting code to fully define a concept can lead to frustration. Instead, we have to accept our active role in understanding a program.

Given this underdetermination of code in defining the bounds of our concepts, what should we do? This depends on the circumstances we find ourselves in. Are we creators of these concepts, or consumers of them? As creators, our work is by no means easy but straightforward. We are responsible for explaining our concept to others, for persuading others to adopt our approach and to listen and consider feedback, refining our concepts as necessary. We are also responsible for considering if/when our concept has overstayed its welcome; when it is time to rid ourselves of this concept. In other words, we should take full responsibility for this concept.

This notion of responsibility is incredibly important. Our concepts, if used, have effects on others. They can enable, or constrain, they can free or restrict, they can bring joy or pain. Responsibility here doesn't mean that all reactions people have towards our concepts are our fault. Perhaps people have taken our concepts and misused or abused them. But it does mean that we ought to consider closely the potential consequences of our ideas. It further suggests that we need to consider how we will receive feedback about the effects our concepts have had on others. Here it might be useful to think about the enterprise architect whose position is to make decisions around concepts for others, impose them, and yet never have to live with their effects.

Consuming Concepts

Being the consumer of a concept is often much harder than the creator. As creators we have certain leeway, we know the boundaries, we know where edges are vague and where they are exact. We are intimately familiar with our concepts. As consumers, we are not affording these luxuries. Instead, we are encountering a concept and trying to employ it to meet our purposes, often not knowing if this is or isn't a good idea.

Understanding of the norms of the person/community that has created this concept can be incredibly important for successful deployment. Does this concept have a canonical definition? Are there "laws" that apply the concept? What is the conceptual background from which this concept was born? Is there a family of related concepts to which this concept belongs? These sorts of questions will help us ensure we do not misuse the concept.

Given all these constraints, how to proceed is far from straightforward and often requires careful consideration and a good amount of taste. What do we do when we find a mismatch between our purpose and the current bounds of a concept? Should we choose some different existing concept, create a wholly new concept, work to extend the bounds of the concept, or change our purpose? Each of these responses (and more besides) can be the correct move to take. When the mismatch between our purposes and our candidate concepts is serious and irreconcilable, it is often the correct course of action to search for a different existing concept. But knowing when something is irreconcilable is the trick.

Responding to Change

As for our other options, we have briefly discussed creating above so we won't add more here. That leaves expansion of the concept and revising of our purposes. Let's begin with the latter. Changing our purposes seems like a rather odd decision to make in light of finding a mismatch between our purposes and some existing concept. Yet, it can often be the right move if we realize that our purpose wasn't independently chosen, but was actually chosen in part because of our misunderstanding of the concept at hand. Imagine we are looking to change some software to meet a business need. We decide to split some part into two by applying a given existing concept in our codebase. So, our immediate purpose is to split something, that was unified, into two parts, but why did we choose to do this? Because of our understanding of the codebase, including our understanding of the concept. When we discover the mismatch, we ought to abandon our immediate purpose and keep in mind our ultimate purpose here, meeting the business need.

This process of updating our purposes as our understanding of concepts expands is incredibly important. But so too, is it important to consider the expansion of our concepts as our purposes change. Our concepts are under-determined by their instances, but more than that their boundaries are fuzzy in many dimensions. Over time we ought to reconsider these edges in light of our present purposes and knowledge. Introducing new concepts is an expensive task; expanding an existing one can be advantageous. When extending an existing concept, we can rely on existing ideas about the concept to help us transfer knowledge to our new extension making the learning curve much less steep. A fantastic example of this is the way in which video game level design can help us discover a new use for an old mechanic that we never had considered before.

Learning from Others

The framework here is rather abstract, and yet I doubt unfamiliar to programmers who have found themselves working in codebases developed by many people over a long period of time. A major part of programming is not just the writing of code, but the understanding, creating, and modifying of concepts. We often find ourselves frustrated in codebases where the various dimensions of concepts have not been properly attended to.

There is more to be said about many of the things mentioned here. For example, how can we understand a concept by reading code and whatever documentation might exist when these things undetermined the concept at hand? How can understanding the norms of a community or the intentions of an author help us here? How can we be sure we are doing justice to a concept in our extension of it? All of these questions and more deserve to be explored. The first step to doing so is to recognize Conceptual Engineering as a crucial part of what we do when engineering a codebase. Creating a codebase is not merely the creation of a bunch of textual artifacts. The norms and intentions we impose on this text are a crucial part of it. The concepts we create, reuse, or modify are incredibly important. If we want to be better programmers, we need to attend to our concepts and consider the question, what should this concept be.

Most importantly though, we aren't in this alone. While our purposes may be particular to our craft, the process is not. We can learn from the philosophers who have explored Conceptual Engineering directly. But we can also share our own experiences, helping to bring our own perspective out into the broader world.

Building Meander in Meander

Meander has been (in my personal opinion) a wonderful success. With meander.epsilon, we can express most of the data transformations we are interested in. There are of course a few rough edges, a few things we'd change. But as more people have begun to use meander and more people present problems they are tackling with meander, it becomes clear that this approach is working. And yet, there is something that isn't working quite the way we'd like; the implementation of the meander compiler itself.

This isn't meant as a diss on the code. Ultimately the organization of the code is actually really nice. There are clear, defined boundaries, there is a clear separation of functionality, the code itself isn't a mess by any standard. Nor is this a diss on the quality of code generated by the meander compiler. There are certainly areas that we could improve, but in general, meander produces code that is fast and small. In all the meander matches we've written, we have never once encountered the "method code too large" error that has plagued complex pattern matches when using libraries like core.match.

But there is still something not right with the meander.epsilon compiler. As you dive into the code base and try to make modifications, it becomes hard to trace the way data is being transformed. The code is littered with if statements needed to inspect the structure of the data we are getting. Then, we have to pull out all the bits and parts we care about. What data is and isn't available at a given point is far from clear. But perhaps even more importantly, the shape of our data is lost. The meander.epsilon compiler is just converting between data structures, we read matches in as data, parse them as an ast, build a matrix-based IR, build a more direct IR, do optimizations, and deduplication, and then generate clojure code (also data). But looking at our compiler all of this is completely opaque, despite our best efforts.

meander.zeta

In meander.zeta we are taking a different approach. All the details haven't been worked out yet, but in this article, I want to share the general approach so that others can understand what we are looking to accomplish. To do that, we are going to build our own mini-meander compiler using meander.epsilon. Our compiler will not be efficient or support many matches. But it will give you a taste of what meander can do, as well as give a taste of how we are using meander to implement itself.

Before we get started though, let's talk about our plan. First to keep our code clear and this article from stretching out forever, we are going to limit our feature set to matching on logic-variables and vectors. For our purposes that should be all we need. Further, we are only going to be implementing the "match" side of meander. Substitution is left as an exercise to the reader. To accomplish all of this clearly, we will start by first implementing a parser. Then taking our AST (abstract syntax tree) produced from our parser, we will implement a simple meander interpreter. Finally, we will show how meander's symbolic nature allows us to easily transform this interpreter into a compiler, with minimal changes. Let's begin.

Parser

Our parser is going to mirror the format used by the meander parser. I will deviate a bit, but the general approach will be the same, so you will begin to see a bit of what the internals of meander looks like. But rather than build the parser first, let's describe the output we would like from our parser.

(parse '?x)
;; =>
{:tag :logic-variable :symbol '?x}

(parse '[?x ?y])
;; =>
{:tag :vector
 :sequence [{:tag :logic-variable :symbol '?x}
            {:tag :logic-variable :symbol '?y}]}

Here we have two very simple examples of the input and output we expect from our parser. Our goal is to take our pattern and turn it into these nice, unambiguous maps. These maps will always have a :tag value and then any other keys they need to record the information our interpreter or compiler might want. So, let's start by writing a parser that can only handle logic-variables, then will figure out how to deal with vectors.

(defn parse [expr]
  (m/rewrite expr
    (m/symbol _ (m/re #"^?.+") :as ?symbol)
    {:tag :logic-variable
     :symbol ?symbol}))

(parse '?x)
;; =>
{:tag :logic-variable :symbol '?x}

(parse '?y)
;; =>
{:tag :logic-variable :symbol '?y}

(parse 'y)
;; =>
nil

This parser is very straightforward. We are taking advantage of meander's symbol and re operators to make sure that we get a symbol whose name starts with a ?. Other than that, we do not match on anything else and so everything else will just return nil. Let's start by trying to extend this to vectors.

(defn parse [expr]
  (m/rewrite expr
    [!xs ...]
    {:tag :vector
     :sequence [(m/cata !xs) ...]}

    (m/symbol _ (m/re #"^?.*") :as ?symbol)
    {:tag :logic-variable
     :symbol ?symbol}))

(parse '[?x ?y])
;; =>
{:tag :vector
 :sequence
 [{:tag :logic-variable, :symbol ?x}
  {:tag :logic-variable, :symbol ?y}]}

Here we pull out all the contents of our vector and use m/cata to recursively parse our input. If you have never used cata you can think of it just like recur but for patterns. With that we have written our parser as far as we need to for our purposes.

Interpreter

Now that we have an ast, we can write a simple interpreter. What our interpreter will do is given an input and a pattern and an environment, we will return an environment with all our logic variables set to some value, or we will return :fail. Rather than try to assemble our interpreter piece by piece, I will begin by showing you the entire thing.

(defn interpret [expr target env]
  (m/match [expr target env]

    [{:tag :logic-variable :symbol ?symbol} ?target ?env]
    (if (contains? ?env ?symbol)
      (if (= ?target (get ?env ?symbol))
        ?env
        :fail)
      (assoc ?env ?symbol ?target))

    [{:tag :vector :sequence ()} ?target ?env]
    ?env

    [{:tag :vector :sequence (?x)} ?target ?env]
    (interpret ?x (nth ?target 0) ?env)

    [{:tag :vector :sequence (?x & ?rest)} ?target ?env]
    (interpret {:tag :vector :sequence ?rest}
               (subvec ?target 1)
               (interpret ?x (nth ?target 0) ?env))))

(interpret (parse '[?x ?y]) [1 2] {})

;; =>
{'?x 1 '?y 2}

If you've written an interpreter before this shouldn't be too surprising. First we handle logic variables by looking them up in the environment. We handle the cases of the logic variable existing in the environment and matching, it existing and not matching, and it not existing. Next we handle some vectors cases. Here we handle the empty case, the single element case, and the case with more than one element. This interpreter does work for the input we've given it. But think about what would happen if we did the same pattern but just passed a single number? We'd throw an error because we never actually check that our input is a vector. We could just go and add a vector check to all of our vector cases, but that means we will be checking that something is a vector for every single element of our vector. So let's try a different approach.

(defn interpret [expr target env]
  (m/match [expr target env]

    [{:tag :logic-variable :symbol ?symbol} ?target ?env]
    (if (contains? ?env ?symbol)
      (if (= ?target (get ?env ?symbol))
        ?env
        :fail)
      (assoc ?env ?symbol ?target))

    ;; Ensure target is a vector
    [{:tag :vector :checked nil :sequence ?sequence} ?target ?env]
    (if (vector? ?target)
      (interpret {:tag :vector :checked true :sequence ?sequence} ?target ?env)
      :fail)

    [{:tag :vector :sequence ()} ?target ?env]
    ?env

    [{:tag :vector :sequence (?x)} ?target ?env]
    (interpret ?x (nth ?target 0) ?env)

    [{:tag :vector :checked ?checked :sequence (?x & ?rest)} ?target ?env]
    (interpret {:tag :vector :checked ?checked :sequence ?rest}
               (subvec ?target 1)
               (interpret ?x (nth ?target 0) ?env))))

Taking advantage of the fact that meander matches are ordered, we added an earlier match that will perform the check for us, and then when we recurse we simply set checked to true. That means, this pattern will no longer match and we can continue with the interpreter as before. There is still a problem with this interpreter that we aren't going to fix in this post, it does not check the size of the vector. For our purposes, doing this would be fairly easy, we check the size of ?sequence and ensure target has the same size. But what would we do if we added repeats (e.g. ..., ..1 etc)? For now, we will leave this off, but this might be a good exercise for thinking about on your own.

Now that we have a working interpreter, let's look at how we can make this a compiler. Doing so with meander will actually be suprisingly easy.

Compiler

Our transition from interpreter to compiler will be simpler than any I have seen before. The translation is basically mechanical. Here is our first version of our compiler.

(defn compile [expr target env]
  (m/rewrite [expr target env]

    [{:tag :logic-variable :symbol ?symbol} ?target ?env]
    (if (contains? ?env ('quote ?symbol))
      (if (= ?target (get ?env ('quote ?symbol)))
        ?env
        :fail)
      (assoc ?env ('quote ?symbol) ?target))

    ;; Ensure target is a vector
    [{:tag :vector :checked nil :sequence ?sequence} ?target ?env]
    (if (vector? ?target)
      (m/cata [{:tag :vector :checked true :sequence ?sequence} ?target ?env])
      :fail)

    [{:tag :vector :sequence ()} ?target ?env]
    ?env

    [{:tag :vector :sequence (?x)} ?target ?env]
    (m/cata [?x (nth ?target 0) ?env])

    [{:tag :vector :checked ?checked :sequence (?x & ?rest)} ?target ?env]
    (m/cata [{:tag :vector :checked ?checked :sequence ?rest}
             (subvec ?target 1)
             (m/cata [?x (nth ?target 0) ?env])])))

At first glance, it might be quite hard to spot the difference. There are only a few differences here. First and most crucially is that we have changed from m/match to m/rewrite. So now instead of our right-hand side being code that will be immediately run, it is actually data that we are outputting, in this case, that data is code. Secondly, we have quoted some of our symbols. Because we will ultimately be outputting this code in a macro and looking up symbols in an environment, they need to be quoted. Finally we have changed from clojure recursive function calls to using meanders recursion operator m/cata. These are our only real changes and given that, we can now make a macro for matching that will compile our pattern.

(defmacro match [target expr]
    (let [target_sym (gensym "target_")
          env_sym (gensym "env_")]
      `(let [~target_sym ~target
             ~env_sym {}]
         ~(compile (parse expr) target_sym env_sym))))

(match [1 2 1] [?x ?y ?x])
;; => {?x 1 ?y 2}
(match [1 2 2] [?x ?y ?x])
;; => :fail

No longer is there a runtime cost to parsing our pattern and then interpretive overhead for crawling through the ast deciding what code to run. All of this happens in our macroexpansion. But there is one small problem. If we look at the code generated by this compiler, we will see that it is rather repetitive and long for what it does. This occurs because of the following clause above:

[{:tag :vector :checked ?checked :sequence (?x & ?rest)} ?target ?env]
(m/cata [{:tag :vector :checked ?checked :sequence ?rest}
         (subvec ?target 1)
         (m/cata [?x (nth ?target 0) ?env])])

Line 4 here is the culprit for our explosion of code. Ultimately our compilation returns us an expression that returns an environment. We build up this environment as we go through our vector. Line 4 allows us to do exactly that, expand into some code that will update our environment. But it does this over and over again. As we continue our compilation our ?env becomes more and more branching code. Luckily there is a pretty simple fix for this.

(m/and [{:tag :vector :checked ?checked :sequence (?x & ?rest)} ?target ?env]
       (m/let [?env-sym (gensym "_env_")]))
     (let [?env-sym (m/cata [?x (nth ?target 0) ?env])]
       (m/cata [{:tag :vector :checked ?checked :sequence ?rest}
                (subvec ?target 1)
                ?env-sym]))

Here is our updated code that no longer creates a huge compilation output. Rather than directly updating our ?env by embedding more and more code. We make a new symbol that will store our environment and pass that down through our recursion. Now that we have solved that problem we end up with some fairly reasonable generated code.

(let
  [target_19362 [1 2 1] env_19363 {}]
  (if (vector? target_19362)
    (let*
      [_env_19364
       (if (contains? env_19363 '?x)
         (if (= (nth target_19362 0) (get env_19363 '?x))
           env_19363
           :fail)
         (assoc env_19363 '?x (nth target_19362 0)))]
      (let
        [_env_19365
         (if (contains? _env_19364 '?y)
           (if (= (nth (subvec target_19362 1) 0) (get _env_19364 '?y))
             _env_19364
             :fail)
           (assoc _env_19364 '?y (nth (subvec target_19362 1) 0)))]
        (if (contains? _env_19365 '?x)
          (if (=
                (nth (subvec (subvec target_19362 1) 1) 0)
                (get _env_19365 '?x))
            _env_19365
            :fail)
          (assoc
            _env_19365
            '?x
            (nth (subvec (subvec target_19362 1) 1) 0)))))
    :fail))

Admittedly this is still quite a bit of code for what we are doing. If you look at it for even a moment you can see some issues. We definitely run nth and subvec entirely too many times. But once we look back at code it becomes pretty obvious that the gensym trick we used before could easy solve that problem. But there is also something still a bit unsatisfying about this generated code. Shouldn't it just be simpler? We know that ?y is only assigned once so why check if it is in the environment or not yet? We also know that the first thing we match on ?x will always succeed. Why are we checking there as well? For the case we are looking at now, we know that our input is a vector and in fact, we know exactly what our output should be at compile because we were passed a literal!

These sorts of optimizations are completely possible with this framework. We don't have the space to fully explore them, but I will just give a general flavor. What if during compile time we also kept a compile-time env of all the things we know? So we know that our input is a vector, so why check that at run time? We know exactly which logic-variables have been bound or not, so ?x can just be directly assigned to (nth ?target 0) right away. Hopefully, you can see that there is nothing about this approach that stops us from making these sorts of optimizations in the future.

Conclusion

I hope that from this post you learned how meander can be incredibly useful when building out a compiler in clojure. Its direct, symbolic, pattern matching approach simplifies a great deal of what goes into writing a compiler for your own customer dsl. It gives you clear and concise code that lets you reason about your cases. As we've built out zeta we've found meander's structured approach to help us understand our code and give us a clear sense of what to do next. Give this technique a try the next time you have a dsl in mind. Why settle for an interpreter when meander makes it this easy to write a compiler?

Card Driven Development

Every company I've worked for has practiced what I call "Card Driven Development" (CDD). CDD is an umbrella term meant to encompass quite a few different development processes. The name comes from the use "cards" (digital or physical) found in the Scrum and Kanban processes as a means of organizing work, but CDD extends further than these two processes. It encompasses formal or informal processes as long as they have a few things in common.

CDD is a way of approaching software development that involves a splitting of responsibilities. In CDD, there must be at least two groups, the first, often called "Engineering", is responsible for technical duties including writing code, maintaining code, and operational concerns like managing servers. The second group, often known as "Product" or informally "the business", is responsible for researching, detailing, and prioritizing what work needs to be done. Product gives work to Engineering, they prioritize what work will be done next, and they message out the status of the work to various interested parties.

This split in responsibility means that the day-to-day work accomplished by Engineering and Product looks very different. Engineering, when operating well, is about shipping working software to customers, whereas Product's work is an act of delegation. This can often lead engineers to believe that Product does nothing, but this could not be further from the truth. Product's days are packed with work. Product's role involves being pulled in many directions, they must keep track of the work in progress, while continuously engaging with the various interested parties vying for their attention; attempting to get their party's particular needs worked on next. Product's work involves status updates, meetings, research, and above all else detailing and prioritizing work for Engineering.

The Card

This last part is where the Card makes its appearance. The work needed to be done is too large to write down and detail in its entirety. Even if this task could be achieved, it isn't desirable because by the time it was written down, the facts on the ground may have changed. So a system of dividing work is adopted. In CDD this system is the card.

Cards are a place to describe work that needs to be done and in what order it should be done. Cards break work into small chunks, typically focused on functionality that some particular party would like. Cards serve as a meeting point for Product and Engineering. Product writes cards, prioritizes cards, and gives them to Engineering as work that needs to be done. Engineering then works in the order given by product. Sometimes this means taking multiple cards at a time and making a time commitment for their completion (often called a sprint). Other times cards are pulled in one by one as time and resources permit (this process is often referred to as a Kanban-style workflow).

As described here the card-making process only involves Product. In reality, there is usually a collaboration between Product and Engineering around cards. Typically cards are reviewed and discussed before they ever are handed to an Engineer to be worked on. Some of these cards may be "technical cards", cards created and owned by Engineering. With the exception of these technical cards though, cards are owned by Product. Product has the final say in the priority of cards and also serves as the "acceptor" for the fact that the work described in cards has been accomplished.

The above is meant to serve as a neutral description of what is entailed by CDD. I imagine the process as I've described above is familiar to most reading this. CDD is a process that many companies adopt and do so for good reason. But despite the best of intentions by companies that adopt CDD, it is a disastrous process that leads to some of the worst issues plaguing modern software development. I contend that companies that adopt CDD will inevitably exacerbate these issues, leading to engineer dissatisfaction and decreased productivity.

A Bit About Creating Software

Creating software is a complex task. Features, that may seem on the surface trivial may involve multiple months of effort, minor modifications to an existing process may cause a cascade of required changes across many codebases, and seemingly innocent changes may introduce pernicious bugs. While there are most certainly ways to mitigate these problems, this is the reality the Software Engineer faces. The development of software does not exist in a vacuum, our new code must continue to work with old code, our legacy systems require continual maintenance and the interactions between pieces of our system continue to grow in complexity.

Any view about how creating software ought to be done needs a realistic picture of the constraints that Software Engineers face. The software creation process at any given time is constrained by all the decisions of past engineers who had a hand in building the software in question. But this set of decisions involves not only all engineers who have worked at the company, but also the authors of the libraries on which the software is built. The design decisions of past engineers provide deep constraints on how much effort a given change will take. The choice of a relational or non-relational datastore limits the sort of data-integrity constraints we can easily enforce, the library we have chosen for our frontend constrains our ability to support things like server-side rendering that may be needed for SSO considerations, and the decision of how to split (or not) our application into services constrains our deployment and delivery strategy, just to name a few examples.

The Problem with Card Driven Development

Card Driven Development, as I have defined it, is concerned primarily with a division of labor. But with this division of labor comes a division of knowledge. Engineering knows the software systems needed for the business to operate and Product has knowledge of the various interested parties' desires. In themselves, these divisions aren't an issue, but as I hope to show the process of CDD limits the sharing of knowledge across these domains, causing issues for both the software being created and the larger goals of the business.

What precisely do I mean by CDD limiting knowledge? As I'm using this term it implies at least two things. First that the knowledge that is shared between Product and Engineering happens at the wrong level and in the wrong order, limiting the usefulness of the knowledge transfer. Second, that the sorts of knowledge shared between Product and Engineering allow neither to accomplish their job as well as they ought. The reciprocal nature of this issue is incredibly important. CDD isn't just bad for Engineering, it is bad for Product and bad for the business.

Wrong Level, Wrong Order

Imagine you are a novice cook looking to tackle a recipe a bit above your skill level. You follow the instructions step by step diligently, spending close to an hour on prepping and cooking the ingredients and as you come to the end of the recipe it says, "To finish the dish, poach an egg and place it on top". You immediately become frustrated. While the recipe is telling you everything needed to make the dish, you have no idea how to poach an egg and even if you figure it out, poaching takes time and your dish is already complete.

Unlike our cooking example, CDD gives engineers knowledge at a far too granular level. Cards are about tiny pieces of functionality. They don't provide the context needed to place a given piece of functionality into its larger context. This focus on granularity can cause the big picture to be lost. It isn't uncommon to implement an entire card based on the details it provided just to find out that it didn't actually solve the problem it was intended to. Had the card been written at a higher level, the constraints on what counted as a solution to the problem would have been clearer.

Not only is the information too granular, but it is also often delivered out of order. It isn't uncommon to work on 2 or 3 cards, make great progress, and then once you reach the next card you find out the work you just implemented was wrong. What you learned on card 4 was the information you really needed at the beginning. This new feature or constraint casts all previous work in a different light. The assumptions you made when implementing the others cards are now invalid. This means you have one of two choices, rework the previous work, or write a hack. I think we all know what happens most often.

Wrong Sorts of Knowledge

Cards detail functional requirements of our software. They are more often than not about the inputs and outputs of our systems. They detail UI components, data requirements, user interactions, etc... And yet these are not what Engineering needs to do its job most effectively. The businesses we create are not about these inputs and outputs. Ultimately as businesses, we are trying to solve problems. Our users face issues that we are providing a service to solve. These users may be our internal customer support, other companies, or consumers, regardless, they have a problem and we are tasked with fixing it.

For these problems to be fixed software needs to be written by Engineering. How can Engineering know if their software is effective? Or in other words, how can Engineering know if their software solves the users' problem? By first understanding at a deep level what problem the user is facing. This is truly the fundamental problem with CDD, it distances engineers from the problems they are solving. No longer do engineers know or care deeply about these problems. No longer do they spend their time thinking about these problems and how best to solve them. Instead, they are consumed by details, consumed by sub-optimal answers.

But this isn't just a problem for Engineering. Product has burdened themselves unnecessarily by adopting CDD. Product's job is to understand the user, but there is never just one user. Product has to understand all these various users with vastly different needs. This is a hard enough task on its own, but with CDD understanding the users is a small part. Instead, Product must spend their time trying to fit these problems into the system that Engineering has developed. They must figure out a way to address the problems the user is facing in a system that is largely opaque to them.

The opaque nature of the system that Engineering has built is a constant frustration for Product. Constantly their best attempts at making small changes are thwarted. Even the most reasonable-sounding request gets push back from Engineering. It becomes harder and harder for product to make the changes they are requesting. Ultimately, in CDD Product is being asked to do something nearly impossible, propose changes to a system that is largely invisible to them. Try as they might, knowing the difference between what changes can easily be made to the system and what are hard are completely opaque to Product, and yet their job depends on knowing exactly that.

Bad for the Business

Engineers who lack understanding of the users' problems may solve the wrong problems. This might mean shipping a feature that no one uses. It might mean removing some important feature. It may mean leaving out a feature that was sorely needed. All of these problems can drive customers away or prevent a company from gaining more customers. If the users of the system are internal, it may cause inefficiencies that cost the company money and can even affect employee morale.

If Product has been tasked with designing new features and lacks the knowledge needed to do so effectively, inefficiencies in the development cycle can appear. As Product proposes more changes, estimates grow. More time is spent in meetings discussing changes than actual time implementing them. As development slows, demands for changes continue to grow. Users talking to Product get increasingly frustrated as to why their changes haven't made it in. More often than not, this means changes to internal tools are deprioritized leading to inefficiencies and frustrating process that can lead to high turnover in customer support.

Strained Relationships and Failure

This lack of knowledge on both sides can lead to friction between Product and Engineering. It is incredibly common for companies to have a strained relationship between Engineering and Product. I think many of these issues stem from this confusion of roles. Both sides feel powerless to achieve the ends they have been tasked with. Over time their jobs get harder and harder and it always seems to be the other group's fault.

A strained Product and Engineering relationship can spell the end for a project or even a whole company. More often than not, strained product relationship leads to frustrated engineers. These engineers leave, taking with them their hard-earned knowledge of the system they helped build. As engineers become frustrated, they often become defensive. When asked for estimates they worry about how the estimates might be used and so they pad them. Knowing that anyone can detect if they constantly over-estimate, they slow their work, meeting nearly exactly the estimates they put forward.

Development time stretches on and on. The system the company has invested so much in grows worse and worse to make changes to. All the original engineers have left. Product is sick and tired of trying to force changes through. Eventually, the business decides to do the big rewrite and the cycle repeats. If we are ever to break this cycle, we must get rid of card-driven development. What I think should replace it is a topic for another post. But whatever the right process is, it must start with an understanding of what programming is.

Introduction to Term Rewriting with Meander

Meander is heavily inspired by the capabilities of term rewriting languages. But sadly, there aren't many introductions to term rewriting aimed at everyday software engineers. Typically introductions to term rewriting immediately dive into discussing mathematical properties or proving theorems. These can be interesting and useful in their own right. But personally, I like to get an intuitive feel for something before diving into a formalism. That is the aim of this post, to help you have a more intuitive understanding of how Term Rewriting works and what it is capable of. This post will not focus on practical uses of meander, if you are interested in that check out Meander for Practical Data Transformation.

The Basics

The goal of Term Rewriting is to take some bit of data and rewrite it into some other bit of data. We accomplish this by writing rules that tell us for a given piece of data what we should turn it into.

(require '[meander.strategy.epsilon :as r]
         '[meander.epsilon :as m])

(def x-to-y
  (r/rewrite
   :x :y))

(x-to-y :x)
;; => :y

Here is the most simple rewrite rule imaginable. If we are given :x we turn it into :y. In term rewriting, the pattern we are using to match is often called the left-hand-side and the data we return is called the right-hand-side. So :x is our left-hand-side and :y is our right-hand-side. The data we pass in to transform is called the reducible-expression (or redex for short).

Admittedly, this seems almost useless, and it really is with this overly simplistic example. But let's take it slow and build it up.

(def rewrite-some-keywords
  (r/rewrite
   :x :y
   :l :q
   :r :t
   :a :c))

(rewrite-some-keywords :a)
;; => :c

Here we've extended our rewrite to have multiple rules. Now we can handle more than just :x. Of course, this is still really limiting. We definitely can't list every single possible input for all of our rules. We need a way to match any input. That is where variables come in.

(def match-any-thing
  (r/rewrite
    ?x [:matched ?x]))

(match-any-thing :a) ;; [:matched :a]
(match-any-thing "hello") ;; [:matched "hello"]
(match-any-thing 1) ;; [:matched 1]

Here we added the variable ?x to our left-hand-side. Variables start with a ? and match any value. Whatever they match is now accessible on the right-hand-side. So we can match anything with ?x and then use it in our output. Let's see a more interesting example.

(def find-x
  (r/rewrite
   [?x] ?x
   [?x ?y] ?x
   [?x ?y ?z] ?x))

(find-x [1]) ;; 1
(find-x [1 2]) ;; 1
(find-x [1 2 3]) ;; 1

Here we can see some really simple rules that work on vectors of various sizes. We can use this to extract the first element from each. In this case, since we only care about ?x, we can actually simplify this code.

(def find-x
  (r/rewrite
   [?x] ?x
   [?x _] ?x
   [?x _ _] ?x))

The _ is a wildcard match that matches anything but doesn't bind at all. What happens if we try to extend this to work for not just vectors, but just a single number?

(def find-x
  (r/rewrite
   ?x ?x
   [?x] ?x
   [?x _] ?x
   [?x _ _] ?x))

(find-x 1) ;; 1
(find-x [1]) ;; [1]

The order of our rules matters, ?x matches anything, so we will always get the first match. We could change the order, or we can constrain the match.

(def find-x  
  (r/rewrite
   (m/pred number? ?x) ?x 
   [?x] ?x
   [?x _] ?x
   [?x _ _] ?x))

(find-x 1) ;; 1
(find-x [1]) ;; 1

Okay, now it works. But many of you are probably thinking "Isn't this just pattern matching?". And in many ways it is. Term Rewriting is a kind of pattern matching. But it doesn't stop with simple pattern matching. Term Rewriting is a way to do all computation through pattern matching. To see that, let's move beyond the basics.

Applying strategies

We've seen that with Meander we can do simple rewrites where we match on the left-hand-side and output a right-hand-side. But just being able to do a single rewrite in this way is really limiting. To see this problem let's consider a classic example in term rewriting.

(def simplify-addition
  (r/rewrite
   (+ ?x 0) ?x
   (+ 0 ?x) ?x))

(simplify-addition '(+ 0 3)) ;; 3
(simplify-addition '(+ 3 0)) ;; 3

Zero added to anything is just that thing. We can easily express this with term rewriting. But what if we have multiple 0's nested?

(simplify-addition '(+ 0 (+ 0 3))) ;; (+ 0 3)

(simplify-addition
 (simplify-addition '(+ 0 (+ 0 3)))) ;; 3

As you can see, the first time we apply our rules we do simplify, but not all the way. If we call our rules again, we fully simplify the expression. But how could we express this with term rewriting? We can use what are called strategies. Strategies let us control how our terms are rewritten. Let's start with an easy strategy the n-times strategy.

(def simplify-twice
  (r/n-times 2 simplify-addition))

(simplify-twice '(+ 0 (+ 0 3))) ;; 3

Strategies wrap our rewriting rules and make them do additional things. In this case, the rewriting will be applied twice. But there are a few problems with the strategy as we've written it. Let's slowly discover those problems together and fix them.

(simplify-twice '(+ 0 3)) ;; #meander.epsilon/fail[]

Our apply-twice strategy works for things that need to be simplified twice, but not for simple cases. We can fix that by using the attempt strategy. It will try to rewrite and if it fails, just return our value.

(def simplify-addition
  (r/n-times 2
    (r/attempt
     (r/rewrite
      (+ ?x 0) ?x
      (+ 0 ?x) ?x))))

(simplify-addition '(+ 0 3)) ;; 3
(simplify-addition '(+ 0 (+ 0 3))) ;; 3
(simplify-addition '(+ (+ 0 (+ 0 3)) 0)) ;; (+ 0 3)

Now it works for both. But having it only rewrite twice is a little arbitrary. What we really want to say is to continue applying our rewrite rules until nothing changes. We can do that by using the (until =) strategy.

(def simplify-addition
  (r/until =
    (r/attempt
     (r/rewrite
      (+ ?x 0) ?x
      (+ 0 ?x) ?x))))

(simplify-addition '(+ (+ 0 (+ 0 3)) 0)) ;; 3
(simplify-addition '(+ (+ 0 (+ 0 (+ 3 (+ 2 0)))) 0)) ;; (+ 3 (+ 2 0))

We can now simplify things no matter how deep they are, but as we can see we didn't fully eliminate 0s from all our expressions. Why is that? Well, our pattern only matches things that are in the outermost expression. We don't look at all at the sub-expressions. We can fix that by applying another strategy. In this case, we will use the bottom-up strategy.

(def simplify-addition
  (r/until =
    (r/bottom-up
     (r/attempt
      (r/rewrite
       (+ ?x 0) ?x
       (+ 0 ?x) ?x)))))

(simplify-addition '(+ (+ 0 (+ 0 (+ 3 (+ 2 0)))) 0)) ;; (+ 3 2)

We have now eliminated all the zeros in our additions no matter where they are in the tree. For the sake of space in our examples, we kept our rules and our strategies together, but these are actually separable. What if we wanted to try the top-down strategy with our rules?

(def simplify-addition
  (r/rewrite
   (+ ?x 0) ?x
   (+ 0 ?x) ?x))

(def simplify-addition-bu
  (r/until =
    (r/bottom-up
     (r/attempt simplify-addition))))

(def simplify-addition-td
  (r/until =
    (r/top-down
     (r/attempt simplify-addition))))

Our rules are completely separate from how we want to apply them. When writing our transformations, we don't have to think at all about the context they live in. We just express our simple rules and later we can apply strategies to them. But what if we want to understand what these strategies are doing? After playing around with things, it seems that the top-down strategy and the bottom-up strategy always give us the same result. But what are they doing that is different? We can inspect our strategies at any point by using the trace strategy.

(def simplify-addition-bu
  (r/until =
    (r/trace
     (r/bottom-up
      (r/attempt simplify-addition)))))

(def simplify-addition-td
  (r/until =
    (r/trace
     (r/top-down
      (r/attempt simplify-addition)))))

So now we have modified our rewrites to trace every time the top-down or bottom-up rules are called. Let's try a fairly complicated expression and see what happens.

(simplify-addition-td '(+ (+ (+ 0 3) (+ 0 (+ 0 (+ 2 0)))) 0)

;; printed
{:id t_20100, :in (+ (+ (+ 0 3) (+ 0 (+ 0 (+ 2 0)))) 0)}
{:id t_20100, :out (+ 3 (+ 0 2))}
{:id t_20100, :in (+ 3 (+ 0 2))}
{:id t_20100, :out (+ 3 2)}
{:id t_20100, :in (+ 3 2)}
{:id t_20100, :out (+ 3 2)}


(simplify-addition-bu '(+ (+ (+ 0 3) (+ 0 (+ 0 (+ 2 0)))) 0)

;;printed
{:id t_20099, :in (+ (+ (+ 0 3) (+ 0 (+ 0 (+ 2 0)))) 0)}
{:id t_20099, :out (+ 3 2)}
{:id t_20099, :in (+ 3 2)}
{:id t_20099, :out (+ 3 2)}

If we look at the top-down approach, we can see that the top-down strategy actually gets called three times. Once it rewrites quite a bit but leaves in a 0 that needs to be rewritten. Then it gets called again, eliminating all zeros. Finally, it is called and nothing changes. Our bottom-up strategy however is only called twice. But we can actually get more fine-grained than this. We can put trace at any point in our strategies.

(def simplify-addition-bu
  (r/until =
    (r/bottom-up
     (r/trace
      (r/attempt simplify-addition)))))

(simplify-addition-bu '(+ (+ 0 3) 0))

;; printed
{:id t_27317, :in +}
{:id t_27317, :out +}
{:id t_27317, :in +}
{:id t_27317, :out +}
{:id t_27317, :in 0}
{:id t_27317, :out 0}
{:id t_27317, :in 3}
{:id t_27317, :out 3}
{:id t_27317, :in (+ 0 3)}
{:id t_27317, :out 3}
{:id t_27317, :in 0}
{:id t_27317, :out 0}
{:id t_27317, :in (+ 3 0)}
{:id t_27317, :out 3}
{:id t_27317, :in 3}
{:id t_27317, :out 3}

Here we moved our trace down outside our attempt strategy. Now we can see the exact order of our bottom-up strategy. Having this sort of visibility into how the process is working is really fantastic.

Rewriting as General Computation

What have been doing so far is interesting, but it falls short of the true power of term rewriting. Term rewriting is a general programming technique. Using it we can compute absolutely anything that is computable. Let's start with a classic example, fibonacci, but to further show general computability, we will make our own numbers instead of relying on Clojure's.

(def fib-rules
  (r/rewrite

   (+ Z ?n) ?n
   (+ ?n Z) ?n

   (+ ?n (succ ?m)) (+ (succ ?n) ?m)

   (fib Z) Z
   (fib (succ Z)) (succ Z)
   (fib (succ (succ ?n))) (+ (fib (succ ?n)) (fib ?n))))


(def run-fib
  (r/until =
    (r/bottom-up
     (r/attempt fib-rules))))

[(run-fib '(fib Z))
 (run-fib '(fib (succ Z)))
 (run-fib '(fib (succ (succ Z))))
 (run-fib '(fib (succ (succ (succ Z)))))
 (run-fib '(fib (succ (succ (succ (succ Z))))))
 (run-fib '(fib (succ (succ (succ (succ (succ Z)))))))
 (run-fib '(fib (succ (succ (succ (succ (succ (succ Z))))))))]

;; [Z
;;  (succ Z)
;;  (succ Z)
;;  (succ (succ Z))
;;  (succ (succ (succ Z)))
;;  (succ (succ (succ (succ (succ Z)))))
;;  (succ (succ (succ (succ (succ (succ (succ (succ Z))))))))]

If you aren't familiar with defining natural numbers via Peano numbers this may be a little bit confusing. But for our purposes all you need to know is that Z means 0 and succ means successor. (succ Z) means 1 (succ (succ Z)) means 2 and so on and so forth. Our fibonacci rules start by defining addition for our Peano numbers. Anything added to 0 is zero. Otherwise, we can add two numbers by moving all the succs to one side until the right hand side equals 0. With those definitions in place, we can define fibonacci, which is basically just the definition of fibonacci. With term rewriting our strategies can enable us to have recursion without directly implementing it. Our rules read like they are recursive. But our rules don't call a function. They don't cause anything to occur. They just return more data. It is the process of interpretation that makes them recursive.

In fact, with Meander, we are limited to what the clojure reader can interpret, but in general, with term rewriting, the syntax doesn't matter. I wrote things as (fib n) merely as convention. I could have writen (n fib). There is nothing special about the syntax other than what rules we apply to it.

Why Should We Care?

Admittedly the example of fibonacci above isn't very useful. And of course, if we had a real language, we would never want a number system like that. So why should we care about term rewriting? Term Rewriting offers a powerful yet simple way of viewing programming. It gives us the potential to take the lisp mantra that code is data and data is code much more seriously. How so? First, in lisps functions might be values, but they are opaque. Evaluating a function definition returns you something that you can't inspect directly. Something you can't directly transform (#function[] in clojure). With term rewriting, things can just remain data, because we have separated execution from description.

Not only can our "code" be data more than it can in lisp, but we can actually have our execution as data. Executing a Term Rewriting rule is just taking in data, matching on it, and producing more data. That means all our intermediate values are data. The entire execution of our program now becomes data. Have you ever run your program and had no idea where a certain value came from? Well, imagine if you could just ask your language to pattern match on every intermediate value that contains that value. Or maybe, give me the last 5 steps that led to this value. With Term Rewriting this is entirely possible.

Term Rewriting also gives us an easy basis for talking about partial programs. Our current programming languages have a problem where if they encounter something they don't understand, they just blow up, not telling us anything. Let's consider the following program:

(+ 3 4 (unimplemented!))

What does the program return? Well as its name is clear, unimplemented is in fact, unimplemented. So most languages will just throw an error. That can be what we want at times. But as people, we can look at that code and tell something else. We know that it will return (+ 7 something). Why can't our languages tell us that? Why can't we start writing partial programs and run them continually refining things as we go? Term Rewriting gives us this ability.

Term Rewriting as Programming Paradigm

Term Rewriting represents a distinct way of programming. It offers us a uniform way of dealing with data. It gives us the ability to think about things as syntactic structures. It offers us a way to truly have code as data, to go beyond the arbitrary distinctions imposed by our languages about what can and cannot be manipulated. It is a fundamental shift in how we view programs. It gives us new perspectives, new ways of thinking about how code executes and what our programs mean.

Meander isn't at that point. But it is the beginning of an exploration into how to get there. In many ways, Meander is a testament to the flexibility of lisps and Clojure in particular. Using Clojure's rich data literals and macros we can embed our own language inside it. Yet at the same time, Meander pushes us beyond the way we've traditionally conceived of programming. Maybe functions aren't the best abstraction for working with data. Could programming be better if we truly had a way to work with data directly? That is Meander's conviction and its chief aim.

Meander for Practical Data Transformation

As Clojure programmers we love data. We believe that, at its core, programming is just data manipulation. To that end, Clojure provides fantastic data literals for its immutable data structures. Moreover core provides tons of functions for the manipulation of data. But as our data grows more complex, things become difficult. Our beautiful declarative data transformation pipeline becomes a nested mess. We wind up, yet again, playing computer in our heads.

In this tutorial, we are going to build up slowly to understand how Meander can be used to solve practical data transformation problems. We will start with simple examples and move to more complicated ones, hopefully choosing problems that reflect the sorts of menial data transformation tasks we all encounter in our day jobs. Let's start with some vanilla Clojure code.

(def person
  {:name "jimmy"
   :preferred-address
   {:address1 "123 street ave"
    :address2 "apt 2"
    :city "Townville"
    :state "IN"
    :zip "46203"}})

(defn reformat-preferred-address [person]
  (let [address (:preferred-address person)]
    {:address {:line1 (:address1 address)
               :line2 (:address2 address)}
     :city-info {:city (:city address)
                 :state (:state address)
                 :zipcode (:zip address)}}))

Here we have a pretty decent Clojure function that converts between two different address formats. This sort of code is fairly common when we need to convert from the data requirements of one system to another. Honestly, with this simple example, the code is fairly straightforward. Our data requirements are simple and so our code isn't difficult. Let's look at how to accomplish this same task in Meander.

(require '[meander.epsilon :as m])

(defn reformat-preferred-address [person]
  (m/match person
    {:preferred-address
     {:address1 ?address1
      :address2 ?address2
      :city ?city
      :state ?state
      :zip ?zip}}
    
    {:address {:line1 ?address1
               :line2 ?address2}
     :city-info {:city ?city
                 :state ?state
                 :zipcode ?zip}}))

Here is code that does the same thing written with Meander. One obvious thing to note is that the Meander version is much longer. Judging code based on the number of lines is not something we are going to do.

Let's explain what is going on. First, we are using the Meander's match feature. match takes the thing that we are matching on (person), a pattern to try to match, and the output. Our pattern here is in the exact shape of the person map we passed in. In order to extract out pieces of this map, we use logic variables (?address1, ?address2, etc). Logic variables are just symbols that start with ?. We can assign values in our data to any logic variables we'd like and then use those logic variables in our output. One thing I love about this simple Meander example is that you can see the exact shape of the input immediately.

Making Our Example Harder

This example while somewhat realistic is very limited. While I like the fact that Meander's match shows us the shape of our data, for simple examples like this, Clojure does pretty well. Let's make things harder.

(def person
  {:name "jimmy"
   :preferred-address
   {:address1 "123 street ave"
    :address2 "apt 2"
    :city "Townville"
    :state "IN"
    :zip "46203"}
   :other-addresses 
   [{:address1 "432 street ave"
     :address2 "apt 7"
     :city "Cityvillage"
     :state "New York"
     :zip "12345"}
    {:address1 "534 street ave"
     :address2 "apt 5"
     :city "Township"
     :state "IN"
     :zip "46203"}]})

In the example above we left out some things. A person has a preferred address, but they also have other addresses. We have a few different things we want to do with this data. First, we want to find all the distinct zip codes that a person has.

(defn distinct-zip-codes [person]
  (let [preferred-address-zip (get-in person [:preferred-address :zip])
        other-zips (map :zip (:other-addresses person))]
    (distinct (cons preferred-address-zip other-zips))))

Here is some pretty straightforward Clojure code for doing exactly that. I'm sure some people could have minor quibbles about how this is written, but I doubt other solutions would be much different. One thing to note here is that we have lost a little bit of the structure of our input data. We could maybe change that up a bit. Maybe using destructuring is the right approach? Regardless, this is a simple and reasonable Clojure function. Now, let's look at the Meander version.

(defn distinct-zip-codes [person]
  (m/match person
    {:preferred-address {:zip !zips}
     :other-addresses [{:zip !zips} ...]}
    (distinct !zips)))

Here is the exact same function, but we've introduced two new concepts. The first one is memory variables, in this case !zip. Memory variables start with ! and remember all the values they match with. The next concept is the zero or more operator (…). The zero or more operator says to repeat the pattern to its left zero or more times. In this case {:zip !zips}. Using these two, we can declaratively gather up all the zip codes in this data structure.

Minor Modifications

What happens if one of our zip codes is nil? Well for both of our functions, nil gets returned in the output. That is probably not what we want. Let's fix that in both versions.

(defn distinct-zip-codes [person]
  (let [preferred-address-zip (get-in person [:preferred-address :zip])
        other-zips (map :zip (:other-addresses person))]
    (filter some? (distinct (cons preferred-address-zip other-zips)))))

(defn distinct-zip-codes [person]
  (m/match person
    {:preferred-address {:zip (m/or nil !zips)}
     :other-addresses [{:zip (m/or nil !zips)} ...]}
    (distinct !zips)))

These two functions aren't that different. In Meander we could have used filter in the exact same way if we wanted. But it's nice that we can set these conditions on the input, which is really more closely stating our intent. Here we used or a short-circuiting operator which says that we should match one of these patterns. Our first pattern is just the literal nil. If it is nil, the pattern will match, but it won't be saved anywhere. If the value isn't nil, it will be saved in our memory variable !zips. Before we move on to more complex examples, let's consider one more modification. This time we want a distinct list of non-nil zips and cities output in a map like this {:zips [] :cities []}.

(defn distinct-zips-and-cities [person]
  (let [preferred-address-zip (get-in person [:preferred-address :zip])
        preferred-address-city (get-in person [:preferred-address :city])
        other-zips (map :zip (:other-addresses person))
        other-cities (map :city (:other-addresses person))]
    {:zips (filter some? (distinct (cons preferred-address-zip other-zips)))
     :cities (filter some? (distinct (cons preferred-address-city other-cities)))}))

(defn distinct-zips-and-cities [person]
  (m/match person
    {:preferred-address {:zip (m/or nil !zips)
                         :city (m/or nil !cities)}
     :other-addresses [{:zip (m/or nil !zips)
                        :city (m/or nil !cities)} ...]}
    {:zips (distinct !zips)
     :cities (distinct !cities)}))

With both of these examples, I extended them in the most obvious way I could think of. I think the Meander held up pretty well, but I wouldn't have written the plain Clojure function that way. Here's what I probably would have done instead.

(defn distinct-zips-and-cities [person]
  (let [addresses (cons (:preferred-address person) 
                        (:other-addresses person))]
    {:zips (filter some? (distinct (map :zip addresses)))
     :cities (filter some? (distinct (map :city addresses)))}))

I think this is a pretty good function. But what I find interesting is that I needed to refactor to get here. It took me a little bit to think this way. Ideally, small changes to output should not require us to restructure our code. In this case, the change is minor. But if we have to change our structure in such small cases, won't we have to change it even more in larger cases?

Searching with Meander

All our examples up until this point have had one answer. Yes, that answer might have been a collection, but there was only one way for our pattern to match. This isn't always the case. To see an example of that, let's write some functions using this data structure.

(def people
  [{:name "jimmy"
    :addresses [{:address1 "123 street ave"
                 :address2 "apt 2"
                 :city "Townville"
                 :state "IN"
                 :zip "46203"
                 :preferred true}
                {:address1 "534 street ave",
                 :address2 "apt 5",
                 :city "Township",
                 :state "IN",
                 :zip "46203"
                 :preferred false}
                {:address1 "543 Other St",
                 :address2 "apt 50",
                 :city "Town",
                 :state "CA",
                 :zip "86753"
                 :preferred false}]}
   {:name "joel"
    :addresses [{:address1 "2026 park ave"
                 :address2 "apt 200"
                 :city "Town"
                 :state "CA"
                 :zip "86753"
                 :preferred true}]}])

I apologize for the amount of room this takes up on the screen, but real-world examples are much larger. I want to try and make something that approaches realistic and to do that our input needs to be a bit bigger. Okay, so what we want to do now is given a zip code, find all people that have an address with that zip code, and for each of the addresses that match that zip code, return a map of{:name <name> :address <address>}. So in this case, if we asked for zip 86753 we should get the following response:

({:name "jimmy",
  :address
  {:address1 "543 Other St",
   :address2 "apt 50",
   :city "Town",
   :state "CA",
   :zip "86753",
   :preferred false}}
 {:name "joel",
  :address
  {:address1 "2026 park ave",
   :address2 "apt 200",
   :city "Town",
   :state "CA",
   :zip "86753",
   :preferred true}})

Okay let's start with the vanilla Clojure example.

(defn find-people-with-zip [people zip]
  (for [person people
        address (:addresses person)
        :when (= (:zip address) zip)]
    {:name (:name person)
     :address address}))

This code might not be very idiomatic. I almost never use for in actual code. But honestly, this was the most succinct way I could think to write it. We could also have written something like this:

(defn person-with-address-comb [person]
  (map (fn [address]
         {:name (:name person)
          :address address})
       (:addresses person)))

(defn find-people-with-zip [people zip]
  (->> people
       (mapcat person-with-address-comb)
       (filter (comp #{zip} :zip :address))))

It seems like there is a better way I'm overlooking. But regardless I think any of these solutions will be a tiny bit complicated. We've lost the shape of the input data. We have some imperative stuff going on here. Let's contrast this with the Meander implementation.

(defn find-people-with-zip [people zip]
  (m/search people
    (m/scan {:name ?name
             :addresses (m/scan {:zip ~zip :as ?address})})
    {:name ?name
     :address ?address}))

This is actually incredibly straight forward even if unfamiliar. We are now using search to find multiple answers. Also note ~zip. The ~ here let's us splice in variables that are in scope. And finally, we can name our whole map using the :as pattern. This code reads like what we conceptually want to do, scan people's addresses looking for zips that match the one passed in. We do not have to think at all about how this code runs.

One Final Example

For our final example of how Meander can be used to perform data manipulation, will show one feature of logic variables that we have left off so far. To do so we need some more complex data.

(def data
  {:people 
   [{:name "jimmy" :id 1}
    {:name "joel" :id 2}
    {:name "tim" :id 3}]
   :addresses
   {1 [{:address1 "123 street ave"
        :address2 "apt 2"
        :city "Townville"
        :state "IN"
        :zip "46203"
        :preferred true}
       {:address1 "534 street ave",
        :address2 "apt 5",
        :city "Township",
        :state "IN",
        :zip "46203"
        :preferred false}]
    2 [{:address1 "2026 park ave"
        :address2 "apt 200"
        :city "Town"
        :state "CA"
        :zip "86753"
        :preferred true}]
    3 [{:address1 "1448 street st"
        :address2 "apt 1"
        :city "City"
        :state "WA"
        :zip "92456"
        :preferred true}]}
   :visits {1 [{:date "12-31-1900"
                :geo-location {:zip "46203"}}]
            2 [{:date "1-1-1970"
                :geo-location {:zip "12345"}}
               {:date "1-1-1970"
                :geo-location {:zip "86753"}}]
            3 [{:date "4-4-4444"
                :geo-location {:zip "54221"}}
               {:date "4-4-4444"
                :geo-location {:zip "92456"}}]}})

Here we have some much more realistic data than anything we've seen before. We have a map with three top-level keys. These represent data we have gathered from various sources. The first key :people is our collection of people with names and ids. The next is the :addresses of these people, indexed by id for efficient lookup. And finally we have :visits, this represents the dates that the users visited our site, again indexed by user-id.

Here's the mock scenario, we've seen suspicious activity on our site and we aren't quite sure how to narrow it down. We are going to start our investigation by finding any users who had visits that were not in the same zip as their preferred address. Because of the nature of our application, we happen to know that it is typically used at the preferred location. Once we know the users affected, we need to return their name, id, the date of access, and the zip code that didn't match. But I want to show that despite this somewhat complicated scenario, we can easily express this using Meander. Before we get there, the Clojure implementation.

(defn find-non-matching-visits [address visits]
  (filter (comp (complement #{(:zip address)}) :zip :geo-location) visits))

(defn find-bad-visits-for-person [addresses visits person]
  (let [preferred-address (first (filter :preferred addresses))
        non-matching (find-non-matching-visits preferred-address visits)]
    (map (fn [visit] {:name (:name person)
                      :id (:id person)
                      :zip (get-in visit [:geo-location :zip])
                      :date (:date visit)})
        non-matching)))

(defn find-potential-bad-visits [{:keys [addresses visits people]}]
  (mapcat (fn [{:keys [id] :as person}] 
            (find-bad-visits-for-person 
             (addresses id)
             (visits id) 
             person))
          people))

I really wanted to come up with a better implementation. for might have been useful here. If any reader has a better implementation, I'm happy to replace this one. But honestly, I think no matter what version we went with, it is going to have the features that make this one less than desirable. Just look at how much of this code is about telling the computer what to do. Let's look at the Meander version now.

(defn find-potential-bad-visits [data]
  (m/search data
    {:people (m/scan {:id ?id :name ?name})
     :addresses {?id (m/scan {:preferred true :zip ?zip})}
     :visits {?id (m/scan {:geo-location {:zip (m/and (m/not ?zip) ?bad-zip)}
                           :date ?date})}}
    {:name ?name
     :id ?id
     :zip ?bad-zip
     :date ?date}))

This is where Meander shines. ?id is being used to join across data structures. We can now find an id in people and use that to index into other collections. This allows us to find out everything we need to know about a person easily. We can also search into any collection and match on data at any level. We don't need to rely on pulling things out into a higher scope by using let bindings, making helper functions to work on sub-collections, or creating a series of transformations to get at the data we care about. Instead, we declare our data needs and the relationships that need to hold between them.

Conclusion

I hope that this has been a good introduction to how Meander can be used for practical data transformation problems. In many of these examples, the vanilla Clojure made for pretty good code. But as the data requirements become more complex, we need tools to handle these. While we may be able to accomplish any of these tasks, the understanding of the structure of our code becomes lost. Looking at the example above, we know so much about what the data coming in looks like. Our code mirrors precisely the shape of data we get in.

Now I do admit, my examples here are a bit contrived. But they are meant to be simple so we don't focus on the examples and instead focus on the code. In coming posts, I will explore more directly various ways we can apply Meander for data transformation. Some ideas I have in mind are using Meander with honeysql to turn our data into sql, transforming a collection of data into hiccup for display as html, and using Meander to scrap the web. I'd also love to do more computer science examples. Using Meander to make a little lisp interpreter, a CEK machine, or basic arithmetic. And yet, Meander goes way beyond all of these things. Meander is about more than practical data manipulation. It is about a new way of programming, a new way of thinking about problems. Hopefully, this introduction will help you to dive in and try it yourself.

Named Function Composition

Some time ago I release a little library on NPM called fluent-compose. I've had some mixed feelings about my creation. I know that no one has, or will use it and if they looked at it would probably dismiss it. In fact, if I hadn't written it, I would do the same. And yet, I think I've stumbled onto a fairly decent idea. By decent idea, I mean a hack. But before we dive into this hack, let's look at the situation that gave rise to it.

There is this fantastic, little known library called Zaphod. The idea behind Zaphod is to mirror Clojure's immutable data API. This makes it incredibly simple to do immutable updates on plain javascript objects.

import { update, inc } from 'zaphod/compat';
const state = { count: 0 };
update(state, 'count', inc) // { count : 1}

The way I've written the code above is actually not the default way Zaphod works. I imported the compat part of zaphod. By default, the functions are exposed to take advantage of the function bind operator ::.

import { update, inc, dec } from 'zaphod';
const state = { count: 0, otherCount: 0 };
state::update('count', inc) // { count : 1}

state
    ::update('count', inc)
    ::update('otherCount', dec)
// {count: 1, otherCount: -1}

This is actually some really neat functionality. It allows you to chain your operators together. We can build pipelines by continuing to bind. Unfortunately, we don't get function bind syntax for free. Function bind is still a stage 0 proposal. This means there is a very good possibility it will never make it into javascript. In fact, after a few years of sitting at stage 0, it is basically considered dead. There is quite a lot of risk involved in using it and more conservative configurations like create-react-app wouldn't use it.

But function bind syntax also has flaws even if it were accepted into the language. Function bind syntax abuses this the most misunderstood keyword in all of javascript. The functions you write with function binding in mind must use this, they can't be normal functions. Of course, you can wrap up those functions, but if we need to wrap functions up, why not wrap them in a way that doesn't require function bind?

This is where fluent-compose comes in. Let's look at an example.

import * as zaphod, { update } from 'zaphod/compat';
import { threadFirst, fluentCompose } from 'fluent-compose';

const transform = fluentCompose(threadFirst(zaphod));

const transformer = transform
    .update('count', inc)
    .update('otherCount', dec)
    .set('anotherCounter', 2)

transformer({count: 0, otherCount: 0, anotherCounter: 0})
// {count: 1, otherCount: -1, anotherCounter: 2}

update({ counters: {count: 0, otherCount: 0}}, 'counters', transformer)
// {counters: {count: 1, otherCount: -1, anotherCounter: 2}}

Here we see the threadFirst function in use. This allows us to take a collection of functions, in this case zaphod, and wrap them up into a fluent interface. But what does this fluent interface do? It is just function composition. After calling it, we get a function back. We can now use this function to pass our data through the pipeline. Since what we get back is just a function, we can also pass this function around. We can see its use on line 14 as just a normal function that lets us perform a series of transformations on data.

This is a fairly simple use of fluent-compose, let's take it one step further.

import * as zaphod from 'zaphod/compat';
import * as lodashFpCollection from 'lodash/fp/collection';
import { threadFirst, threadLast, fluentCompose } from 'fluent-compose';

const zaphodTransform = threadFirst(zaphod);
const lodashTransform = threadLast(lodashFpCollection);

const transform = fluentCompose({
  ...zaphodTransform,
  ...lodashTransform,
})

const transformer = transform
  .map(x => x + 2)
  .filter(x => x % 2 === 2)
  .set(0, 3)

transformer([1,2,3,4]) // [3, 6]

Here we can see a combination of two totally separate libraries. In fact, I even used lodash/fp because rather than taking its primary argument first, it makes it last. Yet, we were still able to compose these libraries in a simple, yet flexible way.

Yet, fluent-compose holds still more power. This time, we will be using some of the lower level features of fluent-compose, explaining them here would be beyond the scope of this post.

import { fluentCompose } from 'fluent-compose';

const baseReducer = (state, action) => state;

const initialState = prev => init => (state, action) =>
  prev(state || init, action);

const reduce = prev => (type, f) => (state, action) => {
  if (action && action.type === type) {
    return f(state, action)
  }
  return prev(state, action)
}

const INCREMENT = 'INCREMENT';
const DECREMENT = 'DECREMENT';

const increment = () => ({
  type: INCREMENT
})

const decrement = () => ({
  type: DECREMENT
})

const reducer = fluentCompose({ initialState, reduce }, baseReducer)
const counter = reducer
  .initialState(0)
  .reduce(INCREMENT, x => x + 1)
  .reduce(DECREMENT, x => x - 1)

console.log(
  counter(0, increment())
)

Using fluent-compose we've made a fluent reducer for redux! No longer would we need to write switch statements in order to make a reducer. In fact, since fluent-compose just makes functions, you can use this reducer with combine reducers. But another really cool thing you can do with it is add on the reducer after the fact. One feature to note with this implementation is that it actually short circuits, as soon as it finds the action that matches the type, it returns, so there is no wasted computation.

Why do I call this a hack?

I really do think this library is really useful, but at the same time, I can't help but feel a little weird about this library. In order to make this library work, I have to take advantage of the fact that functions are objects. I am making a function and then assigning methods to it. This definitely a strange thing to do. Now, I do avoid mutating the functions passed into, I "copy" them before I assign properties to them, but it still feels like the wrong means for accomplishing the task of creating a pipeline.

In fact, that is the thing that makes this library a hack; it is the wrong means. This library was created out of the limitation javascript imposes on us. How would we accomplish similar things in other languages? Here are just a couple of examples.

;; Clojure
(->> (range 100)
     (map (partial + 2))
     (filter even?))

-- Haskell
(|>) :: a -> (a -> b) -> b
(|>) a f = f a

range 100
|> map (+2)
|> filter even

Above we see how we could accomplish similar things in Haskell and Clojure. Almost all functional programming languages have a way to do this. In fact, there are some much more powerful techniques for function composition in both Haskell and Clojure.

Still interesting

At the same time, this method has some interesting features all on its own. What we have done is allow our functions to have special ways in which they compose. Each function can determine for itself special composition points. At each point along the way, we keep these composition properties, allowing us to compose further. Each of these composition methods has a name, hence "named function composition". While born out of necessity an implemented as a hack, there is something here, something interesting that might be worth exploring further. (Addendum: It has been two years and I've yet to explore it further.)

Defending the Incommunicability of Programs

Peter Naur's essay "Programming as Theory Building" is an exploration into the essence of programming. What is the activity of programming? How does this notion affect our practice? As Naur sees it, understanding the nature of this activity is paramount, for, "If our understanding is inappropriate we will misunderstand the difficulties that arise in the activity and our attempts to overcome them will give rise to conflicts and frustrations." Unfortunately, despite Naur's consciousness-raising essay, we find ourselves in exactly this predicament.

As the title suggestions, Naur claims that programming is first and foremost a process of theory building. What Naur means by this is that what is most fundamental to the process of programming is not the source code, the documentation, or even the running program, it is instead the knowledge the programmer has.

One way of stating the main point I want to make is that programming in this sense primarily must be the programmers’ building up knowledge of a certain kind, knowledge taken to be basically the programmers’ immediate possession, any documentation being an auxiliary, secondary product.

This notion of programming as theory building has received in general wide praise in secondary literature. It seems to be recognized widely that Naur's view brings quite a bit to the table. But there is an aspect of Naur's view has not received quite so much praise. This aspect may be called the "incommunicability thesis". Naur puts it this way:

A main claim of the Theory Building View of programming is that an essential part of any program, the theory of it, is something that could not conceivably be expressed, but is inextricably bound to human beings.

This seemingly radical thesis is not just a consequence of Naur's view, but rather a main claim of it. In other words, to view programming as theory building, one must see that these theories are in some sense incommunicable. Given the strong importance Naur places on this thesis, it is surprising that commenters who are even supportive of the view tend to reject this major claim. For instance, Donald Knuth writes:

My experiences agree well with Peter Naur's hypothesis that programming is "theory building," and they strongly support his conclusion that programmers should be accorded high professional status. But I do not share Naur's unproved assertion that "reestablishing the theory of a program merely from the documentation is strictly impossible." On the contrary, I believe that improved methods of documentation are able to communicate everything necessary for the maintenance and modification of programs.

The goal of this essay is to defend Naur's view that theory is incommunicable. Naur himself does provide such a defense, but his defense took for granted philosophical background knowledge not shared by his readers. Not only this, but Naur's defense is indirect. Here we will make explicit what was implicit in Naur, namely his reliance on Gilbert Ryle's notion of theory building and division of knowledge into two distinct kinds. Having explicated the background Naur assumed, we will examine more closely his claims.

Ryle's Theory Building

Gilbert Ryle is most known for his critique of Cartesian Dualism, the notion that the mind is an immaterial substance separate from the body. Ryle's work, while remembered for its groundbreaking critique of dualism has a much broader scope. Ryle's behaviorist theory requires that he separate mental talk from observable behavior. This self-imposed constraint requires Ryle to re-examine intellectual activity broadly and from this, we get beautiful descriptions of various intellectual activities, most important for our purpose, the activity of theory building.

It is here that difficulties arise. Theory is a word with many meanings and uses not all of which match Ryle's. In fact, what we have in Ryle is a technical notion of theory. Theory for Ryle, while multifaceted has a particular meaning, one described in Ryle's work not by definition, but by explicating its relations. In order to understand what Ryle means by a theory, we must pay close attention to these various relations and constraints he places upon it. We must distinguish between the act of building a theory, operations on a theory, and the theory itself.

Ryle's notion of theory extends across disciplines, Marx and Sherlock Holmes, while differing in subject matter and method, built theories. Someone laboring to discover how to lay carpet in a room, making measurements, determining which way the carpet ought to be laid, is caught up in the act of theory building. The historian, as they study the accounts of a battle, is building a theory about the battle's proceedings. Given this broad view of theory building, it should not be a surprise that programming will fall into this category as well.

Theory and Communication

But what sort of thing is a theory that these people are building? We may be too tempted to identify a theory with some set of statements. For example, we may talk about Newton's theory of motion by stating his three laws. We may speak of Sherlock's theory of a case by citing a passage in which he lays out his conclusion as well as the twists and turns along the way that lead him to this conclusion. How can a theory be incommunicable if a theory just is a statement of some position?

This is where Ryle's attention to detail pays off. Ryle helps us by making more precise the notion of theory, separating it from its manifestations and operations upon the theory. First Ryle wants to separate out building a theory from having a theory.

To have a theory or a plan is not itself to be doing or saying anything, any more than to have a pen is to be writing with it. To have a pen is to be in a position to write with it, if occasion arises to do so; and to have a theory or plan is to be prepared either to tell it or to apply it, if occasion arises to do so. The work of building a theory or plan is the work of getting oneself so prepared.

Here Ryle contrasts the process of building the theory, from having a theory by talking about our abilities after we have a theory, the ability to state or apply our theory. From this, it would seem that a theory is some sort of proposition. If this is so, the work of building a theory would be that of memorization in order to recite a propositional statement. But this isn't quite right.

Having a theory or plan is not merely being able to tell what one’s theory or plan is. Being able to tell a theory is, in fact, being able to make just one, namely the didactic exploitation of it. Mastery of Euclid’s theorems is not merely ability to cite them; it is also ability to solve riders to them, meet objections to them and find out the dimensions of fields with their aid.

Having a theory must go beyond mere recitation. Theories are things that can be applied and to have a theory requires the ability to apply that theory. Theories are varied in their presentation and use and a certain level of mastery is required in order to claim possession of a theory. To put these in Ryle's terms, having a theory involves aspects of knowing how and knowing that.

Knowing How and Knowing That

While not the originator of the idea, Ryle offers a spirited defense of the distinction between knowing how and knowing that. Roughing speaking to "know how", is to have the ability to perform an action. Whereas "knowing that" is to justifiably believe a true fact. In The Concept of Mind, Ryle intends to defend two claims concerning these types of knowledge. First, "knowing how" and "knowing that" are not reducible to each other, they are two distinct forms of knowing. Secondly, "knowing how" doesn't require prior instances of "knowing that". This second claim we will not explore further, but in order to understand Ryle's notion of theory, we must explore more Ryle's separation of these two forms of knowing.

The stereotypical example used in philosophy when discussing this topic is that of a juggler. We are led to imagine a skilled juggler, who with ease can juggle various objects in various amounts. His hands rise and fall in perfect time with the objects. This is a perfect example of "knowing how", our juggler has a certain intuitive understanding of the objects they are manipulating. They knows how high to throw an object, how quickly to move his hands to catch them as they fall; his knowledge is made evident by his performance.

There are some who suggest "knowing how" is just a species of "knowing that". In the case of our juggler, the suggestion might be that there are certain propositions that the juggler knows such as "If I am juggling N objects, I need to throw them up X feet at Y angle." In fact, if "knowing how" is reducible to "knowing that", our juggler knows all sorts of propositions of this type. But, as Ryle argues, one could know all possible propositions about juggling and still themselves not be a skilled juggler. Juggling requires practice, it requires "muscle memory", it requires us to build up knowledge of how, not just propositions about the activity of juggling.

To be a Newtonian was not just to say what Newton had said, but also to say and do what Newton would have said and done. Having a theory is being prepared to make a variety of moves, only some of which are teachings; and to teach something to someone, oneself or another, is, in its turn, to prepare him for a variety of tasks, only some of which will be further teachings.

Here we can see Ryle's insistence that theories require a sort of "knowing how". We are to be "prepared to make a variety of moves". Having a theory requires the ability to know how to wield this theory. It requires being able to put the theory into practice, to use it for its particular ends. This is not merely to sit in an armchair and draw conclusions from it, but to know our theory so well, we know how to teach it, how to answer queries about it, how to relate it to other things, how to modify it in the face of new evidence. Each different type of theory has its own use that for which it must be employed.

Sherlock Holmes’ theories were primarily intended to be applied in the apprehension and conviction of criminals, the thwarting of planned crimes and the exculpation of innocent suspects….His theories were applied, if further deductions were actually made from them, and if criminals were arrested and suspects released in accordance with them.

Programming as "Knowing how"

With this philosophical background laid out, it becomes much easier to see what Naur might mean by statements like:

A main claim of the Theory Building View of programming is that an essential part of any program, the theory of it, is something that could not be conceivably be expressed, but is inextricably bound to human beings.

Having a theory is to know how to do something. If a programmer has a theory about a program, they will know how to make changes to that program, we will be able to answer questions about that program, they will be able to explain parts of the program to others. But none of these presentations or applications of the theory of the program are the theory itself

If no presentation of a theory is the theory itself, it follows that no documentation can capture fully the theory of any program. If having a theory requires being able to perform certain moves, in the case of programming, making modifications to a program, then no program contains its own theory. In fact, no text, no video, no media of any sort can contain the theory of a program. Having a theory is something only a human can do.

Implications

The theories of the programs we write live inside us. As we move on to new projects, leave old companies, or forget about code we once wrote, those theories die. The code we wrote might live on. People may continue to run it, read it, modify it, but some understanding of that code lives (or lived) only in our heads. Try as we might, we cannot communicate this theory in its full.

As we inherit code bases written by others, we encounter the same things, codebases whose theories are gone to time or locked away in a location inaccessible to us. Programs that have been modified under many different theories, often incompatible ones. Bugs begin to find their way in, often at the intersection of two theories. Hacks begin to emerge as needs evolve, but theory is lacking. The code becomes worse and worse.

These are the facts we face as professional software engineers. We write code whose theories will one day be lost and we inherit codebases whose theories are long forgotten. If we are to do programming well, we must adopt practices that address these issues.

Basic Functional Studies

In the previous post, we observed some rules on how to get the most out of our functions. In this post, we will do exactly that, get the most out of them. But rather than diving straight into these techniques, we are going to come up with them ourselves. There is no better way to learn a concept than to arrive at it through simple steps. To see how we ourselves could have come up with these techniques.

Map, Getting Rid of 50% of For Loops

Let's imagine we need to add 2 to every single item in a list. How would we write this code? Perhaps something like this.

function add2ToList (list) {
    let temp = [];
    for (let i = 0; i < list.length; i++) {
        temp.push(list[i] + 2);
    }
    return temp;
}

add2ToList([1,2,3])

This function most certainly works. It does precisely what we asked. But it isn't very flexible. What if I want to add 3 to each item in the list? Well, I could go write a function called add3ToList, but that seems cumbersome, so instead let's make this function more generic.

function addNToList (n, list) {
    let temp = [];
    for (let i = 0; i < list.length; i++) {
        temp.push(list[i] + n);
    }
    return temp;
}

addNToList(n, [1,2,3])