
Last year I came across the Recurse Centre's [social rules](https://www.recurse.com/social-rules)\*.

Working with computers day-to-day, it's easy to get carried away with the incidental complexity of our work. I've found that too often we can slip into discourse which at best could be called 'unproductive'. I really like how these social rules neatly and concretely capture a number of behaviours which are self-evidently unproductive and unsupportive.

As a brief summary, the rules are:

### [No well-actually's](https://www.recurse.com/social-rules#no-well-actuallys):

> **Alice:** I just installed Linux on my computer!
>
> **Bob:** It's actually called GNU/Linux

This one really makes me reflect on how often, when we're 90% in agreement with the person we're speaking to, we have the opportunity to either frame our response around points of agreement, or else our — sometimes pedantic — points of disagreement. I don't think this is quite all that was was intended when this rule was written, but it feels valuable to me to reflect that while constructive disagreement can be important, not every point is helpful.

### [No feigning surprise](https://www.recurse.com/social-rules#no-feigning-surprise)

> **Dan:** What's the command line?
>
> **Carol:** Wait, you've never used the command line?

We all learned something once, and you'll understand things better yourself if you always keep that in mind.

### [No backseat driving](https://www.recurse.com/social-rules#no-backseat-driving)

> **Bob:** What's the name of the string copy function?
>
> **Alice:** Strncpy.
>
> **Eve:** (from across the room) You should use strlcpy. It's safer.

This is probably my personal biggest bug-bear.

I'd group with it a class of frustrating behaviours where we insist we have the 'answers' to the problems other people face; even if we haven't taken the time to understand their problems properly\*\*. As such, I think it can be more than just a conversational blooper, but instead can be a long-lived behavioural mistake.

### [No subtle -isms](https://www.recurse.com/social-rules#no-subtle-isms)

> **Carol:** Windows is hard to use.
>
> **Bob:** No way. Windows is so easy to use that even my mom can use it.

It's easy to belittle the complexity of things we've gotten used to, and doing so doesn't help ourselves or others. As a connected thought, I wonder how much incidental complexity could be cut out of computing if we stopped dismissing real issues with the oversimplyfing words "_it's just..._".

That's the last rule, but really I'd encourage you to [read them for yourself](https://www.recurse.com/social-rules).

---

\* I came across the social rules in the [code of conduct](https://courses.joshwcomeau.com/code-of-conduct) for Josh Comeau's excellent [CSS-for-JS](https://css-for-js.dev/) course. I'd recommend it to anyone!

\*\* See also [this post](/posts/2021-11-30-vision-and-decisions)


How we should treat each other


One of my main observations from working in engineering is that there are very few universally good decisions. A choice which has worked well for me in a particular scenario isn't necessarily great advice for others facing their own challenges, and with their own skillsets.

Though this observation applies to engineering, I think it extends to a wide set of problem domains. It strikes me in any situation where we want to solve hard problems, requiring detailed knowledge and skillsets. When solving these problems, the devil is often in the detail. Specific complexities shouldn't always be quickly dismissed and we should make sure to listen to the people who most deeply understand them.

Based on these observations, a natural follow-up question is...

<PullQuote>

_How is it possible to lead a team, when it's so difficult to answer the questions they face?_

</PullQuote>

### In organisations

There's an analagous problem to be faced at an organisational level. Organisations need their people to do work. Fundamentally the challenge is to identify what work those people should do in order to achieve the organisation's goals.

For practical reasons, it's not possible for this to be solely bottom-up activity. If we ask 100 people what they should do, we'll get 101 different answers. Really we need everyone to be working in careful coordination, and for that reason it's important that some appropriate guidance should flow down from the top.

But here it's easy to fall into issue I described above: interesting problems involve a lot of necessary complexity, which only the people working on them deeply understand.

### Passing down a vision

Consequently, I think that in organisations we should aim to pass down a 'vision' of where we want to go, not a set of instructions for what to do. I consider a vision to be a description of the change that we expect to see based on our work.

![Organisation chart with vision arrow pointing down, decisions arrow pointing up](/vision-1-organisation.png)

By passing down a vision, we can begin to let specific decisions be made by the subject matter experts who are closest to their complexities.

This picture of an organisation can be helpful for encouraging a certain set of behaviours in it's leaders:

- **Clearly articulate a vision** - most obviously, a leader should evangalize the direction the organisation wants to go. There are so many subtleties and pitfalls to doing this well that it feels almost trivialising to write this as a single bullet point; the scope goes way beyond this blog post.
- **Focus on behaviours, not solutions** - in organisations, we tend to spend a lot of time discussing concretely _what we've done_, but this lacks a universal application to the rest of the organisation. In the engineering domain this looks like a presentation about a new tech tool, when perhaps we'd be better served talking about the process by which we understood the needs of our dependents. Overall, I think this point is a variant of the idea that 'culture beats process': we should focus so much more on _how_ we solve problems that just a narrative of _what_ we've done.
- **Display judiciousness** - in an organisation where decisions are flowing up there is a lot of responsibility on leaders to assess which decisions will best enact the overarching vision; and to prioritise accordingly. I think this is where we start to see the benefit of experience: I may not know the answer to your question a priori, but with your expert guidance, I can help you get things done.

My take-away is that we should focus on being good at solving problems, not at answering questions.


Leading a team without knowing all the answers

Vision and decisions 👀


This is a follow-up to [Happy path programming (Part 1)](/posts/2021-10-20-happy-path-programming-pt-1), where I considered what makes Data Science 'notebook programming' different from production 'software programming'. The notes I've made here are mostly unstructured, and really are more like a shopping list 🛒 of ideas which I'd be interested to explore in future.

If there's a summary to my thoughts, it would be that "_[simple things should be simple, complex things should be possible](https://en.wikiquote.org/wiki/Alan_Kay)_". Unfortunately, what I actually see right now is:

- Some languages for which _simple things are simple, and complex things impossible_. For example Python, Javascript, Java
- Some languages for which _simple things are difficult, but complex things possible_. For example Haskell, OCaml

<Panel>

In this post, I'd like to use a couple of terms with only an approximation of their full definition:

**"_Compile-time_" or "_Static_"?**

Roughly, I mean the set of 'facts' that can be deduced about the values being passed around in a program, by **only** reading its source code. For example, by reading some code we might know that a 'data table' will be passed around at 'run-time'.

**"_Run-time_" or "_Dynamic_" ?**

Roughly, I mean the actual values which end up being passed around in the program when it actually runs. For example in a notebook, this might be an actual instance of a 'data table', which might have e.g. 4 columns (each with a specific, known name) and 100 rows (each with a specific, known value).

</Panel>

---

## 🛒 Data ergonomics

This is an area where I think languages have to deliver. Manipulation of known or unknown data is a core requirement in data science and modern programming. I've found that many languages promise a lot, but are so hard to use for simple data transformations, that only the most hardy users ever make it past the door.

For starters, my own personal bias is that _anyone_ would benefit from the maximum possible static [type](https://en.wikipedia.org/wiki/Data_type) information being available when writing code. I'd love every developer — at least once — to experience the power of a supportive type-system which is truly helping them write bug-free code. When writing the following code, I have autocompletion, hints and descriptions for every property, since the compiler already knows about the values I'm passing around:

<VideoAutoplay src="/notebooks2-1-editor-support.mp4" />

But in order to make types ergonomic, I think it can be extremely useful to have the 'get out of your way' when you need to. A really nice example of this is [optional chaining, nullish colaescing](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html) and the [non-null assertion operator](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-0.html#non-null-assertion-operator) in Typescript:

```typescript
/** An Author maybe has a name */
type Author = { name: Maybe<string> };
/** A Book has a name and maybe an Author */
type Book = { name: string; author: Maybe<Author> };

const catch22: Book = {
  name: "Catch 22",
  author: { name: "Joseph Heller" }
};

const bible: Book = {
  name: "Bible",
  author: null
};

/** Will have value: "Joseph Heller", type: Maybe<string> */
const catch22Author = catch22?.author?.name;

/** Will have value: "Unknown Author", type: Maybe<string> */
const bibleAuthor = bible?.author.?.name ?? "Unknown Author";

/** Will have value: "Joseph Heller", type: string */
const Author = catch22!.author!.name;
```

There features, are incredibly useful in Typescript when temporarily wanting to consider only the 'happy path' of the function. It is equivalent to saying to the compiler: "_I know about this value, don't get in my way_".

<Collapsible header="More thoughts">

As an aside, one issue I dislike here is how non-null assertions 'infect' a codebase. If we assert a value to be non-null in one place, then the compiler will treat every callsite as non-null, and they'll all become vulnerable to failures. I wonder if instead we could do something like limiting the non-null assertion to [lexical scope](<https://en.wikipedia.org/wiki/Scope_(computer_science)#Lexical_scope_vs._dynamic_scope>), so a value would be treated as non-null within a function/closure, but any caller using that function would still have to handle the null case. An example would be:

```typescript
function getAuthorName(author: Author): Maybe<string> {
  /** This value is of type string (not Maybe<string>) */
  const name = author!.name;
  /** And so we can use it as a string here */
  printStringStrict(name);
  /** But when returning it outside of our lexical scope, the compiler knows it's still a `Maybe<string>` */
  return name;
}
```

</Collapsible>

## 🛒 Code generators

In order to let the compiler support us when we're programming, we need to find a way to tell it information about the values that we're going to handle in our program. I'm almost always suprised how difficult this is.

A few examples of code generators which would be useful:

- Type generators based on static data (e.g. CSV, JSON)
- Type generators based on Rest API responses
- Code generators which create parsers for those types
- Code generators for creating utility functions over those types

For example, wouldn't it be useful if our language or 'notebook' could inspect a CSV file, and tell us that the data frame we've created will have e.g. 4 columns, each with an inferred type?

<Collapsible header="More thoughts">

Haskell and many other languages support this idea of moving from values into types (although that isn't quite what's shown in the snippet below), for example with [Template Haskell](https://en.wikipedia.org/wiki/Template_Haskell) and [Lenses](https://hackage.haskell.org/package/lens). But as with the discussion on 'data ergonomics', it's pretty cryptic (Why do fields have to start with `_`? What does the `''` mean before `User`? How do we handle field name clashes?):

```hs
{-# LANGUAGE TemplateHaskell #-}
import Control.Lens

data Author = Author {
    _authorId :: Int
  , _firstName :: String
  , _lastName :: String
}

makeLenses ''User

-- We'll now have utility functions available for interacting with Author types
```

</Collapsible>

## 🛒 Exceptions

There are lots of problems with exceptions, and you don't have to do much googling to find people talking about them. Here's some example code:

```ts
const response = await fetch("https://www.foo.com/data");
const result = extractResult(response);
```

The compiler can help us avoid errors in this sort of code. By adding compiler annotations, we could let the compiler infer the type of `response` and `result`. But what compilers generally don't do is tells us where exceptions might be thrown. Most programming languages throw [unchecked expections](https://www.baeldung.com/java-checked-unchecked-exceptions), which the compiler does not force us to handle.

But why doesn't the compiler force us to handle exceptions? Well, without having read too much on this topic, I'd imagine:

- Exception handling is boiler-plate heavy - amongst other things they are handled with 'statements' rather than 'expressions'
- Exceptions don't compose well - how do we handle the case where running an operation on each element of a list yeilds some errors and some values?

All of which means that we don't want to force users to handle every possible exception. In principle an exception should only be used if you want your program execution to terminate, but the reality is that they're already used for control flow, and we've simply accepted that it's not ergonomic to enfore compiler checks on this control flow.

<Collapsible>

As an aside, in the defence of exceptions, they do have the interesting property that as well as an error message they contain the execution stack-trace of the point at which they occurred. In functional programming, the advice is to avoid exceptions at all costs and to use [referentially transparent](https://en.wikipedia.org/wiki/Referential_transparency) data types such as `Either`. But stack-traces are really useful as a user.

At the risk of huge cost and magic, I wonder if languages could support some of this behaviour, for example, that an instance of `Error` could preserve a strack-trace.

````ts
// weird-example.ts
type Result<T> = { result: T };
type Error = { message: string; trace: StackTrace };
type Either<T> = Error | Result<T>;

function generateError(): Either<T> {
  return Error("I'm a generated error");
}

let error = generateError();

/**
 * The following line could print something like:
 * ```
 * Error: {
 *   message: "I'm a generated error"
 *   trace: "at generateError (/weird-example.ts:6:10)"
 * }
 * ```
 */
print(error);
````

</Collapsible>

## 🛒 Concurrency

For me, the number one failing of Python is how challenging it is to write concurrent code. Most code written in Python is [synchronous](https://www.outsystems.com/blog/posts/asynchronous-vs-synchronous-programming/) and blocking (with a [global interpreter lock](https://realpython.com/python-gil/)).

Typically a data scientist might write a notebook - used only by them - in which the concurrency problem never appears, but then 'deploy' their work as some kind of webserver. As soon as more than one request is received simultaneously, the application becomes unusably slow.

I won't go into detail here, but I think any 'notebook programming' system of the future should have much better support for runtime concurrency ([something which is also being worked on in Python of course](https://realpython.com/async-io-python/)).

## 🛒 Incrementalism

This idea is the most far-fetched 😅

Another big issue I see in notebooks programming is that cells get executed out of order, and therefore results can't easily be reproduced.

One idea for how this could be avoided is that each value could incrementally update, like a spreadsheet. The implications of making every value reactive - particularly when IO or side-effects are involved - is huge though, and something I'll leave for another day!


What can we learn from notebook programming?

Happy path programming (Part 2)


Like many engineers, I've spent time working with the programming language Python and with interactive [Jupyter Notebooks](https://jupyter.org/). Together, these two tools are the industry-standard toolkit for Data Science and Machine Learning. I really enjoy the interactive nature of notebook coding; but one of the reasons I've focused my career towards software engineering - rather than other specialties - is because I've observed that many organisations have their biggest problem with the quality of their engineering, rather than the ingenuity of their analyses.

In the back of my mind though, this always leaves a big unanswered question:

<PullQuote>

_How do we close the gap between code experiments, and robust software?_

</PullQuote>

In this part of the post, I'd like to start by summarising the problem.

## Notebook programming - it's about the process

Firstly, what does notebook programming look like in the abstract? I'd characterise it as having a few properties:

- **Startpoint** - Typically a 'single' known input, such as a CSV or dataset
- **Endpoint** - Some desired output, such as a chart or trained model
- **Process** - An iterative process of finding a path between the two

![Animated gradual creation of a path between an input point and output point](/notebooks-1-process.gif)

That intentionally oversimplifies a lot! Amongst other things:

- There are often many hidden inputs, like REST APIs or other enrichment dependencies. Those dependencies are alive and working at the point the notebook is created, and so they are broadly treated as additional 'known' inputs
- The 'desired output' may also not be clearly defined at the start, since the work is often investigative. We might only know the 'endpoint' when we reach it.

Broadly, once the notebook process has been completed, we have our happy path, but we probably shouldn't stray from it:

![Path between an input point and output point with fire and mountains either side](/notebooks-2-outcome.png)

## Software programming - it's about the destination

The goal production-ready software is to widen the set of happy paths available. To be more resilient to unexpected inputs, and to produce some desired output a greater percentage of the time:

![Paths between many input points and a single output point](/notebooks-3-software.png)

We want to be able to get to a rational output even when one of our inputs or dependencies isn't working in the way we'd expect. This 'rational output' may simply be an error message, but generating it correctly matters.

From this perspective, we can state the problem as: how can we interactively pursue our happy path, while at the same time ensuring we eventually deliver a wide set of well-handled code paths?

## ⏳ Next time

In [Part 2](/posts/2021-11-22-happy-path-programming-pt-2) of this post, I'll lay out a few of my own disparate ideas about how we could think about bridging the gap between these two software development styles.


Happy path programming (Part 1)


It's really great to be writing this (sort of) first post on my new blog. Like all good engineers, I started off by writing a blog framework 🤦‍♂️ so it's nice to now be writing some content.

My motivations for blogging are broadly two fold:

- Firstly, I'd like to have somewhere to distill and clarify my own thoughts. Much like teaching, I think that writing really pushes you to consider a topic from new perspectives, and I hope to gain some benefit from doing this.
- Secondly, I'd like to have a place where I can archive my own thoughts. Ideally I'll refer back to this blog for my own 'best-practices', and gradually it can be a place where I get to see which of my ideas stand the test of time. I doubt this will help me in the short term, but maybe in 20 years I'll be a better engineer!

So, for the time being my missives will be mostly aimed at my personal development, but perhaps one day - by some happy accident - something I write may prove helpful to others on their own journey.

I don't know yet what I'll want to write about, but I'd say my guiding interest is: _how can 'good' software be written in practice?_ I think that's as much a social problem as it is a technical one, so I hope to capture some dispatches from the frontline of this intersection.

Thanks in anticipation!

---

**\*postscript**

I have already put together one post '[an intuition on eigenvectors](/posts/2021-02-06-an-intuition-for-eigenvectors)' unrelated to the theme I just suggested. It brings together some visualisations I made when playing with the Python library [plotnine](https://plotnine.readthedocs.io/) (a great compositional charting library based on the [Grammar of Graphics](https://www.amazon.co.uk/Grammar-Graphics-Statistics-Computing/dp/0387245448) idea). I enjoyed writing up these notes, and it pushed me to add [maths](https://katex.org/) and inline-video support to the blog.


Welcome to my new blog 🎉


Eigenvectors are a really interesting concept with a wide variety of applications. I originally learnt about them while [studying Physics](http://hyperphysics.phy-astr.gsu.edu/hbase/quantum/eigen.html); then a few years ago they came up again in the context of [Principle Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA), a Data Science technique.

In my experience, it's easy to miss the fundamental concept behind eigenvectors because they are written about so formally. (You might have found the same if you clicked on either of the two links above 👆). And so here I wanted to capture my own limited mental model; focusing on an intuition for what's happening, rather than anything formal.

## What is a matrix?

Here is a (square, two-dimensional) matrix, <InlineMath math="A" />:

<BlockMath
  math="A=\begin{pmatrix}
   1 & 2 \\
   3 & 4
\end{pmatrix}"
/>

A matrix can be thought of as a transformation on vectors in space. That's because, given a vector <InlineMath math="\vec{a}" />:

<BlockMath
  math="\vec{a}=\begin{pmatrix}
   1 \\
   1 
\end{pmatrix}"
/>

It can be transformed by applying some matrix to it:

<BlockMath
  math="\vec{b}=A \cdot \vec{a}=\begin{pmatrix}
   1 & 2 \\
   3 & 4
\end{pmatrix} \cdot \begin{pmatrix}
   1 \\
   1 
\end{pmatrix} = \begin{pmatrix}
   1 \cdot 1 + 2 \cdot 1 \\
   3 \cdot 1 + 4 \cdot 1 
\end{pmatrix} = \begin{pmatrix}
   3 \\
   7 
\end{pmatrix}"
/>

The new vector <InlineMath math="\vec{b}" /> is called the _dot product_ of the matrix and the original vector. And here's what the transformation process looks like:

<VideoAutoplay src="/eigen-1-transforming-a-vector.mp4" />

## What is an eigenvector?

To start answering that, first of all let's imagine applying this matrix to every possible vector in a space. Here's one example of what that might look like:

<VideoAutoplay src="/eigen-2-transforming-all-vectors.mp4" />

Hopefully, looking at this, it's noticeable that in some directions the vectors are _stretched_ (maybe even inverted), but not _skewed_. I've marked those directions with blue arrows:

<VideoAutoplay src="/eigen-3-transforming-all-vectors-with-overlay.mp4" />

These blue arrows are the directions of the eigenvectors of the matrix.

## An intuition

<Panel className="text-lg">

A matrix can be thought of as similar to a lens or a prism; looking through it distorts space. If you imagine letting
a lens rotate in front of you, then there will be certain directions in which you see the world as _stretched_, but
not _skewed_. Those directions are a property of the lens, and we call them eigenvectors.

</Panel>

I purposefully haven't mentioned how to calculate an eigenvector in practice, or why it would be useful to do so. But I hope that this intuitive idea of a lens might help keep a picture in your head if you're ever working with eigenvectors in the future.


Welcome 👋

Blog posts

How we should treat each other

Vision and decisions 👀

Happy path programming (Part 2)

Happy path programming (Part 1)

Welcome to my new blog 🎉

An intuition for eigenvectors 💡