Programming

Everything else being equal, more lines of code is worse than fewer. There are trade offs and caveats of course (so “everything else being equal” is doing some heavy lifting here), but in general, as the amount of code increases:

• The number of locations where the program can break goes up.
• The temporal cost to understand the code goes up.
• The cognitive load on the developer reading the code goes up.
• The cognitive load on the developer reviewing the code goes up, and as that load increases the quality of the code review decreases.
• The overhead to make a change and confirm it works goes up This is particularly pronounced in applications where manual testing is slow, complicated, confusing, expensive, or (as is often the case); all of the above.

A trivial example of this might be: different average calculation. Let’s assume that we’re writing a program that has a bunch of different users, and further assume that for some reason we need to calculate the average age of our users. Maybe we don’t know going into this what the method is, or perhaps we are expecting it to get really complicated. In order to prepare for either of these situations we make a function that takes a list of ages and returns the average. I have also added some typing here since we know those expectations as well.

def get_user_age_avg(ages: List[int, int]) -> float:
    return mean(ages)

This is usable right now, allows the developer to continue building the system, and in the future if the method for getting the average of ages changes, we only have one location in The Program where we’re doing that, so one logical change “Please use median instead of mean” equates to one physical change “update get_userageavg”. Even more important, we can test the method by itself so we can ensure that it continues to work as expected.

Encapsulating correctly, as this should illustrate, is much easier to do if you know what changes are expected; however, it’s also possible to structure code reasonably well without knowing what changes are scheduled. We sometime call this process “abstraction” and will talk about “abstracting the code”. This can be thought of as an iterative process with several steps addressed later.

width_feet = int(width_meters / 0.3048)
height_feet = int(height_meters / 0.3048)
depth_feet = int(depth_meters / 0.3048)

This can be “abstracted” to move the calculation into a function:

def feet_to_meters(meters):
    return int(width_meters / 0.3048)

width_feet = feet_to_meters(width_meters)
height_feet = feet_to_meters(height_meters)
depth_feet = feet_to_meters(depth_meters)

One might immediately notice that we have increased the number of lines of code, which all things being equal is a bad thing; However, things are not all equal here, we have move the “feet to meters” calculation into one place, which means that if we ever want to change that, or if there’s a bug to fix, there’s just one place to fix it. Of course, it’s terribly unlikely that the math for this calculation is ever going to change, so in reality we’re likely going to be attempting to abstract more complexity than just one division operation.

A more complex example might be calculating a GIS region. One might image that a program is written and that part of the expected behavior from the program includes calculating a physical region from some incoming data. Let’s further assume that the stakeholders don’t have hard details about the shape and size of the region, but that it’s going to be contiguious, and mutable; it will be one discrete region and have the ability to change over time.

This is an interesting problem because it has a very simple pseudo solution, but a much more complex rigorous solution (particularly when you add in projection).

Any “blob” shape can be closely estimated by a polygon, with the accuracy of the shape increasing as we add points to the shape. A very simple polygon differs from a very complex polygon in the number of individual points that make up the polygon. Imagine a very large, very complicated GIS region (such as a storm or cloud). Such a region can ultimately be represented, and stored by a list of points (latitude, longitude) at the edges of the region. We don’t need to start with super complex though. We just need a polygon.

Perhaps the most simple solution is calculating a square, and the most straightforward operation is likely to calculate the square inline.

So, let’s calculate a square with sides of 5 from the euclidean origin:

# Comment: calculate a square
SQUARE = [(0, 0), (0, 5), (5, 5), (5, 0)]

>>> print(SQUARE)
[(0, 0), (0, 5), (5, 5), (5, 0)]

This is a very simply shape, but it’s a valid polygon, and placing it at the origin gives us the ability to heuristically define a point that should be inside the region: (2,2). We could now, armed with this expectation, write a unit test for the logic to determine that a point is in a region:

# Comment: calculate a square
def is_point_inside(region, point):
    # TODO: implement me
    return True

>>> assert is_point_inside(square, (2,2))
True

There’s a glaring omission here, the very broad “TODO: implement me”. This is very intentional. What we have done here is to move the “is a given arbitrary point inside a region” calculation (which is non-trivial) into it’s own function. This means that:

• We can reuse the function everywhere we want to perform this calculation.
• We can “check to see” if the function works manually.
• We can write an automated test to ensure that the function continues to work.

This is what a unit test for that calculation might look like. Because we’re passing in simple regions and points we can actually write the tests before we write the code. The second argument to assert is an optional failure message.

# Comment: calculate a square
from main_program import is_point_inside
from fixtures import SQUARE #5x5 at origin

def test_is_point_inside():
    # Confirm that known interior points are "inside"
    assert is_point_inside(SQUARE, (2,2)), "Failed in the middle"
    assert is_point_inside(SQUARE, (5,0)), "Failed on a corner"

    # Confirm that known interior points are "outside"
    assert not is_point_inside(SQUARE, (6,0)), "Failed outside on one axis"
    assert not is_point_inside(SQUARE, (6,6)), "Failed outside on both axis"
    assert not is_point_inside(SQUARE, (-1,2)), "Failed negative axis"

Because we’re using a simple square, we can put together this test (this is a unit test), and define how the real program should behave. This method is called “test driven development”, or TDD. To continue on the TDD trajectory we would then run the test with the expection that it will fail (because we hardcoded it to return true, and there are some test cases where it should return false). However, we don’t have to. If we want to work on logic that adds a new point to a region, we could hardcode is_pointinside to false. For now let’s just add a doc block (and some typing).

# Comment: calculate a square
def is_point_inside(region, point) -> bool:
    """
    Determine if the given point is inside the given region.
    TODO: look for an external library that does this well
    Returns a boolean
    """
    return True

Let’s put that complicated logic down for now and return to our region, and encapsulation.

A square doesn’t really approximate a circle, it has corners that take up more region than we want. At this point, we can further refine the shape, we can add more sides!

Calculate a hexagon with sides of length l around the euclidean origin:

# Comment: calculate a hexawrong
import math
pi = math.pi

hexagon = [(math.cos(2*pi/6*x)*5,math.sin(2*pi/6*x)*5) for x in range(0,6+1)]

print(hexagon)
[(5.0, 4.330127018922193), (2.5000000000000004, 4.443255075045337), (-2.499999999999999, 0.471274906292427), (-5.0, -3.9339932379101574), (-2.500000000000002, -4.722366141717481), (2.4999999999999964, -1.1690173931370136), (5.0, 3.4591205554937012)]

Now one might imagine that there are several places where we are calculating a hexagon from a point. This implementation is fairly brittle, hard to change, and hard to confirm that it’s working or working as expected.

Now let’s assume that we’re calculating multiple regions of different sizes:

# Comment: calculate a hexawrong
import math
pi = math.pi

l = 15
large_hexagon = [(math.cos(2*pi/6*x)*l,math.sin(2*pi/6+x)*l) for x in range(0,6+1)]

l = 6
tiny_hex = [(math.cos(2*pi/6*x)*l,math.sin(2*pi/6*x)*l) for x in range(0,6+1)]

print(tiny_hex)
[(5.0, 4.330127018922193), (2.5000000000000004, 4.443255075045337), (-2.499999999999999, 0.471274906292427), (-5.0, -3.9339932379101574), (-2.500000000000002, -4.722366141717481), (2.4999999999999964, -1.1690173931370136), (5.0, 3.4591205554937012)]

Did you notice that the math is wrong for the large hexagon above? I introduced a typo; some rhetorical questions about this typo:

• Was it immediately obvious?
• Can you find it?
• How long did it take to find?

This is the sort of error that is unlikely to be identified by a peer review, the math still produces a list of points, but one would likely need to plot the shape on a map to detect the issue.

One technique to combat this is to move the calculation into a function, so we only have to get hexagon creation right one time, and can reuse that known good code:

def hexagon(length):
    return [(math.cos(2*pi/6*x)*length,math.sin(2*pi/6*x)*length) for x in range(0,6+1)]

a_hex = hexagon(5)
b_hex = hexagon(6)

Of note: this code is now testable; we can write a “unit test” that confirms that the function always returns a good value:

assert hexagon(5) == KNOWN_GOOD_HEXAGON_SIZE_FIVE

Where the constant is a the known correct value for a five size hexagon.

We can see this in action here:

>>> print(hexagon(5))
[(5.0, 0.0), (2.5000000000000004, 4.330127018922193), (-2.499999999999999, 4.330127018922194), (-5.0, 6.123233995736766e-16), (-2.500000000000002, -4.330127018922192), (2.4999999999999964, -4.330127018922195), (5.0, -1.2246467991473533e-15)]
>>>

You might be familiar with variables and suggest “Perhaps we want to make other shapes”. We can achieve that by moving the number of sided into an argument for the function, of course renaming it along the way to represent what it does:

def polygon(l, n):
    return [(math.cos(2*pi/n*x)*l,math.sin(2*pi/n*x)*l) for x in range(0,n+1)]

Which will produce the same output, but could be used to make regions that are octagons, or any polygon. Now let’s return back to the stakeholders who are not aware or interested in hexagons, they want something closer to a circle.

We can add more sides to the polygon to more closely approximate a circle:

def polygon(l, n):
    return [(math.cos(2*pi/n*x)*l,math.sin(2*pi/n*x)*l) for x in range(0,n+1)]

def circle(l):
    """Return a hundred sided polygon."""
    return polygon(100)

Of course this is all just theoretical until we try to use said circle. As per the stakeholders we’re calculating the “circle” at a point (lat/long), so we actually want to provide a point for the origin. We might see the program start to take shape like this:

def polygon(l, n):
    return [(math.cos(2*pi/n*x)*l,math.sin(2*pi/n*x)*l) for x in range(0,n+1)]

def position_polygon(polygon, point):
    """
    Given a poly and a point, center the polygon on the point
    returns the resulting polygon
    """
    #Assume implementation is here
    return polygon
    
def circle(l):
    """Return a hundred sided polygon."""
    return position_polygon(polygon(100))

I’ve not included the implementation of moving a polygon to a euclidean point. This is, I believe, a good example of “postponing complexity”. The program operates on polygons, it does not care where the polygon is, so we can get everything working end to end, but without the unimplemented part above. Manual testing of this program could occur, and the tester would just need to know “Look for the polygon at null island”.

Now let’s further assume that the initial request from the stakeholders was not a circle, but was in fact any arbitrary region (which is a superset that contains a circle and all the pseudo circles we created above), we could further refine the ask to be:

• Given a point, a width w, and a shape calculate a shape of width w around the point

Of course, size can be a property of a well defined shape, so you could further collapse the ask into:

• Given a point and an arbitrary shape, calculate and return a region centered at said point.

Of course, in the real world, you cannot just map a euclidean shape directly onto a lat/long: that will produce an oval, the size of which varies depending on where it is located on the globe. So there is a further step that needs to be taken to get the circle to the right place and shape: projection.

The model pattern might be the most simple, and perhaps the most elegant. It’s the M in the much vaunted “MVC” pattern. A model represents a discrete “thing” (I know that’s not helpful yet), which can be a real word “thing” like a user, a company, a home, or as in the example above: a geographic region.

class Region():
    """Represents a geographical region made of multiple points."""
    points = []

This pattern is also sometimes known as “Active Model”, generally when the model performs some validation. Here’s an explicit validation method:

class Region():
    """Represents a geographical region made of multiple points."""
    points = []

    def is_valid(self):
        """Confirm that the region is a valid shape."""
        assert is_instance(self.points, List)
        assert len(self.points) > 2
        return True

This allows us to hide complexity in the Region object. Once we have done that, then operations that occur on the Region level become possible, and can expose the business logic while hiding the implementation details. Imagine now that we’re asked to combine regions. If The Program were littered with `square = [(0, 0), (0, l), (l, l), (l, 0)]` snippets all over the place, we’d likely need to tear the whole thing apart! However, if we have a Region class, then we could write a Region.combine method like the following:

class Region():
    def combine(new: Region):
        r = Region(self.points)
        for p in new.points:
            r.append(p)

        return r

This means that our “The Program” code can look like this:

def main(existing_region, new_region, new_point):
    if existing_region.is_point_inside(new_point):
        # point is in existing region return that region
        return existing_region

    if new_region.is_point_inside(new_point):
        # point is in expanded region return that region
        return existing_region.combine(new_region)

    return False

The above has hidden the complexity involved with shapes, regions, projections and is really just the business logic. One can certainly comment on this, but it should be clear from the above code that it:

• Returns the existing region if the new point is in the existing region
• Returns the combined regions if the new point is in the new region
• Returns false if the point is in neither region

This logic does not need to “know” about the details of the shape creation, it just “knows” that it’s manipulating regions.

Models are sometimes referred to as “Active Record” when the model represents a row in a database. In that case you might have some methods in the model that act as a facade for the database operations. An example here is CRUD operations, the business logic very likely does not care if “saving” a region results in an insert into the database or an update of an existing row. The business logic often only wants to ensure that the region is saved, a trivial abstraction of that logic would look like this:

class Region():
    """Represents a geographical region made of multiple points."""
    points = []

    def save(self):
        if self.is_new():
            return self.insert()
        else:
            return self.update()

View pattern, the V in MVC generally speaks to seperation of the display logic from everything else. An example of this might be an api that returns json today, and we might imagine that the ask is to add the ability to return yaml.

The business logic won’t change at all, just the manner in which the client views the response. If the application was written in such a way that the response is format agnostic up until the last second, then it would be trivial to return yaml instead of json, but if The Program is littered with json specific behavior, then The Program might need to be pretty much torn apart.

Controller is the traditional “catch-all” for business logic and it’s where you might look for “The Program”. That might looks something like this:

class WeatherController():
    def main(self):
        pass

That’s an unnecessarily simple example, something that shows off some business logic might look more like this:

from repos import RegionRepository

class WeatherController():
    def grow_active(self):
        for active_region in self.active_regions:
            active_region.expand_by(meters=3)
            active_region.save()

    def main(self):
        if self.is_growing:
            self.grow_active()

Then the entrypoint might looks like this:

from controllers import WeatherController

return WeatherController().main()

The interesting part here is that we don’t care where the region is stored, we don’t care what it’s shape or structure is, all that complexity is hidden elsewhere.

Now we have moved “The Program” from our main.py file and into an object called “Controller”, as such one might be tempted to ask: what does that get me? This structure gives the developer more control over the inputs and outputs from the controller, in the form of unit tests and mocks (TODO: link to mocks) which we will cover later on in the unit test section.

Repository patterns are for getting objects. You might often see these when working with a database or other data store. The business logic does not need to be burdened with where the objects are coming from, or how they are stored! To continue with our Region example, we might want to fetch all the regions that are “active” with the following

class RegionRepository():
    def fetch_active():
        # slug implementation, return a single region in a list
        return [Region()]

Additionally, let’s imagine that there are a dozen or so places where we “expand” a region, so let’s give the region a method to encapsulate that behavior:

class Region():
    def expand_by(meters):
        # TODO: for every given point in the shape, move the point x meters
        # away from the epicenter of the region
        return True

Let’s also combine the controller and repository pattern, by making the repository into a member of the controller:

class WeatherController():
    def __init__(self):
        self.region_repo = RegionRepository()

    def grow_active(self):
        for active_region in self.region_repo.get_active():
            active_region.expand_by(meters=3)
            active_region.save()

    def main(self):
        if self.is_growing():
            self.grow_active()

Now this is getting pretty close to pseudo code. Ideally, I want the main part of a controller to be something that a non-developer can follow.

This very clearly fetches the active regions (we don’t need to know how, or even from where, maybe it’s postgres, maybe it’s dynamo, maybe it’s S3! Those are all implementation details and don’t need to be exposed in the controller). For each of the active regions it then expands the region and saves the region back to wherever it is/was; we don’t need to care about how it expands the region (and perhaps that will change, or isn’t even defined right now! (which is the case with our example).