The writer of the original blog post jokes that actual engineers would probably hate the approach due to how imprecise it is - using estimated measurements, and timings based on senses (what it looks like, smells like, etc).

Well allow me to share an alternative, precise and (over)engineered approach to recipes

The Secret to Comedy*

I’d say I’m a ‘capable’ cook; I’m not skilled, but I can follow instructions.

Except, sometimes the instructions say things like “boil the rice… meanwhile”, and I’m not sure if I’m supposed to start the ‘meanwhile’ right away? Or, if I do that, will the sauce be ready too early? Or should I have already started?!

One solution to this is to plot everything out:

1. How long will each step take?
2. When does each step need to be completed (‘ready’)?
3. Start at the end, and work backwards

Example:

• rice takes 20 mins
• stir fry takes 15 mins
• we want them ready at the same time

0   5  10  15  20
|---+---+---+---| rice
    |---+---+---| stir fry

So, start the stir-fry 5 minutes after the rice. Easy.

This works for simple recipes with only a couple of things going on at once, where most of the things are ‘hands-off’ (like boiling rice), and where everything is combined at the end.

When that isn’t the case, we need a more complicated process. It breaks down into two stages:

1. Dependency graph
2. Timeline

Hot DAGs

First, we have to break down the recipe into discrete steps, and arrange them into a dependency graph.

This means drawing a ‘node’ for each step, with an arrow between the step and any steps that depend on it - you have to break an egg before you can fry your omelette.

[crack egg] -> [whisk egg] -> [fry]

What we end up with is something like a flowchart; in technical terms it’s a ‘directed acyclic graph’ (DAG)

Here I’ve labelled nodes with step numbers; you can also use words. I didn’t draw arrows on the lines because I’m lazy (it’s read top-down).

The point of the graph is, it makes it easier to keep things in the right order when we come to do the timeline.

At this point, I also

• assign approximate times to each step
• identify whether the step is ‘hands on’ (square) or ‘hands off’ (round)
• draw a squiggly line for steps that can be prepared in advance

It’s important to account for the time that every step will take, whether that’s time roasting in the oven, time to chop an onion, or time to heat a drizzle of oil in a pan.

The mistake (one of the mistakes) I made in the past was not accounting for steps that aren’t explicitly timed; the recipe says to add the rice to boiling water and cook for 20 mins, but it doesn’t say how long it takes to bring a pan of water to the boil (it takes a while).

Some of the timing are likely to be different for different people. For example, it takes me a good 5 mins to thinly slice an onion (badly), while I imagine a skilled chef could do it in less than a minute.

Sliding Timeline

This starts out the same as the original, basic approach

• start at the last step(s) and work backwards
• draw a block for each step
  • the block length represents how long the step takes
  • blocks have to be placed behind any steps they depend on (per the DAG)
  • indicate hands-off steps, e.g. with shading/colour

This first pass is likely to be ‘wrong’. In the above, we have multiple (independent) hands-on steps happening at the same time, such as 5c and 6a.

This is where the sliding comes in, and where it helps to use a tablet (or pencil and eraser)

For steps that can be prepared ahead of time, we can slide them as far away from their dependants as we want; an onion can be chopped the night before if need be. Steps which can’t (reasonably) be prepared ahead of time need to slide together with their dependants, e.g. heating oil and frying an egg in that oil.

Honestly, the sliding is not an exact science. To some extend, you just have to feel it out; play around with it until you get something that works/feels right

Here’s what I settled on for the above example

Once you have your timeline, you can rewrite the recipe in this new ordering, or timestamp the steps, or just work off the timeline diagram

(Note, since we start at the last step and work backwards, the timeline is read right-to-left.)

For Software Developers

The process I described is ‘fairly’ mechanical, so it seems natural to try and code it up.

You can see my attempt here - https://github.com/oatzy/recipe-optimiser

It… kind of works. It could be better. (Remember when I said it’s not an exact science)

For the input, we have to re-structure the recipe into json, with steps defined like

{
    "id": "2b",
    "duration": 15,
    "hands_off": true,
    "parents": [
        "1a",
        "2a"
    ],
    "instruction": "Put meatballs in oven"
}

Here’s an example of what the script then generates

[ 0 -  3]  ###                                             (1d) Grate the apple
[ 3 -  6]     ###                                          (1f) Chop potatoes into chunks
[ 6 -  8]        ##                                        (1c) Peel and grate the garlic
[ 8 - 11]          ###                                     (1g) Combine ingredients for meatballs
[11 - 16]             #####                                (2a) Roll mince into evenly-sized balls
[11 - 16]             XXXXX                                (1a) Pre-heat oven to 200
[14 - 19]                XXXXX                             (1b) Bring a large saucepan of water to the boil
[16 - 31]                  XXXXXXXXXXXXXXX                 (2b) Put meatballs in oven
[18 - 22]                    ####                          (1e) Slice the red onion
[19 - 39]                     XXXXXXXXXXXXXXXXXXXX         (2d) Boil the Potatoes
[22 - 24]                        ##                        (3a) Heat oil in a large frying pan
[24 - 34]                          XXXXXXXXXX              (3b) Add onions to pan, fry until golden
[31 - 32]                                 #                (2c) Remove from oven
[32 - 34]                                  ##              (5c) Heat oil in medium sauce pan
[34 - 36]                                    ##            (3c) Add balsamic vinegar and sugar, caramelise
[36 - 37]                                      #           (4a) Add wine stock, etc. to onions
[37 - 43]                                       XXXXXX     (4b) Bring to boil and simmer
[37 - 39]                                       ##         (5d) Add cabbage and stir fry
[39 - 40]                                         #        (5a) Drain potatoes
[40 - 43]                                          ###     (5b) Mash potatoes
[41 - 45]                                           XXXX   (5e) Cover, cook until tender
[43 - 45]                                             ##   (6a) Stir meatballs into gravy
[45 - 46]                                               #  (6b) Serve

(X are hands-off, # are hands-on)

Since recipes generally don’t have that many steps, an alternative approach might be to brute-force the ‘optimal’ recipe.

AI Interlude

As mentioned, the script requires converting the recipe into a particular JSON format. This can be a tedious process.

Then, it occurred to me that parsing prose into json is something an LLM might be good at.

To try this out, I wanted to use a local model. Problem is, my laptop doesn’t have a lot of RAM, so the only model I can run locally is a really small one.

Turns out a really small LLM takes a long time to do a bad job of parsing.

So I asked Gemini to help me improve the prompt, specifically for use with a small model - effectively asking an AI to dumb down the prompt for another, less intelligent AI.

It didn’t help.

And I don’t care enough to spend money on finding out if it works on a larger model, so… moving on.

When a Plan Comes Together

So having gone through all this planning, the whole cooking process runs smoothly now. Right?

The honest answer is: sometimes.

I actually used this approach when cooking Christmas dinner this year

And I have to say, it was pleasing to set a timer going and be able to say “at minute 5 do X, at minute 10 do Y”. A lot less stressful too.

It’s also good for identifying what you need to prepare ahead of time, and what you have time to do while other stuff is cooking.

But the “Cooking for Engineers” guy was on to something - with cooking, you can rarely be so precise.

For example, when a recipe says to fry on “medium high” heat, that’s not an exact setting. Besides which, there are natural variations in people’s equipment, some ovens run hotter than others. Plus, the ingredients may vary in size or shape in a way that subtly affects cooking time.

So sometime you do just have to rely on your senses, make adjustments on the fly, and not get so hung up on following the plan.

Even if you spent more time working out the plan than it took to cook the damn meal…

Chris

[the latest instalment in the “I started a new hobby and tried to solve it with maths” series]

*Timing

In a previous post, I used a simulation to predict how long it would take to complete the Champion of the Sinnoh Region secret mission. It said that I should expect to open around 250 booster packs.

I have now completed that mission! And I finally have the Garchomp emblem I was after

So how long did it actually take?

On my 253rd booster (1265 pack points), I finally pulled the one-star Gastrodon - the last one-star card I needed. I also had enough pack points to buy the missing two-star Cynthia (1250 pp), completing the mission.

I wish I could say I felt more excited when that happened, but mostly I was just relieved that it was done with.

Statistics

So, having opened 253 Space-Time Smackdown booster packs - of which about 200 were the Palkia variants - what did I end up with?

• 152/155 one- to four- diamond cards; 99/99 from Palkia
• 20/24 one star cards; 12/12 from Palkia
• 4/24 two-star cards (not counting Cynthia); 3 from Palkia
• 0 three-star (immersive) cards
• 1 crown (gold) card

Counting duplicates, I pulled 43 one-star cards, 5 two-star cards, and 1,265 cards all together.

Oh, and I also pulled a rare booster. Which was quite a shock! But it didn’t have any of the cards I needed, which made me feel very ungrateful.

How long would it take to complete the set?

Given the above, a natural next question is how long would it take to complete the set?

I’m currently missing 27 of the 207 cards from Space-Time Smackdown.

That doesn’t sound like a lot, but they’re mostly two-star and higher.

The simulation I wrote for the previous post is not suited to answering this question. So I wrote a brand new simulation - source code here

According to that simulation, I should expect to open about 1400 more booster packs to complete the set, which is more than 5 times the number I’ve already opened!

I think you’ll all agree with me when I say: “nah…”

For just the one- to four- diamond cards, I should expect to open about 200 more packs.

Sanity check

But wait a second, I’m only missing 3 of the one- to four- diamond cards. Would it really take that many more packs?

This feels a little off. But how can we check?

An easy set to validate the simulation against in Mythic Island, since it doesn’t have any booster variants.

The new simulation gives us a breakdown of expected packs opened by rarity:

• Crown: 309.2062
• Three Star: 89.547
• Two Star: 656.0182
• One Star: 113.0376
• Four Diamond: 136.5308
• Three Diamond: 87.1258
• Two Diamond: 57.6122
• One Diamond: 43.8046

Immersive

Any easy rarity to validate is the three star ‘immersive’, since there’s only one of them.

In the previous post, I showed how to calculate the probability of pulling a specific card from each booster. For the three star, this comes out to 1.12%

And since pulling the card is a Bernoulli trial, the expected number of boosters is 1/p = 89, which is roughly equal to the simulated number.

Diamond

There’s a 100% chance of pulling 3 one-diamond cards per booster. But for now, it’s easier pretend we pull one at a time.

If we pick one card from a set of 35 with equal probability, how long do we expect until we collect them all?

It seemed likely this was a well know problem with an exact solution, but I wasn’t sure how to identify it. This is the sort of thing AI is actually useful for

“Ah, a classic problem with a rather elegant solution! You’re asking about the Coupon Collector’s Problem”

When it comes to AI, it’s best to check their work. Wikipedia confirms this is indeed the problem I was looking for

The solution is E[t] = n * Hn where Hn is the harmonic number Hn = 1/1 + 1/2 + 1/3 + ... + 1/n

So, for the 35 common cards that comes to 145

And since we pull 3 per booster, that’s 48 boosters.

Again. this agrees quite well with the simulation results.

Star

The probability of pulling any one star is 12.595%, so we expect to pull one on average every 8 packs.

The expected number of pulls to get all 6 one-star cards is 14.7, so the expected number of packs is 8 * 14.7 = 117. Another close match.

And so

You can follow similar logic for the other rarities.

At this point you might be wondering, did I waste my time writing a simulation, when there’s an exact solution?

Maybe.

To be fair, things get more complicated when you take into account variants and the ability to buy cards.

Anyway, the simulation results seem sound. I guess I was just lucky with my pulls. I did also get some of the cards from wonder picks, which probably throws off the numbers too.

Free to play

Now we have the ability to simulate opening boosters, we can start to ask other questions.

For example, if I’m a free-to-play user opening 2 booster packs a day for a month (until the new expansion is released), what cards do I expect to pull? How many rare cards? How close do I get to completion?

Let’s take a more recent expansion: Eevee Grove

If I open 2 * 30 = 60 boosters, I would expect

• 64 of the 69 diamond cards
• 10 one-star or rarer cards (not counting duplicates).
• 1/2 chance of pulling the three-star immersive card
• 1/8 chance of pulling the gold crown card

By comparison, if I go premium and open 3 packs a day, 90 per month, I would expect

• 67 out of 69 diamond cards
• 13 one-star or rarer cards
• 2/3 chance of pulling the immersive card
• 1/6 chance of pulling the crown

As for myself, I did manager to collect all the Eevee Grove diamond cards after 75 packs, which is much less than the expected 185 packs.

But then, a couple of the rare and ex cards I got off wonder picks. Actually, after about 50 packs I was down to 3 cards missing, and just opening pack after pack with nothing new. I wouldn’t be surprised if it had taken another 100+ packs to get those last 3 if I hadn’t wonder picked them.

Feature creep

With a solid base simulation, it’s very easy to keep adding features.

For one, I re-implemented the original mission simulation. Reassuringly, the results agree with the old one.

In fact, I wrote it so that you can define any mission you like in json format. You can also give it an ‘initial state’, what cards you’ve already pulled, to see how much longer it’ll take.

I may have gotten carried away.

I won’t go into the details here, you can have a look for yourself in the repo.

Impossible dream

I’ve been drafting this blog for a while. I was planning to post it in line with the release of the latest set “Wisdom of Sea and Sky”, only to discover that the set introduces a new type of booster: “regular plus one card”.

Naturally, I couldn’t post the blog until I’d updated the simulation to account for these new boosters. It turned out to be a little fiddly.

But now, finally, to wrap up - below are the number of packs you should expect to open to complete each of the expansions that have been released at time of writing:

Considering the (current) release cadence of a new expansion every month, it is practically impossible to complete every set. It’d take a good while to complete just one set! (Unless you’re willing to spend A LOT of money).

Conclusion

In the previous post, I suggested that always opening the same booster would suck the joy out of the game. I definitely felt that, and wouldn’t recommend it.

But then, there’s a two-star full art Garchomp in the Triumphant Light expansion. Two two-star Garchomps in fact!

And because I don’t learn my lessons, I’m at it again. Based on the simulation, I won’t have them any time soon…

Chris.

[I wonder if I can simulate wonder picks]

I was spending a lot of time on Twitter (presently X). And I noticed some patterns, the ways different users behaved. And I wrote a blog post about it.

Some years earlier, when I was in secondary school, I created my first blog on MSN Spaces. It was called “Philosophical Sheep”, after a doodle I would draw on my exam revision notes

(this is a recreation)

I read a lot of ‘popular maths’ books at the time: Fermat’s Last Theorem, Music of the Primes; going back to the books that first sparked my interest in maths: Why Do Buses Come in Threes and its sequel How Long is a Piece of String

I thought that I’d like to do something like that - explain maths-y things in a way that was interesting and accessible to casual readers.

The first blog post was called “How to Weigh the Sun”, making use of Newton’s law of gravitation, angular momentum, and algebra.

The only person who read it (to my knowledge) was my sister. Her comment was along the lines of “you’re such a geek”. Which is hard to argue.

The second, and last, post on that blog was about houses of cards: how many cards do you need to build a house of a given height, and how high of a house you can build with a standard deck (5 rows).

The post ended

And basically, that’s it. Answers to questions you’d probably never ask.

That was where this blog got its name: Questions Nobody Asked

I ultimately went back to university (a different one), and completed my degree. But before that, I wrote a lot more blog posts.

At that time, I thought that the blog might even help me get a job (or at least that’s what I told my parents).

The name of this blog is kind of ironic - each post does start with a question that I asked. Something I’m curious about. Or sometimes the question is just “I wonder if I can”

And I try to answer those questions using the tools I have at my disposal: maths, physics, coding.

If I find an answer, and especially if the answer or the process of finding it is interesting, then I write about it. Because I want to share this cool thing I did.

And I want to explain it in a way that casual readers can understand, and hopefully find interesting.

That’s the reason I keep writing this blog.

When I applied for my current job, I included a link to this blog. On my first day, when I met my line manager, he said that what he found appealing, more than the topics of the posts (which aren’t really applicable to the job), was that it showed how I think through problems.

I’ve been working there almost 10 years now.

Granted, I haven’t written as many posts since then. But then, it’s a lot easier to find the time when you’re unemployed ;)

I started this blog 15 years and 136 posts ago.

I thought I should mark the occasion

Chris.

[I don’t think my sister reads the blog anymore]

Nearly 30 years later, and something of it is still deeply ingrained in my psyche. So when Pokemon TCG Pocket was released, I was an easy mark.

Secret Mission

Garchomp is my favourite Pokemon, so when the second expansion — Space Time Smackdown — was release with a full-art Garchomp, I had to have it.

And I got it!

But then I learned about secret mission: Champion of the Sinnoh Region

Completing this mission would earn me a Garchomp emblem, to proudly display on my player profile.

How hard could it be?

To complete the secret mission, I would need to collect:

• Garchomp (one star)
• Lucario (one star)
• Gastrodon (one star)
• Spiritomb (one star)
• Cynthia (two star)

Luckily, by that point I already had the Garchomp and Lucario, so I only needed 2 more one star cards and a two star card to complete the mission.

Getting cards

There are four ways to acquire a given card

1. pull it from a booster pack
2. wonder pick it
3. trade for it
4. buy it with pack points

We’ll come back to (1)

Wonder Pick

Wonder pick shows you sets of cards that other players pulled when opening a booster pack. You can chose a set, and the cards from that set are placed face-down and shuffled. You then pick one, and get a copy of whatever is revealed.

There are 5 cards to pick from, which means a 1 in 5 chance of getting the card you want, which is much better odds than pulling the card from a booster pack.

However, this only works if the card you want is available to pick at all. This becomes less and less likely over time, as new expansions are released and players open fewer boosters from the older expansions.

For that reason, this method is too unreliable to take into account.

Trade

The trade feature is trash, and we won’t dignify it with further discussion.

[edit]: In August 2025, the trade feature was significantly improved with the introduction of wishlists. This potentially helps with obtaining the one star cards, tho it’s still not possible to trade two star cards.

Pack Points

With each booster pack you open, you earn 5 pack points (pp), and with enough pack points you can out-right buy the card you want.

The number of points required to buy a card depends on its rarity. In my case, the relevance numbers are

• one star: 400 pp
• two star: 1250 pp

So all together, I would need… 2050 pp.

To achieve that many pack points, I would need to open 410 booster packs. Let’s put that in context.

If you’re a free-to-play player, you get to open 2 booster packs a day. That means it would take 205 days to get the required number of points, or almost 7 months!

If, instead, you pay for a premium pass (£7.99/month), you get to open a third pack per day, bringing it down to 137 days or about 4.5 months.

It’s not impossible, but it would suck the joy out of the game; always opening packs from the same expansion, and in most cases getting low value duplicates over and over.

There are various missions and bonuses which allow you to open more booster packs per month, but not enough to move the needle much.

Cold hard cash

What if you wanted to buy the booster packs with real money?

The best deal just now is to buy the 690 Pokemon gold bundle (500 paid + 190 ‘free’) for £79.99.

One booster pack costs 6 gold, so the bundle would give you 115 booster packs at a cost of about 69.5p per pack.

So, to open the 410 booster packs, I would have to pay £285 !

Yeah. That’s a lot of money to spend on an imaginary object.

Booster Packs

But wait, let’s go back to point (1).

Sure, pack points are a guaranteed way to get the cards you want, but if you’re opening 410 booster packs, that’s ample opportunity to just pull some of those cards. Right?

Offering Rates

Helpfully, the game shares it’s “offering rates” — the probability of pulling each card.

So, what’s the probability of getting one or more of the desired cards when opening 410 booster packs?

The relevant offering rates break down like so:

• each booster pack contains 5 cards
• in regular booster packs
  • the first 3 cards are always one diamond ‘common’ cards, so we can ignore them
  • for the 4th card
    • one star - 0.214%
    • two star - 0.041%
  • for the 5th card
    • one star - 0.857%
    • two star - 0.166%
• the probability of opening a ‘rare’ booster - 0.05%
  • in a rare booster there are 5 rare cards
  • all rare cards (including one and two star) have the same probability - 3.846%

Note, these offering rates apply to the Space Time Smackdown booster packs. Other expansions may have different rates.

Pull probabilities

Let’s start by looking at the probability of pulling a one star card from a regular booster

It’s easiest to calculate the probability of NOT pulling the card (not card 4 and not card 5), then subtract from 1

p(one star) = 1 - (1 - p(one star|4)) * (1 - p(one star|5))
            = 1 - (1 - 0.00214) * (1 - 0.00857)
            = 0.0107

Calculated the same way, the two star probability is p(two star) = 0.00207

For rare booster packs, there are 5 opportunities for pulling a given card, with the same probability each time, and a card may appear more than once

p(rare) = 1 - (1 - 0.0384) ^ 5
        = 0.178

The probability is the same for both one and two star cards.

Combining the probabilities for rare and regular boosters, we get

P(one star) = 0.0108
P(two star) = 0.00216

Opening boosters

We want to know the probability of pulling one or more copies of a card while opening a large number of booster packs

As before, it’s easiest to come at it via the probability of not pulling the card in any booster

p(card, N) = 1 - (1 - p(card)) ^ N

As previously mentioned, I would have to open N = 410 booster packs to get the required number of pack points, so we have

p(one star, 410) = 0.988
p(two star, 410) = 0.588

In other words, I’m almost certainly going to pull one or both of the one star cards, but the two star card is only slightly better 50:50

The probability of pulling all three is 58.7%

But, if I do pull a one star card, I won’t need to buy it with pack points, so then don’t need to open as many boosters. But opening fewer boosters decreases the probability of pulling the card. Is there some sweet spot?

Simulation

How many boosters should I expect to open before I either pull all the cards, or else have enough points to buy the cards I haven’t pulled yet?

This is more difficult to calculate from pure probability, so I used a simulation instead — source code here.

The results looks like this

# Average boosters opened - 205.0471

# Most likely number opened - 250 (0.5076)
# Probability above most common - 0.0789

# Average copies of each card
 - Cynthia: 0.4393
 - Spiritomb: 2.2016
 - Gastrodon: 2.2031

# Probabilities of card sets
 - Cynthia: 0.0116
 - Spiritomb: 0.0137
 - Gastrodon: 0.0159
 - Cynthia, Spiritomb: 0.0925
 - Cynthia, Gastrodon: 0.0928
 - Cynthia, Gastrodon, Spiritomb: 0.2265
 - Gastrodon, Spiritomb: 0.547

I should expect to open ~205 booster packs before getting all three cards.

The most likely outcome is I’ll pull the 2 one stars and will have to buy the two star, with probability about 50%. The probability of having to open more than 250 packs is only 7.89%

There’s roughly a 1 in 5 chance of pulling all 3 cards before getting to the required pack points. The probability of pulling none of the cards is so unlikely it didn’t happen in 10,000 simulations (but it’s not impossible).

If we plot a histogram, we get a clearer picture of what’s going on

There are four clear peaks corresponding to 80, 160, 250, and 330 packs — or 400 pp, 800pp, 1250 pp, and 1650 pp. Those numbers should look familiar.

Head start

When I starting this mission I had already opened about 100 boosters (500 pp).

Taking that into account, the results become

# Average boosters opened - 142.9071

# Most likely number opened - 150 (0.4728)
# Probability above most common - 0.2493

# Average copies of each card
 - Cynthia: 0.3124
 - Gastrodon: 1.5405
 - Spiritomb: 1.5657

# Probabilities of card sets
 - (none): 0.0007
 - Spiritomb: 0.0464
 - Gastrodon: 0.0482
 - Cynthia: 0.056
 - Cynthia, Gastrodon, Spiritomb: 0.078
 - Cynthia, Gastrodon: 0.0826
 - Cynthia, Spiritomb: 0.0911
 - Gastrodon, Spiritomb: 0.597

Now the expected number of boosters is 145. Adding that to the 100 I already opened comes to 245, which is more than the expected number when starting from zero.

But as before, I will most likely have to buy the two star card with pack points, and it’s highly likely that I will pull the one star cards in the meantime.

Gotta catch ‘em all

In case you were wondering, the expected number of packs to get all 5 Champion of Sinnoh cards is 215 (1075 pp), which is much less than the 2850 pp required to buy the cards. It’s less even than the cost of just the two star card.

But that calculation isn’t quite correct — Lucario is only available in the Space Time Smackdown Dialga boosters, while the rest are in Palkia. That means, if we only open Palkia boosters, there’s zero chance of pulling Lucario.

Figuring out the optimal strategy in that case is left as an exercise for the reader.

Conclusion

No matter how we come at the calculation, the most likely outcome is that I’ll have to buy the two star Cynthia with pack points, but there’s a good chance of pulling the one star cards. This comes down to two star cards being much rarer than one star cards; the probability of pulling Cynthia in any given pack is only about 1 in 500.

Since I started drafting this post, I’ve gotten up to 900 pack points, and I managed to pull one of the one star cards (Spiritomb) along the way.

Re-running the simulation, the expected number of boosters in now down to 85.

Still some way to go…

Chris.

Oh, and if you want to add me, my user name is ‘oatzy’ and my friend code is 0625-2776-5308-5489

I’m too much of a coward for Versus battles (PvP), but if you offer me a trade I will offer you something in return, even if I don’t need what you offered.

Let’s focus on the dice - what probabilities can we generate with a dice roll?

• 1 in 6 is easy, if you roll a 1 you win
• for 1 in 2, if the result is even – 2, 4, or 6 – you win
• for 1 in 3, if the result is divisible by three – 3 or 6 – you win
• 1 in 4 is less obvious, but since 1/4 = 1/2 * 1/2, we can roll the dice twice and win if both rolls come up even.

But what about 1 in 5 ?

Keep rollin’ rollin’ rollin’ rollin’

Let’s start by stubbornly pretending that the dice only has five side.

• If you roll a 1, you win.
• If you roll 2, 3, 4, or 5, you lose.

Problem solved.

But wait! What if you roll a 6?

In that case, let’s try rolling again – same rules apply – and keep rolling until we win or lose.

What’s the probability of winning?

• the probability of winning – rolling a 1 – on the first roll is p(1) = 1/6
• the probability of winning on the second roll is the probability of rolling a 6 on the first roll (1/6) multiplied by the probability of rolling a 1 on the second roll (1/6)
  • p(2) = 1/6 * 1/6 = 1/36
  • the probability of winning on first OR second roll is the sum of the probabilities
    • p(1||2) = p(1) + p(2) = 1/6 + 1/36 = 0.19444..
• the probability of winning on the third roll, following the same logic, is
  • p(3) = 1/6 * 1/6 * 1/6 = (1/6)^3 = 1/216
  • the probability of wining on the first, second, or third roll is then
    • p(1||2||3) = 1/6 + 1/36 + 1/216 = 0.19907..

This is looking promising – the probability is getting closer and closer to 0.2 (= 1/5)

So, the probability of winning at all, in any round, is the sum of the probabilities of winning on any of a potentially infinite number of rounds

p = p(1) + p(2) + p(3) + ... p(i) + ... + p(inf)

This a a geometric series, for which there’s a simple formula

And substituting x = 1/6 we get… p = 1/5

tada!

Granted, this approach could result in rolling dice forever, without ever winning or losing (by rolling infinite 6s). But the probability of that happening is diminishingly small; the probability of rolling just four 6s in a row is less than 1%

Roll again

Now, we might well wonder: Was this a fluke? Or does this work for other numbers?

Let’s try for 1 in 4.

• roll 1 -> win
• roll 2, 3, or 4 -> lose
• roll 5, or 6 -> re-roll

The probability is calculated the same as before

• the probability of winning – rolling a 1 – on the first roll is p(1) = 1/6
• the probability of winning on the second roll is the probability of a re-roll (2/6 = 1/3) multiplied by the probability of winning on the second roll (1/6)
  • p(2) = 1/6 * 1/3
• the probability of winning on the third roll is p(3) = 1/6 * 1/3 * 1/3 = 1/6 * (1/3)^2
• and so on

Adding up, this time, we have

Just like we wanted.

[NOTE: this time, the sum is from i=0 rather than i=1]

Prove it

And since we’re on a roll (pun intended), let’s see if this always works.

Suppose we have a dice with N sides, and we want to roll for a 1/r probability, with integer r <= N

Probability of a re-roll is

And following through the same logic as before

QED

Can you take me higher

So with our six-sided dice, we can get probabilities for 1/2 to 1/6. But what about 1/7?

Our algorithm only lets us represent probabilities 1/r for integer r <= N, where N=6 for a standard six-sided (d6) dice.

Sure, we could go up to a dice with more sides, like an octahedral (d8). But I don’t have one of those.

How about this – roll the d6 dice twice; each roll represents a digit in a two-digit, base-6 number.

We’ll treat 6 as zero. So if we roll [2][5] we have hexary (sexary?) number 25 = (2*6) + 5 = 17 in decimal

Now we can represent 36 values. Can you see where this is going?

So we roll our dice twice

• if the result is two 6s – [0][0] = decimal 0 – then we win
• if the result is decimal 1 to 6 – hexary [0][1] to [1][0] – we lose
• otherwise, re-roll.

As previously demonstrated, we know this will converge on 1/7. Job done.

Is this practical? Not really. The re-roll probability in this case is 29/36 ~ 80%, which means potentially a lot of re-rolls.

But, it does work.

Great expectations

Actually, how many rolls would we expect?

We can think of rolling and re-rolling as a Bernoulli trial, where the probability of the trials ending (not re-rolling) is given by p(end) = 1 - q = r/N

The expected number of trials is then given by T = 1/p(end) = N/r

For the two dice 1/7 example, that comes out to 36/7 = 5.1428.. rolls (or 10.28.., since we’re rolling two dice for each trial).

By comparison, the single dice 1/5 probability would expect only T = 6/5 = 1.2 rolls

We can get an improvement on two dice for 1/7 by instead pairing the dice with a coin – flip a coin for the first digit and roll a dice for the second. This gives us 12 possible values and means we expect only 12/7 = 1.714.. trials (rolls and flips), which is much more reasonable.

Flippin’ heck

There’s question that had been floating around in my head for a while, before the dream – can we get a probability of 1 in 3 with one or more coin flips?

Following the same algorithm as for the dice, we can flip two coins (or one coin twice). The result is interpreted as a two digit binary number – heads = 0, tails = 1, giving us 4 possible values.

• If we get two heads – (0)(0) – we win!
• If we get one head and one tail – (0)(1) or (1)(0) (1 or 2 in decimal) – we lose.
• If we get two tails – (1)(1) (decimal 3) – flip again.

Repeat until the game ends. Surprisingly simple.

The expected total number of coin flips is 2 * N/r = 2 * 4/3 = 2.66..

So there you go - you can generate any probability with a single dice or coin.

Wait, why are we only half way through this post?

Re-numeration

All the probabilities we’ve looked at so far have been one-in-X. What about higher numerators? Can we get 2-in-5 with a dice?

If the trick worked for 1/5, why not 2/5?

• roll a 1 or 2 -> win
• roll 3, 4, 5 -> lose
• roll 6 -> re-roll

Now

• the probability of winning on the first roll – rolling 1 or 2 – is p(1) = 2/6 (= 1/3)
• the probability of a re-roll is 1/6, so the probability of winning on the second roll is p(2) = 1/6 * 1/3

You should know the words by now

Likewise, if we have an N sided dice and want probability p = a/b, where integers 0 <= a < b <= N

The probability of winning on the first roll is a/N and the probability of a re-roll is q = (N - b) / N

So the total probability of winning is

QED again.

Okay, now we’ve covered everything. Right?

Stop making sense

“But wait…“, you say, “what about irrational probabilities?”

“Oh…“, I say, “right…“ :/

An irrational probability would be something like 1/pi or 1/e. Can we roll that with a six-sided dice?

So far, we’ve been assuming the probabilities are the same each round – for example, if we roll a 1 we win, if we roll a 6 we re-roll, regardless of how many rolls we’ve already thrown.

What if we change things up. Let’s stick with re-rolling when we get a 6, but vary the win condition on each round.

For example, suppose we did the following

• if we roll 1 on the first round we win
• if we roll 1, 2, 3, 4, or 5 on the second round we win
• if we roll 1 or 2 on the third round we win
• and so on

The probability is then

And we can see this inching upwards

p(1)          = 0.1666..
p(1||2)       = 0.3055..
p(1||2||3)    = 0.3148..
p(1||2||3||4) = 0.3179..
...

towards p = 0.3183... = 1/pi

All your base

We can generalise this approach like so – we have an N sided dice, and to win on the i-th throw, we need to roll ai or lower, where ai is an integer 0 <= ai < N

The total probability of winning is then

And what we’re describing here is actually the base-N representation of p. It’s correct by construction, so no proof required.

The coefficients ai can be calculated like so

def tobase(p, N):
    while p > 0:
        a, p = divmod(p * N, 1)
        yield int(a)

# list(itertools.islice(tobase(1/math.pi, 6), 10))

(for p < 1)

So, for example, the first 10 values of ai for 1/pi in base-6 are

1, 5, 2, 4, 3, 1, 0, 2, 2, 1, ...

and for 1/e in base-6

2, 1, 1, 2, 4, 3, 4, 4, 1, 1, ...

The same works for coin flips as well – 1/pi in base-2 is

0, 1, 0, 1, 0, 0, 0, 1, 0, 1, ...

Is this practical – memorising an infinite sequence of coefficients? Almost certainly not.

Sixes and sevens

Incidentally, this approach does also works for rational numbers.

And it gives us an interesting, alternative way of getting 1/7 from a dice roll.

The coefficients (ai) for 1/7 in base 6 are

0, 5, 0, 5, 0, 5, 0, 5, 0, 5, ...

alternating 0 and 5

In terms of dice rolls, 0 means we lose if we roll anything other than 6 (re-roll), and 5 means we win if we roll anything other than 6

This translate to:

• Roll the dice until you get anything other than 6
• If you rolled an odd number of times, you lose.
• If you rolling an even number of times, you win.

For example, if you rolled 6, 6, 3, then you stopped on the third roll, which is odd, so you lose. Sorry.

But if you rolled 6, 1, that’s an even number of rolls, so you win. Woo!

This is much easier to comprehend than the previously discussed 1/7 methods, and it has an expected number of rolls T = 6/5 = 1.2. So it would be my preferred method.

There’s a similarly nice, repeating pattern for 1/7 in base 2

0, 0, 1, 0, 0, 1, 0, 0, 1, ...

This means we can flip a coin until we gets a heads, and if the number of flips was divisible by 3, we win.

Back to dreaming

Now are we done?

What about imaginary numbers? Can we do imaginary probabilities?

…

Merry Christmas.

Chris.

[yes, I know the singular of ‘dice’ is ‘die’]

trakt

One of the first options, on creating a letterboxd account, is importing data from other services - one of them being trakt.

I don’t use trakt directly. For many years, I’ve been using SeriesGuide to keep track of TV shows. I used this in combination with trakt because I could then use the trakt api to get a list of new episodes ‘today’ for completely legal reasons.

SeriesGuide also lets you track films.

So I exported my data and imported it into Letterboxd - roughly 950 films

And what’s interesting about this is trakt had recorded when I’d watched each film, and Letterboxd displays this in a ‘diary’.

This is quite pleasing.

The only problem is, the earliest film I had recorded in trakt was Interstellar in November 2014.

I could do better. I could go farther.

Too far? Who’s to say…

eTickets

There were a couple films - Spider-man: No Way Home and Venom: Let There be carnage - which I hadn’t logged. Luckily, I have etickets, so just had to search my emails to get the exact date.

But I only started buying tickets online in 2021, when movies started to come back after covid

[the first film I saw, post-covid, was Black Widow]

Ticket stubs

My next thought was

you know… I’m the sort of person who would keep old ticket stubs; and I bet I know where they are…

And reader, I am, and I did

I was surprised by how far back they go - the oldest is for The World is Not Enough in 1999.

The most recent is for Amazing Spider-Man 2 in April 2014.

I was also surprised by some of the films, for example I have no memory of seeing Lilo & Stitch at the cinema, but I have a ticket stub that says otherwise.

On the other hand, I was disappointed by how much was missing. I saw Pokemon: The Movie 2000 at the cinema - I have the promo cards to prove it - but sadly no ticket stub to say when.

Notion

While looking over my watch history from trakt, I noticed a weird empty spot from November-December 2022. This didn’t look right, because I’m in the habit of watching movies every weekend.

Maybe I was lazy those two months, or maybe trakt was having some issues? I dunno.

I started using notion for note keeping in 2020, and I had a movie watching list in there. And when I watched a movie, I would check a box to hide it.

Now, I didn’t record when exactly I watched the films, but notion does auto-track the last edit time on each item - such as when a checkbox is checked.

I was a little lax about when I checked the box on notion - sometimes it would be days later - but it was enough to tell me which films I was missing from that period.

[I also did star ratings in notion, which I could manually copy across]

Netflix

Netflix keeps a watch history, and even better lets you download it as a CSV file

But not a CSV that can be imported to letterboxd.

It only tells you title and watch date, and annoyingly combines tv shows and movies.

The quick and dirty work around was to grep out words like “Episode”, “Season”, “Series”

Starting from the bottom, the earliest entry was Cloudy With a Chance of Meatballs in September 2014

I only had to add a few entries from 2014 before I hit the movies I’d recorded in trakt.

Prime Video

Amazon also keeps a watch history, but you have to make a formal request and wait a couple of days for the download.

It’s CSV again, and with a similar issue as netflix of mixing tv and movies. And to make matters worse, it seems to record trailers, including the ones that auto-play in the app

I only got Amazon Prime when I entered full time employment (end of 2015), and 2015 onwards I had in trakt. So most of it was already covered.

There were a couple of early films I hadn’t logged. The first film I watched on Prime was It Follows on Boxing Day 2015.

I’ve watched films on other streaming services, but none before 2015. I did have to look up one on Crunchyroll, which sadly does not provide a downloadable history.

LoveFilm

…was a DVD rental service, like the UK equivalent of the original incarnation of netflix.

I signed up for an account in 2010. LoveFilm was later bought out by Amazon, who ultimately discontinued the service in 2017

Unfortunately, I don’t have any record of what films I rented or when. I searched my email and found two “Your next rental is on the way…” messages (Twelve Monkeys and Creation)

Apparently, I deleted all the subsequent emails.

IIRC, my subscription was 3 DVDs a month, and I had the account ~7 years - totalling about 250 films, which is a pretty major loss of data.

When I was requesting my Amazon data, I also enquired about my LoveFilm rental history. But they just sent me another copy of my prime video data (which doesn’t include Lovefilm). So one has to assume that data no longer exists.

Foursquare

Clutching at straws, I was starting to wonder if I could use my phone’s location data (google maps) to figure out when I visited (or was near) a cinema. Then I remembered I had something even better.

There was a period, 2010-2014, when I used foursquare to record everywhere I went - including cinema visits, and including what film I saw

(2014 conveniently being when I started tracking films in SeriesGuide, more or less)

And the data still exists on the foursquare website (tho it was hard to find the right login page from google). Once I reset my password, it was a simple matter of filtering on “Category: Movie Theatre”.

But it turned out I didn’t log the specific film for my first 3 recorded cinema visits.

For the first one, I added the comment “Mind suitably braced for being blown…” and safely assumed this was Inception (it aligned with the release date)

Similarly, I remembered seeing Source Code around this time, and it wasn’t mentioned in any of my other check-ins. And again the release date lines up.

For the last one I wasn’t sure what it could be. I had a scroll through old blog posts from around that time for clues, and lo and behold - “Tron Legacy: A Review”

Incidentally, there are a couple other old posts on this blog for film reviews

Twitter

Between 2009 and 2015, I was a prolific tweeter, so it seemed likely I would have mentioned watching a bunch of films.

To check, I downloaded my twitter data. I can find quite a few relevant tweets by grepping: ‘watch’ (watched, watching), ‘cinema’, ‘film’, ‘movie’, ‘see’ (seen), ‘saw’ …

Watched 'Non Stop'. Was better than I expected. As if any movie where Liam Neeson plays an action hero could be bad.
-- Sat Aug 16 21:55:48 +0000 2014

[I guess a lot can change in 10 years :p]

But this isn’t foolproof, it’s hard to pick out all mentions or allusions to films

Who is this old joker? #BucketList
-- Sat Sep 11 21:06:15 +0000 2010

Maybe one could use machine learning to pick out movie titles.

Then there’s this spicy take

I've got nothing against subtitled film, but they're a pain in the arse if you like to watch films while you eat. #RemakeAllTheForeignFilms
-- Sun Feb 26 13:59:20 +0000 2012

I remembered I was referring to The Extraordinary Adventures of Adele Blac-Sec - or as I typed it into google, since I couldn’t remember the title, “french comic movie adel”.

Side note: one set of tweets shows me starting to watch Michael Bay’s Transformers, and giving up after ~1 hour. Should that count?

MySpace

Before twitter, I had a MySpace blog, and from 2007-2008 I updated it almost daily.

In a way, the blog was like a proto-twitter, in that it was a collection of disconnected thoughts, autobiography, and random quotes and song lyrics.

I kept a backup of all my posts when MySpace went out of fashion; the live blog no longer exists.

So I fetched the backup and, as with the tweets, grepped for pertinent words

2 Aug 2007
    [Another chapter of my pathetic life]

    I'm disinclined to acquiesce to your request.

    Means no.

    Due to the lack of decent TV before the setting of the sun, I've rediscovered my DVD collection.
    [I've got perhaps more Johnny Depp DVDs than is normal for a boy of my age]
    Today I watched 'Pirates of the Caribbean: Curse of the Balck Pearl' [longest name ever], and tomorrow I intend to watch 'Deadman's Chest'.

    Other movies I've watched over the past week [most on TV]:
    'Edward Scissorhand', 'Charlie's Angels', 'Secretary', the back end of Napolean 'Dynamite', 'Charlie and the Chocolate Factory', 'Constantine', 'Dodge Ball'...
    can't think of any others off the top of my head.

Frustratingly, a few time I mention going to the cinema, without saying which film I saw

I went to the movies. But I don't need to tell you about that, 'cause most of my 'core' readers were there.

[Reading back over those blogs, I was surprised by how often I wrote about getting drunk. Ah, to be 18 again.]

Memory?

It’s frustrating, knowing you saw a film at the cinema, but not being able to pin down an actual date.

I mean, it’s going to be within, say, 2 weeks of release. But I’m not sure I can allow myself that big a margin of error.

Compromise

Of course, I’m never going to be able to figure out when I saw every film I ever saw.

Beyond a certain point I just have to settle for logging that I saw a film ‘at some point’.

Even this is harder than it seems.

Given a specific film, it’s fairly easy to say whether or not I’ve seen it. So one approach was to browse through popular films, directors, actors. But there are going to be things you miss from this approach - the unpopular, the obscure.

The harder approach is trying to randomly remember films I’ve seen.

[“Ooh, what was that vampire film with Saoirse Ronan?”]

This mostly works by association - I’ve seen this film, oh that reminds me of this other film.

I’ve reached the point where I think I’ve covered maybe 99% - I struggle to think of new ones.

Then occasionally I’ll be like “wait, how did I forget Who Framed Roger Rabbit?”

Fool’s Errand

One thing I’ve learned from the films I had concretely recorded - there are films I’ve seen that I have no memory of seeing. Which means there are probably films I don’t remember, which I don’t have any record of.

And if you don’t remember seeing a film, does it really count?

To date, I’ve managed to log ~1,800 films, of which ~1,200 have a definitive watch date. So I managed to pin down ~250 films in addition to those from trakt.

Which is not too shabby. Not an entirely fruitless endeavour.

Chris.

[It really is a shame I couldn’t get that LoveFilm data]

My hair’s been getting thinner for a while now, so I thought I’d try going all in.

I’m still getting used to it.

But one thing I did expect, was that my head would feel colder now. So I needed a hat.

I looked over my save crochet patterns and found this one - Widcombe C2C crochet hat

It uses a crochet technique called ‘corner-to-corner’ (c2c) - effectively the fabric is made up of squares, worked in diagonal rows.

The design as written is okay, but a little generic. I wanted something more meaningful.

The c2c effectively gives us a grid, and in the original design the middle band is a repeating pattern of 6x10 square blocks.

So the question was, what could I do with this?

Well a row of 6 squares gives us 6 bits, allowing us to represent 64 values - more than enough to encode characters of the (latin) alphabet.

Meanwhile we have 10 rows, and it just so happens that my name contains 10 letters - CHRIS OATES

In fact, we only need 5 bits to represent the alphabet, so we have one spare. I considered leaving it blank, or using it to represent uppercase/lowercase. But none of those looked very good, so I just used repeating blocks of 5x10.

The complete design looks like this

And here’s the completed hat

Following the pattern diagonally was… fun.

But at least my head isn’t cold anymore.

Chris.

I got maybe 20 rows into it, then got bored. I ended up frogging it.

But now I have these three balls of super-soft aran yarn, and what to do with them?

I wondered if I could make the pattern more interesting by cycling - blue, grey, cream, blue, grey, … - but that’s not much more exciting.

I had the thought that I could completely randomise the colours - for each row, pick one of the three colours at random (roll a dice, even).

I like the idea of encoding information in arts and crafts. In my temperature blanket I encoded temperatures as different colours, and months as white and coloured rings representing binary numbers.

And here I have 3 colours - a base 3 encoding - and, conveniently, three base-3 bits (trits?) can represent 27 values - enough for all 26 letters, plus a whitespace character.

A simple plan

So suppose we want to make a scarf.

We have three colours, which we might map as

0 -> cream
1 -> blue
2 -> grey

The most simple mapping of characters to numbers is to make A=1, B=2, C=3, etc. saving 0 for the whitespace character.

As mentioned, we’re representing 27 characters so we need 3 rows per character.

If we take my name, CHRIS OATES, and translate it to ternary as above, we get

010 022 200 100 201 000 120 001 202 012 201

Or in colours

Not too bad. But doesn’t it feel a little… unbalanced?

Ignoring the space character (000) we have 14 cream, 7 blue, and 9 grey.

Cream (zeroes) are way over-represented.

Every possible 3 bit value is represented in the simple encoding, so in a random sequence of letters, we would expect all 3 colours to appear with roughly equal frequency… except, English isn’t a random sequence of letters.

Popularity contest

Some letters appear in the English language a lot more than others.

So maybe we want to account for this in our encoding to get a more even balance of colours.

For example, if we assigned E as 222 then 2 would end up very much over represented. It would be better to assign E a value with all three bits like 012

The 3 bit ternary numbers can be split into 4 broad groups

• 6 are one of each digit, e.g. 012
• 3 are all the same digit (of which 000 is our space)
• 6 are two the same, split up, e.g. 010
• 12 are two the same together; 6 left e.g. 001, and 6 rights e.g. 100

So the obvious thing is to assign the 6 one-of-each codes to the six most common letters (E, T, A, S, …), and likewise the 2 all-the-same codes to the two least common letters (Q, Z)

The six two-split codes can be assigned to the 7th-12th most common letters, on the basis that colours clumped together are less pleasing.

How the actual codes within those groups are assigned is largely arbitrary (more on that later).

Here’s one possible encoding ¹

e -> 012   t -> 120   i -> 201
a -> 021   n -> 102   m -> 210

s -> 010   u -> 121   r -> 202
w -> 020   d -> 101   k -> 212

g -> 011   o -> 122   h -> 200
v -> 022   f -> 100   l -> 211

p -> 001   j -> 110   b -> 221
x -> 002   c -> 112   y -> 220

z -> 111   q -> 222

And here’s my name again, using this encoding

112 200 202 201 010 000 122 021 120 012 010

and in colour

Hmm… this looks even less balanced than before.

However, this time we have 11 cream, 9 blue, and 10 grey, not counting the space block. Almost perfectly balanced.

So what’s the issue? The problem this time is that colours aren’t well distributed. We have lots of repeated digits (10, including the white space block), and the blues are biased towards the right.

Going wider

I said in the previous section that how the codes are assigned to letters within each group is arbitrary.

We can do better than that.

Suppose we look at pairs of letters - can we assign codes so as to maintain ‘evenness’ across pairs? And will that make the encoding more even across whole texts? ²

For example, in the above we assigned Q=222. Now Q is (almost) always followed by a U, so it would be foolish to assign e.g. U=220 as we would then get a run of five 2s in a row 222 220

It would be better to give U a code with no 2s, say 101 -> 222 101

We should also take into account the white space character. I want to keep white space pinned as 000, so letters which tend to appear at the end of words should not end with 0s, and letters which tend to appear at the start of words should not start with 0s.

For example, if we assigned Y=100 and D=001, we might suddenly discover a run of 7 zeros - 100 000 001

A more perfect encoding

To figure out the ‘best’ encoding, we need a way to quantify or ‘score’ each possible encoding.

We did this implicitly, above, for the 3 bit codes - i.e. the all-different codes are higher scoring than the all-same codes because they have a higher variety of digits/colours.

Likewise, the two-same-split codes are higher value than 2-same-together codes because the colours are more spread out.

We just need to extend that logic to pairs of 3 bit codes (or equivalently, 6 bit codes) and come up with an empirical ‘score’ function, alike

score(123, 213) = 1
score(111, 111) = 0

For scoring the ‘spread’ of digits, we can

1. look at each pair of bits
2. add 1 if different, else add 0 if same
3. divide by the total number of pairs (5)

e.g.

220 021 -> 22, 20, 00, 02, 21 -> 0 + 1 + 0 + 1 + 1 = 3 -> 0.6

For ‘balance’ we can use entropy with a base of 3

- n0/6 * log3(n0/6) - n1/6 * log3(n1/6) - n2/6 * log3(n2/6)

where n0 is the number of 0 digits, etc. ³

For example, in the worst case scenario, where all the digits are the same, we get

6/6 * log3(6/6) + 0 + 0 = log3(1) = 0

and in the best case, with two of each digit, we get

3 * ( -2/6 * log3(2/6) ) = (3 * -1/3) * log3(1/3) = -1 * -1 = 1

So now we can smash (multiply) the balance score together with the spread score to get a combined score for a given pair of codes, e.g.

s(121, 102) = (4/5) * (log3(6)/6 + log3(2)/2 + log3(3)/3)
            = 0.8 * 0.921
            = 0.736

Now we have to take into account how those codes are assigned to actual letter pairs. That is, it’s better to assign high scoring codes to common pairs (EA) than to uncommon pairs (ZW)

Similar to how we can calculate the frequency of individual letters in the English language - by counting occurences in a text - we can also calculate the frequency of letter pairs in English.

We can then use these frequencies as a weighting, by multipling the letter pair frequency by the score of the assigned codes.

So if we come up with a possible mapping, we can calculate the score for each possible letter pair and take the sum, giving us the total score for that mapping

total score = sum [ f(c_i, c_j) * s(c_i, c_j) ]

For example, in the A=1 encoding, we would calculate

total score = f(a, a) * s(001, 001) + f(a, b) * s(001, 002) + ... + f(z, z) * s(222, 222)

This is our metric for comparing and finding the best encoding.

The best enough

What now?

Here’s the tricky part - we have a method of scoring any given encoding, but how do we find the ‘best’ one?

The problem is, there are 26! (factorial) possible ways of assigning codes to characters - 4 x 10^26 - that’s more than something something in the universe! 🤯

Suffice to say, finding an optimal solution ⁴ is not viable.

Instead, we can look at a heuristic solution.

The approach that worked the best for me was

1. generate a random starting mapping
2. for each character in the mapping, find the swap which produces the best score
3. repeat 2 until the score doesn’t increase anymore
4. repeat 1+2 multiple times and pick the best scoring result

You can find my code here. I won’t go into detail about the code in this post, as it is already quite long. I have provided code comments.

For my experiments, I grabbed a copy of Frankenstein off Project Gutenberg to generate the bigram frequencies. This gives us 426_160 letter pairs

As the baseline, the scores for the mappings we already discussed are

• A=1 -> 0.51581
• letter frequency-based -> 0.58679 ⁵

I tried multiple runs and the best score I got was 0.66869

Here’s what that mapping looks like

a -> 210   b -> 122   c -> 022
d -> 121   e -> 021   f -> 221
g -> 011   h -> 012   i -> 101
j -> 100   k -> 001   l -> 020
m -> 220   n -> 202   o -> 120
p -> 110   q -> 222   r -> 201
s -> 212   t -> 102   u -> 010
v -> 002   w -> 211   x -> 200
y -> 112   z -> 111

I tried running it a few times, and it consistently turned up this mapping as the best, which suggests it’s the most optimal. Or perhaps it’s the most optimal solution that this method can generate; maybe a different optimisation method could generate an even better solution. ⁶

Here’s what my name looks like with the optimised mapping

022 012 201 101 212 000 120 210 102 021 212

and here it is in colours

Doesn’t that look so much more balanced?

Here we have 11 cream, 10 blue, and 12 grey, this time including the space block since it was part of the optimisation. It’s almost perfectly balanced, and this time it also has a much better spread of colours; we have only 5 repeated digits (of which 2 are in the space block)

Spinning out

This idea can be extended to different bases. For example, with 4 colours and 2 rows per colour you get 64 possibilities - enough for upper and lower case, 10 digits, 1 space, and 1 left over for a period (or exclamation mark!)

The principle remains the same, you just have more character combinations to deal with.

Similarly, I’ve been talking about a scarf, but this would work just as well for a blanket. For, what is a blanket if not a really wide scarf?

Alternatively, we could construct a blanket of squares - similar to my temperature blanket - with one block per character comprised of 3 bands of colour representing the 3 bits.

This can make for a much more striking design

Tho in the squares case, the criteria for what is optimal is slightly different.

For example, in the above the second and third to last squares are 202 012 which has no repeated digits

But with squares

22222 22222
20002 21112
20202 21012
20002 21112
22222 22222

the last digits ⁷ - the outer rings of 2s - are adjacent.

And that’s only thinking one-dimensionally. For a blanket, we’re going to have a grid, with the text split across multiple rows. How do we account for that?

And notice that in a square the outer ring would use a lot more yarn than the inner ring. So maybe we’d like to ensure that each digit is represented in each position roughly equally - that we use roughly equal amounts of each colour.

Maybe I’ll come back to this idea in a future blog…

Unravelled

At this point, you’re expecting to see a completed scarf?

The truth is, I already used some of the yarn I mentioned to make a sock. Yes, just the one sock. And maybe someday I’ll get around to making it a matching pair.

And maybe someday I’ll even make a ternary scarf.

But first I need to figure out what message is worth wearing around one’s neck…

Chris.

[was this all just a waste of time? you must be new here]

Epilogue: Structure and interpretation of scarf

Suppose you’re presented with a scarf. It’s made up of stripes of three different colours, but the patterns seem… odd. Random? But why would anyone make a random patterned scarf?

Given that there are three colours, you think maybe it’s a base 3 encoding, and you intuit that 3 rows gives you 27 sequences - enough for 26 letters and a space.

You might assume a basic encoding - A=1, B=2, etc. But you don’t know how the colours map to bits 0, 1, 2

But then, there are only 6 possible arrangements, so you can just try them all, and see what yields a meaningful message. Perhaps you notice a particular colour regularly appears in blocks of 3, and you think “maybe that’s a space character”. You call that colour 0 and now you only need to figure out which colour is 1 and which is 2 - two possibilities.

It’s not trivial, but it’s possible. This feels ideal to me.

By comparison, you couldn’t infer the frequency-based or optimal encoding in the same way - in the first one, some of the assignments are arbitrary, and in the second the assignment is heuristic so even following the same procedure you might not reproduce the same mapping.

If the message (scarf) is long enough, you can ignore the base 3 aspect, treat each 3-row sequence as an arbitrary symbol and perform frequency analysis to figure out the letter mapping.

But we’re talking a Tom Baker length of scarf, at least.

Appendix: the example stitches

Each of the examples uses a different stitch/technique. Some may call this an unfair comparison, but it made things more interesting for me ;)

The first example, for the A=1 encoding, was done in tunisian crochet, using repeated tunisian simple stitch (TSS)

The second example, for the frequency encoding, was knitted in a basic stockinette - 1 row knit stitch + 1 row purl stitch.

In this case there are actually two rows per bit (1 knit + 1 purl). This was mostly so the colour change would always happened along the same edge, for my convenience.

The last example, for the optimal encoding, was done in regular crochet, using repeated half-treble (Htr) crochet stitches in UK notation, or half-double (Hdc) in US notation.

I didn’t do an example for the squares because that would have required cutting the yarn (and also I’m lazy).

Footnotes

For the full background and design of my blanket, see the previous blog post

God laughs

When we last left our temperature blanket, it was the end of July, and I said

we haven’t made it into the oranges/reds […] it would be nice to see those colours incorporated

Turns out I got my wish, but when I least expected it.

Going back to the original layout

the plan was that when I got to the start of September, I would move to the top right, following a truncated z-ordered curve.

But then, come the start of September, we had an unexpected heatwave - including the hottest day of the year! (22.2C)

Following the original plan would have put these yellows and oranges next to the blues of March/April, which felt to me like it wouldn’t look so good.

Back to the old drawing board

The most obvious redesign - instead of shifting to top right, we simply continue downward, but otherwise adhering to the z-order layout.

In retrospect, this feels much more natural.

Doing this, the completed blanket becomes 16x24 squares. This is slightly longer and thinner than the original design, and comes out to 384 squares total.

The original plan had 365 day + 12 months = 377 with 1 spare to represent the year (378 total). In the new plan, I was left with 7 spare squares.

7 is a bit of an ‘odd’ number, so I decided to use one to ‘sign’ the piece, and then use the other 6 as a block for representing the year as a whole.

The signature square is straightforward - it’s my initials, CO. I chose a shade of grey so that it would be cohesive with the overall design, without standing out or being mistaken for any of the other colours used.

The year marker is more interesting. The original thought was to represent the year ‘2023’ in a similar way to the month markers, which are the month numbers in binary.

I hit on the fact that 2023 in hexadecimal is 7e7 which is conveniently 3 digits (and I have 6 squares to fill). It’s also a palindrome, which makes for a nice, symmetrical pattern.

Each square represents 2 bits - an outer ring and the centre, plus a separating ring which is always white. And each vertical pair together represent one hex digit (nibble?)

The bits are read outside-in - so for example, top left is white outer and solid centre = 01, bottom left is solid outer and centre = 11; so all together the pair is 0111 or 7 in hex.

The complete pattern also looks kind of like a smiley face ° □ °

Naturally, the colour represents the average temperature across the whole year (9.9C).

I also broke the z-ordering a little - the December marker (binary of 12) looks too similar to the 10 year square, so I wanted to move it away from the year block so as not to confuse the design.

(If I’d thought of it sooner, I would have put the signature as the bottom-left square of the complete blanket, in the October region)

Wrapped up

Without further rambling, here’s the completed blanket

(rotated 90 degrees, January at top right)

The completed blanket is ~118x170cm

Overall, it wasn’t too difficult on a technical level, but boy did it take a lot of time. Including joining and weaving in ends it took ~ 25mins per square, which comes out at ~157 hours (!) total, or ~3 hours a week.

As for the scarf I mentioned in the previous post; well… the blanket alone was a lot of effort. And I couldn’t muster the enthusiasm to keep working on it. Two year-long projects at once was a little ambitious, oh well.

Chris.

[Now I’ve got to figure out what to do with the damn thing…]

I wasn’t sure I’d bother at all this year. I was trying to think if there was a way to spice it up - doing it in python is dull, I write python almost every day.

Then one day at work I was doing some API stuff on the command line with curl and jq, and it got me thinking…

jq

I’d wager most developers have heard of jq, the “lightweight and flexible command-line JSON processor”.

Odds are, if you’ve used it you’ve probably not done anything more exotic than picking out fields

curl localhost:8000/auth/token/ -d '{"username": "foo", "password": "bar"}' | jq .access_token

That was most of what I did. Occasionally, I’d try to do something more complex, usually with liberal help from google.

Heck, I’ve seen coworkers use jq to pretty print json, then use grep and sed to grab fields.

Anyway, it seemed like it would be fun to try and do some AoC in jq

Before we get to specific puzzles, lets look at…

General stuff

Not json

The first thing is, of course, that jq is for processing json.

On the other hand, AoC puzzles are rarely (never?) json formatted. Usually the input is lines of plain text.

A little googling tells us the way to deal with this

jq -Rn 'inputs | ...'

The -R flag means “don’t try to parse this as json”. The -n flag is needed for reasons I don’t fully understand.

Then the inputs filter is how you actually get at the input - it’s a generator of lines. Alternatively, you can do [inputs] to get an array of lines

A slight variation is when the input isn’t one-per-line, but multi-line blocks, separated by a double newline.

In that case, we don’t want the input to be split line-wise. So instead we use the -s (slurp) flag to pull in the whole input. We then get the full input with input singular and do our own splitting

... | jq -Rns 'inputs | rtrimstr("\n") | split("\n\n") | ...

rtrimstr gets rid of any trailing new line, otherwise we usually end up with an empty string somewhere down the line, which causes confusing errors

Scripts

If you’ve used jq, you’ve probably used it directly on the command line ... | jq '.[] | .count'

This is fine for simple stuff. But as things get more complex, especially when you start introducing functions, it makes more sense to put everything into a script file. The script file can then be passed to jq with the -f flag - ... | jq -f script.jq

But we can do one better - we can set a ‘shebang’

#!/usr/bin/env -S jq -f

You can use the path of jq directly rather than env if you prefer. -S on env allows us to pass flags to jq

Then it’s just a matter of setting the script to executable (chmod +x), then you can invoke the scripts directly

./script.jq < input.txt

This is what you’ll find in my solutions repo.

Functional programming

One thing I didn’t notice about jq until I started using it in earnest, is that it’s a functional programming language.

My experience with functional programming is all that passes for FP in python, and a failed attempt to learn Haskell. But I’ve picked up a few tricks along the way.

The thing that took some adjusting to is not having a for-loop, but rather having to think in terms of recursion.

Also mutation is not so straightforward. I only used it once.

Assignment

Variables can be assigned in the middle of a pipeline, for example

$ echo '[1,2,3,4,5]' | jq '(length | debug) as $l | debug | add / $l'
["DEBUG:",5]
["DEBUG:",[1,2,3,4,5]]
3

It’s not needed in that example, but you get the idea. It allows you to perform some calculation on the current value and capture that into a variable. It then passes along the original current value unchanged.

This is convenient if a value needs to be reused, or just to make the code more readable

Debugging

The errors from jq tend to be terse, and often not that helpful.

In this case, it’s useful to throw in some debug statements

$ printf "1\n2\n3\n" | jq -Rn 'inputs | debug | tonumber'
["DEBUG:","1"]
1
["DEBUG:","2"]
2
["DEBUG:","3"]
3

Yes, this is akin to putting print statements everywhere. You work with what you’ve got :)

Basically, it prints out the input value then passes the input along unchanged.

There’s a variation where you can pass it a message e.g. debug("hello") but that isn’t supported in the version installed on my laptop.

Documentation

The manual is a bit hit and miss.

For example, the search box is more like ‘jump to heading’

Long story short, Ctrl+F and google are your friends.

Formatting

As far as I can find, there’s no standard formatter in the vein of black, gofmt, prettier for jq

So for my scripts I had to go with what felt right to me.

The Puzzles

Day 1

For part 1 we need to pick out the first and last digit from a string, the wrinkle being there may be only one digit present.

Regular expressions are the obvious choice for this task - scan("\\d") - returns an array of digits (strings), from which we can grab the first .[0] and last .[-1] (or indeed ‘first’ and ‘last’)

For part 2, we also have to account for digits written out as letters, and the ‘obvious’ solution is to string replace words for digits, at which point the rest of the solution is the same as for part 1.

The tricky bit is that digit names may overlap, e.g. in eightwothree ‘eight’ is the first digit name, but if we substitute digit names in numerical order, we’d replace ‘two’ to get eigh2three, losing ‘eight’

To workaround this, I had a sudden flash of inspiration while brushing my teeth (as one often does) - what if we substitute the digit numeral, wrapped in its name.

For example we replace two with two2two. When we do that in the example, we get eightwo2twothree. Now we haven’t lost ‘eight’.

The final piece (arguably unnecessary) is solving both parts in one.

As noted, after transforming the input for part 2, it’s solved in the same way as part 1

So for each line, we create an array of [., sub_numbers] ([part1, part2]), find digits, and transpose from

[[line1-part1, line1-part2], [line2-part1, line2-part2], ...]

into

[[line1-part1, line2-part1, ...], [line1-part2, line2-part2, ...]]

then sum up each part.

This transpose trick comes up often.

Day 2

This one looks hard at first glance. The trick is parsing it, with regex and lots of splitting, into the right structure

The key bit is getting an array of [colour, count] pairs into a mapping (object) of {colour: count} using from_entries

Once you have that, the actual solution is straightforward, just applying a couple of functions.

Day 4

An observation which makes this one easier - any given number will only appear once on either side of the |, so we just parse all the numbers in a line into a single array, group the numbers together, then look for the ones there are two of.

$ echo '[1,2,3,4,2,5,1]' | jq -c 'group_by(.) | debug | map(length)'
["DEBUG:",[[1,1],[2,2],[3],[4],[5]]]
[2,2,1,1,1]

(-c means compact format; otherwise the result would be pretty-printed/split across multiple lines)

Part 2 was more interesting. We start with a list of (wins, count) for each ticket. For each ticket we add count to the wins number of subsequent tickets, then return the number of this ticket plus a recursive call, e.g.

  r([(2,1), (1,1), (0,1)])
= 1 + r([(1, 1+1), (0, 1+1])
= 1 + 2 + r([(0, 2+2)])
= 1 + 2 + 4
= 7

Day 5

This is another one I didn’t think I could do. But again, once you get past parsing the input it’s a lot clearer.

Then it’s just raw calculation.

I didn’t manage to solve part 2. I did try, but my solution didn’t scale.

But while we’re on the subject, surprisingly difficult was splitting an array into chunks; I’m surprised there isn’t a built in for it.

The solution I came up with was a sliding window using foreach, which emits the current pair every other iteration.

echo '[1,2,3,4,5,6]' | jq -c 'foreach .[] as $i ([0, null, null]; [.[0] + 1, .[2], $i]; debug | if .[0] % 2 == 1 then ("skipped" | debug | empty) else .[1:] end)'
["DEBUG:",[1,null,1]]
["DEBUG:","skipped"]
["DEBUG:",[2,1,2]]
[1,2]
["DEBUG:",[3,2,3]]
["DEBUG:","skipped"]
["DEBUG:",[4,3,4]]
[3,4]
["DEBUG:",[5,4,5]]
["DEBUG:","skipped"]
["DEBUG:",[6,5,6]]
[5,6]

Day 6

If you write out the formula for distance vs time, what we want to find is (T - x) * x > D or x^2 - Tx + D < 0

In other words, it’s a quadratic equation (inequation?), and we want to find the integer values of x which give a value less 0, which we can get with the quadratic formula

T - sqrt(T^2 - 4D)        T + sqrt(T^2 - 4D)
------------------ <  x < ------------------
        2                         2

To get the count, we take the difference of (the floor of the larger value) and (the ceil of the smaller value), plus one. There’s an edge case where this doesn’t work, when the bounds themselves are integers, as is the case with one of the examples.

But that wasn’t the case in any of my puzzle inputs, so I ignored it :D

Day 7

My first thought, taking inspiration from day 4, was to ‘quantify’ each hand using group_by and length. But how to order them? I wrote out the possibilities

5
4,1
3,2
3,1,1
2,2,1
2,1,1,1
1,1,1,1,1

These are all the ways to partition 5 (not counting permutations). Not that that helps here. But it occurred to me, if I pad them with 0 until they’re all length 5, then they sort in the right order, i.e. 50000 > 41000 > 32000 > 31100, etc

But what about the values of the cards themselves? As they are, they’re not sortable because e.g. king is higher valued than queen, but K is less than Q lexically.

The dumb solution I came up with was to translate the face cards into their equivalent hex value, i.e. T -> A, J -> B, etc.

We then concatenate the hand type with the hexified cards to get a ‘canonical’ form, e.g.

32T3K -> 2111032A3D
T55J5 -> 31100A55B5
KK677 -> 22100DD677
KTJJT -> 22100DABBA
QQQJA -> 31100CCCBE

Then finding out the ‘power’ ordering is a simple lexical sort

For part 2, we count the Js, quantify the hand without them, then add the J count to the largest of the remaining groups, e.g.

KTJJT - > 2 + KTT -> 2 + (2,1) -> (4,1)

and when converting to hex we replace J with 1 instead of B

Then the rest works the same as part 1

pad is another function which is surprisingly absent from jq. Additionally, the repeat method is unbounded. So I used range + foreach to generate an array of 5 zeros, then zipped (transposed) that with the input, which pads the input with null

$ echo '[1,2,3]' | jq -c '[., [foreach range(5) as $i (0; .)]] | debug | transpose'
["DEBUG:",[[1,2,3],[0,0,0,0,0]]]
[[1,0],[2,0],[3,0],[null,0],[null,0]]

then use max to take advantage of the fact null is less than any other value.

Day 8

Part 1 is a fairly straightforward parsing of a tree structure into an object - from_entries is our friend - then a recursive walk for the length.

Part 2 is a classic AoC trap. You try to play it out, then realise it’s going to take forever for it to complete executing that way, and actually the different paths are looping, so you just need to find when the loops coincide.

For that we need to calculate the lowest common multiple of the loop lengths, and to my great shame I had to look up the formula on wikipedia (probably the last time I did an LCM was AoC 2021).

Day 9

To get the pair-wise difference, we ‘zip’ the input with itself offset by one

$ echo '[1,2,3,4,5]' | jq -c '[.[:-1], .[1:]] | debug | transpose'
["DEBUG:",[[1,2,3,4],[2,3,4,5]]]
[[1,2],[2,3],[3,4],[4,5]]

Otherwise it’s just implementing the procedure as described in the puzzle.

Day 12

This one I did by brute force - that is, replace each ? with a # or . and see if it matches the pattern.

It was slow - took something like 10mins - but it got there in the end. And more to the point, it was easy to implement.

It did not, however, scale for part 2. I didn’t even bother trying, given how long part 1 took.

Day 13

For this, finding the horizontal reflections didn’t seem too bad - slice each line, does the first half match the reverse of the second half.

But what about vertical reflection?

Then I remembered the trusty transpose function, which turns the vertical problem into the horizontal problem again. Easy.

Part 2, not so much.

Day 15

For this we have the handy explode function, which turns a string into an array of ‘code points’, which are conveniently the same as ASCII values (yay, unicode)

Day 24

Finding where (and when) the paths collide can be found algebraically.

Having said that, translating said algebraic solution into jq was horrendous. Seriously, that script should come with a content warning :p

jq-wise, we have the convenient combinations function to generate all the pairs of hailstones. You just need to remove self-pairs and reverse pairs

$ echo '[1,2]' | jq -c '[. ,.] | combinations'
[1,1]
[1,2]
[2,1]
[2,2]

i.e. [1,1] is a number paired with itself, and [1,2] and [2,1] are the same just in opposite orders.

Conclusion

Well, it was fun while it lasted. Given jq is Turing complete [citation needed], it is theoretically possible to solve all the days with jq, but I’m afraid that’s beyond my skills/determination.

In the end, I got 19 out of 50 stars (38%), which I’m pretty sure is a failing grade. Oh well :)

The sad thing is, having learned all this jq, I’ll probably never use it professionally. Anything which requires more complex jq processing than what will fit on a single line would just raise the question - why not write it in python instead? After all, python is more readable and more testable.

To that point, as I was writing this blog I was looking back at my solutions and thinking, “erm.. how does this work again?”

Still, a fine way to pass the time before Christmas.

Chris.

[And I did also find the time to crochet more Christmas decorations]