GitHub link: github.com/brege/image-activity

Overview

• Generate heatmaps and histograms of image saving activity over hours, days, and months
• Use file timestamps, modified-times, EXIF, and regex parsing for refined image discovery
• Add bands and markers for major life events

Background

I wanted to determine if my image activity is dependent on major events and device purchases in my life.

• do I tend to take more pictures during certain times of year?
• how has my screenshot usage evolved over the last 15 years?
• do I have “honeymoon” periods after a device purchase?
• in what ways has my camera and screenshot usage changed between being an academic, chef, and developer?

I’m not a social media person, although my mastodon did see an uptick of usage following my hip surgery, where I began hiking and foraging a lot.

My image activity fits in three main categories:

1. camera: storage of camera photos from my phone
2. screenshots: screenshots on both my laptop and phone
3. internet: pictures downloaded from the internet

Gallery

I’ve marked in these first line charts, Camera Usage and Image Capture Concurrency, times when I’ve purchased a major device (a new phone or laptop) and a couple key periods of my life. These plots have all been normalized to a 0-100 photo count scale.

Camera Usage

From 2010 to 2017 I was a Physics TA and, following my 2014 physics prelims, a computational astrophysics doctoral researcher. I began attending conferences in 2015, exploring places around Pullman, WA during these researcher years there.

At the end of 2017, I left that life. I embraced my love of food and cooking and became a professional chef for a number of years thereafter, including the Covid-19 pandemic. This period of my life saw a greater number of photos taken: pictures of plates, menus, schedules, etc. My camera photos before this time were mostly non-work-related: travel, events, and pets drove image origination.

Image Capture Concurrency

Heatmaps

I only have one experience with online coursework: the data science bootcamp I attended in the fall of 2023. This period did not have a major impact on my screenshotting habits. There are three principal areas in which screenshot usage was more frequent:

1. The creation of my website brege.org around August 2016.
2. As an executive chef, screenshotting is recurrent for scheduling, text message records, receipts/purchase dates, etc.
3. Agentic-driven coding workflows, beginning midway through 2025, saw a surge in screenshot usage. Screenshots have become a large part of my front-end debugging workflow for web app development–extending well beyond data-structured Cypress end-to-end tests.

I did not find my screenshot usage noticeably change during my brief stint with online coursework.

In general, it appears that I take more screenshots on desktop earlier in the week and in the afternoon (averaged over the last ~15 years). To my surprise, the heatmap for screenshots on my phone have nearly identical densities. I assumed this would be biased toward the weekend and closer to 17:00 because of sports and restaurant dinner service.

Camera usage frequency, on the other hand, is made distinct by day of week only on density during Thursday evening and Saturday afternoon. It’s especially featured in both my Chef days and post-op mobility.

Histograms

By device and source, then binned on hours of the day, day of the week, and month of the year, histograms provide a finer distribution in one dimension.

For the hourly concentration of all three photo habits, my activity roughly follows a Boltzmann distribution.

These distributions generally peak at two distinct hours:

• camera photos and screenshots center around 15:00
• internet photos are generally concentrated around 20:00

Each bin is averaged for each picture type over the last 15 years, regardless of timezone.

Image activity generally increases at the beginning and end of standard university semesters, which also include the height of summer and the holiday period when I am always always travelling. Screenshotting is highest in the fall to mid-winter.

In my experience, restaurants are historically busier between, roughly, Friendsgiving and Father’s Day. Camera usage also largest during high summer. Beach. Hiking. Produce selection during chef years.

I have survey fatigue

Surveys are marketing emails. I can’t believe I used to take the time to respond to some of them. An analysis on my email history has shown that my hunch on survey spam is correct. Around 2023, I began marking all surveys as spam, and I’ve got the data to prove just how rampant companies have used surveys to get their brand in your inbox.

Emails that matter

My main goal in exploring my emails in-depth was to build a predictor of whether an email had any future usefulness.

Mail from friends and family, correspondence with students, receipts and financial records, etc. fit the binary of keep. After I manually processed this massive backlog of email (with great help of Thunderbird’s filters), what I found I discarded most, besides spam, were surveys, newsletters, and other mass mailers.

What I found, qualitatively, was:

• imperfect spelling, capitalization, and grammar
• little-to-no HTML markup
• all emails meant only for me sans phishing and spam

These traits defined true keepsakes.

Introducing sanoma

en.wiktionary.org/wiki/sanoma

sanoma (noun) Finnish 
  message, communication (a communication or the content of a physical
  message; also the message contained in some act or expression such as a
  work of art)

sanoma (github.com/brege/sanoma) uses YAML workflows to define multi-step analysis pipelines. The workflow runner automatically discovers and executes tools from the sanoma/analysis/ and sanoma/plot/ directories, making it easy to chain data extraction, filtering, analysis, and visualization into reproducible pipelines.

I developed this YAML workflow method in my Markdown-to-PDF project–oshea–where I realized comprehensive end-to-end tests were just manifest workflows. It’s an intuitive way to string command line sequences together. The pipeline term in machine learning/data science is congruent to this system.

Data Mining

While much of this can be done in a Jupyter notebook (far easier to refresh plots this way, although :MarkdownPreview in Neovim is sufficient), I built this project as a way to data-mine my own activity. I also want to create a visualization harness for many things on my computer:

• text message history
• email history
• screenshot frequency
• browser history and bookmarks

Because email is text-based, and because my first concept of “AI” was the need for combative spam filters that have been built over the last thirty years, email felt like a good starting point.

Grad-school Emails

The monthly timeline reveals the academic year rhythm: high volume during active semesters with dramatic drops during summer breaks and winter holidays. The 2016-2017 dip corresponds to the dissertation defense period, where militant email sanitation was a reprieve from LaTeX and simulation monitoring–hence the dip.

My personal dataset has about 35K emails between my grad-school emails and my current website’s personal email. Not included are my Gmail and undergrad email(s). I plan on synchronizing those at a later date.

Grad-school Timeline Seasonality

WSU’s Okta system required changing passwords every 6 months, and some time after my defense my account died. I am thankful that I had a Thunderbird profile tucked away on a drive that allowed me to recover all of my university emails.

Grad-school and onward Histogram

The year-over-year histogram demonstrates consistent academic seasonality, with September-April peaks and May – mid-August valleys across all years of graduate study. Even with teaching summer labs, the bureaucratic pressure in the summertime dies. I loved teaching in the summer.

Spam, Marketing, and Surveys

The spam timeline shows minimal marketing emails pre-2010, followed by a sharp increase around university enrollment. By 2015, spam reached 60-80% of all emails and has remained consistently high. The GDPR implementation around 2018 created a spike in unsubscribe language as companies scrambled to comply with new regulations.

Marketing Spam Trends

The tail in the beginning of this timeline is presented for context. It only includes a “purified” hotmail account mailbox from my teenage years that extended a bit into my undergrad years. Those years overlap with Gmail usage (not integrated into this data) and my GVSU university email.

Keyword Buckets

Another useful filter for spam emails is checking for keywords like unsubscribe in the message body.

unsubscribe_bait dominates with over 12,500 matches, followed by satisfaction surveys (~8k) and direct “survey” requests (~4k). This reveals how modern marketing shifted from direct sales to engagement-focused tactics requesting feedback and reviews.

Conclusion: Satisfaction Surveys are the new email cancer

The heatmap (filtered to post-2010) shows “satisfaction” spam as the most persistent threat, maintaining 20-25% frequency from 2012 onwards. Survey-based spam shows steady growth, intensifying after 2020, when both GDPR constraints pressured companies to invent new angles of attack, becoming increasingly desperate for customer “feedback” (attention) during the pandemic. Satisfaction feedback surveys are advertisements.

Most anyone in the gym has heard the adage to eat “1g of protein per pound of (goal) bodyweight per day.” Research has shown this figure isn’t accurate; it’s more like 60–70% of that (and that’s at peak resistance training intensity) for ideal muscle mass gains ^{1 2 3 4 5}. For example, if you weigh 175 lbs, this guideline states you should eat somewhere between

$$ \begin{aligned} \frac{\textrm{protein}}{\textrm{day}} &= \left[0.35,0.75\right] \cdot \left( \text{g} / \text{lbs} / \text{day} \right) \cdot 175 \textrm{lbs} \\ &= ~ \left[60, 140\right] \textrm{g/day} \end{aligned} $$

where we have used the shorthand

$$ \text{g} / \text{lbs} / \text{day} \coloneqq \frac{\text{protein (g)}}{\text{bodyweight (lbs)} \cdot \text{day}} $$

But for nearly everyone landing on these pages, the goal is fat loss. Higher protein (and fiber) intake will help you feel fuller for longer and can help people new to fitness and diet control their cravings.

Following the guidelines is also made more tedious by the near-constant arithmetic at the grocery store. While it’s good to check the labels to learn what’s actually in your food, keeping a running total and hitting your target each day becomes a challenge. You wind up relying on apps, breaking meals into chunks: “10 grams here, 25g scoop there… how much protein is in a palm-sized chunk of chicken breast again??” And then there’s the protein in bread, grains, bars. It’s nearly impossible to balance all that against metabolic calories if you’re even trying a little.

Compositional Thinking Strategy

Here’s my take: Ratio counting. As a chef, there’s a reason the imperial, fractional system works: ratio. It’s easier for me to think about doubling, tripling, or halving depending on what I’m buying and who I’m cooking for. Our days are divided into twelves and therefore 2’s, 3’s, and 4’s. Naturally, our meals are spaced that way too. The value of our base10 system cannot be overstated, but I have a difficult time what eating or preparing 10% less of a meal or recipe means vs doing a quarter or half.

For the 175 lbs individual, the ranges of protein consumption needed are pinned by three characteristic quantities:

• $\textbf{MAX} = $ 100% efficiency in protein synthesis $\approx 1.6 \text{g} / \text{kg} / \text{day}$
• 0.75 $\coloneqq 0.75 \times \text{MAX} \approx 1.2 \text{g} / \text{kg} / \text{day}$
• $\textbf{RDA} \coloneqq 0.5 \times \text{MAX} \approx 0.8 \text{g} / \text{kg} / \text{day}$

where $\text{RDA}$ is the Recommended Daily Allowance, and $\text{MAX}$ is the Maximum Efficient Intake. The “$0.75 \times \text{MAX} \approx 1.2 \text{lbs}$” is roughly the target for most people who weigh 175 lbs (80 kg), on average. ¹

Implementation: The Protein Efficiency Calculator

The table below answers a simple question: how much of one single food do you need to eat to hit your daily protein target? Input your weight, toggle metric or imperial, and adjust units per row—grams, ounces, scoops, each. It’ll show you the amount needed to hit benchmarks in the range of $\text{RDA}$ to the $(\text{MAX})$ intake. Based on resistance training status and the latest research, a sliding scale in this range will allow you to estimate your required protein demands.

Quantity: the number of units in the Unit/Size column you need to eat to acheive the chosen protein target, per day, for your input weight.

Adjusting the Protein target slider can be thought of as a direct conversion of protein to training intensity, from sedentary/$\text{RDA}$ all the way to the $\text{MAX}$ intensity threshold.

I’ve also included the daily Calories you’d likewise take on per food item. You’ll note that high carbohydrate (peas, chili) and especially high fat content (mixed nuts) greatly diminish the remaining calorie budget in the day. Conversely, protein powders, including the vegan kind, substantially lower the overall caloric footprint.

The Protein Efficiency Matrix

While the above tool scales protein needs to an individual’s body composition, it’s also worth viewing high-protein foods in a way that’s independent of bodyweight. The protein-to-calorie efficiency of a food source is a fixed property — it doesn’t change based on who’s eating it.

With that in mind, the Protein Efficiency Matrix below expresses the relative caloric density of each food source. It compares the efficiency of each against the others in the table.

Your weight and training status don’t matter here — this table is the same for everyone, assuming equal daily protein intake. The first column shows a simple mean of each row’s relative efficiency compared to others — a rough estimate of how each food stacks up overall. Smaller numbers are better. High-fat items like salmon and nuts, and high-carb ones like peas and chili, form relative “islands of inefficiency” among high-protein sources.

You can even think of this matrix as a way to zone: cutting / maintaining / bulking cycles using blues / greens / reds as a visual cue for how your shopping list and pantry pars might shift depending on your current training status.

For example, Greek yogurt has over five times the protein density of tree nuts by volume. If you’re trying to control weight and meal prep, having this kind of visual measure might help you balance portions — say, between yogurt and nuts in your breakfast bowl.

Note also that the red, lower-in-protein efficiency items are still quite efficient. For example, gas station treats like trollies and doritos have relative efficiency values order(s) of magnitude higher. Compared to greek yogurt, their protein efficiencies are astonomically bad $\text{eff.}\left(\text{trollies}:\text{yogurt}\right)$ $\approx 32$ and $\text{eff.}\left(\text{doritos}:\text{yogurt}\right)$ $\approx 13$. Abysmal.

Discussion

This lens is useful because it translates the quantities into grocery store units—like pack sizes. About a dozen eggs is one day; two pounds of beef and two pounds of chicken is about three days; a tub of cottage cheese (24oz) and a tub of Greek yogurt (1qt) is just under two days. Then 3–4 scoops of protein powder can help balance that week’s diet.

There is also an upper limit to the amount of protein available on the planet. 70% of freshwater is already in use; 50% of global land space is already dedicated to agriculture. It’s a 60–40 split between plant- and animal-sourced proteins occupying this space—and both the global population and protein demand will increase by 20–25% in the next 25 years ⁶.

From a purely resource standpoint, there will need to be a lab-grown protein synthesis renaissance. And it’s this fact alone that makes the “$1 \text{g} \ \text{protein} / \text{lbs} / \text{day}$” myth not only overly generous, but also inconsiderate of the planet’s biological limits. One-third of that is the target if you aren’t lifting. Half to two-thirds is more realistic if you’re engaged in moderate to heavy resistance training.

References

$$ W_\text{1RM} = W \cdot \left(1 + \frac{R}{30}\right) $$

To compute your one-rep max weight for a given lift, $W_{1\text{RM}}$, you simply input the weight lifted, $W$, and the number of reps performed before failure, $R$.

This formula has been empirically validated ¹ and is a useful tool not just for estimating your max weight for a given weight-rep pair $\left(W, R\right)$, but also to compute the number of predictable reps you can do at any other given weight, relative to your one-rep max.

This is quite useful for plateaus. For example, if you get stuck progressing the Overhead Press to 115 lbs for 5 reps after successfully performing the OHP for 110 lbs for 5 reps, you may need to strategize. Using the Epley formula, to go from $$ \left( 110\text{lb}, 5 \text{reps} \right) \rightarrow \left(115 \text{lb}, 5 \text{reps}\right), $$ you are effectively increasing your one-rep max: $W_\text{1RM}: 128 \text{lbs} \rightarrow 134 \text{lbs}$. Our goal here is to effectively increment your $W_\text{1RM}$ in smaller steps: $$ \small \left( 110\text{lb}, 5 \text{reps} \right) \rightarrow \left(115 \text{lb}, 4 \text{reps}\right) \rightarrow \left(110 \text{lb}, 6 \text{reps}\right) \rightarrow \left(105 \text{lb}, 8 \text{reps}\right) \rightarrow \left(115 \text{lb}, 5 \text{reps}\right) $$ because this is the same as $$ W_{1\text{RM}}: 128.33 \text{lb} \rightarrow 130.33 \text{lb} \rightarrow 132.00 \text{lb} \rightarrow 133.00 \text{lb} \rightarrow 134.17 \text{lb} \\ W_{5\text{RM}}: 110.00 \text{lb} \rightarrow 111.71 \text{lb} \rightarrow 113.14 \text{lb} \rightarrow 114.00 \text{lb} \rightarrow 115.00 \text{lb} $$ I refer to this as the Epley pathway.

Because the Epley formula is nonlinear, this requires an iterative approach. I’ve made a calculator that will help you find this path over different $\left(W, R\right)$ pairs. It defaults to using a range of reps roughly half that of your initial number, $R_0$, and a few “sweet spot” rep values.

More details of different, specific formulae for Epley ² Brzycki ³, Lombardi ⁴ and O’Connor ⁵, and several complimentary research articles ¹ and ⁶, are available.

My Progression Scheme

This makes it straightforward for me to also compute the warmup sets needed for my leading barbell exercise. Recently, I’ve found good success in pyramiding warmups using my calculated $\left(30, 20, 10\right)\text{RM}$’s for $\left(12, 8, 4\right) \text{reps}$, respectively.

In addition, another scheme I’ve been using for strength-focused exercises has been, after 3–6 reps of heavy weight for three sets, to drop set into $\left(10, 20\right)\text{RM}$ for $\left(8, 12\ldots\text{amrap}\right)$, respectively, to increase my work volume without chewing up my joints, leaving $\sim \text{2}$ $\text{rir}$. Here, $\text{amrap} = $ “as many reps as possible” and $\text{rir} =$ “reps in reserve”. My thoughts here reflect the goal to cover warmup (3), strength (3–4), and hypertrophy (2), in a single leading compound exercise.

Possible Improvements

• Create a similar pathway table for the goal of adding a rep, instead of adding $5 \text{lbs}$ to the bar. Will take a good amount of effort.

• Implement a pyramid scheme with broader rep ranges in the progression chart histogram. I haven’t thought much about this yet, but when I can hit 7 reps on heavy compound lifts, I feel like I’m entering a moderately safe 7–12 rep range. At that point, the effort level shifts to something different.

• The Epley model is just one of many. I haven’t explored the others yet because I could solve this with fractional arithmetic in my head while “ape-brained” in the gym. However, adding a dropdown toggle to choose different models, and referring to recent literature on advancements in curve fitting, would make this tool more robust. Similar to how the Opus audio codec performs at different fidelities with various bitrates, I’m sure that the pathway from 1RM to 30RM follows a more complex, multi-fitted approach.

References

The Books

The Books were a musical duo that made music from found sound recordings. Like a collage, bits of old cassette tapes discovered at garage sales, thrift stores, and thrown out are pieced together through melody with minimal instrumention to create emotional, sonically beautiful tracks.

This mosaic of found sound and vocal samples in large part created a music genre of its own. Well, mostly. I think most of us have heard those tracks that kickstart or bridge their songs with an audio clip taken from some movie or speech. But The Books did this to the extreme, mixing these sounds into their songs as their own instrument.

In fact, each track is filled with an unimaginable number of recordings–sounds that bleed with every emotion of the original, forgotten storyteller within them. There are also aleatoric takes (re: “by digitizing sound and thunder”) that fold more natural sounds in, but not like Stomp or Anathallo.

What compelled me most was the unscripted essence of the recordings. I wondered about the people that may have listened to some of those source tapes on long road trips, or recorded there. Like archives of thoughts from carpeted basements, strung together from one fried thought to the next.

Found Magazine

Found Magazine was created by another duo, based in Ann Arbor, Michigan whose pages were each a collage of scrap notes and photos sent in to them. In it, glimpses of everyday minutia are scanned into its pages, without continuity, often bearing the weather and stains obtained in journey from the trash.

Source

Amid a probable banality, the subjects of these clippings range anywhere from odd grocery store requests to people dealing with heavy topics like abortion, divorce, and suicidal thoughts. I was in my very early twenties when my friends introduced me to it, and while I had every opinion imaginable about the world, I certainly didn’t suffer any of the experiences these notes conveyed.

There was sheer earnesty to them: a glimpse into worlds you couldn’t ever possibly live. People’s experiences simultaneously made out to be both relatable and unlivable.

Source

It is no wonder why this magazine was so popular in the US penal system. Shy of living in it, it’s these discards that are made to share our bond. To me, there is an inner nature here lost in most documentaries (ugh, Netflix), let alone all reality TV shows: they are over prompted and epicly driven. But the stories in Found were of a different species, one that was free of slant, rich in humor, tied deeply in emotion and all of our imperfections.

How To with John Wilson

Source

Arguably the most inaccurately titled show ever, John Wilson sets out in New York City with nothing but his video camera, as he has done since he was a child, recording literally everything. In his docuseries How To with John Wilson, there are shots of spills and overflowing trash, candid public moments and private invitations, people spontaneously napping or ritualistic maneuvers that all walks of city life take. But moreso, it’s the profoundly deep cuts into the human experience accompanying the imagery that make it so special.

What I loved about this show, which thankfully made it to (but sadly ended in) three seasons, was the juxtaposition of Wilson’s unpolished interview/questioning skills with a carefully well thought-out narrative. There is an incredible mastery of using random, second-long bits to string together larger, much broader points about our experiences in the world.

The travelogues benefit from people living in tight proximity in an entirely manmade habitat: The City. Even when journeying out West (I forgot why?) to fail in filming Burning Man, Wilson somehow winds up inside an underground, decommissioned nuclear launch bunker that feels just as claustrophobic and unnatural as the city he’s escaped.

Conclusions

While unfortunately these mediums aren’t widely known nor around anymore, I think about them often when I forage for different things on my hikes. So maybe I’m intrigued by The Books, Found, and John Wilson for a similar lust, that it hits such a similar psycological mechanism is probably no accident.

I think about random notes I’ve found, the content of which often surprised me. I also imagine all the little notes I’ve lost whose finder must have been just as confused upon reading. I guess that’s my larger wonder. We don’t always know who our audience is. And I think that’s actually quiet wonderful.

Sweetness is a part of the flavor balancing equation. Sugar is energy dense. It stores well. It is the second fastest “macro” metabolized by our bodies. Like the fastest (alcohol), we associate its taste with peak ripeness since it expresses itself in fruit through juiciness which elicits immediate opportunity modes within our survival brain.

One can practice a great deal of creativity with simple table sugar and their stove top range alone. In addition to practically any sweet confection, sugar is melted into water for simple syrup in cocktails, into caramels and all its derivative forms (butterscotch, toffee, gastriques ¹, etc.), and heated in puddings, pot de creme and creme anglaise. All in all, understanding that sugar has a particular behavior when it melts carries over to practically every dessert sauce in existence. Melting sugar to different temperatures and playing with ratios of milk fats and proteins explodes a whole catalog of dessert sauces.

Molasses is the leftover byproduct from refining sugar from sugar cane, pomegranate, or beets, each of which imparts complex flavor profiles and adds many of its nutrients to the syrup. Molasses is the brown in brown sugar. Most barbecue sauces call for molasses and/or brown sugar. I tend to not carry brown sugar in my kitchens because of its propensity to harden into an annoying brick–opting instead for molasses and white sugar always, which allows you to play with different types of molasseses and provides an easy to adjust bitterness knob.

You can also “make” brown sugar-adjacent substances by mixing honey, maple syrup or agave with granulated sugar (try it!). In truth, I find there is little advantage keeping brown sugar on hand all together; its shelf life is an order of magnitude shorter than its two parent items, recipes often call for light or dark brown sugar ($\sim$ 1 or 2 Tbsp of molasses per cup of sugar, respectively), adding needless inventory. Perhaps an issue is the variability of molasses spoonfuls, but then again brown sugar introduces pack as a new parameter.

Two especially irreplaceable sugar vehicles come in the form of honey and maple syrup, notably for breakfast items but just as essential for pastries, coffee drinks, and cocktails. I use honey in every vinaigrette recipe. Maple syrup has a high warming association not replaceable with spirits or bitters. Advocating for the use of maple, like honey, requires our agricultural industries to plant more trees and plow fewer fields, creating habitat for creatures that take flight. As a sweetener, honey is better for tea and maple for coffee. Flipped the other way, I find the flavor combination of coffee and honey just as odd as that of a vodka and coke! ²

With all that said: consuming less sugar is probably a good idea. Any modern diet requires an elimination of sugar. I have made an effort in previous kitchens to un-crutch myself from its use, and to always explore its substitution via honey or maple or juice reductions where ever I can (this is a great way to make friends with bartenders too, by the way). Corn syrup is mostly employed because it doesn’t crystalize, opacify, or dye candies, but is largely unnecessary otherwise. Now that its production is tied to the energy sector–which has made the price of an ear of corn skyrocket in the last twenty years–its use in cooking and baking is, in my opinion, best avoided altogether.

This tool allows you to explore the flavor network, a social graph for flavor profiles. The network is based on the Flavor Bible and soon the companion book What to Drink with What You Eat.

Search for an ingredient you like, and the graph will refine to give you a web of ingredients that share highly similar flavor profiles. Then, click on a new ingredient in the network to add it to your recipe above the search box (or to remove it). Clicking on a recipe item or a node has the same effect. Search is not sorted by the flavor metric, it is instead sorted lexically.

In this way, you can start building out recipes, menu items and tastings from a consensus of flavor combinations.

Overview

What you are seeing:

• the nodes with color are your recipe ingredients
• the suggested ingredients are determined by Jaccard similarity (default) or by one of the other options in the ‘lens’ dropdown
• if you choose the hybrid option, the suggested ingredients are fiducially split between:
  • the most similar ingredients in the flavor metric (similarity)
  • the most similar ingredients by text ranking (affinity)
• the edges from one ingredient to another are weighted by a consensus of chef and expert opinion ¹
• if your ingredient is missing, it was likely missing in the book (quinoa) or was pruned because its mentions were too sparse ²

If a node is present without an edge, it means that the ingredient has a very good similarity with your recipe, but wasn’t mentioned (connected) in its book-entry literally. Reconstructing ‘ghost’ entries and connections by training a model with listed affinities is one of the ultimate goals of this project.

The amount of suggestions gradually decreases as you add more ingredients to your recipe. This is for performance reasons, as with the physics simulation disabling itself at destabilization. When that happens, maybe you discovered a flavor affinity.

I have included autogenerated links from your recipe basket to a few popular recipe resources above the network graph, including a database for cocktail mixing. ³

Why and how

Understanding why I chose this text for the dataset is probably already apparent to its readers, but the key thing to take away is that the authors did a fine job formatting something computer readable and human usable–a rare feat! Most importantly, it is aggregated from chefs, from real humans in kitchens doing what works, what’s delicious, and what’s in season. Recipe API’s don’t have this kind of granularity, many rely too heavily on user data to seed recommendations. To my knowledge, this is the only dataset of this kind.

Technical tools only involve vis.js for visualization and BeautifulSoup for parsing. The data is scraped from the Flavor Bible, and the similarity matrix is calculated using Jaccard similarity for pairwise comparisons. I am working on cleaning up the initial data with some mix of modern techniques with some concoction of nltk, fuzzywuzzy and/or Bert. The current form was done entirely with regexp/bs4 parsing. The suggested nodes can be improved by using a weighted Jaccard probability distribution (arXiv). Source code for this part of the calculation (the text → dataset chain) is available on GitHub.

Inspiration

In 2019, I was helping fellow chefs come up with new specials. At this point in time, we were rolling about four-ten new specials as a team every week, ranging from brunch, cocktails, lunch, football apps and our highly anticipated farm-to-fork pop-up dinners. But sometimes you just get plain stuck. A good trick, at least for creativity, is to set rules so you have some boundaries to push. But if you are going to set rules, they should at least solve a few things:

1. do something new
2. use something old
3. feature three things in season

I hate having extra stuff around, but I love new stuff coming in, yet I don’t like wasting things, but then I actually look forward to doing inventory. Ah, Schrodinger’s cook.

Specials: we would work out new ideas together over the prep table. Sometimes ideas required working things out on paper, usually butcher’s, and occoasionally crude graphs of our plate setups evolved. These were sketches of sauce and protein layouts, heavy edges between ideas if their pairing ‘sang’, then as a guide hanging from the ticket rail on the night a feature debuted.

Karen and Andrew’s book was gifted to me later that year, and it changed my game. It finally put in words a mental ranking of flavor profiles based on ingredient query. I had a good resource that gave answers, and especially new ideas, quickly. And it was thorough enough to trust.

This method was so helpful, I started dreaming of a computer tool to help me sketch out this process. I remembered a reddit post that spurred others to lay out some of the underlying ideas here: overlapping communities :: compatible flavors.

Broader thoughts

I believe the impact of mathematical concepts to the broader culinary scope to be a major upgrade in our thoughtfulness about food. To extend its application, in creativity and clarity, not abused in statistics to pressure a sale and disable the creative mind. While I do see how a tool like this could provide immense practical application in the distribution world, my focus here is to empower chefs, bartenders, brewers, baristas, and sommeliers to create new things.

When it comes to tools available to chefs, compared to musicians, writers, and artists, chef’s are unfortunately at a disadvantage creatively. Yes, we have recipes, but those are instructions, and do little to help us build on ratio or balance. What might be more helpful, I think, is a playground for putting new food ideas together.

The graph may take a moment to load.

The search bar is the major addition to the graphing methods. Nodes can be clicked and added to a subgraph builder. You can continue to search for new node members in the search bar (which has a rudimentary autofill that’s a straight json query) and clicking on them will add them to the builder. Simultaneously, the graph will reduce to a graph containing only all nodes with edges linked to nodes in the builder.

Items can be removed from the builder either by clicking the little builder tabs or re-clicking the node. Clearing the builder bar completely will redraw the whole graph.

Testing and development was done on the mini pesto data set I made for What is Pesto?. Recipe builder coming soon(!)

Please email me at wyatt@brege.org with any questions.

Dataset can be found here:

• nodes.json
• edges.json

Lingering annoyances:

• Slow
• Javascript needs clean up
• I have great fear running this on my 700x3000 dataset..

1. Data for the network is stored in two separate JSON files in this page bundle:

   • nodes.json
   • edges.json

2. The shortcode and post-local javascript work together:

   • fruit-network.html
   • fruit-network.js

     {{ $nodesPath := .Get "nodesPath" }}
     {{ $edgesPath := .Get "edgesPath" }}
     
     <style>
       #mynetwork {
         background-color: #f5f5f5; /* a medium gray color */
         border-radius: 10px;
         border: 1px solid #cccccc;
         margin: 5px 0 40px 0;
       }
     </style>
     
     <div id="mynetwork" data-nodes-path={{ $nodesPath }} data-edges-path={{ $edgesPath }}></div>
     
     <script src="https://visjs.github.io/vis-network/standalone/umd/vis-network.min.js"></script>
     <script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
     <script src="js/fruit-network.js"></script>
     

This will provide network graph physics where the nodes are images (all sourced from Wikipedia. Hugo template for completeness:

{{< fruit-network nodesPath="data/nodes.json" edgesPath="data/edges.json" scriptPath="js/fruit-network.js" >}}

Classification

Basil pesto is the common pesto. Its recipe typically calls for five key ingredients: $$ \begin{aligned} \textrm{basil pesto} = \textrm{basil} &+ \textrm{extra virgin olive oil} + \textrm{garlic} \\ &+ \textrm{parmesan cheese} + \textrm{pine nuts} \end{aligned} $$

Chimichurri includes a vinegar, usually red wine vinegar, which pesto does not, although pesto can suspend acid through citrus. Chimichurri often uses rough chopped or ground herbs, almost always oregano and parsley, but some modern applications will puree chimichurri as one does with pesto.¹

Pesto is also different from gremolata, which like chimi is also made with parsley, so you may refer to carrot-family herb sauces as either gremolatas or chimis, and mint-family herb sauces as pestos, and chefs will know what you mean, even if they don’t quite agree with the abstractions.

A table is provided as a summary of these three sauce’s distinctions.

“Green Sauces. The author made spring sauce up, but it’s very good on crostini.”

But there is a small catch, because some chefs may also say ‘arugula pesto,’ which might have no mint-family herb mixed in. This leafy sauce resembles pesto in process and appearence, and provides the effect of coating pastas in bright green colors, especially when suspended in stablized fat like reduced cream or mayonaisse. A chef can employ arugula for its liquorice and peppercorn aspects, or baby spinach which adds tang (like nori) when cooked, but she will most likely apply it because of its food coloring properties (vibrancy). Similarly, ‘spring pea pesto’ is made with a puree of spring peas and mint, and on crostini is a fantastic crudité.

In this way, pesto* is generalized from its elements, describing a sauce of process and appearence while flexible in its flavor balancing rules. This is pesto* as a category, not pesto as a set. For all intents and purposes, chimis and gremolatas are pestos*.

Ingredients

The key to most pestos is to use a motor driven device for pureeing the sauce. The two best tools available and in common use both in commericial and residential kitchens are the vitamix or immersion blender. Food processors and ninjas work just as well.

Hint: always add cheese last

Puree everything besides cheese first. You don’t want to gum up your blender, so you can fold the proto-pesto with grated parmesan in a mixing bowl after pureeing the produce. In restaurants, you can make a really fine proto-pesto (i.e., loose pesto) by pureeing the produce and oil for an extended period of time. It is easier on the motor and on the blades. You can be lazier picking your leaves on your basil or include more stems, because the cheese absent creates less friction. Save the cheese for last.

Greens

Most herbs from either the carrot- or mint-families will work great, and many green tops of root vegetables (turnip, radish) are, while not interchangeable, usable and often complimentary in pestos. Caution: while tarragon can be made into a pesto*, its deployment in cold-prep sauces is rare (marinades, green goddess dressing) and is typically utilized better in hot-prep sauces–especially béarnaise. Sage, rosemary, and lemon verbena require special practice. Thyme is time. Use edible garnishes. Keep away from the ornamental “perfumes”.

Recommend: basil, mint, parsley, oregano, marjoram, arugula
Avoid: lavendar, tarragon, lemon verbena, rosemary

Cheese

Try to stick with a hard cheese that has longer room temperature stability. Cheeses with hard interiors and developed rinds will impart the necessary fats and impactful aromas while also being more conservative in the funky bleu cheeses, chalky goat cheeses, or sour yet creamy compositions of farmer/cottage style cheeses–namely, ricotta, cotija or paneer. An old queso fresco that has been air dried in the fridge, however, is a mighty substitute to parmesan on a budget.

Recommend: parmesan, hard rind, feta, crumbly
Avoid: soft, bleu

Citrus?

Of course, pesto is also seasoned (sodiums, peppercorn) and lemon zest or lemon juice is an especially common addition. Adding citrus tunes flavor, because it combats salinity (parmesan saltiness is not equal between products) and piquancy (garlic cloves grown in drought can be remarkably ‘hot’). Because lemons are a hybrid of pomelo and citron, grapefruit juice is an equally applicable substitute.

Recommend: lemon zest, grapefruit juice, kombucha vinegars
Avoid: orange, distilled vinegars

Nuts

Pine nuts are the nut of choice in recipes from canonical sources. People don’t have pine nuts in their house. They do have other nuts leftover from baked good preperations. In restaurants, pine nuts are toasted in hot ovens. They are applied whole to pizzas and salads, so because they can live in a recipe and two auxillary dishes, they define themselves as roleplayers in menus and inventories. However, pine nuts are expensive. Because they are normally toasted, errant chefs may leave them in ovens for moments too long, destroying a product whose cost is equivalent to whole numbers of the average hourly wages. This is not good for anyone. I’ve never done this before, I swear.

Instead, some nontropical tree nuts are immediate substitutes if you are concerned only with the context of pesto: walnut, almond, and pistachio. This does not mean cashews are off limits, nor peanuts. Consider the role they play with basil and different oils in Thai cuisine. But maybe we don’t often think of peanut butter and parmesan together, so we avoid peanuts in pesto. That doesn’t mean you are violating international treaties by substituting deluxed mixed nuts from the grocery store in place of pine nuts. Pine nuts are the most replaceable element of classical pesto.

Recommend: pine nut, walnut, almond, pistachio
Avoid: peanut, cashew, macadamia, brazil, pecan

Oils

It is by no coincidence that extra virgin olive oil is the go to substrate (notice also olives and the alternative nuts previously mentioned grow adjacent to each other). Its Mediterranean origin is the required consideration for almost all member dishes in its cuisine. Some oils are better than others when olive oil is out: avocado, walnut, grapeseed. Avoid soy and canola oils which are better suited for frying, and harm basil less in lite applications like salad (Italian) dressing. Coconut oil can be used, but is not ideal because of its high melting point. However, you can create a spreadable pesto by using coconut oil–if the proto-pesto wets are tempered into coconut oil carefully. Coarser nuts in the spread make for a provocative but health-conscious party favor.

Recommend: extra virgin, walnut, avocado
Avoid: canola, soy, peanut

Alliums

Lastly, we generalized garlic to alliums above for good reason. You can make garlic scape pesto, which uses the green tops grown above earth from the garlic bulb. But, you can also use spring onions and/or shallot bulbs in place of garlic, which is especially more pleasant with lakefish and seafood. Although, garlic is preferable to onion in the regime of shellfish and, notably, the mollusk. You can also use radish bulbs, as well as radish or mustard greens in pesto, which simulates “bite”. In place of garlic, this is a quite useful substitution when pestos are applied in dressings for leafy salads or pasta salads.

Recommend: fresh garlic, young onions, shallot, some pickled
Avoid: large onions, leek, pearl onion, cooked garlic

Recipe

I’ve included a basic production recipe for Basil Pesto that creates a structurally “loose” pesto. My application could be added to heavy mayo for basil aioli, brushed on bread for grilling, and thickened with reduced (50%) cream for penne pasta dishes (bechamel cheese sauces work there, too). Cheese plays a more vital role when pesto is a topping where the goal is to gratinize, or in the presence of crackers.

What is sauce?

When you play this game of generalizing sauces into irreducible primitives, you eventually question with good reason why the five mother sauces are the way that they are.

If postulated today, there would no doubt be endless criticism of its organization, lack of other fundamentally different sauces (gastriques?), exclusion of other world foods, and few would be convinced of any universality. But Escoffier’s postulate has historical importance defining the role of these hot-held sauces in a greater culinary scope. Less valuable now as theory, it is more so enforced as a performance lesson: a chef must learn how to make the mother sauces to be the best in her trade.

If you were to reduce every cold sauce into fundamentals (vinaigrette, aioli, coulis, etc), to what extent can you? There is at least some argmunent: the preperation does require special work. While the heated sauces have an additional thermal parameter (‘careful not to scald, to break..’), the cold sauces spend energy kinetically: emulsify, puree, grind (‘careful not to overwhip, to break..’). In this sense, attention isn’t voluntary, so it separates itself from chopping, fermenting, or stirring to mix, giving an argument of practicality some teeth in the need for order. No, Ranch is not a mother cold sauce, but mayo does require special work and is an elemental part of Ranch.

In my mind, I do consider some categories of cold sauces as fundamental to the education of the garde (those mentioned, probably closer to seven in total). So with liberty, let’s for sake of argument hold salsas and pestos in different regard. I cannot speak to the genre it belongs, but: if you have never tried blending tomatillo and basil together, you are missing out on one of life’s best kept secrets.

Relative to the root of the Hugo website directory, here’s some basic files to make this interactive. Note that The JSON data and CSS is added inline here to make the scope of this tutorial focus on Hugo-specific structures.

1. The javascript file lives in this page bundle:

   • toy-network.js

2. This file accesses data for the nodes and edges from two JSON files in this page bundle:

   • nodes.json
   • edges.json

3. In the shortcodes directory /layouts/shortcodes/:

   • toy-network.html

     <div id="mynetwork" data-nodes-path="data/nodes.json" data-edges-path="data/edges.json">
         <script src="https://visjs.github.io/vis-network/standalone/umd/vis-network.min.js"></script>
         <script src="js/toy-network.js"></script>
     </div>
     

4. Do the normal way of making a post in Hugo, but invoke the shortcode within the body of your markdown:

   • index.md

     {{< toy-network nodesPath="data/nodes.json" edgesPath="data/edges.json" scriptPath="js/toy-network.js" >}}
     

This will provide the simple network graph above.

🔗 RDR2 Completionist Checklist (Google Sheets)

2025 Note on Google Drive/Sheets

In order for you to edit this, you need to:

1. Open the spreadsheet in 🔗 Google Sheets while signed in with your account
2. File > Make a copy

This will add a completely editable copy in your Google Drive. I’ve gotten a lot of requests to edit this spreadsheet!

If you do not have a Google account: you can use this PDF version instead. On my last playthrough, I made sure the page breaks were well placed so I could print them out and mark them off as I played.

While not a replacement by any means for the various user made maps and guides out there, I wanted everything consolodated in a form similar to what I had already been doing with a little notebook while I played.

• Hunting: these are animals that you need for the Trapper, Mr. Pearson, Ms. Hobbs, and the legendaries. Daggers † are reserved for the epilogue, officially, and have a grey background.
• Plants + Fish: combined on the same sheet so they can be printed together. Also, there’s some markers in there that indicate challenges, recipe usage/bait types, and some other information available :on hover:
• Horses: this was actually a major pain to put together! With Rockstar updates and online content being incosistent, there is a definite need out there for these to be consolodated and made tidy.
• Missables: sorted first by character; this includes:
  • camp requests
  • companion activities
  • missable compendium weapons and “unique item” satchel contents
• Collectables: a grid to help demarcate collectable discovery (again, some have grey backgrounds if inaccesible for much of the story)
• Weapons: see key for more details

This guide can be used to checkoff discovered recipes and items, avoiding purchase from the fence, but mostly to fully complete the compendium (and sometimes it takes too many clicks and scrolls in the pause menu). Items that are bolded are uniquely obtained, some of which are missable, or can be sold and never obtained again.

There are page breaks to make printing easier. Saving a copy of the Google Sheet will also allow you to use it on your phone quite well. Or, here’s the full PDF Checklist ⬇️

I take as input the spreadsheet provided by Robert Vollman, which has not been updated with GVT data yet. I made minor modifications to his spreadsheet in LibreOffice Calc to make it export to the CSV file format well. The code calculates OGVT by player, which is weighted against his own team’s Threshold Offensive Contribution by forwards ($TOC_F$), or defensemen ($TOC_D$), per minute, rather than league wide.

To get an estimate of how good a goal is compared to an assist, we estimate that a goal scored contributes 1.5 times as much as an assist contributes to a goal. Therefore, the calculated goal value (or assist) scored by an entity $x$ is $$ \begin{aligned} GV_x &= \frac{1.5 G_x}{A_x + 1.5 G_x}, \\ AV_x &= \frac{GV_x}{1.5} \end{aligned} $$ where $G_x$ is goals scored by either an individual, $x=i$, team, $x=T$, or the league as a whole, $x=L$, and $A_x$ are the assists scored by those subcategories.

The total offensive contribution of all forwards, $TOC_F$, is determined by

$$ TOC_F = \frac{\sum_{f \in T} G_f \times GV_T + A_f \times AV_T}{\sum_{f \in T} MP_f} \times OTV$$ where $MP_f$ is the minutes by forward, and the offensive threshold value is $OTV = 0.75$ via Tom Awad or $0.58$ via Alan Ryder (I chose the former). I chose an uppercase $F$ so that one may distinguish this value, which applies to all forwards on the team, from an individual forward, $f$.

The final formula to calculate $OGVT$ for each forward $f$ is, according to Awad, then $$ OGVT = G_f \times GV_f + A_f \times AV_f - MP_f \times TOC_F $$

Additionally, I wanted to get a sense for one player’s value to the team in relation to his salary cap hit. Here, I show from the 2016-17 NHL regular season $OGVT$ versus Salary Cap for the Stanley Cup Champion Pittsburgh Penguins, the cap-troubled Detroit Red Wings, and the young Edmonton Oilers with generational talent Connor McDavid (only forward skaters).

However, in debugging my code, something seemed strange to me. This first term in the $OGVT$ expression, with some math, reduces to the number of goals by that individual: $$ \begin{aligned} G_f \times GV_f + A_f \times AV_f &= G_f \times GV_f + A_f \times \frac{GV_f}{1.5} \\ &= \left( G_f + \frac{A_f}{1.5}\right ) \times GV_f \\ &= \left( G_f + \frac{A_f}{1.5}\right ) \times \left( \frac{1.5 G_f}{A_f + 1.5 G_f} \right) \\ &= \left( 1.5 G_f + A_f \right) \times \left( \frac{G_f}{A_f + 1.5 G_f} \right) \\ &= G_f. \end{aligned} $$ So, unless I’m misunderstanding Tom Awad’s definition of terms here:

A player’s OGVT is therefore:

OGVT = (G x GV) + (A x AV) - (MP x TOC)

Where G is the player’s goals, A his assists, MP his minutes played, GV his goal value, AV his assist value, and TOC the Threshold offensive contribution value for his position.

I don’t quite understand how this first set of terms is relevant, as it essentially removes the direct value of a skater’s assists in the calculation of this catch-all offensive statistic.

    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;


    # intermediate configuration. tweak to your needs.
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
    ssl_prefer_server_ciphers on;


    # to score better grades on
    #  https://observatory.mozilla.org/analyze.html?host=brege.org

    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Content-Security-Policy "default-src 'self';";
    add_header Referrer-Policy "no-referrer, strict-origin-when-cross-origin";

    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;
    ssl_stapling_verify on;

We include the headers we obtained in the last post in here as well. The main configuration file /etc/nginx/brege.org contains:

server {
    server_name *.brege.org;
    return http://brege.org/;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    server_name  brege.org;
    # enforce https
    return 301 https://$host$request_uri;

    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot

}

server {
    listen 443 default ssl;
    listen [::]:443 default ssl;

    root /usr/share/nginx/brege.org;
    index index.html index.htm index.php;

    server_name brege.org;

    ssl_certificate /etc/letsencrypt/live/brege.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/brege.org/privkey.pem; # managed by Certbot

    include security_headers.conf;
    include letsencrypt.conf;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }

    location ~ ^/(?:feed|feeds|rss)  {
        return 301 /index.xml;
    }

}

Note that first server block does not work as expected. On a separate instance, I tested this and confirmed that http://abc.mysite.org does get redirected to https://mysite.org, but I think that since my site has been HSTS Preloaded, this doesn’t allow http://abc.brege.org to redirect to https://brege.org.

After some trial & error and searching around, I came up with the following to be placed in my Nginx server{...} block:

    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Content-Security-Policy "default-src 'self';";

Now I am receiving an “A+” from the observatory!

I also went to Google’s submission page to have your site put on the HSTS preloaded list, so that all users of Google Chrome access your site through HTTPS by default.

I tried implementing this method, but it is very long and requires many modifications to the post-receive hook to get it working on a per-user basis.

Here I outline a way to do this in a more conservative manner.

Installation on development machine

1. On a development laptop, Fedora, I installed Go and Hugo via

sudo dnf install golang
mkdir -p ~/build/go
export GOPATH=$HOME/build/go
go get -v github.com/spf13/hugo

2. Clone repository

cd ~/build/
git clone git://github.com/brege/brege.org.git

New workflow for publishing an article

1. I once again use Hugo to create my article

cd ~/build/brege.org
hugo new post/new-way-forward-for-deploying-brege.org.md

2. Add content, run hugo server, then preview http://localhost:1313 in a browser to determine what needs to be edited. The cool thing about this is you can edit/add a file while the test server is running. You don’t even need to refresh the browser page whenever you save the file!

3. Run rm -r public/ ; hugo to remove the old and create the new public/ directory on the development machine.

4. Ship the changes to the production server (DO droplet) with rsync:

rsync -avP --delete --exclude=".well-known" public/ brege.org:/usr/share/nginx/brege.org/

The new article will now show up on the https://brege.org/post/new-way-forward-for-deploying-brege/

5. Finally, commit the new article to GitHub:

git add content/post/new-way-forward-for-deploying-brege.org.md
git commit -m "Add article"
git push -u origin master

Much nicer way to go about it. Images and other assets can be more readily copied and modified. I can edit a webpage on the fly before publishing layout-bugs and typos to the world (or, at least to a lesser extent).

dfox="~/Preview/firefox-developer/firefox --profile ~/.mozilla/firefox/7ahl24yk.default/'

I was able to launch Firefox Developer quickly enough. Unfortunately, I found it annoying that I couldn’t just type [Super Key] + “Fire” in GNOME and be on my way to browsing the web.

Adding a launcher icon for Firefox Developer

I wanted to add a desktop entry/launcher icon for Firefox Developer. I searched and found the Alacarte package, which is the GUI-way of doing this. Alas, adding entries in Alacarte led to the following problem:

Problem 1: Alacarte didn’t allow me to actually click “OK”. Both with the version from the Fedora repositories, alacarte-3.11.91-4.fc24 via sudo dnf install alacarte, as well as a version built from source, 1bb265f7 via git clone git://git.gnome.org/alacarte, did I have these issues.

Solution: Create the .desktop file by hand. (Also, probably submit a bug report.)

In ~/.local/share/applications/firefox.desktop, I added the following lines:

[Desktop Entry]
Comment=Browse the World Wide Web
Terminal=false
Name=Firefox Developer Browser
Exec=~/Preview/firefox-developer/firefox --profile ~/.mozilla/firefox/xxxxxxxx.default %U
Type=Application
Icon=~/Preview/firefox-developer/browser/icons/mozicon128.png
Categories=Network;

where xxxxxxxx.default is my default profile directory. My starting point was browsing a few .desktop files in ~/.local/share/applications/ (user installed applications) and /usr/share/applications/ (system-wide applications), for reference.

I ran into an issue, however:

Problem 2: If I added Firefox to my Favorites (the dock on the left where openened and manually added applications sit), I would get a duplicated menu entry.

Solution: Add StartupWMClass=Firefox to ~/.local/share/applications/firefox.desktop

I came acrossed a proper solution here. In short, I fired up Firefox Developer, ran xprop WM_CLASS in the terminal, clicked the Firefox window, and added the entry to firefox.desktop

My final file looks like this:

[Desktop Entry]
Comment=Browse the World Wide Web
Terminal=false
Name=Firefox Developer Browser
Exec=~/Preview/firefox-developer/firefox --profile ~/.mozilla/firefox/xxxxxxxx.default %U
Type=Application
Icon=~/Preview/firefox-developer/browser/icons/mozicon128.png
Categories=Network;
StartupWMClass=Firefox

Exercise - Repeating the process for Earlybird (Thunderbird Alpha)

1. Go to the Thunderbird Release Channel and grab the package for Earlybird (or the the Beta, if you choose):

   

   Or, from the command line:

   cd ~/Preview`
   wget https://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-aurora/thunderbird-51.0a2.en-US.linux-x86_64.tar.bz2`
   tar -xvf thunderbird-51.0a2.en-US.linux-x86_64.tar.bz2 -C thunderbird
   

If you use Thunderbird already, determine your profile: ls ~/.thunderbird/*.default

2. Create the desktop entry, ~/.local/share/applications/thunderbird.desktop, with the following contents:

[Desktop Entry]
Name=Thunderbird
Comment=Send and receive mail with Thunderbird
Type=Application
Exec=~/Preview/thunderbird/thunderbird --profile ~/.thunderbird/xxxxxxxx.default %u
Icon=~/Preview/thunderbird/chrome/icons/default/default256.png
Categories=Network;
StartupWMClass=Thunderbird

Replace xxxxxxxx.default with your profile.

For any other application like this, I would just copy firefox.desktop to myapplication.desktop and adjust the contents to suit your needs. Then, in general, launch the application and run xprop WM_CLASS to determine the value to put in the StartupWMClass= field.

https://github.com/brege/brege.org

    location ~ ^/(?:feed|feeds|rss)  {
         return 301 /post/index.xml;
    }

Now people can go to https://brege.org/rss to find the RSS feed. This rewrite rule could be expanded to match other common feed URI guesses.

• In /etc/systemd/system/certbot.service:

[Unit]
Description=Let's Encrypt renewal

[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet --agree-tos

• In /etc/systemd/system/certbot.timer:

[Unit]
Description=Daily renewal of Let's Encrypt's certificates

[Timer]
OnCalendar=daily
RandomizedDelaySec=1day
Persistent=true

[Install]
WantedBy=timers.target

• Enable and start the timer:

sudo systemctl enable certbot.timer
sudo systemctl start certbot.timer

brege.org is deployed with the following services:

• Namecheap - DNS Registrar
• Let’s Encrypt - Free SSL certificates
• Digital Ocean - Droplets for $5 a month

brege.org makes use of the following tools:

• Nginx - Fast reverse proxy
• Debian Jessie - Operating system running on our DO droplet
• Hugo - Static site generator

Hosting

First thing I did was register my domain on Namecheap. It was around $12 to register brege.org.

Next bit was deciding on a hosting provider. I gave Amazon S3 some consideration, since it’s practically free for a static site, but after playing around with their tools it seemed far too confusing to use. I thought I’d try Digital Ocean. Their tutorials on deploying various web apps have been quite helpful in previous searches, and they host good things like lkml.org. The $5 droplet service offers 512 MB RAM, 20 GB SSD, and 1 TB of bandwidth. Good enough.

I selected the debian installation on the “Create Droplet” page and created a new ssh key:

ssh-keygen -t rsa -b 4096 -C "wyatt@brege.org"

I like to have a different key for each remote service I use (GitHub, Clusters, Servers, etc). So for the bit about saving the key, I just substituted the default id_rsa with do_rsa, chose a password to unlock my key, and uploaded do_rsa.pub in the DO web interface.

This gets added to /root/.ssh/authorized_keys on your DO droplet. I added another user on my droplet so I wouldn’t always be logging in as root. You can copy this key from your workstation to the user’s home directory on your droplet via

ssh-copy-id -i ~/.ssh/do_rsa newuser@ip.of.my.droplet

DNS settings can be confusing. Setting “Custom DNS” in Namecheap and setting the nameservers to DO’s nameservers was simple enough. Setting records on namecheap is kind of annoying. On DO it is really intuitive to set up records.

Nginx

I’ve been using nginx on my personal home server and quite like its configuration and speed.

sudo apt-get install nginx

I’m not exactly sure what the consensus is on where your site should live in your debian install. /usr/share/nginx/? /var/www/? The html/ directory in either of those? The way I treat it is that my static site will live in /usr/share/nginx/brege.org/. If I want to password protect any section of my site, my .htpasswd will live in user www-data’s home directory, /var/www/. If I want to host some other site for abc.brege.org, it will live in /usr/share/nginx/abc.brege.org/.

I removed the default site configuration file installed on debian.

rm /etc/nginx/sites-available/default
rm /etc/nginx/sites-enabled/default

I made a new configuration file /etc/nginx/sites-available/brege.org with the following settings:

server {
    server_name *.brege.org;
    return http://brege.org/;
}

server {
    listen 80;
    listen [::]:80 default_server;

    server_name  brege.org;
    # enforce https
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 default ssl;
    listen [::]:443 default_server;

    root /usr/share/nginx/brege.org;
    index index.html index.htm index.php;

    server_name brege.org;

    ssl_certificate /etc/letsencrypt/live/brege.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/brege.org/privkey.pem;

    include letsencrypt.conf;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }

}

You can split each server{ ... } block into its own config file if you like. One redirects all *.brege.org requests to brege.org, including www.brege.org. The second one redirects all requests from http://brege.org to https://brege.org. The third is the main configuration for the site. This is where you put reverse proxies that redirect internal ports to subdirectories on your site.

Link the new configuration file and restart nginx:

sudo ln -s /etc/nginx/sites-available/brege.org /etc/nginx/sites-enabled/
sudo systemctl restart nginx.service

Let’s Encrypt

In /etc/nginx/letsencrypt.conf, I have:

location ^~ /.well-known/acme-challenge {
    alias /var/lib/letsencrypt/.well-known/acme-challenge;
    default_type "text/plain";
    try_files $uri =404;
}

After making any changes to your nginx configuration, restart the service:

sudo systemctl restart nginx.service

We need to generate a free SSL certificate from Let’s Encrypt. I followed the instructions on https://certbot.eff.org/#debianjessie-nginx to install certbot on Debian Jessie.

As root, I ran

certbot certonly --webroot -w /usr/share/nginx/brege.org -d brege.org -d www.brege.org

and completed the wizard. YMWV

Hugo

Since I’ve been getting into the habit of writing my notes and personal documentation in Markdown, I wanted to be able to quickly copy and paste snippets to-and-from this website. I found Hugo, which is written in Go. It is very fast to build a static website under. Every post is written in markdown, and running the hugo command only takes ~50ms to generate all the html/css/js files. No more having to fiddle with html/css/js.

First, the version of Go required to run Hugo must be 1.5 or greater at the time of writing. I went to https://golang.org/dl/ and grabbed the latest release:

cd ~/
wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz
tar xvf go1.7.1.linux-amd64.tar.gz
rm go1.7.1.linux-amd64.tar.gz
sudo mv go /opt/
sudo chown -R newuser:newuser go/
cd go/ 

I added the following to /etc/profile

export PATH=$PATH:/usr/local/go/bin
export PATH=$PATH:/opt/go/bin

Next, install Hugo via git:

export GOPATH=/opt/go/bin
go get -v github.com/spf13/hugo

More information available here.

Checkout https://themes.gohugo.io/ to see if there’s a theme that you like. I thought the academic theme looked pretty nice. I followed the instructions in the README.md to create my site.

The workflow is typically:

1. Creating a new post

hugo new post/hello-world.md

2. Adding what I want to say with simple markdown format in content/post/hellow-world.md after the +++ ... +++ front matter.

3. Running hugo, which creates all the files for your site. It may be wise to remove the public/ directory and the /usr/share/nginx/brege.org/* subdirectories to prune any orphaned articles.

4. Copying the generated public/ folder over to the document root.

sudo rsync -avP public/ /usr/share/nginx/brege.org/

5. Visit brege.org in my browser.

In fact, there is probably no need to run go/hugo on my DO droplet at all. The proper way is probably to do everything on a local machine, add a new article and check at localhost:1313, then rsync the contents of public/ to your document root on your droplet.

Putting the source of this site on github might be even better. I’m already keeping track of my changes with a local git repository, so I might as well make it available on GitHub. Plus it seems worth it just for the sake of writing articles locally.

Update: I am getting closer to implementing my initial thoughts here: https://brege.org/post/new-way-forward-for-deploying-brege/