Byrne Hobart
The Economics of Engineering Blogs
A company is valuable if it does something that customers value, and that other companies can't do. A consequence of this is that companies have a vested interest in keeping secret anything that would help their competitors copy them better or would help their customers solve the same problem in-house. So oil and gas companies usually don't go around telling everyone about which leases are undervalued, hardware companies try keep new devices under wraps, and traders don't talk about trades they're making (but do, of course, talk about trades they've already made from time to time—if you’ve made a bet that the consensus on something is wrong, popularizing your thesis is the fastest way to make it right), etc.
And then there are software companies: a Google engineer is perfectly able to read Meta telling them how to scale a global network of datacenters, Netflix will happily share some tips YouTube can use to automate the detection of scene breaks in shows, Shopify wants you to know how to improve Ruby's garbage collection, GitHub will teach GitLab how to improve natural language searches of source code, Spotify will give a boost to Apple Music's efforts to customize a playlist, and there's nothing stopping someone at Indeed from perusing LinkedIn's post on how to deliver code faster.
It's a big enough category of content that there's even an aggregator devoted to engineering blogs, a surprise since it’s counterintuitive that these exist at all—tech companies are constantly at each other's throats. In some categories, they compete head to head, and in others, they're in adjacent layers of a supply chain and thus have a strong incentive to commoditize one another ($, Diff). Many of them offer products that could, at least in principle, be replaced with some homegrown solution. And here they are, sharing all kinds of helpful writeups about their successes and failures, without bothering to geofence their competitors out.
Why?
There are theories of varying levels of cynicism here. Two closely-related ones are the intimidation model and the wild goose chase approach. Why does Meta want you to know how to optimize datacenters? So you come away impressed with the cost advantage they've acquired, and worried about how much better they'll be by the time you catch up. You can read any “How we succeeded” post as a warning not to even try, at least in that domain. The wild goose chase model suggests that engineering blogs focus on projects that are much harder than they seem to be, and that require extensive upfront work just to demonstrate how much time the finished product will take. In this model, the result of a fascinating writeup on a cool project is wasting their competitor’s time. There are precedents for this; allegedly, the multi-armed bandit problem was so time-consuming to Allied researchers in the Second World War that they looked into ways to get the Germans to try to solve it specifically to waste their time.
But these both point to a better model: engineering blogs focus on problems where the solution is a necessary but not sufficient part of what they do. And, ideally, they focus on problems that are complementary to scale that only the publisher of that post has. Take Netflix's recommendation, or pretty much any case where a huge company talks about applying machine learning at scale: the safest time to do this is when the scale of the problem is big enough that smaller companies wouldn't get similar results.[1]
That would explain why publishing doesn't have much downside. But what about the upside? There are a few cases here:
- It's good for recruiting: people want to work on interesting problems, and sharing the results is one way to attract them. (In a way, this is the entire model of academia: a published paper is partly an invitation to build on or refute its results.)
- It's a way to get feedback, which could be a subset of recruiting, but not necessarily. Most of the time, there's a good answer to "Why didn't you just do X?" but occasionally there is some X that was actually worth doing. (This can be especially interesting when there's an obscure theoretical result that has relevance to the practical implementation. Especially in ML/AI, the pace of publications is too much to keep up with, and the lead time between starting and finishing a project can mean that the state of the art changes over the course of the implementation.)
- It helps with retention: there's some amount of labor alienation in being one of thousands of people making incremental contributions to some massive system. There are engineers at big companies who have, single-handedly, created human lifetimes worth of free time for people just by reducing the latency of some app.[2] The numbers are similarly staggering when you consider monetary impact; spend enough time among people who work at big companies and you will meet individuals who have shipped features or even tweaks that can be rigorously shown to have produced hundreds of millions of dollars in value. Giving them an opportunity to take some credit for this in a blog post—to be able to tell friends and family members who use the service they contribute to that they helped to make it what it is—gives people another reason to keep doing more of the same.
- Some engineering blog posts help demystify the product, or allay conspiracy theories. Why does Meta walk through the ranking algorithm for the news feed? To make it slightly harder to claim that somewhere in the code there's a line like
if source == russian_propaganda_mill then pin_to_top_of_feed. And even there, the point isn't to stop people from saying that, but to make those people seem less technically sophisticated than the people who don't think it's happening.[3] - Earlier, we noted that traders don't publicize their ideas while they're putting on a trade, but will sometimes share them later. This doesn't apply to most engineering blogs, but it does work in some cases: publishing good documentation about impressive projects done using the company's chosen tech stack is a way to promote it as a standard and own part of that standard ($, Diff). It's bad for Shopify if someone else writes a giant e-commerce suite in Ruby, but it's overall good for them if more companies are using Ruby for huge projects, since it deepens the hiring pool. (Ocaml would probably be a much more obscure language without Jane Street's promotion (plus, they went even further than blogging and wrote a book, too). So there are a few cases of hyperstition going on, though it's less about blogging something into existence and more about using the blog to set a more favorable equilibrium.
Using an engineering blog as a recruiting tool essentially forces the company to raise its talent metabolism: it's easier to attract good people, but outside recruiters can see who published what, and the more impressive the project is the more likely its participants are to get attention from other employers.
But for historical reasons, the marginal cost of this is lower in software than in other industries. The biggest cluster of software companies globally is in and around Silicon Valley, which means it's subject to California's labor laws—which include a notorious reluctance to enforce noncompete agreements. It's a small world, and there's a whole cohort of people whose job is to simultaneously be well-informed, be well-connected, and to catalyze people either leaving companies or starting them. So tech's talent metabolism has always been running hot, even before the software industry as such existed. As Tom Wolfe put it, writing about the early days of the semiconductor business: "Every year there was some place, the Wagon Wheel, Chez Yvonne, Rickey's, the Roundhouse, where members of this esoteric fraternity, the young men and women of the semiconductor industry, would head after work to have a drink and gossip and brag and trade war stories about phase jitters, phantom circuits, bubble memories, pulse trains, bounceless contacts, burst modes, leapfrog tests, p-n junctions, sleeping-sickness modes, slow-death episodes, RAMs, NAKs, MOSes, PCMs, PROMs, PROM blowers, PROM burners, PROM blasters, and teramagnitudes, meaning multiples of a million millions."
It's a lucky situation. Industries can thrive by keeping secrets, but if they're at least partly competing on openness, overall progress happens a lot faster. More industries are slowly moving in that direction; software had a head start because the barriers to entry are so low and because so much of the tacit knowledge can be passed down in text form. Video has made it easier for other fields to share information the same way. It won't go as far in other fields, both because of the path dependence behind software's openness and because nearly-zero marginal costs mean that when a software product gets commoditized, the price doesn't drop by 10% or 20% or 50%, but goes straight to zero and stays there forever. But in tech, it's not an advantage to practice this kind of openness; it's a competitive necessity. Which means that in other fields, being the first to do this, and to do it well, is a meaningful advantage.
Thanks to Gergely Orosz of Pragmatic Engineer for the tweet that inspired this.
Disclosure: Long META, MSFT.
In fact, this can be an implicit version of the wild goose chase thesis, if the effectiveness of some technique starts out lower but reaches a higher ceiling. And that seems to be the case generally; if the corpus of text is 1,000 tokens, you can get a better language model by hand-crafting some rules than by applying a model optimized for a larger dataset. And applying the big model will require more expensive overhead. It's fun to speculate here: what if one step the company takes before writing about some technique is to run it on a subset of their data equal to what their next-biggest competitor has, and confirm that the results are bad enough not to be worth it. ↩︎
The threshold of human perception is around 100 milliseconds. There are ~1.3bn iPhones in use today, and the average user checks their phone ~60 times a day. Suppose some Apple engineer speeds up the process of opening the phone by 10ms, i.e. an order of magnitude smaller than the amount a person can detect. The aggregate impact of this is that it saves roughly a decade of user-time every single day. ↩︎
You can't stop people from hating your company even for fairly ludicrous reasons, but you can arm the moderates with better data and nicer-looking diagrams. Silencing negative conspiracies is impossible, but it is at least possible to make their supporters look relatively more like stereotypical conspiracy theorists. ↩︎
A Word From Our Sponsors
Meet Fin. Intercom’s breakthrough AI bot for your support team. Fin can resolve half of your support tickets instantly, solving complex problems and providing safer, more accurate answers than any other AI bot. Simply pair Fin with your help center to learn your written history and hold natural conversations with your customers. Visit the website to learn more.
Elsewhere
Art Exchange
A company called Artex is trying to create an exchange for equity in paintings, starting with a $55m offering for shares of a portrait by Francis Bacon ($, WSJ). Art is a strange asset class. In one sense, it's an inflation hedge, since the supply is finite. On the other hand, it's a sort of supercharged bet on market beta, since historical peaks of art prices have coincided not just with market peaks but with cases where newly-rich people made their money very differently from whoever came before them, and want to demonstrate that they're still part of the establishment. (This works best for art prices when the establishment, too, can afford to make a bid.) J.P. Morgan collected art and books in part because he didn't have an aristocratic lineage to fall back on, and was able to do so because the people with the pedigree and the art collections were being left behind economically by increasingly wealthy Americans. Something similar happened to Americans, briefly, in the late 1980s; the record amount paid for a work of art tripled in 1987 when a Japanese insurance company bought Van Gogh's Vase with Fifteen Sunflowers for ~$40m. Art investments are hard to underwrite because it's unclear what will stay relevant—Sunflowers, for example, may have been painted by Emile Schuffenecker instead. And art has a carrying cost; the paintings have to be preserved, and shares aren't worth much if the painting gets stolen. (Unless, of course, it's 2021 and your shares are nonfungible tokens.)
Fraud
A few weeks after job training company Bitwise shut down amid accusations of fraud, events company IRL has wound down, saying that 19m of its 20m users were fake ($, The Information). It's extremely hard to bootstrap a user-generated content company without some amount of inorganic posting, whether that's from the founders using sockpuppet accounts (the Reddit strategy), actual bots (a common dating site scam), or even gray areas like paying users to post (Yelp was able to cultivate more content than users organically wanted to produce through its Yelp Elite parties—so the users were real, but they had a cost to acquire that would have to drop for the model to work long-term). These are all on the same continuum, and while all of them are in some sense unsustainable, they still make it possible for a site to get off the ground and reach a point where users do the work without being prodded by gifts or fake users. So the real gap between fraudulent companies and social media companies with an effective growth model is whether you can plot fake engagement against organic engagement and find a line of best fit that points to the fake engagement eventually going to zero.
Airlines
Boeing and Airbus have received record-breaking orders and can't make planes nearly fast enough to satisfy demand ($, WSJ), which has extended order backlogs into the 2030s. The pandemic was a massive demand shock, but also a bit of a reset that led some airlines to accelerate the retirement of planes. Naturally, once things returned to semi-normal, this caused a squeeze. It's generally harder to model supply than demand, because some of the factors behind supply seem unstoppable and others are intrinsically hard to predict. The supply side can be trickier, but it gets easier to assess when demand is backlogs are long. The market is more or less on board with the view that this demand swing will reverse: Boeing and Airbus shares naturally tanked right after the pandemic started, but, unlike many other companies, they're still trading below where they were in early 2020.
The Long Tail
Modern Retail has a quick profile of influencer marketing startup Kale, which tries to identify social media users who are already posting about brands and reward them for this. There's a porous boundary between organic marketing from unprompted customer endorsements and paid marketing that's part of a formal campaign. And one thing that determines the boundary is the transaction costs, both for identifying potential product endorsers and for encouraging them to share what they like about the product. In a sense, this is a productized version of the Etsy seller practice of giving customers a handwritten note asking them for a review. But as happens in many other cases, a linear decline in the cost to reach people can lead to an exponential increase in the size of the potential audience.
Chinese IPOs
Swiss chemicals company Syngenta was acquired by ChemChina in 2017, and is now being taken public in Shanghai, but American banks increasingly doubt that they'll be able to participate in the offering at all ($, FT). China's government has been able to encourage a large equity market that also functions as a tool to direct capital towards state goals—by promising large subsidies to strategic industries but not picking the winners all at once, for example, they can use equity markets as a proxy for which firms in their chosen category are likely to be the eventual winners. One ingredient of that is having a domestic investment banking industry that can handle big underwritings. In this case, the company was public before, so already evidence of investor demand, meaning it's a good opportunity for the government to give domestic underwriters experience with larger deals.
Diff Jobs
Companies in the Diff network are actively looking for talent. A sampling of current open roles:
- A proprietary trading firm is seeking systematic-oriented traders with ML experience—ideally someone who has displayed excellence in DS and ML, like a Kaggle Master. (Montreal)
- A company building tools to enable zero-knowledge proofs is looking for multiple roles, including a full stack engineer. (Remote)
- A company building ML-powered tools to accelerate developer productivity is looking for a mathematician. (Washington DC area)
- A startup building a new financial market within a multi-trillion dollar asset class is looking for a senior ML engineer, especially someone interested in using LLMs to make unstructured data more tractable. (US, remote.)
- A hedge fund is looking for an experienced alternative data analyst who can help incorporate novel datasets into systematic strategies (NYC).
Even if you don't see an exact match for your skills and interests right now, we're happy to talk early so we can let you know if a good opportunity comes up.
If you’re at a company that's looking for talent, we should talk! Diff Jobs works with companies across fintech, hard tech, consumer software, enterprise software, and other areas—any company where finding unusually effective people is a top priority.