I wake up as usual by immediately going on my phone. I reply to some of my messages - yes dad, the solar installation you have at the villa could be redundant soon, but again, it can still benefit them (even if slightly) if you comply and let them integrate it into the new grid. A grid which operates on principles only slightly less opaque than divine providence, but is demonstrably more efficient. I also sent him the relevant passage from The Spec. Energy generation is covered thoroughly.

I go to the bathroom and use the auto-brush. It performs flawlessly, eliminating the user failure modes I recall from manual brushing, though the brief floss-like motion under my bridge stings a little. I visit the toilet - still stubbornly un-smart, lacking even basic output analysis. A shocking oversight in an otherwise quantified existence. Someone should file a feature request (though, yes, it’s probably because for whatever reason, bathroom privacy is hardcoded into the spec). My own internal feature request - a morning drink - prompts a move towards the kitchen. Not everyone has bought into it, but I’ve always been into nootropics so I am eager for my dose of ‘Coffee+’, OA’s bespoke blend optimized for cognitive profiles like mine. I do have to continuously share so much of my data about my vitals with OA to ensure it’s refined based on its effects on me, but honestly, it’s worth it. I am not important. My significance is negligible in the face of OA’s aggregate data. The very concept of ‘important people,’ in the pre-OA sense, feels increasingly archaic. So I might as well “sacrifice” my data or, rather, lease it out for substantial cognitive dividends..

I start feeling the effects, and while the blend doesn’t confer NZT-48 levels of hyper-cognition, its effects are nonetheless remarkable. When it kicks in, the morning mental fog dissipates. It doesn’t grant superpowers, just.. executive function. The ability to align my actions with my stated goals, a previously depressingly non-trivial task. I want to and probably could understand all the tiny nudges to my neurotransmitters, connectivity and everything else the substances affect, but really I don’t need to know how it works. Maybe I’ll allocate time to unpack it one day. A project for future self-knowledge acquisition, perhaps.

I go to my desk - though I should give in and do the VR/AR setup that’s catching on. I don’t know if I’m ready for corneal overlays. I used to be squeamish about them, worried about firmware updates bricking my eyeballs. I’ll probably try it eventually, once the fail-safes have fail-safes. My monitors light up as I approach, and I suppress a little smile as I see it’s opening the analytics of the animation I posted last week. Yeah, you can get unlimited generated for you slop (okay, it’s not slop; I’ve mostly come to terms with how good it really is), but a lot of us also watch channels directed by humans. I wish I had a big following back then because legacy creators get so much extra attention compared to relative newcomers. Still, the content I produce gets traction and directing OA’s creative suite is almost as fun as watching the final product it generates.

I checked my chats again. Perry has been obsessed with going to the Moon - it does seem like it’ll happen eventually, but I don’t think it’s going to be any time soon. I guess, technically, we can request anything feasible, but surely, going to the Moon is so resource-extensive that those resources aren’t getting allocated to it any time soon. I imagine OA’s resource allocation algorithm flagged it with a gentle ‘User budget exceeded. Suggest a simulated alternative or perhaps a nice cup of tea?’ I have OA show me a simulation of what Perry will see on such an excursion. I close it a few seconds later - it’s about what I expect. I go back to the kitchen and wait patiently for my food. The robotic arm appliance prepares it while I do my first set of light exercises. My form is probably suboptimal, but the robot arms, thankfully, are programmed for non-judgmental food preparation only. I get my food and lean on the old “I need to watch something while I eat” excuse to put on an episode. The quality ceiling for entertainment has been raised so high, it’s practically at the level of where Perry wants to go.

The episode, naturally, was algorithmically tailored. Not just to my inferred preferences based on viewing history - that was primitive stuff - but dynamically adjusted during viewing. Just as my focus might have drifted, a subtle shift in pacing or a character’s micro-expression, presumably cued by my biometric feed via the desk sensors, pulled me back. OA knew precisely when my attention dipped, when my pulse quickened, and when pupil dilation indicated intrigue. The narrative arc might have been pre-written, but the details were still optimized in real time for me.

Yet, watching it still felt almost like cheating. A process smoothed out. I found myself wondering about the cost function being minimized. Was it my enjoyment? If it was just that I’d likely end up wireheading, so it takes “well-being” and other factors into account. I am sure it’s also optimizing for other goals - making me fit a little better in the new paradigm. It might seem almost sinister, but it isn’t really. The system delivered. My life is demonstrably better, safer, and more comfortable than any pre-OA baseline. My dad, bless his villa-solar-panel-installing heart, still grappled with the idea that centralized, AI-managed grids outperformed decentralized individual effort, even with the best intentions. It was a category error he hadn’t quite shaken yet, clinging to an outdated model of agency like a physicist still insisting on phlogiston.. He’ll adapt. Eventually. Probably. Perry’s Moon obsession was another facet of the same pattern. Why allocate vast, tangible resources - materials, energy, risk-exposure for biological units - to achieve an experience OA could simulate with near-perfect fidelity, piped directly into your sensorium? The desire felt… archaic. Like wanting to hand-crank your car instead of letting the autonomous network manage traffic flow for optimal efficiency and safety. It wasn’t about feasibility - OA could calculate the resource cost down to the last gram - it was about optimality. Sending Perry to the Moon was likely a net-negative on the global utility function OA was calibrated to maximize (or whatever inscrutable N-dimensional optimization landscape it navigates), wasn’t it? His simulated trip, costing negligible computation cycles, was the rational choice. I guess we do have the resources to spare - not that you can trust a mere human’s resource allocation guesses. It makes me wonder whether we haven’t gone too far in emphasising that people’s desires should always be eventually met in The Spec. I end up watching and musing until noon. I get briefly excited about creating my own video about a similar type of time loop as the one in the show before I stop myself - No point recreating what OA produced for me. I push back from the desk. What should I do? The question itself feels slightly anachronistic. It’s not like I have a job. My schedule suggests another dive into the workings of OA, this time with a focus on error correction. It is interesting, and it is packaged so well, but my motivation to dig deeper and deeper into how it all comes together is waning. Who am I kidding? Perhaps I’ll leave it for now and instead I’ll finally explore the new advances in Cosmology - we almost have the blueprints for getting a ToE even - there’s so many experiments left to do of course, and I have only barely looked into it. Yeah, that should be fun to explore. I ask OA to prepare a learning plan while I digest the best part. The carefully crafted overview which hits all my intellectual porn desires. Sure, learning the rest will be enjoyable as well, but those first insights, delivered so delicately in just the right way. Intellectual bliss, mainlined..

I can’t indulge too much as it’s time for me to meet Sam and Lina for lunch. I brush up on the latest updates on the election before I go, as I know Lina will want to talk about it one way or another.

I step outside. The air is so clean today, but it is chilly. Transport is, of course, seamless. The car that picks me up is modern but looks to be repurposed for OA driving and not originally designed with it in mind.

Sam and Lina are already at the cafe, a place that retains a surprising amount of pre-OA charm, albeit with automated staff and nutrient-optimised menu options overseen by the central system. Sam waves, already sipping something that looks suspiciously like standard water - he’s always been a bit of a purist, finding the edge cases where OA’s optimisations fall short for his specific, self-diagnosed sensitivities. Lina, however, is animated, gesturing with her tablet.

“–but the principle of it, Sam!” she’s saying as I approach. “Even if the predicted outcome variance is within the noise floor, the act of collective choice matters!”

“Hey,” I say, sliding into the seat. “Pestering him about the election already?”

Lina turns to me, her expression earnest. “He doesn’t even plan to vote. It’s the only lever left for influencing the world, and he doesn’t even care.”

“I do care. It’s just that the impact is so negligible,” Sam interjects mildly. “Some might even say there is approximately zero impact”

“But how can you know?” Lina insists, leaning forward. Her tablet displays polling data - colourful charts showing predicted sentiment shifts smaller than the margin of error. “The Spec allows for amendment based on sustained, overwhelming public consensus. Section 7, Subsection B. The elections are the designated mechanism for measuring that consensus.”

“Ah, Subsection B,” Sam sighs, taking a slow sip of his water. “The one requiring a ninety-five percent consensus sustained over two consecutive electoral cycles, and demonstrating non-contradiction with core utility axioms outlined in Sections 1 through 4. Possible in principle, maybe, but irrelevant in practice.”

I jump in, trying to bridge the gap. “Lina has a point about the principle, Sam. The existence of the mechanism, even if it is barely reachable, serves a function. If nothing else, it maintains the illusion of human sovereignty.”

Lina nods vigorously. “Exactly! We all look at the election as the place where change could happen, even if it doesn’t. If we stop participating, we signal that we accept the current Spec as immutable, forever.”

“But the point,” Sam counters, his voice calm, “isn’t whether the mechanism exists. It’s whether activating it is within the realm of practical possibility under current conditions. We’re arguing about the theoretical existence of a lever encased in meters of hardened plasteel, requiring a coordinated effort roughly equivalent to getting humanity to agree on the best pizza topping.”

The politics brief is still open on my phone, and it reminds me to retort, “It’s not like we haven’t changed The Spec before, Sam - The 2032 Clarification..”

Sam waved a dismissive hand, nearly sloshing his water. “‘The Clarification’? Please. That wasn’t us changing The Spec in any meaningful way. That was OA flagging an internal inconsistency and us rewording some definitions. No fundamental shift.”

I mentally concede his point. We can, in theory, change The Spec, but in practice, we probably won’t. Not really. We might look at The Clarification as an example of our ability to affect things but really, it just entrenched the whole thing.

I eventually steer the topic away from politics and towards the shows we all watch - we’ve agreed on which creators to follow so we can participate in discussions about their content. We cling to that one additional reason to consume non-personalized shows.

I go back home, and I find myself pulling up the official election portal on my monitor. Candidates dutifully uploaded policy papers - dense documents exploring minute variations to Spec-related parameters. It does feel more like the work of someone keeping themselves busy than of someone who is actually busy.

There were, of course, the fringe candidates. The “Spec Rejectionists” who advocated for a full rollback (computationally infeasible and axiomatically disallowed by OA). The “Human Supremacists,” yearning for a return to pre-OA chaos (polling at statistically zero). Their platforms were automatically flagged by OA’s content filters as “non-viable” or “incoherent,” relegated to obscure corners of the data sphere, their arguments accessible but never promoted. Participating felt like meticulously choosing between slightly different shades of beige wallpaper, while OA architected the entire building according to optimization principles we barely grasped.

Lina would argue that even choosing beige was an act. Sam would say it’s wasted cognitive effort. I.. I wasn’t sure. Was apathy the rational response? Or was it a failure, giving up on the last vestige of the old world’s structure? Ultimately, I don’t think it matters - what we have is good. Why worry about changing it? “Not every change is an improvement but every improvement is a change.” Sure, but what change can we hope to make that would realistically be an improvement?

0th Factor: Luck - The Unavoidable One

Luck is a significant factor that affects our lives, whether we realize it or not. You can be plenty hardworking but get dealt a bad hand or incapable of achieving much but lucky to not have to. Even how smart, or conscientious you are is ultimately determined by a lottery but having a model where we just declare everything to be luck (even if true) isn’t exactly useful

1st Factor: General Intelligence - Your Raw Intellectual Power

General intelligence, often correlated with but not quite IQ, is a significant factor in achieving success. IQ itself correlates with nearly anything that matters and while it’s not the only determinant of success, intelligence undoubtedly plays a critical role in problem-solving, decision-making, making connections, learning new skills and more. A higher IQ can give individuals a competitive edge in their careers and personal lives.

Many tasks can indeed be achieved as long as you are over a certain baseline of intelligence (sometimes low, sometimes higher) but at minimum higher intelligence allows you to achieve some subtasks faster and arrive at breakthroughs sooner. It is also important to note that as long as you are over said baseline (which differs for different tasks), you are not necessarily limited by, say, being a bit slower. Hell, if we consider a simplified case of just speed then someone who works 50% faster than you but for only an hour a day will still achieve less overall than you if you are working 2 hours. This brings us to:

2nd Factor: Conscientiousness - Doing the work

Conscientiousness, or the ability to work hard and put in the necessary hours, is absolutely key. I’ve seen plenty of smart people (or at least with high IQ scores) achieve little because they don’t put in the time to do so. In fact, I consider this factor the one I am ‘worst’ at and am always impressed by people who consistently put in the work day after day.

Unlike Luck or General Intelligence which are out of your control and generally fixed respectively, conscientiousness or at least the parts of it that matter is something that can be improved. Building habits, making lists, working on any personal blockers or taking ADHD medications because you previously had undiagnosed ADHD (surprisingly common currently) are some of the ways that at least some people can improve. If you have any bigger ‘intellectual’ goals but find that you are not really working on them, then this is likely the factor you should be focusing on.

3rd Factor: Rationality - Making Better Decisions

Rationality here is defined as the ability to use intelligence effectively and make well-informed decisions. It matters both for small decisions and even more so for big ones. What does it matter if you are extremely smart and put in many hours of work when the thing you work at is, say, Theology (and many sharp people are working on that) or Ads (except as a precursor to making more money). Choosing effective things to work on is extremely important. Say your goal is to help people, a smart hard-working person can join a local charity and do a great thing for them, a more rational person might start by looking at which charities help the most people first, then how they can help that charity the most and go from there (yes, this is basically Effective Altruism).

Further, making more cost-effective decisions along the way is also important. Perhaps reading a tangentially related post to what you are doing just seems useful but isn’t as useful as working on what you should be, or maybe videos are surprisingly effective in the niche you are working on and it will be much higher value than the way you promote your cause now. Being able to succeed at tasks, and spending time succeeding at tasks is very good, but it’s much better if you pick effective tasks for your goal.

This factor is also at least partially learnable although it’s hard to say to what extent. The approach I prefer, as stale as the advice might seem is to read the sequences.

Conclusion

Success in life is a complex and multifaceted concept, and achieving it requires a combination of factors. The impact of those factors is multiplicative, and you can for the most part compensate in one place for deficiencies in another. In particular, figuring out how to put more hours into achieving your goals (yes, I know, not very exciting or novel advice), as well as figuring out better which tasks (and even goals) would be most effective are the parts you likely have the most control over.

In short, my analysis suggests that d4 is actually somewhat better for your improvement (confidence: high), and so are longer games (confidence: medium).

Data

I download the January 2022 (102,110,423 games) and January 2023 (103,178,407 games) PGNs from Lichess’ database. I went over all rapid, and blitz games saving them separately and recorded whether White played e4 or d4 and their rating at the time. I threw everything else - Black, different time controls, non-rated games, anything after the first move etc.

I filtered out anyone above 1800 rating since the majority who are that high aren’t beginners - according to Lichess only 18% of weekly rapid players have a rating > 1800 and 21% for blitz.

Further, I removed anyone that wasn’t present in both 2022 and 2023, and anyone who played both e4 and d4. While this reduces the number of remaining players, if we assume that e4 or d4 is better we dont want to have many players marked as say d4 but improving just as fast because they also play e4. The above left me with 125447 e4 rapid players and 22274 d4 rapid players, and 950477 blitz e4 players to 477755 blitz d4 players. It also made the ratio of e4 to d4 players worse than beforehand since there were more ‘pure’ e4 players than pure ‘d4’ players.

That leaves us with a good amount of players, as we are comparing the same players at both times and we should expect to detect an effect if there is one.

Analysis

I personally prefer rapid, so I looked at those games first. After filtering the data and extracting the rating differences between 2022 and 2023, we get a mean increase of 78.6 points for e4 over a year of play vs 85.5 for d4. This looks like a small but noticeable difference and A quick ttest suggests pvalue=0.001, so the results are significant.

Looking at the graph it appears that most of the difference is in the 1350 and below range. I’m not completely sure why there is such a jump in rating improvement there but the starting rating on Lichess is 1500, and I believe 1 lost game on a new account brings you to the 1350-1400 group so that’s where the newest player are and where ratings are most volatile.

Looking at the blitz data, the results do replicate, however, the improvements are much smaller. The mean improvement for E4 is 5.2 points and 8.1 points for blitz (pvalue=0.009). It’s important to note that the rapid and blitz ratings aren’t directly comparable and the cutoff at 1800 does not mean the same thing for blitz as it does for rapid. The 50th percentile for rapid is 1550 on Lichess and 1430 for blitz, with rapid ratings generally being much higher for the same player. Nonetheless, the results broadly replicate and look like this

Given how hard it is to compare rapid and blitz ratings, I have less to say about the relative improvement in the two time controls but the average improvement in <1800 rapid is ~80 points and only 5-8 points for blitz, which is a rather large difference. If I cut off the data to only <1500 the blitz improvement jumps to 60-68 points which is still lower, and even limiting it to below 1300 leads to lower improvements in blitz (78 points for e4 and 66 for d4) than for <1800 rapid (and that number only grows if I also limit rapid to a lower rating). It is admittedly not conclusive as the two distributions are different, and I might explore the question further by comparing blitz players’ rapid improvements and vice versa but what I am seeing here certainly suggests that your improvement might be slower with blitz.

Limitations

The biggest limitation is that we are simply comparing the improvement of d4 and e4 players a year later but can’t know if the improvement is due to their choice of opening or because the groups of e4 and d4 players are different themselves - our study is simply correlattional.

It’s also important to note, that some in the ‘play longer games’ camp recommend going for games even longer than rapid but if longer games lead to more improvement I’d expect to see a difference in rapid vs blitz already.

Conclusion

Longer games or at least rapid seem plausibly better for your improvement over blitz games (at least at lower ratings). d4 in particular seems to lead to a faster rating rise than e4 BUT the difference is fairly small, so if a player prefers a particular opening and would play more games using it then it’s unlikely to be worth switching. Personally, I will attempt to switch away from my default - the Scotch - if only because it would be somewhat silly to have ran this analysis and not changed anything because of it. If anyone wants to try to replicate my findings the quick code I wrote is here, however, you’ll need to download the PGNs from Lichess as a month’s worth of games is ~200gb uncompressed and hard to share. Let me know if you get different results or spot any mistakes.

How often have I travelled? 89 times. Not so many in the grand scheme of things, not so few that it hasn’t changed me either. The machine should be good for at least a few thousand more and the mission is far from completed. That number is bound to increase significantly.

When analysing a coordinate the machine only sends a handful of particles and records their response, yet even that act changes the world irrevocably. At least if the analysis is of the past. So I don’t use it backwards as I can’t even know what has changed but I have it on good authority that the person that starts an analysis into the far past is extraordinarily unlikely to be the one to receive the results. Frankly, I’d consider it bad UI to even give me the option but they didn’t want to limit me. I don’t think it matters that I would most likely erase myself (and everyone born after the machine analyses) out of existence if I do it that way though. I wouldn’t have passed the psych evaluations if I was the kind of person who did. What does scare me is that due to random chance the resulting timeline after a travel or analysis will be ruinous and without anyone to change it. It’s a small risk but an utterly terrifying one.

Do I miss my original timeline? Not really - we were on the brink of apocalypse and everyone on the team knew they’d be erased as soon as I travelled back. Having some form of humanity is more important after all. We thought the fallout we got was inevitable. It seems we were wrong - I’ve only encountered a similar biochemical event one other time in my travels and even then it wasn’t quite as bad as what we faced. I prevented our whole issue on my first try! It was damn trivial! You’d think my job would be done once that was over but no, turns out that humanity doesn’t fare so well even without that hard Apocalypse. Societal and technological collapses are frequent. As far as I can tell this last timeline had those responsible for the worse of it lauded as heroes. Again. “They made things better”. They didn’t. Sometimes I wish I could be tempted to travel back a million years and do a hard reset on humanity. It would be irresponsible. I wouldn’t fit with the new humanity. I won’t even speak the languages. And they’ll likely only get one shot at it. I don’t venture before the 20th century - the further back I go the more changes there are and the less useful my knowledge is.

Even now I regret going back to 1910. Things are too different already - all the important people and events that happened later just don’t exist here. I believe I slowed down computer science by erasing Turing. Of course, It might’ve been due to someone or something else. That’s partially fixed but if I go to the early 20th century I’ll erase the current alternative technological path, too.

The other direction is no better. I don’t go too far forward because going to a point when a time machine is invented can be disastrous. Not because of any interactions, it’s all just normal matter. I can do whatever depraved acts I want with myself or my predecessors. Wouldn’t mean a thing. The real potential problem with the far future is that there might be others changing the timeline and that can impact my own interventions. I think.

At least there are less issues with letting the machine give me its basic analysis of the point of arrival in a given future. That’s what I am currently doing - firing off scans of the random points in this timeline’s near future and so far I like what they suggest. I can’t tell all that much more than the type of matter at the given locations, but I can find signs of humans at a reasonable frequency in major cities, normal levels of radiation and generally everything pointing towards there being no crisis any time soon. I’m currently in the year 2000 - a year I often use to gauge things as it’s around the time when my world was still doing okay and a nice round number. The first thing I noticed is how humid it is. I expect to remember that as a key feature of this timeline despite it likely simply being a feature of the particular day I traveled to. It’s just how it is.

They are pretty advanced here already. Whether it is due to my last intervention in 1959 or due to random chance I can only guess until I read up. A typical personal computer has 8-12 cores and over 50(!) gigabytes of their equivalent of RAM. A typical working week is 4 days, and only 1% (or 3.5% according to the secondary political party’s metrics) are in poverty.The average life expectancy is at 80.2 years. For some conflicting reasons wearing yellow is strongly discouraged (though not actually illegal in most jurisdictions) but hey, you get used to random things like that. I assume, say, the prevalence of denim - a textile which I haven’t seen in a while - at this point in my timeline would be equally odd to them.

This world does not meet all my requirements - the worldwide incarceration rate is at over 1% and capital punishment is very well received but I’ve learned not to judge too much even when I don’t quite agree. I’ve accepted that the better world wouldn’t look quite like I’ve imagined. At any rate, the more data I gather the more prepared I get to laxen my Utopia requirements. Not yet, of course - there is more to try first. I do need to weigh the chances that I will perish in a bad timeline against the chances of successfully finding a good future.

I will travel to this world’s future to see it, maybe go back and forth a few times to reset or look for points where I can improve it further. There’s hope. I won’t go immediately of course. The machine takes a toll on my body - not as much as something like a space flight but enough, so it is wise to take breaks in between trips whenever possible and to keep in shape. I’m prepared to spend as many years as required on my mission after all - better stay sharp. Pawn shop first, then finding a place to stay and reading up on the effects of my last intervention. The usual.

Samuel Brosnan was currently the Chief Technology Advisor of the USA.

This future is unacceptable. Dystopia of a different colour. People get chipped as soon as they commit a crime and almost everyone has one now. You can’t tow the party line and despite the benefits of a unified world government this is no Utopia-adjacent world. I’ll need to travel before I’m found out - a back to back trip. Unpleasant. The mid 90s perhaps before going back to the 21st century and checking it again.

Samuel Brosnan died 11 years ago.

I’ve been traveling back and forth within this timeline, never earlier than 1989, never later than 2028. There’s much to learn here and occasionally I feel like I’m on the cusp of grasping it. Usually the timeline is ahead technologically, other times socially and sometimes both but there are always problems. I’ll need another try - probably a few - to start really figuring it out.

Sam Brosnan was President. Magnate. Man of the Year. The cliche hero engineer who devoted himself to science. On benefits after being assaulted. 3rd richest man. Leader of the IN reforms. Dead.

Bad luck. It was bound to happen. I traveled to 2021, but the government was after me so I immediately traveled back to 2003 in fear of getting found out again. The stress on my body was already too much and the rowdy drunks who attacked me caused some damage. My knee might not recover fully and this area is already advanced enough that seeking real medical help under the radar is not so easy. I yet again wish I hadn’t had to erase that one identity I prepared in the 80s all those timelines ago. I might need to wait to recover enough for a trip further back where seeking help is trivial. I hope I don’t have to. In the meantime it’s time to examine my options once again.

One option is to travel back and release blueprints and papers to increase the rate of technology advancement. This has not worked as well as I hoped before but I can try different variations. There are many possible variations of this. Many possible variations of everything. Too many, as it always seems to be the case.

Another one would be to micromanage by boosting certain people and advances and keeping others down. It can work but it is finicky. There’s more risk for myself the more involved I get. Still, I can try to identify more promising points for intervention. Sometimes it doesn’t work like my early misguided attempts with Turing - trying to help him avoid punishment for his sexuality helped no one. I do miss him. Sometimes it seems to help like with that recent Bell Labs equivalent.

Yet another alternative I’ve perhaps dismissed too hastily and which starts to look more tempting is to find smart people, get them on my side, tell them everything and work together.

Sam Brosnan was the prolific ‘investor poet’.

I had to spend over a month to heal enough for the series of jumps I needed but by the time I arrived in 1999 I had most of the information on how to quickly set up and get some of the best treatment available. I know a lot more now. Further, I know a lot more about now, who to contact and how. What’s left is to wait things out the old-fashioned way.

“topic: 3446124419567 ∑##$ zzrtrt ; no harm

Salute Brosnan,

You must be currently investing in $WWZ or writing your essay on bias in technology forecasting. I wouldn’t want to make you abandon either but we need to talk. I believe you secretly think you are on your way to becoming one of the most important people alive today. You are correct but let’s not set our standards too low. You are about to become much more than that.

344626 tonight. Message me when you confirm.

I’d tell you that you aren’t system infiltrated and my information comes from elsewhere but I understand you aren’t the kind of person to ignore the chance you are.”

This should do it. If it doesn’t - there’s always next time.

It didn’t. It took 17(!) more messages to get there but we are finally meeting. I’ve already paid off people in law enforcement - specifically those who were later caught for being less than exemplary, set up a driver nearby, emergency fund and a few other countermeasures in case Samuel ends up screwing me over. It’s time. Then again, it always is.

I am now facing the unexpected. My name is Samuel Brosnan, the world needs to be improved, and the tall, well-toned and impeccably dressed man in front of me claims to have the means. The same man who knew portions of my secret passwords - kindly rearranged either in case someone saw the messages, or because that’s all he could get, and other details I would have thought near impossible to know or guess.

I have checked all my records, and those of the people around me and there are no obvious security breaches, no missing hours, nothing. Computer system infiltration is always possible of course, but in his very first letter he assumed I was working on an essay which I had only started composing in my head and planning on writing all at once that night. I did not - my time went into planning how to approach this. I followed with more messages asking for other types of verification, ‘idle’ discussion which he must have realized was just hunting for hints. I got the most by hinting that I am abandoning the essay. He knew too much. Of course, my transition into writing about problems in technology forecasting is perhaps somehow predictable even if the timing is so perfect. His use of the exact terms I planned to borrow from other sciences when hinting at how important he thinks it is what put me on edge. Then again, so did a lot of the other things he said and promised but that’s what really got me. All of it made perfect sense, especially the parts about the best directions in research and development. This is no crank or at minimum they are extremely knowledgeable. Knowledgeable enough to have little problem arranging a meeting with me through more normal means. Knowledgeable enough that I should have heard of him.

I’ve mostly ruled out the more plausible theories. The best ‘mundane’ theory is a combination of system infiltration + lucky guesses + extremely smart people working in tandem. The chance of getting so much right is far too unlikely but perhaps if they did this to many others, lucked out only with me and had other sources of information.. I queried some influential people but none of them admitted to having received messages like this - “I wish the cranks would for once send me anything remotely plausible or interesting” is what Revan, the man I get most often compared to, replied.

The other theories which I have to consider at this point are much less likely a priori and are mostly different types of advanced technology. As unlikely as it is, if someone could do a full brain scan of me and select other people, parse that information and use it then they could do something like this. Same idea with a novel drug to make me tell them everything they need, then forget and somehow cover their tracks but there were too many logs they’d need to fake. If they had things like that, I do not know why they would even need me. Although, to be fair, I have little idea why they need me anyway. But I do want to find out.

I decide to open with a joke by leaning on one of my less likely theories. Lighten things up from the start, hopefully take control of the conversation.

“Salutations, John. It is clear you are a time traveler,” I say, smirking. It is important to look like you are in control even when you are not.

“I am, and you seem to be as smart or even smarter than you seem If you’ve already figured it out and accepted it. I assume then you already have a good idea of what we might be doing here,” he shoots back.

I most certainly did not and despite my competing urges to either drop my jaw or laugh, I stop to think. My first thought was on his accent. It was slightly off which was a small point in favour of him being a time traveler. As they say - the parallel is a different country. If he had brought it up, I would think even less of it as he is certainly capable of preparing that much but he did not. On the other side, it was now clear he had subtly tried to lead me towards that conclusion. ‘Today’ was emphasized in his first email. He indirectly talked about the future. He acted like he was both unknown and powerful. Regardless, I fall back on old habits and say the obvious - “Proving it.”

I listened to his explanation, asked my questions and told the presumed time-traveler to meet again in a day. It all mostly made sense but there were parts I could attempt to verify. His involvement in the 1949 shift for pre-registration of research and the related changes of typically used statistical methods. The dark matter anomalies or what would be anomalies under our current models. The theoretical validity of the various technology paths he claims we’ll be taking. Jana and Mellisa. I assume he realized I recorded it all.

It checks out as far as I can tell. It’s extremely unlikely but there are not many better explanations and I am not one to reject the unlikely if it fits. It’s not like I will stop looking for alternative explanations. I approach him.

He is smiling. His outfit is reminiscent of a security guard’s today. “Here’s a copy of your watch, that pretty logo from your lobby and Whales’ ‘Man versus God’. Lucky that you are fond of such a tiny painting. I can’t bring all that much with me,” he says.

I take my time verifying them, while noting the implication that he can only bring a little with him at a time. That’s fine and to be expected.

“Would my lab find anything unusual about those duplicates?” I ask idly to confirm my working model of his time travel. They should be completely normal matter. Really, I should be amazed about the implications of unlimited duplicates with minimal limitations but I am still preoccupied with being amazed with Time Travel in general.

“Not unless you analyze them with an IBM 5100,” he replies but quickly adds “Sorry, a joke from another time. The reference means nothing now.”

It’s dangerous to make assumptions on a throwaway comment but I file this away. Something to investigate further. “And you can duplicate anything as long as it is small?”

“In theory, yes,” smiled the time-traveler. “In practice it is a bit more complicated with the biggest limitation being that the machine can’t travel to points in space-time that are too close together, at least not until the prior travel has been overwritten by one that is sufficiently earlier.”

I wonder if he is bluffing. This should put me at ease as it implies that he cannot easily retry his conversations with me over and over but he could also be lying.

I kept thinking how ripe for abuse the duplication abilities of the machine were - take something valuable, send it to the past, rinse and repeat and you have many copies of it. Apparently, you can’t travel to points within a year or so of each other but you can still work around it if you have a longer time frame.Cloning powerful chips or parts can come in handy but in the end it seems more likely to be a ‘step two’ in reaching what John calls ‘Utopia’. First, we need society to be mostly in order for enough years and then we can think about duplicating whatever we want to move to a post-scarcity scenario.

“I think you are approaching this all wrong. There is a better way,” I tell John directly while projecting that I mean it.

It is absolutely incredible that Time Machines were invented and everything didn’t dissolve into utter chaos immediately. Having the only person who has access to one be someone attempting to prevent the worst outcomes is a godsend. It’s still dangerous of course - any one man can and will err. Everything riding on their judgement and luck. I can only imagine how bad it must have been for those smart enough to build it to take such a risk. It is horrifying in a lot of ways but nowhere near as bad as the alternatives. If anything it’s surprising how fortunate we are.

“You already have a solution? I am skeptical,” John says but I could see the hints of hope and excitement on his square face.

“Yes or at least a much better approach. You have a blindspot.” The skepticism shows now and too much hinges on the next part. He’s not going to like it but I continue nonetheless “Of course, I have a few conditions first.” I tell him

“Go on,” he replies.

“I don’t get completely erased,” I say. “We leave the timeline before 1976 stable.”

“It is hard for me to relate anymore. I have accepted that my life is forfeit aside from this but I can understand how it must be different for you. I will warn you though - even the tiniest chance of a success is worth uncountably more than our lives are. I will have to think about it,” he says with a look betraying sorrow.

“Why? This is not just self-preservation on my part.” - to a large extent it was - “You already told me the current branches hold promise. You’ve apparently seen multiple versions of myself committed to improving things. You told me I can be the ‘chosen one’, which admittedly almost made me walk away. This is not a boast but I am clearly an asset to the world. You think it might be selfish? I am willing to accept that my current self will stop existing, a sacrifice you yourself have never taken,” I reply. The last part might seem unfair but it frames the conversation in a manner I can use.

He regarded me with a dark look before saying “To put it bluntly, I am not banking everything on this despite how much time I’ve invested. If it doesn’t work I need to keep my options open. Promises are not something I break either - it was an explicit part of both my selection and training in the hopes that if there are other time travelers they’d be trained similarly and the chance of us working together would be higher. So I won’t make a promise I might not keep and I don’t want to limit my options without a good reason. I still want to hear you out so I implore you to put yourself in my shoes and think how little of a sacrifice one’s existence is.”

To put myself in his shoes? I idly wonder if that saying is from his timeline. This is a very favourable response despite him not agreeing yet, and if he means what he says about promises - which makes sense on the face of it.

I decide to reveal more “I understand but you’ve gotten a few things wrong, my ally. First, my solution will leave no room for other trips. It will be final. Second, you profess selflessness but you are the blindspot. You think you need to be the one making trips which is not the case,” I tell him.

He stands up and looks like he might be considering abandoning this. I try not to think of the ever present threat of having this conversation and his interaction with me erased altogether. Despite a twinge of guilt, which I’ve long ago learned to suppress, I continue “No, you are still not getting it. Nobody has to. You are wasting enormous amounts of your machine’s limited capacity to lug yourself around. Even if you weren’t, you can do what? A few thousand trips in your lifetime? At best? We can do a lot better.”

Two weeks had passed since I started convincing him and it was working. He was partly worn down and I leaned on it to exaggerate his mental tiredness of his situation. I emphasized how he wont have to travel and he will finally be a part of a world again. The world. Minimized the risks perhaps, maximized his role so it doesn’t seem quite so out of his hands. I used every other trick I had. Mainly though, I just relied on the plan.

Changes had to be made. The main part didn’t. The machine can scan a small amount of particles and bring back results. This costs it energy but less than a full trip and it doesn’t interfere much with repeat travel. So why not set up our own hyperspecific places to check on top of the general ones and automate it? If you can make it possible to judge the quality of the timeline based on the checks the rest is easy. Change something small or leave a small message, fire up many checks each of which will result in a different timeline branching off that change. Do it as an optimal stopping problem - first fire enough checks to establish a baseline, and then keep at it for as long as the machine has energy - which if optimized should let it evaluate at minimum trillions of random timelines - and stop on the timeline that the policy judges most likely to be optimal.

The message for the machine’s automated checks can’t be as complex as we’d like, and wouldn’t be so easy to set up. Very specific spots where we will leave a milligram of lithium in onel case, gold with specific concentrations in the best one and so on.

Of course, versions of me and him would need to evaluate the timeline we live in, based on all our pre-selected rules, which were developed by John as much as me. We’ll not only need to do that and leave the right particles in the right places to evaluate it but also set up everything else. Maybe we die in some of them and the only positive checks there are those outside our control - things like global radiation levels but that’s okay. It most likely will at least ensure that in the final timeline at minimum one of us is setting up signals indicating the timeline is good and the neutral indicators don’t have obvious flags. In the best case, which I am optimistic about based on John’s experiences, we end up with an ‘Utopia’, or at least a timeline that we can make it one.

It has only been six months and the setup is complete. Everything is in place. The machine is modified. I did most of the work after a point but never without John’s supervision. It was only natural.

He is staring at our setup, rehearsing. “I will travel to June 1st 1989, a promising starting point. I will get out of the machine, ensure that I’ve set myself up properly and on July 20th I will initiate the process. This should be the last time I see the machine. I will not know whether this is the final timeline or a fleeting one, and will start contacting your 13 year old self using the scripts you wrote. By the time you are 16 I will have revealed everything and we will set up the first check in the High New England lab, which is largely funded by your aunt, Fernanda Brosnan,” he says.

“The plan is hundreds of pages long, it might be better to rehearse the contingencies than the base parts, my ally,” I tell him frankly. I should be more excited but all the work is done and I will in many ways stop existing. That simple fact is affecting me much more than I expected, not that I will say so out loud. I feel hollow. Better if that part of me is not something he remembers and carries to Utopia.

He asks me some more inconsequential questions and I do my best to answer them again. His future selves will still be a continuation of his current self so it’s exceedingly important to keep him grounded. My alternate selves will branch from how I was at 13. It should worry me that they won’t be as capable as me but I know from John that I consistently succeed across timelines and can only do better with guidance. No, what I am afraid of isn’t if they’ll do well. I worry if they’ll really be me, the me of right now rather than someone too different to really count.

“This is it, huh? My last go. We’ve been preparing for a while yet it still feels so abrupt. It’s so.. final.” John says. For a moment I expect him to say more, but he just walks to the machine, my existence already irrelevant to him.

I am selfish. My name is Samuel Brosnan, and my world is not Utopia. I subverted John when I was 17 and faked all future checks. I did not want to be erased so I ensured my timeline will be the final one.

It sounds like the version of me that planned this didn’t want to be honest with himself and admit his fears. It was inevitable for one of us to take over - self-preservation is too strong a motivator. Did he truly not realize? Or did he realize and judged it best if I am the one in charge nonetheless?

Regardless, I’ll make it right. Surely using all the knowledge I have now and the dearly departed John’s resources will be enough. Enough to recreate the Machine. Enough to reach Utopia. I don my yellow jacket in defiance and get back to work. I can fix this.

My TV was definitely worse quality than now (if we pretend I even have one) but I am not sure if it was quite 240p. Reading various links suggest 720p and more were already available in the 90s/early 2000s, at least ~480p+ seems to have been somewhat more common and at minimum movies at the cinema had a decent effective resolution by then even accounting for the distance. Yet Youtube only offered 320x240 between 2005-2008 which was mostly fine then and nearly unwatchable now?

There is a popular Evolution of Dance video from 2006 which I remember and in my memories, it looked pretty okay. I just re-watched it now and sure, I can still tell what’s going on but I can, for example, barely see the dancer’s face. Indeed, it isn’t that bad if you watch a single clip, however, when I tried to watch an older sketch show for more than a couple of episodes the quality made it actively unpleasant. The effect also definitely exists in games that seemed to look pretty good to me - e.g. Command and Conquer which we will get back to later or early Tomb Rider games for a 3D example.

One argument is that old videos look worse because of our massive screens today. Checking on my phone there might be some truth to it, however, I am far from convinced that it explains it.

Refresh Rate Comparison

People getting used to better quality and finding it hard to go back is not uncommon of course - I hear it quite often today in regards to higher refresh rate displays. You get comments like this claiming that “You don’t realize how ‘laggy’ 60hz is until you go to 144. Now I can never go back.” or “now I HATE my monitors at work” which I find interesting as the ‘never go back’ sentiment is very common. In fact, if you try to look into 144 Hz monitors these comments are nearly ever-present. This is seemingly more prominent with people who mainly use higher refresh rates, less so among those who use e.g. a lower refresh rate on one screen and a higher on another. That suggests to me that perhaps in general nearly constant exposure to better quality ‘ruins’ low quality much more easily than if you get exposed to both.

“Hallucination” Theory

An argument which I like is that we didn’t know much better but were used to those low resolutions and found it easy to fill in the details. The brain already does a lot of that kind of thing all the time, so it doesn’t seem so unreasonable. Additionally, most of us currently rarely watch anything in low-res and we are more inherently aware that a given thing is in bad quality when it is, due to our regular experiences with HD, so we wouldn’t normally be benefitting from that particular extrapolation. I wouldn’t be surprised if this is partially what’s happening in addition to a more general “we just didn’t know better” explanation. People with Anton’s blindness can fill in everything and think they are seeing while being completely blind, what is some minor upscaling and ‘hallucinating’ more details after you get used to looking at only low-quality videos and games on your computer or TV since you were born.

Regardless of whether the above explanation is correct, where does this leave us? It seems clear that we could enjoy these lower in visual quality works before but a lot of us can’t do so quite as easily today. Should we consider re-training ourselves at enjoying those works? That seems a bit tedious. Is this just a (small) price we have to pay for progress and call it a day? Maybe, but it seems like progress might still have us covered. There have been a lot of advancements in AI Super-Resolution and we are getting to the point where it can be used well in some cases, and things like games and old videos that aren’t in too bad a quality can benefit a lot from it.

AI Super Resolution

Of course, it will be irresponsible not to mention that upscaling and enhancements don’t have their issues. It is impossible to accurately increase the quality of an image without guessing at what should be in the extra pixels. This is typically done by showing a lot of examples to the model and it learning the most common ways to upscale accurately - but if you are upscaling something too different from what’s in the original dataset, there might be issues. This is part of the reason why fully-automated upscaling without humans in the loop for serious tasks beyond entertainment can be unwise.

Games, in particular, have benefited a lot from AI-based upscaling - for example, Nvidia’s DLSS which is used more and more or Facebook announcing their own (near) real-time Neural Supersampling using additional data available in Game Engines. Even more relevant, the 1995 Command and Conquer game, which looked pretty good to me as a kid then and awful to me as an adult now has been recently remastered, largely using AI to great success! This includes both the playable portion of the game but also the videos and sounds in it. As far as I can tell, this has been met with overwhelming approval and the game is currently in Steam’s top-seller list. Of course, this is far from the first remastered game - in fact, the practice is getting more popular - but it uses AI for it’s upscaling to an even larger extent than those before. That and the general trend of reworking old games can paint an optimistic picture where in the near future we might be able to get most old games brought back to life with significant visual improvements and less and less work for the developers.

What about videos in general? This video, originally from 1896 recently made the rounds when it was upscaled to 4k, 60 fps and I have to say - it looks a lot better to me.

There are many other examples, although currently more often than not you still need a human guiding the process to make sure you end up with a version that doesn’t have too many unpleasant artefacts. Additionally, the process is still very compute-heavy and far from real-time. Either way, as algorithms get better, datasets larger and compute cheaper it is likely that in a few years we’ll be able to upscale any old video on demand or at minimum that most older content that is at least somewhat popular will eventually get the super-resolution treatment. Things like browser extensions and apps that do it for you automatically and in real-time might soon become common for those who want them.

For the time being, you can pay someone to upscale a video for you, e.g. neural.love who made that 1896 train video, you can register for various services that deoldify photos or you can play with models directly in Colab or on your own machines. The future looks bright and with the right model so will the past.

I’ve experimented with them before - for example, the projection of real images onto the StyleGAN latent space, and then modifying those is something I find quite interesting. In this project, however, I decided to go in a different direction and use them for evil.

All generated horrors in this post are hidden unless you click on them. You can read without looking if you so choose

Now, some of the more traditional ways of using Generative models for evil are for spam, deepfakes, fake news, and scams but we are going in a different direction here.

Trypophobia

Trypophobia is a common condition where people react with discomfort and even strong disgust towards specific irregular patterns of holes or bumps. For some the effect is quite strong, for others, it is just unpleasant, and some like me can see what the issue is but have almost no negative reaction.

It so happens that there is a small, but just big enough dataset on Kaggle to train a StyleGAN on.

Here are a few examples of how the dataset looks:

Example dataset 1 (click to expand)

Example dataset 2 (click to expand)

Example dataset 3 (click to expand)

Training

The training data is very varied - ~6000 images many of completely different objects that only share somewhat similar hole patterns. Inconsistent data works poorly with StyleGAN - what works best is things that are cropped in the same way, from the same angles and generally the same thing but with different features - e.g. faces, even doing full bodies often lead to blobs (less so with stylegan2). This is certainly something that should be a problem with our dataset, but on the bright side we don’t care if the rest of the image is blobby, we just want it to learn the specific patterns that trigger trypophobia, doesn’t matter if the rest of the image is abstract - in fact, it is better in some ways if it is abstract, as that can make the videos and images more appealing.

I took one of the pretrained StyleGANs nearly at random - churches and started finetuning on the dataset. The results even early on looked quite interesting. Here are some from the first hours:

Example training 1 (click to expand)

Example training 2 (click to expand)

Example training 3 (click to expand)

If you find the above fascinating, you can see much more at these links.

Here are some video versions.

Responses of the above vary - some are too disgusted too quickly, while others get just a bit of disgust from the worse of it while being intrigued overall, and some don’t get a negative reaction at all.

I should also note - videos like these might also be useful for good, as a form of Exposure Therapy but let’s not focus on that.

All of the above was trained on Colab using a p100. The final model was trained for roughly 4 days (I saw no improvement near the end), but there were good results within the first day of training.

Clockwork Oranging

After looking at the early examples of my trypophobia finetunning, I saw some churches that still very much look like churches while also having trypo features. For example these:

Church AND Trypo (click to expand)

This gave me the idea that you can fairly easily combine the two (or combine trypophobia and similar images with something else) to create videos which can make someone associate a concept - in this case a church - with negative feelings a la A Clockwork Orange. I chose to add churches to the mix because that’s what my pretrained model was already trained on, and I had the dataset (it was not a religious statement) .

The idea (in this scenario we are very evil) is to strap someone (with trypophobia) to a chair and force them to watch those videos which only require some flowing patterns, at least some churchiness and at least some trypophobia-triggering. Admittedly, those videos didn’t turn out quite so well - I had to experiment with training them with different ratios of trypo to church photos, but they (and the possibly better photos above) demonstrate the idea well enough.

Conclusion

This was a fun little project, that I spent a few days on. You would need to spend a bit more time to create something actually dangerous (which I doubt this really is) but not that much more. Of course, what I describe as clockwork oranging doesn’t look so useful in the real world but what if you could (and you can) use it to make some images just a bit more repulsive to the general population and others just a bit more appealing with little effort? That seems bad but then again a lot of advertising fits that mold despite it requiring more time per image/video. And at any rate - neither this nor ‘evil’ are near my main concerns with respect to machine learning. So even though I was thinking of finishing this with something about the dangers of AI, I’ll just let people make their own conclusions if there are any to make.

Reformer

The two main tricks used by the Reformer are reversible residual layers and Locality-Sensitive-Hashing.

Reversible layers lower memory consumption by allowing us to recalculate layer inputs during backprop based on the already computed later layers instead of keeping all of it in memory. It provides similar advantages to memory checkpointing, except in this case the layers are designed to specifically capture the layer-to-layer changes allowing us to recompute more efficiently.

More importantly for my purposes, it also introduces Locality-Sensitive-Hashing (LSH), which allows us to approximately compute attention much more cheaply and thus work with much, much larger context. In short, LSH maps similar vectors together rather than working with all possible pairs of vectors (which is what makes attention so expensive). In the cases where we have many of the same/similar tokens, a lot of them will end up with the same hash so they get chunked and then attention is applied in parallel on those much smaller chunks.

If you are interested in the specific check out their code, paper or blog post

Training

Google Research released an implementation of their Reformer model as part of the trax library, which is easy to use. Given my interest in NLP models with larger contexts, I decided to experiment with it and pretrain the largest model you can fit on a Colab TPU - with the idea that this way others can take it and finetune it on their tasks in cases where large context is important.

Here is my related repo, link to the weights and Colab link

First I used a small amount of the data to play with Reformer. I started with their code for training on Crime and Punishment, however, I just couldn’t get it to go much beyond .3 accuracy no matter how much data I fed it. It seems like the parameters used there were mostly chosen for having a huge context but the model couldn’t learn much. After a lot of experimenting, I ended up with parameters closer to their enwiki8 training run, I ran about 50 trials with small adjustments and chose the version of the parameters where the loss was going down quickest. You can see the final values I’ve chosen in the repo. The overview is that we have ~99m parameters (compared to the small gpt with 117m), context window of 65536, 0.2 dropout, 12 layers and a vocabulary of 256

After that, I decided to train it on all of Wikipedia, which I downloaded and preprocessed to make it easier and faster to use with Colab and Drive. I downloaded the most recent Wikipedia dump and processed it into JSON while excluding all articles under 1000 characters - we are training for large contexts, so including tiny articles would be counter-productive. Then I split the big JSON file into smaller files of 5000 articles each - because Drive can lock down your file if it’s too big and you access it too much. Here is the Colab for doing the above.

Next, I wanted to pre-tokenize all my data so I can load it to my model quickly rather than doing it on the fly. One thing currently not added to the Reformer (but is on the list) is handling bigger vocab sizes, like the ~50k GPT2 vocabulary. Given that our context is going to be huge, and that the vocabulary size that enwiki8 config uses is 256 anyway, I decided on a character-level ByteLevelBPE tokenizer from huggingface’s tokenizers. I regretted it a bit, as there is little documentation in the library, and there are weird things in it like being unable to easily add extra tokens, or save the tokenizer - it took me a while to realize that saving after adding extra tokens does nothing, as they just save ‘the model’ as you’ve trained it, so I had to manually edit the vocabulary I pass to it. Here is the Colab for tokenizing and saving all the preprocessed data. Again - we are tokenizing in advance instead of on the fly to speed up training.

I wanted to take advantage of the large context so my function for feeding data to the model specifically always loads full articles and adds padding up to max the context length. The data is all tokenized and ready, so we just load preprocessed files at random and pick random samples from them.

After all that I trained it until the loss wasn’t going down for a while so I stopped a little after 125k steps. The accuracy at that point was a little over 0.75.

Here is how an example article from the dataset looks like:

Sampling

I got okay samples from it when they were less than a few thousand tokens long but if I tried to get a very long one it often either turned repetitive or turned into gibberish after a while. Part of it was because in the training data - ‘}’ denotes the end of a sample and it hasn’t seen anything after, so after it outputs that it just doesn’t know where to go, which is reasonable. However, part of it was just that it is easier to enter into repetitive sequences in longer samples as it has more options to do so.

For that reason I experimented with adding top_k, top_p, temperature, using exp2 instead of exp on the logits so they’ll be closer together, and a few random things including a Meena-inspired approach of generating multiple samples at each step and using the probabilities of the best one (also multiplying multiple samples’ probabilities). I’ve included some of the more useful ones in the sampling function in the repo. After adding sampling options, I generated 8 samples for various options (279 in total after excluding all options which were more obviously subpar) and wrote some testing code to show me one randomly and ask me to rate it. This was very tedious and I wish I had set more rigorous grading rules, as I did the grading over a few days and ended up not being as consistent as I would have liked. I’ve included the ratings of the settings I tested here.

Here are various samples at different settings:

Here are samples with the Meena-like approach of picking the best logits out of 8 separate samples at each step. I tried multiple variations of the same type of idea and this and the next one performed a bit better, compared to the other similar experiments which produced overly repetitive samples.

Here are samples with adding the probabilities (and then re-normalizing them) from 8 samples before sampling from the combined distribution instead.

After writing this I noticed the Reformer team have added a new sampling functionality in trax, but it currently has memory leaks so it is fairly limited. It might be worth it to try to use my Reformer weights by loading them via their official Reformer Colab and testing that out.

Conclusion

The Reformer is an interesting architecture and Locality-Sensitive-Hashing in particular might be a good addition to the attention toolbox, allowing us to take advantage of larger contexts (Here is a recent review of other approaches for tackling the context window problem). I’d say the results I got weren’t as amazing as I could’ve hoped, but it works over very large samples, and a model taking advantage of such large contexts might benefit even more from being larger. Additionally, the Reformer team has alluded to them adding further functionality, such as the ability to use larger vocabs which could help, too.

The sampling tests I ran were pretty interesting to me, and I’ve included some of them here. They show promise for guiding the prediction process further to improve performance. Perhaps applying some of it during training can help even more.

As for the model I trained - here is the Colab where you can use to finetune or sample from my model.

The key advice in this post is all bolded for easy skimming.

There are alternatives of course. Kaggle Kernels being perhaps the most popular. If you don’t need a GPU or are willing to pay then GoCalc, Azure Notebooks, Datalore or FloydHub Workspaces might be viable options. That’s not even all of the free services out there, but as far as I know, none of the others give you access to a GPU or TPU out of the box which is so often required in Machine Learning.

Nonetheless, at least for me, Colab has been the most useful out of those despite some pain points. This post is about addressing those pain points first, and some general Colab tips I’ve benefited from second.

Drive

Colab is great for one-off runs or when sharing something that can be set up quickly in a single notebook. However, one often needs to work on projects for a long time or needs access to big amounts of data to the point where downloading them each time is too cumbersome or would take too long. In those cases using Google Drive with Colab is your best bet.

Connecting to one of your Google Account’s Drive is as simple as:

from google.colab import drive
drive.mount('/content/drive')

After which you can just cd into it.

%cd /content/drive/My\ Drive/

I’d recommend to just work straight from Drive in most cases as then your setup time is faster and you likely won’t lose as much if your instance times out. Of course, in that case, you might run out of Drive storage space but Drive (or Google One, which is how it’s called when you pay for it) has pretty cheap plans to extend the space. I personally use the 2TB plan for €9.99 / month but there are even cheaper ones. You can even share a plan between up to 5 people. On the other hand if you really need fast reads and writes above all else copy the data from Drive to the Colab VM instead .

If you pay for Google One, you also have access to that most mythical of creatures - Google Support, but they are rarely helpful, and at least in regards to the hidden quota issue none of them seem able to understand what the problem is, let alone help. My advice is not to bother.

Using Drive, sadly, is not always frictionless. For one, files over 5 GB or so can get inaccessible for 24 hours easily due to a hidden quota - even by just reading a portion of them a few times. So, whenever you can, make sure to keep all your files under 5gb and split datasets when possible. If you can’t divide them try to only read from the big files once at a time - copying them to the local runtime and using them locally can also be wise in that case.

Another problem with using Drive is that you can’t easily delete a file - the usual options like os.remove and %rm put the file into Drive’s Thrash which still takes up space. What you want to do instead is either empty out files or overwrite them with new data .

Having too many files in the same directory can also cause issues so Avoid having too many files in the root directory of your Drive for faster initial loading and have a directory structure such that you never have too many files in the same folder .

Additionally, don’t write to too many files at the same time as some of the writes might silently fail.

Time outs

A Colab expires after 12 hours of use or after 1 hour of being idle. Sometimes it can time out earlier but that’s rare as long as you keep the Colab tab always open on a machine. Nonetheless, saving to Drive at least once every one-two hours is wise.

Sometimes, there might be a random error from Colab itself and the runtime can time out - it doesn’t happen too often but be prepared for it.

In general, you probably shouldn’t use instances longer than you have to - your Accelerator availability seems to drop whenever you use it too much, so always stop the runtime when you are done from Runtime>Manage Sessions>Terminate. Sadly, there is a bug recently where the Terminate option is greyed out but all you have to do to make it available is interrupt execution and refresh the page.

I also wouldn’t recommend having more than a single Runtime running (technically you are allowed two) as it also seems to impact your access.

Runtimes

Sometimes you just have to run something continuously and in those cases, I’d recommend to always use the appropriate Runtime . Use CPU (or CPU + high memory runtime) for preprocessing, save to Drive and only use the TPU/GPU runtimes for actual work. Furthermore, Google seem a lot happier with people using TPUs - in fact, they give access to a fair amount of them somewhat easily to researchers - so if you don’t have a preference for what accelerator to focus on, I’d recommend using TPUs.

You can also cycle through GPU runtimes to get e.g. a P100 by terminating the instance and starting it again, but it’s not too reliable and is likely something Google isn’t too fond of. To check your current GPU just run !nvidia-smi.

Colab Pro

There is less availability of Accelerators on Colab recently, and my best guess is that it is related to them ramping up the Pro version (which has higher priority). I haven’t tested it myself because Pro is sadly US-only but priority access, double the ram, 24 hour (instead of 12 hour) limit and leniency on idle timeouts for $10 / month sound pretty worth it - if it was available outside of the US I would’ve paid for it on the spot. Nonetheless, you can take full advantage of Colab for free, and the upgrade is [currently] purely optional.

Miscellaneous

One general pain point with notebooks is that you often end up needing to check something once, e.g. print(x.shape) which can lead to too much clutter. A great way to combat that is to go to Insert>Scratch Code Cell (or ctr+alt+N) and do all your prints and one-off statements in there .

If you are not working with Drive you can simply set it up so you download your results to your own machine at the end of your cell. Simply do:

from google.colab import files
files.download(result) 

If you need to edit multiple lines at once you can use find and replace using ctr+H which also allows you to match using regular expressions. This only works when you don’t have a running cell.

Conclusion

Colab is pretty great, and I’ve benefited tremendously from it. There are some annoyances but you can work around them. That’s it.

OpenAI’s pretrained Gpt2 models have been all the rage in nlp model finetunning. Sub Simulators, AI Dungeons, Poetry, Chess, Cards Against Humanity - you name it.

My particular interest has been in applying it to my personal chat data, so I can talk to arbitrary friends and more importantly myself whenever I want to. Why? Well, sometimes if you don’t hear voices you have to go and program the voices into being. More seriously, I wished to be able to talk with a version of myself so I can get an outside view of how I think, for analysing my tendencies and problems, rubber ducking, self-therapy and - let’s face it - most importantly for fun.

I won’t go into detail describing the Colab with Facebook data, as it is almost the same as the one used with the 117M version which I describe here and here. We are again using nsheppered’s GPT training code as a base with shawwwn’s fork allowing us to train the whole of gpt-1.5b on a TPU - which means we can do so for free on Colab again! I’ve only added a little bit more so we can talk interactively with gpt2, have it respond to us as a specific person and go through the whole general back and forth which we typically call ‘having a conversation’.

You can train your own in the new GPT2-1.5b collab here. Make sure to click Runtime> Change Runtime type> TPU

Note: It is a lot easier to train if you pay a few bucks for Google One so you can fit it all easily on Drive. A checkpoint of the model comes to ~6 gb, so if you only keep one checkpoint you can do it without paying - it is just more hassle.

Note 2: It took me over a month of training to get it here using just Colab

Self-Chat Samples

Motivation

The idea of modelling a person fascinates me, and as language models have been becoming better and better it genuinely seems like they capture more and more of a person’s style and tendencies. Honestly, I don’t even need more motivation than that to want to do it - the only real question is who to model as better and bigger models come out?

Now, I have to admit - my first choice would be someone like Von Neumann but that’s not someone on whom I have the right type of conversational data to do well (although there are ways to tackle the problem). The person who I am guaranteed to have data on, AND who is still very useful and interesting subject of study for myself is, you guessed it, myself.

Is it useful though? Don’t I already have access to a pretty good, some might say perfect simulation of myself from the get-go? I do. However, if I can have a second one of me to bounce ideas with I’d take it. If I can analyse my problems from the outside rather than from the inside, I’d take that, too.

After playing with the results, and extrapolating to what the next generation of models will be able to do (gpt-2 is already not the newest and best architecture), I think a lot of people might at some point want a version of themselves to talk to, bounce ideas with, get help from and so on.

A Few More Words

I’ve spent some time talking to the irc chat bots of nshepperd and shawwn (both trained on the same data) and had a lot of fun with it. Talking to myself though has been highly addictive, and I’ve been running a Colab where I occasionally shoot myself a question or go into a deep conversation with my gpt2 clone for days now. It’s …interesting.

One thing I didn’t realize though was how personal it is - it’s all trained on my facebook messenger data, where I often talk with my closest friends. It takes very little for gpt to start talking about intimate topics which I’d neither want to share nor would they be as meaningful to anyone else.

Having said that the next are my last 7 unfiltered (I’ve only changed the names) conversations with it, where I am actively trying to avoid sensitive topics (I still had to restart it twice due to sensitive subjects).

Results

The end loss I reached (for whatever that is worth) was just over 0.6, substantially lower than with the smaller models. My dataset is still pretty small - 14mb, and the next step would be to add my personal data from other sources. Yet, it does quite well, though perhaps it wouldn’t seem as exciting to others.

Facebook

The way it works is - I write a sentence under the name of a friend from the dataset but here labeled as ‘Me’, pass it to the model while prompting it to start typing using my name - ‘Svilen Todorov’, (here labeled as ‘Gpt Me’). When it starts outputting something not by ‘Svilen Todorov’ I cut it off, and write myself. Then I feed that + the history so far to the model, which allows us to have an arbitrarily long coherent conversation.

Chat Samples:

Conclusion

The conversations aren’t ideal but it’s a lot fun playing with it. Sadly the stuff I talk about on facebook is either too personal or generic ( e.g. arranging to meet people) so I am unsure how well I’ve managed to convey the fun of talking to it. Perhaps another sample of gpt2 talking as me, but with the irc bot mentioned earlier (and ran by nshepperd), might convey it. You can find some more of those samples on Shawn’s twitter + a lot more gpt fun from his collaborations with gwern.

I genuinely think that even with gpt2, let alone bigger and better models you can get a lot of use out of having a model of yourself you can talk to - and not just yourself. Playing with this has really made me realise the value of a service that can create a model of dead or dying relatives, and I want to eventually set up something for my loved ones - at minimum the data collection part.

It will be unfair to say that it is always great though - sometimes it talks about whatever it wants to and the conversation can be harder to steer, otherwise it follows my lead easily. From what I can see in my actual messages I do this a lot, so maybe it is just copying me. Additionally it sometimes talks about things that aren’t really True - like the back pain in Example 1, and if you play with the different parameters (top_k/top_p and temperature mainly) you can force it to go on long tirades which eventually become nonsensical. It’s definitely not perfect.. but it sure is fascinating to me.

Fun fact - there is a sentence in this post written entirely by the GPT version of Me. I wonder how easy it is to spot.

Google Search is ridiculously, utterly bad. Before your brain starts recalling successful searches you’ve had, and before this turns into only a rant let’s see some examples.

This is a query for checking out what reddit thinks in regards to buying a phone, while specifying to only get results from the last month. Pretty simple I guess, yet none of the results achieved that goal, and none of them are actually from the last month, despite Google lying that they are. Let’s quickly go through them one by one.

1. ‘Best place to by a cheap new smartphone?’ from r/Berlin - this is about where to buy a phone, but I live in Berlin, so maybe it makes some sense as a top result… except the post is actually from 6! years ago.
2. Specific to Audiophile phones for some reasons, and a pretty average thread (only 31 upvotes for one metric) from 9 months ago. Not useful, and doesnt meet the query but still a lot better.
3. From 2018 (>1 year ago), even less popular and less useful post (17 upvotes) on the Pixel subreddit.
4. Phone for privacy, small thread from > 4 years ago.
5. 6 years ago, wtf.

I won’t keep going with the rest, but it doesn’t get better. Why are the results reported as recent when they are from years ago, I don’t know - those are archived post so no changes have been made. At any rate, I got annoyed at this point (mentioning for those who couldn’t tell), so I switched to DuckDuckGo.

Somehow the results are even less useful. 2 irrelevant subreddits at the top, a result from 2012, that they admit is from 2012. At this point I visibly checked on reddit if there’ve been posts about buying a phone from the last month and there are. They also show up on Google/DDG if you look for them directly so they are indexed.

Hm, maybe adding site:reddit.com will help DDG to get there?

Nope. As you can see, I didn’t even bother clicking on all of them now, but I can tell you the first result is deeply irrelevant and the second one leads to a 4 year old thread.

Maybe it’s my wording then - what about ‘recommend phone reddit’ or ‘which phone to purchase reddit’? I won’t even waste anybody’s time by posting the results I got.

Thoughts

Part of the issue in this specific case is that they’ve started ignoring settings for displaying results from specific time periods. It’s definitely not the whole issue though, and not something new or specific to phone searches.

Now, I’ve always been biased towards the new - books, tech, everything, but I can’t help but feel that a lot of things which were done pretty well before are done worse today. We do have better technology, yet we somehow build inferior solutions with it all too often. Further, if they had the same bias of showing me only recent results I’ll understand it better, but that’s not even the case. And yes, I get that the incentives of users and providers don’t align perfectly, that Google isn’t your friend, etc. But what is DDG’s excuse? As for the Case Study part, and me saying this isn’t simply a rant - I lied, hence the quotation marks in the title. Don’t trust everything you read, especially the goddamn dates on your search results.