<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
  <title>River&#39;s Blog</title>
  <subtitle>A blog about whatever technical stuff I feel like talking about today</subtitle>
  <link href="https://riverseeber.net/blog/feed.xml" rel="self" />
  <link href="https://riverseeber.net/blog/" />
  <updated>2026-08-08T00:00:00Z</updated>
  <id>https://riverseeber.net/blog/</id>
  <author>
    <name>River Seeber</name>
    <email>riverseeber12@gmail.com</email>
  </author>
  <entry>
    <title>The Internet was Supposed to be a Community</title>
    <link href="https://riverseeber.net/blog/post/the-internet-was-supposed-to-be-a-community/" />
    <updated>2026-08-08T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/the-internet-was-supposed-to-be-a-community/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;August 8, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The internet nowadays has become saturated in influencers. Whether that&#39;s the people we typically call influencers — TikTok and Instagram stars — or other people who basically maintain the same social role — like big YouTubers, or other big accounts on literally any other social media platform. To me, an influencer is just a person on the internet who has a sufficiently large enough follower so that they&#39;re interactions form a parasocial relationship, rather than a standard social interaction. The only caveat I&#39;ll add to this definition is that they need to be famous &lt;em&gt;because of&lt;/em&gt; their online personality on social media. Thus, Jimmy Kimmel — with his 4 million followers on Instagram — wouldn&#39;t be an influencer, but Hank Green — with 3 million YouTube subscribers — would be. That&#39;s because Jimmy Kimmel has a television show that makes him famous, whereas Hank Green is famous specifically for the work he does on various social media platforms.&lt;/p&gt;
&lt;p&gt;Additionally, Cory Doctorow would also not really be an influencer, since even though he is famous for &lt;em&gt;online&lt;/em&gt; work (blogs, webpages, etc), he isn&#39;t particularly well-known for things that he does on social media platforms (although he does spend a good amount of time &lt;em&gt;talking about&lt;/em&gt; those platforms).&lt;/p&gt;
&lt;p&gt;Here&#39;s the thing, it&#39;s &lt;em&gt;really weird&lt;/em&gt; how centered our online platforms are around influencers. It&#39;s weird that we&#39;ve created an &lt;em&gt;amazing&lt;/em&gt; technology called the internet, a technology that allows us to engage in easy, &lt;a href=&quot;https://www.youtube.com/watch?v=d8PndpFPL8g&quot;&gt;many-to-many communication&lt;/a&gt; amongst ourselves with practically no friction. A technology that subverts the old model of only having a small number of &amp;quot;thought leaders&amp;quot; and creators. A technology that enables anyone to publish their ideas and works, enables us all to be part of the global conversation. And yet what have we coalesced around? We&#39;ve re-invented the damn thought leaders! We&#39;ve organized our entire online ecosystems to focus on influencers!&lt;/p&gt;
&lt;p&gt;We&#39;ve created convoluted social networks, and reduced the friction for any user to create a relationship with another. The original idea about all of this was that it would uplift the significance of the average user. Make each voice more powerful. Get us to understand what the true will of the people was. And somehow, what we&#39;ve ended up with is a system that obsesses over the most-followed users on the networks — often neglecting all of the users who follow them.&lt;/p&gt;
&lt;p&gt;Not to say that those influencers are in a position of power per se. Sure, being well connected gives them a certain level of influence (an influencer having influence? Wow, what a concept), but it&#39;s not like they&#39;re the ones who built the platform. Both the followers (who have been converted into consumers) and influencers (converted into producers) are helpless to the changes that the platform undergoes. The owners of YouTube, Facebook, Instagram, Twitter, Bluesky — these are the people who are driving the changes. And these changes are often done in service to &lt;a href=&quot;https://en.wikipedia.org/wiki/Enshittification&quot;&gt;increasing their own wealth&lt;/a&gt; — not as a way to improve the social network.&lt;/p&gt;
&lt;h1&gt;Returning to the indie internet&lt;/h1&gt;
&lt;p&gt;I&#39;ve been considering what the internet might look like if we moved away from a centering on influencer culture. I love Hank Green and many other &amp;quot;high influence creators,&amp;quot; but surely this isn&#39;t the only fruit that the internet could yield? Take the &lt;a href=&quot;https://www.youtube.com/watch?v=Tv223kX0SRg&quot;&gt;indieweb&lt;/a&gt;, for example. This is one of those few places online that really do exemplify what I&#39;m talking about. Lots and lots and lots of different individuals all learned how to make a website, and put something that they created out onto the internet. Some websites involve a lot of effort, some involve a bit less. Regardless, all of them exist and get seen by various people. None of these website owners become celebrities. There&#39;s no algorithm that gets them millions of followers. They made the website for the same reason that somebody might crochet a scarf, or paint a sunset — because they&#39;re a human. These people &lt;em&gt;aren&#39;t after anything&lt;/em&gt;. They&#39;re creating for the act of creation. They&#39;re not chasing an audience in the way that an influencer would.&lt;/p&gt;
&lt;p&gt;&lt;img class=&quot;introImg&quot; src=&quot;https://archive.org/download/influencers_graphic/influencers_graphic.jpg&quot; alt=&quot;An image showing a small collection of 3 nodes, each connected to a *lot* of smaller nodes. Caption is &amp;quot;Small number of high-influence users&amp;quot; Below it, an image of a larger collection of 7 nodes, each connected to a few smaller nodes each. Caption is &amp;quot;A large number of low-influence users&amp;quot;.&quot;&gt;
&lt;/p&gt;
&lt;p&gt;What if we imported this approach to other areas? Web design is a very particular type of medium. What if we brought this approach to social media? Replace the small number of big influencers we have today with a large number of small artists, all just creating for the love of creation. I wanna see a video platform like YouTube where people post fun little vlogs about their day to their 200 or less followers. Not those overly polished vlogs, the ones that the algorithm props up. I&#39;m talking about the kind of vlog that made vlogging popular in the first place — a regular person, speaking to the camera, being honest and open, not performative and perfect. Sometimes with a shitty camera and microphone, showing off the typical parts of their day. And it wouldn&#39;t be to build an audience or a brand. It would literally be because humans have an intrinsic need to create. You put a human in the worst situations imaginable, and they will still create art, because art is part of the human experience. I want to see the internet become a place that it&#39;s normal to create art for, without needing to expect fame and massive engagement from it.&lt;/p&gt;
&lt;p&gt;According to the very rudimentary analytics that I employ on this site&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/the-internet-was-supposed-to-be-a-community/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;, I get maybe 20 people or so to read my articles, if I&#39;m lucky. I&#39;ve had 2 articles blow up and reach a few hundred viewers, but it&#39;s not common. I would love to get more readership, but I&#39;m not milking the algorithm day after day to chase after that. Unlike some of the things you might hear from other people in the indieweb community, I don&#39;t necessarily want to see more people making personal websites. Like, yes, okay, I totally think it should be easier to do (we need better tools for it), and I think people who are drawn to it totally should come join us. But also, I wanna see the people who aren&#39;t into web design go do the same thing we&#39;re doing, but in a new medium! I wanna see the film and photography people making videos or sharing their images. I wanna see visual art people sharing their pieces, maybe even showing their process. I wanna see writers posting to easy blogs they made without needing to learn how to code. I wanna hear music that people recorded in their bedroom on their cellphone.&lt;/p&gt;
&lt;p&gt;I know that there are already people doing this, but the thing that upsets me is that our social platforms make it difficult to find these folks. The fact that the platforms are designed to orbit around the large creators means that it&#39;s hard for people to actually prioritize smaller creators in their feeds. This is especially true if the small creator isn&#39;t making an active effort to climb the engagement ramp to become a big influencer. The algorithm punishes casual users. In order to be seen &lt;em&gt;at all&lt;/em&gt;, you have to be trying to be seen &lt;em&gt;by everyone&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;I&#39;m tired of it.&lt;/p&gt;
&lt;h1&gt;Can it be done?&lt;/h1&gt;
&lt;p&gt;Well, the question is, can we create this indie culture that I&#39;m advocating for, and if so, how? I think it is possible, but we can&#39;t do it on the corporate platforms. Those platforms are designed to increase &amp;quot;engagement&amp;quot;, and casual indie creators aren&#39;t exactly useful towards that end. Instead, we should form these communities on the Social Web. The Social Web is like if social media wasn&#39;t owned by a corporation — instead being controlled and owned collectively by the community. The Social Web is up and running, and has lots of users already on it. One of the early things you&#39;ll notice while using this network is that the influencer culture is largely gone. It turns out, when you don&#39;t have an algorithm constantly trying to convince you that the producer-consumer relationship of influencers and their followers is natural, you don&#39;t end up falling into those categories.&lt;/p&gt;
&lt;p&gt;Sure, the Social Web has bigger and smaller accounts with varying degrees of influence, but even the biggest accounts don&#39;t fall so hard into the parasocial weirdness that goes on on Instagram or Twitter or whatever else. And a &lt;em&gt;lot&lt;/em&gt; of the accounts that people tend to follow are smaller accounts. It&#39;s common to follow someone who has 200 followers or less. The moment you remove an algorithm that prioritizes parasociality, people stop creating those dynamics.&lt;/p&gt;
&lt;p&gt;There&#39;s still some work to be done, though. The biggest problem on the Social Web is that there&#39;s only a few types of dorks on there, still. Mostly, it&#39;s computer dorks. We desperately need more enthusiasts across the spectrum of interests. History dorks, art dorks, film, photography, music, sound design. We need you here, because we can&#39;t find your stuff when you&#39;re on Instagram, and there&#39;s not enough of you on the Social Web yet for us to get our fill.&lt;/p&gt;
&lt;p&gt;If you&#39;re somebody who creates art — that is to say, if you&#39;re human — you should come share it on the Social Web. It isn&#39;t so parasocial and weird over there as it is on literally every other part of the interactive internet. I&#39;ve got a short guide for &lt;a href=&quot;https://riverseeber.net/blog/post/so-you-want-to-join-the-social-web/&quot;&gt;how to get started on the Social Web&lt;/a&gt; that you can check out. There&#39;s also plenty of other resources you can find with just a simple web search.&lt;/p&gt;
&lt;p&gt;The internet of today often feels dead and hollow, but a better alternative is possible — we can switch to tools that enable community and trust, instead of those that destroy it. The Social Web is far from perfect, but it&#39;s one of the better options available, given the current bleak situation.&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;By the way, if you&#39;re uncomfortable knowing I use analytics, enabling a simple adblocker in your browser should prevent it from sending data. The service I use is called &lt;a href=&quot;https://www.goatcounter.com/&quot;&gt;Goatcounter&lt;/a&gt;. It&#39;s an indieweb service, and it doesn&#39;t track you around the web like Google Analytics does, or anything. &lt;a href=&quot;https://riverseeber.net/blog/post/the-internet-was-supposed-to-be-a-community/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>Accessibility in Tech Communication</title>
    <link href="https://riverseeber.net/blog/post/accessibility-in-tech-communication/" />
    <updated>2026-07-19T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/accessibility-in-tech-communication/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 19, 2026&lt;/em&gt;&lt;/p&gt;
&lt;img alt=&quot;XKCD&#39;s comic titled &#39;Average Familiarity&#39;. Even when they&#39;re trying to compensate for it, experts in anything wildly overestimate the average person&#39;s familiarity with their field.&quot; src=&quot;https://imgs.xkcd.com/comics/average_familiarity_2x.png&quot; class=&quot;introImg&quot;&gt;
&lt;p&gt;Regardless of what the domain of knowledge we&#39;re talking about, there&#39;s a common tendency for people to forget what it was like to be a non-expert.&lt;/p&gt;
&lt;p&gt;It&#39;s practically a universal fact that the more experienced you become in a particular field, the &lt;em&gt;worse&lt;/em&gt; you become at explaining those concepts to regular folks. This comes as somewhat of a paradox at first. Shouldn&#39;t the people who know the &lt;em&gt;most&lt;/em&gt; about a subject be most qualified to teach it?&lt;/p&gt;
&lt;p&gt;Until graduating from community college this semester, I&#39;ve spent the last 3 years working as a peer tutor through the school. That is, I was a student, and once I would finish a class (with a good grade), students could come to me for questions in that class. It was often the case at the Learning Center (the building where tutoring happened) that students had a &lt;em&gt;much&lt;/em&gt; easier time understanding concepts when we — college students without degrees — explained them, as opposed to the professor — who would have at least a Master&#39;s or PhD in the subject.&lt;/p&gt;
&lt;p&gt;You see, after spending over a decade as an expert, they&#39;ve forgotten the type of misunderstandings that a normal person tends to have when approaching the subject. A student who took the class a year or two ago though? They have both things; enough expertise in the subjects covered to explain them, but also a recent memory of what it was like to be a learner — what it was like not to understand the subject yet.&lt;/p&gt;
&lt;p&gt;I don&#39;t want to give the impression that experts are helpless to this situation, however. It&#39;s largely caused by the amount of time they spend around other experts, and how little time they spend in comparison trying to understand how non-experts think about their field.  There&#39;s also probably something to be said about how little time STEM major (who tend to be most prone to this effect) spend on learning language and communication skills.&lt;/p&gt;
&lt;h1&gt;Tech communication&lt;/h1&gt;
&lt;p&gt;This isn&#39;t just an issue for college students, though. This phenomenon becomes incredibly consequential when it comes to the intersection between tech and politics. There&#39;s a lot of really important issues in tech that regular people ought to be paying attention to, but the issues are just too complicated for them to understand on their own. They need somebody more experienced to explain it. The problem? The experts often do a really bad job explaining things.&lt;/p&gt;
&lt;p&gt;In the summer of 2013, news broke to the world that the United States government had been conducting a mass surveillance program against its own citizens. The information came as part of a giant leak by NSA whistleblower Edward Snowden. Snowden had given the media everything. Mountains and mountains of evidence. Journalists had to pour through this, discovering which documents were important for the American public to be aware of, and which were important to remain private to protect national security.&lt;/p&gt;
&lt;p&gt;Finally, the importance of the revelations had to be explained in a way that could be understood by regular American people, not just the computer folks. Journalist Nicole Perlroth describes being pulled in to work on the project in her 2021 book &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/This_Is_How_They_Tell_Me_the_World_Ends&quot;&gt;This Is How They Tell Me the World Ends&lt;/a&gt;&lt;/em&gt;. She was &lt;em&gt;not&lt;/em&gt; a computer person, and that&#39;s kind of why she had been tasked with covering cybersecurity stories previously, and why she was involved in this project too.&lt;/p&gt;
&lt;p&gt;She wasn&#39;t a computer person, but she was smart enough that she could understand the importance of computer-related topics, and then explain it back to other regular people. A computer person would&#39;ve just thrown a word salad of jargon at the reader. Somebody like Perlroth is able to explain it in terms that can be understood by the masses.&lt;/p&gt;
&lt;p&gt;As we enter into the enshittocene, the era of mass &lt;a href=&quot;https://en.wikipedia.org/wiki/Enshittification&quot;&gt;enshittification&lt;/a&gt; — the phenomenon where all parts of the internet we used to really enjoy is turning into shit — it becomes incredibly important that we take note of how much or little of our message is understood by regular people. This is especially true because of the somewhat complicated causes of enshittification.&lt;/p&gt;
&lt;p&gt;I don&#39;t think the answer is for us computer people to not be the ones talking about this, but I do think that we ought to spend as much time as possible trying to understand how regular folks actually think about these issues, so that we can improve our messaging and actually be understood, instead of just using the most technically correct jargon that gets misunderstood.&lt;/p&gt;
&lt;p&gt;I find failures in communication all over the place in tech-related topics. I wrote a &lt;a href=&quot;https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/&quot;&gt;whole blog post&lt;/a&gt; about how bad most people are at explaining the Fediverse (or &amp;quot;Social Web&amp;quot; as I like to call it) to people stuck on Big Tech social media.&lt;/p&gt;
&lt;p&gt;But this isn&#39;t the only place we suck at this. It&#39;s everywhere, and the more time you spend actually explaining these ideas to your non-expert friends and family, the more you&#39;ll start to notice how bad the communication is.&lt;/p&gt;
&lt;h1&gt;What Apple did right&lt;/h1&gt;
&lt;p&gt;We need our message to be accessible to the people who it matters to, and to the people who can help us organize to work on the issue. When we talk about privacy, that conversation should be accessible to anyone smart enough to own and use a cellphone or computer, at &lt;strong&gt;minimum&lt;/strong&gt;! If we&#39;re talking about Big Tech or its alternatives, that should be understandable by &lt;em&gt;anyone who uses those Big Tech products&lt;/em&gt;! This should be intuitive, but it&#39;s not.&lt;/p&gt;
&lt;p&gt;Just look at the way &lt;a href=&quot;https://friendi.ca/&quot;&gt;Friendica&lt;/a&gt; — an open source, non-Big Tech alternative to Facebook — advertises itself on its own website:&lt;/p&gt;
&lt;blockquote&gt;
&lt;h2&gt;Decentralization&lt;/h2&gt;
&lt;p&gt;Decentralized architecture with no central authority or ownership.&lt;/p&gt;
&lt;p&gt;Relationships can be made across any compatible system, creating a network of Internet scale made up of smaller sites.&lt;/p&gt;
&lt;p&gt;Seamless wall-to-wall posts and remote comments, even across different network nodes.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;blockquote&gt;
&lt;h2&gt;Interoperability&lt;/h2&gt;
&lt;p&gt;Built-in support for ActivityPub (e.g. Funkwhale, Hubzilla, Mastodon, Pleroma, Pixelfed), OStatus (e.g. StatusNet, GNU social, Quitter) and diaspora* protocols.&lt;/p&gt;
&lt;p&gt;Support for email contacts and communications (two-way) via IMAP4rev1/ESMTP.&lt;/p&gt;
&lt;p&gt;Import arbitrary websites and blogs into your social stream via RSS/Atom feeds.&lt;/p&gt;
&lt;p&gt;Support for other services via plugins.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This is a terrible advertisement for 90% of the people who actually use Facebook. The only people who even understand what this is saying are people who are &lt;em&gt;heavily&lt;/em&gt; involved in the tech world. What if we catered to the 90% of people who don&#39;t know about this stuff, instead of fighting for the 10% or less that know what &amp;quot;federation&amp;quot; means?&lt;/p&gt;
&lt;p&gt;We ought to be acting more like Apple used to when they would sell new tech products. Apple invented a lot of things, but on top of that, they were incredibly successful at &lt;em&gt;popularizing&lt;/em&gt; their version of a product that already existed among niche enthusiasts.&lt;/p&gt;
&lt;p&gt;Apple wasn&#39;t the first to sell an MP3 player, but they were the ones to get it adopted by a mass audience. As &lt;em&gt;Dan Fixes Coin Ops&lt;/em&gt; &lt;a href=&quot;https://retro.social/@ifixcoinops/116712444755950963&quot;&gt;wrote&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Instead of saying stuff like &amp;quot;four gigs of space, six hour battery life&amp;quot; they said &amp;quot;A thousand songs in your pocket and you don&#39;t have to fuck around with CDs,&amp;quot; while all the other MP3 player makers were fighting over people who already knew about MP3 players, apple targeted people who&#39;d never used an MP3 player before&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The technical details were there to be found for those interested, but they weren&#39;t the headline of the advertisement. The pitch involved why people should care. It was up to them after that to go look up the specs if they wanted to.&lt;/p&gt;
&lt;p&gt;We need to take this page from Apple in how we advocate for open source alternatives to Big Tech. We also can use it when we explain concepts like interoperability (a term I hate saying, and would love it if somebody could come up with a better phrase) and enshittification.&lt;/p&gt;
&lt;p&gt;The revolution of the nerds is never going to win, because nerds are not a big enough slice of the population. If we want to improve things, we need to start targeting the general public; getting them to understand and care about the issues in tech, and then getting them to organize with us to fight back!&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Why We Fight For The Internet</title>
    <link href="https://riverseeber.net/blog/post/why-we-fight-for-the-internet/" />
    <updated>2026-07-18T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/why-we-fight-for-the-internet/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 18, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;There are a lot of things in this world worth fighting for. Things we would like to see improved in our society. Wealth inequality, corporate and political corruption, civil rights for queer people, especially trans and gender non-conforming folks. An end to the daily kidnappings of our neighbors by masked feds, who have been authorized to perform &lt;a href=&quot;https://en.wikipedia.org/wiki/Kavanaugh_stop&quot;&gt;Kavanaugh stops&lt;/a&gt; — detaining people on account of their accent or other racially-based indicators. We&#39;re witnessing a mad king punish disfavourable media outlets, and &lt;a href=&quot;https://www.npr.org/2026/04/29/nx-s1-5804483/8647-meaning-comey-indictment&quot;&gt;political dissidents&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Why then, might you ask, should anyone be focusing on the politics of &lt;em&gt;the internet&lt;/em&gt;?! Believe it or not, it&#39;s my opinion that fighting to improve the internet is one of the most important fights we can be having right now, in order to maximize our political effect.&lt;/p&gt;
&lt;h1&gt;Means and Ends&lt;/h1&gt;
&lt;p&gt;In the early 20th century, a new information medium was hot, spreading across the United States. It was called radio. It allowed a speaker to reach thousands, maybe millions of listeners all at once, if he was incredibly successful. And it was all real time. Prior to radio, the only way to spread a message would be to organize inside of some auditorium and speak to a few hundred people who were able to get themselves physically into the event. Now, you could reach listeners sitting anywhere between Los Angeles and New York City.&lt;/p&gt;
&lt;p&gt;President FDR famously delivered his Fireside Chats over radio to assure the country during the hardships of the Great Depression and subsequently World War II.&lt;/p&gt;
&lt;p&gt;The radio was more than just a medium for entertainment. Listening to music at any time was great, but the part that really made this technology so valuable was the way it enabled mass communication across the country, which allowed us to organize and confront our problems as a nation together. It&#39;s become clear that radio has had a monumental impact on public opinion in the 20th century.&lt;/p&gt;
&lt;p&gt;Similarly, the internet is not worth fighting for just for the sake of the internet. Rather, the internet should be fought for as a &lt;a href=&quot;https://pluralistic.net/2025/02/13/digital-rights/&quot;&gt;means towards the end of maintaining a healthy democracy&lt;/a&gt;. As a tool for organizing and communicating about the issues of our day.&lt;/p&gt;
&lt;p&gt;Humans are a social species, and as it turns out, we&#39;re much better at thinking as a collective than we are as individuals. The internet should help &lt;a href=&quot;https://www.programmablemutter.com/p/were-getting-the-social-media-crisis&quot;&gt;inform us on the public opinion of that collective&lt;/a&gt;. If our means of communicating with the collective is being skewed by a CEO like Elon Musk or Mark Zuckerberg, our thoughts will also be skewed in that direction.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;A healthy internet should enable people to make easy, intentional decisions about which online communities they would like to be a part of. A healthy internet allows for disagreements between people, and subsequently enables constructive discussions between those people, should they so choose to engage with one another.&lt;/p&gt;
&lt;p&gt;But a healthy internet also allows people to avoid discussions they don&#39;t want to be a part of. The consequence of not giving people this kind of freedom to not engage is what we see on the internet now: everyone is frustrated and mad, getting into flame wars and screaming competitions. I&#39;d wager a lot of the people spreading negativity in certain comments sections are doing so because an opaque algorithm keeps feeding them content that they vehemently disagree with, and at a certain point, they feel the need to correct the narrative.&lt;/p&gt;
&lt;h1&gt;The Social Web&lt;/h1&gt;
&lt;p&gt;So, I&#39;ve discussed some of why the internet is important to fight for, and I&#39;ve hinted at some of the complaints I have against the internet as it exists today. You may now be wondering what can be done.&lt;/p&gt;
&lt;p&gt;Probably the most succinct summary of the problems with the internet, as well solutions for improving it, can be found in Cory Doctorow&#39;s theory of &lt;a href=&quot;https://en.wikipedia.org/wiki/Enshittification&quot;&gt;Enshittification&lt;/a&gt;. In short, big platforms like Twitter and Facebook have become so dominant as to maintain monopolistic power over their sector. No longer disciplined by the forces of market competition, these oligarchic corporations have been allowed to degrade their platforms and tug on the strings without facing negative consequences on their end. The solution then is to leverage antitrust policy and enforcement against them, in order to reduce their power over us.&lt;/p&gt;
&lt;p&gt;While I don&#39;t want to minimize the importance of policy-based solutions to this problem, it&#39;s also not the only action we can pursue.&lt;/p&gt;
&lt;p&gt;We&#39;re currently sitting on the precipice of a new digital revolution that&#39;s just waiting to happen. And unlike the fake &amp;quot;revolutions&amp;quot; of NFTs, Bitcoin, and &amp;quot;AI&amp;quot;, this new revolution promises to actually improve the lives of regular people, not just function as a power grab for the ultra wealthy. It&#39;s called the Social Web.&lt;/p&gt;
&lt;p&gt;It&#39;s important to remember that the problems we face with the internet aren&#39;t actually about the internet at all — they&#39;re issues caused by the popular online ecosystems. There does, in fact, exist an internet outside of the &amp;quot;5 giant websites filled with screenshots of the other 4&amp;quot;. The Social Web exists outside of this Big Tech sphere. As I explained in one of my &lt;a href=&quot;https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/&quot;&gt;previous blogs&lt;/a&gt;, it works like this:&lt;/p&gt;
&lt;p&gt;The Social Web is just a bunch of different websites on the internet. You pick just one website to sign up with. From that site, you can interact with people who signed up on &lt;em&gt;any other website&lt;/em&gt; on the Social Web. If you ever don&#39;t like the way moderation decisions or other actions are taking place on the website you chose, you can always pick up your account and move it to a different website.&lt;/p&gt;
&lt;p&gt;On the Social Web, moderation decisions are handled by individual sites. That way, you&#39;re able to move to a site that aligns with your values, without losing access to your friends who signed up through a website with slightly different moderation rules. The result of all this is that the Social Web isn&#39;t owned by any company, it&#39;s controlled by the community, not by Big Tech.&lt;/p&gt;
&lt;p&gt;The best part is that this isn&#39;t some theoretical technology. This is a living, active social media ecosystem that is just waiting for you to join it! It might not have as many people yet as platforms like Instagram or Facebook, but it&#39;s not empty either. There&#39;s lots of people interacting with one another all the time, every single day on the Social Web, and they would love to see you join them. It&#39;s an incredibly welcoming community, and it&#39;s just so kind and nice there. It&#39;s nothing like anything I&#39;ve ever seen on social media anywhere else. The model really does seem to prove to make people interact in much more positive, constructive ways.&lt;/p&gt;
&lt;p&gt;I recently wrote a blog post explaining &lt;a href=&quot;https://riverseeber.net/blog/post/so-you-want-to-join-the-social-web&quot;&gt;how to get onto the Social Web&lt;/a&gt;, which you can read in full, but the passage you&#39;re probably looking for is here:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The quickest and easiest way to get on the Social Web is to just make an account on Mastodon. You can either pick the default site &lt;a href=&quot;https://joinmastodon.org/&quot;&gt;here&lt;/a&gt; to sign up, or if you&#39;d like to sign up with a specific community oriented around a hobby, identity, culture, or whatever else, you can check out &lt;a href=&quot;https://fedi.garden/&quot;&gt;this page&lt;/a&gt; to see if something appeals to you.&lt;/p&gt;
&lt;p&gt;There&#39;s also &lt;a href=&quot;https://fedi.tips/&quot;&gt;fedi.tips&lt;/a&gt;, which is a very helpful resource for answering any kind of questions you might have while getting started on the Social Web (fedi.tips refers to the Social Web as &amp;quot;the fediverse&amp;quot;).&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I also recommend following as many interesting people as you can, since there really is no algorithm to push content onto your feed. The &amp;quot;algorithm&amp;quot; on the Social Web is the people you follow, and the things they post or repost. You can find my account &lt;a href=&quot;https://riverseeber.net/follow-me/#panel1&quot;&gt;here&lt;/a&gt;, by the way.&lt;/p&gt;
&lt;h1&gt;Conclusion&lt;/h1&gt;
&lt;p&gt;The internet is worth fighting for not because it is important in its own right, but because it&#39;s an important tool that allows us to organize, discuss, and fight for all nature of issues that &lt;em&gt;do&lt;/em&gt; matter in their own right.&lt;/p&gt;
&lt;p&gt;The most important thing we can do to fix the internet is to improve policy and enforcement related to competition law. Secondary to that, we really ought to be moving to communication mediums that are owned by the people and the community, not by a vanishingly small number of oligarchic billionaires.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Against Pessimism</title>
    <link href="https://riverseeber.net/blog/post/against-pessimism/" />
    <updated>2026-07-17T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/against-pessimism/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 17, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;We live in a pretty scary part of history right now. Global fascism is on the rise, aided by a digital environment that seems to be &lt;a href=&quot;https://www.youtube.com/watch?v=d8PndpFPL8g&quot;&gt;purposefully engineered&lt;/a&gt; to &lt;a href=&quot;https://www.youtube.com/watch?v=9zi0ogvPfCA&quot;&gt;devour public trust&lt;/a&gt;. We&#39;re watching civil liberties beginning to be peeled away, and a scare campaign be deployed against our neighbors. All the while, democracies around the world are embracing mass surveillance tools in an attempt to spy on their own citizens in nearly all aspects of our lives.&lt;/p&gt;
&lt;p&gt;In this environment, it&#39;s easy to spread fear and hopelessness. Pointing out small victories can feel wrong. It can feel like spreading fear and uncertainty is important. As if the most important thing we can be doing is worrying about everything wrong with the world.&lt;/p&gt;
&lt;p&gt;I feel this way too sometimes, and I witness the tendency a lot online. But, I&#39;d like to provide an alternative approach to our political era. Rather than focusing on the issues, we ought to spend our energies on solutions. Not just implementing solutions, but also on the solutions that have already been implemented. The victories we&#39;ve already won. We should look at how those victories happened.&lt;/p&gt;
&lt;p&gt;I know that it feels wrong to celebrate a small victory when so much else wrong is going on, but I promise you we won&#39;t be able to make it to the end if this is our outlook on life. If everything is terrible and nothing good ever happens or is talked about, then &lt;em&gt;why should we&lt;/em&gt; even fight for the next victory? If nothing good ever happens, why should we come together and stand in front of Flock headquarters, &lt;a href=&quot;https://www.eff.org/deeplinks/2026/07/victory-flock-ends-rollout-audio-distress-detection-human-voices&quot;&gt;demanding better from them&lt;/a&gt;? If nothing good ever happens, why should anyone fight for their privacy rights to be respected, from &lt;a href=&quot;https://www.eff.org/deeplinks/2026/06/victory-supreme-court-says-constitution-protects-peoples-location-data&quot;&gt;location data&lt;/a&gt;, &lt;a href=&quot;https://www.eff.org/deeplinks/2026/06/victory-meta-strips-facial-recognition-code-smart-glasses-app-after-public-outcry&quot;&gt;facial recognition surveillance&lt;/a&gt;, &lt;a href=&quot;https://www.eff.org/deeplinks/2026/07/california-steps-back-dangerous-expansion-its-age-gating-law&quot;&gt;age verification&lt;/a&gt;, or &lt;a href=&quot;https://www.eff.org/deeplinks/2026/06/get-flock-out-here&quot;&gt;otherwise&lt;/a&gt;? If nothing good ever happens, why should anyone show up to the polls to vote in a monumental victory for New York City&#39;s first ever Democratic Socialist mayor, against the power of incumbency and established parties?&lt;/p&gt;
&lt;p&gt;Clearly, good things happen, but as Hank Green often talks about, &lt;a href=&quot;https://www.youtube.com/watch?v=qVu2qnlQcvM&quot;&gt;good things don&#39;t naturally attract attention&lt;/a&gt;. Even worse, people often feel the need to spread negativity when something good does happen, because there&#39;s always something more that could happen.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&quot;http://eff.org/&quot;&gt;EFF&lt;/a&gt; is a non-profit that fights for digital rights around the world; like opposing surveillance and government censorship. Nearly every single time I see one of their blog posts shared around on social media where they celebrated a victory they&#39;ve made, there&#39;s always somebody in the comments complaining about how it didn&#39;t go far enough, that we still haven&#39;t done enough. And they&#39;re right in a sense — we &lt;em&gt;do&lt;/em&gt; still need to keep going, keep fighting. This victory alone is only a step along the path! But why do you feel it is important to share that right now? Why must we pollute every victory with some sense of additional fear and worry? Can we not celebrate a victory, understand its impacts, and say &amp;quot;wow, isn&#39;t it incredible what we can accomplish when we come together and organize? Let&#39;s keep going, use this victory to spur encouragement and belief in the power of collective action!&amp;quot;&lt;/p&gt;
&lt;p&gt;If you&#39;re in the comments explaining how we still haven&#39;t done enough, even after this victory: Yes you are technically correct, but you are doing serious damage to the very thing you want to achieve.&lt;/p&gt;
&lt;p&gt;In order to make progress, we need to get organized in collective action, and in order for people to be motivated to do that, they need to feel like victory is possible. When you sit in the comments and shit on everything good that ever happens, explaining how it&#39;s not enough, it makes people feel like success isn&#39;t possible.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;Look, in all honesty, I think the people bringing negativity are trying to do the right thing. They want people to realize how much work still needs to be done. They don&#39;t want people to feel comfortable enough to just give up. But it&#39;s not working. You are numbing people to the horrors, when what we should really be doing is numbing ourselves to pessimism. We need to wake up and realize that when we work together to get things done, we win.&lt;/p&gt;
&lt;p&gt;If we stick it through, if we don&#39;t sabotage one another, if we stand in solidarity with each other, we start to win. And each small victory starts to make our movement more and more powerful, until the victories stop being small.&lt;/p&gt;
&lt;p&gt;For decades, we&#39;ve been living in America with a cultural idea that collective organizing isn&#39;t important. That all the important issues are somehow going to be solved through individual actions and &amp;quot;&lt;a href=&quot;https://pluralistic.net/2025/09/13/consumption-choices/&quot;&gt;voting with your wallet&lt;/a&gt;.&amp;quot; If we build a culture that encourages ourselves when we achieve things that we fought for — instead of belittling ourselves as not having done enough — we can start to build real momentum behind our movements.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>So You Want To Join The Social Web</title>
    <link href="https://riverseeber.net/blog/post/so-you-want-to-join-the-social-web/" />
    <updated>2026-07-16T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/so-you-want-to-join-the-social-web/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 16, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;So, you&#39;ve been hearing more and more lately about a free and better alternative to commercial social media, called &amp;quot;the Open Social Web&amp;quot;, &amp;quot;the Social Web&amp;quot; or sometimes just &amp;quot;the Fediverse&amp;quot;. You want to join this better, less terrible social network, and see if it&#39;s any good. This blog post is a guide for you.&lt;/p&gt;
&lt;p&gt;I&#39;ll cover the basics of what the Social Web is and how it works, where to make an account, and what kind of stuff is even on this Social Web. I&#39;ll also provide links to various resources along the way.&lt;/p&gt;
&lt;h1&gt;What the heck is it, and why is it different from Facebook or Instagram?&lt;/h1&gt;
&lt;p&gt;&lt;img class=&quot;introImg&quot; src=&quot;https://ia600501.us.archive.org/30/items/social-web-graphic/Social%20Web%20graphic.jpg&quot; alt=&quot;A diagram depicting two situations. Top: A collection of users all collected around a big &amp;quot;Facebook&amp;quot; circle. Text says &amp;quot;Everyone has to go through Facebook to hang out&amp;quot; right arrow &amp;quot;Facebook has control over you&amp;quot;. Bottom: A collection of many circles, all of which are connected to each other. This graphic looks very similar to the logo for the fediverse. Each circle has a few users on it. Text says &amp;quot;You can communicate on the Social Web from any site you would like&amp;quot; right arrow &amp;quot;Leads to community ownership of the internet&amp;quot;&quot;&gt;
&lt;/p&gt;
&lt;p&gt;A lot of people like to give a technical answer to this question, but I don&#39;t think that&#39;s very helpful to do, so I&#39;ll explain why it&#39;s different for a regular user like you, not a computer programmer.&lt;/p&gt;
&lt;p&gt;Unlike Facebook, the Social Web is owned by the community, not just one group, organization, or billionaire. Moderation rules are set and enforced by the community. Each website on the social web (also called an &amp;quot;instance&amp;quot; or &amp;quot;node&amp;quot;) can decide on it&#39;s own rules, and anyone is free to start their own website if they don&#39;t like what the current offerings are.&lt;/p&gt;
&lt;p&gt;The Social Web is made up of many different websites, such as &lt;a href=&quot;https://mastodon.social&quot;&gt;mastodon.social&lt;/a&gt;, or &lt;a href=&quot;https://lgbtqia.space&quot;&gt;lgbtqia.space&lt;/a&gt;. Even though there&#39;s a lot of different sites, you don&#39;t really need to think about it, since you only need to make an account on just one site, and you can talk and interact with people who made account on &lt;em&gt;any other site&lt;/em&gt; on the Social Web. It&#39;s like how you have an iPhone and I have an Android, but we can still text each other even though they&#39;re different phones. You have an account on one site, I have an account on another, but we can still follow each other and reply to stuff.&lt;/p&gt;
&lt;p&gt;From your perspective, it won&#39;t feel like there&#39;s multiple sites at all, it will all behave just the same as on Facebook or Instagram where everyone uses the same website or app. The only difference being that the community has control over it, not a corporation with bad moderation.&lt;/p&gt;
&lt;h1&gt;What does it look like?&lt;/h1&gt;
&lt;p&gt;You might be wondering what the format is, does it emulate the feel of Reddit, Facebook, Instagram, or something else entirely? Well, as is often the case online, it depends which website you use on the Social Web. There&#39;s many different &amp;quot;types&amp;quot; of sites. The most popular type is &lt;a href=&quot;https://joinmastodon.org/&quot;&gt;Mastodon&lt;/a&gt; sites, which look a lot like Bluesky or Twitter. But there&#39;s other options too. &lt;a href=&quot;https://join-lemmy.org/&quot;&gt;Lemmy&lt;/a&gt; is a forum site type that is very similar to how Reddit works. There&#39;s also &lt;a href=&quot;https://pixelfed.org/&quot;&gt;Pixelfed&lt;/a&gt;, which is meant to be similar to Instagram. Finally, &lt;a href=&quot;https://friendi.ca/&quot;&gt;Friendica&lt;/a&gt; acts a lot like Facebook, although it&#39;s got some fancy features that allows it connect your account to more places than just the Social Web, so that might be confusing if you&#39;re a beginner user.&lt;/p&gt;
&lt;p&gt;Again, even though there are a lot of different types of sites on the social web, you can &lt;em&gt;usually&lt;/em&gt; interact with users who signed up on a different site type than you did. In fact, plenty of people do this without even realizing it. You just hit follow and you look at their posts. Everything behaves very intuitively.&lt;/p&gt;
&lt;h1&gt;Why&#39;s it better than commercial social media?&lt;/h1&gt;
&lt;p&gt;The Social Web isn&#39;t designed to make somebody rich. It&#39;s designed to work. It&#39;s understood to be infrastructure, like roads and telephone poles, but for online communication.&lt;/p&gt;
&lt;p&gt;There are no exploitative algorithms. It shows you everything that your friends posted in the order they posted it, and it doesn&#39;t show you things you didn&#39;t ask to see. There are explore features on most sites, but they&#39;re not designed to maximize engagement like the ones on most other apps.&lt;/p&gt;
&lt;p&gt;You repost or &amp;quot;boost&amp;quot; things that you think are important so that your friends can see them, rather than relying on an algorithm to deliver content.&lt;/p&gt;
&lt;h1&gt;How do I get started?&lt;/h1&gt;
&lt;p&gt;The quickest and easiest way to get on the Social Web is to just make an account on Mastodon. You can either pick the default site &lt;a href=&quot;https://joinmastodon.org/&quot;&gt;here&lt;/a&gt; to sign up, or if you&#39;d like to sign up with a specific community oriented around a hobby, identity, culture, or whatever else, you can check out &lt;a href=&quot;https://fedi.garden/&quot;&gt;this page&lt;/a&gt; to see if something appeals to you.&lt;/p&gt;
&lt;p&gt;There&#39;s also &lt;a href=&quot;https://fedi.tips/&quot;&gt;fedi.tips&lt;/a&gt;, which is a very helpful resource for answering any kind of questions you might have while getting started on the Social Web (fedi.tips refers to the Social Web as &amp;quot;the fediverse&amp;quot;).&lt;/p&gt;
&lt;h1&gt;What&#39;s the culture like?&lt;/h1&gt;
&lt;p&gt;It really does depend on who you follow, but it tends to have a lot of nerds right now, although you don&#39;t have to be one to join! Most people on the Social Web tend to value creativity, human expression, and social acceptance. The more people who get online, the more sub-communitities can form online, so the best thing you can do is just get started and go looking for things that interest you.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>A Reasonable Analysis of the Social Web</title>
    <link href="https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/" />
    <updated>2026-06-23T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;June 23, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;I think it&#39;s become clear to a lot of people that the internet is in a state of decay. Platforms keep us &lt;a href=&quot;https://pluralistic.net/2024/03/21/involuntary-die-hards/&quot;&gt;locked in&lt;/a&gt; as they slowly make the experience worse and worse. And yet we find we can&#39;t leave. We are stuck, because, as it turns out, the internet is a wildly useful tool. It is not an option to leave the internet entirely, yet everybody is stuck in the walled gardens of Google, Facebook, Instagram, and Twitter. You might on your own be able to look and find some better platform that doesn&#39;t pump algorithmic ragebait content on your feed every other post, but if none of your friends are on there, you might as well not be online at all. The point of the internet is to connect you to people. This is the reason walled gardens work. If you can&#39;t leave a platform without leaving your friends, you&#39;re probably not going to leave unless things get really bad.&lt;/p&gt;
&lt;p&gt;Well, I&#39;m here to tell you that things have gotten really bad online. The people are &lt;em&gt;dying&lt;/em&gt; for a way to get away from Big Tech social media, but they don&#39;t want to give up the utility that the platforms initially lured them in with. We want to get away from the trolls, algorithms, and bad moderation of Twitter, Facebook, and Instagram, but we don&#39;t want to have to go back to using old-school methods like email or IRC to talk to our contacts. There has to be a better way.&lt;/p&gt;
&lt;h1&gt;The new good internet&lt;/h1&gt;
&lt;p&gt;&lt;img class=&quot;introImg&quot; src=&quot;https://ia600501.us.archive.org/30/items/social-web-graphic/Social%20Web%20graphic.jpg&quot; alt=&quot;A diagram depicting two situations. Top: A collection of users all collected around a big &amp;quot;Facebook&amp;quot; circle. Text says &amp;quot;Everyone has to go through Facebook to hang out&amp;quot; right arrow &amp;quot;Facebook has control over you&amp;quot;. Bottom: A collection of many circles, all of which are connected to each other. This graphic looks very similar to the logo for the fediverse. Each circle has a few users on it. Text says &amp;quot;You can communicate on the Social Web from any site you would like&amp;quot; right arrow &amp;quot;Leads to community ownership of the internet&amp;quot;&quot;&gt;
&lt;/p&gt;
&lt;p&gt;I&#39;ve already &lt;a href=&quot;https://riverseeber.net/blog/post/big-tech-mutual-aid-and-the-social-web/&quot;&gt;made a post recently&lt;/a&gt; about a new movement online to create something called the &amp;quot;Social Web&amp;quot; (or more formally, the &amp;quot;&lt;em&gt;Open&lt;/em&gt; Social Web&amp;quot;)&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;, which is an idea where you can access social media by making an account on one of many possible sites. Then, from that site, you can interact with people across the entire network, including those who made their account on a different site than you. Since you can talk to users across different sites, it means you always have the option to change sites if the one you&#39;re on begins to &lt;a href=&quot;https://en.wikipedia.org/wiki/Enshittification&quot;&gt;enshittify&lt;/a&gt;. Thus, under this model, it will be functionally impossible for a site to become a walled garden. Everything is open, and anyone can prop up their own new site on the Social Web without needing to get permission from anybody else.&lt;/p&gt;
&lt;p&gt;I want to use this post to talk less about what the Social Web is, and more about how we (the nerds who are going to be building it, and also the nerds who are going to be early to adopting it) need to be acting in order to make sure this ship actually takes off so we can break Big Tech&#39;s walled garden. I&#39;d like to proceed with a number of questions we should be asking ourselves, whose answers need to dictate the direction we push design and community organization around this new online paradigm.&lt;/p&gt;
&lt;h2&gt;Who&#39;s going to be using it?&lt;/h2&gt;
&lt;p&gt;This is perhaps the single most important question you can ask when you&#39;re building something new, especially a digital technology. Who is this Social Web going to be for? If we look at who&#39;s using it currently, we see lots of software developers, computer geeks, and lots of other types of nerds. But surely that&#39;s not the only people who this tech is being designed for, is it? I would argue that a freer internet should be for everybody — not just nerds, and thus, the Social Web should be too.&lt;/p&gt;
&lt;p&gt;I think that most people involved in trying to make the Social Web happen would probably agree with me, but the way they act tends to not agree. If the Social Web isn&#39;t just for nerds, why are we saying stuff like this to newcomers:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Okay, so the fediverse&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/#fn2&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt; is like, a bunch of interconnected servers, all talking to each other. They all use the same open protocol, so the whole network is interoperable, and anybody can self-host their own new server whenever they want. You can get on the fediverse through any server. We call these servers &amp;quot;instances&amp;quot;. If you wanna join fedi, you can just go to &lt;a href=&quot;https://joinmastodon.com&quot;&gt;this website&lt;/a&gt; and pick an instance to sign up through. All the instances are referred to by names like &amp;quot;mastodon.social&amp;quot; because we like to use domain names for some reason.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Does this sound like a place where normal people are going to have a good time hanging out?! What are we doing, y&#39;all? This is simply too complicated for regular people to digest. It&#39;s not possible for normal people to understand what you&#39;re talking about when you say it like this.&lt;/p&gt;
&lt;p&gt;Anytime communication takes place between individuals, there is an unspoken tension between the speaker, who wants to rely on big words in order to more efficiently communicate their idea, and the audience, who only has a vague grasp on some of those bigger words. The speaker often knows more about their topic than the audience (which is why they&#39;re talking about it), and so are usually going to know the exact meaning of technical terms more than their audience. They&#39;re using these terms because they&#39;re really potent when speaking with &lt;em&gt;other experts&lt;/em&gt;. When speaking with a general audience, the speaker must use more everyday examples, lest their idea be lost to the words they try to use when expressing it.&lt;/p&gt;
&lt;p&gt;We &lt;em&gt;must&lt;/em&gt; keep this in mind when trying to sell the Social Web to people. Normal people don&#39;t know what &amp;quot;federated&amp;quot; means — they didn&#39;t watch Star Trek, they just think you&#39;re a cop. Normal people &lt;em&gt;really&lt;/em&gt; don&#39;t know what a protocol is, or what interoperability means. Stop talking like this. I also don&#39;t know why we&#39;re using the word &amp;quot;instance&amp;quot; to refer to servers. &amp;quot;Server&amp;quot; is literally easier to understand even for tech people. Even better than &amp;quot;server&amp;quot; would be &amp;quot;site,&amp;quot; which is the terminology that I tend to use. Here&#39;s a better explanation of the Social Web that actually communicates the idea to a normal person:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The Social Web is like social media away from Big Tech. Instead of being a walled garden where you&#39;re trapped, the Social Web is like a bunch of different social media websites connected together. You can pick any site to sign up with, and then you can follow people who signed up on &lt;em&gt;other&lt;/em&gt; sites too, so it kind of forms one big network, or online space. If you wanna sign up, you can make a Mastodon account (Mastodon is like, one of the categories/types of sites on the Social Web) at &lt;a href=&quot;https://joinmastodon.com&quot;&gt;this website&lt;/a&gt;. You can use the default option, or pick a different &amp;quot;instance&amp;quot; (site) if you want, too. It doesn&#39;t really matter which site you decide to sign up with, since you can always change your mind and move to a different one later.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Notice how the hardest parts for a regular person to understand were the parts where I had to translate the &lt;em&gt;terrible&lt;/em&gt; terminology that&#39;s commonly being used right now? If you think the first version is better, go ahead and read it to one of your friends who isn&#39;t into computer science, then ask them to explain it back to you. Then, try and read the second version and see how much better they understand it.&lt;/p&gt;
&lt;p&gt;There are probably technical fixes that could be made to improve the actual &lt;em&gt;experience&lt;/em&gt; of Mastodon and other Social Web services for a regular user, but those are engineering problems which are probably quite hard to figure out, and I don&#39;t have any specific complaints off the top of my head anyways, so I&#39;ll just leave that as a thing for people to consider whether its worth looking into or not. But one thing&#39;s for sure is that our communication needs to be improved &lt;em&gt;significantly&lt;/em&gt; before this thing can take off properly.&lt;/p&gt;
&lt;h2&gt;What does it do?&lt;/h2&gt;
&lt;p&gt;Next, I think it&#39;s time to consider what kind of services ought to exist on the Social Web. Is the Social Web just a place for social media (Twitter/Tumblr-style microblogging, Instagram-style image posts, etc), or should this model also expand to other types of online services? As much as we could say that Instagram has created a walled garden around social media, we surely could say the same of Microsoft Office and Google Drive when it comes to document collaboration, couldn&#39;t we? There are exciting things going on right now with Libre Office — the open source document editor — &lt;a href=&quot;https://www.howtogeek.com/dont-sleep-on-libreoffices-web-app-its-coming-back-and-this-time-it-might-actually-work/&quot;&gt;reviving their web service&lt;/a&gt;, and allowing any organization to host it. This could lead to good places, but for now, it still retains the problem that when you and another person are using Libre Office from different websites, they&#39;re unlikely to work with each other, unless they decide to implement a Social Web approach to the software, allowing different websites to communicate with each other. That way, I can collaborate on a document with a user on a different website, since both sites communicate to one another.&lt;/p&gt;
&lt;p&gt;This would be exactly the same thing as what happens on the existing parts of the Social Web, except instead of having users interact over text and image posts, they would interact over text documents and slideshows and such.&lt;/p&gt;
&lt;p&gt;I&#39;m not saying that this particular suggestion needs to be implemented right immediately now. The point of saying this is to point out that the Social Web idea can be extended beyond just what is thought of when we talk about &amp;quot;social media&amp;quot; services. The Social Web can free people from nearly every form of walled garden on the internet right now. Some of these services might not end up getting the same kind of popular support as the social media variants (very few people are passionate about online document collaboration), but the sites that show up could rely on an economic model somewhat similar to how Linux gets open source contributions — by providing an essential service, leading to corporations who use it contributing to the project, either financially, or by sending in bug fixes for the source code. It&#39;s not going to be a one-to-one matching, but I&#39;m sure whatever ends up happening will take some inspiration from the Linux model.&lt;/p&gt;
&lt;p&gt;The point is that this new good internet isn&#39;t just going to be on social media. This can be extended all across the internet. This can show up with blogging services (which provide an alternative to Substack and Medium). We could create something like StackOverflow/StackExchange on the Social Web, so that developers wouldn&#39;t need to rely on the crap moderation of the site owners in order to work on their projects and get help for their questions. We already have a Social Web built around online git repositories (although people don&#39;t often think of it as being on the Social Web, but you can use GitHub to make commits to a repo that&#39;s hosted on GitLab or Codeberg). If this really took off, it might eventually be possible to do streaming for movies and TV on the Social Web, although that gets complicated as you have to get permission from the movie studios in order to share the movie, but in theory, you might be able to make it at least a technical possibility. Same for ebooks and audiobooks, too.&lt;/p&gt;
&lt;p&gt;If we built an online feed reader on the Social Web for RSS and Atom feeds, we might be able to achieve network effects where enough people have an easy to access feed reader, so websites start putting a button that adds their site to your feed reader, leading to more people actually reading things online in long form instead of just scrolling social media for their news summaries (a thing that is all too common, and incredibly worrying as a trend).&lt;/p&gt;
&lt;p&gt;Heck, you might even be able to create a whole shopping site, which allows buyers to &lt;a href=&quot;https://pluralistic.net/2022/07/10/view-a-sku/&quot;&gt;get directly connected to sellers&lt;/a&gt;, skipping the whole middleman of Amazon, who scrapes off a fee on every single transaction on their site. If there were a free (or even just cheaper) network that just let customers buy things directly from sellers, our &lt;a href=&quot;https://pluralistic.net/2026/02/25/most-favored-nation/&quot;&gt;&lt;em&gt;whole economy&lt;/em&gt; would get cheaper&lt;/a&gt;. We&#39;d probably still have to develop some kind of system for escrow (some kind of assurance that a customer isn&#39;t getting scammed by a fake seller), but I have confidence that solutions could be found if we worked on it.&lt;/p&gt;
&lt;h2&gt;How do we advertise it?&lt;/h2&gt;
&lt;p&gt;Now, I don&#39;t want people to read this question in the sense of &lt;em&gt;corporate&lt;/em&gt; advertising. Think of it more like the way that the library will put up fliers about the various free services they offer. Those fliers are advertisements, but they&#39;re not trying to get you to buy anything. We should be doing the same. We&#39;re trying to build something really special, something to get people free of Big Tech. Well, that mission is going to fail if we aren&#39;t reaching out to people who have never heard of these options before. We need to reach out and actually get those regular folks into the fold.&lt;/p&gt;
&lt;p&gt;I&#39;m reminded of something written by &lt;em&gt;Dan Fixes Coin Ops&lt;/em&gt; &lt;a href=&quot;https://retro.social/@ifixcoinops/116712444755950963&quot;&gt;on Mastodon&lt;/a&gt; (&lt;a href=&quot;https://web.archive.org/web/20260615092859/https://retro.social/@ifixcoinops/116712444755950963&quot;&gt;archived version&lt;/a&gt;). He&#39;s describing the history of MP3 players and smartphones. Apple wasn&#39;t the first to invent these things, they were just the first to get them in the hands of regular people:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;See the iPhone was fucking shit compared to other smartphones at the time, like it was YEARS behind, but apple did that thing where, like, remember how they did it with MP3 players?  Instead of saying stuff like &amp;quot;four gigs of space, six hour battery life&amp;quot; they said &amp;quot;A thousand songs in your pocket and you don&#39;t have to fuck around with CDs,&amp;quot; while all the other MP3 player makers were fighting over people who already knew about MP3 players, apple targeted people who&#39;d never used an MP3 player before, and that&#39;s what they did with their smartphone.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;If our &amp;quot;advertising&amp;quot; is to talk about how cool it is to have decentralized platforms and federation and open protocols, we&#39;re fighting over a very small slice of the population (the slice who are probably more likely to discover this technology on their own anyways, if we&#39;re being honest). We should be trying to reach out to typical users of existing social media platforms. We need to promote the Social Web in a way that is accessible to regular people, or else we will fail, and the new good internet will not come to fruition.&lt;/p&gt;
&lt;h1&gt;Conclusion&lt;/h1&gt;
&lt;p&gt;I think there&#39;s a lot of cool things happening right now in the open source community that are pushing us towards a freer internet, but the Social Web is largely not ready yet for most regular users (otherwise, they would already be here). Thus, we need to focus on how to fix our communication so that regular people can understand it and want to join us! We also should be looking in to the future, and thinking about what kinds of things we are capable of creating, in order to further chip away at Big Tech&#39;s control over our lives.&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Some people like to call it &amp;quot;The Fediverse&amp;quot; or &amp;quot;Fedi&amp;quot;, but those terms are confusing and overly technical-sounding. You can read more in this &lt;a href=&quot;https://forbetter.ghost.io/the-normal-response-to-the-social-web/&quot;&gt;good blog post&lt;/a&gt;. &lt;a href=&quot;https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;This is the nerd term for the Social Web &lt;a href=&quot;https://riverseeber.net/blog/post/a-reasonable-analysis-of-the-social-web/#fnref2&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>Big Tech, Mutual Aid, and the Social Web</title>
    <link href="https://riverseeber.net/blog/post/big-tech-mutual-aid-and-the-social-web/" />
    <updated>2026-06-18T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/big-tech-mutual-aid-and-the-social-web/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;June 18, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;I&#39;ve been thinking a lot about Big Tech and the various ways people have proposed fighting back. I think most people would agree that what&#39;s been happening to the internet over the last few years has been egregious, and definitely crossed a line. I also think that in recent years, especially after the pandemic, we&#39;re all online a lot more than we used to be. That&#39;s not necessarily a bad thing — the internet is an easy place to have access to all kinds of educational information — but it does mean that we can no longer skirt by on the delusion that the politics of the internet don&#39;t matter.&lt;/p&gt;
&lt;p&gt;As we see traditional news and journalistic institutions being threatened by online platforms, we can&#39;t sit idly by when the infrastructure that&#39;s replacing it starts getting worse. I say this because it&#39;s easy to feel like the internet is just a silly thing we use to pass the time. It&#39;s not actually as important as &lt;em&gt;real&lt;/em&gt; problems, like bias in news media, or historic instances of the FBI spying on suspected political dissidents. And yet, the internet has become the new medium through which many people receive news, and it&#39;s an apparatus that has allowed the US government and mega-corporations to conduct leaps and bounds more intensive surveillance on their own citizens then even the dictators of old-fashioned regimes could&#39;ve ever dreamed of doing. Additionally, due to the bottleneck that has occurred with online platforms, in order to say something online, it has to be approved (or at least, not struck down) by Big Tech. If Facebook or YouTube has a problem with what you&#39;re posting, guess what? It&#39;s not going to be seen.&lt;/p&gt;
&lt;p&gt;Okay, enough justifying the &lt;em&gt;importance&lt;/em&gt; of these issues, let&#39;s actually discuss them, and also, let&#39;s talk about how I think some of us should rise to the moment.&lt;/p&gt;
&lt;div id=&quot;diagnosis&quot;&gt;
&lt;h1&gt;Diagnosis&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;So, I don&#39;t think it&#39;s really an isolated incident that basically everybody right now gets this feeling that just about everything online that they regularly interact with is kind of awful. Everything&#39;s extractive, provocative, intellectually hollow, or otherwise designed without the interests of actual people at heart. I&#39;m not going to be arguing that point forward, since it seems most people are at least vaguely on board with that feeling.&lt;/p&gt;
&lt;p&gt;What&#39;s less clear to folks, is where the root problem is, and what to tackle, if we want change? Some (knobheads) think the issue is that the internet is too free, and that we need to lock it down by making everybody upload their government ID&#39;s in order to go online. This is obviously stupid, but they&#39;re promoting it under the guise of &amp;quot;child safety&amp;quot;, so naturally, it&#39;s captured the imaginations of whole populations, and is being &lt;a href=&quot;https://www.eff.org/issues/age-verification&quot;&gt;rolled out in various countries&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Another approach — one that I actually respect quite a great deal — is that promoted by Cory Doctorow in his analysis of his theory of &lt;a href=&quot;https://en.wikipedia.org/wiki/Enshittification&quot;&gt;Enshittification&lt;/a&gt;. He posits that the problem with the internet lies with the consolidation of power — the way in which the entire internet was able to captured by a handful of monopolistic companies. He&#39;s absolutely right, and he calls for action at the policy level. Unfortunately, I&#39;m not a politician, so I only can do so much to push on that lever, although being aware of the issue is of course incredibly important. My thinking mirrors his somewhat, but it has a more community-centered focus. My analysis is compatible with his, but I&#39;m trying to emphasize different aspects and facts.&lt;/p&gt;
&lt;p&gt;My thinking is that the biggest issue driving the destruction of the internet right now is that we don&#39;t control the platforms that we communicate on. Perhaps that sounds silly to read. &amp;quot;Of course we don&#39;t control the platforms we use, that&#39;s why they&#39;re called platforms!&amp;quot; Well, of course, we as &lt;em&gt;individuals&lt;/em&gt; are never going to have total control over the entire platform we&#39;re on — we share them with other people who don&#39;t always see the same as us! That&#39;s the price of living in a society with other humans. But that&#39;s not usually how the power play works out in disagreements about social media. It&#39;s not usually that Alice thinks they should do one thing, while Bob thinks they should do a different thing. More often, in fact almost always, it is the case that some billionaire or trillionaire and a board of shareholders have decided to make a change that will enrich their pockets, while Alice, Bob, and every other user is along for the ride, as everything that was originally good about the platform slowly burns.&lt;/p&gt;
&lt;p&gt;I&#39;m of the opinion that there&#39;s a better way than this. I believe that we could collectively own the platforms we were on, so that they actually work &lt;em&gt;for&lt;/em&gt; us, not against us. And the great thing is that everyone really would have control and the ability to make changes &amp;quot;locally&amp;quot; to whatever online communities they&#39;re a part of. It doesn&#39;t have to be a machine too big for individuals to affect. And really, if we can get our heads straight and actually push this forward, there wouldn&#39;t be (much) that Big Tech could try to do to stop us from liberating ourselves from their walled garden.&lt;/p&gt;
&lt;div id=&quot;community-ownership&quot;&gt;
&lt;h1&gt;Community Ownership&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;The platforms we should be on ought to be owned by the community. We often think of online communities as being organized around a single platform, like Facebook, Twitter, or Instagram. We are organized in such a way that a single platform has nearly infinite control over community because it&#39;s the only place that it exists — the platform has a monopoly on our community, and so they can do whatever they want to us without feeling much punishment in return.&lt;/p&gt;
&lt;p&gt;But there&#39;s a strain of thought that&#39;s been picking up lately that promotes a different model. There&#39;s a lot of names for this, but I like calling it the &lt;em&gt;Social Web&lt;/em&gt;. The Social Web is kind of like the World Wide Web, in that it&#39;s not just one thing. It&#39;s made up of a bunch of different pieces all working together, where no single piece is so crucial that we couldn&#39;t get along if it broke.&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/big-tech-mutual-aid-and-the-social-web/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;&lt;img class=&quot;introImg&quot; src=&quot;https://ia600501.us.archive.org/30/items/social-web-graphic/Social%20Web%20graphic.jpg&quot;&gt;&lt;/p&gt;
&lt;p&gt;The idea of the Social Web isn&#39;t just to spread everything out so thin that there&#39;s nobody to talk to though. The idea is that from any one place on the Social Web, you can talk to &lt;em&gt;anybody else&lt;/em&gt;, even people who are on &lt;em&gt;different parts&lt;/em&gt; of the social web than you. Beyond just that, if a group of people is ever unhappy about the current offerings of sites where you can talk to your friends (perhaps they don&#39;t like the way moderation is being handled, for instance), they can get together and just make their own little space on network. Then they and anyone else who shares their opinions can hangout together under the same policies. Those who disagree can still talk to them, but from a different place on the network.&lt;/p&gt;
&lt;p&gt;It&#39;s kind of like the Social Web is like Europe, and each little site is a country inside of it, which has its own rules, but each one still works together and interacts with the whole. Sometimes, a user from one site might even migrate to another site because they prefer something about that new place more — but they of course don&#39;t have to leave their old friends, since they can still communicate with anyone on the network.&lt;/p&gt;
&lt;p&gt;Currently, the best way to enjoy the Social Web is on &lt;a href=&quot;https://joinmastodon.org/&quot;&gt;Mastodon&lt;/a&gt;. Pick a site you wanna sign up through, then hangout. It&#39;s like Twitter, but with less character limits, and no algorithm. I like it there, but it tends to be much less populated than regular social media apps. It&#39;s also mostly made up of nerdy computer scientists who like to use words like &amp;quot;protocol&amp;quot;, &amp;quot;instance&amp;quot;, &amp;quot;ActivityPub&amp;quot;, and &amp;quot;fediverse&amp;quot; — which is somewhat problematic, if we want to actually get this stuff adopted by the world.&lt;/p&gt;
&lt;div id=&quot;mutual-aid&quot;&gt;
&lt;h1&gt;Mutual aid, and independent online communities&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;I think that if we actually want to rescue our society, we need to lean into the Social Web concept. We need to own our communication. We can&#39;t allow some company to shove harmful algorithms down our throats. In reality, I think governments should put up funding for various groups to host sites on the Social Web (they&#39;re very cheap to run, it wouldn&#39;t be expensive to fund this kind of thing). Government is expected to fund all kind of infrastructure like roads and telephone lines. Why shouldn&#39;t they also fund the infrastructure we use to organize and communicate together online?&lt;/p&gt;
&lt;p&gt;Unfortunately, I don&#39;t think in the US government is going to be picking up that project anytime soon. Short of that, I think it falls down to the community to run these sites. Think of it like mutual aid. We don&#39;t want to see our neighbors and friends and family getting stuck watching mind-numbing AI slop that&#39;s fed to them by an algorithm that hates them. I don&#39;t want to be stuck spending my time interacting with the algorithmic feed, but I&#39;m stuck on it cuz that&#39;s the only place that works for talking to my friends. We need to band together and find ways to get our non-technical friends onto the Social Web, and away from corporate social media.&lt;/p&gt;
&lt;p&gt;As good as Mastodon and some of these places are, it really could be a lot better. The only people who are on Mastodon are people who already don&#39;t like existing social media, and who are technical enough to try to figure it out. That&#39;s an issue, because a lot of people are not very technical, but really deserve to have an escape from corporate social media just as much as the computer science nerds. Unfortunately, it&#39;s the nerds who are building these apps, and who are hosting the sites. I&#39;m one of these nerds, so I&#39;m deeply aware of how blind this community is to the needs and skills of typical internet users. Those of us who are technical enough to building the software for the Social Web need to be more keen towards the needs of typical users, because those are usually the kinds of people you want to hangout with online.&lt;/p&gt;
&lt;div id=&quot;walled-gardens&quot;&gt;
&lt;h1&gt;Walled gardens&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;A &amp;quot;walled garden&amp;quot; in software describes a situation where some company provides a service to users, but those users are forced to interact with that service only in ways that the company allows them too. Furthermore, since we&#39;re talking about the internet, the company can change the rules on everybody overnight, after lots of people have already joined in. For instance, Facebook starts up in 2004, provides a social media service to a bunch of people, and promises to never be shitty to its users. A bunch of people join, and eventually, Facebook has practically captured an entire generation. Then they decide to go be shitty like they said they wouldn&#39;t do. Nobody on there is able to disable the bad features. Your only option is to leave all your friends, or just put up with it. You have no control in this situation, and Facebook has all the control.&lt;/p&gt;
&lt;p&gt;In silicon valley, investors call this &lt;em&gt;Software-as-a-Service&lt;/em&gt;, or SaaS. It&#39;s a situation where a company creates a server that you have to connect to in order to get what you want. Business people love this, because it means that we&#39;re all entirely beholden to them, and they can change the rules on us whenever they want, and we can&#39;t do anything about it. Every new tech startup in the past decade or more has operated under this model. It&#39;s what social media is, it&#39;s what cloud hosting is, it&#39;s how &amp;quot;smart&amp;quot; fridges and toasters and washing machines all work. It&#39;s how streaming music and movies and audiobooks works. Every AI company is using this model as well, and it works spectacularly for that industry, because of just how hard it is to run AI software on most hardware.&lt;/p&gt;
&lt;p&gt;Not that AI is relevant to this post, though. What we should be focusing on is how walled gardens operate. Walled gardens are the result of a Software as a Service business model. If we want to resist Big Tech, we need to be finding ways to flip that business model on its head. Sometimes, we can do that by providing software that lets people do all the computation locally on their own computer for some given task. But a lot of tasks still require access to the internet in order to work. This is the more complicated side. How do we build our own services that free people from Big Tech, without just building new walled gardens of our own?&lt;/p&gt;
&lt;p&gt;Well, the important thing is that your service needs to be open. Other services should be able to interact with it (nerds like to refer to this as &lt;a href=&quot;https://www.eff.org/deeplinks/2019/10/adversarial-interoperability&quot;&gt;interoperability&lt;/a&gt;), allowing for users to escape the garden, while still participating. It should be possible for anyone to spin up their own site on the Social Web that plugs into the existing sites, and lets somebody look at posts made by users on other sites.&lt;/p&gt;
&lt;div id=&quot;conclusion&quot;&gt;
&lt;h1&gt;Conclusion&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;The point of what we should be building is something that is lightweight, and which can easily be shared across a community. Whatever we make to add to the Social Web, it needs to be easy for an individual (hopefully one who is only mildly technical) to host themselves for their community. That way, we can have a large number of different sites operating on the overall network. More nodes (sites) means a stronger network, and less prone to Big Tech coming in and trying to break something.&lt;/p&gt;
&lt;p&gt;One of the biggest issues with the Social Web right now is how awful it is for non-technical users to understand. Half of that&#39;s a communication thing. For instance, almost none of the language I&#39;ve used in this blog post is the language that most Social Web users use. They like to use the language of &amp;quot;open protocols&amp;quot; and &amp;quot;federated&amp;quot; or &amp;quot;distributed&amp;quot; networks. This is great for them, but it&#39;s not how you start a grassroots movement to push away from Big Tech. The other half of the problem is that it&#39;s genuinely just hard from a software point of view. Instead of one site (&amp;quot;go to facebook dot com!&amp;quot;), there&#39;s now a bunch of sites, and you have to pick one, even though you&#39;ve got no experience about which ones are better, or what criteria you should even be picking based off of. I don&#39;t know what the solution to that second problem is, but I&#39;m sure it&#39;s something that can be figured out if we apply ourselves. On the other hand, the communication problem is easy to solve. People need to stop trying to sell the Social Web using fancy computer science terms. Nobody wants to join your nerd revolution.&lt;/p&gt;
&lt;p&gt;People are &lt;em&gt;yearning&lt;/em&gt; for a way to escape Big Tech. We cannot afford to waste this golden opportunity of the current moment by being pedantic. We need to invite our less-technical friends into the Social Web without making them feel unwelcome by overloading them with technical jargon. Give them the basics, enough for a regular person to understand, and then help them make an account. This stuff can&#39;t be demonstrated as being confusing, or else nobody will want to join us as we push away from Big Tech.&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Please save your jokes about how &amp;quot;that version of the Web died with US-East-1&amp;quot; for the end of the blog. &lt;a href=&quot;https://riverseeber.net/blog/post/big-tech-mutual-aid-and-the-social-web/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>An Exploration into Reinforcement Learning</title>
    <link href="https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/" />
    <updated>2026-06-07T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;June 7, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;I&#39;ve been looking into and learning a lot about Reinforcement Learning lately — which is a method of Machine Learning which doesn&#39;t rely on the copying of original human work. Modern &amp;quot;AI&amp;quot; systems like LLMs as well as the various image/video generation tools both rely on a Machine Learning technique known as &amp;quot;Structured Learning&amp;quot;. In Structured Learning, the agent is provided with a (usually quite large) corpus of labeled data: a set of inputs (like prompts, for example), and the outputs the machine is expected to produce in that scenario (such as a response, or an image).&lt;/p&gt;
&lt;p&gt;Structured Learning is entirely dependent upon the sum of all that is in order to produce results, just by its very nature. These types of machines can never engage in a single act of creation. They can never make something new, or discover something that they weren&#39;t directly shown during training. As it happens, they&#39;re also quite bad at learning in general. They shut their minds off entirely once they&#39;re no longer in training. Once deployed, they cannot learn. In the case of an LLM, it might be able to simulate learning by retaining something you told it in its context window, but once the conversation gets long enough, that info will be dropped, and it never gets integrated into the model itself. It has no ability to learn new things in a long-term way once its no longer training.&lt;/p&gt;
&lt;p&gt;Contrast that with the way Reinforcement Learning (RL) works. An RL agent has no substantive distinction between training and deployment. Sure, you can go in and artificially disable learning after you feel you&#39;ve &amp;quot;got it right&amp;quot; (especially if you&#39;re deploying into a high risk application and don&#39;t want it to act unpredictably down the road), but it&#39;s not the natural state of things for it to do that. RL agents are always learning, and they learn through &lt;em&gt;experience&lt;/em&gt;, not by being fed a list of &amp;quot;right&amp;quot; and &amp;quot;wrong&amp;quot; answers.&lt;/p&gt;
&lt;p&gt;This simple fact seems (to me) like a very good argument for saying that Reinforcement Learning is the only way forward for AI systems. I also happen to feel as though this kind of system would be much more of an ethical technology than the current fad of AI tech being sold to us right now. For one thing, it wouldn&#39;t require Big Tech companies to collect all kinds of data on us as a means to train them. Additionally, the machine wouldn&#39;t be regurgitating remixed versions of the creative outputs of artists, authors, musicians and the like. It&#39;s not a technology that suited to mimicry. Reinforcement Learning AIs aren&#39;t really capable of becoming plagiarism bots, at least in the sense that it wouldn&#39;t be reproducing the work of others without providing citations.&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt; It&#39;s also probably not going to be very good at language for quite a long time. After all, language is merely the expression of a number of deep, very complicated intellectual processes that are going on inside us. It&#39;s a tool for communicating those intellectual concepts to one another. LLMs are only able to get away with language because they took the easy way out. They skip the thinking, and go straight to reproducing words in orders that seem right based on their training data. LLMs are not aware of the words they&#39;re typing. It&#39;s the &lt;a href=&quot;https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thought_experiment&quot;&gt;Chinese Room thought experiment&lt;/a&gt;. LLMs are given a set of tokens (numbers), which correspond to words or half-words. Your prompt is a string of numbers to it, and it responds with the string of numbers back that seems right.&lt;/p&gt;
&lt;p&gt;If a Reinforcement Learning agent were to perform language, it would not do so like this. It would learn language as a tool for communication regarding whatever task was at hand. It would learn to communicate in the same way that biological organisms in nature did.&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn2&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt; The agents would exist in the world, and then slowly develop an ability to communicate threats or describe resources. Over time, the language would become more sophisticated. Having it learn a human language like English or what have you might be more difficult of a task, although I don&#39;t see any reason why it couldn&#39;t be persuaded to use noises humans understand instead of whatever medium it would have done otherwise.&lt;/p&gt;
&lt;p&gt;I hope so far that I&#39;ve convinced you of the ways that Reinforcement Learning is uniquely separate from what we&#39;ve lately come to understand &amp;quot;AI&amp;quot; to be. We think of AI today to be lazy, plagaristic slop-bots, that steal work from real humans, memorizing data in their weights. We also think of it as being unoriginal, and incapable of dealing with a novel problem in a novel way. It is only able to solve new problems by applying old methods. This is all true of the short-sighted approach that current offerings have taken. But there is a whole field of Machine Learning which seeks to produce machines that are not like this.&lt;/p&gt;
&lt;p&gt;I would also like to say right here up front that Reinforcement Learning doesn&#39;t seem to me (a very much non-expert) to be well-suited to the task of doing difficult things like writing essays or screenplays or other tasks that our bosses seem to be frothing at the mouth at, in hopes of firing all of the worlds workforce. I (again, just an undergraduate college student) feel like this kind of tech would be way better as a tool for handling truly boring, data-driven tasks. It might one day become capable enough to maybe perform tasks that require certain amounts of very basic intuition, but even that seems to be a stretch, given the enormous computational constraints we&#39;re already starting to discover with LLMs. I don&#39;t think a sober economy is going to be willing to put up a massive data center in every city in America to fund a technology that isn&#39;t sellable as &amp;quot;we&#39;re building god&amp;quot;. In any case, I do find this field to be really interesting. I think it would be fun to make some little digital robot creatures that could actually do interesting stuff, instead of just kind of ruining the internet.&lt;/p&gt;
&lt;div id=&quot;so_how_does_it_work&quot;&gt;
&lt;h1&gt;So how does it work?&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;Well, that was a long winded way for me to introduce the concept, and defend it against some (possibly quite reasonable) ethical concerns. I hope I&#39;ve done it justice. In this section, I&#39;d like to go over a bit of how it actually works. Fair warning, there&#39;s gonna be a bit of computer math involved, but I hope you&#39;ll be able to push through it — I find this stuff really quite interesting.&lt;/p&gt;
&lt;p&gt;A lot of my knowledge on this topic comes from Richard Sutton&#39;s &lt;em&gt;Reinforcement Learning: An Introduction&lt;/em&gt; textbook, which has a &lt;a href=&quot;https://github.com/MrinmoiHossain/Reinforcement-Learning-Specialization-Coursera/blob/master/Book/Reinforcement%20Learning%20An%20introduction%20(Second%20Edition)%20by%20Richard%20S.%20Sutton%20and%20Andrew%20G.%20Barto.pdf&quot;&gt;free PDF&lt;/a&gt; you can legally share with your friends, thanks to being licensed under CC BY-NC-ND 2.0. I hope that this blog post is much more to the point than the textbook, with less technical detours. I&#39;m no expert, so take everything with a grain of salt, but all of what I&#39;m talking about I have successfully implemented for a simple agent (which I discuss at the end).&lt;/p&gt;
&lt;div id=&quot;guessing_rewards&quot;&gt;
&lt;h2&gt;Guessing rewards&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;The basic principle of Reinforcement Learning (in my opinion) is an algorithm that tries to guess the reward it will receive for taking various actions available to it. Rewards are expressed as a numerical value. The most intuitive way to try to guess rewards is to just keep track of an average of rewards you&#39;ve received for each action. Use the past as a way to try to predict the future. If we define a &lt;em&gt;Quality function&lt;/em&gt; &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; to represent our predicted reward we will receive for taking some action &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, we can express it as:&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mtext&gt;(sum of rewards)&lt;/mtext&gt;&lt;mtext&gt;(# of attempts)&lt;/mtext&gt;&lt;/mfrac&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;munderover&gt;&lt;mo&gt;∑&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;/munderover&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;/mstyle&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;/mfrac&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a) = &#92;frac{&#92;text{(sum of rewards)}}{&#92;text{(&#92;# of attempts)}} = &#92;frac{&#92;displaystyle&#92;sum_{i=1}^{N} R_i}{N},&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1.53em;vertical-align:-0.52em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.01em;&quot;&gt;&lt;span style=&quot;top:-2.655em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;(# of attempts)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.485em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;(sum of rewards)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.52em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:3.761em;vertical-align:-0.345em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:3.416em;&quot;&gt;&lt;span style=&quot;top:-3.4833em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.8283em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-4.0583em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.8283em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-5.416em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.8283em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mop op-limits sizing reset-size3 size6&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.8283em;&quot;&gt;&lt;span style=&quot;top:-1.8723em;margin-left:0em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.05em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;mrel mtight&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.05em;&quot;&gt;&lt;/span&gt;&lt;span&gt;&lt;span class=&quot;mop op-symbol large-op&quot;&gt;∑&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-4.3em;margin-left:0em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.05em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.2777em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord sizing reset-size3 size6&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.345em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R_i&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is the reward received at some step &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;i&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6595em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;However, there is a slightly easier way to handle calculating a rolling average. When taking one data point &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R_i&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; at a time, and trying to calculate the average &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; after some number of data points &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;n&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;n&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, you can use the following assignment&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn3&quot; id=&quot;fnref3&quot;&gt;[3]&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mtext&gt;new&lt;/mtext&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mtext&gt;old&lt;/mtext&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mtext&gt;old&lt;/mtext&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q_&#92;text{new}(a) = Q_&#92;text{old}(a) + &#92;frac{1}{n+1} [R_i - Q_&#92;text{old}(a)]
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.1514em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;new&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3361em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;old&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:2.0908em;vertical-align:-0.7693em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.3214em;&quot;&gt;&lt;span style=&quot;top:-2.314em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.677em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.7693em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3361em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;old&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;Where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;n&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;n&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is the number of datapoints included in &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mtext&gt;old&lt;/mtext&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q_&#92;text{old}(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3361em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;old&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. You&#39;ll notice that if there have been no datapoints yet (i.e., &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;n=0&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6444em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;), then the value of &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mtext&gt;old&lt;/mtext&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q_&#92;text{old}(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3361em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;old&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; has no impact on &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mtext&gt;new&lt;/mtext&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q_&#92;text{new}(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.1514em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord text mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;new&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. In the literature, this assignment is often rewritten further as&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn4&quot; id=&quot;fnref4&quot;&gt;[4]&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi&gt;α&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a) &#92;leftarrow Q(a) + &#92;alpha [R_i - Q(a)]
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0037em;&quot;&gt;α&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;With the assumption&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn5&quot; id=&quot;fnref5&quot;&gt;[5]&lt;/a&gt;&lt;/sup&gt; that &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;α&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;alpha = &#92;frac{1}{n+1}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0037em;&quot;&gt;α&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1.2484em;vertical-align:-0.4033em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8451em;&quot;&gt;&lt;span style=&quot;top:-2.655em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.394em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.4033em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. As it would happen, this single, simple seeming equation actually forms basically the entire basis of Reinforcement Learning. We still have no mechanism for actually &lt;em&gt;picking&lt;/em&gt; actions, but this &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8778em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; function is our way of &lt;em&gt;ranking&lt;/em&gt; actions against each other. More complexity is eventually introduced to this function in order to account for cause-and-effect, but for now, we can operate with just this.&lt;/p&gt;
&lt;div id=&quot;example_k-armed_bandit&quot;&gt;
&lt;h3&gt;Example: K-Armed Bandit&lt;/h3&gt;
&lt;/div&gt;
&lt;p&gt;Suppose you&#39;re a Reinforcement Learning agent. In front of you are a number of slot machines (&amp;quot;one-armed bandits&amp;quot;). Each slot machine returns a random value centered on a given (unknown) setpoint. Some machines will have higher setpoints than others. You want to choose a course of action such that you maximize your long-term yield from the machines after some number of turns. You can only pull the arm of one machine at a time. We refer to this collection of machines as a &amp;quot;K-armed bandit&amp;quot;, where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;k&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0315em;&quot;&gt;k&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; represents the number of machines total. Basically, the strategy you want to employ is to discover the machine with the highest setpoint, so as to maximize your expected yield.&lt;/p&gt;
&lt;p&gt;Let&#39;s ignore for now &lt;em&gt;how&lt;/em&gt; you would pick an action, and instead just focus on how you might keep track of your &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; function.&lt;/p&gt;
&lt;p&gt;You start by initializing all your &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8778em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; values at any arbitrary value (it will be overwritten soon, since &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;n=0&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6444em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;). Then, when you pull on some machine &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mo&gt;…&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a=0,1,2,&#92;ldots&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8389em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;…&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; you update your quality function by including that reward &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6833em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; from action &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; in your average for &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. Here&#39;s how that would look as an algorithm:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mtable rowspacing=&quot;0.25em&quot; columnalign=&quot;right left&quot; columnspacing=&quot;0em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;//initialize values&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;for &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mo&gt;…&lt;/mo&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;mo&gt;:&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;//start playing&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;loop:&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mtext&gt;(pick some action)&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mtext&gt;takeAction_getReward&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//increment &lt;/mtext&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mtext&gt; for this action&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//update the new running average of rewards for &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;begin{aligned}
	&amp;amp;&#92;text{//initialize values}&#92;newline
	&amp;amp;&#92;text{for }a = 0, 1, 2, &#92;ldots, k: &#92;newline
	&amp;amp;&#92;quad Q(a) &#92;leftarrow 0&#92;newline
	&amp;amp;&#92;quad N(a) &#92;leftarrow 0&#92;newline
	&amp;amp;&#92;text{//start playing}&#92;newline
	&amp;amp;&#92;text{loop:}&#92;newline
	&amp;amp;&#92;quad	a &#92;leftarrow &#92;text{(pick some action)}&#92;newline
	&amp;amp;&#92;quad	R &#92;leftarrow &#92;text{takeAction&#92;_getReward}(a)&#92;newline
	&amp;amp;&#92;quad	&#92;text{//increment }N&#92;text{ for this action}&#92;newline
	&amp;amp;&#92;quad	N(a) &#92;leftarrow N(a) + 1&#92;newline
	&amp;amp;&#92;quad	&#92;text{//update the new running average of rewards for }a&#92;newline
	&amp;amp;&#92;quad	Q(a) &#92;leftarrow Q(a) + &#92;frac{1}{N(a)}[R - Q(a)]
&#92;end{aligned}
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:18.7574em;vertical-align:-9.1287em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-r&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:9.6287em;&quot;&gt;&lt;span style=&quot;top:-12.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-10.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-9.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-7.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-6.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-4.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-1.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-0.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:1.3898em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:2.8898em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:4.8713em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:9.1287em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:9.6287em;&quot;&gt;&lt;span style=&quot;top:-12.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//initialize values&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-10.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;for &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;…&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0315em;&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;:&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-9.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-7.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-6.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//start playing&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-4.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;loop:&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;(pick some action)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-1.6102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;takeAction_getReward&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-0.1102em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//increment &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt; for this action&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:1.3898em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:2.8898em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//update the new running average of rewards for &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:4.8713em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.3214em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.3214em;&quot;&gt;&lt;span style=&quot;top:-2.314em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.677em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.936em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:9.1287em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;br&gt;
&lt;div id=&quot;picking_an_action&quot;&gt;
&lt;h2&gt;Picking an action&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;The only thing we really have to do after getting an up-to-date guess about the value of each action &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is to determine which one to pick. In most circumstances, we should just pick whatever is highest. In the literature, they refer to this as &amp;quot;exploiting&amp;quot; your prior knowledge/experience, and it&#39;s a good thing to do. However, if you exploit at all times even when you&#39;ve only just started out, you run the risk of settling into a sub-optimal routine. Something that&#39;s good, but not the best possible.&lt;/p&gt;
&lt;p&gt;Better than this would be to exploit &lt;em&gt;most&lt;/em&gt; of the time, and then picking a random action with some very small probability &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;varepsilon&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;0&amp;lt;&#92;varepsilon&amp;lt;1&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6835em;vertical-align:-0.0391em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.5782em;vertical-align:-0.0391em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6444em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. This taking of a random action is called &amp;quot;exploring&amp;quot;. We can then pick our action based on the following:&lt;/p&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mtext&gt;action&lt;/mtext&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo fence=&quot;true&quot;&gt;{&lt;/mo&gt;&lt;mtable rowspacing=&quot;0.36em&quot; columnalign=&quot;left left&quot; columnspacing=&quot;1em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;mtext&gt;(Exploit), with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;mtext&gt;(Explore), with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{action} = 
&#92;begin{cases}
	&#92;text{(Exploit), with } p = 1-&#92;varepsilon	&#92;newline
	&#92;text{(Explore), with } p = &#92;varepsilon
&#92;end{cases}
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6679em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;action&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:3em;vertical-align:-1.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;&lt;span class=&quot;mopen delimcenter&quot; style=&quot;top:0em;&quot;&gt;&lt;span class=&quot;delimsizing size4&quot;&gt;{&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.69em;&quot;&gt;&lt;span style=&quot;top:-3.69em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;(Exploit), with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-2.25em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;(Explore), with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.19em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;So that within our code, we can assign our action as:&lt;/p&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mtext&gt;action&lt;/mtext&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mrow&gt;&lt;mo fence=&quot;true&quot;&gt;{&lt;/mo&gt;&lt;mtable rowspacing=&quot;0.36em&quot; columnalign=&quot;left left&quot; columnspacing=&quot;1em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mtext&gt;argmax&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mtext&gt;, with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;mtext&gt;random(A), with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{action} &#92;leftarrow 
&#92;begin{cases}
	&#92;text{argmax}_a Q(a)&#92;text{, with } p = 1-&#92;varepsilon	&#92;newline
	&#92;text{random(A), with } p = &#92;varepsilon
&#92;end{cases}
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6679em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;action&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:3em;vertical-align:-1.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;&lt;span class=&quot;mopen delimcenter&quot; style=&quot;top:0em;&quot;&gt;&lt;span class=&quot;delimsizing size4&quot;&gt;{&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.69em;&quot;&gt;&lt;span style=&quot;top:-3.69em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;argmax&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.0573em;&quot;&gt;&lt;span style=&quot;top:-2.4559em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2441em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;, with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-2.25em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;random(A), with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.19em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;Where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mtext&gt;argmax&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{argmax}_a Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;argmax&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.0573em;&quot;&gt;&lt;span style=&quot;top:-2.4559em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2441em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; simply returns the &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; such that &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; has the largest value. If &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8778em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is an array, and &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is the index corresponding to the value, then &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mtext&gt;argmax&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{argmax}_a Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;argmax&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.0573em;&quot;&gt;&lt;span style=&quot;top:-2.4559em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2441em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; just returns the index of the largest value in the array. Additionally, &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mtext&gt;random(A)&lt;/mtext&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{random(A)}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;random(A)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; returns a random action in the set of all possible actions, &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;A&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;A&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6833em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;A&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. So to update our k-armed bandit algorithm, we can express it as:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mtable rowspacing=&quot;0.25em&quot; columnalign=&quot;right left&quot; columnspacing=&quot;0em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;//initialize values&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;for &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mo&gt;…&lt;/mo&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;mo&gt;:&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;//start playing&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;loop:&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mrow&gt;&lt;mo fence=&quot;true&quot;&gt;{&lt;/mo&gt;&lt;mtable rowspacing=&quot;0.36em&quot; columnalign=&quot;left left&quot; columnspacing=&quot;1em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mtext&gt;argmax&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mtext&gt;, with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;mtext&gt;random(A), with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mtext&gt;takeAction_getReward&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//increment &lt;/mtext&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mtext&gt; for this action&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//update the new running average of rewards for &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;begin{aligned}
	&amp;amp;&#92;text{//initialize values}&#92;newline
	&amp;amp;&#92;text{for }a = 0, 1, 2, &#92;ldots, k: &#92;newline
	&amp;amp;&#92;quad Q(a) &#92;leftarrow 0&#92;newline
	&amp;amp;&#92;quad N(a) &#92;leftarrow 0&#92;newline
	&amp;amp;&#92;text{//start playing}&#92;newline
	&amp;amp;&#92;text{loop:}&#92;newline
	&amp;amp;&#92;quad a &#92;leftarrow 
		&#92;begin{cases}
			&#92;text{argmax}_a Q(a)&#92;text{, with } p = 1-&#92;varepsilon	&#92;newline
			&#92;text{random(A), with } p = &#92;varepsilon
		&#92;end{cases}&#92;newline
	&amp;amp;&#92;quad	R &#92;leftarrow &#92;text{takeAction&#92;_getReward}(a)&#92;newline
	&amp;amp;&#92;quad	&#92;text{//increment }N&#92;text{ for this action}&#92;newline
	&amp;amp;&#92;quad	N(a) &#92;leftarrow N(a) + 1&#92;newline
	&amp;amp;&#92;quad	&#92;text{//update the new running average of rewards for }a&#92;newline
	&amp;amp;&#92;quad	Q(a) &#92;leftarrow Q(a) + &#92;frac{1}{N(a)}[R - Q(a)]
&#92;end{aligned}
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:20.5575em;vertical-align:-10.0287em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-r&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:10.5287em;&quot;&gt;&lt;span style=&quot;top:-13.4387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-11.9387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-10.4387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-8.9387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-7.4387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-5.9387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.5287em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-1.1387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:0.3613em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:1.8613em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:3.3613em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:5.3427em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:10.0287em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:10.5287em;&quot;&gt;&lt;span style=&quot;top:-13.4387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//initialize values&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-11.9387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;for &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;…&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0315em;&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;:&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-10.4387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-8.9387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-7.4387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//start playing&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-5.9387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;loop:&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.5287em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;&lt;span class=&quot;mopen delimcenter&quot; style=&quot;top:0em;&quot;&gt;&lt;span class=&quot;delimsizing size4&quot;&gt;{&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.69em;&quot;&gt;&lt;span style=&quot;top:-3.69em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;argmax&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.0573em;&quot;&gt;&lt;span style=&quot;top:-2.4559em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2441em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;, with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-2.25em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;random(A), with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.19em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-1.1387em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;takeAction_getReward&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:0.3613em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//increment &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt; for this action&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:1.8613em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:3.3613em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//update the new running average of rewards for &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:5.3427em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.3214em;&quot;&gt;&lt;span style=&quot;top:-2.314em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.677em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.936em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:10.0287em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;You&#39;ll notice that the only undefined function now is the &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mtext&gt;takeAction_getReward&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{takeAction&#92;_getReward}(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1.06em;vertical-align:-0.31em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;takeAction_getReward&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; function. This function will be wired directly into the agent&#39;s external environment. It is responsible for making the agent actually pull the levers. In more complicated applications, it may be the function responsible for actually sending electrical signals to the hardware of a robot, or for sending APIs in order to take actions in an online chess or poker game.&lt;/p&gt;
&lt;div id=&quot;rewards&quot;&gt;
&lt;h2&gt;Rewards&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;With what we have already, we&#39;ve basically already defined our agent completely. However, outside of the agent, we do still need to define how our rewards will be given to the agent. It&#39;s important to realize about these rewards is that they aren&#39;t just helpful clues for the agent. The reward signal is, in fact, the sole thing that the agent will ever care about. Hook up the rewards to something you want it to do, and you&#39;ll be pleased with the results. Hook it up to something that it can find a loophole to exploit without accomplishing your task, and you will find yourself disappointed. I feel I&#39;ve already over extended myself in terms of technical depth in this post, so I won&#39;t be giving any specific pointers on how to choose reward drops, but take comfort in the fact that this has hardly anything to do with difficult math, which is a nice contrast compared to the previous two sections.&lt;/p&gt;
&lt;div id=&quot;gridworld_proof_of_concept&quot;&gt;
&lt;h1&gt;Gridworld Proof-of-concept&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;So in learning this, I decided to actually code something up that implements the Reinforcement Learning algorithm. I created a simple simulated world in which an agent is able to control an &amp;quot;animal&amp;quot;. The animal can move in 4 directions. The animal is given state information &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;s&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; based on the tiles that are nearby it within a given radius. Most tiles are empty, but some have food. If the animal moves to a food tile, the food is consumed, and the agent receives a reward. At each step where the agent moves somewhere and doesn&#39;t get food, they receive a very small negative reward. As the animal moves around it runs out of energy, which is regained by eating food. If the animal runs out of food completely, it dies. If the animal passes a certain threshold of high energy, it reproduces, losing half of its energy to the baby. The child then has a clone of the parent&#39;s agent placed into it (as well as a copy of the parent&#39;s quality function &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8778em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;). As time goes on, food is spawned into the world.&lt;/p&gt;
&lt;p&gt;Since this example actually has state information, we need to keep track of that in our quality function as well. This is actually quite simple, as it turns out. We just go from using &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; to using &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(s, a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;s&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is the state we are in. This increases the problem space in a big way, but all the same methods are still perfectly workable, as long as we&#39;re okay waiting a while for it to learn by trying things.&lt;/p&gt;
&lt;p&gt;Since in gridworld, one state leads into another, causing consequences down the road, we also had to introduce an estimate, &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;Σ&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R_&#92;Sigma&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3283em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;Σ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, for how much reward we &lt;em&gt;think&lt;/em&gt; we&#39;ll achieve long-term following an action. This estimate is is based on our current values for the quality function, &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8778em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, meaning that as we improve the accuracy of our quality function, we&#39;ll also improve the accuracy of &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;Σ&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R_&#92;Sigma&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3283em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;Σ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. So as not to over-value the less immediate rewards, we introduce a &lt;em&gt;discounting factor&lt;/em&gt;, &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;0 &amp;lt; &#92;gamma &amp;lt; 1&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6835em;vertical-align:-0.0391em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.7335em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6444em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, so that we could define:&lt;/p&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;Σ&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;msub&gt;&lt;mtext&gt;max&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;≈&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;msup&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/msup&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;.&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;.&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;.&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;∑&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;∞&lt;/mi&gt;&lt;/munderover&gt;&lt;msup&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msup&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R_&#92;Sigma = R_t + &#92;gamma&#92;text{max}_a Q(s_{t+1}, a_{t+1}) &#92;approx R_{t} + &#92;gamma R_{t+1} + &#92;gamma^2 R_{t+2} + ... = &#92;displaystyle{&#92;sum_{i=0}^{&#92;infty} &#92;gamma^i R_{t+i}}
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3283em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;Σ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2806em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;max&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.1514em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3011em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2083em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3011em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2083em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;≈&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2806em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8917em;vertical-align:-0.2083em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3011em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2083em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1.0724em;vertical-align:-0.2083em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8641em;&quot;&gt;&lt;span style=&quot;top:-3.113em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;2&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3011em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;2&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2083em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.3669em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;...&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:2.9291em;vertical-align:-1.2777em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mop op-limits&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.6514em;&quot;&gt;&lt;span style=&quot;top:-1.8723em;margin-left:0em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.05em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;mrel mtight&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.05em;&quot;&gt;&lt;/span&gt;&lt;span&gt;&lt;span class=&quot;mop op-symbol large-op&quot;&gt;∑&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-4.3em;margin-left:0em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.05em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;∞&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.2777em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8747em;&quot;&gt;&lt;span style=&quot;top:-3.113em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2083em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;All this really does is basically say that &amp;quot;even though literally taking random actions will &lt;em&gt;eventually&lt;/em&gt; result in getting infinite rewards (after infinite time), we would actually prefer sooner rewards over later rewards.&amp;quot; Basically, every subsequent reward is less valuable to us than the one we&#39;re about to receive (since &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;0 &amp;lt; &#92;gamma &amp;lt; 1&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6835em;vertical-align:-0.0391em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.7335em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6444em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;). Your choice of &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;gamma&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.625em;vertical-align:-0.1944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is a matter of preference. Tweak it as needed.&lt;/p&gt;
&lt;p&gt;For ease of syntax, we can define &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;S&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;S&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6833em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0576em;&quot;&gt;S&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; as the set of all possible states, and &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;A&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;A&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6833em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;A&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; the set of all possible actions (in our case, &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;A&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext&gt;{up, down, left, right}&lt;/mtext&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;A = &#92;text{&#92;{up, down, left, right&#92;}}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6833em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;{up, down, left, right}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;). Our algorithm for gridworld is:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p class=&quot;katex-block&quot;&gt;&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mtable rowspacing=&quot;0.25em&quot; columnalign=&quot;right left&quot; columnspacing=&quot;0em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;//initialize values&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;for each &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;∈&lt;/mo&gt;&lt;mi&gt;A&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo&gt;∈&lt;/mo&gt;&lt;mi&gt;S&lt;/mi&gt;&lt;mo&gt;:&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;//start playing&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;loop:&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mtext&gt;getState&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mrow&gt;&lt;mo fence=&quot;true&quot;&gt;{&lt;/mo&gt;&lt;mtable rowspacing=&quot;0.36em&quot; columnalign=&quot;left left&quot; columnspacing=&quot;1em&quot;&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mtext&gt;argmax&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mtext&gt;, with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;false&quot;&gt;&lt;mrow&gt;&lt;mtext&gt;random(A), with &lt;/mtext&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mtext&gt;takeAction_getReward&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//&lt;/mtext&gt;&lt;msup&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo mathvariant=&quot;normal&quot; lspace=&quot;0em&quot; rspace=&quot;0em&quot;&gt;′&lt;/mo&gt;&lt;/msup&gt;&lt;mtext&gt; is our state after taking action &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;msup&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo mathvariant=&quot;normal&quot; lspace=&quot;0em&quot; rspace=&quot;0em&quot;&gt;′&lt;/mo&gt;&lt;/msup&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mtext&gt;getState&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//&lt;/mtext&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;Σ&lt;/mi&gt;&lt;/msub&gt;&lt;mtext&gt; is our expected long term rewards for taking action &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;Σ&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi&gt;γ&lt;/mi&gt;&lt;msub&gt;&lt;mtext&gt;max&lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/msub&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;msup&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo mathvariant=&quot;normal&quot; lspace=&quot;0em&quot; rspace=&quot;0em&quot;&gt;′&lt;/mo&gt;&lt;/msup&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//increment &lt;/mtext&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mtext&gt; for this action&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mtext&gt;//update the new running average of rewards for &lt;/mtext&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mstyle scriptlevel=&quot;0&quot; displaystyle=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;/mrow&gt;&lt;mspace width=&quot;1em&quot;&gt;&lt;/mspace&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi mathvariant=&quot;normal&quot;&gt;Σ&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;mo separator=&quot;true&quot;&gt;,&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mstyle&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;begin{aligned}
	&amp;amp;&#92;text{//initialize values}&#92;newline
	&amp;amp;&#92;text{for each }a&#92;in A, s&#92;in S: &#92;newline
	&amp;amp;&#92;quad Q(s, a) &#92;leftarrow 0&#92;newline
	&amp;amp;&#92;quad N(s, a) &#92;leftarrow 0&#92;newline
	&amp;amp;&#92;text{//start playing}&#92;newline
	&amp;amp;&#92;text{loop:}&#92;newline
	&amp;amp;&#92;quad	s &#92;leftarrow &#92;text{getState}()&#92;newline
	&amp;amp;&#92;quad a &#92;leftarrow 
		&#92;begin{cases}
			&#92;text{argmax}_a Q(s, a)&#92;text{, with } p = 1-&#92;varepsilon	&#92;newline
			&#92;text{random(A), with } p = &#92;varepsilon
		&#92;end{cases}&#92;newline
	&amp;amp;&#92;quad	R &#92;leftarrow &#92;text{takeAction&#92;_getReward}(a)&#92;newline
	&amp;amp;&#92;quad	&#92;text{//}s&amp;#x27;&#92;text{ is our state after taking action }a&#92;newline
	&amp;amp;&#92;quad	s&amp;#x27; &#92;leftarrow &#92;text{getState}()&#92;newline
	&amp;amp;&#92;quad	&#92;text{//}R_&#92;Sigma &#92;text{ is our expected long term rewards for taking action }a&#92;newline
	&amp;amp;&#92;quad	R_&#92;Sigma &#92;leftarrow R + &#92;gamma&#92;text{max}_a Q(s&amp;#x27;, a)&#92;newline
	&amp;amp;&#92;quad	&#92;text{//increment }N&#92;text{ for this action}&#92;newline
	&amp;amp;&#92;quad	N(s, a) &#92;leftarrow N(s, a) + 1&#92;newline
	&amp;amp;&#92;quad	&#92;text{//update the new running average of rewards for }a&#92;newline
	&amp;amp;&#92;quad	Q(s,a) &#92;leftarrow Q(s, a) + &#92;frac{1}{N(s, a)}[R_&#92;Sigma - Q(s, a)]
&#92;end{aligned}
&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:28.0575em;vertical-align:-13.7787em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-r&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:14.2787em;&quot;&gt;&lt;span style=&quot;top:-17.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-15.6887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-14.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-12.6887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-11.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-9.6887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-8.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-5.7787em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.3887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-1.8887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-0.3887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:1.1113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:2.6113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:4.1113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:5.6113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:7.1113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:9.0927em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:13.7787em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:14.2787em;&quot;&gt;&lt;span style=&quot;top:-17.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//initialize values&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-15.6887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;for each &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;∈&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;∈&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0576em;&quot;&gt;S&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;:&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-14.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-12.6887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-11.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//start playing&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-9.6887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;loop:&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-8.1887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;getState&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-5.7787em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;minner&quot;&gt;&lt;span class=&quot;mopen delimcenter&quot; style=&quot;top:0em;&quot;&gt;&lt;span class=&quot;delimsizing size4&quot;&gt;{&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mtable&quot;&gt;&lt;span class=&quot;col-align-l&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.69em;&quot;&gt;&lt;span style=&quot;top:-3.69em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;argmax&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.0573em;&quot;&gt;&lt;span style=&quot;top:-2.4559em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.2441em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;, with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-2.25em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.008em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;random(A), with &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.19em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.3887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;takeAction_getReward&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-1.8887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8019em;&quot;&gt;&lt;span style=&quot;top:-3.113em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;′&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt; is our state after taking action &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-0.3887em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8019em;&quot;&gt;&lt;span style=&quot;top:-3.113em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;′&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;getState&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:1.1113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3283em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;Σ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt; is our expected long term rewards for taking action &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:2.6113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3283em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;Σ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0556em;&quot;&gt;γ&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;max&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.1514em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8019em;&quot;&gt;&lt;span style=&quot;top:-3.113em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;′&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:4.1113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//increment &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt; for this action&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:5.6113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:7.1113em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;//update the new running average of rewards for &lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:9.0927em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3.75em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:1em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:1.3214em;&quot;&gt;&lt;span style=&quot;top:-2.314em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.109em;&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.677em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.936em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3283em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;Σ&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;mpunct&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.1667em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:13.7787em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This new algorithm requires us to define a new function &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mtext&gt;getState&lt;/mtext&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;text{getState}()&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord text&quot;&gt;&lt;span class=&quot;mord&quot;&gt;getState&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; which returns the state in some format that you believe to be most appropriate for the agent to digest. This state will usually be a vector (a list of numbers). You&#39;ll notice that very little changed really. The overall structure of the algorithm remains the same for state or stateless agents. Of course, very few stateless problems need to be solved with Reinforcement Learning, and so we fix our gaze on state-based problems.&lt;/p&gt;
&lt;p&gt;You&#39;ll also notice that since the actions available to us remains the same regardless of the state, we didn&#39;t need to specify &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;s&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; when referring to &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;A&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;A&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6833em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;A&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. If that weren&#39;t the case, we would need to refer to our set of possible actions as &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;A&lt;/mi&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;A_s&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.1514em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:0em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;s&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, updating our algorithm accordingly.&lt;/p&gt;
&lt;div id=&quot;see_it_in_action&quot;&gt;
&lt;h2&gt;See it in action&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;So, let&#39;s get a look at how this thing works.&lt;/p&gt;
&lt;p&gt;All the code for the project is available from the &lt;a href=&quot;https://github.com/rseeber/gridWorld-rl&quot;&gt;GitHub repo&lt;/a&gt;. If you&#39;d like, you can download the code and then run it by pasting the following into your terminal:&lt;/p&gt;
&lt;blockquote&gt;
&lt;pre&gt;&lt;code&gt;git clone https://github.com/rseeber/gridWorld-rl
cd gridWorld-rl
python agent_Q.py
&lt;/code&gt;&lt;/pre&gt;
&lt;/blockquote&gt;
&lt;p&gt;It&#39;s going to prompt you, asking how many training rounds you want it to run. Remember, these agents literally are acting entirely randomly before being trained, so pick a number large enough for them to properly learn their environment from experience. I&#39;ve found that a number somewhere in the range of 10,000 or 50,000 tends to be enough for them to properly learn their environment.&lt;/p&gt;
&lt;p&gt;This &amp;quot;training&amp;quot; stage is no different from the part afterwards, except that it doesn&#39;t show what&#39;s going on to the user until after the initial training stage.&lt;/p&gt;
&lt;div id=&quot;analysis&quot;&gt;
&lt;h2&gt;Analysis&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;There&#39;s lots of behaviors that the agents engage in which raise certain questions — most often, &amp;quot;are they behaving this way because that&#39;s actually beneficial in this environment, or have they just not had enough time to fully learn that there&#39;s better ways to behave?&amp;quot; This is mostly a question when I see them doing inefficient things, like wasting movement going nowhere instead of heading towards food that they can see.&lt;/p&gt;
&lt;p&gt;One possible explanation is that they&#39;ve learned over time that going too aggressively for food results in explosions in population (since they cannot control when they reproduce — it occurs spontaneously once they&#39;ve stored enough food energy), inevitably leading to overgrazing and a subsequent shortage of food for the parent.&lt;/p&gt;
&lt;p&gt;Thus, the agent &amp;quot;decides&amp;quot;&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn6&quot; id=&quot;fnref6&quot;&gt;[6]&lt;/a&gt;&lt;/sup&gt; it would be more optimal to take a conservative strategy — wasting time, so as not to exceed the carrying capacity of the environment. It&#39;s interesting, since the behaviors are being determined based on both Natural Selection, but also based on the goals (which come from our reward function we set) of the agent. So the agent is evolving to behave in a way that results in less of what you might call &amp;quot;suffering&amp;quot;&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fn7&quot; id=&quot;fnref7&quot;&gt;[7]&lt;/a&gt;&lt;/sup&gt; for the agent.&lt;/p&gt;
&lt;p&gt;To my own limited understanding of Biology, this seems to me to be fairly unique. Since heritable traits (which I shall call &amp;quot;genes&amp;quot;) are almost always passed down through DNA, it means that your genes cannot be pointed in the direction of behaving in ways that reduce suffering. They merely point towards behaving in ways that keep you and your family/tribe/species alive. The only exception to this would be the extent to which culture is also a method for passing down heritable traits. Humans are very strongly influenced by culture, as are many other animals to what is likely a lesser extent. Individuals within a population can choose to neglect inheriting a cultural trait if they feel that doing so would reduce their own personal suffering (even if it would have no effect on survival of the individual or the group). This is not a possibility with DNA, but it is a possibility in cultural inheritance.&lt;/p&gt;
&lt;p&gt;These RL agents are then made &lt;em&gt;entirely&lt;/em&gt; out of genes that can be modified in such ways as to reduce the &amp;quot;suffering&amp;quot; (I squirm to use the word a second time, I shouldn&#39;t hope to give the impression that these things are actually alive or aware of anything) of the agents. Individual agents can realize that certain behaviors (coded as genes) lead to less rewards than other options — despite the fact that neither option changes the survival rates for the group or individual. The agent adjusts their quality function following experience, and just like that, they update their genome to prefer higher rewards. The agents are able to evolve via the reward function, not just through Natural Selection.&lt;/p&gt;
&lt;div id=&quot;conclusion&quot;&gt;
&lt;h1&gt;Conclusion&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;Well, maybe that got a little philosophical there, but I hope the significance of that fact is maybe noteworthy to some people. Humans are capable of defying our own hardwired habits, but that doesn&#39;t mean it&#39;s always easy! Yet for these agents, at least in this particular environment, their main mechanism of evolution seems to be based on behaving in ways that maximize their goals (their reward function), not just what results in higher biological &lt;a href=&quot;https://en.wikipedia.org/wiki/Fitness_(biology)&quot;&gt;fitness&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;So, to recap what we covered:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Reinforcement Learning is a experience-based method for doing Machine Learning.&lt;/li&gt;
&lt;li&gt;At least in simpler applications, the basic strategy is to label the &amp;quot;quality&amp;quot; of various actions, or state-action pairs. That is, we decide what the quality is for taking some action &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, in a given state &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;s&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;s&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;s&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;In order to avoid sub-optimal (but still fairly okay) paths, we occasionally decide to &amp;quot;Explore&amp;quot; a random path, rather than &amp;quot;Exploiting&amp;quot; our previous knowledge. We Exploit most of the time, but we occasionally Explore, with some probability &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;varepsilon&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, where &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mi&gt;ε&lt;/mi&gt;&lt;mo&gt;&amp;lt;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;0 &amp;lt; &#92;varepsilon &amp;lt; 1&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6835em;vertical-align:-0.0391em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.5782em;vertical-align:-0.0391em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;ε&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6444em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There&#39;s of course a lot more complexity in the theory than what I showed off, for instance, I only described the basic details of how Q-learning and TD-learning work (which are the methods I used for the gridworld algorithm). If any of this was confusing (I&#39;m sure some of it was), you can totally feel free to &lt;a href=&quot;https://riverseeber.net/about&quot;&gt;email me&lt;/a&gt;. I love talking about this stuff, and I&#39;m sure it would also help me refine my own understanding!&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;I suppose there&#39;s nothing necessarily stopping a student from stealing the (original) work of an RL agent, and passing it off as their own. Though I&#39;d argue that would be a lot less detrimental to society than what is currently happening: a student stealing the output of an LLM, which in turn was stolen from a number of different human authors. Though, I don&#39;t know we&#39;ll ever create RL-agents that can write essays anyways. &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;I say organisms plural because humans are not, in fact, the only species to develop communication. We (might) be the most advanced at it, but many a species communicate with one another. Additionally, many communication calls are cultural, varying not by species, but by region. Birds of the same species in one part of the US have different mating calls than those in another part of the US. This would seem to indicate a kind of dialect or primitive &amp;quot;language&amp;quot; among animals (if you&#39;ll allow a slight abuse of the term). &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref2&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn3&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Where this equation comes from might be a bit mysterious to some readers. You can actually prove that it&#39;s true by starting at the first equation and slowly rewriting it until you get to this one. If you&#39;d like some more intuition about this version, you can think of the &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;−&lt;/mo&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;[R_i - Q(a)]&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mbin&quot;&gt;−&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2222em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; as the &amp;quot;error correction&amp;quot; from your predicted value &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;Q&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;(&lt;/mo&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo stretchy=&quot;false&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;Q(a)&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1em;vertical-align:-0.25em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;Q&lt;/span&gt;&lt;span class=&quot;mopen&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mclose&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, and a datapoint &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;R_i&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.8333em;vertical-align:-0.15em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0077em;&quot;&gt;R&lt;/span&gt;&lt;span class=&quot;msupsub&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.3117em;&quot;&gt;&lt;span style=&quot;top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:2.7em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;i&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.15em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, so that you&#39;re just incrementing your function by &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;frac{1}{n+1}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1.2484em;vertical-align:-0.4033em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8451em;&quot;&gt;&lt;span style=&quot;top:-2.655em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.394em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.4033em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; of that correction. &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref3&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn4&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Note that &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;mo&gt;←&lt;/mo&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;a &#92;leftarrow b&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;←&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.6944em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;b&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; is equivalent to &lt;code&gt;a = b&lt;/code&gt; in popular modern programming languages like C or Python. &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref4&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn5&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;By changing the value of &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;α&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;alpha&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0037em;&quot;&gt;α&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;, we&#39;re actually able to achieve various results, such as introducing a recency bias to the agent by setting &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;α&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;alpha&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0037em;&quot;&gt;α&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; to be a constant value. There are also various things you can do if you set &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;α&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;alpha&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0037em;&quot;&gt;α&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; to be some function of &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;n&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot;&gt;n&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; other than &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;α&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;&#92;alpha = &#92;frac{1}{n+1}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class=&quot;katex-html&quot; aria-hidden=&quot;true&quot;&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:0.4306em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord mathnormal&quot; style=&quot;margin-right:0.0037em;&quot;&gt;α&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mrel&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mspace&quot; style=&quot;margin-right:0.2778em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;base&quot;&gt;&lt;span class=&quot;strut&quot; style=&quot;height:1.2484em;vertical-align:-0.4033em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mord&quot;&gt;&lt;span class=&quot;mopen nulldelimiter&quot;&gt;&lt;/span&gt;&lt;span class=&quot;mfrac&quot;&gt;&lt;span class=&quot;vlist-t vlist-t2&quot;&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.8451em;&quot;&gt;&lt;span style=&quot;top:-2.655em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mathnormal mtight&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;mbin mtight&quot;&gt;+&lt;/span&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.23em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;frac-line&quot; style=&quot;border-bottom-width:0.04em;&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;top:-3.394em;&quot;&gt;&lt;span class=&quot;pstrut&quot; style=&quot;height:3em;&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sizing reset-size6 size3 mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;&lt;span class=&quot;mord mtight&quot;&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-s&quot;&gt;​&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;vlist-r&quot;&gt;&lt;span class=&quot;vlist&quot; style=&quot;height:0.4033em;&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;mclose nulldelimiter&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;. &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref5&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn6&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Perhaps it&#39;s not the &lt;em&gt;agent&lt;/em&gt; deciding. The way I&#39;m thinking of it is in terms of the way the Evolutionary Biologist Richard Dawkins &lt;a href=&quot;https://en.wikipedia.org/wiki/Gene-centered_view_of_evolution&quot;&gt;frames genetic evolution&lt;/a&gt; in &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Selfish_Gene&quot;&gt;&lt;em&gt;The Selfish Gene&lt;/em&gt;&lt;/a&gt;. Here, the quality function would be our so-called &amp;quot;genetic code&amp;quot;. Since this genome is passed down (perfectly identically) to children (but with the ability to mutate following the child&#39;s own unique experiences), I think it works in an analogous way. That is to say, maybe it&#39;s not the agent (the organism which carries genetic code) deciding to behave one way or another. Perhaps it is the genetic code deciding to code for inefficient paths, since those genes that code for excessive food consumption tend to result in the parent and children going hungry longer, and thus the gene either being overwritten, or else the organism starving to death, ending the entire bloodline. &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref6&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn7&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Not to say the agent is sentient or conscious in any meaningful way, but no word seems to exist for &amp;quot;experience a denial of what you want&amp;quot; that is meant to be used for non-sentient agents. &lt;a href=&quot;https://riverseeber.net/blog/post/an-exploration-into-reinforcement-learning/#fnref7&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>SEO And Independent Websites</title>
    <link href="https://riverseeber.net/blog/post/seo-and-independent-websites/" />
    <updated>2026-03-23T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/seo-and-independent-websites/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;March 23, 2026&lt;/em&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This post ended up being a lot more of a speculative exploration of ideas, rather than being a concrete declaration of opinions, so do keep that in mind while reading.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I spend a lot of my time on the more independent sides of the internet, away from all the Big Tech of Google, Facebook, and that type of thing. Out here on fringe, there&#39;s a lot of tendencies of the mainstream web that are (rightfully) frowned upon. Tracking and ads, for instance, are basically a no-go in these spaces. If you run an independent website for your blog and you&#39;re serving ads, people don&#39;t like that. Those ads are being used to track you as you browse the web, mapping out every single site you visit, basically feeding a profile of your likes and dislikes straight into Google&#39;s gaping eye of surveillance.&lt;/p&gt;
&lt;p&gt;So we don&#39;t like that. We talk bad on it. For good reason.&lt;/p&gt;
&lt;p&gt;Another one we don&#39;t like is known as Search Engine Optimization, or SEO. If you&#39;ve ever read a blog that was 50% just repeating the title to you every single paragraph, that&#39;s SEO-slop. Content which is trying to communicate as effectively as possible to the &lt;em&gt;search engine&lt;/em&gt; you&#39;re using, while fully missing the part where it needs to actually be readable by you once you click the link. With the advent of LLMs, SEO-slop has gotten a lot more pervasive, and as a result, search has gotten worse.&lt;/p&gt;
&lt;p&gt;Even if you&#39;re using a &lt;em&gt;less&lt;/em&gt; shitty search engine like DuckDuckGo or Kagi or what have you, the results are still likely being polluted with SEO-slop that is utterly terrible to read and extract meaning from. Take a look at this little blurb from 404Media, the independent news site that publishes stories mostly about the role of tech in politics and our lives:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&amp;quot;404 Media is an independent website whose work is written, reported, and owned by human journalists. Our intended audience is real people, not AI scrapers, bots, or a search algorithm.&amp;quot;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://www.404media.co/about/&quot;&gt;https://www.404media.co/about/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;This is the general attitude of people who veer away from the centralized hubs of online Big Tech. No matter where you go, you&#39;ll see the same general essence: &lt;em&gt;We are not the machine, and we don&#39;t create content to be consumed by the machine. We&#39;re humans, making human content, so that other humans will see it. We don&#39;t pervert our content for a machine so that it will distribute it to more humans. The machine is a tool that takes human expression and sends it out as is. We will not let it putrefy our art.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;I agree with this, by the way. If you&#39;re destroying the product of your art in order for it to be seen by more people, then you&#39;re literally defeating the purpose. Also, if you&#39;re a company who is doing SEO so that you&#39;ll get me to buy something, I literally despise that. Putting out fake review articles that are actually just ads is deceitful and awful.&lt;/p&gt;
&lt;p&gt;On the other hand, if you create a piece of art on the web that you really feel proud of, like an article or blog post, I wouldn&#39;t blame you for trying to communicate a few basic facts about the post to search engines so it will show it to people who want to see it.&lt;/p&gt;
&lt;p&gt;What does that mean? It could mean a few things. Picking a good title, one that people might end up actually searching for (especially if you&#39;re making tutorials, since that&#39;s the main tool you&#39;ll use when looking for tutorials). Throw some of the main keywords somewhere in the top, if you want people to be able to find it based on that.&lt;/p&gt;
&lt;p&gt;Obviously don&#39;t slopify it by spending 5 paragraphs getting to the point. Keep it human, but do some things so the search engines promote it, and people see your high-quality page, instead of having to gruel through the AI-generated article that was optimized to the point of unreadability.&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;Search engines do a number of things to figure out who to put on the top of a particular search. The obvious thing they do is read the page, which we&#39;ve already talked about. Another is that they&#39;ll look at how many &lt;em&gt;other&lt;/em&gt; pages have linked to your page or site. (They also look at how many times your own site links to a given page, but they don&#39;t take it into much consideration compared to other pages).&lt;/p&gt;
&lt;p&gt;While I&#39;m not recommending link spam, I think it would be worth making more of an effort to link to sites and pages that we like more often. Webrings have somewhat of this effect, but obviously those little widgets aren&#39;t exactly encoding the links of everybody on the page. They&#39;re quite useful for humans already on the site, but if you want search engines to be able to find your friends&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fn2&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt;, you should include links directly into your site somewhere.&lt;/p&gt;
&lt;p&gt;I try to make a point about linking to &lt;em&gt;as much stuff as possible&lt;/em&gt; on my blog. If you&#39;ve read even just one of my posts, you&#39;ll probably be able to guess how much I like to link to Cory Doctorow&#39;s blog over at Pluralistic. Something I really like about his blog is just how much he links to other pages. To the extent that reading a blog post of his slowly turns into a journey across the whole web.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/&quot;&gt;https://pluralistic.net/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;A lot of the blogs I interact with by less IndieWeb folks tend to be pretty bad about this. There&#39;s lots of good blogs on sites like Substack, but they&#39;re usually not very good about including abundant links in their posts, despite the high quality otherwise.&lt;/p&gt;
&lt;p&gt;I wish the blogging world had a better tendency to link around. In fact, I&#39;m currently considering different creative options for displaying links to all the new posts by blogs I appreciate. I&#39;m probably going to program a little Eleventy function that updates the list with new posts every time I rebuild my site.&lt;/p&gt;
&lt;p&gt;On the other hand, maybe I&#39;ll just stick to a hand-curated collection by shouting them out at the end of my blog, or else mentioning them within the blog.&lt;/p&gt;
&lt;p&gt;The web only works as well as it does because of how many pages make extensive use of links, allowing us to easily navigate from one page to another, without needing every hop to involve a search engine and a question. Sometimes (often) people want to find things that they never knew to ask for. The less that websites include links to other pages, the less spontaneous and creative the web becomes&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fn3&quot; id=&quot;fnref3&quot;&gt;[3]&lt;/a&gt;&lt;/sup&gt;. I guess that&#39;s getting a little off track of the topic of linking to good pages so search engines will show them to more people, though.&lt;/p&gt;
&lt;h1&gt;It bears asking&lt;/h1&gt;
&lt;p&gt;All this discussion of search and the degradation of it leaves me wondering if perhaps the way we&#39;re discovering new content on the web isn&#39;t a little fragile? As broken as Google is right now, it still works. To this day, if you search &amp;quot;vimwiki&amp;quot; on its own in any search engine, my post&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fn4&quot; id=&quot;fnref4&quot;&gt;[4]&lt;/a&gt;&lt;/sup&gt; shows up just after all the official resources, and underneath the much more relevant tutorial by Veronica Explains and a few others. So they still do show tiny little blogs if you search the right (niche) terms.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://vkc.sh/vimwiki-101/&quot;&gt;https://vkc.sh/vimwiki-101/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;But the fact that things still &lt;em&gt;kind of&lt;/em&gt; work now is a very fragile assurance that they won&#39;t get worse tomorrow. Earlier this year there was an incident where all of neocities (including both the site itself, and all subdomains hosted on it) got blocked from Bing search results, which had the effect of also removing them from other engines such as DuckDuckGo. The problem has since been resolved, but it goes to show the fragility of the system we&#39;re dealing with. What if this happened to somebody smaller who didn&#39;t have as large of a community to pull from?&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://blog.neocities.org/blog/2026/01/27/bing-block&quot;&gt;https://blog.neocities.org/blog/2026/01/27/bing-block&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;I don&#39;t have the solution. A diversity of search engines is a step in the right direction, but creating your own web index is so costly that basically only the major tech companies are able to run them, and all the other engines just proxy through them. Perhaps a tool like Marginalia Search could become useful:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://marginalia-search.com/&quot;&gt;https://marginalia-search.com/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;I also wonder if human curation by the community might somehow be able to improve things. After all, I don&#39;t think we&#39;re living in an ecosystem where tech companies are bending over backwards to make interesting or better things for us to use. If we want the web to work, we&#39;re going to have to work on this ourselves.&lt;/p&gt;
&lt;p&gt;I saw this thread on Mastodon about good wikis and information preservation in the context of technical communities, and it made me wonder if the same concept (which has been tried and tested to some extent) could maybe be expanded to a wider category of general web exploration? Maybe that&#39;s too ambitious and too prone to bad actors ruining everyone&#39;s fun, I&#39;m not really sure, but it was an interesting idea nonetheless.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://retro.social/@ifixcoinops/116132773124280983&quot;&gt;https://retro.social/@ifixcoinops/116132773124280983&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Veronica Explains also made a very interesting post about quitting search for 3 months a while back, which I found to be insightful, as well as the link to the Wikipedia page for the Google effect that she included.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://vkc.sh/quit-search/&quot;&gt;https://vkc.sh/quit-search/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Google_effect&quot;&gt;https://en.wikipedia.org/wiki/Google_effect&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;I don&#39;t really know where we should be &lt;em&gt;going&lt;/em&gt;, in terms of what future tech we should use to traverse the web, but as things remain currently, we really need to build a stronger community. More connection between independent sites. I won&#39;t tell you whether you should or shouldn&#39;t be trying to search-optimize your site, but if you want to do it, I don&#39;t think you should feel bad about it.&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Most people would probably argue that this level of tweaking isn&#39;t even SEO. It&#39;s just good design. Using metadata and titles and including some basic keywords at the top of an article in order to communicate to the search engines without degrading the experience for the user is hardly the same thing that the actual SEO folks are doing. Just something to keep in mind. Most people are not talking about this kind of simple thing when they talk about SEO. They&#39;re talking about wholesale sloppification of their content in order to sell it to the search engine, rather than trying to sell it to the reader. &lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Not something everybody on the IndieWeb is necessarily aiming for, but plenty are! It just depends what the user&#39;s goals are with their website. I do wanna get discovered on search engines, so I do things here and there to make sure my page shows up on relevant searches for my name. To each their own. &lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fnref2&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn3&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;I will say that one thing I particularly like about (some) social media platforms is how good they are at pushing links around. Back when I was on Reddit, they were pretty good about this (despite some other flaws), and now that I&#39;m on Mastodon, they&#39;re even better about this. Part of my experience is related to the fact that I just follow a lot of bloggers, but hey, so what? The point is that it&#39;s a pretty effective tool for pushing around interesting links. Also, Hacker News tends to be pretty good as well (just try not to spend too much time in the comment section). &lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fnref3&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn4&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Sidenote, I really ought to make a follow-up to that post. I wasn&#39;t really sure what I wanted my blog to be yet at that point (and I still don&#39;t, not fully!), and so it didn&#39;t really seem to be driving towards any specific purpose. Maybe I might enjoy writing some really solid tutorial posts? &lt;a href=&quot;https://riverseeber.net/blog/post/seo-and-independent-websites/#fnref4&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>Individualism Won&#39;t Solve Our Problems</title>
    <link href="https://riverseeber.net/blog/post/individualism-wont-solve-our-problems/" />
    <updated>2026-02-01T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/individualism-wont-solve-our-problems/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;February 01, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Last week, I published a blog post titled &lt;em&gt;The Internet Doesn&#39;t Suck: Blame Big Tech, Not The Internet&lt;/em&gt;. In it, I argued that the annoyances or issues we find with the internet are not inherent to the technology we use. Rather, Big Tech is designing these platforms in their own interests, which contradict our own — leading to the problems we&#39;re all quite familiar with. I wrote this to contradict a popular sentiment among Social Media critics, digital minimalists, and modern-day Transcendentalists that somehow &amp;quot;social media is the problem.&amp;quot; I fear this is just not the case. I would much rather people place the blame where it truly belongs: on the corporations who are intentionally turning the knobs to slowly make digital platforms more and more unbearable.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/the-internet-doesnt-suck/&quot;&gt;https://riverseeber.net/blog/post/the-internet-doesnt-suck/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Of course, though it is important, the management of social media platforms is by no means the only issue related to tech that plagues our society. I chose to use it as  an entry point because it proves useful for dissecting the underlying mechanism that has allowed all of tech to get worse suddenly. It&#39;s also easy because people have a very visceral connection with social media. Everyone knows that social media is getting crappy, so you don&#39;t really have to sell them very hard on the premise.&lt;/p&gt;
&lt;h1&gt;Enshittification&lt;/h1&gt;
&lt;p&gt;Before heading any further, I want to go over Cory Doctorow&#39;s description of the process of &amp;quot;enshittification&amp;quot;, which is his model of how and why platforms and services decay. Here&#39;s how he spoke of it during his talk at DEFCON 31:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&amp;quot;First, [a platform] is good to its users, then, it abuses its users to make things better for its business customers; finally, it abuses those business customers to claw back all the value for itself. Then, it dies.&amp;quot;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2023/08/27/an-audacious-plan-to-halt-the-internets-enshittification-and-throw-it-into-reverse/&quot;&gt;https://pluralistic.net/2023/08/27/an-audacious-plan-to-halt-the-internets-enshittification-and-throw-it-into-reverse/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;I&#39;m going to try not to get to caught up in the weeds in talking about this, but if you&#39;d like to learn a bit more, totally feel free to read or watch his full speech, linked above (or read his new book, &lt;em&gt;Enshittification: Why Everything Suddenly Went Wrong and What To Do About It&lt;/em&gt;. I haven&#39;t read it yet, but it&#39;s definitely on my list).&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://us.macmillan.com/books/9780374619329/enshittification/&quot;&gt;https://us.macmillan.com/books/9780374619329/enshittification/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;So, as stated, things start out good at first. When Facebook started, it really wasn&#39;t a bad platform. It showed you exactly what you wanted from people you followed, and didn&#39;t show you crap you didn&#39;t ask for from content creators you don&#39;t know. This was the era where &amp;quot;social media&amp;quot; wasn&#39;t a misnomer. You actually interacted mostly with your friends on these platforms.&lt;/p&gt;
&lt;p&gt;When Instagram started, it was being propped up as an alternative to the (now enshittified) Facebook. It was better to its users because it had to be in order to ignite an exodus from the incumbent platform.&lt;/p&gt;
&lt;p&gt;Of course, we know how the story goes. Instagram, just like Facebook, slowly started to enshittify. Whereas they used to only show posts from people you actually follow in your feeds, in a reverse chronological order — now most of the posts in your feed come from random &amp;quot;featured&amp;quot; accounts that you didn&#39;t follow or ask to see&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/individualism-wont-solve-our-problems/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;. Ads have seemingly increased at an exponential rate. They&#39;ve added all kinds of features directly designed to distract you into using the app longer than you had intended to.&lt;/p&gt;
&lt;p&gt;Moderation is also important, as it&#39;s the thing that keeps the interactions between users from degrading into a bar fight, though that&#39;s a highly detailed discussion that I won&#39;t be able to do justice here. Perhaps I&#39;ll make a post on it in the future. Suffice it to say that a company which feels the need to treat its users well will also invest into developing good moderation policies in order to prevent the more nasty elements of user interaction online.&lt;/p&gt;
&lt;h1&gt;The Montgomery Bus Boycotts&lt;/h1&gt;
&lt;p&gt;So what&#39;s it going to take to fix this system?&lt;/p&gt;
&lt;p&gt;Well, I&#39;ll tell you what&#39;s &lt;em&gt;not&lt;/em&gt; going to work. Trying to &amp;quot;boycott&amp;quot; Instagram isn&#39;t going to solve it. &lt;em&gt;Especially&lt;/em&gt; if you do it on your own. The issues with Big Tech are giant and messy and deeply entrenched, and simply avoiding the platform isn&#39;t going to solve those systemic issues on its own.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2025/07/31/unsatisfying-answers/&quot;&gt;https://pluralistic.net/2025/07/31/unsatisfying-answers/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;When black organizers in Montgomery, Alabama were fed up with the racism of the public transit system and decided to engage in a bus boycott, they didn&#39;t do it by all just waking up one day and deciding &amp;quot;dammit, these buses are pissing me off. I&#39;m just gonna walk to work instead.&amp;quot;&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Montgomery_bus_boycott&quot;&gt;https://en.wikipedia.org/wiki/Montgomery_bus_boycott&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The Montgomery Bus boycott was an &lt;em&gt;organized, collective movement&lt;/em&gt;. It saw it&#39;s members acting as a &lt;em&gt;group&lt;/em&gt;, not as individuals. The boycott was organized, and happened all at once, and thus could also be called off all at once (once their goals were achieved). It involved grassroots organizing, cooperation with labor unions and organizations like the NAACP, as well as the creation of an ad hoc organization, the &lt;em&gt;Mongomery Improvement Association&lt;/em&gt; (MIA) to coordinate the boycotts. This was not an individualistic boycott. Had the community simply just casually decided to not individually support the buses, their efforts would not have succeeded the way that they did.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://encyclopediaofalabama.org/article/montgomery-bus-boycott/&quot;&gt;https://encyclopediaofalabama.org/article/montgomery-bus-boycott/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;An individualistic boycott has absolutely &lt;em&gt;no hope of succeeding&lt;/em&gt;, because it tries to levy our own (admittedly weak) individual force against a giant systemic problem, rather than organizing together into a group which can multiply our forces together into something much stronger than the sum of its parts.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&amp;quot;Yet what force on Earth is weaker than the feeble strength of one? But the union makes us strong!&amp;quot;&lt;/p&gt;
&lt;p&gt;  — &lt;em&gt;Solidarity Forever&lt;/em&gt;, traditional folk song&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=Rgo9L_2l7QU&quot;&gt;https://www.youtube.com/watch?v=Rgo9L_2l7QU&lt;/a&gt;&lt;/p&gt;
&lt;h1&gt;So let&#39;s fight back&lt;/h1&gt;
&lt;p&gt;We need to take a stab at Big Tech from a collective position, acting as a &lt;em&gt;group&lt;/em&gt;, not as individuals. There&#39;s a few ways we can do that.&lt;/p&gt;
&lt;h2&gt;Anti-trust&lt;/h2&gt;
&lt;p&gt;First and foremost, we can work together to push for &lt;em&gt;good&lt;/em&gt; policy and enforcement. Get the FTC to actually enforce anti-trust law again. The FTC is an agency that was created for the sole purpose of policing the misconduct of corporations — such as when they try to monopolize, or when they harm the consumer in other ways. Since the Reagan-era, under presidents both Red and Blue, the FTC has been neutered. For the past 40 years, the agency has largely operated under the theory that unless a monopoly has direct, (easily) visible harms to the consumer, we shouldn&#39;t stop them. This has the end result of allowing corporations to consolidate into gigantic centers of unstoppable power.&lt;/p&gt;
&lt;p&gt;For a very brief 4 years under the Biden administration (problematic as he might&#39;ve been otherwise), we saw the revival of true anti-trust action. He appointed Lina Khan to chair the FTC, and under her leadership, we saw incredible strides for consumer rights, and punishment for monopolies. We saw a gigantic case against Google for operating an illegal monopoly — and the court even found them guilty!&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/individualism-wont-solve-our-problems/#fn2&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Lina_Khan#Actions_and_policies&quot;&gt;https://en.wikipedia.org/wiki/Lina_Khan#Actions_and_policies&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Of course, that all came to an end under the Trump administration, who has predictably returned the FTC to its role of playing lapdog for mega-corporations (as long as they fill his pockets with enough shitcoins to make him happy).&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.citationneeded.news/trump-memecoin-dinner-guests/&quot;&gt;https://www.citationneeded.news/trump-memecoin-dinner-guests/&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;Access Control laws&lt;/h2&gt;
&lt;p&gt;Beyond just anti-trust, there&#39;s several other things we could do to topple the Big Tech empire. It seems fairly clear to me (though I admit some readers might not agree) that the root cause of these issues is the mass consolidation of power and wealth by corporate tech giants, and so that&#39;s where I&#39;m aiming my attacks. If we can get tech companies into a position where they actually have to compete more ardently with one another in order to stay in business, I think we&#39;ll begin to see a lot more good things start happening in our world.&lt;/p&gt;
&lt;p&gt;The United States has instituted something known as an &amp;quot;anti-circumvention law&amp;quot; in the form of DMCA 1201. What this law does is make it a crime to bypass an &amp;quot;access control&amp;quot; (a piece of software designed to prevent you from modifying your device). For instance, if Apple doesn&#39;t want you installing 3rd party apps on your phone, all they do is create an access control on the iOS operating system. Even though it&#39;s actually quite trivial for hackers and individuals to invent workarounds for these access controls, DMCA 1201 makes it a federal crime to bypass them. It&#39;s also a federal crime to even describe &lt;em&gt;how&lt;/em&gt; you might bypass them. So what we&#39;ve basically created is a special kind of &amp;quot;legal saran wrap&amp;quot; where access controls are quite easy to bypass, except for that doing so is arbitrarily illegal.&lt;/p&gt;
&lt;p&gt;The power that this gives to tech companies is immense. It means they can prevent you using your own computer how you want to. Anytime there&#39;s a way to use some piece of software in a way that benefits the user to the detriment of the corporation, they can simply send out an update where they prevent that usage. And they further prevent &lt;em&gt;undoing&lt;/em&gt; that update by wrapping it in legal saran wrap that makes it a crime to undo.&lt;/p&gt;
&lt;p&gt;This is why Apple is able to charge a 30% fee on all transactions that happen in apps that come from their App Store. If it were legal to install apps through a different app store, everyone would use that app store instead. But because it&#39;s impossible to do so, that means that Apple can screw you over without losing your business. As a result, you pay an extra 30% on all transactions through your phone.&lt;/p&gt;
&lt;p&gt;This kind of anti-consumer bogus has embedded itself into everything from smartphones to tractors to medical equipment to smart fridges. A repeal of DMCA 1201 (or it&#39;s international equivalents in other countries) would mean that tech companies actually have to compete for your business, instead of making it a crime to use competitors on the devices they sold to you.&lt;/p&gt;
&lt;h2&gt;Privacy&lt;/h2&gt;
&lt;p&gt;By now I hope you&#39;re starting to realize that tech-related social issues are important for more reasons than just &amp;quot;Instagram is annoying to use&amp;quot;. This next point is going to start dipping into some of the heavier implications of Big Tech being so powerful.&lt;/p&gt;
&lt;p&gt;One of the many reasons why these corporations have so much power is because of the vast amount of &lt;em&gt;data&lt;/em&gt; they can collect on us. The fact that they&#39;re even &lt;em&gt;allowed&lt;/em&gt; to collect and use all of this is really beyond ludicrous. One of the most egregious applications of all this data is in the practice of surveillance pricing. Surveillance pricing is the act in which a corporation will spy on you, spy on how desperate you are, how long you&#39;ve gone without a paycheck, how bad your credit score is, how many mouths depend on you, and how stressed out you are in general with the weight of being broke, and they will selective jack up the price of goods or services for you based on all that data that they spied on you. Alternatively, some surveillance pricing systems work on the employment side of the economy instead, algorithmically reducing your pay when they know you can&#39;t afford to turn the work down. The end result in either case is that they get more of your money.&lt;/p&gt;
&lt;p&gt;For example, a 2023 report &amp;quot;&lt;em&gt;On Algorithmic Wage Discrimination&lt;/em&gt;&amp;quot; by Veena Dubal, a professor at UC Law San Francisco describes the way that Uber algorithmically discriminates against you in your wage based on your behavior patterns:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4331080&quot;&gt;https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4331080&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The same job, in the same time frame, and the same quality, will yield two &lt;em&gt;different&lt;/em&gt; payouts based on who&#39;s doing the work. Uber&#39;s algorithm for payment calculation is incredibly opaque, so it&#39;s difficult to understand what all goes into it, but it&#39;s clear to see that at the very least, they&#39;re using the data they collect on you when you use the app itself to determine your rate.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.latimes.com/business/technology/story/2023-04-11/algorithmic-wage-discrimination&quot;&gt;https://www.latimes.com/business/technology/story/2023-04-11/algorithmic-wage-discrimination&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;There&#39;s also nothing really to stop apps like this from approaching the data brokerage market, buying your data for pennies, and using that info to more accurately gauge your desperation. On the flip side, there&#39;s also nothing to stop Uber from selling &lt;em&gt;their&lt;/em&gt; data about you back into the industry for other corporations to have.&lt;/p&gt;
&lt;p&gt;Gig work isn&#39;t the only place seeing this type of thing, either. It&#39;s also happening in shopping, where your dollars are being priced at a different value because you&#39;re more or less desperate. Take Instacart, which was recently found to be charging different prices for different shoppers of the &lt;em&gt;same goods&lt;/em&gt;. The study was conducted by Groundwork Collaborative, Consumer Reports, and More Perfect Union, and they found differences in pricing that could add up to as much as $1,200 a year between various families purchasing the same exact products.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://groundworkcollaborative.org/work/instacart/&quot;&gt;https://groundworkcollaborative.org/work/instacart/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Again, we don&#39;t entirely know &lt;em&gt;what&lt;/em&gt; is behind the variation in pricing, but internal data collection is a likely candidate, as is data bought for cheap from 3rd parties.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.nytimes.com/2025/12/09/business/instacart-algorithmic-pricing.html&quot;&gt;https://www.nytimes.com/2025/12/09/business/instacart-algorithmic-pricing.html&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The &lt;em&gt;only&lt;/em&gt; reason this kind of algorithmic discrimination is even possible at all is because US citizens do not have federal privacy rights. The US hasn&#39;t passed a new federal privacy law since the 1988 Video Privacy Protection Act, which was a law to protect the privacy of your Blockbuster checkouts! Clearly, the mass adoption of the internet has changed a few things, and we need to pass new laws to protect us against new threats. The shady dealings of the data brokerage industry need to be heavily stomped out.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act&quot;&gt;https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Additionally, internal data retention of this scale should also be regulated, in order to make it more difficult for this kind of twiddling to take place. If Uber isn&#39;t allowed to store every micro-decision you&#39;ve ever made on their app, they won&#39;t be able to create an algorithm in order to determine the worst conditions under which you&#39;ll continue to work for them. If Instacart can&#39;t run experiments on what the highest price you&#39;ll pay for a carton of eggs is, they can&#39;t screw you out of being able to put food on the table for your family.&lt;/p&gt;
&lt;h1&gt;Solidarity Forever&lt;/h1&gt;
&lt;p&gt;So that&#39;s &lt;em&gt;what&lt;/em&gt; to fight for. I&#39;m not going to lie, it&#39;s a big to-do list. I don&#39;t know exactly how we&#39;re going to get all of this done, or what techniques for inspiring change we should use in favor others. I do know one thing thing though, and it&#39;s that we&#39;re not going to be able do it &lt;em&gt;on our own&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;If you want things to change, you must dismiss from your head the idea that your solo actions will affect any kind of real change. We simply cannot afford to think that we&#39;re all going to just &lt;em&gt;individually&lt;/em&gt; act in ways that eventually add up to a systemic overhaul. That&#39;s not how any of this works. The only way we can change society is by participating in it &lt;em&gt;as a society&lt;/em&gt;. By being larger than ourselves. By acting in &lt;em&gt;organizations&lt;/em&gt;, coordinating and timing our efforts with one another in order to make the largest impact. When we work together, our strength multiplies, and becomes greater than the sum of what we can all do on our own.&lt;/p&gt;
&lt;p&gt;It&#39;s become this kind of assumption in society that individual action is at the heart of social change. This is a piece of literal poison that comes from Neo-Liberalism — a school of thought pioneered by none other than Ronald Reagan and Margret Thatcher. They want you to focus more energy on not buying supplies from Amazon than you do on using those supplies to protest systemic problems. They want you to spend more time finding apps other than Twitter to organize your movement under than you do &lt;em&gt;actually organizing your movement&lt;/em&gt;. As Cory Doctorow has eloquently put it:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&amp;quot;Squabbling over whether using a social media network makes you a Nazi generates far more heat than light – so much heat that it incinerates the solidarity you need to actually fight Nazis.&amp;quot;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2026/01/22/optimized-for-unoptimizability/&quot;&gt;https://pluralistic.net/2026/01/22/optimized-for-unoptimizability/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Do not buy into the lies of &amp;quot;voting with your wallet&amp;quot;. That is right-wing intellectual poison that the left has bit down on, and we need to spit it out post haste! If the effective way to make change is to &amp;quot;vote&amp;quot; with your wallet, then that means people with thicker wallets get more votes than you. It also means that the people who have no reason to want social change are &lt;em&gt;almost always&lt;/em&gt; going to have more votes than the people suffering who need change in order to survive.&lt;/p&gt;
&lt;p&gt;So what do we do instead? I&#39;ll be honest, I&#39;ve got very little experience in this field. I&#39;m a young college student living in a tiny town that sees very little political activity. Still, here&#39;s some of the advice that I personally am going to be trying out myself based on stuff I&#39;ve read several places online.&lt;/p&gt;
&lt;p&gt;The big picture is that we need to get organized and working with other people. Tech-related activism groups, your local DSA, a workers union if that&#39;s something you can safely do, whatever. We need to be working with each other. We can&#39;t expect to solve things on our own. Don&#39;t feel obligated to do everything though, our strength comes from our numbers. Just do something, anything, and do it with other people.&lt;/p&gt;
&lt;p&gt;One resource that is incredibly useful if you&#39;re going to be pushing on tech-related issues is the &lt;em&gt;Electronic Frontier Foundation&lt;/em&gt;. They&#39;re a digital rights non-profit that has resources for grassroots organizing, as well as putting out news on items that demand our attention, like when Congress is voting on a critical tech bill. They make it easy to contact your local reps when that happens, and all you need to do is subscribe to their email updates.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://efa.eff.org/toolkits&quot;&gt;https://efa.eff.org/toolkits&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://act.eff.org/&quot;&gt;https://act.eff.org/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.eff.org/deeplinks&quot;&gt;https://www.eff.org/deeplinks&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;If you don&#39;t have the time or energy to join a proper organizing group, then just reach out and connect with community for the sake of getting to know people. Just knowing people at all means that you have something to draw from when you do want to get involved later.&lt;/p&gt;
&lt;p&gt;Also, of course, not every fight needs to be about tech. If you want to help out at a local community fridge or soup kitchen, that&#39;s fine! It means more people have their basic needs covered. It means you&#39;re reaching out into the community, and strengthening a local network by being involved. That&#39;s good both because getting people fed is good, and also because it means you&#39;re connected to a group of people who might be willing to organize together in other ways in the future. It gets you plugged into the community.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://infosec.exchange/@tinker/113589807117870451&quot;&gt;https://infosec.exchange/@tinker/113589807117870451&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;We don&#39;t need to all be superman. We just need to all be one of the millions of people who are pushing on an issue. Just being involved counts for something.&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Technically, Instagram has a way to (kind of?) disable this feature. They&#39;ve designed it in the most cruel way possible for the kinds of people who might find distracting features difficult to counter: The setting only disables the feature for &lt;em&gt;30 days&lt;/em&gt;. After that, featured posts are enabled again, secretly, without a notification. If you want to keep this feature disabled, you have to change your settings &lt;em&gt;every 30 days&lt;/em&gt;, because it KEEPS. TURNING. ITSELF. BACK. ON... &lt;a href=&quot;https://riverseeber.net/blog/post/individualism-wont-solve-our-problems/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Unfortunately, the remedy was incredibly lackluster. This was a great start, but it wasn&#39;t enough. I hope to see more energy out of them in future administrations. See Cory Doctorow&#39;s post about this for more info (&lt;a href=&quot;https://pluralistic.net/2025/09/03/unpunishing-process/&quot;&gt;pluralistic.net&lt;/a&gt;) &lt;a href=&quot;https://riverseeber.net/blog/post/individualism-wont-solve-our-problems/#fnref2&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>The Internet Doesn&#39;t Suck</title>
    <link href="https://riverseeber.net/blog/post/the-internet-doesnt-suck/" />
    <updated>2026-01-23T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/the-internet-doesnt-suck/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;January 23, 2026&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;I think about social media a lot. I&#39;m a computer science student at an undergraduate college, and I also happen to have an intense interest in political science and some of the humanities. I feel like it&#39;s almost natural from there that I would spend a lot of time thinking about the way we as humans use all this technology that makes up our world and our lives.&lt;/p&gt;
&lt;p&gt;Back when computers were just a footnote in our everyday lives, back when only your more well-off friends had smartphones, I could forgive you for seeing deep analysis of the way humans interact online as a trite or unimportant endeavor. Today, however, I think it&#39;s well understood the amount of impact that these online networks can have on our real world.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2025/02/13/digital-rights/&quot;&gt;https://pluralistic.net/2025/02/13/digital-rights/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;I&#39;ve often criticized current social media platforms for being designed in the interests of profit, which directly contradicts the interests of the actual people who use those platforms. Often a response to these criticisms — and indeed, I&#39;ve been in this camp myself previously — is to decry the very &lt;em&gt;idea&lt;/em&gt; of social media entirely.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;&amp;quot;Hear ye, hear ye, brethren, as I shout from the rooftops! It is time to disavow from the oppressive state of social media platforms! Indeed, social media is simply a tool of the bourgeoisie to push down on the proletariat in order to further oppress the little man!&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The only thing we have to lose are our chains! Disavow of all social technologies today, my brothers and sisters! Delete Instagram!&amp;quot;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;  — Me, probably. Circa 2024 (dramatization)&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Is that really a good idea though? Should we be disposing of the very idea of a social networking technology that allows you to see baby photos of your out-of-state sister&#39;s first-born child on your smartphone while scrolling your feed? A technology that allows you to reconnect with the people you went to high school with, or to easily contact people on the fringes of your social circle? I don&#39;t think I&#39;ve ever heard somebody criticize &lt;em&gt;those&lt;/em&gt; features, and yet, they&#39;re the very stuff that allows us to call the platforms we criticize &amp;quot;&lt;em&gt;social&lt;/em&gt; media&amp;quot;.&lt;/p&gt;
&lt;p&gt;In all fairness of course, the &lt;em&gt;options&lt;/em&gt; of what actually exists are, well — abysmal, to say the least. I don&#39;t think that it&#39;s in any way unfair for somebody to look at the landscape of platforms that exist right now and to say &amp;quot;wow, social media is awful. These apps were a mistake,&amp;quot; because, well, they are awful. And the fact they have been allowed to get this bad has been a detriment to our society.&lt;/p&gt;
&lt;p&gt;I&#39;m largely avoiding answering the question of &amp;quot;why do you think social media sucks,&amp;quot; because if you&#39;re reading a blog on a website in the year of our lord 2026, I&#39;m going to guess that you&#39;re at least partially clued in to why some people might be upset. For those who aren&#39;t, I invite you to open your screen time statistics for your phone. It should be somewhere in the settings. I&#39;m going to guess that if you&#39;re in your 20s or 30s, you probably have a very strong desire &lt;em&gt;not&lt;/em&gt; to look at that screen, for fear of how bad it might look. If you&#39;re one of the few people whose daily screentime tends to stay under an hour or less per day, then I guess you&#39;re better off than most of us. Most of my friends have daily times of several hours spent just scrolling on social media, wasting time, and not necessarily even having that much fun while doing it.&lt;/p&gt;
&lt;h1&gt;Why is the internet worse?&lt;/h1&gt;
&lt;p&gt;Okay, so we&#39;ve talked about how social media as a &lt;em&gt;concept&lt;/em&gt; is great, but that basically every single implementation of it is &lt;em&gt;terrible&lt;/em&gt; in practice. While those who are not involved in technology might simply throw out the idea with the crappy execution, those of us who are actually involved in creating technology might want to keep our minds open.&lt;/p&gt;
&lt;p&gt;In the world of software, we are only limited by our imaginations and our expertise. We can try to imagine tools that provide the utility we seek, without implementing the features that irk people. We can imagine a social media that doesn&#39;t play games with the &amp;quot;attention economy&amp;quot;, trying to &amp;quot;increase retention&amp;quot; — corpo-speak for &amp;quot;waste more of your time, when you otherwise would&#39;ve closed the app&amp;quot;. I&#39;d argue that Mastodon is pretty good at hitting all the positive user-experience metrics, without adding in any of the negative features you tend to see on the major platforms, but that&#39;s not really the point of this post.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://joinmastodon.org/&quot;&gt;https://joinmastodon.org/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The point is that I think people need to increase their creativity and imaginations. To criticize existing apps is fine, but I fear too many people forget that it&#39;s not really a problem with the &lt;em&gt;format&lt;/em&gt; of the apps or tools, but rather, it&#39;s a problem that originates with the &lt;em&gt;corporate, monopolistic structure&lt;/em&gt; of the designers of these tools.&lt;/p&gt;
&lt;p&gt;Facebook doesn&#39;t suck because it&#39;s social media. Facebook sucks because Mark Zuckerberg and the Meta shareholders demand that the company make more and more money from this product every year. It sucks because they were allowed to buy up or corner nearly all their competition without drawing the ire of the FTC in the form of an anti-trust case.&lt;/p&gt;
&lt;p&gt;The internet isn&#39;t worse today than it was 10 years ago because of the format of our communication tools. It&#39;s worse today because the companies who design our online experiences have been allowed to consolidate to the point of no longer needing to compete in the market. As a result, these major companies no longer need to create better products so that you will use them instead of a competitor, because now, &lt;em&gt;there is no competitor to speak of!&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;A lack of privacy rights for US citizens means that these tech firms are also able hoover up all kinds of data about you, using that to enrich their own pockets on data brokerage market, as well as in targeted ad placements. The US hasn&#39;t passed a new federal privacy law since the Video Privacy Protection Act of 1988. It was a Reagan-era, pre-internet law that made it illegal for the clerks at &lt;em&gt;Blockbuster&lt;/em&gt; to tell journalists what videos you&#39;ve checked out at their store (really, Blockbuster!).&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act&quot;&gt;https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;We deserve more than this, and for a number of reasons, but if for nothing else, we need it so that tech firms cannot use all of our data as leverage in their eternal struggle to enlarge themselves.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/&quot;&gt;https://riverseeber.net/blog/post/data-rights/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Anti-circumvention laws such as DMCA 1201 also play into this. They make it impossible for people to modify apps and software whose creators have a vested interest in you &lt;em&gt;not being able to modify it&lt;/em&gt;. Companies are able to wrap their apps in something known as Digital Rights Management, or DRM, suddenly making it a crime to modify the app. The power this gives to firms is immense. It locks up our devices, so that they cannot behave in ways that benefit the user at the cost of the corporation.&lt;/p&gt;
&lt;p&gt;All of these factors have led to the slow and continual enshittification of everything online, to the point where some people now negatively associate the internet and (certain types of) technology entirely.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=4EmstuO0Em8&quot;&gt;https://www.youtube.com/watch?v=4EmstuO0Em8&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;So to recap: Technology and the internet is a good thing. Not all the apps and software we use in our daily lives is good though. That&#39;s largely because they&#39;ve slowly enshittified over time due to their corporate ownership. What we need is to fix the system to prevent this enshittification from happening in the first place, not tell people to embrace mountain-man lifestyles or Butlerian Jihad (though I&#39;m sure both have a proper time and place associated with them).&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Data Rights</title>
    <link href="https://riverseeber.net/blog/post/data-rights/" />
    <updated>2025-12-23T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/data-rights/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;December 23, 2025&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Anybody who&#39;s even slightly involved in tech-related activism will know that there is just simply a &lt;em&gt;lot&lt;/em&gt; of jargon and technical details involved in the issues. Worse still, there&#39;s also just a lot of distinct issues: There&#39;s the fight for Net Neutrality, the right to use secure encryption tools, the right not to be spied on in public, the list goes on and on.&lt;/p&gt;
&lt;p&gt;It tends to be understood by those more deeply involved that all of the distinct issues related to tech are actually interconnected — it&#39;s no coincidence that people of the same creed tend to fall on the same side on a lot of these issues — the difficulty is of course in trying to convey that to the uninitiated. You and your hacker buddies might see the connection between Right To Repair, DRM, surveillance pricing, and enshittification; but the rest of the world just doesn&#39;t see it, and so our movement gets splintered as we lose people to the sheer breadth of the discussion.&lt;/p&gt;
&lt;p&gt;I propose that what we need is a unifying set of basic virtues (&lt;em&gt;basic for non-tech people!!&lt;/em&gt;) that makes it easier for the uninitiated to parse out which side is fighting for freedom, and which is pushing to hamper personal liberties. Something that makes it easy for people to talk about all aspects of issues related to tech: enshittification, data brokerage, surveillance pricing, right to repair, and even the high-tech spying being conducted by ICE and other federal agencies. It&#39;s all one big related movement, and these issues are part of the same fight, even if their effects seem wildly distinct from one another.&lt;/p&gt;
&lt;h1&gt;Data Rights&lt;/h1&gt;
&lt;p&gt;Data Rights is the name I would give to this model of virtues. Here&#39;s the definition I&#39;m working with currently:&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&amp;quot;&lt;strong&gt;Data Rights&lt;/strong&gt; is a movement that pushes for self-determination over your data, freedom over how you can access that data, and what you do with it.&amp;quot;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;That covers several domains of tech issues into a single big topic issue.&lt;/p&gt;
&lt;p&gt;This movement isn&#39;t really new. Lots of people are of course trying to push for this stuff already. Lots of people already understand why the issues are interconnected. What I&#39;m doing here is just presenting one way to verbalize that idea. If you think my approach is inefficient or not as good as other approaches, feel free to disregard it, and go on your own existing ways. This is mostly just me throwing this out into the air to see if it sticks or not.&lt;/p&gt;
&lt;p&gt;With that out of the way, lets look at how we can use the Data Rights approach to look at various different issues:&lt;/p&gt;
&lt;h2&gt;Privacy&lt;/h2&gt;
&lt;p&gt;Self-determination over your data means that you should be able to control who gets your data, and more importantly, who &lt;em&gt;doesn&#39;t&lt;/em&gt;. We need meaningful consent systems where it is possible to avoid being spied on without needing to exclude yourself from society. It&#39;s not enough to say &amp;quot;if you don&#39;t want Facebook tracking you, don&#39;t use Facebook&amp;quot;. That&#39;s not good consent. Facebook is where people are. You might be on Facebook so you can follow updates from your local community — support groups, your kid&#39;s after-school sports group, or just out of state family. You can&#39;t expect a person to have to choose between ditching all that social infrastructure, or else being spied on by Mark Zuckerberg. That&#39;s not consent, that&#39;s coercion.&lt;/p&gt;
&lt;h2&gt;Interoperability&lt;/h2&gt;
&lt;p&gt;Beyond just controlling who has access to your data, you should also have a right to interact with that data in any manner that you choose. Facebook should not be allowed to dictate how you access your account and feed. If you want to use an alternative Facebook client to access your account or view your friends posts, that needs to be legal&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/#fn1&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;. If you want to read your audiobooks using your old iPod, or on a different app than Audible, that should be legal&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/#fn2&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt;. DRM that locks your data up and tells you that you can only interact with it using special programs go against this virtue. It needs to be legal to unlock DRM.&lt;/p&gt;
&lt;h2&gt;Encryption and Freedom of Speech&lt;/h2&gt;
&lt;p&gt;We can even argue for certain free-speech goals under this model. So long as the content I&#39;m sharing is legal, I should have a right to transfer that data to interested parties who would like to have it. If I want to text my friend using an End-To-End Encrypted system (such as Signal, WhatsApp, or Matrix), I should be legally allowed to do this&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/#fn3&quot; id=&quot;fnref3&quot;&gt;[3]&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;h1&gt;It&#39;s all connected&lt;/h1&gt;
&lt;p&gt;Right now, ICE agents are prowling around the United States like secret police, arresting our neighbors who don&#39;t look white enough, asking questions later. Oftentimes these people are either US citizens, or else immigrants who were staying here legally, but weren&#39;t given due process.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.propublica.org/article/immigration-agents-detained-mistreated-citizens-congressional-investigation&quot;&gt;https://www.propublica.org/article/immigration-agents-detained-mistreated-citizens-congressional-investigation&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;ICE agents are sweeping into communities with such a high level of effectiveness thanks in part to the amount of surveillance tech they are deploying (or hijacking) on our communities. By fighting this surveillance tech, we make wins both for general privacy, as well as wins in stopping ICE from invading our communities.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://techcrunch.com/2025/10/16/amazons-ring-to-partner-with-flock-a-network-of-ai-cameras-used-by-ice-feds-and-police/&quot;&gt;https://techcrunch.com/2025/10/16/amazons-ring-to-partner-with-flock-a-network-of-ai-cameras-used-by-ice-feds-and-police/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;By fighting for DRM-breaking to legalized, we make wins both for general consumers of electronic media, but also anybody who owns hardware controlled by software, such as iPhones, smart fridges, insulin pumps, printers, hospital equipment, John Deere tractors, Teslas, and just about anything else in our computer focused world that&#39;s currently being locked down by software that&#39;s illegal to modify. Not just that though, it also creates a new niche for small businesses to pop up, who would be able to apply that DRM-breaking for users for a small fee.&lt;/p&gt;
&lt;p&gt;DRM-breaking would also mean you could install alternative app stores, which aren&#39;t controlled by Apple or Android — meaning Apple wouldn&#39;t be able to stop you from downloading apps like &lt;em&gt;ICEBlock&lt;/em&gt;, which allow vulnerable people to keep track of where ICE agents are, so you can stay safely away from them.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://apnews.com/article/apple-ice-iphone-app-immigration-fb6a404d3e977516d66d470585071bcc&quot;&gt;https://apnews.com/article/apple-ice-iphone-app-immigration-fb6a404d3e977516d66d470585071bcc&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h1&gt;Is Data Rights the only issue?&lt;/h1&gt;
&lt;p&gt;So, Data Rights for sure is limited to the extent through which technology interacts with society. Data has to be involved in some way, but since basically any kind of thinking that a computer does is going to involve data, any breaches of human decency committed by a computer are going to included. Tracking you around the world, trying to decrypt your private messages, preventing you from using your tech the way you want to use it — all that stuff is connected to Data Rights.&lt;/p&gt;
&lt;p&gt;At the same time, there&#39;s a much larger issue going on under the surface of all this. The reason that any of these issues around tech even exist has &lt;em&gt;nothing to do inherently with tech&lt;/em&gt;. The issue stems from corporate power, and the gap between the interests of these corporations versus the interests of regular humans. If Amazon actually cared about regular people, they wouldn&#39;t be nonconsensually handing the camera feed of your front porch over to law enforcement so that they could track license plates and pedestrian movements across the entire city. If John Deere wasn&#39;t single-handedly focused on money, they wouldn&#39;t be wrapping their tractor controllers in DRM that made it impossible to use 3rd party manufactured replacement parts.&lt;/p&gt;
&lt;p&gt;Not that these companies are ever going to care about people over money — they&#39;re companies, after all. But it illustrates the point that the enemy here is in big corporations that are run by billionaires who do not care about how many people they hurt.&lt;/p&gt;
&lt;h1&gt;Anti-Trust&lt;/h1&gt;
&lt;p&gt;Truly, what needs to happen is that the US (and other countries, too) need to enforce anti-trust law against these companies. Though he was spectacularly disappointing in most areas, the FTC under President Biden was actually the most energized and vigorous we&#39;ve seen it since the pre-Reagan era — largely thanks to the chair he appointed, Lina Khan.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2025/04/10/solidarity-forever-2/&quot;&gt;https://pluralistic.net/2025/04/10/solidarity-forever-2/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;We need to keep that kind of pressure up. First of all, it stops companies from hurting regular people (by punishing them when they misbehave), and it also forces them to actually compete with each other (meaning they have to actually follow the model proposed by Adam Smith — y&#39;know, offering us better products at market in order to out-sell their rivals). It also means they&#39;d have a lot less idle money sitting around waiting to be used for lobbying, bribes, or other nefarious purposes.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2024/08/14/the-price-is-wright&quot;&gt;https://pluralistic.net/2024/08/14/the-price-is-wright&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;While anti-trust enforcement might be the most important thing we should be doing against the threat of corporate power against regular people, that doesn&#39;t mean it&#39;s the only tool at our disposal. There are so many others things we can do in the mean time while waiting for anti-trust cases to move forward. Given the ways that nearly every major corporation has been using tech to advance its influence, it might then be worthwhile to take a look at the ways we allow or don&#39;t allow that tech to be used — both by regular people, and by the corporations themselves.&lt;/p&gt;
&lt;h1&gt;The big picture&lt;/h1&gt;
&lt;p&gt;So here&#39;s the full picture. You, me, our various communities, we&#39;re all involved in this struggle between the 99% of us who have our humanity, who care about other humans, and then there&#39;s the 1% (the billionaires and owning class) who see humanity as just something that effects their profit margins. We&#39;re involved in a fight against them. I&#39;m not talking about a coordinated anti-capitalist revolution fight or anything — just the eternal struggle between working class interests (keep prices affordable, provide livable wages, etc) and corporate interests (make union-busting legal, legalize price-fixing, defund the FTC, etc).&lt;/p&gt;
&lt;p&gt;Along that fight, there are a number of levers we can pull on to get what we want. The anti-trust lever works, and it&#39;s by far the most effective of them all, but it&#39;s not the only thing we can do. Among the other levers involves the whole array of switches in Data Rights.&lt;/p&gt;
&lt;p&gt;We can pass privacy laws that would make it illegal for companies to be collecting these vast stores of data on us. Without all that data, they wouldn&#39;t be able to analyze our levels of financial desperation in order to increase prices and decrease our wages when we&#39;re more desperate. (Uber does this, and there was a recent study that showed Instacart was too).&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2025/12/11/nothing-personal/&quot;&gt;https://pluralistic.net/2025/12/11/nothing-personal/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;We haven&#39;t passed a new federal privacy law since 1988, and that was a law banning journalists from getting access to the movies you checked out at &lt;em&gt;Blockbuster&lt;/em&gt;! We are 6 presidents removed from the last law actually protecting regular humans from being spied on. The internet has clearly created some changes that need to be addressed by further legislation.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act&quot;&gt;https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;We also need to remove some legislation, such as the DMCA 1201, which makes it illegal to circumvent DRM locks. It&#39;s the thing that prevents you from installing alternative app stores on your iPhone that aren&#39;t controlled by Apple. It&#39;s the reason you can&#39;t move your audiobook library to a competing app. It&#39;s the reason John Deere is able to permanently &lt;em&gt;brick all the tractors in an entire country&lt;/em&gt; from a server room in Los Angeles if Uncle Sam tells them to (likely with a tariff- or tax-shaped punishment if they don&#39;t comply). Not that I don&#39;t want to see Russian thieves thwarted when they steal from Ukrainian farmers — but it seems a worrying power to place in the hands of any single government or corporation. Imagine the Machiavellian things that could be done with that kind of power, especially under our current president.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.cnn.com/2022/05/01/europe/russia-farm-vehicles-ukraine-disabled-melitopol-intl/index.html&quot;&gt;https://www.cnn.com/2022/05/01/europe/russia-farm-vehicles-ukraine-disabled-melitopol-intl/index.html&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Fighting for changes to our digital world is not the primary fight, but it provides a hell of a lot of levers and controls that will help us pull through in that much more important fight of us vs the 1% who would see us toiling in squalor and chains.&lt;/p&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;In the US, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act&quot;&gt;Computer Fraud and Abuse Act of 1986&lt;/a&gt; makes Terms of Service violations a federal crime. Thus, Facebook is able to easily criminalize the usage of alternative Facebook clients by banning their usage in their Terms of Service. See also: &lt;a href=&quot;https://www.eff.org/files/2025/08/19/rebooting_the_computer_fraud_and_abuse_act.pdf&quot;&gt;A really good EFF one pager&lt;/a&gt; on the subject. &lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/#fnref1&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;A law known as DMCA 1201 makes it a federal crime to circumvent &lt;a href=&quot;https://en.wikipedia.org/wiki/Digital_rights_management&quot;&gt;digital locks&lt;/a&gt;. As a result, companies like Amazon have placed digital locks around products that &lt;em&gt;you&#39;ve paid for&lt;/em&gt;, which make it impossible to read your audiobooks (for example), unless you do it in their store. It also means that if they ever want to delete all your books in your library if you stop paying your ever increasing subscription fee, they&#39;re perfectly capable of doing that, and you can&#39;t (legally) do anything to recover those books without just buying them again somewhere else. &lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/#fnref2&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn3&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;Though not illegal in the US currently, End-To-End Encryption (E2EE) is constantly under threat by very real bills that would see the technology banned if passed. Luckily, none of these bills have passed the President&#39;s desk to become laws, yet. &lt;a href=&quot;https://riverseeber.net/blog/post/data-rights/#fnref3&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
  </entry>
  <entry>
    <title>Post-Semester Linkdump</title>
    <link href="https://riverseeber.net/blog/post/post-semester-linkdump/" />
    <updated>2025-12-08T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/post-semester-linkdump/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;December 08, 2025&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Well, it&#39;s been a minute since I&#39;ve made a post. I&#39;ve been &lt;em&gt;busy busy busy&lt;/em&gt; going through a crazy semester in college. There was nothing in particular that made it this stressful, just a large collection of several independent factors all dragging me down in tiny ways. With the semester now coming to a close and everything sorting itself out, I wanted to come back and make another post.&lt;/p&gt;
&lt;p&gt;I haven&#39;t really been doing a deep level of creative thinking lately, and haven&#39;t had the time to read any new books, so this post is just going to be a linkdump, since that seems easy and fun for me to write. I&#39;m mostly copying the idea from Cory Doctorow, but who&#39;s keeping track anyways?&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/tag/linkdump/&quot;&gt;https://pluralistic.net/tag/linkdump/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;With us on the topic, I recently caught up on a good post he made last week, &amp;quot;Normie diffusion and technophilia&amp;quot;. The basic idea is that &amp;quot;early adapters&amp;quot; of new technologies aren&#39;t just doing it because they&#39;re &amp;quot;tech-savvy,&amp;quot; if fact, most often they &lt;em&gt;aren&#39;t&lt;/em&gt; savvy at all -- at least not innately. The majority of people who switch to using new technologies, especially communication technologies, do so because they have some intrinsic need to switch &lt;em&gt;away&lt;/em&gt; from what already exists.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2025/11/27/early-adapters/&quot;&gt;https://pluralistic.net/2025/11/27/early-adapters/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Speaking of new communication tech, I&#39;ve recently started using Mastodon. It&#39;s a decentralized social microblogging network. It&#39;s designed with fire exits built in -- that is to say, if or when Mastodon (the company) gets bought up by private equity and decides it wants to screw the user instead of being nice to them (much like Facebook, Instagram, Threads, X, TikTok, Tumblr, etc are screwing their users already), users of Mastodon (the product/network/platform) can leave to a server that isn&#39;t as shitty. When they leave, they just add back all their old friends again -- even the friends who &lt;em&gt;didn&#39;t&lt;/em&gt; switch to the new server.&lt;/p&gt;
&lt;p&gt;It&#39;s a lot like how you have an Android, your cousin has an iPhone, and your uncle still has a flip-phone for some reason, and yet all of you can talk in the same group chat together. Now apply that concept to social media and that&#39;s the Mastodon network. If Mastodon (the company) goes full Elon-mode, you still have other options on the Mastodon network you can switch to, without losing access to your friends. It promotes competition, and keeps the server owners from going crazy like they have with all the major corporate platforms.&lt;/p&gt;
&lt;p&gt;Anyways, I make a few tweets (I&#39;ve been informed they&#39;re called &amp;quot;toots&amp;quot;) on there from time to time, and I plan to post links each time I make a new blog post, so if RSS isn&#39;t your thing but you still want to follow my blog posts, you can follow my account on Mastodon. The cool thing about open platforms is how easy it is to syndicate in many different places from a single account. Check out some of the other places it&#39;s possible to follow the account from over on my Follow Me page.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://riverseeber.net/follow-me/&quot;&gt;https://riverseeber.net/follow-me/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Since we&#39;re on the topic of Mastodon, here&#39;s a cool little site I found that tracks how decentralized in practice the various platforms are that purport to be theoretically decentralized. It shows the Mastodon network being fairly decentralized, Bluesky/AT Proto being highly concentrated. It&#39;s also got some stats on other networked services like Git or Web Hosting that you can look into as well.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://arewedecentralizedyet.online/&quot;&gt;https://arewedecentralizedyet.online/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;I recently read a pretty good post by Robert Kingett about web accessibility and his experience as a blind user trying to get corporations to fix their unusable shit -- and actually succeeding and seeing results. If you create things that get used by people with accessibility needs -- that is to say, if you create things -- you should really give this a look. Accessibility is heavily under looked, and the few advances we&#39;ve made as a society since Bush senior passed the ADA are being threatened to be pulled back as Republican politicians have been criticizing the measure in recent years. Now is probably a good time to start paying attention to people&#39;s accessibility needs.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://sightlessscribbles.com/posts/20250724/&quot;&gt;https://sightlessscribbles.com/posts/20250724/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Awhile back, I watched a great video about writing and AI (but mostly just about writing). The video is titled &amp;quot;You are a better writer than AI. (Yes, you.)&amp;quot; on YouTube. It took things pretty foundational, and has changed how I think about a lot of things when it comes to communication. I recently gave it a rewatch and managed to still get something out of the second go around. I don&#39;t really know how else to explain it other than by letting you watch it.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=V5wLQ-8eyQI&quot;&gt;https://www.youtube.com/watch?v=V5wLQ-8eyQI&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;It seems we&#39;re getting into rapid-fire mode here. In case you&#39;ve never looked into it before, here&#39;s some reasons why Salvation Army is kind of an awful organization that doesn&#39;t deserve your money (hint: it&#39;s not even primarily just the homophobia).&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://libcom.org/article/starvation-army-twelve-reasons-reject-salvation-army&quot;&gt;https://libcom.org/article/starvation-army-twelve-reasons-reject-salvation-army&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;While I don&#39;t necessarily agree with point #3 in the article (I&#39;d say they&#39;re being overly harsh on religion as a concept), what I&#39;m trying to share is that Salvation Army has made a mission out of upholding the status quo of economic inequality, actively making things worse for poor people on an institutional level. They&#39;ve been fighting with the working class since their founding. As far back as in the early 20th century, they would engage in constant altercations with union workers and organizers. The Union organizers would counter protest the Army by singing their songs back at them with new lyrics satirizing them. Joe Hill wrote &amp;quot;The Preacher and the Slave&amp;quot; in this fashion back in 1911. Here&#39;s a good performance of it by legendary folk singer Pete Seeger, along with some good commentary and history.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=cj02R-R8Who&quot;&gt;https://www.youtube.com/watch?v=cj02R-R8Who&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Since we&#39;re somewhat on the topic of history, how about I include this blog post I read from an actual historian about how &amp;quot;The Dark Ages&amp;quot; is a stupid term when used in the pop culture sense, and how the fall of the Roman Empire was a good thing, actually. All delivered in the format of a rant against an annoying Twitter (well, Bluesky) interaction.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://going-medieval.com/2025/11/25/on-contrarian-history/&quot;&gt;https://going-medieval.com/2025/11/25/on-contrarian-history/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Going from that, here&#39;s an interesting little short blog post about excellence, discussing Mamdani&#39;s victory speech, and drawing ties to the writings of Marx. Not much to say on it, it&#39;s super short, just read it if you&#39;re interested.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://coreyrobin.com/2025/11/15/excellence-over-mediocrity-from-mamdani-to-marx-to-food/&quot;&gt;https://coreyrobin.com/2025/11/15/excellence-over-mediocrity-from-mamdani-to-marx-to-food/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Finally, let&#39;s round things off with a fun card game I found while doing some research for my tabletop RPG games. It&#39;s called &lt;em&gt;Scoundrel&lt;/em&gt;, and it&#39;s a solo dungeon crawling game. It can easily be modified to be played with multiple players, or to include additional mechanics. I&#39;ll be using it at some point for sure. Just thought I&#39;d drop the link for anyone else who plays TTRPG games.&lt;/p&gt;
&lt;br&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=7fP-QLtWQZs&quot;&gt;https://www.youtube.com/watch?v=7fP-QLtWQZs&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Isaac Asimov, Robots, AI, and Luddism</title>
    <link href="https://riverseeber.net/blog/post/isaac-asimov-robots-ai-and-luddism/" />
    <updated>2025-09-24T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/isaac-asimov-robots-ai-and-luddism/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;September 24, 2025&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;img class=&quot;introImg&quot; src=&quot;https://ia800402.us.archive.org/view_archive.php?archive=/12/items/olcovers662/olcovers662-L.zip&amp;file=6621014-L.jpg&quot; alt=&quot;Book cover for &#39;The Caves of Steel&#39;&quot;&gt;&lt;/p&gt;
&lt;p&gt;I just finished reading &lt;em&gt;The Caves of Steel&lt;/em&gt; (1953) by Isaac Asimov. I really enjoyed it. I guess it&#39;s a little weird to write a review for a 70 year old book, so instead I decided to put together whatever the hell this post is. We go over the book, but then also its implications on modern society, as well as draw-backs to the 19th century Luddite movement that occurred during the Industrial Revolution. Hope you enjoy, it&#39;s a little non-traditional.&lt;/p&gt;
&lt;h1&gt;The plot&lt;/h1&gt;
&lt;p&gt;So this book was actually my first Asimov novel. The plot follows that of one Lije Baley, a detective who lives and works in the &lt;em&gt;heavily&lt;/em&gt; socially stratified big city of New York. The highly dense and compact nature of this city in the future means that most people don&#39;t have kitchens or bathrooms in their own houses or apartments, and must go to communal places for this. The entire city is encased in a dome, and shielded from the air and sunlight of the outside world.&lt;/p&gt;
&lt;p&gt;Understandably, this unnatural way of living lends itself to some backlash movements. The Medievalists, as they&#39;re called, want a return to a time they refer to as the &amp;quot;Medieval&amp;quot; age. It becomes clear that what they mean by this is the 20th century or so, which to them is long, long past.&lt;/p&gt;
&lt;p&gt;This society is slowly being introduced to some very primitive robots — working in retail centers, and one in the detective office. All of them are quite dull, unable to perform advanced tasks. Even still, this represents a threat to the labor force, especially in a society such as this, where getting fired means decent from all of the comforts in life that you know, total abandonment at the bottom of society. (Are we sure this was written in the 1950s not the 2020s?) Anyways, this fear of unemployment creates a sense of Luddism within the minds of many Medievalists or even just regular folks.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://thetechbubble.substack.com/p/on-the-origins-of-dunes-butlerian&quot;&gt;https://thetechbubble.substack.com/p/on-the-origins-of-dunes-butlerian&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;On top of all this, the culture of Earth must tolerate some pompous, upper class society, called the &amp;quot;Spacers&amp;quot;, who live, you guessed it, up in space right above Earth. The Spacers are of high social standing, and tensions between the cultures are high, not least because of the push by the Spacers to introduce robots into the Earth culture.&lt;/p&gt;
&lt;p&gt;It is inside of this context that one of the Spacers are murdered within their own home in Spacetown. The Spacers expect it to be someone from Earth. They start up a joint investigation, with Lije Baley, an Earth man, being partnered up with R. Daneel Olivaw, a &lt;em&gt;very&lt;/em&gt; advanced robot from Spacetown. In fact, Daneel even passes for a human upon reasonably close inspection.&lt;/p&gt;
&lt;p&gt;The investigation follows through looking into Medievalist secret societies, conspiracies, corruption, and Luddite aversion to automation technology which threatens the livelihood of the working class. By the end of the novel, Lije comes to change his view from that of a vague Medievalist, instead to believe in the cooperation between robots and humans, a transition to a so-called &amp;quot;C/Fe society&amp;quot; (Carbon-based life integrating with Iron-based life).&lt;/p&gt;
&lt;h1&gt;Luddism&lt;/h1&gt;
&lt;p&gt;The word Luddism is, of course, not actually used within the novel, but I mention it because I feel like the themes are overtly present in the Medievalist backlash to robots working positions once held by humans. In the real world, Luddites were a group of textile makers during the industrial revolution. They stood opposed to the &lt;em&gt;ways in which textile automation technology was being used&lt;/em&gt;. More specifically, the tools were being used to take power away from the textile workers, and it was outputting lower quality products for consumers than when it was all just made by hand. Luddites had no issue with the &lt;em&gt;idea&lt;/em&gt; of automation. What they didn&#39;t like was the way it was being used as a means by the owning upper-class to take away power from the poor working-class.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In truth, Luddism and science fiction concern themselves with the same questions: not merely &lt;em&gt;what&lt;/em&gt; the technology does, but who it does it &lt;em&gt;for&lt;/em&gt; and who it does it &lt;em&gt;to&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt; — Cory Doctorow, Locus Magazine&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://locusmag.com/feature/cory-doctorow-science-fiction-is-a-luddite-literature/&quot;&gt;https://locusmag.com/feature/cory-doctorow-science-fiction-is-a-luddite-literature/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Even back in the 1950s, the awareness of where computers were headed was presently in the minds of many, who perhaps might have reason to worry about being replaced and laid off. In a certain capacity, those fears were justified, as the computer would become more and more advanced, and take a seat in various positions once ran by humans. Of course, at the same time, it brought in a whole new set of positions needing humans to fill. So life goes on.&lt;/p&gt;
&lt;p&gt;The bringing in of the computer was in some ways used as a threat to workers, but for the most part, it was simply done out of pragmatism. Humans tend to be pretty bad at performing repetitive, pattern-based actions which do not require discretion. On the other hand, computers are pretty good at these kinds of activities. What computers &lt;em&gt;aren&#39;t&lt;/em&gt; good at is activities that require a certain level of situational context and discretion. Computers aren&#39;t good at determining whether or not you&#39;re returning this library book late for good reasons or not. They just know that its late. That&#39;s why we created self-checkout machines at the library, but still have a staff of librarians which can be a human to help you escape the strict rigidity of the computer bureaucracy. They can listen to you explain that your car broke down so you couldn&#39;t get it returned, and they can override the fees on your account, giving you amnesty. A computer is not suited towards that kind of task.&lt;/p&gt;
&lt;p&gt;Right now, AI is being peddled as some kind of technology that attempts to replace humans in their everyday jobs. At the same time, AI is pretty bad at any task where it needs to &lt;em&gt;replace&lt;/em&gt; a human. Humans just perform too many small, discretionary decisions that current AI models are simply not possible of even realizing they should consider. The models just don&#39;t compare to the human mind if they&#39;re being made to try to &lt;em&gt;replace&lt;/em&gt; the human mind. That&#39;s not what this tool is ever going to do, not in the near future anyways.&lt;/p&gt;
&lt;p&gt;But AI is not useless. AI has potential for &lt;em&gt;assisting&lt;/em&gt; humans, when it&#39;s done organically. When your boss fires half your department, but tells you that you need to offset the workload onto &amp;quot;AI integrations&amp;quot; or something like that, the AI is going to perform badly. The department is going to put out shit content compared to what they used to. On the other hand, when you just have a thing you need to do, and you realize that an AI can make a template for this thoughtless task to help speed it up, that&#39;s a genuine boost. But those boosts are, much to the chagrin of the employer class, not actually that dramatic, and definitely don&#39;t create the kind of productivity boosts that allow you to fire half of every department.&lt;/p&gt;
&lt;p&gt;In his blog, Cory Doctorow often explores the ideas of &amp;quot;Centaurs&amp;quot; and &amp;quot;Reverse-Centaurs&amp;quot; when it comes to automation technology. A centaur is a person who, in the natural course of their duties, finds a productive way to use the automation technology to offload some of their own work, making them more productive at the job they&#39;ve always been doing. It lets them get more done, but at the end of the day, they are free to use the automation tool where they feel it is appropriate, and abandon it in instances where it is not useful or needed. They are given control and discretion over its use. A centaur is a human standing atop a robot body which they can control.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://pluralistic.net/2025/09/11/vulgar-thatcherism/&quot;&gt;https://pluralistic.net/2025/09/11/vulgar-thatcherism/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;A reverse-centaur, on the other hand, is a squishy human being piloted and controlled &lt;em&gt;by the robot!&lt;/em&gt; The reverse-centaur is a worker whose boss just fired a bunch of their coworkers, and expects them to pick up the slack by using AI. When the quality of the work goes down, the boss will either not care, or, worse still, they might blame the remaining workers for being so careless, despite the ludicrously cramped deadlines and under-staffing.&lt;/p&gt;
&lt;h1&gt;Weren&#39;t we supposed to be talking about Isaac Asimov?&lt;/h1&gt;
&lt;p&gt;Right. Back to the book.&lt;/p&gt;
&lt;p&gt;It&#39;s important to remember that the conclusion of the book involved an acceptance that &lt;em&gt;humans need to learn to cooperate with robots&lt;/em&gt;. In the backdrop of the 1950s, it might be easy to mistake calls for robot integration with that of racial integration that were going on at the time. This was my initial viewpoint of the themes of the so-called &amp;quot;C/Fe culture&amp;quot; that the pro-robot faction was calling for. Upon further inspection, it becomes clear that this is not the true nature of things, and the comparison really isn&#39;t fair. These robots &lt;em&gt;do not form a culture of their own&lt;/em&gt;. The robots are not sentient, and they do not pretend to be. The robots are machines, and do not have desires of their own, besides in how they seek to serve humans.&lt;/p&gt;
&lt;p&gt;Despite this, the characters in the book often saw the &amp;quot;C/Fe culture&amp;quot; stuff as being a cultural mixing, notably the Medievalists and others of the anti-robot factions. They feared that these newfangled robots would demand to fill human jobs and be treated with humanity. All this on top of their utter superiority at performing certain kinds of tasks that humans could never hope to achieve (think: basically anything a computer can do better than a human, but then give that computer opposable thumbs and a body to walk around in).&lt;/p&gt;
&lt;p&gt;This is the mistaken way that the Earth society looks at robots in. They see them as a threat, as some kind of culture of their own. Oftentimes within the book, Daneel Olivaw — the advanced robot from Spacetown — would shock the humans around him by his utter lack of care for human prejudice against robots. He is &lt;em&gt;not&lt;/em&gt; sentient. He does not have feelings to hurt. When Lije Baley admitted that he was prejudiced against robots, Daneel said it didn&#39;t matter, so long as they could work together effectively. The robot had no sense of cultural identity that needed to be defended from trampling.&lt;/p&gt;
&lt;p&gt;At one point, a Medievalist radical took a swing against Daneel. The robot didn&#39;t see it coming, but did his best to back away to soften the blow — not to himself, but to protect the hand of the attacker from being hurt by his metal frame. He didn&#39;t care that he was being attacked, he just needed to follow the First Law of Robotics: to protect the human.&lt;/p&gt;
&lt;p&gt;At other times, people would express hatred towards the more primitive robots in front of Daneel, then apologize, as if they had offended him. He told them he didn&#39;t care, he couldn&#39;t be offended even if he tried — he&#39;s a robot.&lt;/p&gt;
&lt;h2&gt;What does this tell us about automation and technology?&lt;/h2&gt;
&lt;p&gt;The answer is that technology does not &lt;em&gt;inherently&lt;/em&gt; threaten humanity. It simply doesn&#39;t. Technology is created by humans, in order to best suit humans. Within this story, we are to believe that the people who create and distribute the robots (Spacetown) genuinely hold the best interests of the Earthmen in mind when creating their robots. This is the premise upon which the acceptance of all of this lies. This suits the story well, though it would incorrect to say that it is the only way it could have been.&lt;/p&gt;
&lt;p&gt;If, for instance, the story took place in a society that had a fairly strong working class, and which the owning class wanted to assert their dominance by breaking up unions, firing workers for expressing discontent with their conditions, and otherwise just putting down the labor force, the story might be different. It might then be more plausible to say that the owning class is pushing robots down on society in order to take away agency from the working class. In that case, the argument that &amp;quot;by just simply cooperating with the robots, humanity (and the working class) can be better off&amp;quot; would fall flat. You cannot cooperate with a technology that was expressly designed to destroy you.&lt;/p&gt;
&lt;p&gt;And this is where we stand in our real world in the year 2025. We must ask ourselves who it is that is designing these so-called &amp;quot;Artificial Intelligence&amp;quot; tools? Are they people who have the best interests of regular people, or are they designed by corporate tech monopolies, which have demonstrated time and time again that they will sacrifice the well-being of countless workers in order to further their power, wealth, and influence?&lt;/p&gt;
&lt;p&gt;The AI technologies we are being fed at current time are created by money-hungry monopolists who would love nothing more than to have us all at their mercy. They are forced upon us (often, but not always) by bosses who want to speed up our work speed, while sacrificing both output quality and our own well-being, just in the name of increased profits. This has the result of decreasing quality of life for both consumers and employees.&lt;/p&gt;
&lt;p&gt;But it doesn&#39;t have to be this way, not inherently. What we are witnessing is a result of our economic system, not a result of the technology itself. The extreme stratification of our economic environment is what leads all of this. AI models are massive and bulky, and are pretty damn difficult to run on cheap hardware, but as the economic incentives begin to make corporations try to get AI models to run on consumer hardware (be it out of the chase of the AI bubble, or simply a desire to get AI spyware running on everybody&#39;s smartphone), we&#39;re seeing models pop up that are actually reasonably competent that can run on low specs.&lt;/p&gt;
&lt;p&gt;What we absolutely &lt;em&gt;need&lt;/em&gt; right now, is for more of these reasonably sized tools to keep coming out. Open source models, runnable by anybody. Tools that aren&#39;t being gatekept behind a singular service provider, who can refuse access for particular uses it deems inappropriate. We&#39;ve got a decent amount of tools like this, but it&#39;s limited. To be fair, the actual utility of AI in general is limited. But still, the idea that this technology keeps coming out and we&#39;re all just going to be better off for it all, even though it&#39;s all held up and controlled by our corporate overlords is ridiculous. If this is going to be a Good Technology, then it&#39;s going to need to be a technology that People Actually Own, not one that they rent access to, one prompt at a time. Even better than that would be if it were being &lt;em&gt;designed&lt;/em&gt; by people who have a shared interest in the collective well-being.&lt;/p&gt;
&lt;p&gt;I genuinely think that Machine Learning (it&#39;s not actually AI, btw, that would require a level of &#39;intelligence&#39;) has a lot of real potential, especially the LLMs. It&#39;s just that everybody has got their heads so far up their own asses about whatever the hell the AI bros &lt;em&gt;think&lt;/em&gt; that it&#39;s going to revolutionize, that they can&#39;t see the potential of what it might &lt;em&gt;actually&lt;/em&gt; be able to do if we just stopped fantasizing about firing everyone for just 10 seconds.&lt;/p&gt;
&lt;h1&gt;Wrap up&lt;/h1&gt;
&lt;p&gt;Well, not exactly a traditional book review post, but I feel like I covered all the important parts. Major themes, general plot, implied conclusions about technology, and also the way we can turn the text to a modern interpretation. I hope yall enjoyed this less-than-traditional post from me. I enjoyed reading this book, and looking at the way that robots were thought of &lt;em&gt;before&lt;/em&gt; the major advent of the computer was interesting, in comparison to how we think of them now within pop culture.&lt;/p&gt;
&lt;hr&gt;
&lt;h1&gt;Check this out&lt;/h1&gt;
&lt;h2&gt;Some links after the blog&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Pluralistic: &lt;em&gt;It&#39;s still censorship (even if it doesn&#39;t violate the First Amendment)&lt;/em&gt; &lt;a href=&quot;https://pluralistic.net/2025/09/22/one-throat-to-choke/&quot;&gt;https://pluralistic.net/2025/09/22/one-throat-to-choke/&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;More Perfect Union: &lt;em&gt;I Took Bernie Into Deep Trump Country. Can He Win Them Over?&lt;/em&gt; &lt;a href=&quot;https://www.youtube.com/watch?v=RP8Oxe6OxJc&quot;&gt;https://www.youtube.com/watch?v=RP8Oxe6OxJc&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Snopes: &lt;em&gt;DOJ removed study from website showing most domestic terrorism is right-wing&lt;/em&gt; &lt;a href=&quot;https://www.snopes.com/fact-check/doj-removes-study-website/&quot;&gt;https://www.snopes.com/fact-check/doj-removes-study-website/&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The study in question:&lt;/li&gt;
&lt;li&gt;&lt;em&gt;What NIJ Research Tells Us About Domestic Terrorism&lt;/em&gt; &lt;a href=&quot;https://web.archive.org/web/20250911165140/https://www.ojp.gov/pdffiles1/nij/306123.pdf&quot;&gt;https://web.archive.org/web/20250911165140/https://www.ojp.gov/pdffiles1/nij/306123.pdf&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Vox: &lt;em&gt;Why obvious lies make great propaganda&lt;/em&gt; (2018) &lt;a href=&quot;https://www.youtube.com/watch?v=nknYtlOvaQ0&quot;&gt;https://www.youtube.com/watch?v=nknYtlOvaQ0&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
</content>
  </entry>
  <entry>
    <title>Happy Constitution Day</title>
    <link href="https://riverseeber.net/blog/post/happy-constitution-day/" />
    <updated>2025-09-17T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/happy-constitution-day/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;September 17, 2025&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Happy Constitution Day to everyone living in the United States! On this day, September 17, all the way back in 1787, the final draft of the US Constitution was signed, and would slowly make its way to the Congress of the Confederation, being ratified by the 13 states a year later, replacing the Articles of the Confederation.&lt;/p&gt;
&lt;p&gt;By the way, if you&#39;ve never the Constitution in full, or you haven&#39;t done it recently, I highly recommend you take today as an excuse to do so. It&#39;s fairly short, and can be easily read in a single sitting. It&#39;s one of the most important documents that relates to our current legal system in the US.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://web.archive.org/web/20240930125328/https://constitution.congress.gov/constitution/&quot;&gt;https://web.archive.org/web/20240930125328/https://constitution.congress.gov/constitution/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;During deliberation over whether the US should accept this new constitution or not, there were 2 factions that emerged on the issue. The Federalists, who promoted the constitution (which enacted a &amp;quot;federalist&amp;quot; system of governance), and the less creatively named Anti-Federalists, who were weary of potential structural dangers that might come associated with this new system. In order to quell some of the fears by the Anti-Federalists of authoritarian rule and the trampling of civil liberties, Congress passed the Bill of Rights — 10 amendments, codifying certain basic liberties as constitutional rights. The Federalists felt as though this was unnecessary, claiming instead that the &lt;em&gt;structure&lt;/em&gt; of the Constitution itself serves as a bill of rights. In Federalist 84, Alexander Hamilton wrote this on the subject:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;And the proposed Constitution, if adopted, will be the bill of rights of the Union. Is it one object of a bill of rights to declare and specify the political privileges of the citizens in the structure and administration of the government? This is done in the most ample and precise manner in the plan of the convention; comprehending various precautions for the public security, which are not to be found in any of the State constitutions. Is another object of a bill of rights to define certain immunities and modes of proceeding, which are relative to personal and private concerns? This we have seen has also been attended to, in a variety of cases, in the same plan. Adverting therefore to the substantial meaning of a bill of rights, it is absurd to allege that it is not to be found in the work of the convention.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://www.gutenberg.org/files/1404/1404-h/1404-h.htm#link2H_4_0084&quot;&gt;https://www.gutenberg.org/files/1404/1404-h/1404-h.htm#link2H_4_0084&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Hamilton was right in some ways. He&#39;s saying that the Constitution dictates how the government can operate. The government is to be a body composed of publicly elected and liable individuals consisting of varying constituencies from one representative to another. On top of that, it was specifically engineered to create contradictions of power between the different bodies of government, so as to prevent any individual body becoming too over-zealous with power. It creates limitations on which bodies, if any, can conduct certain types of actions. It bans certain actions altogether, such as retroactive application of laws (called &#39;ex post facto laws&#39;), while at the same time guaranteeing certain rights to citizens, such as the right to trial, (called the right to a &#39;writ of habeas corpus&#39;). It is a system that has been engineered to prevent tyrannical takeover.&lt;/p&gt;
&lt;p&gt;I want to go over that point again, because I think it&#39;s important. &lt;strong&gt;Our government was designed, in our Constitution, to prevent wannabe tyrants and dictators from trampling over the people, and to prevent them from seizing too much power.&lt;/strong&gt; Our Constitution was designed with an adversarial threat model for its own leaders. The way the Founders saw it, one of the biggest threats to The United States was our own elected representatives. They designed an entire system, explicitly in order to prevent those representatives — Congressmen, Judges, &lt;em&gt;Presidents&lt;/em&gt; — from becoming too powerful.&lt;/p&gt;
&lt;h1&gt;Testing Our Efforts&lt;/h1&gt;
&lt;p&gt;Right now, our country is getting a real life test of those safety measures. We&#39;re only 9 months into the second presidency of Donald Trump, and he&#39;s already ran through a number of structural barriers that have never before been circumvented. Structural barriers considered to be load bearing supports for a functioning democracy.&lt;/p&gt;
&lt;p&gt;He&#39;s suspended the right to trial for suspected immigrants, making it impossible for a legal citizen to prove that they are not, in fact, the immigrant that ICE thinks them to be:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Deportation_of_Kilmar_Abrego_Garcia&quot;&gt;https://en.wikipedia.org/wiki/Deportation_of_Kilmar_Abrego_Garcia&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;He&#39;s ignored court orders from judges because he didn&#39;t like their rulings, and had to be fought hand over fist to actually obey them finally:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://apnews.com/article/aclu-trump-deportations-el-salvador-boasberg-e447c61de031150669d01687edc4b11b&quot;&gt;https://apnews.com/article/aclu-trump-deportations-el-salvador-boasberg-e447c61de031150669d01687edc4b11b&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;He&#39;s been found to have illegally deployed National Guard troops on Los Angeles, with public plans to deploy on other big US cities (more specifically, big cities whose local politics swing left-wing or anti-Trump):&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://apnews.com/article/trump-newsom-los-angeles-national-guard-d6c8450a3ac2de34e669ef0836d22cbc&quot;&gt;https://apnews.com/article/trump-newsom-los-angeles-national-guard-d6c8450a3ac2de34e669ef0836d22cbc&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;He&#39;s fired certain public officials even when Congress has passed explicit laws insulating them from being fired by the President, such as members of the NLRB. He&#39;s discussing doing the same for members on The Federal Reserve, despite there being laws preventing such thing unless there is good cause to remove (not just partisan disagreement).&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.npr.org/2025/01/28/nx-s1-5277103/nlrb-trump-wilcox-abruzzo-democrats-labor&quot;&gt;https://www.npr.org/2025/01/28/nx-s1-5277103/nlrb-trump-wilcox-abruzzo-democrats-labor&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h1&gt;The Power of the Pen&lt;/h1&gt;
&lt;p&gt;In the more abstract, his slew of Executive Orders in the first months of his second presidency have been very explicitly clear grasps by the president for legislative power. The reason why this in particular is worrying is that the Executive and Legislative branches are setup in very structurally different ways, so as to create two very different modes of operation.&lt;/p&gt;
&lt;p&gt;I&#39;d like to draw once again upon The Federalist Papers, this time Federalist 70, another one of Hamilton&#39;s:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In the legislature, promptitude of decision is oftener an evil than a benefit. The differences of opinion, and the jarrings of parties in that department of the government, though they may sometimes obstruct salutary plans, yet often promote deliberation and circumspection, and serve to check excesses in the majority. When a resolution too is once taken, the opposition must be at an end.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;What he&#39;s describing here is the inherent &lt;em&gt;innefficiency&lt;/em&gt; of Congress. That inefficiency is the point. We shouldn&#39;t be passing laws in a &amp;quot;move fast and break things&amp;quot; kind of way. Laws are universally applicable, and if passed without much consideration, could have disastrous ramifications for all kinds of people. The great innovation of our legislative system in the United States is that it is so darn easy to prevent a &lt;em&gt;bad&lt;/em&gt; bill from getting passed. This way, disadvantaged minorities can leverage that ability to stop the bill in order to force the majority to add concessions to the bill before you allow it to go through and become law.&lt;/p&gt;
&lt;p&gt;In all of this fighting, a great amount of thinking about the bill must occur, and it will continuously go through various improvements, shedding off bad provisions, and perhaps even gaining new good ones that couldn&#39;t have been imagined until it came to the debate floor.&lt;/p&gt;
&lt;p&gt;On the other hand, once the bill has passed, Hamilton notes that it must be enforced without this same level of deep introspection. The law is the law, and once we&#39;ve slowly decided what the law is, we must now move to &lt;em&gt;quickly&lt;/em&gt; enforce it with haste and vigor. Thus, he argues, the Executive branch must have much energy. This forms the basis for the Framer&#39;s argument to have a single person at the head of the Executive branch — The President.&lt;/p&gt;
&lt;p&gt;Now, all of this talk about energy in the Executive might seem to support everything Trump is doing in all his vigorous actions as President. The Framers never argued against an energetic Presidency, so this is in line with their vision of America, surely?&lt;/p&gt;
&lt;p&gt;The issue is that Donald Trump is not just merely applying the laws that Congress has passed in an energetic manner. If he were simply executing the resolutions which had already gone through intense debate and compromise before becoming law, then the vigor would be appreciated. But Donald Trump is largely creating his own new plans for this country entirely independent of Congress or that deliberatory process. In many cases, as already stated, he is going directly &lt;em&gt;against&lt;/em&gt; policies mandated by Congress. What we are witnessing is not &#39;Energy in the Executive&#39;, but rather, &lt;em&gt;legislative energy&lt;/em&gt;, which just so happens to originate from the Executive branch.&lt;/p&gt;
&lt;h1&gt;Is This What America Needs Right Now?&lt;/h1&gt;
&lt;p&gt;There&#39;s an argument going around right now that our country needs all of this. That we need somebody like Trump to come in, destroying and subverting all limitations on his power, in order to actually accomplish important change in our country which has been tied up in governmental gridlock for the past 2 decades. Then, at the end of his 4 years, he&#39;s going to be all done with his title of supreme leader, and he&#39;s going to leave. Then, when the next President is elected, they&#39;ll just never be able to pull the same power plays as Trump, because apparently it&#39;s only possible for Trump to do it, and nobody else is charismatic enough to repeat the process after he&#39;s already set the precedent.&lt;/p&gt;
&lt;p&gt;The idea that &amp;quot;Too Much Democracy is hindering America, and is at the root of all our problems&amp;quot; is a common idea in the minds of a lot of Conservatives right now. Not to say that they &lt;em&gt;don&#39;t&lt;/em&gt; believe in Democracy, it&#39;s just that I&#39;ve personally seen a lot of complaining about &amp;quot;too much red tape in the system&amp;quot; coming from that side of the aisle. I mean, that&#39;s kind of what the whole DOGE project with Elon Musk was about. But it&#39;s also an underscore of the entire way Trump 2.0 is behaving.&lt;/p&gt;
&lt;p&gt;This is a recent turn of events for Conservatives. Traditionally, Conservatives have believed strongly in the core mission of the Constitution, or at least if they didn&#39;t, they pretended to. Back in 2011, Supreme Court Justice Antonin Scalia — a &lt;em&gt;famous&lt;/em&gt; Conservative justice who was appointed by none other than Ronald Reagan — sat before the Senate to talk about the role of Judiciary in today&#39;s world. He spoke about the way that the US is uniquely set up to &lt;em&gt;contradict power&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;  &lt;a href=&quot;https://www.youtube.com/watch?v=Ggz_gd--UO0&quot;&gt;https://www.youtube.com/watch?v=Ggz_gd--UO0&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;In the United States, a bill must pass before &lt;em&gt;two&lt;/em&gt; legislative bodies, with the same wording, before it is allowed to become a law. Those two bodies are elected in &lt;em&gt;very&lt;/em&gt; different manners. Then it is passed to the President, who can veto it if he doesn&#39;t like it. Again, the President is elected in a very different manner than either legislative house. There could be immense ideological disagreement between all 3 of them. Scalia explains that to people living in Europe, and sometimes even those within the United States, this entire system seems to amount to nothing more than a big machine that creates endless gridlock. That we can&#39;t get anything done.&lt;/p&gt;
&lt;p&gt;He says that&#39;s the wrong way to look at it. That we&#39;re supposed to love the gridlock. The gridlock is the point. He&#39;s echoing the same point that Hamilton was making in Federalist 70 where he wrote about the deliberation that was supposed to occur in Congress before a bill could get passed.&lt;/p&gt;
&lt;p&gt;Scalia was a &lt;em&gt;staunch&lt;/em&gt; Conservative. And his beliefs on this topic are part of what made him a &lt;em&gt;hero&lt;/em&gt; to the Conservative side of politics. What I&#39;m trying to demonstrate here is that there&#39;s nothing inherently Conservative about believing that we need to brush through this whole system of &lt;em&gt;intentionally engineered friction&lt;/em&gt; and lather it in lubrication in order to expedite the process. We do not want the process to be systemically expedited. We want the laws of this land to have been passed before the great machine of debate and deliberation that is our Legislative process. And so if these ideas are not inherent to Conservatism, then I&#39;ve shown that I&#39;m not attacking Trump for being Conservative, or rapidly instituting Conservative policy. I may dislike those things, but it is what it is. What I&#39;m criticizing him for here — and in the strongest way I know how — is for subverting America&#39;s founding principles of contradiction of powers, of checks and balances, of everything that the drafters of the Constitution were fighting for.&lt;/p&gt;
&lt;h1&gt;He Fancies Himself a King&lt;/h1&gt;
&lt;p&gt;Trump has decided that his preferred method of governing in this country is to ignore laws passed by Congress, and when Congress has decided not to act in a particular area (perhaps because their deliberation has led them to the conclusion that it would be improper to do so), he finds ways to inaccurately interpret laws that were never meant to serve for those purposes in order to create a legal basis to enforce his private policies.&lt;/p&gt;
&lt;p&gt;His basis for trying to invade US cities with the National Guard are prime examples. No law allows him to do this, except for if he claims there to be some kind of national &amp;quot;crime emergency&amp;quot; going on in these cities, a fact that is simply not true. He&#39;s also used a similarly faulty basis in how he&#39;s announced there to be a &amp;quot;foreign invasion of immigrants&amp;quot; into this country, which is the basis for which he is suspending the right to a fair trial before being deported. Without this trial, there is no way for legitimate citizens to prove that they are in this country legally. If you don&#39;t give immigrants a right to a trial before deporting them, it means you&#39;re fine deporting US citizens without a trial, since they will have no basis upon which they can prove their legal status if they&#39;re denied a court trial.&lt;/p&gt;
&lt;p&gt;The fact of the matter is that our President is governing by creating his own legislation. That is the way that kings of old ran their kingdoms. That&#39;s not how Democracies run, and it&#39;s definitely not how The United States of America runs. We do not have kings in this country, but it seems that our President disagrees with that fact.&lt;/p&gt;
&lt;p&gt;We&#39;re in a very scary place right now. I don&#39;t want to minimize the threat, but I&#39;d like to point out the fact that this is the one thing that this country was designed for. We&#39;re seeing a lot of fights right now legally against the president. Culturally, every time Trump gets held up for breaking the law, he decries it as a moment of a &amp;quot;rouge judge&amp;quot; trying to be a dictator over him. So he simply refuses to accept the role of checks and balances, so he tries to delegitimize it. He&#39;s ignoring and going against large portions of the law right now, but it&#39;s important to remember that he&#39;s still being held up. Courts are finding his actions to be illegal and forcing him to stop. There are still moments where Trump does not ultimately prevail.&lt;/p&gt;
&lt;p&gt;Just this month, a federal judge found that Trump&#39;s use of the California National Guard in Los Angeles was illegal. This is of course coming after most of the damage has already been done, but the ruling sets a useful precedent that this kind of thing will not be allowed to happen in other cities, such as Chicago, which the President has boldly asserted that he will be sending troops to at some undisclosed point in the future.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://thehill.com/regulation/court-battles/5481178-judge-rules-trumps-use-of-us-military-in-la-was-illegal/&quot;&gt;https://thehill.com/regulation/court-battles/5481178-judge-rules-trumps-use-of-us-military-in-la-was-illegal/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;People are fighting back against the administration, and some of them are actually winning some ground, even if it is an entirely unfair uphill fight against authoritarianism. The point is that as terrible of a predicament our country is in right now, &lt;em&gt;we are not helpless!&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;At the same time, I want to urge people to realize that the words coming out of Trump and his supporters mouths are explicitly &lt;em&gt;anti-American and unpatriotic&lt;/em&gt;. To break so many of the institutional limits within our Constitution, to decry the concept of checks and balances, it is to spit in the face of our founders and the American promise of freedom. The MAGA party loves to call themselves patriots, and to criticize Trump&#39;s opponents on supposed unpatriotic grounds. But the issue is that MAGA stands for values that are antithetical to America and our values laid out in the Constitution. Donald Trump thinks that anyone who stops him from assuming complete and utter control of the government is a rogue agent scheming to become their own dictator. Donald Trump believes that anybody in power who stops him from carrying out his illegal plans has overstepped their bounds. Donald Trump believes that he is entitled to do anything he wants. If you agree with him, you do not believe in America. I&#39;m tired of pretending that the MAGA faction that is destroying the infrastructure of checks and balances in this country &lt;em&gt;isn&#39;t&lt;/em&gt; un-American.&lt;/p&gt;
&lt;h1&gt;Fighting Back&lt;/h1&gt;
&lt;p&gt;If you&#39;re a concerned citizen who is scared of the way Trump is proceeding, might I perhaps persuade you to &lt;em&gt;do something about it!&lt;/em&gt; If you feel like Trump has massively overstepped his bounds as President, I can&#39;t urge you enough, &lt;em&gt;please&lt;/em&gt;, we need to go out and get organized! So, what on Earth can we do to stop this man from tearing up our Democracy?&lt;/p&gt;
&lt;h2&gt;No Kings&lt;/h2&gt;
&lt;p&gt;If you remember the &#39;No Kings&#39; protests that took place all across the country in June earlier this year to protest Trump, you might like to know that another day of protest has been planned, this time falling on October 18. If you&#39;re tired of President Trump acting as a dictator, you should go to this event. There are so many of these going on across the country. Even in my small, relatively Conservative town within California, there&#39;s a local event, so I don&#39;t have to drive to a big city to partake. You can check out different locations on the map they provide on their website:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.nokings.org/#map&quot;&gt;https://www.nokings.org/#map&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;If you do go, one thing I must urge you to do is to represent an energy of peace. In addition, don&#39;t allow them to label you as insurrectionists who hate America. We do not hate America. We love America, and we&#39;re showing up to this event because of that. We&#39;re showing up to protest Donald Trump because his actions ring in contradiction of this country&#39;s founding principles. We say No Kings because we believe that Donald Trump represents everything that this country is &lt;em&gt;not&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Bring American flags, bring printouts of the Constitution, The Federalist Papers, the Declaration of Independence. Don&#39;t burn flags, don&#39;t hang them upside down. Even if your intentions are patriotic or pro-American, that&#39;s not how it gets interpreted when those who disagree with you see it on social media. Do your best to represent what the anti-Trump faction really is.&lt;/p&gt;
&lt;h2&gt;Call Your Representatives&lt;/h2&gt;
&lt;p&gt;I know you&#39;ve heard this a million times before, but I&#39;m serious, call them. You can send emails too, though I&#39;ve heard that phone calls tend to be more effective. Pressure them to take action against Trump. Pressure them to speak out publicly on an issue that they&#39;re neglecting, even if the Conservative majority in Congress prevents them exerting much direct power against the administration. If your representatives are Conservative, &lt;em&gt;especially&lt;/em&gt; make a point to call them. Tell them that their support of Trump&#39;s undemocratic actions makes you lose faith in them. Tell them you want them to stop standing in line behind a man who hates this country. Tell them your vote is only cast for people who love this country.&lt;/p&gt;
&lt;p&gt;If you&#39;ve never called your representatives before, it&#39;s super easy. Here&#39;s a really good video tutorial on how to do it, if you&#39;re nervous. Don&#39;t worry, your representatives aren&#39;t going to be the ones picking up the phone, it&#39;ll just be an office secretary who takes down your information.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=GX_ERS7U_WQ&quot;&gt;https://www.youtube.com/watch?v=GX_ERS7U_WQ&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;Vote&lt;/h2&gt;
&lt;p&gt;This is a little preemptive, but it&#39;s important. The November 2026 Congressional election is going to be crucial if we want to stop some of the worse effects of the Trump takeover. It&#39;s common for there to be a blue wave in midterms during Republican presidencies, and vice versa. At the same time, there&#39;s an ongoing effort, initiated by Trump, to get red states like Texas to Gerrymander their district map outside of a census year, in order to squeeze out some extra Republican seats in the House. There&#39;s talk about a counter effort being taken on by various blue states, including California, but in either case, the vote for Congress incredibly important this midterm. If you want Congress to be able to actually fight back against this takeover, having a Democratic majority is going to be vitally important, since we know at this point that Republicans will only ever comply.&lt;/p&gt;
&lt;p&gt;I recommend making sure that you&#39;re registered to vote, even if you&#39;ve registered already. Databases can become corrupted, and you might need to register again. You also might need to update some information like your address or vote-by-mail preferences. You can check your registration in just a few easy clicks here. Just click &#39;Voter Registration Status&#39;, select your state, and it will take you to your state&#39;s voter registration website. If you&#39;d like to register, click &#39;Register To Vote,&#39; and proceed in the same manner.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.nass.org/can-I-vote&quot;&gt;https://www.nass.org/can-I-vote&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;You might not love Democrats, I sure as hell don&#39;t, but don&#39;t let that stop you from voting against a Republican who is only ever going to bend the knee to Trump. Vote for somebody who is not happy with the way things are going on right now in DC. Be a single-issue voter, where that issue is the maintenance of Democracy and our Constitution.&lt;/p&gt;
&lt;hr&gt;
&lt;h1&gt;Check this out&lt;/h1&gt;
&lt;h3&gt;&lt;em&gt;Some links after the blog&lt;/em&gt;&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;AP News: &lt;em&gt;Tracking the lawsuits against the Trump administration&lt;/em&gt; &lt;a href=&quot;https://apnews.com/projects/trump-executive-order-lawsuit-tracker/&quot;&gt;https://apnews.com/projects/trump-executive-order-lawsuit-tracker/&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;em&gt;On Not Surrendering in Advance (Or At Any Point Thereafter)&lt;/em&gt; &lt;a href=&quot;https://www.meditationsinanemergency.com/on-not-surrendering-in-advance-or-during-or-at-any-point-thereafter/&quot;&gt;https://www.meditationsinanemergency.com/on-not-surrendering-in-advance-or-during-or-at-any-point-thereafter/&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The Atlantic: &lt;em&gt;Apologies: You Have Reached the End of Your Free-Trial Period of America!&lt;/em&gt; &lt;a href=&quot;https://www.theatlantic.com/newsletters/archive/2025/09/america-free-trial-services/684072/?gift=jQN1t1D1nkO2TQodBiz5KLmz9qdi35_pconlf7F6jjg&quot;&gt;https://www.theatlantic.com/newsletters/archive/2025/09/america-free-trial-services/684072/?gift=jQN1t1D1nkO2TQodBiz5KLmz9qdi35_pconlf7F6jjg&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;em&gt;A notional design studio.&lt;/em&gt; &lt;a href=&quot;https://ethanmarcotte.com/wrote/a-notional-design-studio/&quot;&gt;https://ethanmarcotte.com/wrote/a-notional-design-studio/&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
</content>
  </entry>
  <entry>
    <title>The Web Was Designed to Be Open</title>
    <link href="https://riverseeber.net/blog/post/the-web-was-designed-to-be-open/" />
    <updated>2025-08-11T02:25:20Z</updated>
    <id>https://riverseeber.net/blog/post/the-web-was-designed-to-be-open/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 13, 2025&lt;/em&gt;&lt;/p&gt;
&lt;h1&gt;The Free and Open Web&lt;/h1&gt;
&lt;p&gt;The internet is a truly incredible thing. Founded on the principle of shared ownership, the internet does not belong to any one group or person, it is shared by all of us. There is no single group you must go through in order to access the internet, nor in order to host an online service. There is no single set of rules governing what is and is not allowed to be posted on the internet. The internet deters censorship because new sovereign territory can be created online simply by hosting a server. With that server, you are created a new space on the internet, a new node added to the network. Within that node, you are free to do anything you like, so long as it is legal in the country you&#39;re hosting it in -- otherwise you might get some knocks on your door IRL.&lt;/p&gt;
&lt;p&gt;The internet being this way means that it cannot be controlled, truncated, censored, or destroyed by any overzealous entities -- be they politicians, tech corporations, or otherwise anyone else with the power to attempt a move like that. The internet remains strong, unfettered by the whims of would-be autocrats.&lt;/p&gt;
&lt;p&gt;Contrast this model with the model we use for many of the online serves common throughout our society. Social Media is almost universally done using a centralized model. You use some service, say Facebook for example. You download an app provided to you by the Facebook corporation. When you want to post your social content online, you upload it through that app up to the Facebook servers. No new or sovereign land is created in this process, all media is retained on the already existing Facebook node. There is no way to host a new node, and let the content on that node be visible as regular Facebook content through the app.&lt;/p&gt;
&lt;p&gt;Even if you wanted to host a new server, some piece of sovereign territory upon the web. You would be free to do so, but your content would not show up on the Facebook network. Users would have to leave the Facebook app in order to view your content. There would be no way to view both Facebook content and your content using the same app.&lt;/p&gt;
&lt;h1&gt;The Free and Open Social Net?&lt;/h1&gt;
&lt;p&gt;But it doesn&#39;t have to be this way. What if we created a social media that worked the same way that the web does? Where the social network consists of distinct nodes controlled and operated by separate people? Those people could each set the rules of what is and is not allowed to exist on their node -- rather than having all content moderation be mandated by a single controlling authority like Facebook? Much like the web, we could create open standards for this technology. That way, anyone can create a program that can view this content. For the web, we called it Web Browsers, for social media, it will be a Social Media Client. A developer will have access to the standard, and will be free to implement any additional features, or organize the app in any way they see fit. We will not be bound and stuck with the design decisions of a single corporation. Users with different tastes will be enabled to cater to their own preferences for visual design of the interface that they want to use.&lt;/p&gt;
&lt;p&gt;Many people reading this who have been in tech and hacker spaces long enough may be thinking right now of the decentralized alternatives to corporate social media. Mastodon and Bluesky both come to mind as possible contenders. For Bluesky, I will be brief and say that it is 1) not ready yet as a developed platform, and 2) it is not making enough significant strides for true decentralization. If Bluesky were to disappear from the internet tomorrow, there would not be a feasible way for the rest of the community to continue hosting the network without updating the protocol first.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://tormentnexus.substack.com/i/152261629/a-protocol-not-a-platform&quot;&gt;https://tormentnexus.substack.com/i/152261629/a-protocol-not-a-platform&lt;/a&gt;&lt;/p&gt;
&lt;br&gt;
As for Mastodon, while it is a truly decentralized network that can be entirely run without the help of any centralized authority, the underlying ActivityPub protocol that it relies on is both complicated and confusing, as well as overly vague, leaving many important questions unanswered in the specification, leading to much confusion for developers. The protocol is perhaps a good basis on which to build an actually concrete protocol, but in my own honest opinion, it is overly complicated, and a drain of the valuable time and resources of an open source community of intelligent developers. 
&lt;p&gt;Dennis Schubert, a project manager for the Diaspora* project -- another attempt at a social media network, although much smaller that both Bluesky or ActivityPub -- wrote a pretty good article on the issues with the ActivityPub protocol. In that article, he also lays out some of the bigger flaws in the existing Diaspora* project that make it also unsuitable in many ways, but I&#39;ll leave that to you to look into.&lt;br&gt;
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://overengineer.dev/blog/2019/01/13/activitypub-final-thoughts-one-year-later/&quot;&gt;https://overengineer.dev/blog/2019/01/13/activitypub-final-thoughts-one-year-later/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h2&gt;A Clean, Simple Design&lt;/h2&gt;
&lt;p&gt;I propose a simpler way of doing things. I&#39;m not going to put forward a concrete protocol, as I&#39;m only just starting to work on these ideas myself, but I&#39;d like to lay down the groundwork for what I think should someday become a solid protocol. The so-called &amp;quot;UNIX design principles&amp;quot; promote simplicity and modularity in all things that you create. Each module of your project should do one thing, and do that one thing well. Further modules can be appended to the overall project in order to create a program (or protocol, in our case) that can do many things. Our new protocol would work something like this. You may ignore the more specific details of this description in favor of the general principles I&#39;m trying to isolate in how I create this example. The point is simplicity, which we will achieve through modularity.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://en.wikipedia.org/wiki/Unix_philosophy&quot;&gt;https://en.wikipedia.org/wiki/Unix_philosophy&lt;/a&gt;&lt;/p&gt;
&lt;br&gt;
You would have two sides to the network. There would be clients, AKA &quot;users&quot;, and then there are the servers, which host the content for the users. A user would be free to host their content in any number of ways. It would be a very similar experience to that of hosting a web server. You are free to host it yourself if you know how to, or you may turn to any number of middle men who will do it for you. In the world of the web, this would be services like Wordpress, Medium, NeoCities, GitHub Pages, etc. Unlike ActivityPub, very little server-to-server interaction would occur, if any. When you as a user want to look at a post by your friend, your client software would request the content directly from their server. If you want to interact with that content (commenting, liking, etc), that interaction would be sent to and stored directly on the server where the original post is stored. The post would not be shared around with other servers like it does in the ActivityPub protocol. This is done to simplify things, and reduce scaling difficulties.
&lt;p&gt;As for the design of the clients, and the specific toolings used, feed content would be generated as an RSS feed for each user. So if you are following 5 friends, your client would fetch 5 RSS feeds at refresh time, looking for any posts inside of those 5 feeds that you have not yet seen, and then putting it on your timeline. Content itself would likely be stored as HTML or XML, or something of the sort. The benefit being that we do not need to teach the client software how to handle all 48 thousand content types, from image posts, to videos, to microblogs, to audio notes, to story posts. Instead, we could just encode the content using an extensible format like html or xml that support embedding tags for any content type imaginable. If you want to do an image post, your client software will simply embed an &lt;code&gt;&amp;lt;img&amp;gt;&lt;/code&gt; tag into the post. For microblogging, it will put text inside of the &lt;code&gt;&amp;lt;p&amp;gt;&lt;/code&gt; tag. Use a format like html and you can encode anything you desire, even things which the designers of this future protocol couldn&#39;t have imagined.&lt;/p&gt;
&lt;p&gt;The idea here is that we&#39;re re-using existing protocols and tools. This lets us re-use technologies for understanding those protocols. Web apps already know how to render HTML, we can simply use the existing engine for reading it. There are plenty of libraries around for reading RSS and XML feeds as machine readable data.&lt;/p&gt;
&lt;p&gt;In keeping with this idea, we would also want to create some kind of direct messaging service within the standard, for when users would like to communicate with one another. Perhaps we could use a protocol like XMPP or something similar. The fantastic benefit of this is that smaller servers which don&#39;t want to handle storing direct messages could offshore their implementation of XMPP to another server that they simply point to in their metadata file for the user. Perhaps a server might have a user called Alice. She has an account on this server at &lt;code&gt;https://example.com/alice&lt;/code&gt;, but the metadata file (perhaps &lt;code&gt;https://example.com/alice/index.xml&lt;/code&gt;), lists that her messaging endpoint is at &lt;code&gt;https://external-server.net/alice&lt;/code&gt;. Thus, when a user wants to message Alice, they will send an XMPP message directly to Alice at that endpoint on a separate server. Perhaps our example server would even handle the user credentials for the external server, making sure login details always match with the main server.&lt;/p&gt;
&lt;p&gt;Should they wish, a user would even be free to hand pick the XMPP server they want to use. Maybe they could do so in order to reduce the number of message inboxes they have, so they can re-use an already existing account to use for their social media DMs as well.&lt;/p&gt;
&lt;h1&gt;Closing&lt;/h1&gt;
&lt;p&gt;Perhaps some of the specific recommendations for the protocol wouldn&#39;t be perfect. A lot more thinking needs to go into this kind of thing. But as it stands right now, we are very dry on alternatives to the corporate social media platforms that are seen by many as the only real choice available. Creating social media in a way that mimics the shape of the free and open Web in my estimations would leave the world better off, assuming we could actually get it to take off from the ground, achieving a tangible level of success in terms of users and community support.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Use Technology Creatively</title>
    <link href="https://riverseeber.net/blog/post/use-technology-creatively/" />
    <updated>2025-08-11T02:25:17Z</updated>
    <id>https://riverseeber.net/blog/post/use-technology-creatively/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 16, 2025&lt;/em&gt;&lt;/p&gt;
&lt;h1&gt;Email is magic&lt;/h1&gt;
&lt;p&gt;A good email client is like magic. As technologists, it&#39;s easy to lean into the world of the terminal and operating system, messing around with our Linux installations, Desktop Environments, or other things. It&#39;s easy to lean into this world and see this part as the &amp;quot;real world of computing&amp;quot;, and then look at many of the more business and corporate tools such as email and Office Suites (like Microsoft Office or LibreOffice) as the boring technology, stripped of life.&lt;/p&gt;
&lt;p&gt;After all, look at your email inbox that you&#39;re forced to use for every service you sign up for. It&#39;s littered with corporate newsletters you didn&#39;t sign up for, advertisements from companies you don&#39;t even have an account with, and political updates from politicians who you thoughts you would care about when you signed up, but you&#39;ve received over 25 messages in a row so far that have remained unread. (Well, that&#39;s my experience anyways).&lt;/p&gt;
&lt;p&gt;It&#39;s easy to look at this and simply blame the corporate style of technology. But email isn&#39;t actually a bad technology, it&#39;s just that most people don&#39;t realize how good it can be if you simply take a few small proactive steps to improve your experience.&lt;/p&gt;
&lt;h2&gt;Manage Your Subscriptions&lt;/h2&gt;
&lt;p&gt;Step number one to gain control is to manage how many new messages are coming in to your inbox. My preferred method for this is to maintain two email accounts. The first account you identify as &lt;em&gt;your email account&lt;/em&gt;. It&#39;s the email you list on your website, and it&#39;s the one you give to contacts. You also will give it to any services for which you &lt;em&gt;actively want to recieve updates from&lt;/em&gt;. Not weird marketing emails from a service you don&#39;t care to use for more than a week after signing up. Not sales ads from an online store you bought one thing from 2 months ago. Updates that you specifically, intentionally thought to yourself, &amp;quot;yeah, I think I want to get emails from these people, that would enrich my life or help me out in some way.&amp;quot; For the other services, the ones that send you junk mail you don&#39;t care for, but which you still need an email to sign up with, you sign up through your second email. This inbox does not get checked unless they tell you to check it for something like a 2FA code or a password reset or something like that.&lt;/p&gt;
&lt;p&gt;The majority of services you use shouldn&#39;t get access to your &amp;quot;real&amp;quot; inbox, they get access to the secondary one. The one you don&#39;t check. This reduces the amount of messages you get in your first inbox, allowing you to actually read those few numbers of emails that actually &lt;em&gt;do matter&lt;/em&gt;. It also means you can confidently hand your email out to contacts and not get stressed out to the prospect of having to check it once in a while so that you don&#39;t miss their message. Herman from Bearblogs mentions something along the lines of this strategy in his post about digital hygiene.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://herman.bearblog.dev/digital-hygiene-emails/&quot;&gt;https://herman.bearblog.dev/digital-hygiene-emails/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;This method can be further enhanced by using email filters to automatically sort emails into different folders of your inbox, though I personally don&#39;t receive enough emails in my &amp;quot;real&amp;quot; inbox to warrant that level of organization. The 4 emails a week I get are easy enough to handle without the extra tooling. That may change if I begin receiving more emails from contacts and real humans.&lt;/p&gt;
&lt;h1&gt;It&#39;s Not Just Email&lt;/h1&gt;
&lt;p&gt;Fixing your email is a great move, it means you don&#39;t miss things. You come off to others as a more put together person. It reduces stress personally from your life. And the fact that it&#39;s such a simple move really is a major selling point on this idea. You don&#39;t need to do anything drastic, just make a new email (or clean up an existing one, unsubscribing from a few emails you&#39;re currently getting on it), and maybe apply some inbox filters if you plan on getting a lot of important messages. This idea is great, and it has a lot of room for further application. We can use the full potential of computing to become more efficient in as many ways as we feel creative, inspired, and compelled to explore. I&#39;d like to detail some of these other tricks you can use. All of these are going to be 1) simple and 2) offer non-negligible payouts for minimal effort.&lt;/p&gt;
&lt;h1&gt;Text Documents&lt;/h1&gt;
&lt;p&gt;Having a folder on your computer, just for general working notes is really important to me. I think anyone who works on a computer should have such a thing. Inside of it, you can include all kinds of things. There&#39;s of course your general todo lists (I do my day-to-day lists analog, keeping only big picture stuff digital), but then there&#39;s all the notes you need for remembering small details on a project you&#39;re handling, or for creative thinking while working through a fuzzy problem, or for keeping track of changing details regarding something as time goes by.&lt;/p&gt;
&lt;p&gt;I personally advocate for using them as simple text files, using .txt or .md extensions. You don&#39;t need to use a Word Document file to do this. Text files are easy to open -- virtually any program will accept them -- and they don&#39;t obscure anything. Their pure simplicity is a factor of freedom, not constraint. The fact that it&#39;s just text, often rendered in a monospace font, means you are given a free canvas with which to format and organize your document. The simplicity is the point.&lt;/p&gt;
&lt;p&gt;If you don&#39;t want to use a raw text file, don&#39;t. But if you&#39;re like me, and are comfortable editing files from raw text editors like vim, or on other operating systems, Notepad, I recommend doing it this way.&lt;/p&gt;
&lt;p&gt;(As a side note, if you happen to be looking for a better way to keep all of your endless notes organized, I recommend giving VimWiki a look. It&#39;s an easy to install extension for Vim, the terminal-based text editor. It allows you to put hyperlinks into your text documents that point to one another. It&#39;s also got a lot of quality of life features that are great, but I don&#39;t want to derail too far by going into depth about it. I&#39;ll let you look into it yourself.)
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://vkc.sh/vimwiki-101/&quot;&gt;https://vkc.sh/vimwiki-101/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://tinkerbetter.tube/w/6hrXmYYNMjBGJnSE8iJ6NM&quot;&gt;https://tinkerbetter.tube/w/6hrXmYYNMjBGJnSE8iJ6NM&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://vimwiki.github.io/&quot;&gt;https://vimwiki.github.io/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;The use of these files is incredible. Not just for writing simple notes, but you can generate detailed logs which will be useful to you as a reference guide. Take, for instance, the idea of a Suspense File. This list is used in order to track not your own progress, but to track things you&#39;re waiting on others to do. It&#39;s not designed to track them in some controlling, &amp;quot;hurry up and stop making me wait&amp;quot; kind of way. It&#39;s more like &amp;quot;oh wow, me and Rachel were talking about doing xyz, but it&#39;s been 2 weeks since any news from her. She&#39;s probably too busy to make it all work out in time. Let&#39;s take a look. If she&#39;s making progress, it would be good to know about it, and if not, I&#39;ll give her an out and let her off the hook if she needs it.&amp;quot; The idea is that people are busy, and the natural tendencies of people working together on a stagnant project is to just let it silently fall through. The point of the suspense file is to make sure if it does fall through, it&#39;s because you identified that one of you lacked in the time or resources necessary to make it work, and explicitly communicated that you wanted to drop the project.&lt;/p&gt;
&lt;p&gt;That tends to be the more useful aspect of using lists. Lists don&#39;t help you know what to do, they help you decide what you &lt;em&gt;aren&#39;t&lt;/em&gt; going to do. Which projects you don&#39;t have time for, and need to ditch.&lt;/p&gt;
&lt;p&gt;The suspense file works like this. Each line is a new entry. You can organize it however you like, but this is the method I use, and it&#39;s the one Cory Doctorow describes, which is where I learned of the concept in the first place.&lt;/p&gt;
&lt;p&gt;Each line starts with &amp;quot;&lt;code&gt;WAITING&lt;/code&gt;&amp;quot;, followed the category. After that, you describe who you&#39;re waiting on, and for what specifically. Finally, add a trailing string of dates for whenever a new development is made. If you communicate back and forth with someone for 3 weeks on a project, you might see over a dozen dates appended to the end of this line. The idea is that it gives you an idea of how long you&#39;ve been trying to get the ball rolling on something, and how many interactions have taken place so far.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://pluralistic.net/2024/10/26/one-weird-trick/&quot;&gt;https://pluralistic.net/2024/10/26/one-weird-trick/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;An example (fake) entry might look like this:&lt;/p&gt;
&lt;p&gt;  &lt;code&gt;WAITING Email from Prof. J on CS Club availability for advisorship 2/12, 2/16, 2/20&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;After 3 separate interactions about it, I might decide that she&#39;s too busy and start looking into other options. On the other hand, she might have just been temporarily bogged down, but will be free again starting next week, and that fact just needs to be communicated. Either way, it helps me identify the progress or stagnation of a plan, and easily track whether I need to reach out.&lt;/p&gt;
&lt;p&gt;Again, the point isn&#39;t to control anyone, or be an incessant nag (that would be weird). The point is to identify which projects are lagging behind secretly without being noticed, so I can communicate with others involved, and see if this is something that can be remedied somehow, or if we need to drop the project in pursuit of other aims that are demanding more of our time.&lt;/p&gt;
&lt;h1&gt;Keeping Up With Updates&lt;/h1&gt;
&lt;p&gt;In our age of information overload, where we are often expected to stay up to date with a number of ever changing events at 400 words a second, it can be helpful to institute some simple practices that greatly help us to manage all that information we&#39;re expected to take in regularly. We want both to be able to quickly have access to that information without too much mental strain on actually &lt;em&gt;trying to find it&lt;/em&gt;, as well as wanting a system in place that allows us to know which information sources we &lt;em&gt;still haven&#39;t looked at yet&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;The first is simple, we want some database of all our sources like blogs, news sites, product updates, etc in a singular place where we can quickly check all of them. The second goal is a little more nuanced. Perhaps we spent the first 15 minutes of the day checking through our updates, but get cut off before checking the last one. Or, for instance, perhaps we&#39;re following a large number of publication sources, but each source only puts out a small number of articles per month. Many times we check, there will be nothing new. It would be worth having a kind of system that keeps track of which source has received updates lately, and which ones we don&#39;t need to bother checking in on yet.&lt;/p&gt;
&lt;p&gt;Lucky for us, a tool for this already exists that covers both of our needs at once. It&#39;s called RSS feeds (or sometimes &amp;quot;Atom feeds&amp;quot;. I think there&#39;s a nuanced difference between them, but from the perspective of a user, they function exactly the same). You can follow the RSS feed of a publication source online, such as a blog or many news sites. You will store these feed links into an application called an RSS Reader. There&#39;s a lot of apps to choose from, but I personally like to use Mozilla Thunderbird for it. That way, it&#39;s stored in a folder right next to my emails inside of the same program. You can use something else if you like, there&#39;s plenty of options to choose from if you just care to do a quick search online.&lt;/p&gt;
&lt;p&gt;When you subscribe to a particular feed through your app, it shows up as a little folder of updates. When you click on the folder, it shows you any recent posts from that source. Unread posts are highlighted a different color so you know you haven&#39;t seen them yet. You can read the title of each post and decide if it&#39;s worth your time or not. I&#39;m currently following over a dozen feeds, though checking through them often only takes a little bit of time each day. Some days, I use it to keep up with light, enjoyable reading from popular posts on the Bearblogs blogging platform.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://bearblog.dev/discover/&quot;&gt;https://bearblog.dev/discover/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;I also use it to keep up with the news, dev updates relating to Arch Linux, or other projects I&#39;m following, podcasts I listen to, and noteworthy blogs I follow outside of the Bearblogs trending page.&lt;/p&gt;
&lt;p&gt;Most sites that post regular articles will have an RSS feed you can follow somewhere. Heck, even YouTube provides one for each channel, even if they don&#39;t necessarily advertise about it, or make it easy to find. I use an online tool to convert channel links into RSS feeds.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://ytrss.pesky.moe/&quot;&gt;https://ytrss.pesky.moe/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h2&gt;Without RSS&lt;/h2&gt;
&lt;p&gt;If you don&#39;t have an RSS feed for a site, you get two options. Either don&#39;t follow their updates, or find a way to follow it anyways. An easy way to do the second option is to create a basic &amp;quot;startup&amp;quot; bookmarks folder in your web browser. Every time you check your RSS feeds, you&#39;ll also open that startup folder, comb through if anything demands your attention, closing each tab once you&#39;re finished. This should take usually take a few minutes each day, unless you&#39;re putting a bunch of news sources into this folder, which I would not recommend.&lt;/p&gt;
&lt;p&gt;You can also use these folders to check up on special pages that require you to log in to your account to view. If you&#39;re trying to follow the shipping on a particular product you&#39;ve purchased, or check if some important lab result from your hospital has been posted to your account yet, rather than stressing about it, you can just right click your folder and &amp;quot;Open All Bookmarks&amp;quot;. Comb through if an update has been posted, and be done in less than 5 minutes. Once you&#39;ve received the update you were waiting on, you can delete the bookmark if you no longer need to check it regularly. You can always add it back again later if you get more labs done, or buy another product.&lt;/p&gt;
&lt;h1&gt;Conclusion&lt;/h1&gt;
&lt;p&gt;The point is that each of these tricks are &lt;em&gt;dead simple&lt;/em&gt;. They each provide moderate to high payouts for realistically small effort required to institute them. They utilize the power of computing to solve problems that computers are uniquely capable of solving efficiently.&lt;/p&gt;
&lt;p&gt;This article has largely been inspired by the general principles laid out in Cory Doctorow&#39;s post on Suspense Files (linked above), as well as his Lifehacking liveblog he links to at the beginning of that post. The idea that people who are keen with technology can use that technology in ways beyond just what everyone else is already doing (often without much thinking on the matter). To actually think critically about the tools at your disposal and find clever ways to use them that makes you accomplish more, it&#39;s a simple idea but it&#39;s also incredibly powerful.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://craphound.com/lifehacksetcon04.txt&quot;&gt;https://craphound.com/lifehacksetcon04.txt&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;I&#39;m only really just starting down the path of experimenting with and discovering creative ideas like the ones I&#39;ve laid out in this post. If you&#39;ve got any of your own that I didn&#39;t mention and you think I&#39;d like it, I&#39;d be happy to hear from you. You can feel free to shoot me an &lt;a href=&quot;https://rseeber.github.io/about.html&quot;&gt;email&lt;/a&gt; telling me all about it. I love getting emails from real people -- it&#39;s the explicit reason that I ever went about cleaning my inbox in the first place.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Get Off Social Media, Embrace The Web</title>
    <link href="https://riverseeber.net/blog/post/embrace-the-web/" />
    <updated>2025-08-11T02:25:14Z</updated>
    <id>https://riverseeber.net/blog/post/embrace-the-web/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;July 27, 2025&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Lots of people right now feel inundated by information. In our digital age, it&#39;s like everything is flying at us all at once, and we&#39;re always just trying to sip from this super-powered firehouse of information. This is largely the fault of social media. Some people today, especially those of us in our 20s, are looking for an escape from this. People are trying to get away from social media, but we don&#39;t want to stop connecting with our friends. We all know that the internet is a great thing! We just want to get away from some of the ills presented by the usage of social media (which is just one of &lt;em&gt;many&lt;/em&gt; ways to use the internet).&lt;/p&gt;
&lt;p&gt;Many people will have many different strategies for doing this, but in this blog, I&#39;ll be presenting a kind of guide for how to use the World Wide Web as a replacement technology -- a new (old) way to stay online, without having to conform to the limiting format of a centralized service like Facebook or Instagram.&lt;/p&gt;
&lt;h1&gt;Why The Web?&lt;/h1&gt;
&lt;p&gt;The point of the internet is to facilitate the mass spread of information, without necessarily always needing to go through traditional publishers or gatekeepers. Anyone with an idea can spread it. But when all of that information goes through the bottleneck of a small collection of major platforms -- platforms that can control what is and isn&#39;t allowed -- doesn&#39;t it seem that we&#39;ve lost the point of it all?&lt;/p&gt;
&lt;p&gt;And even if you&#39;re not going at it for publishing subversive ideas or something like that, the web is still better than our existing social media platforms. The web is an open format. The web is less conducive to mindless consumption. The web encourages more intentional use. Every post you see is because you clicked a link and went there. Content exploration is also easier. You get recommended things online by people who share links to them. Those links post to external sites, not just more content hosted on the same service. There&#39;s no monolithic algorithms over-analyzing your decisions and coming up with recommendations that show up on your &amp;quot;feed&amp;quot;. You have to explore. This already weeds out a lot of the more negative elements of social media, since it requires you to put thought into what you do.&lt;/p&gt;
&lt;p&gt;It leads to less negativity from people in the comments, and less anxiety created by impulsive scrolling.&lt;/p&gt;
&lt;p&gt;But all of this requires there to be a community on the web. A community of people who put their things on the web instead of social media, and who offer out links to interesting things they found recently.&lt;/p&gt;
&lt;p&gt;So let&#39;s begin, how do we get on the web, and what can we do once we&#39;re there?&lt;/p&gt;
&lt;h1&gt;Getting on the web&lt;/h1&gt;
&lt;p&gt;The easiest method for getting on the web is to just sign up for a blog. You don&#39;t really have to make it function like a traditional blog though. Blogs are just websites that don&#39;t require you to learn HTML in order to create. You don&#39;t need to be writing entire articles and essays on here. You can just use it as an easy way to have a webpage and make social posts about yourself.&lt;/p&gt;
&lt;p&gt;On Instagram, perhaps you might have posted a dozen pictures you took when you went to an event, with a one or two sentence long caption underneath. You could do the same thing here. Post your images, and include some text at the bottom (read ahead for how to embed your images if your blog provider doesn&#39;t host them for you). But you&#39;re also free to expand from this if you like. You&#39;re no longer creating something for a bored, apathetic audience with a low attention-span, who will ultimately be looking at your post through an algorithmic feed of junk. On the web, the people looking at your posts are likely to have longer attention-spans. If you wanted to write a whole paragraph or two about your experience, you&#39;d be free to do so, without feeling as though people will scroll past and not read what you&#39;re trying to express.&lt;/p&gt;
&lt;p&gt;Basically, you can use a blog, or the web in general, as a way to post the same type of thing you would normally post on social media. The benefit, however, is that you don&#39;t have to cater to the low-attention span that social media creates in people while they&#39;re using it. When you&#39;re on your computer browsing the web, you&#39;re often willing to spend more time looking at a single post than you would be if you were scrolling your phone while bored in line, for instance.&lt;/p&gt;
&lt;p&gt;So, let&#39;s get into the more specific guide of &lt;em&gt;how to actually do it&lt;/em&gt;.&lt;/p&gt;
&lt;h1&gt;The Guide&lt;/h1&gt;
&lt;h2&gt;The Blog&lt;/h2&gt;
&lt;p&gt;First thing&#39;s first, you need to sign up for a blog service. There&#39;s a lot of options out there of course. Honestly, if you wanted, you could even just run a blog by making your own website, but doing all that isn&#39;t necessary. My personal recommendation is to sign up with Bear Blog. They&#39;re great. It&#39;s a small, independent blogging service. They&#39;re simple, but offer plenty of customization. And the whole thing is entirely free. There&#39;s a paid tier that unlocks some extra stuff, but all of the core features you expect from a blog are available without needing to pay for anything.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://bearblog.dev/&quot;&gt;https://bearblog.dev/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Anyways, you&#39;re going to sign up with something. If you&#39;d prefer a bigger name, you can use something like Substack, Medium, Blogger, or anything else. All these services offer more less the same thing -- a place online that belongs to you.&lt;/p&gt;
&lt;p&gt;You might also go ahead and pick a theme for your blog, or even make a custom one yourself, but that&#39;s not strictly necessary, it&#39;s just nice is all. Some blogging services have a few pre-built themes you can just click and apply to your page, which makes things a lot easier.&lt;/p&gt;
&lt;h2&gt;Hosting Images&lt;/h2&gt;
&lt;p&gt;At a certain point, you might get the idea that you want to post some images. This is where we&#39;ll need to invoke the use of another service, since most blogs limit the amount of images that they&#39;ll host for you. We&#39;ll want to find a service that hosts images that can be referenced with a permalink to the image file. The best option I&#39;m aware of for this is Imgur, although alternatives definitely exist. We can make an account on Imgur, upload our images, then right click &amp;quot;open image in new tab&amp;quot; and copy the image URL.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://imgur.com/&quot;&gt;https://imgur.com/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;This URL is what we will use to embed it into our blog post. This way, the reader can see the image without having to click on any link. If you&#39;re using a service that uses a markdown format, you can embed an image like so:&lt;/p&gt;
&lt;p&gt;  &lt;code&gt;![Image description](image url)&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Where you replace &#39;image URL&#39; with the URL that you copied, and ideally also fill in an actual description as well.&lt;/p&gt;
&lt;p&gt;If you&#39;re using a visual editor for your blog, you&#39;ll have to dig around for yourself for any image embed buttons.&lt;/p&gt;
&lt;h1&gt;Examples&lt;/h1&gt;
&lt;p&gt;Okay, so you&#39;ve seen the guide, but now you wanna see what the final product should look like in action. I&#39;ve included two examples. The first is a blog designed by me. You can notice how I&#39;m not trying to be professional or sound like an essayist or anything. I&#39;m just writing like as if this were Twitter or Mastodon or something. There&#39;s no reason to try to be super polished about things unless that&#39;s what you wanna do. You have to think about this in the same way you would think about social media. Do you feel the need for everything on Instagram to be high fidelity content? If not, why should you feel any differently about a blog where you more less post the same type of stuff.&lt;/p&gt;
&lt;p&gt;The second example was done by my friend Trezha, who worked on this idea with me of how the web can be used as a better way to connect online.&lt;/p&gt;
&lt;p&gt;&amp;quot;Post-Roadtrip Thoughts&amp;quot; (by me)
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://riverpunk.bearblog.dev/post-roadtrip-thoughts/&quot;&gt;https://riverpunk.bearblog.dev/post-roadtrip-thoughts/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;&amp;quot;I went to a Concert&amp;quot; (by Trezha):
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://myfirst.bearblog.dev/i-went-to-a-concert/&quot;&gt;https://myfirst.bearblog.dev/i-went-to-a-concert/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h1&gt;Closing&lt;/h1&gt;
&lt;p&gt;The web is an amazing tool that allows for personal creative expression, and enables us to share those expressions with our friends. It&#39;s a tool that should connect us together. I find that social media doesn&#39;t always promote meaningful connection, and a lot of people my age seem to agree with me. We feel that there are better potential ways to connect with one another than just through the walled garden of social media.&lt;/p&gt;
&lt;p&gt;I hope you&#39;ll consider creating a space for yourself on the free and open web, as opposed to the locked up platforms of Instagram, Facebook, Twitter, or otherwise. You don&#39;t have to leave your social media accounts if you get on the web, you can have both if you like. I just feel that society would be better off if more people felt free to create and post things on the web, and so that content could become linked together. We would develop longer attention spans, the things we post would become more meaningful (even if the content itself didn&#39;t change much), and people would be free to view it without needing to sign up through an authoritarian tech platform that values money over people.&lt;/p&gt;
&lt;hr&gt;
&lt;br&gt;
&lt;p&gt;&lt;em&gt;This article, as well as mine and Trezha&#39;s example posts, were made as part of our team&#39;s submission for the &lt;a href=&quot;https://csgirlies.devpost.com/&quot;&gt;CS Girlies Hackathon&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>How I use VimWiki for note taking</title>
    <link href="https://riverseeber.net/blog/post/how-i-use-vimwiki-for-note-taking/" />
    <updated>2025-08-10T00:00:00Z</updated>
    <id>https://riverseeber.net/blog/post/how-i-use-vimwiki-for-note-taking/</id>
    <content type="html">&lt;p&gt;&lt;em&gt;August 10, 2025&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;I made an earlier post where I tried my best to inspire folks to use simple tech solutions in strategically useful ways in order to get large gains out of the tech they already have. Among one of the things I discussed was the power of using simple text files to keep track of things. I&#39;d like to expand on that point here and give some of my own technical details of what I use. Of course, any old text editor will work for the job. I&#39;m going above and beyond so I can have extra features and efficiency.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://rseeber.github.io/blog/post/use-technology-creatively/&quot;&gt;https://rseeber.github.io/blog/post/use-technology-creatively/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h1&gt;VimWiki&lt;/h1&gt;
&lt;p&gt;So we&#39;re going to be using VimWiki. It&#39;s an extension for the terminal-based text editor vim. First, install vim. After that, the README for the VimWiki project tells you to simply git clone the project into a plugins folder, and run a special command to generate the help pages. It&#39;s all just copy paste into the terminal, super easy stuff.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://github.com/vimwiki/vimwiki?tab=readme-ov-file#installation&quot;&gt;https://github.com/vimwiki/vimwiki?tab=readme-ov-file#installation&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Veronica Explains did a pretty good blog post and video tutorial about using VimWiki, which you can take a look at if the official documentation isn&#39;t enough for you. My post is mostly going to be about what you can &lt;em&gt;do with it&lt;/em&gt;, not necessarily a tutorial on the details.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://vkc.sh/vimwiki-101/&quot;&gt;https://vkc.sh/vimwiki-101/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://www.youtube.com/watch?v=RmEtH5FQs28&quot;&gt;https://www.youtube.com/watch?v=RmEtH5FQs28&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h1&gt;Using it&lt;/h1&gt;
&lt;p&gt;Okay, so now you&#39;ve got it installed. Open vim, and type &amp;quot;&lt;code&gt;&#92;ww&lt;/code&gt;&amp;quot; to enter VimWiki. It will create a folder in your home directory called &lt;code&gt;~/vimwiki/&lt;/code&gt; where all files will go by default. VimWiki is basically just vim with internal hyperlinks to other text files on your computer. When you create a link to a file and open it, it will automatically &lt;em&gt;create&lt;/em&gt; the file, if it doesn&#39;t already exist. You never need to leave the editor in order to add new files.&lt;/p&gt;
&lt;p&gt;You can of course use nested directories for different types of notes. For instance, you might use &lt;code&gt;~/vimwiki/blog/&lt;/code&gt; for writing up your blog posts, and &lt;code&gt;~/vimwiki/daily/&lt;/code&gt; for all the notes you need at the beginning of your day. That kind of thing. All links are relative, so if you add an index page to &lt;code&gt;~/vimwiki/blog/&lt;/code&gt;, you can refer to all posts as simply their filename without the full path.&lt;/p&gt;
&lt;h1&gt;How I use it&lt;/h1&gt;
&lt;p&gt;A natural hierarchical order comes out of this system. You can begin at the top level index page, which lists sub-pages of different categories. Each sub-page lists either actual pages, or even more detailed categories, until finally you dig all the way down to a file where you&#39;re actually writing in, not just listing links. The hierarchy is saved entirely in your &lt;code&gt;~/vimwiki/&lt;/code&gt; folder. It&#39;s all just files and folders. So if you ever want to switch to a different management system and ditch vimwiki, there&#39;s no conversion tool needed, everything is already is an easily readable format. There&#39;s no special vimwiki-specific metadata or databases or anything.&lt;/p&gt;
&lt;p&gt;I have a few big category pages, including &amp;quot;Journals&amp;quot;, &amp;quot;Writing&amp;quot;, &amp;quot;Blog&amp;quot;, and some for different projects I&#39;m working on, like school clubs or things.&lt;/p&gt;
&lt;p&gt;I use the journals folder to keep all my generic notes. There&#39;s your typical examples of just brief ideas or little details I need to record for later, but I also like to keep links to or names of media I&#39;ve enjoyed, or that I plan to take a look at later when I have time. I also keep a todo list and a suspense file.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://pluralistic.net/2024/10/26/one-weird-trick/&quot;&gt;https://pluralistic.net/2024/10/26/one-weird-trick/&lt;/a&gt;
&lt;br&gt;
&lt;br&gt;&lt;/p&gt;
&lt;h2&gt;Blog posts + static site generator&lt;/h2&gt;
&lt;p&gt;Next, I also have the blog folder. This site is self-run using a static site hosting service (GitHub pages currently), and I use the static site generator eleventy to turn my markdown files into html pages based on the template for the site. All I do is install the node package into &lt;code&gt;~/vimwiki/blog/&lt;/code&gt; and have it generate my site whenever I finish a new post. I&#39;ve got it configured to output files to a custom directory where I keep the git repo for the website. So when I run the command, it updates my website repo. When I like how it looks, I commit and push the changes, and they become public within a few moments.&lt;/p&gt;
&lt;p&gt;Just for convenience sake, I also keep a &lt;code&gt;~/vimwiki/blog/drafts/&lt;/code&gt; folder which I&#39;ve instructed eleventy to ignore. Once I finish writing a draft, I move it into the parent folder where it can be seen by eleventy.&lt;/p&gt;
&lt;p&gt;Of course, you&#39;ll probably want to configure your VimWiki to use markdown files if you&#39;re doing this, rather than it&#39;s default &lt;code&gt;.wiki&lt;/code&gt; files, which I&#39;m unaware as to whether eleventy can support, but I would assume markdown has better support in any case. You also don&#39;t necessarily need to use eleventy -- any static site generator that is comfortable for you will do.
&lt;br&gt;
&lt;br&gt;
  &lt;a href=&quot;https://www.11ty.dev/&quot;&gt;https://www.11ty.dev/&lt;/a&gt;
&lt;br&gt;&lt;/p&gt;
&lt;p&gt;The idea to this whole system is that all your external tools for memory are stored into a singular place. Rather than having to think about where you keep movie suggestions, read later list, todo lists, favorite quotes, and notes on different projects, you can just know that everything is in your VimWiki folder. Just start following links down your category until you land on what you&#39;re looking for. It eases the friction for getting started on a task, and it also makes it easier to transition from one task to another, since your notes are going to be all stored in the same place.&lt;/p&gt;
&lt;h1&gt;Cloud sync + mobile access&lt;/h1&gt;
&lt;p&gt;Okay, this is getting into perhaps the more technical, but still doable. You can actually sync your entire &lt;code&gt;~/vimwiki/&lt;/code&gt; folder to the so-called &lt;em&gt;&amp;quot;&amp;quot;cloud&amp;quot;&amp;quot;&lt;/em&gt; using git. Initialize the repo, and commit all changes. Pick a server you want to use for syncing. I use GitHub because it&#39;s what I know, but some may prefer to use GitLab, or even a VPS or something like that if you&#39;re hoping for some privacy from Big Tech services.&lt;/p&gt;
&lt;p&gt;Anyhow, simply set up a &lt;em&gt;private&lt;/em&gt; git repo and push your changes. You&#39;ll need to of course authenticate first to prove you&#39;re allowed access, but after that it should all work. You want it to be this way so that nobody can read your notes or drafts unless they have access.&lt;/p&gt;
&lt;p&gt;If you want to access it from somewhere else, simply git clone the folder to a separate device, edit the files, then commit and push. The benefit of using simple text files instead of more advanced files like Word Documents is that you can edit them from basically any device, even a phone. A lot of file managers for your phone will come with a text editor pre-installed.&lt;/p&gt;
&lt;p&gt;You can get even fancier than that though. If you have an Android, you can install a terminal emulator like Termux, which enables you to install packages such as git and vim on your phone. That&#39;s what I do, I git pull my vimwiki folder to get the latest version, and then I open it with vim. I even have the VimWiki plugin installed on it too. I make my edits, then commit and push back up. Of course, that&#39;s not your only option. All you need is a way to push and pull a git repo to your phone, and some kind of generic text editor.&lt;/p&gt;
&lt;h1&gt;Closing&lt;/h1&gt;
&lt;p&gt;I really enjoy my experience with VimWiki, creating a hierarchical note system. I try not to get carried away with the data structure of it all, instead focusing on just keeping notes on important stuff. If you have any other tricks how to make the most out of this tool, feel free to let me know about it! I love endlessly tweaking with my workflow on my laptop to optimize efficiency (or often just coolness factor).&lt;/p&gt;
</content>
  </entry>
</feed>