{"id":467,"date":"2024-10-08T00:46:49","date_gmt":"2024-10-08T00:46:49","guid":{"rendered":"https:\/\/liverickson.com\/blog\/?p=467"},"modified":"2024-10-08T00:46:49","modified_gmt":"2024-10-08T00:46:49","slug":"incident-report-site-outage-07-october","status":"publish","type":"post","link":"https:\/\/liverickson.com\/blog\/?p=467","title":{"rendered":"Incident Report: Site Outage 07 October"},"content":{"rendered":"\n<p>Multitasking got to the best of me today when I distractedly decided to update my web host server from Ubuntu 20.04 to 24.04. <\/p>\n\n\n\n<p>At or around 11:15am EST, I brazenly went forth on my attempt to update my AWS instance. As I was running <code>apt update<\/code> I realized that I should created at least one backup of the instance, and took a snapshot. It&#8217;s a good thing I did, or I wouldn&#8217;t be here today. <\/p>\n\n\n\n<p>The initial upgrade from 20.04 to 22.04 went smoothly. So, I decided (with far more confidence than I had any right to) that I would go ahead and update all the way to 24.04. Did I take another snapshot? No, no I did not. Had I not been smack in the middle of building an analytics dashboard in R and trying to give feedback on a strategy document, I might have thought to. Oh well. <\/p>\n\n\n\n<p>When I was going through the process of upgrading to 24.04, I got a message that told me that I had different ssh configuration files on my host machine and the remote update server. There were a bunch of options to choose from. I carefully took the time to read each one and analyze the differences before I made a decision about how to resolve the conflict. <\/p>\n\n\n\n<p>JK, I just hit OK and replaced the one that was local in favor of the shiny new one. <\/p>\n\n\n\n<p>Folks, don&#8217;t do that. <\/p>\n\n\n\n<p>I&#8217;m not skilled enough with ssh to know whether or not there was a better way to fix things when I suddenly could no longer get into my web server. How do you get into a cloud server when you can&#8217;t ssh in? Rather than try and figure it out, I just decided to recreate and migrate the site. If you were on the M60 with me today, I promise I wasn&#8217;t hacking or anything, I was just trying to figure out whether or not my web server config files needed to be updated with my new static IP address.<\/p>\n\n\n\n<p>By the time I got home, I had successfully created a new instance from the original snapshot with a new static IP address. I was able to confirm that my domains were successfully pointing to the new machine, but the site was still broken. Then, I had to go to a bunch of back to back meetings, and wasn&#8217;t able to do the literal last step, which was to reopen the port for HTTPS to actually allow web traffic back in. <\/p>\n\n\n\n<p>I wouldn&#8217;t normally write an incident report for my personal website. It doesn&#8217;t really matter to anyone other than me, but today marked a really meaningful part of my journey as a developer: <em>for the first time, I knew I would be able to fix it on my own<\/em>. <\/p>\n\n\n\n<p>I&#8217;ve been running my site for over ten years now. It&#8217;s migrated from a hosted WordPress site to a managed self-hosted site on Azure and now to me running it myself on AWS with a server that I manage. And even though sometimes I end up in rough situations like these, I surprised myself today. Maybe it&#8217;s because I&#8217;ve been using Ubuntu as my desktop operating system as well as my web host, but today I finally got to experience the satisfaction that came with taking down &#8211; and then bringing back up &#8211; my own site, without having to ask for help. <\/p>\n\n\n\n<p>That rules.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Multitasking got to the best of me today when I distractedly decided to update my web host server from Ubuntu 20.04 to 24.04. At or around 11:15am EST, I brazenly went forth on my attempt to update my AWS instance. As I was running apt update I realized that I should created at least one backup of the instance, and took a snapshot. It&#8217;s a good thing I did, or I wouldn&#8217;t be here today.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":0,"activitypub_interaction_policy_quote":"","activitypub_status":"","footnotes":""},"categories":[3],"tags":[],"class_list":["post-467","post","type-post","status-publish","format-standard","hentry","category-development"],"_links":{"self":[{"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/467","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=467"}],"version-history":[{"count":1,"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/467\/revisions"}],"predecessor-version":[{"id":468,"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/467\/revisions\/468"}],"wp:attachment":[{"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=467"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=467"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/liverickson.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=467"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}