While working on a bunch of tools for WaffleHacks, a hackathon I helped organize that recently finished, I kept running\ninto issues deciding when to deploy something onto our server. No matter whether it was a small or large change, I would\nstill need to go through the same arduous process of:
\nDepending on how long I waited between deployments, this could take anywhere from 5 minutes to an hour. Time I could be\nspending being more productive, like working on new features or maybe doing homework.
\nThis same process continued for around 2 months when I decided to attempt to automate this workflow. I began working on\na quick and dirty tool I sensibly named AutoDeploy. Using webhooks from GitHub and a configuration file located\nat the root of the repository, it would pull and run a set of predefined commands on every push. It worked well enough\nfor a while, but it was very brittle. If any dependencies changed or a new application needed to be added, AutoDeploy\nwould blindly run through its commands and the deployment would most likely fail. Not to mention that there was no way\nto know whether the deployment was successful because AutoDeploy never reported its status anywhere.
\nShortly after setting up AutoDeploy, I stopped making and deploying as many changes since it was becoming more of a\nburden to deploy than before. I quickly realized I needed a new solution.
\n\n\nA system development/management pattern where
\n\n
\n- git is the SINGLE source of truth for a system
\n- git is the SINGLE place where we operate ALL environments
\n- ALL changes are observable and verifiable
\n
Using something like ArgoCD or Flux seemed like the perfect solution to my deployments problem. Just\ncontainerize all the applications and use a single repository to store all the Kubernetes manifests.
\nAll done, right? Unfortunately not.
\nEverything seemed great until I took a look at the cost of running this. At the minimum it would cost double, maybe\neven triple what we were paying for our server, granted that was only $20/month. However, for a bunch of broke college\nstudents trying to run a hackthon with, at the time, no sponsors, this was a no-go. So I went searching for a smaller\nsolution that could be used on a single server, or could scale down to a single server.
\nAfter searching for about a week, I couldn't find anything. Either I was using the wrong terms, nothing like it exists,\nor they are all environment specific solutions that are not public. So, Like any overly enthusiastic programmer, I\ndecided to write one myself.
\nAround 3 months later, after weeks of on-off work interspersed throughout my internship, WaffleMaker was born. Why\nthe name? One of my inspirations for the tool was Beekeeper made by HackGT, so WaffleMaker seemed fitting given\nthe hackathon's name was WaffleHacks.
\nSimilarly to Flux and ArgoCD, WaffleMaker allows using a single git repository as the single source of truth for\ndeployments. However, since it does not run on Kuberentes, it can't benefit from having a pre-defined manifest format\nfor any resources, or a nice API to hook into for deploying them. As such, WaffleMaker is what you get when you smash\nKubernetes, ArgoCD, and Vault together into one application. It does some light container monitoring to prevent an application from being inaccessible, receives webhook events\nwhenever a push is received on the source repository, and injects secrets into containers from Vault.
\nI wrote a custom manifest format to define each service (example manifest). It allows specifying the base image,\nany configuration through environment variables, secrets to pull from Vault, and the image to deploy from. The benefit of\nusing Vault for secret management, is that you can generate credentials for things on the fly. In the case of WaffleMaker,\nI use it to generate credentials for AWS and a PostgreSQL database. This means less credential management, and it still\nstays secure.
\nIn addition to WaffleMaker, we are using GitHub Actions at WaffleHacks to automatically build container images on each\npush to any branch. This allows us to automatically update the running container ensuring the latest version is deployed.\nOur tagging scheme is such that each image is tagged with the commit hash and the branch it was pushed to. When\nconfiguring the service, we can specify which tags are allowed to be updated from, preventing development versions from\naccidentally being deployed.
\nCurrently, WaffleMaker is deployed and being used (our source of truth). Looking ahead, there are a handful of\nchanges and process improvements I would like to make:
\nThat's all for this post, thanks for reading! If you know of any existing solutions or see anywhere WaffleMaker could be\nimproved, feel free to comment below.
", "url": "https://krantz-dev.krantz.workers.dev/blog/small-scale-gitops", "title": "Small Scale GitOps", "summary": "Implementing GitOps principles for a small organization", "date_modified": "2021-09-11T00:00:00.000Z", "date_published": "2021-09-11T00:00:00.000Z" }, { "id": "https://krantz-dev.krantz.workers.dev/blog/ssh-certificates", "content_html": "First off, hello to anyone reading this.\nWelcome to my personal website/blog where I'll be posting every-so-often about stuff I've been working on or find interesting.\nIf you're curious about me, head over to my about page.
\n\n\nBefore I start this, I should say that I am by no means endorsed by Smallstep.\nI am simply a big fan of their SSH certificates tool.
\n
For the past 3ish years, I've been renting a few VPSes from various providers to host services for myself and, at one time, this site.\nI have a tendency to run with a particular system for a while and then completely wipe it to start from scratch when I've found a better way to do something.\nEverytime I've provisioned a new server, I always have to go through the same, tedious process of: assigning the root public key, creating a new user, adding a public key to that user, and giving the user sudo privileges.\nThis process appears to be unavoidable, unless you only want to use root (which you ABSOLUTELY SHOULD NOT BE DOING!).
\nAt the start of the lockdown in March, I had a desire to completely redo my server setup since my mail server was being a pain.\nIn my research to find the services I wanted to run and how to best do it, I came across this blog post about SSH certificates.\nTLDR: SSH certificates allow you to have a single certificate authority that signs all your public keys allowing you to use any private key you approve of, and you should be using them.\nAfter reading that, I sought to implement it for my own servers.
\nNow, I should note that I am by no means running anything at scale.\nI was planning on using 5 servers with 3 in a Docker Swarm cluster, 1 dedicated for mail, and the other for general testing (I have since cut back to 3 standalone servers due to changes in pricing).\nI chose to use SSH certificates mainly out of curiosity about the technology.\nRather than using Smallstep's solution that they were pushing in that blog post, I chose to almost roll my own version of it.\nDon't worry, I didn't try to roll my own crypto and create a fully custom implementation of it.\nI instead used Hashicorp Vault, a general purpose secret store, and a tool I wrote to sign the public keys.
\nThis worked, but it was a bit clunky.\nVault is really intended for more than just signing SSH certificates so that was a bit overkill.\nAnd the tool I wrote required me to login to Vault's web UI about once a month to reset the authentication token since it had a 30-day TTL, and I didn't really want to figure out how to implement auto-renewal.\nIt was mostly just a slap-dash implementation to get this whole system working.\nAdding on top of this pile, if the server running Vault ever went down, I would essentially be locked out of all my servers.
\nAs I mentioned before, I reduced my server count from 5 to 3 in an effort to save a good chunk of change.\nThis resulted in me changing my hosting provider (since my previous one, Scaleway, raised their prices) to DigitalOcean.\nI decided to do away with that old system and switch to Smallstep's all-in-one solution.\nUnfortunately, they only allow you to use external providers on their platform if you pay $10/host which is WAY too expensive for me.\nAs such, I'm hosting it myself which unfortunately means that I have to find some authentication source to use.
\nI've been following the development of ORY for a while now, their Hydra product in particular since it provides a light-weight OAuth/OIDC server.\nHowever, it still seems a bit too new to me, and I'm honestly not entirely sure how to add users to it.\nThen there's also Keycloak that provides similar features, but is a bit of a memory hog which I'd rather not have since I'm in a memory-constrained environment.\nI couldn't find any other services that seemed like they fit my needs, so I opted to use Auth0 instead.\nMy goal is to host most of my own services, so I'll probably be switching out Auth0 for ORY Hydra at some point.\nIn the meantime, Auth0 works very well and is free for up to 7000 users.
\nThe Smallstep CLI and CA are quite easy to setup.\nI was mostly following this guide and made a few modifications, especially since I'm not on AWS.\nUsing Smallstep is especially nice since I am able to just use SSH as I normally would, and it automatically hooks into the step CLI to issue certificates when needed.
\n# Before\nvssh profiles connect krantz.dev/mail\n\n# After\nssh mailserver.domain\nThis is especially nice since I don't need to use two different tools for connecting to any servers.\nIt also is semi-redundant if the CA server is down since an offline copy of the database is stored locally, meaning I can still access my servers so long as my certificate hasn't expired.
\nIt's been about a week now of using this setup, and I'm very happy with it.\nThis whole process of setting up the CA, getting my servers connected to it for verifying the certificates, and getting my desktop and laptop setup to request certificates took about a week.\nWhile that may sound like a while, keep in mind that I'm a student and don't have too much time on my hands.\nYou could probably get it done in a weekend if you had a bunch of free time.\nJust make sure to always leave an SSH connection open or have some way of accessing your server, so you can still access it in case you mess something up.
\nAnyway, I'd highly recommend anyone who has at least 2 servers they own/rent to use SSH certificates.\nBye for now!
", "url": "https://krantz-dev.krantz.workers.dev/blog/ssh-certificates", "title": "Securing servers with SSH certificates", "summary": "Learn how to SSH certificates instead of public keys using SmallStep", "date_modified": "2020-11-13T00:00:00.000Z", "date_published": "2020-11-13T00:00:00.000Z" } ] }