canonical

Learning to speak Juju

Learning to speak Juju

One of my favorite quotes is that there are two hard problems in tech, cache invalidation and naming things. Naming things is fun and you quickly realize that in any tech community there’s a vocabulary you need to understand in order to participate. Programming languages, technical tools, and even just communities all build entire languages you need to understand to participate. Juju is no different here. When you look at Juju you’ll find there’s a small vocabulary that it really helps to understand. 

There are only two hard things in Computer Science: cache invalidation and naming things.
— Phil Karlton

The Juju Client

$ juju --version

We start with Juju itself. Now honestly, there’s a few layers of Juju you end up working with but let’s start with the command line client. When you invoke Juju from the CLI you’re running a local client that communicates with APIs that perform the real work. Clients are available on all major systems and you can even think of things like the Juju GUI as a web based client. You might be asked what version of Juju you’re running and the best place to start is with the version of your client. 

Cloud

$ juju clouds

Cloud        Regions  Default        Type        Description
aws               14  us-east-1      ec2         Amazon Web Services
azure             26  centralus      azure       Microsoft Azure
google             8  us-east1       gce         Google Cloud Platform
localhost          1  localhost      lxd         LXD Container Hypervisor
guimaas            0                 maas        Metal As A Service
...

Just what is the Cloud? In our ecosystem a Cloud is any API that will provision machines for running software on. There are a number of public clouds and each has an APIs, such as AWS, GCE, and Azure. There are private clouds that you can operate software on, such as OpenStack or MAAS. There’s even a local cloud using the LXD API to provide a cloud experience on your local system controlling LXD containers. The key thing is that Juju is primarily intended to abstract away the details of the various cloud APIs so that the operations work you need to perform is repeatable and consistent regardless on which cloud you choose to use. 

Controller

$ juju controllers

Controller   Model        User               Access     Cloud/Region   Models  Machines    HA  Version
guimaas      -            admin              superuser  guimaas             2         1  none  2.2.4
jaas*        k8-test      rharding@external  (unknown)                      -         -     -  2.2.2
jujuteamops  teamwebsite  admin              superuser  aws/us-east-1       2         1  none  2.2.4

A controller is the main brain of Juju. It holds the database of what is deployed, what configuration is expected, what is the current status of everything that’s been and being deployed and running over time. The controller houses the users and permissions and it’s the API endpoint that the Juju Client communicates with. The controller runs on the same cloud as the workloads that will be operated and can be scaled for HA purposes as it’s important infrastructure in the Juju system. You can see, in the above snippet, I’m running controllers on my local MAAS, AWS, and using the JAAS hosted controller. 

Model

juju models

Controller: jujuteamops

Model            Cloud/Region   Status     Machines  Cores  Access  Last connection
ci-everything    aws/us-east-1  available         0      -  admin   never connected
ricks-test-case  aws/us-east-1  available         3      3  admin   2017-09-28
teamwebsite*     aws/us-east-1  available         1      1  admin   2017-09-28

In Juju we talk a lot about modeling and models. Models are namespaces that work is done within. A single Controller can manage many many Models. Each Model tracks the state of what’s going on inside of it, it’s able to be ACL’d using the users known to the Juju Controller, and you can dump out the contents of a Model into a reusable format. In the above snippet you can see I have several Models running on a Controller in AWS right now. One is handling some CI work, another I’m using to test out a bug fix, and a third is the Model where we operate the software required to provide our website. 

Users are able to switch between models and it really helps provide a great level of focus on what a user is working on at any given moment in time. 

Charm

$ juju deploy mysql

Located charm "cs:mysql-57".
Deploying charm "cs:mysql-57".

A Charm is the component that users use to deploy software. Each software component is deployed using a ZIP file that contains all of the scripts, configuration information, event handling, and more that might be required to operate the software over time. The URL of the Charm that’s been found breaks down like so:

  • cs: = “charm store” (vs local file on disk)
  • mysql = name of the charm
  • 57 = what revision of the charm did we find. (latest when not specified)

Charms are just chunks of software themselves. They are the set of YAML, scripts, and templates that makes it possible to operate things. In the above case the Charm provides all the software required to operate MySQL over time. This includes configuration information, backup scripts, and code that handles the provisioning of databases to other software in the Model. 

Application, Unit, and Machine

$ juju status mysql

...
App    Version  Status   Scale  Charm  Store       Rev  OS      Notes
mysql           waiting    0/1  mysql  jujucharms   57  ubuntu

Unit     Workload  Agent       Machine  Public address  Ports  Message
mysql/0  waiting   allocating  1                               waiting for machine

Machine  State    DNS  Inst id              Series  AZ          Message
1        pending       i-0524bda18a2bb0cb9  xenial  us-east-1b  Start instance attempt 1
...

When you deploy MySQL you are asking that a single Machine is provisioned by the Controller using the Cloud API and that the Charm is placed on that machine and, once there, it’s executed. However, once that Charm added to the Model, we call MySQL an Application. The Application is an abstraction that is used to control how we operate our MySQL. This includes adding additional MySQL servers and building a MySQL cluster. If we state that the data for MySQL should be in /srv vs /opt we want that to be set on the Application and then each MySQL in the cluster will update and follow suit. Each member of the cluster is a Unit of the MySQL Application. We can talk to each one individually when necessary, but really we want them to treat them as a single Application in our Model. 

In this way we say that an Application consists of one or more Units that are located on a Machine. The Machine is the actual hardware or VM allocated for the Unit to run on. If we were to remove the Machine #1 in the above example, we’d also lose our unit mysql/0 but the Application would still be in the model. The state details are still in the model and any new Units added would pick up where the first one left off. 

Relation

juju relate mysql:db gypsy-danger

Having MySQL servers running is great but I have some Python software that would love to store some data into that MySQL cluster. In order to do this I need to write out the MySQL DSN to a config file that my application knows how to read. To facilitate this I’ll create a Charm for my gypsy-danger Application and in my Charm I’ll declare that I know how to speak the “db” language. Funny enough, the MySQL charm already knows how to speak the “db” language as well. This means that I can create a relation in the model that states that MySQL and my python project gypsy-danger can communicate using the “db” language. 

This allows Juju to facilitate a conversation between Units of both Applications. gypsy-danger will ask MySQL for a database. MySQL will then create a fresh new database with a unique username and password and send those details back to the gypsy-danger Application. The scripts contained in the Charm will then make sure each Unit of the Application gypsy-danger updates their Python configuration files with those MySQL connection details. 

As an operator, I don’t want to get into the business of specifying things that are transient or different from one Model to another. That just reduces the reusability of what I’ve built. Relations allow Applications to communicate and self-coordinate to perform actions needed to go from independent software components into a working collection of software that performs a useful function. What’s neat is that other Charms can claim to speak “db” and so one MySQL cluster can be used to server out databases to many other Applications in a Model. You can even have different servers provide those databases, such as Percona or MariaDB. By speaking a common protocol Relations provide a great way for Models to stay agile. 

 

Bundle

applications: 
  mysql: 
    charm: "cs:mysql-57"
    num_units: 1
  "gypsy-danger": 
    charm: "cs:~rharding/gypsy-danger-5"
    num_units: 1
relations: 
  - - "mysql:db"
    - "gypsy-danger:db"
$ juju add-model testbundle
$ juju deploy mybundle.yaml

A bundle is basically a static dump of a Model. It takes all the Applications that are running, how many Units are there, what configuration is specified, what Relations exist in the Model, and dumps it to a clean reusable form. In this way, you can replicate this Model in a new Model, or on another Cloud, or just save it for later. It deploys just like a single Charm would, except that it is going to be performing a lot more work. We have bundles of all shapes and sizes and when you see examples of easily deploying a Kubernetes cluster or an OpenStack installation it’s typically done using a reusable Bundle. 

There is our crash course in the language of Juju. Every project has a language, and we all get really creative naming things in the tech world. Juju is no exception. Hopefully this will help you better understand how the parts fit together as you dive into using Juju to operate your own software. 

Terms to remember:

  • Juju (and the client)
  • Cloud
  • Controller
  • Model
  • Charm
  • Application
  • Unit
  • Machine
  • Relation
  • Bundle

Operations in Production

Operations in Production

As we setup our important infrastructure we set up monitoring, alerting, and keep a close eye on them. As the services we provide grow and need to expand, or failures in hardware attempt to wreck havoc, we’re ready because of the due diligence that’s gone into monitoring the infrastructure and applications deployed. To do this monitoring, we often bake it into our deployment and configuration management tooling. One thing I often see is that folks forget to monitor the tools that coordinate all of that deployment and configuration management. It’s a bit of a case of “who watches the watcher?”

Working at Canonical, three years in. a.k.a wtf just happened?

A couple of people have reached out to me via LinkedIn and reminded me that my three year work anniversary happened last Friday. Three years since I left my job at a local place to go work for the Canonical where I got the chance to be paid to work on open source software and better my Python skills with the team working on Launchpad. My wife wasn't quite sure. "You've only been at your job a year and a half, and your last one was only two years. What makes this different?" What's amazing, looking back, is just how *right* the decision turned out to be. I was nervous at the time. I really wasn't Launchpad's biggest fan. However, the team I interviewed with held this promise of making me a better developer. They were doing code reviews of every branch that went up to land. They had automated testing, and they firmly believed in unit and functional tests of the code. It was a case of the product didn't excite me, but the environment, working with smart developers from across the globe, was exactly what I felt like I needed to move forward with my career, my craft.

2013-09-02 18.17.47

I joined my team on Launchpad in a squad of four other developers. It was funny. When I joined I felt so lost. Launchpad is an amazing and huge bit of software, and I knew I was in over my head. I talked with my manager at the time, Deryck, and he told me "Don't worry, it'll take you about a year to get really productive working on Launchpad." A year! Surely you jest, and if you're not jesting...wtf did I just get myself into?

It was a long road and over time I learned how to take a code review (a really hard skill for many of us), how to do one, and how to talk with other smart and opinionated developers. I learned the value of the daily standup, how to manage work across a kanban board. I learned to really learn from others. Up until this point I'd always been the big fish in a small pond and suddenly I was the minnow hiding in the shallows. Forget books on how to code, just look at the diff in the code review you're reading right now. Learn!

My boss was right, it was nearly ten months before I really felt like I could be asked to do most things in Launchpad and get them done in an efficient way. Soon our team was moved on from Launchpad to other projects. It was actually pretty great. On the one hand, "Hey! I just got the hang of this thing" but, on the other hand, we were moving on to new things. Development life here has never been one of sitting still. We sit down and work on the Ubuntu cycle of six month plans, and it's funny because even that is such a long time. Do you really know what you'll be doing six months from now?

P1100197.jpg

Since that time in Launchpad I've gotten work on several different projects and I ended up switching teams to work on the Juju Gui. I didn't really know a lot about this Juju thing, but the Gui was a fascinating project. It's a really large scale JavaScript application. This is no "toss some jQuery on a web page" thing here.

I also moved to work under a new manager Gary. As my second manager since starting at Canonical and I was amazed at my luck. Here I've had two great mentors that made huge strides in teaching me how to work with other developers, how to do the fun stuff, the mundane, and how to take pride in the accomplishments of the team. I sit down at my computer every day and I've got the brain power of amazing people at my disposal over irc, Google Hangouts, email, and more. It's amazing to think that at these sprints we do, I'm pretty much never the smartest person in the room. However, that's what's so great. It's never boring and when there's a problem the key is that we put our joint brilliant minds to the problem. In every hard problem we've faced I've never found that a single person had the one true solution. What we come up with together is always better than what any of us had apart.

When Gary left and there was a void for team lead and it was something I was interested in. I really can't say enough awesome things about the team of folks I work with. I wanted to keep us all together and I felt like it would be great for us to try to keep things going. It was kind of a "well I'll just try not to $#@$@# it up" situation. That was more than nine months ago now. Gary and Deryck taught me so much, and I still have to bite my tongue and ask myself "What would Gary do" at times. I've kept some things the same, but I've also brought my own flavor into the team a bit, at least I like to think so. These days my Github profile doesn't show me landing a branch a day, but I take great pride in the progress of the team as a whole each and every week.

The team I run now is as awesome a group of people, the best I could hope to work for. I do mean that, I work for my team. It's never the other way around and that's one lesson I definitely picked up from my previous leads. The projects we're working on are exciting and new and are really important to Canonical. I get to sit in and have discussions and planning meetings with Canonical super genius veterans like Kapil, Gustavo, and occasionally Mark Shuttleworth himself.

Looking back I've spent the last three years becoming a better developer, getting an on the job training course on leading a team of brilliant people, and crash course on thinking about the project, not just as the bugs or features for the week, but for the project as it needs to exist in three to six months. I've spent three years bouncing between "what have I gotten myself into, this is beyond my abilities" to "I've got this. You can't find someone else to do this better". I always tell people that if you're not swimming as hard as you can to keep up, find another job. I feel like three years ago I did that and I've been swimming ever since.

P1040511.jpg

Three years is a long time in a career these days. It's been a wild ride and I can't thank the folks that let me in the door, taught me, and have given me the power to do great things with my work enough. I've worked by butt off in Budapest, Copenhagen, Cape Town, Brussels, North Carolina, London, Vegas, and the bay area a few times. Will I be here three years from now? Who knows, but I know I've got an awesome team to work with on Monday and we'll be building an awesome product to keep building. I'm going to really enjoy doing work that's challenging and fulfilling every step of the way.

DSC00329

Been a good summer, fitness, woodworking, and new job coming soon...

I'm very late on a bunch of weekly status reports, but I've got good reason I promise. This summer has been a bit of a "summer of change" for me. Perhaps more renovation than change.

First, I decided I was sick of looking and feeling like crap and got into a weight loss program. I've finished that program today with some really good results. I'm down 37lbs, with another 19 to go until I hit my goal. Here's hoping I can get there before the year is out. This has really helped me feel a ton better. I did a lot of biking this summer, including a 48mile monster that beat me up, but man was that fun.

Next up, I finally managed to clean the garage out. That means the woodworking tools are open for business. I've only started working on putting some finish on some new oak closet doors, but that's a huge move forward. I've not done any woodworking since the boy was born and now that the space is cleared out I'm looking forward to getting back to making some shavings.

Finally, the big news. I've put in my two weeks notice with Morpace and have accepted an offer to work for the Launchpad team with Canonical.

I can't express how excited I am to get this chance to work with a team of smart people that are really working hard to build something awesome. In many ways this is the culmination of a goal I set years ago to get paid to work on open source software, and not only that, but to get paid to work with Python as well.

Morpace was really good to me and I feel sad to leave them. They gave me my first chance to prove that I could be a productive Python developer. I've grown though and the opportunity to work with all of the great people at Canonical is just too much to pass up. I can't wait to jump in and see what I can help with.

It's been a good summer. I can only hope that each year continues to be as fulfilling.