Open Source software and operational usability

A friend of mine linked me to Yaron Haviv's article "Did Amazon Just Kill Open Source" and I can't help but want to shout a reply


There is something in premise though, and it's something I keep trying to push as it's directly related to the work I do in my day job at Canonical.

Clouds have been shifting from IaaS to SaaS as sure as can be. They've gone from just getting your quick access to hardware on a "pay for what you use" system to providing you full self service access to applications you used to have to run on that hardware. They've been doing this by taking Open Source software and wrapping it with their own API layers and charging you for their operating of your essential services.

When I look at RDS I see lovely images for PostgreSQL, MySQL, and MariaDB. I can find Elasticsearch, Hadoop, and more. It's a good carrot. Use our APIs and you don't have to operate that software on the EC2 platform any more. You don't have to worry about software upgrades, backups, or scale out operational concerns. Amazon isn't the only cloud doing this either. Each cloud if finding the services it can provide directly to the developers to build products on. 

They've taken Open Source software and fixed delivery to make it easy to consume for most folks. That directly ties into the trend we have been working on at Canonical. As software has moved more and more to Open Source and the cost of software in the average IT budget drops, the costs of finding folks that can operate it at production scale has gotten much more expensive. How many folks can really say they can run Hadoop and Kubernetes and Elasticsearch at professional production levels? How hard is it to find them and how expensive is it to retain them? 

We need to focus on how we can provide this same service, but straight from the Open Source communities. If you want users of your software to be users of YOUR software, and not some wrapped API service, then you need to take those operational concerns into account. We can do more than Open Source the code that is running, we can work together as a community to Open Source the best practices for operating that software over the full life cycle of it. Too often, projects stop short at how to install it. The user has to worry about it long after it's installed. 

I have hope that tools like Juju and the Kubernetes can provide a way for the community around the software we use and love to contribute and avoid the lock in of some vendored solution of the Open Source projects we participate in today. 

Giving Gitlab an afternoon spin

I've been meaning to check out Gitlab lately. I hear a lot about folks replacing their internal systems with it. It has an awesome checkbox of features for managing your internal code with public and private repos,  and enables you to build the best practices out there around code reviews, automated testing and gated landing. I'm lucky and work in Open Source for my work, but until recently all the code I worked on sat on some internal Git or SVN system. A better application for performing that task is exciting to a lot of users out there. Of course, the best way for me to test it out was to grab the Gitlab Juju charm and toss it up on GCE with JAAS

While looking at that I noticed that a member of the community had actually created a bundle of two applications that made the Gitlab experience even better. It includes a new charm from another community member that allows using Let's Encrypt. Cool! This means that I not only get a Gitlab instance to test with, but I can use it with others with SSL encryption so that the login credentials and such aren't passed around in clear text. 

Let's get testing by adding a new model in JAAS and deploying this bundle into GCE.

juju add-model gitlab google
juju deploy cs:~spiculecharms/bundle/gitlab-ssl
juju show-machine 0

The show-machine command here is useful because I need to go add a new DNS entry for Let's Encrypt to use. Since I'm testing out a scenario of hosting all of my Open Source project's code here I added an A record for to point to the public IP address of the ssl termination application. With that DNS entry setup I need to tell the termination application what its URL is meant to be. 

juju config ssl-termination-proxy

A few minutes later and I had a working Gitlab deploy to play with. I imported all of my old Bookie code projects from Github and tested out cloning repos and such. One issue I did hit was that since my Gitlab application was proxied I needed to tell Gitlab about the URL it's actually under for end users. Do do that you need to edit a config file on the Gitlab application. 

juju ssh gitlab-server/0

sudo vim /etc/gitlab/gitlab.rb

# Change this external_url config
external_url ''
sudo gitlab-ctl reconfigure

I can now move forward with testing out the gitlab supplied tools for testing and building releases. If it works out I can then deploy this in my private MAAS or internal OpenStack with the same ease because Juju provides such a wide array of options. 

I want to thank the great folks at Spicule that put together the Gitlab charm and the folks at Tengu for the ssl-termination-proxy charm. I find that second one really interesting and will cover that in a follow up blog post. 

CoffeeHouseCoders 11/23/11: YUI Theater group viewing

Just a heads up, this week's CoffeeHouseCoders (CHC) Detroit-ish will be a bit different. One of the goals of moving the location to the new Caribou was that we get access to the meeting room. This opens up the opportunity for us to have some group discussion and such around various topics. We're going to give that a shot this week with a group viewing of YUI Theater video viewings and JavaScript discussion.

Most of us do at least some JavaScript in our work and projects so I think it's relevant and should be fun to geek out before the holidays start up. I'll have a little projector and speaker and with the new videos from YUIConf 2011 going up, it'll be nice to set aside some time to catch up on some of the recorded presentations. Take a peek and set aside one or two "must watch" videos for Wed night! Not all of the videos are YUI specific, so it should be useful for all of us doing JavaScript.

Bookie status report: Aug 14th 2011

Updates this week

Finally, the new and fancy Bookie API is reworked and better than ever. The live running site is updated to use the new api and there's a new chrome extension that's update for it as well. Make sure you get the latest dev version of the Chrome extension.

With the api updated, it's time to get to work on finishing up 0.3 for release. This means that we need to finish updating the mobile site and some rework of how the readable content is stored. Since we pull the content out of the page based on actual page you see, we should store these separated from user to user. We don't want to leak logged in info from one user to another.

Farewell Firefox

With the api update, the Firefox is so far out of date as to be unusable. There's some work going on to help catch it up, but I think I'm going to remove it as a feature in the docs and such. We just can't claim it at this time.

New testing

With the new api we hit some testing issues. Since there's no good way I've found to functional test the extension, the tests aren't great and reliable. I also don't have tests setup on some shared code like the bookie.api.js code. I'm thinking of setting up a running instance with some live data with the sole purpose of running tests on the JS code against it. We might even go so far as to setup the extension html files as an actual website and write some functional tests around that. After all, I can load popup.html and do some testing of that as a normal web page. We'll be missing the interactions with background.html and such, but might be worth some attempt.

Any ideas and help with this is most welcome.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github:

  • Current Chrome Extension: 0.3.11
  • Most recent code updates: develop branch
  • Current working branch: develop

YUI, how dare you make me rethink

So I've been spending time working on moving ahead with my project using YUI. So it's been a lot of trying to port over old jQuery code into YUI, going through docs on the various widgets and plugins, and just trying to figure out how to rearrange how I think with this stuff.

The hard part is that YUI3 enforces something we all consider good practice. Don't pollute the global namespace. What's the bad in that? Well, it presents some challenges with the way I used to do things.

[sourcecode lang="javascript"] MyApp = {};

MyApp.home_page = {

'init': function () { // do some stuff here // like load some social content or crap MyApp.home_page.bind_stuff(); },

'bind_stuff': function () { // hey, we have fancy events we bind/catch // woooo } };

// and then in the page onReady I just MyApp.home_page.init(); [/sourcecode]

So that's how I have been doing things. Parts get abstracted out, shared, but basically each page tends to have an object for each page and any extra handling of events or whatever that requires.

However, YUI3 does this cool thing that breaks that. When you use YUI3 you have a YUI().use() and everything gets encapsulated within that bit of code. So if I were to just move the MyApp code into the YUI block I'm no longer able to access it from the page of content I want to run JS on. So this is good, no JS globals, but it's bad...not the way I've been doing things.

So I've had to figure out how to adjust my old code to the new YUI3 way, and I came across this great video on structuring your code a bit better. It makes some sense, but it seemed to just scratch the surface. Unfortunately, his example is a single page application so it's a bit simpler than what I needed, but it got the point across for me.

So what's the new stuff look like? It looks like YUI3 modules

[sourcecode lang="javascript"] YUI.add('myapp', function (Y) { var MyApp,

MyApp = function (config) { MyApp.superclass.constructor.apply(this, arguments); };

/** Static **/ MyApp.NAME = 'MyApp'; MyApp.ATTRS = {


Y.extend(MyApp, Y.Base, { initializer : function (config) { },

destructor : function () {},

/** Public Methods **/ 'some_page': function () { var that = {};

that.init = function () { // do some stuff that.bind_stuff();


that.bind_stuff = function () { // and we're doing stuff };

// return the page object with all the controls/method return that;



Y.namespace('MyApp'); Y.MyApp.Window = MyApp;

}, '0.1', { requires: ['base', 'node', 'my-ui', 'yui2-datatable', 'yui2-button'] });

// and then I use it from my page with YUI({ }).use('myapp', function (Y) { win = new Y.MyApp.Window(); my_page = win.qcontent_admin() my_page.init(); });


So this has me creating a module for my application code. It allows me to specify versions, requirements, etc. It keeps the YUI3 way of not polluting any global namespace for the application, and while a bit more verbose, seems to be working out. I like this since it's getting to what I was talking about back before when I was looking at moving to YUI from jQuery. This feels more like treating the JS code like application code.

I could split up the code into different modules based on different logical sections of the application and I might be able to reuse bits of this down the road. For instance, we use a common base bit of application code for our authentication across apps. I could break out the JS code for the auth pages into their own module, version it, and reuse it much easier.

What do you think? Am I heading in the right direction or am I just making life more complicated than it's supposed to be?

Howdy YUI, looking fine...ish

I subscribe to some RSS feeds and I've seen some cool videos/posts about YUI3 and it looked kind of interesting. I've always felt like I should be using/checking out YUI more. Everything seems very well thought out, tested, and I love that they have a whole package. Everything from DOM tools, Ajax, CSS Grids all the way to ui widgets. I've always thought they had the best autocomplete widget around from the demo. Tab should complete the current selection, it just should. So I picked up a copy of YUI2.8 Learning the Library and while it's not YUI3, it's a good view into the minds of things and how they fit together. So I decided to port over a work project I'm working on to YUI. It's kind of CMS-y so I want to setup grids just to help flexibility as things go on. (Although...I'm a grid hater for the record) I also want to take advantage of the UI collection for auto complete, progress bar, text editor, treeview and possibly datagrid.

Today was day one and so far so good. It takes a bit of work to get your head moved over from jQuery. I've been using it for a while and really the thought process in jQuery is different. Go find about any tutorial and you'll see things like:

[sourcecode lang="javascript"] // find all the instance of class="important" // and then make sure they're marked with a bolder font $('.important').css('font-weight', 'bold'); [/sourcecode]

This view is kind of against the ideas of mixing logic and template, in a way. I mean this is all view information here. Instead I think I should have things more like:

[sourcecode lang="javascript"] var highlight_important = function (identifier) { var default_id, id;

default_id = '.important';

if (identifier !== undefined) { id = identifier; } else { id = default_id; }

$(id).css('font-weight', 'bold'); }; [/sourcecode]

Now, ideally this would take more config options for just what class property to set and what value to set, all optional of course. And you have this reusable component with a sensible function name you can reuse throughout the app.

This is probably a bad example, but the point is that jQuery comes across as 'find the html and do stuff' where frameworks that come across more verbose/complex, like Dojo and YUI, seem to present things as 'write JS logic' instead.

For instance, I love the idea of the modules in YUI3. I used to just have a custom namespace object like:

[sourcecode lang="javascript"] mystuff = {

'ui': { 'zebra': function () { ... }, },

'form': { 'init': function () { ... },

} [/sourcecode]

This is fine, but I like the idea of the modules you can treat like other YUI modules. For instance my form code turns into something like:

[sourcecode lang="javascript"] YUI.add('myform', function (Y) {

Y.myform = function () { // private property to track the forms we've processed forms_formatted = {};

var that = {};

that.init = function () { Y.myform.maxlabels(); },

that.dosomething = function () { ... }

return that; }(); }, '0.0.1', { requires: ['node'] });

// and use it like var login_page = function () { YUI().use("node", "yui2-button", 'myform', function(Y) {

var Y2 = Y.YUI2;

//YUI 2 implementation code var button = new Y2.widget.Button("submit");

// init the form ui tweaks Y.myform.init(); }); }; [/sourcecode]

This also shows off how I can do things like include the YUI2 widgets.

So people are asking "Why use YUI over jQuery" and it really comes down to 'total package' with the overall completeness of things, 'well thought outness' of the library. In reading things I'm always hitting points like "man, that's a really good way to do/handle that". Finally, I want to try out and seem to like how it makes you 'think' when writing the code.

Will it stick, who knows, but I've learned enough JS to want to try out something a bit higher up. I consider it a bit like going from webob to Pylons. They're both JS underneath and all, but one includes a lot more bits than the other at the cost of some framework complexity.