Code

Working at Canonical, three years in. a.k.a wtf just happened?

A couple of people have reached out to me via LinkedIn and reminded me that my three year work anniversary happened last Friday. Three years since I left my job at a local place to go work for the Canonical where I got the chance to be paid to work on open source software and better my Python skills with the team working on Launchpad. My wife wasn't quite sure. "You've only been at your job a year and a half, and your last one was only two years. What makes this different?" What's amazing, looking back, is just how *right* the decision turned out to be. I was nervous at the time. I really wasn't Launchpad's biggest fan. However, the team I interviewed with held this promise of making me a better developer. They were doing code reviews of every branch that went up to land. They had automated testing, and they firmly believed in unit and functional tests of the code. It was a case of the product didn't excite me, but the environment, working with smart developers from across the globe, was exactly what I felt like I needed to move forward with my career, my craft.

2013-09-02 18.17.47

I joined my team on Launchpad in a squad of four other developers. It was funny. When I joined I felt so lost. Launchpad is an amazing and huge bit of software, and I knew I was in over my head. I talked with my manager at the time, Deryck, and he told me "Don't worry, it'll take you about a year to get really productive working on Launchpad." A year! Surely you jest, and if you're not jesting...wtf did I just get myself into?

It was a long road and over time I learned how to take a code review (a really hard skill for many of us), how to do one, and how to talk with other smart and opinionated developers. I learned the value of the daily standup, how to manage work across a kanban board. I learned to really learn from others. Up until this point I'd always been the big fish in a small pond and suddenly I was the minnow hiding in the shallows. Forget books on how to code, just look at the diff in the code review you're reading right now. Learn!

My boss was right, it was nearly ten months before I really felt like I could be asked to do most things in Launchpad and get them done in an efficient way. Soon our team was moved on from Launchpad to other projects. It was actually pretty great. On the one hand, "Hey! I just got the hang of this thing" but, on the other hand, we were moving on to new things. Development life here has never been one of sitting still. We sit down and work on the Ubuntu cycle of six month plans, and it's funny because even that is such a long time. Do you really know what you'll be doing six months from now?

P1100197.jpg

Since that time in Launchpad I've gotten work on several different projects and I ended up switching teams to work on the Juju Gui. I didn't really know a lot about this Juju thing, but the Gui was a fascinating project. It's a really large scale JavaScript application. This is no "toss some jQuery on a web page" thing here.

I also moved to work under a new manager Gary. As my second manager since starting at Canonical and I was amazed at my luck. Here I've had two great mentors that made huge strides in teaching me how to work with other developers, how to do the fun stuff, the mundane, and how to take pride in the accomplishments of the team. I sit down at my computer every day and I've got the brain power of amazing people at my disposal over irc, Google Hangouts, email, and more. It's amazing to think that at these sprints we do, I'm pretty much never the smartest person in the room. However, that's what's so great. It's never boring and when there's a problem the key is that we put our joint brilliant minds to the problem. In every hard problem we've faced I've never found that a single person had the one true solution. What we come up with together is always better than what any of us had apart.

When Gary left and there was a void for team lead and it was something I was interested in. I really can't say enough awesome things about the team of folks I work with. I wanted to keep us all together and I felt like it would be great for us to try to keep things going. It was kind of a "well I'll just try not to $#@$@# it up" situation. That was more than nine months ago now. Gary and Deryck taught me so much, and I still have to bite my tongue and ask myself "What would Gary do" at times. I've kept some things the same, but I've also brought my own flavor into the team a bit, at least I like to think so. These days my Github profile doesn't show me landing a branch a day, but I take great pride in the progress of the team as a whole each and every week.

The team I run now is as awesome a group of people, the best I could hope to work for. I do mean that, I work for my team. It's never the other way around and that's one lesson I definitely picked up from my previous leads. The projects we're working on are exciting and new and are really important to Canonical. I get to sit in and have discussions and planning meetings with Canonical super genius veterans like Kapil, Gustavo, and occasionally Mark Shuttleworth himself.

Looking back I've spent the last three years becoming a better developer, getting an on the job training course on leading a team of brilliant people, and crash course on thinking about the project, not just as the bugs or features for the week, but for the project as it needs to exist in three to six months. I've spent three years bouncing between "what have I gotten myself into, this is beyond my abilities" to "I've got this. You can't find someone else to do this better". I always tell people that if you're not swimming as hard as you can to keep up, find another job. I feel like three years ago I did that and I've been swimming ever since.

P1040511.jpg

Three years is a long time in a career these days. It's been a wild ride and I can't thank the folks that let me in the door, taught me, and have given me the power to do great things with my work enough. I've worked by butt off in Budapest, Copenhagen, Cape Town, Brussels, North Carolina, London, Vegas, and the bay area a few times. Will I be here three years from now? Who knows, but I know I've got an awesome team to work with on Monday and we'll be building an awesome product to keep building. I'm going to really enjoy doing work that's challenging and fulfilling every step of the way.

DSC00329

Bookie meets Google Summer of Code 2014

Today the Google Summer of Code student selections were announced, and with that announcement Bookie revealed our selections for the slots allocated for each of our two mentors. This announcement highlights an amazing round of participation in Bookie as an open source project. Twenty people participated and landed over 110 commits worth of patches in Bookie since the opening of GSoC. That is AMAZING! In less than a week every bite-sized bug evaporated from the issue tracker. Also amazing is the quality and effort that everyone put into their work. Everyone was eager to learn how to add tests to their patches, and they worked so hard to get their code landed. Bookie emerges a better open source tool for managing bookmarks than it was 2 months ago, and that is because of the hard work and dedication of all of the participating students.

Students did more than land branches; they invigorated the community. We had many users jump into IRC to answer questions and guide students through the process. They also performed QA and did code reviews of their work. The enthusiasm the students brought to Bookie motivated me to make the time to help move things forward. After all: if a student spent 3 days figuring out how to fix a bug, write a test for it, commit the fixes to git, and get it up for code review; then I can manage to find the 30 minutes to pull the commit, review the code, and QA the work. This period motivated me to update documentation and ensure the install process worked for a wider audience. Additional motivation came from knowing that Bookie is interesting as a tool to other people besides myself.

I want every student not selected for Google Summer of Code to know their work and effort is greatly appreciated. I and the other members of the Bookie community enjoyed working with everyone who participated. Bookie had 32 applications for 2 available spots. In conversations with other organizations Bookie had a comparatively crazy amount of competition for few allocated spaces. I wish Bookie had a dozen or so more mentors and slots as over half of Bookie's proposals would have easily been accepted. Culling the dozens of great proposals into two positions was a very difficult process for us. It's hard to say "not right now" when there's so many great offers by so many eager and capable students.

Regardless of whether you were selected for Google Summer of Code the fun doesn't have to end. If you found the time contributing to Bookie valuable; if you learned something new, gained some material for that resume, or just had a good time: PLEASE DON'T STOP! Bookie isn't going anywhere or closing up shop; we're more than happy to continue mentoring and working with you all. We worked hard during this process to ensure all students were given the best chance to take something positive away from this application process. With your continued participation in the Bookie project we'd like to continue to mentor and provide guidance for you.

One area of guidance we owe all students relates to your proposal. Should you want any explanation of what you could do differently with your proposal / application please let us know. I'll be honest though: most of the applications we received were quite good, so there's little to critique. The scoring method we used put most of the applications within a few points of each other. But if you'd like to know more please feel free to ping me in irc and ask me anything you'd like.

Finally we'd like to congratulate Sambuddha and Pradyumna for their outstanding work leading up to this announcement, and we look forward to the results of their proposals for adding great features to Bookie over the summer. If you find the work interesting, please come help them out. Feel free to get involved, help with the work, the code reviews, and the testing of the new features. Maybe you'll be helping mentor Bookie next year? Who knows? :)

This was our first year participating in Google Summer of Code, but you can be assured it will not be the last.We'd like to thank all of the students for flooding our channels and making this not only an amazingly crazy and busy time but also an immensely rewarding period in Bookie's history. You are all part of Bookie's history and we look forward to seeing you as part of Bookie's future. Thank you.

Bookie Sprint - Aug 31st

It's time for another Bookie sprint! When - Saturday August 31st

What time - Starts at 11am

Where - my house! Ping me for address/map info if you're coming along. Map out to Clarkston, MI.

What will we be working on?

The goal is to work on test coverage and breadability article parsing. Are you new to application testing? Come out and learn while helping out an open source project.

If you want to participate online please join our irc channel #bookie on freenode.net. If there's something else you'd rather work on then please let me know and I'll be happy to do whatever I can to aid in participation.

Bookie Weekly Status Report Returns! - April 15 2012

Ok, I'm overdue for a 'weekly' status report. I'm going to try to kick this back into gear as it helps you out there track things and me feel like I'm moving forward by writing down all the little things I've done over the last bit.

Trello board to keep up to date: https://trello.com/board/bookie/4f18c1ac96c79ec27105f228

New Projects

In an effort to add some features to Bookie I've ended up starting two new repos of code meant to interact with Bookie.

  1. Bookie Parser

This is meant to start taking over the work of reading the page content and readable parsing the important content out. It was a chance to play with Tornado and Heroku. This also means that in the future I'll be able to scale out the readable processing serperatly from the main Bookie website and host. It's pretty bare bones right now and doesn't directly talk to Bookie, but I'll look at adding that integration soon as the API stabilizes and I get more tests going in it.

So far the Heroku bit has been pretty awesome. I have to deal with the fact that the app gets shut down and has to restart on first request, but hopefully that gets better as traffic and use picks up. You can tinker with it at http://readable.bmark.us

  1. Bookie Api

I've been wanting to start up a command line client for some of the Bookie work. The big thing is that I need tools to help manage invites and such. So it's currently very admin centric, but eventually I'd like to get this into a ncurses cool command line interface to pull up recent bookmarks and even do some quick searches via the API. Aren't API's cool. This will also contain the reference Python API implementations so we'll have two implementations soon. One in JS and one in Python.

I've got a beta version (which is really an alpha) up on PyPi so you can

$ pip install bookie_api
$ bookie ping

Build baby build

I spent some quality time with http://build.bmark.us to get the JS tests running via grover and phantomjs and that's awesome. I also added the new projects into the builder as well. So, while I don't have all the tests I need, at least now the ones I do have run consistantly.

Other little tweaks

  • Prettied up the new user invite email and landing page
  • Fixed a bug with dupe tags in the tagcontroller
  • Added more icons from the fontawesome set to pretty up the ui, especially the account page.
  • Lots of changes to the make/build steps for JS and CSS including actually doing the pyscss transition.
  • Everything is now on the final stable release of YUI 3.5. It's been a good ride through the development releases.

Upcoming events

I'll be giving a talk at Penguicon on using YUI for JS app development. If you're in the area stop by. This is Friday April 27th, at 6pm. Then on Saturday I've got a Bookie mini-sprint going on. I'll probably be hacking most of the weekend. Feel free to stop by and check things out.

Bookie Sprint Summary

Bookie Sprint

Today we had the first Bookie sprint of the year. The five of us got together at my place for the main purpose of working on starting a Firefox extension for Bookie.

We ended up pairing up into a couple different tasks. Will and Adam started work on the firefox extension. It looks like they started to get a feel around the sdk and some real progress got forward on that front. It's not quite doing API calls, but I'm excited about how that's going together now.

Matt and Craig worked on some updates for the Chrome extension. We pushed a new updated 0.4 dev version with the delete functions restored and some spinner updates working. It took a bit of time to get things figured out, but what's nice is that the change was really pretty small once everyone got their heads around the JS driving the extension. In a way I like that adding in a feature was so little amount of code. Hopefully we'll be doing some more cool things with the extension going forward.

I spent time working with helping people get up to speed. I also finally updated all the Python side unit/functional tests to pass after all the changes that have gone on under the hood. I also worked on fixing a bug where the API calls wouldn't work unless you had logged into the bmark.us website. Now API calls with valid username/api_key credentials should work just fine. This will hopefully reduce some confusion with getting the extensions going.

We're down to the last few items to getting the 0.4 release out the door. I'm excited with the changes in this release and really excited to start tackling some exciting new changes in my time at PyCon next month. I hope to sprint on some user registration backend and some background processing of tasks. That should get me back into some fun Python side of things vs all the JS code I've been doing recently.

As always, we'd love to have help with any of these projects and anything else you'd like to see updated in Bookie. Hop in irc at #bookie and let's chat.

Thanks again to Will, Craig, Matt, and Adam for a great time sprinting and getting a lot done today. Always good to see others getting interested and involved in the work you do. Maybe one day, we'll get a real community going around Bookie. That'd be awesome.

Book status report...the JS UI is alive!

Well that took a while. Back in September I opened a branch of Bookie to try to do some cool Backbone driven UI stuff. I decided that maintaining a separate mobile UI was going to kill me. So I needed something I could tweak to make mobile friendly without dual sites. Between a JS drive UI and responsive design techniques for the CSS, I should be able to make things not suck so much.

So today, the first part of that has gone live. https://bmark.us now has a UI driven by API calls through Javascript. It's using the YUI MVC stuff that's in their 3.5pr. You might have noticed that Backbone isn't in there any more. I started the app with jQuery thinking it would be a bit more friendly to outside contributions. However, after trying to put together jQuery, Backbone, HistoryJS, etc. I kind of got tired of making due and moved to YUI. It's my choice for JS frameworks and since there's not exactly a pouring of external commits, moving to YUI doesn't hurt me much.

So finally, after waaaay to many months I feel like the site is moving forward. An updated UI that needs some responsive love. An API that need some more methods, logging, etc. Phew.

There are a number of bugs and tweaks that still need working on. I'm going through them and to help be transparent I've started a public Trello board for Bookie so that everyone can see exactly what I'm in the process of, what's next on the board, etc. Hey, if you see something on the board that's on your hot button list, feel free to take it and run with it. Patches welcome :-)

CoffeeHouseCoders 11/23/11: YUI Theater group viewing

Just a heads up, this week's CoffeeHouseCoders (CHC) Detroit-ish will be a bit different. One of the goals of moving the location to the new Caribou was that we get access to the meeting room. This opens up the opportunity for us to have some group discussion and such around various topics. We're going to give that a shot this week with a group viewing of YUI Theater video viewings and JavaScript discussion.

Most of us do at least some JavaScript in our work and projects so I think it's relevant and should be fun to geek out before the holidays start up. I'll have a little projector and speaker and with the new videos from YUIConf 2011 going up, it'll be nice to set aside some time to catch up on some of the recorded presentations. Take a peek and set aside one or two "must watch" videos for Wed night! Not all of the videos are YUI specific, so it should be useful for all of us doing JavaScript.

Bookie weekly status report: Sept 18 2011

Updates this week

Slow week. The changes to the fulltext search involved a small bug in updating the tag_str property of the bookmark. I got this fixed and added a migration to rebuilt the tag_str for all bookmarks in the system. The big sign of this was that the edit ui wasn't showing the tags on the bookmark for edit.

Part of fixing that was a full update to SqlAlchemy 0.7 and moving all post/pre hooks into the new events system SqlAlchemy provides.

Along with that I also updated things to use the new release of Pyramid 1.2. This big thing with this is the new debug toolbar is available for development. It's definitely pretty cool and helps provide some timing/insight as testing requests.

What's next?

There's a ton to work on. The celery task runner, the backbone.js in place edit ui, and investigating using tagger to provide some tag suggestions according to the content.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie weekly status report: Sept 7 2011

Updates this week

Sorry, I'm late on the weekly report. The holiday weekend threw me off. However, I've got good stuff for you. I completed the port of the fulltext search from being database specific to being Whoosh which is pure Python. This should be cool because it reduces code complexity trying to change things up based on the database used. It also been performing really well. The hosted http://bmark.us site is using it for everyone's content.

Be aware, if you do the migration, it takes a while to run. It has to go through and processes every readable record one by one. So when you upgrade, let it run.

What's next

Aside from that, I started playing with an all Javascript front end using Backbone.js. The idea is to see about setting up a very interactive front end that would allow in place edits of bookmark data easily using the existing API. We'll see how it works out.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 28th 2011

Updates this week

With 0.3 out and gone, it's time to move forward on 0.4 tickets. I started out with the big one, edit/add form ui for bookmarks on the page. This now works and when you're browsing urls from your page (/{username}/recent) you'll see a new "E" icon next to the "R" one. You'll also see a "Add" button at the top left.

Once we had the add/edit form, I could setup a bookmarklet to store your current page there. If you log into the site and go to your account page, a bookmarklet url is there for the taking. Drag it to your browser bookmark bar. On some platforms (hey Android!) you can't save a bookmarklet to the bookmarks. You first have to create a normal bookmark, and then edit the link after the fact. To help that out, there's a copy/pastable version of the bookmarklet on the account page. Just long press the text area, select all, and then paste into your bookmark from earlier.

It's my hope that this bookmarklet will help users on Firefox and other browsers use Bookie more. It's not the full extension experience, but hopefully holds people over for now. Note that when storing bookmarks in this way, the content is not immediately available as from the Chrome extension. The cron job that does fulltext parsing runs at 4am each morning.

Next I started some css work. The readable page no longer uses the Cabin font. It's also got some default styles for H1 and such. Let me know if any content looks funny and we can look to add some additional styles going forward.

Next up

For the week ahead, I'll be mainly working on tests. I don't have enough for the new add/edit feature. I'm still working on adding live api tests to the phantom.js test suite. If I get time, I also hope to start work on a private bookmark feature.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 22 2011 (0.3 yay!)

Updates this week: 0.3 out the door

We had to do some hacking of dying limbs to get there, but 0.3 is out the door. This was the biggest release cycle since things started up. However, the features of an API and multi user auth were worth it. There were a ton of small tweaks long the way as well. Here's hoping that 0.4 is a bit faster in the moving and releasing

You'll notice that we're backtracking on having a Firefox extension and a mobile site. Both are horribly broken at the moment. I really want to get this going though and they will be specific todo items in up coming releases. Well, except the Firefox extension part. Honestly, every time I try to get started with the Firefox extension I feel like the team there is just damn hostile to developers. It's beyond ridiculous how unable to get a handle on things there it's been. Confusing docs all over the place, an add on sdk that tries to run a virtualenv, but you don't built it from any of your local python's so your fubar if you have the wrong one, the fact that it's all getting redone again in JS and not Python...Sorry. I'm just not willing to be a fulltime firefox dev to make the dippy thing work.

Links

Next up

  • Work on more of the live api tests. It's running via the latest build of phantom.js and working well
  • Start work on the 0.4 todo list
  • Especially the save/edit ui and a bookmarklet for those.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

CoffeeHouseCoders Detroit hiring a moving truck

Coffee House Coders is moving

Coffee House Coders has been meeting every Wed night for something near two years now. I've not been able to find the announcement for the first one. However, starting August 24th (note, that's a week away, not in two days) we'll make our first location move.

Why? Well, as those of you that show up know, we don't always have all the space that we need. Sometimes we get the place to ourselves, sometimes it's crazy and hard to get seats for everyone. The new location has a private meeting room. So we can be sure we have a set of space available.

Hopefully, this will help attract some of the irregular visitors and give us room to grow. I've always wanted to try to do some more things with CHC. It's great to have that set time each week, but I get the feeling that it would be cool to actually do some mini-user group content. We don't have an east side Python user group, but we have a ton of people interested in that content. We don't have an east side Web Developers group, but again, we have potential interest. Using this meeting location, we might be able to expand and actually do some user group like material in the future.

What is this Coffee do dad?

The idea of CHC is easy, we all want to code/geek out on fun stuff. Every Wed night, from 8pm until 10pm we meet at the coffee shop and sometimes write code and sometimes just BS the time away with fellow geeks. We find it's great to have some time set aside each week for personal projects, learning something new, or whining about what that co-worker did to your code while you weren't looking.

When again?

Our first week in the new location is August 24th. We'll be in the new location from then on as long as things work out. As before, we meet 8pm - 10pm. The last Wed of each month we add an extra hour so get there at 7pm. The 31st will be our next "long edition" CHC and that will be at the new location.

Where is this?

Caribou Coffee
31901 Woodward Avenue
Royal Oak, MI 48073-0984

[googlemaps http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=Caribou+Coffee,+Woodward+Avenue,+Royal+Oak,+MI&aq=0&sll=42.677261,-83.160496&sspn=0.113078,0.099049&vpsrc=0&ie=UTF8&hq=Caribou+Coffee,+Woodward+Avenue,+Royal+Oak,&hnear=Michigan&cid=8777002513835499183&z=13&iwloc=A&output=embed&w=425&h=350" /]

View Larger Map

If you have any questions or have any ideas on topics/wishlist for a local programmers group in the area, feel free to let me know.

Bookie status report: Aug 14th 2011

Updates this week

Finally, the new and fancy Bookie API is reworked and better than ever. The live running site is updated to use the new api and there's a new chrome extension that's update for it as well. Make sure you get the latest dev version of the Chrome extension.

With the api updated, it's time to get to work on finishing up 0.3 for release. This means that we need to finish updating the mobile site and some rework of how the readable content is stored. Since we pull the content out of the page based on actual page you see, we should store these separated from user to user. We don't want to leak logged in info from one user to another.

Farewell Firefox

With the api update, the Firefox is so far out of date as to be unusable. There's some work going on to help catch it up, but I think I'm going to remove it as a feature in the docs and such. We just can't claim it at this time.

New testing

With the new api we hit some testing issues. Since there's no good way I've found to functional test the extension, the tests aren't great and reliable. I also don't have tests setup on some shared code like the bookie.api.js code. I'm thinking of setting up a running instance with some live data with the sole purpose of running tests on the JS code against it. We might even go so far as to setup the extension html files as an actual website and write some functional tests around that. After all, I can load popup.html and do some testing of that as a normal web page. We'll be missing the interactions with background.html and such, but might be worth some attempt.

Any ideas and help with this is most welcome.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.11
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 7th 2011

See, I can do these things once a week, I promise!

Updates this week

Progress this week is all in the api branch. It's taken some time to figure out just what things are going to look like, and I started with the docs. Lots and lots of docs updates for the api. You can check out how things look so far.

  • Gone is the custom json response object. We now send back an http status code and possibly a json message
  • The auth code is all updated. We now use a new api_auth decorator in lib/access.py that will check the api call and take care of the 403 response if things fail.
  • Most of the tests are commented out while we enable/fix them one by one as the new api methods get filled in. Once we're sure thing are where we want, we'll revisit the rest of the apis. Since parts of the functioning site use the api it really needed to be done.
  • The first few new api methods are working correctly and tests are passing in the api branch. This is mostly the /{username}/bmark/XXX urls. So adding, updating, deleting, and getting a bookmark for a user is set.

To make some of the new request changes we needed to update to Pyramid 1.1. This gave us things like request.response.status_int and some changes to the way the http exceptions work.

The goal for the next week is to keep plugging at the api. If you've got any feedback let me know. I'm really hoping to do a really solid job on this updated api and it will help with the extensions, feature updates, and encouraging new developers to get involved.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

Bookie Status Report: Aug 1st 2011

Phew, with PyOhio out of the way and my tutorial done, I can get back to Bookie. Sorry that I missed the last couple of weekly status reports. Honestly, there wasn't a ton, but there was some work going on.

So first, let's go over what's been going on.

Mobile View

I put in a bunch of work redoing the mobile view. It's still using jQuery Mobile, but I redid the javascript that ran it, I implemented an HTML5 history api to doing the tracking as you go from "page" to "page". I put that in quotes because pages are driven by just json api calls. There's no real new html page load after it gets started.

Of course, it's not totally released yet. There's a bug using it with Android. It works in Google Chrome and it seems to work on iOS. The most current version is live on the htp://bmark.us site.

Docs

There's been a lot of work to help prepare the docs for the 0.3 release coming. Craig did some great work helping going through parts of it and finding things I had missed.

API

Ok, here's the one that's going to hurt. I fleshed out most of the rest of the api to get the mobile site updated and such. There's still one view left out there, but the api is going to get reworked. Issac Kelly's talk at PyOhio reminded me that an api is all about resources, not just implementing what your app does via json. So I really need to rethink things. Of course, since the web site, the mobile site, and the extensions all use the api, there will be some work there to get everything updated. This means 0.3 is going to be delayed form where I wanted.

The docs are also pitiful when it comes to the api. Once things are updated I really want to get them really solid. A good api I think will be a big boost to Bookie going forward. So this is another heads up that 0.3 will be a bit delayed, but hopefully this will all pay off.

PyOhio Sprints

The sprints went awesome. While I was hoping to get a lot of coding myself, that was a poor expectation. I spent most of my time working on the mobile site updates and the documentation. We had some six or so people that managed to get it install. It really proved that there's some work to get for easing the installation process.

What was really cool was that once people got it installed, there was work getting done. Dave Brondsema added the urls to the main ui so you can see the url. He also added a branch flag for the bootstap.py script which should help enable bootstrapping both dev and stable versions easily.

I want to really thank everyone for taking part in the sprints and I really hope there are some people that find the project interesting and want to take part going forward.

Todo

Next up on the todo list:

  • redo the api and docs
  • fix this android mobile issue
  • keep working on the install docs. The bootstrap vs the long way was really confusing to users.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

PyOhio 2011: Another year, another great time.

Phew, another PyOhio has come and gone. This year was a great event. I can't say enough good things about the group that puts it together. It's really nice to have something somewhat local to head to every year.

Talks:

  • Data-Transfer Objects Are a Disease. Meet the cure.

    The first I went to on a whim, I wasn't really sure what kind of Data Transfer Object (DTO) was going to be discussed. It turns out, the talk was a praising of the NamedTuple as a great way to pass data in Python. It's got some nice sanity checks, is very lightweight, and helps prevent developers from going crazy adding all kinds of business logic to simple data containers. I'm not sure how they're passing them around their applications, but I can see the appeal. I know I've been bit a few times when unserializing some JSON and a typo in a dict key bites me. I wonder if there's a way to easily get back a list of NamedTuples vs dicts when loading up some JSON transferred around.

  • Aspen: A Next-generation Web Framework

    I checked out the Aspen web framework talk just to check out what new ideas people are playing with when it comes to web tools. I wasn't disappointed, Aspen has some very interesting ideas. The author has done some work to bring back the url meaning something on the project layout. The idea of your layers of the app in a single file is kind of interesting, and I can see how that'd be helpful in some development cases. I end up sitting my controller/template code side by side when I work anyway.

    The weakness I see, is that it's got the same issue PHP has when it comes to helping new developers start good practices. One of the great things about web frameworks, is that they help tell you where and how to organize your code. They give you test directories out of the box, they help bootstrap a good way to get your database connection up in a way that avoids shared sessions across requests, etc. Aspen does a lot less of that, and I could see a younger dev doing way more copy/paste of code than I'd want. It's a little bit of a bare framework, which is great to help integrate you tools of choice, but also provides a barrier to entry for new developers at times.

  • Django and Google App Engine: Why I'm using Flask and EC2

    This talk was from a friend on the west side of the state. As I've been part of the IRC discussions where he's been trying to go through various tools to build some small and simple web apps, it was cool to see the story in one swoop. He's a fan of the microframework. For his use cases, the full stack just threw hurdles in front of him. It's great to hear he's found a tool he loves in flask.

  • Evolving an internal web service

    Taavi gave a great talk on how they worked to rebuild an aging PHP app in Python over a long period of time. It's a great example of what I've been trying to get going for a while, APIs all the time. Everything seems to talk to the new application base via an array of remote methods and this great decoupling has been a boon for them to help provide data to several different systems in a clean way.

    I wish I had more time to try to chat with him. They're doing some cool things with SqlAlchemy, migrating data from old systems to new, and just some very good work and testing done on performance.

  • Creating web apis that are a joy to use

    This was something I really was interested in. Since I've been pushing apis like mad at work and I've been working on my first public one for Bookie, I knew I needed some help/guidance on this. Issac did a great job hitting home the big rules I kind of knew, but wasn't following that well.

    First, document, document, document. I loved his graph of user happiness vs amount of documentation. Users of your api don't get happy until the docs are near complete. Until then, it's just as bad as no documentation. I've got some typing ahead of me.

    The second point was something I was battling with a bit. I tend to think of the api in terms of usage. "You want to do task XXX". At that point, I'm deciding what the user wants to do. In reality it's more about the term "resouce". A resource can be data, a function (send email), or something else along those lines. However you want to expose them via the api in simple distint manner. Just taking your current html view you push to browser users and building an api that is the same doesn't work. After all, the great thing about the api is that people build and do things you didn't think to do on your current application implementation.

  • My Talk, Sqlalchemy Tutorial

    Finally, I was of course at my talk. This year I decided to really didn't like myself and I should do more than a talk, but a two hour tutorial with some hands on coding exercises. The room was full of people of all levels and was a bit more challenging than I originally thought it would be. On top of that, the AC broke and the rooms were over 85 degrees which made holding an audience's attention all that much more challenging.

    In the end, I think things came out ok. I've heard from an array of people that they enjoyed the talk. Once the first hour/talk part was over, most of the room left. We had about seven people that did the hands on code for the second hour. In the end, it sucks, but I can't blame them. If it wasn't my talk I'd have searched out cooler air as well. I hope that people still take the time to try out the hands on code and let me know if they run into any issues. If you do, feel free to email me rharding@mitechie.com.

    Thanks to PyOhio for letting me take a shot at something more classroom like. It's a new challenge to go from a talk to a tutorial and I encourage people to try it out.

Networking

I think it's true what they say, as you go to more and more conference you tend to do and learn less in the talks themselves, but make up for it with the great networking. This year it was very noticeable. What was great was that I got a chance to meet several people I've been following on Twitter for a while. These are people that are interesting, respected, and meeting them was kind of mini-starlet moments.

I got to have a great discussion around apis and self bootstrapping application installs with Issac Kelly. I met Michael Trier whos been a great Python presence on Twitter for a while now. I also caught up with the Ohio crew and guys Dave, Mike, and Catherine. If you run into these guys start a conversation, it'll be worth it.

I also had a ton of great conversations on things from new people that I really wish I did a better job of remembering and tracking down names for. Sorry that I don't call you all out.

All the hallway stuff really helped make this PyOhio a great one for me.

Sprints

Another great thing PyOhio does are the sprints. Unfortunately I could only do them on Saturday, but man what fun it was. I think we managed to get 6 or so people with their own up and running Bookie instance running. We had nearly a dozen people hacking on things at one point. We had some fun hammering pypi from the wifi network and some really good ideas came up to help make the installs a bit easier. At the end of the night we had a pull request and some definite interest going forward. I hope that the people that sprinted on Bookie found it interesting to take part in and maybe learned something. I know I've got a lot of work to do still

I'll have a separate Bookie status report out later with some details on changes and things.

A reminder as well, if you'd like to have a hosted Bookie account on bmark.us just sign up to the waiting list here: http://goo.gl/BBn2b

Bookie Status Report: July 6th 2011

A slow plodding week in Bookie progress this week. Thanks to Greg and Craig for some help in cleaning up a feature.

  • New version of the Chrome extension that will do a much better job of error checking on the options page. Make sure to get the updated extension.
  • The Account page has been cleaned up. You can now view your Api Key, change your password, and update your base account information such as name and email. I also did some ui cleanup so things should start to be a bit prettier. Let me know if you have any suggestions to the account ui.
  • I fixed the Google fonts used so things might look a bit different. I'm using the Cabin font for the body. I think I want to change it for the readable version body though. It's not great for large bodies of text. What do you think?
  • I also updated the mobile view to the jQuery Mobile beta release. I finally found the cause of a nasty bug and things should work ok. I still need to write my own history api wrapper. So navigating back and such is pretty broken still.
  • Per request of Greg and Craig, when the bookmark isn't for the current year we now add the '10 or whatever to the calendar graphic. You can see an example over here.

Next up on the todo list:

  • We need a password reset feature for those that have forgotten their passwords. There's some database side start to this, but no ui for it yet.
  • I want to start on a page in accounts for viewing your readable results such as displaying 404 counts and such. See ticket #47

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.4
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 29 2011

Shorter update list for Bookie this week. Took some vacation and not a ton of time hacking.

  • Fixed a bug in the readable/existing.py handling a non-asci url
  • Fixed all of the js/css to come via https so site should be happy lock for you
  • Added some app logging, log failed logins, imports, exports of bookmarks
  • Start the /{username}/account section with it's own sub nav and some css for that. The logout/export/import moved there

Most of this got pushed out in an incomplete form onto the live site. From here I need to add the edit form for the user's details and clean up the navigation to be reusable across the different pages.

I also want to keep adding some logging and stats counting so we can start to provide some data porn to both myself and users of their info.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 22nd 2011

Phew, it's been a good week with Bookie. First, we have some new ways to contact and follow Bookie. There's now a Google Group for better long form discussions and assistance. You have a feature idea, question, or just feedback? Then go ahead and send it to the list. Don't forget to follow BookieBmarks on Twitter. I want to start using these things to bounce ideas off of people as we move forward.

We managed to get user authentication working and added support across the site for routes for each user in the system. This was great because it means that http://bmark.us is a live running instance with a few different real users. So far we're running seven different users on the system. Part of that was getting all the routes updated, the queries, the templates to only show links for import/export if you're logged in, all of that stuff. The extension needed a little bit of updates, but the only big thing is changing your API URL to have your username in it.

We moved the documentation to http://docs.bmark.us and I tried to update the links in the docs as well. Much as Identi.ca is a running instance of Status.net, Bmark.us is a running instance of Bookie. I just can't get a cool url for Bookie such as http://bookie.net.

With the new site I purchased an SSL certificate. So you'll notice that the site is now all behind https. Part of the other sysadmin items were to setup twice daily pgdumps of the data which is sync'd off to S3, munin to monitor the server resources, including plugins for Nginx and Postgresql, and some Nginx config tweaks which I still need to document in the docs.

All of the code updates are available in the develop branch on Github. If you've got any questions feel free to hop into #bookie on irc and let me know.

With all this going on I'm looking for some early alpha testers. If you're interested in Bookie, but couldn't, or didn't want to, setup your own instance, let me know. I'd love to get another 6-10 people on this instance. The feedback of current users has already paid dividends. You can thank them for a pair of new features in the Chrome extension, including a new keyboard shortcut (ctrl-alt-d) and support for helping auto complete recently used tags.

Coming up, we need to work on moving the app forward by adding a user account page, creating add/edit abilities for authenticated users, and a bookmarklet that uses the add ability for mobile devices and other browsers.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 15th 2011

I just finished up reading Start Small, Stay Small and there were some good points in there. One is that writing about your progress on a project each week helps people move forward. There is something about putting down what you've accomplished and what you plan into the public that helps keep the motivation motor running.

In an effort to keep Bookie from stagnating, I think that's a good thing to start doing. Count this as the first of a series of weekly progress reports I'm going to be doing. I also like that it helps show, beyond links to commit logs, that Bookie is moving forward and getting updates.

This past week has been a bit crazy. There hasn't been a ton of time to put into things, but I've managed to move a few big things forward:

First, work on making Bookie work via user accounts and logins is moving forward. Basically all of the urls in the application needed to be updated. Currently there are two sets. If you leave out a username from the url, you get overall, full site info.

In this way, a url of /recent will pull the 50 most recent bookmarks from all users on the site. However, /rick/recent will only pull the 50 most recent bookmarks of the user rick. The API urls needed to be updated as well. There's a ton of work in getting this going, but it's a major step of progress to allowing me to host a version of Bookie that other users can sign up for. Since that's really the big goal that I've set myself by the end of the year, I'm feeling good on this one.

The idea of multiple users has me realizing that my little readability.py script that fetches url content from bookmarks and stores the clean, readble parsed html for that page needs some work. It'll never scale that way. So I've split the work into a couple of parts.

One part is a node.js script that will fetch a list of urls to go fetch and asyncronously goes out and fetches the html content. It then shoves the bookmark id and the content into a beanstalkd queue for processing. The queue is polled by a python script that then calls a new Bookie API call with the content and the id. Bookie then runs the parsing code against the content and stores it in the database. The async code on node.js can fetch the html content in a hurry. In testing with my SSD hard drive and sqlite, I'm able to pull, process, and store more than one url per second. This is with 1 node.js producer and two running instances of the python consumer.

I'm definitely looking forward to ramping this up on a real server with Postgresql running. I'd love to be able to pull down and parse content at some decent rates to be able to cope with new users signing up to the service.

So that's this week's report. Next up is more work on the multi user setup. The tag urls still need work and all of the unit tests that I had need to be updated to test the new urls. This also means some duplicate tests to check both with/without usernames in the urls. Work is never done!

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie