Python

Bookie Weekly Status Report Returns! - April 15 2012

Ok, I'm overdue for a 'weekly' status report. I'm going to try to kick this back into gear as it helps you out there track things and me feel like I'm moving forward by writing down all the little things I've done over the last bit.

Trello board to keep up to date: https://trello.com/board/bookie/4f18c1ac96c79ec27105f228

New Projects

In an effort to add some features to Bookie I've ended up starting two new repos of code meant to interact with Bookie.

  1. Bookie Parser

This is meant to start taking over the work of reading the page content and readable parsing the important content out. It was a chance to play with Tornado and Heroku. This also means that in the future I'll be able to scale out the readable processing serperatly from the main Bookie website and host. It's pretty bare bones right now and doesn't directly talk to Bookie, but I'll look at adding that integration soon as the API stabilizes and I get more tests going in it.

So far the Heroku bit has been pretty awesome. I have to deal with the fact that the app gets shut down and has to restart on first request, but hopefully that gets better as traffic and use picks up. You can tinker with it at http://readable.bmark.us

  1. Bookie Api

I've been wanting to start up a command line client for some of the Bookie work. The big thing is that I need tools to help manage invites and such. So it's currently very admin centric, but eventually I'd like to get this into a ncurses cool command line interface to pull up recent bookmarks and even do some quick searches via the API. Aren't API's cool. This will also contain the reference Python API implementations so we'll have two implementations soon. One in JS and one in Python.

I've got a beta version (which is really an alpha) up on PyPi so you can

$ pip install bookie_api
$ bookie ping

Build baby build

I spent some quality time with http://build.bmark.us to get the JS tests running via grover and phantomjs and that's awesome. I also added the new projects into the builder as well. So, while I don't have all the tests I need, at least now the ones I do have run consistantly.

Other little tweaks

  • Prettied up the new user invite email and landing page
  • Fixed a bug with dupe tags in the tagcontroller
  • Added more icons from the fontawesome set to pretty up the ui, especially the account page.
  • Lots of changes to the make/build steps for JS and CSS including actually doing the pyscss transition.
  • Everything is now on the final stable release of YUI 3.5. It's been a good ride through the development releases.

Upcoming events

I'll be giving a talk at Penguicon on using YUI for JS app development. If you're in the area stop by. This is Friday April 27th, at 6pm. Then on Saturday I've got a Bookie mini-sprint going on. I'll probably be hacking most of the weekend. Feel free to stop by and check things out.

A few ideas, quick ways to get on the Bookie contrib list

Quick ideas for improving Bookie

Well, with all the great stuff going on with Bookie, I've gotten a bit buried in some big changes. The background processing and importing updates are going to take a bit to get right.

This means, there's a great chance for others to hack up the little tweaks that we need to really add some polish to Bookie. So below I've listed a few ideas that should be pretty simple things to add, but with a really good positive and visible effect on the site.

  • Add notification that user has invites

    Now that invites are there, we should highlight a user's account navigation link to let them know they have invites available. I'll periodically add them to the system, and we don't want users to have to go to their account page each time to see they've got invites. I think a simple adding of one of the envelope/message type icons from our font-icon set would be perfect, with some sort of hover message to start. We might also want to highlight the block in their account page so it stands out that the invites are available.

  • Flash message system.

    We want to be able to let users know things have happened successfully after doing something that redirected them. Imports are going to be doing this, saving/updating bookmarks, etc. It'd be nice to have a consistent type of ui to drop flash messages in and them to show after a redirect.

  • Show new user message if self bookmarks page has no results

    When a new users starts up and logs in, they default to their own page of bookmarks...which is going to be empty. So we should detect this in our JS code that fetches the results and displays a set of default content with links to things like importing instructions, where to get the chrome extensions, and other handy new user tips.

    Some of this might also be nice to use for the email that a new user gets when they've been invited to Bookie.

  • Add firefox bookmark importer

    Ok, so this one is a bit more involved, but really, it's a single class and a couple of Python methods. The hard part is reading in and figuring out how to match bookmarks to tags in Firefox's JSON dump of bookmarks. Once we get the Firefox extension rolling, it'll be great to have a good import system for the browser as well.

Well, here are four things I'd love to see happen in the near future to help make the experience a level nicer for everyone. If you're interested in all or have any questions, ping me in #bookie in irc or shoot me a comment below. I'd be happy to help walk anyone through these or any other ideas you might have.

Bookie PyCon 2012 Sprint Report

http://www.flickr.com/photos/66176388@N00/4592858614

So PyCon sprints, what can you say? You go, you hack...and hack...and some some point you take a break for a beverage...and hack some more.

Last year I got the real movement behind Bookie. So this really marks the one year anniversary for Bookie. I've had it as a side project to hack on in my spare time for the last whole year. In that time, honestly it's not crazy different from where it started. However, it's gone through multiple JS rewrites, two different UI designs, and a whole lot more. I've really learned a lot about development, testing, and making some hard choices over the last year. I hope by this time next year I'll have announced Bookie on some big site (reddit/hacker news?) and survived.

This sprint though wasn't the time. So what did I get done?

tl;dr

Bookie got improvements

  • Better JS tests
  • Better PY tests
  • Start of HTML5 history
  • Invite system
  • Threaded content fetcher
  • Start of celery background runner

First, I started out by working on getting the html5 history stuff going. It's not perfect yet, but it's started and I really realized I needed to have a better way to do JS tests, so...

Next I redid the JS tests. I don't want to have to fire up the application in order to run my JS tests. I also don't want to have to hit the database and such. This means I had to change the API tests. Rather than making real requests/responses, I test that the classes build the right type of requests. I verify the url, data payload, etc are correct.

Once the JS tests were redone, I realized that I hated how I had yXXX.js as the filenames and redid those as well. While I was cleaning up I dumped a bunch of old code we no longer needed. Basically tons of gardening cleaning out the weeds.

With that out of the way, the next day of sprints was all about getting an invite system underway. I originally wanted to do a throttled signup process, so anyone could sign up, but then I realized that really, invites will work better. The people using Bookie now will know who'd be interested in testing and if someone really wants to get in, they'll contact me in some fashion.

With that up, I got to spend the next day fixing bugs in everything. Wheeee! What was cool was that I managed to get a few people at the sprints curious about Bookie and testing things out. Nothing exposes bugs like new users. During this process I spent some time cleaning house on the Python side of things and making tests easier to run/write.

Finally, I've started work on the background processing using Celery. I've got a big hurdle in that, but my cron'd stats processes are working and I've almost got imports running as out of process celery children. That should really help with new users. You know at some point they'll come flooding in right? :-)

Overall, while I didn't get a ton of new user facing features going, I did a TON of clean up and maintenance. As one person expressed "Wow, there's a lot more in here than I expected when you said it was a bookmark application". Bookie has really grown over the last year and she needs me to spend some time giving things some love before moving forward too fast. The sprints really gave me a chance to do that, all while hanging out and chatting with really smart people. What more can I ask for?

PyCon 2012: What a ride!

Phew, tiring trip to PyCon this year. This was my second year after hitting up my first last year. The conference definitely felt larger than last year as they crossed 2,200 attendees. It's unbelievable to see how large the Python community has gotten. I can't stress what great job the people that put this together.

Last year I hardly knew anyone. This year, however, I got to put faces to people I've interacted with over the last year, welcome back those I met last year, and get some face to face time with new co-workers from Canonical. The social aspect was a larger chunk of my time this year for sure.

Side note, I listen to The Changelog podcast from time to time, and I love their question on who you'd love to pair up/hack with as a programming hero type question. I got to meet and greet mine at this PyCon by meeting up with Mike Bayer. He's behind some great tools like SqlAlchemy and Mako. What I love is that, not only does he rock the code part, but the community part as well. I'm always amazed to see the time he puts into his responses to questions and support avenues. Highlight of my PyCon for sure.

I'll post a seperate blog post on my sprint notes. I feel that if you're going to go, you might as well stay for sprints. I get as much out of that as the conference parts itself. I think I made some good progress on things for Bookie this year. The big thing is that an invite system is in place, so if you'd like an account on Bmark.us let me know and I'll toss an invite your way.

Notes

  • Introduction to Metaclasses
    • Basic but reminded me how the bits worked and had some good examples. I like this because I often write 'the code I want to be writing' and then write my modules/etc to fit and metaclasses help with this sometimes.
  • Fast Test, Slow Test
    • Just a reminder that fast tests are true unit tests and run during dev which helps make things easier/faster as you go vs the whole 'mad code' then wait for feedback on how wrong you are.
  • Practical Machine Learning in Python
    • mloss.org - check out for lots of notes/etc on ML in OSS
    • ml-class.org - teach me some ML please
    • sluggerml - app he built as a ML demo
    • scikit-learn : lots of potential, very active right now
  • Introduction to PDB
    • whoa...where have you been all my life 'until' command?
    • use 'where' more to move up stack vs adding more debug lines
  • Flexing SQLAlchemy's Relational Power
  • Hand Coded Applications with SQLAlchemy
    • <3 SqlAchemy. Some really good examples of writing less code by automating the biolerplate with conventions.
  • Web Server Bottlenecks And Performance Tuning
    • lesson: if you think it's apache's fault think again. You're probably doing it wrong.
  • Advanced Celery
    • check out cyme https://github.com/celery/cyme, possible way to more easily run/distribute celery work?
    • cool to see implementations of map/reduce using celery
    • chords and groups are good, check them out more
  • Building A Python-Based Search Engine
    • Good talk for into into terms and such for fulltext search
  • Lighting talks of note
    • py3 porting docs: http://docs.python.org/howto/pyporting
    • bpython rewind feature is full of win over ipython
    • 'new virtualenv' trying to get into stdlib for py3.3, cool!
    • asyncdynamo cool example of async boto requests for high performance working with AWS api (uses tornado)
    • I WANT the concurrent.features library...but it's Python 3 :(

Bookie Sprint Summary

Bookie Sprint

Today we had the first Bookie sprint of the year. The five of us got together at my place for the main purpose of working on starting a Firefox extension for Bookie.

We ended up pairing up into a couple different tasks. Will and Adam started work on the firefox extension. It looks like they started to get a feel around the sdk and some real progress got forward on that front. It's not quite doing API calls, but I'm excited about how that's going together now.

Matt and Craig worked on some updates for the Chrome extension. We pushed a new updated 0.4 dev version with the delete functions restored and some spinner updates working. It took a bit of time to get things figured out, but what's nice is that the change was really pretty small once everyone got their heads around the JS driving the extension. In a way I like that adding in a feature was so little amount of code. Hopefully we'll be doing some more cool things with the extension going forward.

I spent time working with helping people get up to speed. I also finally updated all the Python side unit/functional tests to pass after all the changes that have gone on under the hood. I also worked on fixing a bug where the API calls wouldn't work unless you had logged into the bmark.us website. Now API calls with valid username/api_key credentials should work just fine. This will hopefully reduce some confusion with getting the extensions going.

We're down to the last few items to getting the 0.4 release out the door. I'm excited with the changes in this release and really excited to start tackling some exciting new changes in my time at PyCon next month. I hope to sprint on some user registration backend and some background processing of tasks. That should get me back into some fun Python side of things vs all the JS code I've been doing recently.

As always, we'd love to have help with any of these projects and anything else you'd like to see updated in Bookie. Hop in irc at #bookie and let's chat.

Thanks again to Will, Craig, Matt, and Adam for a great time sprinting and getting a lot done today. Always good to see others getting interested and involved in the work you do. Maybe one day, we'll get a real community going around Bookie. That'd be awesome.

Bookie weekly status report: Sept 18 2011

Updates this week

Slow week. The changes to the fulltext search involved a small bug in updating the tag_str property of the bookmark. I got this fixed and added a migration to rebuilt the tag_str for all bookmarks in the system. The big sign of this was that the edit ui wasn't showing the tags on the bookmark for edit.

Part of fixing that was a full update to SqlAlchemy 0.7 and moving all post/pre hooks into the new events system SqlAlchemy provides.

Along with that I also updated things to use the new release of Pyramid 1.2. This big thing with this is the new debug toolbar is available for development. It's definitely pretty cool and helps provide some timing/insight as testing requests.

What's next?

There's a ton to work on. The celery task runner, the backbone.js in place edit ui, and investigating using tagger to provide some tag suggestions according to the content.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 14th 2011

Updates this week

Finally, the new and fancy Bookie API is reworked and better than ever. The live running site is updated to use the new api and there's a new chrome extension that's update for it as well. Make sure you get the latest dev version of the Chrome extension.

With the api updated, it's time to get to work on finishing up 0.3 for release. This means that we need to finish updating the mobile site and some rework of how the readable content is stored. Since we pull the content out of the page based on actual page you see, we should store these separated from user to user. We don't want to leak logged in info from one user to another.

Farewell Firefox

With the api update, the Firefox is so far out of date as to be unusable. There's some work going on to help catch it up, but I think I'm going to remove it as a feature in the docs and such. We just can't claim it at this time.

New testing

With the new api we hit some testing issues. Since there's no good way I've found to functional test the extension, the tests aren't great and reliable. I also don't have tests setup on some shared code like the bookie.api.js code. I'm thinking of setting up a running instance with some live data with the sole purpose of running tests on the JS code against it. We might even go so far as to setup the extension html files as an actual website and write some functional tests around that. After all, I can load popup.html and do some testing of that as a normal web page. We'll be missing the interactions with background.html and such, but might be worth some attempt.

Any ideas and help with this is most welcome.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.11
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 7th 2011

See, I can do these things once a week, I promise!

Updates this week

Progress this week is all in the api branch. It's taken some time to figure out just what things are going to look like, and I started with the docs. Lots and lots of docs updates for the api. You can check out how things look so far.

  • Gone is the custom json response object. We now send back an http status code and possibly a json message
  • The auth code is all updated. We now use a new api_auth decorator in lib/access.py that will check the api call and take care of the 403 response if things fail.
  • Most of the tests are commented out while we enable/fix them one by one as the new api methods get filled in. Once we're sure thing are where we want, we'll revisit the rest of the apis. Since parts of the functioning site use the api it really needed to be done.
  • The first few new api methods are working correctly and tests are passing in the api branch. This is mostly the /{username}/bmark/XXX urls. So adding, updating, deleting, and getting a bookmark for a user is set.

To make some of the new request changes we needed to update to Pyramid 1.1. This gave us things like request.response.status_int and some changes to the way the http exceptions work.

The goal for the next week is to keep plugging at the api. If you've got any feedback let me know. I'm really hoping to do a really solid job on this updated api and it will help with the extensions, feature updates, and encouraging new developers to get involved.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

Bookie Status Report: Aug 1st 2011

Phew, with PyOhio out of the way and my tutorial done, I can get back to Bookie. Sorry that I missed the last couple of weekly status reports. Honestly, there wasn't a ton, but there was some work going on.

So first, let's go over what's been going on.

Mobile View

I put in a bunch of work redoing the mobile view. It's still using jQuery Mobile, but I redid the javascript that ran it, I implemented an HTML5 history api to doing the tracking as you go from "page" to "page". I put that in quotes because pages are driven by just json api calls. There's no real new html page load after it gets started.

Of course, it's not totally released yet. There's a bug using it with Android. It works in Google Chrome and it seems to work on iOS. The most current version is live on the htp://bmark.us site.

Docs

There's been a lot of work to help prepare the docs for the 0.3 release coming. Craig did some great work helping going through parts of it and finding things I had missed.

API

Ok, here's the one that's going to hurt. I fleshed out most of the rest of the api to get the mobile site updated and such. There's still one view left out there, but the api is going to get reworked. Issac Kelly's talk at PyOhio reminded me that an api is all about resources, not just implementing what your app does via json. So I really need to rethink things. Of course, since the web site, the mobile site, and the extensions all use the api, there will be some work there to get everything updated. This means 0.3 is going to be delayed form where I wanted.

The docs are also pitiful when it comes to the api. Once things are updated I really want to get them really solid. A good api I think will be a big boost to Bookie going forward. So this is another heads up that 0.3 will be a bit delayed, but hopefully this will all pay off.

PyOhio Sprints

The sprints went awesome. While I was hoping to get a lot of coding myself, that was a poor expectation. I spent most of my time working on the mobile site updates and the documentation. We had some six or so people that managed to get it install. It really proved that there's some work to get for easing the installation process.

What was really cool was that once people got it installed, there was work getting done. Dave Brondsema added the urls to the main ui so you can see the url. He also added a branch flag for the bootstap.py script which should help enable bootstrapping both dev and stable versions easily.

I want to really thank everyone for taking part in the sprints and I really hope there are some people that find the project interesting and want to take part going forward.

Todo

Next up on the todo list:

  • redo the api and docs
  • fix this android mobile issue
  • keep working on the install docs. The bootstrap vs the long way was really confusing to users.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

PyOhio 2011: Another year, another great time.

Phew, another PyOhio has come and gone. This year was a great event. I can't say enough good things about the group that puts it together. It's really nice to have something somewhat local to head to every year.

Talks:

  • Data-Transfer Objects Are a Disease. Meet the cure.

    The first I went to on a whim, I wasn't really sure what kind of Data Transfer Object (DTO) was going to be discussed. It turns out, the talk was a praising of the NamedTuple as a great way to pass data in Python. It's got some nice sanity checks, is very lightweight, and helps prevent developers from going crazy adding all kinds of business logic to simple data containers. I'm not sure how they're passing them around their applications, but I can see the appeal. I know I've been bit a few times when unserializing some JSON and a typo in a dict key bites me. I wonder if there's a way to easily get back a list of NamedTuples vs dicts when loading up some JSON transferred around.

  • Aspen: A Next-generation Web Framework

    I checked out the Aspen web framework talk just to check out what new ideas people are playing with when it comes to web tools. I wasn't disappointed, Aspen has some very interesting ideas. The author has done some work to bring back the url meaning something on the project layout. The idea of your layers of the app in a single file is kind of interesting, and I can see how that'd be helpful in some development cases. I end up sitting my controller/template code side by side when I work anyway.

    The weakness I see, is that it's got the same issue PHP has when it comes to helping new developers start good practices. One of the great things about web frameworks, is that they help tell you where and how to organize your code. They give you test directories out of the box, they help bootstrap a good way to get your database connection up in a way that avoids shared sessions across requests, etc. Aspen does a lot less of that, and I could see a younger dev doing way more copy/paste of code than I'd want. It's a little bit of a bare framework, which is great to help integrate you tools of choice, but also provides a barrier to entry for new developers at times.

  • Django and Google App Engine: Why I'm using Flask and EC2

    This talk was from a friend on the west side of the state. As I've been part of the IRC discussions where he's been trying to go through various tools to build some small and simple web apps, it was cool to see the story in one swoop. He's a fan of the microframework. For his use cases, the full stack just threw hurdles in front of him. It's great to hear he's found a tool he loves in flask.

  • Evolving an internal web service

    Taavi gave a great talk on how they worked to rebuild an aging PHP app in Python over a long period of time. It's a great example of what I've been trying to get going for a while, APIs all the time. Everything seems to talk to the new application base via an array of remote methods and this great decoupling has been a boon for them to help provide data to several different systems in a clean way.

    I wish I had more time to try to chat with him. They're doing some cool things with SqlAlchemy, migrating data from old systems to new, and just some very good work and testing done on performance.

  • Creating web apis that are a joy to use

    This was something I really was interested in. Since I've been pushing apis like mad at work and I've been working on my first public one for Bookie, I knew I needed some help/guidance on this. Issac did a great job hitting home the big rules I kind of knew, but wasn't following that well.

    First, document, document, document. I loved his graph of user happiness vs amount of documentation. Users of your api don't get happy until the docs are near complete. Until then, it's just as bad as no documentation. I've got some typing ahead of me.

    The second point was something I was battling with a bit. I tend to think of the api in terms of usage. "You want to do task XXX". At that point, I'm deciding what the user wants to do. In reality it's more about the term "resouce". A resource can be data, a function (send email), or something else along those lines. However you want to expose them via the api in simple distint manner. Just taking your current html view you push to browser users and building an api that is the same doesn't work. After all, the great thing about the api is that people build and do things you didn't think to do on your current application implementation.

  • My Talk, Sqlalchemy Tutorial

    Finally, I was of course at my talk. This year I decided to really didn't like myself and I should do more than a talk, but a two hour tutorial with some hands on coding exercises. The room was full of people of all levels and was a bit more challenging than I originally thought it would be. On top of that, the AC broke and the rooms were over 85 degrees which made holding an audience's attention all that much more challenging.

    In the end, I think things came out ok. I've heard from an array of people that they enjoyed the talk. Once the first hour/talk part was over, most of the room left. We had about seven people that did the hands on code for the second hour. In the end, it sucks, but I can't blame them. If it wasn't my talk I'd have searched out cooler air as well. I hope that people still take the time to try out the hands on code and let me know if they run into any issues. If you do, feel free to email me rharding@mitechie.com.

    Thanks to PyOhio for letting me take a shot at something more classroom like. It's a new challenge to go from a talk to a tutorial and I encourage people to try it out.

Networking

I think it's true what they say, as you go to more and more conference you tend to do and learn less in the talks themselves, but make up for it with the great networking. This year it was very noticeable. What was great was that I got a chance to meet several people I've been following on Twitter for a while. These are people that are interesting, respected, and meeting them was kind of mini-starlet moments.

I got to have a great discussion around apis and self bootstrapping application installs with Issac Kelly. I met Michael Trier whos been a great Python presence on Twitter for a while now. I also caught up with the Ohio crew and guys Dave, Mike, and Catherine. If you run into these guys start a conversation, it'll be worth it.

I also had a ton of great conversations on things from new people that I really wish I did a better job of remembering and tracking down names for. Sorry that I don't call you all out.

All the hallway stuff really helped make this PyOhio a great one for me.

Sprints

Another great thing PyOhio does are the sprints. Unfortunately I could only do them on Saturday, but man what fun it was. I think we managed to get 6 or so people with their own up and running Bookie instance running. We had nearly a dozen people hacking on things at one point. We had some fun hammering pypi from the wifi network and some really good ideas came up to help make the installs a bit easier. At the end of the night we had a pull request and some definite interest going forward. I hope that the people that sprinted on Bookie found it interesting to take part in and maybe learned something. I know I've got a lot of work to do still

I'll have a separate Bookie status report out later with some details on changes and things.

A reminder as well, if you'd like to have a hosted Bookie account on bmark.us just sign up to the waiting list here: http://goo.gl/BBn2b

Bookie Status Report: July 6th 2011

A slow plodding week in Bookie progress this week. Thanks to Greg and Craig for some help in cleaning up a feature.

  • New version of the Chrome extension that will do a much better job of error checking on the options page. Make sure to get the updated extension.
  • The Account page has been cleaned up. You can now view your Api Key, change your password, and update your base account information such as name and email. I also did some ui cleanup so things should start to be a bit prettier. Let me know if you have any suggestions to the account ui.
  • I fixed the Google fonts used so things might look a bit different. I'm using the Cabin font for the body. I think I want to change it for the readable version body though. It's not great for large bodies of text. What do you think?
  • I also updated the mobile view to the jQuery Mobile beta release. I finally found the cause of a nasty bug and things should work ok. I still need to write my own history api wrapper. So navigating back and such is pretty broken still.
  • Per request of Greg and Craig, when the bookmark isn't for the current year we now add the '10 or whatever to the calendar graphic. You can see an example over here.

Next up on the todo list:

  • We need a password reset feature for those that have forgotten their passwords. There's some database side start to this, but no ui for it yet.
  • I want to start on a page in accounts for viewing your readable results such as displaying 404 counts and such. See ticket #47

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.4
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 29 2011

Shorter update list for Bookie this week. Took some vacation and not a ton of time hacking.

  • Fixed a bug in the readable/existing.py handling a non-asci url
  • Fixed all of the js/css to come via https so site should be happy lock for you
  • Added some app logging, log failed logins, imports, exports of bookmarks
  • Start the /{username}/account section with it's own sub nav and some css for that. The logout/export/import moved there

Most of this got pushed out in an incomplete form onto the live site. From here I need to add the edit form for the user's details and clean up the navigation to be reusable across the different pages.

I also want to keep adding some logging and stats counting so we can start to provide some data porn to both myself and users of their info.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 22nd 2011

Phew, it's been a good week with Bookie. First, we have some new ways to contact and follow Bookie. There's now a Google Group for better long form discussions and assistance. You have a feature idea, question, or just feedback? Then go ahead and send it to the list. Don't forget to follow BookieBmarks on Twitter. I want to start using these things to bounce ideas off of people as we move forward.

We managed to get user authentication working and added support across the site for routes for each user in the system. This was great because it means that http://bmark.us is a live running instance with a few different real users. So far we're running seven different users on the system. Part of that was getting all the routes updated, the queries, the templates to only show links for import/export if you're logged in, all of that stuff. The extension needed a little bit of updates, but the only big thing is changing your API URL to have your username in it.

We moved the documentation to http://docs.bmark.us and I tried to update the links in the docs as well. Much as Identi.ca is a running instance of Status.net, Bmark.us is a running instance of Bookie. I just can't get a cool url for Bookie such as http://bookie.net.

With the new site I purchased an SSL certificate. So you'll notice that the site is now all behind https. Part of the other sysadmin items were to setup twice daily pgdumps of the data which is sync'd off to S3, munin to monitor the server resources, including plugins for Nginx and Postgresql, and some Nginx config tweaks which I still need to document in the docs.

All of the code updates are available in the develop branch on Github. If you've got any questions feel free to hop into #bookie on irc and let me know.

With all this going on I'm looking for some early alpha testers. If you're interested in Bookie, but couldn't, or didn't want to, setup your own instance, let me know. I'd love to get another 6-10 people on this instance. The feedback of current users has already paid dividends. You can thank them for a pair of new features in the Chrome extension, including a new keyboard shortcut (ctrl-alt-d) and support for helping auto complete recently used tags.

Coming up, we need to work on moving the app forward by adding a user account page, creating add/edit abilities for authenticated users, and a bookmarklet that uses the add ability for mobile devices and other browsers.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 15th 2011

I just finished up reading Start Small, Stay Small and there were some good points in there. One is that writing about your progress on a project each week helps people move forward. There is something about putting down what you've accomplished and what you plan into the public that helps keep the motivation motor running.

In an effort to keep Bookie from stagnating, I think that's a good thing to start doing. Count this as the first of a series of weekly progress reports I'm going to be doing. I also like that it helps show, beyond links to commit logs, that Bookie is moving forward and getting updates.

This past week has been a bit crazy. There hasn't been a ton of time to put into things, but I've managed to move a few big things forward:

First, work on making Bookie work via user accounts and logins is moving forward. Basically all of the urls in the application needed to be updated. Currently there are two sets. If you leave out a username from the url, you get overall, full site info.

In this way, a url of /recent will pull the 50 most recent bookmarks from all users on the site. However, /rick/recent will only pull the 50 most recent bookmarks of the user rick. The API urls needed to be updated as well. There's a ton of work in getting this going, but it's a major step of progress to allowing me to host a version of Bookie that other users can sign up for. Since that's really the big goal that I've set myself by the end of the year, I'm feeling good on this one.

The idea of multiple users has me realizing that my little readability.py script that fetches url content from bookmarks and stores the clean, readble parsed html for that page needs some work. It'll never scale that way. So I've split the work into a couple of parts.

One part is a node.js script that will fetch a list of urls to go fetch and asyncronously goes out and fetches the html content. It then shoves the bookmark id and the content into a beanstalkd queue for processing. The queue is polled by a python script that then calls a new Bookie API call with the content and the id. Bookie then runs the parsing code against the content and stores it in the database. The async code on node.js can fetch the html content in a hurry. In testing with my SSD hard drive and sqlite, I'm able to pull, process, and store more than one url per second. This is with 1 node.js producer and two running instances of the python consumer.

I'm definitely looking forward to ramping this up on a real server with Postgresql running. I'd love to be able to pull down and parse content at some decent rates to be able to cope with new users signing up to the service.

So that's this week's report. Next up is more work on the multi user setup. The tag urls still need work and all of the unit tests that I had need to be updated to test the new urls. This also means some duplicate tests to check both with/without usernames in the urls. Work is never done!

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

In search of better enums

I've been fighting now to deal with a pretty common use case in my Pylons applications. I often have a db model (with the marvelous SqlAlchemy) and the object in question has some type of limited set of options for a field. It's basically an enum field, but I don't want to deal with the enum on the DB end. It makes it hard to support with different databases and such.

One solution is to actually create a table for the options. For instance you might have a state table that is joined to your object. It has all the US states in there for you and whenever you query you have to do a join on the state table to get the pretty name. The use cases I'm talking about are sets of 2->5 options. The models might have 5 or more of these as well. So I don't want to deal with creating tables, migrations, and joins for them all.

I had a few failed attempts, but I ended up learning some great new things about SqlAlchemy and have something working. First up I need to create an object that contains the options. Forgive the cheesy names, but let's say we have a table with a Severity field in it.

[sourcecode lang="python"] # first we want to inherit from a common base for all of these enums

class MyEnum(object): """Enum-like object type

Each implementation need only extend MyEnum and define the dict _enum

""" def __getattribute__(self, name): if name in object.__getattribute__(self, '_enum'): return object.__getattribute__(self, '_enum')[name] else: # Default behaviour return object.__getattribute__(self, name)

def to_list(self): """Generate a list of tuples we use to quickly create <select> elements""" return [(val, string) for val, string in self._enum.iteritems()]

class Severity(MyEnum):

_enum = { 'low': 'Low', 'med': 'Medium', 'high': 'High', } [/sourcecode]

So the base MyEnum object gives up a method of accessing the values via an object like interface.

[sourcecode lang="python"] my_instance = SomeModel() my_instance.severity = Severity().low [/sourcecode]

Now this is cool because it helps us have nice refactorable code to reference these strings we're storing into the database for the column. Yep, I know that having lots of strings is more resource intensive on the database. In most of these low scale tables though I'd rather have the table easy to read. Not only that, when I get the values out they're actually the prettified version of the string. So output is much easier.

Also just to note, in order for the MyEnum.__getattribute__ stuff to work you have to have an instance of the object. So that's why you have to enter Severity().high because Severity.high won't hit __getattribute__ and you'll get that there's no attribute by the name 'high'.

I also added a nice to_list() method because we use the webhelpers package to generate html elements and it'll accept a list of value tuples. I can just shoot Severity().to_list() to the webhelper for generation now

This was a good start and works well, but the one other thing I get nervous about is making sure the database doesn't take in garbage, it should be able to make sure that the option I'm setting is in fact in the Severity list of options. In order to do this I added a custom SqlAlchemy column type using the TypeDecorator. In doing this I basically have access to when the value is pulled from the database and placed upon the instance of the model, as well as when the value gets put onto the model from the code side. A database getter/setter in practice.

[sourcecode lang="python"] class MyEnumType(types.TypeDecorator): """SqlAlchemy Custom Column Type MyEnumType

This works in partnership with a MyEnum object

"""

# so this is basically usable anywhere I have a column type of Unicode impl = types.Unicode

def __init__(self, enum_obj, *args, **kwargs): """Init method

:param enum_obj: The enum class that this column should be limited to

e.g. severity = Column(MyEnumType(Severity, 255)) for a unicode column that has a length allowance of 255 characters

""" types.TypeDecorator.__init__(self, *args, **kwargs) self.enum_obj = enum_obj

def process_bind_param(self, value, dialect): """When setting the value make sure it's valid for our MyEnum

# allow setting to None for empty value if value is None or value in self.enum_obj._enum.itervalues(): return value else: assert False, "%s -- %s" % (value, self._enum)

def process_result_value(self, value, dialect): """Going out to the database just return our clean value""" return value

# brief usage example class SupportTicket(meta.Base): __tablename__ == 'tickets'

severity = Column(MyEnumType(Severity, 255)) [/sourcecode]

So now I have things tied into my database model as well. Hopefully this holds up as a valid method for doing this. It seems like it'd be a common use case. What I love is that my code never has to get down to specifying strings. If I want to limit a column or check a value I can use somewhat cleaner (imho that is)

[sourcecode lang="python"] if some_ticket.severity == Severity().low: ... do something

# or in a query results = SupportTicket.query.\ filter(SupportTicket.severity == Severity().high).\ all()

[/sourcecode]

Anyway, food for though. let me know if you've got an alternative I'm missing that I should be using.

Working on OSS @ Work: Dozer profiling Pylons

It's been a fun little bit working on helping speed up a Pylons app at work. Performance needed improvement, and while we knew a couple of big places, I also wanted to look at getting some profiling in place. Fortunately I recently ran across Marius's post on profiling Pylons applications with Dozer. Now Dozer was originally meant for viewing app logs and memory checking, but it seemed that the in dev work added some profiling ability. So away I went checking out his code and seeing if I could get it to run. Once I realized that you had to setup the middleware for each type of activity you wanted to perform (cProfile, memory monitor, log view) I got things running. Very cool!

Right off the bat I realized I might be able to help a bit. The links were dark blue on black, there seemed to be a some issues with the ui on the profiling detail screen. Since we were using this for work I took it a chance to help improve things upstream. I moved the css images to get the arrow indicators working right, cycled the background rows vs hard coding them, and did some small ui tweaks I liked. I also coded up a set of "show/hide all" links to quickly expand things out and back.

Of course, it's not all roses. The show all pretty much times out in anything but Chrome. There's still more ui bits I think that could be improved.

Now that I had tinkered though it was time to add in some features we could use. First up, log view for json requests. We have some timer code in our SqlAlchemy logs and so I want to be able to view those in browser, but also for json requests. So I tweaked it up so that on json requests it adds a _dozer_logview object to the response. This then shows the various log entries along with the time diff that the html view shows.

Once that was going we decided this would be great to put on a staging version of our web application that the client has access to. The nice thing is that the staging is as close to production as possible, so some profiling information there would be very helpful. We don't want the html logview information visible to the client though. It detracts from the ui. To help with this I added a ipfilter config option for the LogView feature. In this way we can limit it to a couple of testing dev boxes and not worry about the client getting access.

I've pushed the code up into a bitbucket fork for Marius's repository. Hopefully this is useful and it's awesome that I got to spend some work time working on code that I can hopefully give back. I love this stuff.

Ubuntu: Help Wanted

So Jono wants to know how we can get more developers interested in Ubuntu. We got talking about it in the #ubuntu-us-mi loco and it sparked an interesting debate. Just who are you looking for? Jono mentions "people interested in getting involved in packaging, fixing bugs, and joining our community", but that's not just developers. I mean, our guy that started the Ubuntu Bug Jams doesn't do much development for Ubuntu. Our debate also had us wondering if packaging was really a developer task or if it was better left to the non-developers so as to free up valuable developer time to add more features and fix more bugs. Personally, I took the idea of the Bug Jams and decided to start my own Packaging Jam. I wanted to to learn to be a MOTU and thought it would be great to bring in someone who knew the ropes. For me it's hard to go through the documentation and figure out how things work. Especially since I was a PHP developer. What do I know of build systems? I just deploy from source code via scripts.

I never ended up reaching that goal of becoming a MOTU. It was just too hard to get through all the docs and learn all the rules. There are so many different tools. People think Git is a series of disjointed commands? Look at the list of things in bold here.

It's insane. I've moved on now and I do Python programming and I've had to learn how to build packages for Python tools, but it seems a bit easier using tools that you already use for Python. I haven't gone the extra mile to package things up into .deb files. I think that the popularity of scripting languages with their own build/distribution tools have taken a big hit on your pool of potential packagers and such.

Well anyway, back to Jono's request. I think there are a few things. First, narrow down your post. What are you looking for, more help packaging existing app? Help doing bugs? Help developing awesome apps? Each are a bit different. Based on our discussion I'm going to run with packagers.

The two things I think that would help potential packagers getting going would be

  1. A more formal filled mentoring program. I think if existing packagers brought in trainees to help watch and build the packages during the alpha stage of a release, over time they'd pick things up. People need to learn best practices, how the tools fit together, and it's best to pick this up through practical use under a guiding eye. The one trick is I wonder is how it would be best for two parties to share a common build/package environment. Developers can share an editor via various online tools, I'm not sure how to share a pbuilder instance.
  2. I once mentioned how I think one could build a practical web application to walk someone through packaging step by step. Imagine that you get walked through getting source, making changes to the changelog, and rebuilding the package. It could even be tied into the ppa system so that your 'project' can be built as part of walking through the various steps. It's a lot of work, but I could see it as something that has different modules and as you work through them you learn a new trick with each.

Lococast.net back, now with screencast goodness

A quick Lococast.net update. After PyOhio ate up one episode and then my illness ate up another, Craig and I finally got back behind the mics for episode 4. Thanks for everyone that checked in on us and sorry for the delay.

Along with that I managed to release something new. A screencast! That's right, I'm getting tired of repeatedly trying to show the same tricks and decided if I screencast the stuff I can just point people at the videos. Video is a whole new ballgame to me, and I'm cheating by using this script from the great Ubuntu Screencast community.

My first episode is naturally one on Vim and the awesomeness of splits for very day use. I'm starting planning on my next Vim one now. Make sure to check it out. The videos are available from blip.tv (with better audio) and Youtube.

Let me know if there's anything in particular you'd like to see covered and make sure you subscribe to the Lococast.net RSS feed to keep up on updates.

p.s. go check out my talk and the other great ones from PyOhio at: http://python.mirocommunity.org/category/pyohio-2010

Insert Giant PyOhio 2010 Recap Here

Ugggg, zombie day today as I attempt to get work done after a crazy PyOhio this year. Just like the last two this was a great regional conference I'm so glad that people put on. So congrats to the team for a 3rd year and some great stuff. The nice thing this year is that I managed to finally reach a goal of mine to get a job doing Python. After being a wanna-be the last two years, I got a chance to give back a bit more than I took for the first time. So below is a brain dump of my view of the conference. I want to thank all the awesome people that came to the open spaces, provided great insight, and to the speakers I attended for showing me cool stuff I've yet to see. If you've got any questions or corrections from my dump please let me know. And if you were the one giving the talk, or want to add let me know and I'll toss an link/edit in here going forward.

The full schedule is up at: http://www.pyohio.org/2010/Schedule

Day 1

Python Koans - Tutorial

I started out with the Koans tutorial track. You can get the koans from: http://bitbucket.org/mcrute/python_koans/src/tip/python%202_6/

The idea is that you get the source code which is setup as a series of unit tests. Each test is broken and you need to fix it in order to understand some aspect of Python. They've got a ton of work in here and it's a really fun way to learn about Python.

For instance there's a whole test file for tuples. You have to go in and correct the unit tests so they pass and as you correct them you learn how tuples work and some of the ways you can use them.

You can see all the koans in their directory: http://bitbucket.org/mcrute/python_koans/src/tip/python%202_6/koans/

So one example:

[sourcecode lang="python"] def test_tuples_are_immutable_so_item_assignment_is_not_possible(self): count_of_three = (1, 2, 5) try: count_of_three[2] = "three" except TypeError as ex: self.assertMatch(__, ex[0]) [/sourcecode]

Here it's demonstrating that tuples are immutable. You'll get an exception if you try to edit the tuple in place like this. Your job is to replace the '__' with the name of the exception you'll end up getting.

The koans are small and you can easily test these in a python terminal to help you figure things out. Lots of fun.

Log Analysis with Python

This turned out to not be what I was looking for. It seemed like a chance for the writer of this petit tool to show off his tool. I was hoping for more about how to analyze, generate trends, etc from log files. More on the math/methods. His library seems cool and all, but unless a lot of regular time going over log files I'm not sure how handy it would be.

http://crunchtools.com/software/petit/

Open Space - Message Queues

I attended an open space on Message Queues. There were only a couple of people there and we did some chatting, one guy was a heavy ZeroMQ user, but wasn't that happy and was looking at rabbitmq. So not a lot of new notes, but seems the rabbitmq route is the most popular.

Wrangling the bits, standardizing how apps get built - My Talk

I gave my talk at the end of the day. It was about my work's venv.py setup we use and how it enables us to easily create new apps/projects in a hurry in a standard way so that we all know what to expect. It seemed to go over really well. I had a number of people comment after the talk how they thought it was really good.  It's basically notes on how to customize virtualenv.py with your own actions/features making life a bit easier.
I've put my talk slides up in case you're interested, but there's not a lot of meat to them unfortunately:

Sprints

That night we met up at Subway to do some sprints on projects. We didn't get a whole lot done. I did start to checkout the source code for Pypi, but didn't do much with it.

On a side note, all hail the awesome subway lady that got serving us all. I think we might have been some 25 people strong showing up at once and hammered here running the store all on her own. She didn't complain once and was just pure awesome.

Day 2

Vim open space

The talks were so-so and I decided to instead lead an open space on vim and using it for python work.  There were a bunch of people in there and it went over really well. Everyone learned a few new tricks in there. One thing I've updated is to use the plugin pylint.vim so that I can get in-line syntax error checking.

Some of the plugins brought up:

And another link to my vim config: http://github.com/mitechie/pyvim

Controlling Unix Processes with Supervisord

This was a pretty good talk. There were several things I didn't realize that supervisor could do such as writing your own custom event handlers. For instance they have a custom thing that checks for some message that causes them to spin up more web servers to handle an increased server load.

Some of the tricks like tying all your services together under supervisor was pretty neat. For instance they tie in their mysql servers, web app servers, solr full text servers into one supervisor control setup. So they all startup/work together.

Some of the things they do are because they can provide a user control interface for clients. They create something like a cPanel-like interface for users on their network which allows them to restart/stop servers at will.

Code with Style

This talk basically a walk through Pep8. I try hard to keep to Pep8 and use a vim plugin to check my files as I work on them. The two places I have a hard time with it is the line length when you have long single line messages and with complicated list comprehensions.

He demonstrated a couple of tricks:

  • for list comprehensions that are complicated you can turn it into a function that you then run across your list/etc via the filter() function.
  • for long strings you can avoid the \ at the end of line by using ()

[sourcecode lang="python"] s = ("my one long" " line of text" " is this thing here") [/sourcecode]

will come out as: "my one long line of text is this thing here"

So now I have some work to do to clean up some things I've left behind.

Fabric - Open Space

I wasn't into the next set of talks so I again hosted an open space on fabric.  This one got interesting as many people in the room handle large server farms of hundreds of machines and have very separate ops/dev people. So a developer is separated from some of the things I was showing in fabric like stopping/restarting celery, resetting the web app, etc. There was also some question as to the advantage of using something like Fabric over just bash/shell scripts.

So it was interesting to see where fabric fits in, where it falls short, and while it's cool in some aspects, there are a number of limitations and concerns in several use cases.

The one regret was that with the discussion going on we never got other people up front showing off their fabric setups which I would have liked to see. I guess we had a hard time keeping the discussion going in line and I'd have liked to see more stuff from other fabric fans.

Making it go faster

This was the last talk I went to. It started out looking at how to profile your code using cProfile and pstats. I had messed with this, but then he covered using runsnakerun in order to do the profiling which gives you pretty graphs that represent the time spend in functions as a series of shells like this:

http://www.vrplumber.com/programming/runsnakerun/screenshot-2.0.png

You can see the site there:

http://www.vrplumber.com/programming/runsnakerun/

So it's a decent wrapper around pstats for some more visual/easy to view performance stats. I'll have to try that out in the future.