An updated email config, 2 offlineimap, mutt, and dovecot ftw!

Since joining the Launchpad team my email has been flooded. I've always been pretty careful to keep my email clean and I've been a bit overwhelmed with all the new mailing lists. There are a bunch of people working on things, as you can imagine. So the email never stops. I'm still working on figuring out what I need to know, what I can ignore, and what should be filed away for later.

Another thing I'm finding is that I've got emails in both of my accounts around a single topic. For instance, I have to do some traveling. I've got emails on both my Gmail (personal) and Canonical (work) accounts that I really want to keep together in a single travel bucket.

I currently have offlineimap pull both of work and personal accounts down into a single folder on my machine ~/.email/. So I've got a ~/.email/work and a ~/.email/personal. I use mutt then to open the root there and to work through email. It works pretty well. Since I really wanted a global "travel" folder, I figured I'd just created one. So that works. I end up with a directory structure like:

  • personal
  • travel
  • work

The problem

Of course the issue here is that when offlineimap runs again it sees the email is no longer in the personal or work accounts and removes them from the server. And the travel folder isn't a part of any server side account so it's not backed up or synced anywhere. This means Gmail no longer sees things, my phone no longer sees them, and I've got no backups. Oops!

Solution start

So to fix that, my new directory structure needs to become an account. So I setup dovecot on my colo server. This way I could have an imap account that I could do whatever with. To get my email into there, I setup offlineimap on my colo to pull personal and work down as I had on my laptop. So I still have things in a ~/.email that's from the accounts and then dovecot is keeping all of my email in ~/email (not a hidden dir). To get my email into there, I symlinked the ~/.email/personal/INBOX to ~/email/personal and did the same with the work account. Now the two accounts are just extra folders in my dovecot setup.

So there we go, colo is pulling my email, and I changed my laptop to offlineimap sync with the new dovecot server. In this way, I've got a single combined email account on my laptop using mutt. I then also setup my phone with an imap client to talk directly to the dovecot server. Sweet, this is getting closer to what I really want.

Issues start, who am I

Of course, once this started working I realized I had to find a way to make sure I sent email as the right person. I'd previously just told mutt if I was in the personal account to use that address and if in the work account use that one. Fortunately, we can help make mutt a bit more intelligent about things.

First, we want to have mutt check the To/CC headers to determine who this email was to, if it was me, then use that address as a From during replies.

mutt config:

# I have to set these defaults because when you first startup mutt
# it's not running folder hooks. It just starts in a folder
set from="rharding@mitechie.com"
# Reply with the address used in the TO/CC header
set reverse_name=yes
alternates "rick.stuff@canonical.com|deuce868@gmail.com"

This is a start, but it fails when sending new email. It's not sure who I should be still. So I want a way to manually switch who the active From use is. These macros give me the ability to swap using the keybindings Alt-1 and Alt-2.

mutt config:

macro index,pager \e1 ":set from=rharding@mitechie.com\n:set status_format=\"-%r-rharding@mitechie.com: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b?%?l? %l?]---(%s/%S)-%>-(%P)---\"\n" "Switch to rharding@mitechie.com"
macro index,pager \e2 ":set from=rick.stuff@canonical.com\n:set status_format=\"-%r-rick.stuff@canonical.com: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b?%?l? %l?]---(%s/%S)-%>-(%P)---\"\n" "Switch to rick.stuff@canonical.com"

That's kind of cool, and it shows in the top of my window who I am set to. Hmm, even that fails if I've started an email and want to switch who I am on the fly. There is a way to change that though, so another macro to the rescue, this time for the compose ui in mutt.

mutt config:

macro compose \e1 "<esc>f ^U Rick Harding <rharding@mitechie.com>\n"
macro compose \e2 "<esc>f ^U Rick Harding <rick.stuff@canonical.com>\n"

There, now even if I'm in the middle of creating an email I can switch who it's sent as. It's not perfect, and I know I'll screw up at some point, but hopefully this is close enough.

Firming up with folder hooks

Finally, if I know the folder I'm in is ONLY for one account or the other, I can use folder hooks to fix that up for me.

mutt config:

folder-hook +personal.* set from="rharding@mitechie.com"
folder-hook +personal.* set signature=$HOME/.mutt/signature-mitechie
folder-hook +personal.* set query_command='"goobook query \'%s\'"'

So there, if I'm in my personal account, set the from, the signature, and change mutt to complete my addresses from goobook instead of the ldap completion I use for work addresses.

Not all roses

There are still a few issues. I lose webmail. After all, mail goes into my Gmail Inbox and then from there into various folders of my dovecot server. Honestly though, I don't think this will be an issue. I tend to use my phone more and more for email management so as long as that works, I can get at things.

I also lose Gmail search for a large portion of my email. Again, it's not killer. On my laptop I've been using notmuch (Xapian backed) for fulltext search and it's been doing a pretty good job for me. However, I can't run that on my phone. So searching for mail on there is going to get harder. Hopefully having a decent folder structure will help though.

I've also noticed that the K-9 mail client is a bit flaky with syncing changes up on things. Gmail, mutt, and I've also setup Thunderbird all seem to sync up ok without issue, so I think this is K-9 specific.

That brings up the issue of creating new folders. Offlineimap won't pick up new folders I create from within mutt. It won't push those up as new imap folders for some reason. I have to first create them using thunderbird, which sets up the folder server side for me. Then everything works ok. It's a PITA, but hopefully I can find a better way to do this. Maybe even a Python script to hook into a mutt macro or something.

Wrap Up

So there we are. Next up is to setup imapfilter to help me pre-filter the email as it comes in. Now that all email is in one place that should be nice and easy. I can run that on my colo server and it'll be quick.

This is obviously more trouble than most people want to go through to setup email, but hey, maybe someone will find this interesting or have some of their own ideas to share.

CoffeeHouseCoders 11/23/11: YUI Theater group viewing

Just a heads up, this week's CoffeeHouseCoders (CHC) Detroit-ish will be a bit different. One of the goals of moving the location to the new Caribou was that we get access to the meeting room. This opens up the opportunity for us to have some group discussion and such around various topics. We're going to give that a shot this week with a group viewing of YUI Theater video viewings and JavaScript discussion.

Most of us do at least some JavaScript in our work and projects so I think it's relevant and should be fun to geek out before the holidays start up. I'll have a little projector and speaker and with the new videos from YUIConf 2011 going up, it'll be nice to set aside some time to catch up on some of the recorded presentations. Take a peek and set aside one or two "must watch" videos for Wed night! Not all of the videos are YUI specific, so it should be useful for all of us doing JavaScript.

Launchpad Team: Day 1 complete

Phew, well one day down. I dove head first into Canonical and Launchpad today. It's a bit amazing the amount of information and parts there are to everything. Everyone welcoming me throughout the day was great, but my head is still spinning a bit for sure.

I managed to get a nice starter walk-through of Launchpad and find my way through a superficial bugfix and merge request. So hey, that wasn't so bad heh. It's kind of exciting to throw out all my usual tools I've been mastering for a while and start over. Make files, zpt files, ZCA, and YUI run the show. Time to see how people get things done without Fabric, Mako, and SqlAlchemy.

I'm really excited to get to some real change and hope to pick things up quickly. I know a while ago I was disappointed that Launchpad wasn't taking advantage of some of the Javascript driven UI enhancements that we can do these days. The change of that is already in full swing and my team is looking to land a nice chunk on the bugs UI shortly. Let's get to work!

Mutli-monitor on the go, DisplayLink usb monitor

Now that I'm going to be working for Canonical I've got to get ready for some week long sprints. Now, I LOVE my 12" X201 thinkpad. It's the best laptop I've ever owned, and I've owned macs, toshibas, dells, and larger thinkpads. When I'm home, I dock and use dual 21" displays. When I travel, it's the perfect size, yet still packs an i5 and 8GB of ram.

However, a week of living inside a 12" display has me a little claustrophobic. So I decided I should do something about it. They make some decent looking USB powered external monitors that seemed like they'd travel pretty nice. Once cable, no external power needs, and another 1024px is a good thing. Mobile mutli-head..sweet!

Getting it working

The trouble is getting it working. The USB monitors use something called DisplayLink to work. Getting that to work is a little tricky. Fortunately, a few brave souls paved my way. You can see the links I checked out to get started

Once you install xserver-xorg-video-displaylink you'll need an xorg.conf file since DisplayLink doesn't work with all the hotplug business that makes modern video work.

It's alive!

The problems

The real trouble is that I ONLY want this xorg.conf file when I'm actually using the USB display. When I'm docked at home or using a projector, I don't need it. Right now I've just written a toggle script that I use to flip the xorg back and forth.

There are also some usage issues. It only works for me if the DisplayLink monitor is the primary one in the xorg server setup. This means the LightDM is running on there and it's running in just a few hundred pixels of the display. I can still log in though so it's not killer.

You can't drag windows back and forth among them, which isn't the end of the world, but some apps (like Google Chrome) only launch on the same display as the currently running instances. So I can't find a way to get Chrome to run on both monitors. For now I just run Firefox on one and Chrome on the other.

Overall, I'm really digging the setup. I'm sitting at the bar with irc on a second display while I write this blog post out all unplugged from any power. I think it'll make life a LOT nicer for road traveling for any extended time.

[sourcecode lang="bash"] #!/bin/sh -e # toggle xorg.conf on/off so I can add/remove it as needed # requires a reboot after running to take effect

if [ -e /etc/X11/xorg.conf ] then echo "Removing xorg.conf file" sudo mv /etc/X11/xorg.conf /etc/X11/xorg.conf.usb else echo "Setting xorg.conf file" sudo mv /etc/X11/xorg.conf.usb /etc/X11/xorg.conf fi [/sourcecode]

My xorg.conf file I'm using [sourcecode] ############ Original Video Settings ###########

Section "Files" ModulePath "/usr/lib/xorg/modules" ModulePath "/usr/lib/xorg/modules/drivers" EndSection

##################################################

Section "Device" Identifier "Configured Video Device" Driver "intel" EndSection

Section "Monitor" Identifier "Configured Monitor" EndSection

Section "Screen" Identifier "Default Screen" Monitor "Configured Monitor" Device "Configured Video Device" SubSection "Display" Depth 24 Modes "1280x800" EndSubSection EndSection

############### DisplayLink Stuff ###############

Section "Device" Identifier "DisplayLinkDevice" driver "displaylink" Option "fbdev" "/dev/fb0" EndSection

Section "Monitor" Identifier "DisplayLinkMonitor" EndSection

Section "Screen" Identifier "DisplayLinkScreen" Device "DisplayLinkDevice" Monitor "DisplayLinkMonitor" SubSection "Display" Depth 16 Modes "1024x600" EndSubSection EndSection

#################################################

Section "ServerLayout" Identifier "Server Layout" Screen 0 "DisplayLinkScreen" 0 0 Screen 1 "Default Screen" RightOf "DisplayLinkScreen" Option "Xinerama" "Off" EndSection [/sourcecode]

Visual of the display issue with LightDM

Been a good summer, fitness, woodworking, and new job coming soon...

I'm very late on a bunch of weekly status reports, but I've got good reason I promise. This summer has been a bit of a "summer of change" for me. Perhaps more renovation than change.

First, I decided I was sick of looking and feeling like crap and got into a weight loss program. I've finished that program today with some really good results. I'm down 37lbs, with another 19 to go until I hit my goal. Here's hoping I can get there before the year is out. This has really helped me feel a ton better. I did a lot of biking this summer, including a 48mile monster that beat me up, but man was that fun.

Next up, I finally managed to clean the garage out. That means the woodworking tools are open for business. I've only started working on putting some finish on some new oak closet doors, but that's a huge move forward. I've not done any woodworking since the boy was born and now that the space is cleared out I'm looking forward to getting back to making some shavings.

Finally, the big news. I've put in my two weeks notice with Morpace and have accepted an offer to work for the Launchpad team with Canonical.

I can't express how excited I am to get this chance to work with a team of smart people that are really working hard to build something awesome. In many ways this is the culmination of a goal I set years ago to get paid to work on open source software, and not only that, but to get paid to work with Python as well.

Morpace was really good to me and I feel sad to leave them. They gave me my first chance to prove that I could be a productive Python developer. I've grown though and the opportunity to work with all of the great people at Canonical is just too much to pass up. I can't wait to jump in and see what I can help with.

It's been a good summer. I can only hope that each year continues to be as fulfilling.

Bookie weekly status report: Sept 18 2011

Updates this week

Slow week. The changes to the fulltext search involved a small bug in updating the tag_str property of the bookmark. I got this fixed and added a migration to rebuilt the tag_str for all bookmarks in the system. The big sign of this was that the edit ui wasn't showing the tags on the bookmark for edit.

Part of fixing that was a full update to SqlAlchemy 0.7 and moving all post/pre hooks into the new events system SqlAlchemy provides.

Along with that I also updated things to use the new release of Pyramid 1.2. This big thing with this is the new debug toolbar is available for development. It's definitely pretty cool and helps provide some timing/insight as testing requests.

What's next?

There's a ton to work on. The celery task runner, the backbone.js in place edit ui, and investigating using tagger to provide some tag suggestions according to the content.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie weekly status report: Sept 7 2011

Updates this week

Sorry, I'm late on the weekly report. The holiday weekend threw me off. However, I've got good stuff for you. I completed the port of the fulltext search from being database specific to being Whoosh which is pure Python. This should be cool because it reduces code complexity trying to change things up based on the database used. It also been performing really well. The hosted http://bmark.us site is using it for everyone's content.

Be aware, if you do the migration, it takes a while to run. It has to go through and processes every readable record one by one. So when you upgrade, let it run.

What's next

Aside from that, I started playing with an all Javascript front end using Backbone.js. The idea is to see about setting up a very interactive front end that would allow in place edits of bookmark data easily using the existing API. We'll see how it works out.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 28th 2011

Updates this week

With 0.3 out and gone, it's time to move forward on 0.4 tickets. I started out with the big one, edit/add form ui for bookmarks on the page. This now works and when you're browsing urls from your page (/{username}/recent) you'll see a new "E" icon next to the "R" one. You'll also see a "Add" button at the top left.

Once we had the add/edit form, I could setup a bookmarklet to store your current page there. If you log into the site and go to your account page, a bookmarklet url is there for the taking. Drag it to your browser bookmark bar. On some platforms (hey Android!) you can't save a bookmarklet to the bookmarks. You first have to create a normal bookmark, and then edit the link after the fact. To help that out, there's a copy/pastable version of the bookmarklet on the account page. Just long press the text area, select all, and then paste into your bookmark from earlier.

It's my hope that this bookmarklet will help users on Firefox and other browsers use Bookie more. It's not the full extension experience, but hopefully holds people over for now. Note that when storing bookmarks in this way, the content is not immediately available as from the Chrome extension. The cron job that does fulltext parsing runs at 4am each morning.

Next I started some css work. The readable page no longer uses the Cabin font. It's also got some default styles for H1 and such. Let me know if any content looks funny and we can look to add some additional styles going forward.

Next up

For the week ahead, I'll be mainly working on tests. I don't have enough for the new add/edit feature. I'm still working on adding live api tests to the phantom.js test suite. If I get time, I also hope to start work on a private bookmark feature.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 22 2011 (0.3 yay!)

Updates this week: 0.3 out the door

We had to do some hacking of dying limbs to get there, but 0.3 is out the door. This was the biggest release cycle since things started up. However, the features of an API and multi user auth were worth it. There were a ton of small tweaks long the way as well. Here's hoping that 0.4 is a bit faster in the moving and releasing

You'll notice that we're backtracking on having a Firefox extension and a mobile site. Both are horribly broken at the moment. I really want to get this going though and they will be specific todo items in up coming releases. Well, except the Firefox extension part. Honestly, every time I try to get started with the Firefox extension I feel like the team there is just damn hostile to developers. It's beyond ridiculous how unable to get a handle on things there it's been. Confusing docs all over the place, an add on sdk that tries to run a virtualenv, but you don't built it from any of your local python's so your fubar if you have the wrong one, the fact that it's all getting redone again in JS and not Python...Sorry. I'm just not willing to be a fulltime firefox dev to make the dippy thing work.

Links

Next up

  • Work on more of the live api tests. It's running via the latest build of phantom.js and working well
  • Start work on the 0.4 todo list
  • Especially the save/edit ui and a bookmarklet for those.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

CoffeeHouseCoders Detroit hiring a moving truck

Coffee House Coders is moving

Coffee House Coders has been meeting every Wed night for something near two years now. I've not been able to find the announcement for the first one. However, starting August 24th (note, that's a week away, not in two days) we'll make our first location move.

Why? Well, as those of you that show up know, we don't always have all the space that we need. Sometimes we get the place to ourselves, sometimes it's crazy and hard to get seats for everyone. The new location has a private meeting room. So we can be sure we have a set of space available.

Hopefully, this will help attract some of the irregular visitors and give us room to grow. I've always wanted to try to do some more things with CHC. It's great to have that set time each week, but I get the feeling that it would be cool to actually do some mini-user group content. We don't have an east side Python user group, but we have a ton of people interested in that content. We don't have an east side Web Developers group, but again, we have potential interest. Using this meeting location, we might be able to expand and actually do some user group like material in the future.

What is this Coffee do dad?

The idea of CHC is easy, we all want to code/geek out on fun stuff. Every Wed night, from 8pm until 10pm we meet at the coffee shop and sometimes write code and sometimes just BS the time away with fellow geeks. We find it's great to have some time set aside each week for personal projects, learning something new, or whining about what that co-worker did to your code while you weren't looking.

When again?

Our first week in the new location is August 24th. We'll be in the new location from then on as long as things work out. As before, we meet 8pm - 10pm. The last Wed of each month we add an extra hour so get there at 7pm. The 31st will be our next "long edition" CHC and that will be at the new location.

Where is this?

Caribou Coffee
31901 Woodward Avenue
Royal Oak, MI 48073-0984

[googlemaps http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=Caribou+Coffee,+Woodward+Avenue,+Royal+Oak,+MI&aq=0&sll=42.677261,-83.160496&sspn=0.113078,0.099049&vpsrc=0&ie=UTF8&hq=Caribou+Coffee,+Woodward+Avenue,+Royal+Oak,&hnear=Michigan&cid=8777002513835499183&z=13&iwloc=A&output=embed&w=425&h=350" /]

View Larger Map

If you have any questions or have any ideas on topics/wishlist for a local programmers group in the area, feel free to let me know.

Bookie status report: Aug 14th 2011

Updates this week

Finally, the new and fancy Bookie API is reworked and better than ever. The live running site is updated to use the new api and there's a new chrome extension that's update for it as well. Make sure you get the latest dev version of the Chrome extension.

With the api updated, it's time to get to work on finishing up 0.3 for release. This means that we need to finish updating the mobile site and some rework of how the readable content is stored. Since we pull the content out of the page based on actual page you see, we should store these separated from user to user. We don't want to leak logged in info from one user to another.

Farewell Firefox

With the api update, the Firefox is so far out of date as to be unusable. There's some work going on to help catch it up, but I think I'm going to remove it as a feature in the docs and such. We just can't claim it at this time.

New testing

With the new api we hit some testing issues. Since there's no good way I've found to functional test the extension, the tests aren't great and reliable. I also don't have tests setup on some shared code like the bookie.api.js code. I'm thinking of setting up a running instance with some live data with the sole purpose of running tests on the JS code against it. We might even go so far as to setup the extension html files as an actual website and write some functional tests around that. After all, I can load popup.html and do some testing of that as a normal web page. We'll be missing the interactions with background.html and such, but might be worth some attempt.

Any ideas and help with this is most welcome.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.11
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 7th 2011

See, I can do these things once a week, I promise!

Updates this week

Progress this week is all in the api branch. It's taken some time to figure out just what things are going to look like, and I started with the docs. Lots and lots of docs updates for the api. You can check out how things look so far.

  • Gone is the custom json response object. We now send back an http status code and possibly a json message
  • The auth code is all updated. We now use a new api_auth decorator in lib/access.py that will check the api call and take care of the 403 response if things fail.
  • Most of the tests are commented out while we enable/fix them one by one as the new api methods get filled in. Once we're sure thing are where we want, we'll revisit the rest of the apis. Since parts of the functioning site use the api it really needed to be done.
  • The first few new api methods are working correctly and tests are passing in the api branch. This is mostly the /{username}/bmark/XXX urls. So adding, updating, deleting, and getting a bookmark for a user is set.

To make some of the new request changes we needed to update to Pyramid 1.1. This gave us things like request.response.status_int and some changes to the way the http exceptions work.

The goal for the next week is to keep plugging at the api. If you've got any feedback let me know. I'm really hoping to do a really solid job on this updated api and it will help with the extensions, feature updates, and encouraging new developers to get involved.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

Bookie Status Report: Aug 1st 2011

Phew, with PyOhio out of the way and my tutorial done, I can get back to Bookie. Sorry that I missed the last couple of weekly status reports. Honestly, there wasn't a ton, but there was some work going on.

So first, let's go over what's been going on.

Mobile View

I put in a bunch of work redoing the mobile view. It's still using jQuery Mobile, but I redid the javascript that ran it, I implemented an HTML5 history api to doing the tracking as you go from "page" to "page". I put that in quotes because pages are driven by just json api calls. There's no real new html page load after it gets started.

Of course, it's not totally released yet. There's a bug using it with Android. It works in Google Chrome and it seems to work on iOS. The most current version is live on the htp://bmark.us site.

Docs

There's been a lot of work to help prepare the docs for the 0.3 release coming. Craig did some great work helping going through parts of it and finding things I had missed.

API

Ok, here's the one that's going to hurt. I fleshed out most of the rest of the api to get the mobile site updated and such. There's still one view left out there, but the api is going to get reworked. Issac Kelly's talk at PyOhio reminded me that an api is all about resources, not just implementing what your app does via json. So I really need to rethink things. Of course, since the web site, the mobile site, and the extensions all use the api, there will be some work there to get everything updated. This means 0.3 is going to be delayed form where I wanted.

The docs are also pitiful when it comes to the api. Once things are updated I really want to get them really solid. A good api I think will be a big boost to Bookie going forward. So this is another heads up that 0.3 will be a bit delayed, but hopefully this will all pay off.

PyOhio Sprints

The sprints went awesome. While I was hoping to get a lot of coding myself, that was a poor expectation. I spent most of my time working on the mobile site updates and the documentation. We had some six or so people that managed to get it install. It really proved that there's some work to get for easing the installation process.

What was really cool was that once people got it installed, there was work getting done. Dave Brondsema added the urls to the main ui so you can see the url. He also added a branch flag for the bootstap.py script which should help enable bootstrapping both dev and stable versions easily.

I want to really thank everyone for taking part in the sprints and I really hope there are some people that find the project interesting and want to take part going forward.

Todo

Next up on the todo list:

  • redo the api and docs
  • fix this android mobile issue
  • keep working on the install docs. The bootstrap vs the long way was really confusing to users.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

PyOhio 2011: Another year, another great time.

Phew, another PyOhio has come and gone. This year was a great event. I can't say enough good things about the group that puts it together. It's really nice to have something somewhat local to head to every year.

Talks:

  • Data-Transfer Objects Are a Disease. Meet the cure.

    The first I went to on a whim, I wasn't really sure what kind of Data Transfer Object (DTO) was going to be discussed. It turns out, the talk was a praising of the NamedTuple as a great way to pass data in Python. It's got some nice sanity checks, is very lightweight, and helps prevent developers from going crazy adding all kinds of business logic to simple data containers. I'm not sure how they're passing them around their applications, but I can see the appeal. I know I've been bit a few times when unserializing some JSON and a typo in a dict key bites me. I wonder if there's a way to easily get back a list of NamedTuples vs dicts when loading up some JSON transferred around.

  • Aspen: A Next-generation Web Framework

    I checked out the Aspen web framework talk just to check out what new ideas people are playing with when it comes to web tools. I wasn't disappointed, Aspen has some very interesting ideas. The author has done some work to bring back the url meaning something on the project layout. The idea of your layers of the app in a single file is kind of interesting, and I can see how that'd be helpful in some development cases. I end up sitting my controller/template code side by side when I work anyway.

    The weakness I see, is that it's got the same issue PHP has when it comes to helping new developers start good practices. One of the great things about web frameworks, is that they help tell you where and how to organize your code. They give you test directories out of the box, they help bootstrap a good way to get your database connection up in a way that avoids shared sessions across requests, etc. Aspen does a lot less of that, and I could see a younger dev doing way more copy/paste of code than I'd want. It's a little bit of a bare framework, which is great to help integrate you tools of choice, but also provides a barrier to entry for new developers at times.

  • Django and Google App Engine: Why I'm using Flask and EC2

    This talk was from a friend on the west side of the state. As I've been part of the IRC discussions where he's been trying to go through various tools to build some small and simple web apps, it was cool to see the story in one swoop. He's a fan of the microframework. For his use cases, the full stack just threw hurdles in front of him. It's great to hear he's found a tool he loves in flask.

  • Evolving an internal web service

    Taavi gave a great talk on how they worked to rebuild an aging PHP app in Python over a long period of time. It's a great example of what I've been trying to get going for a while, APIs all the time. Everything seems to talk to the new application base via an array of remote methods and this great decoupling has been a boon for them to help provide data to several different systems in a clean way.

    I wish I had more time to try to chat with him. They're doing some cool things with SqlAlchemy, migrating data from old systems to new, and just some very good work and testing done on performance.

  • Creating web apis that are a joy to use

    This was something I really was interested in. Since I've been pushing apis like mad at work and I've been working on my first public one for Bookie, I knew I needed some help/guidance on this. Issac did a great job hitting home the big rules I kind of knew, but wasn't following that well.

    First, document, document, document. I loved his graph of user happiness vs amount of documentation. Users of your api don't get happy until the docs are near complete. Until then, it's just as bad as no documentation. I've got some typing ahead of me.

    The second point was something I was battling with a bit. I tend to think of the api in terms of usage. "You want to do task XXX". At that point, I'm deciding what the user wants to do. In reality it's more about the term "resouce". A resource can be data, a function (send email), or something else along those lines. However you want to expose them via the api in simple distint manner. Just taking your current html view you push to browser users and building an api that is the same doesn't work. After all, the great thing about the api is that people build and do things you didn't think to do on your current application implementation.

  • My Talk, Sqlalchemy Tutorial

    Finally, I was of course at my talk. This year I decided to really didn't like myself and I should do more than a talk, but a two hour tutorial with some hands on coding exercises. The room was full of people of all levels and was a bit more challenging than I originally thought it would be. On top of that, the AC broke and the rooms were over 85 degrees which made holding an audience's attention all that much more challenging.

    In the end, I think things came out ok. I've heard from an array of people that they enjoyed the talk. Once the first hour/talk part was over, most of the room left. We had about seven people that did the hands on code for the second hour. In the end, it sucks, but I can't blame them. If it wasn't my talk I'd have searched out cooler air as well. I hope that people still take the time to try out the hands on code and let me know if they run into any issues. If you do, feel free to email me rharding@mitechie.com.

    Thanks to PyOhio for letting me take a shot at something more classroom like. It's a new challenge to go from a talk to a tutorial and I encourage people to try it out.

Networking

I think it's true what they say, as you go to more and more conference you tend to do and learn less in the talks themselves, but make up for it with the great networking. This year it was very noticeable. What was great was that I got a chance to meet several people I've been following on Twitter for a while. These are people that are interesting, respected, and meeting them was kind of mini-starlet moments.

I got to have a great discussion around apis and self bootstrapping application installs with Issac Kelly. I met Michael Trier whos been a great Python presence on Twitter for a while now. I also caught up with the Ohio crew and guys Dave, Mike, and Catherine. If you run into these guys start a conversation, it'll be worth it.

I also had a ton of great conversations on things from new people that I really wish I did a better job of remembering and tracking down names for. Sorry that I don't call you all out.

All the hallway stuff really helped make this PyOhio a great one for me.

Sprints

Another great thing PyOhio does are the sprints. Unfortunately I could only do them on Saturday, but man what fun it was. I think we managed to get 6 or so people with their own up and running Bookie instance running. We had nearly a dozen people hacking on things at one point. We had some fun hammering pypi from the wifi network and some really good ideas came up to help make the installs a bit easier. At the end of the night we had a pull request and some definite interest going forward. I hope that the people that sprinted on Bookie found it interesting to take part in and maybe learned something. I know I've got a lot of work to do still

I'll have a separate Bookie status report out later with some details on changes and things.

A reminder as well, if you'd like to have a hosted Bookie account on bmark.us just sign up to the waiting list here: http://goo.gl/BBn2b

Bookie Status Report: July 6th 2011

A slow plodding week in Bookie progress this week. Thanks to Greg and Craig for some help in cleaning up a feature.

  • New version of the Chrome extension that will do a much better job of error checking on the options page. Make sure to get the updated extension.
  • The Account page has been cleaned up. You can now view your Api Key, change your password, and update your base account information such as name and email. I also did some ui cleanup so things should start to be a bit prettier. Let me know if you have any suggestions to the account ui.
  • I fixed the Google fonts used so things might look a bit different. I'm using the Cabin font for the body. I think I want to change it for the readable version body though. It's not great for large bodies of text. What do you think?
  • I also updated the mobile view to the jQuery Mobile beta release. I finally found the cause of a nasty bug and things should work ok. I still need to write my own history api wrapper. So navigating back and such is pretty broken still.
  • Per request of Greg and Craig, when the bookmark isn't for the current year we now add the '10 or whatever to the calendar graphic. You can see an example over here.

Next up on the todo list:

  • We need a password reset feature for those that have forgotten their passwords. There's some database side start to this, but no ui for it yet.
  • I want to start on a page in accounts for viewing your readable results such as displaying 404 counts and such. See ticket #47

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.4
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 29 2011

Shorter update list for Bookie this week. Took some vacation and not a ton of time hacking.

  • Fixed a bug in the readable/existing.py handling a non-asci url
  • Fixed all of the js/css to come via https so site should be happy lock for you
  • Added some app logging, log failed logins, imports, exports of bookmarks
  • Start the /{username}/account section with it's own sub nav and some css for that. The logout/export/import moved there

Most of this got pushed out in an incomplete form onto the live site. From here I need to add the edit form for the user's details and clean up the navigation to be reusable across the different pages.

I also want to keep adding some logging and stats counting so we can start to provide some data porn to both myself and users of their info.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 22nd 2011

Phew, it's been a good week with Bookie. First, we have some new ways to contact and follow Bookie. There's now a Google Group for better long form discussions and assistance. You have a feature idea, question, or just feedback? Then go ahead and send it to the list. Don't forget to follow BookieBmarks on Twitter. I want to start using these things to bounce ideas off of people as we move forward.

We managed to get user authentication working and added support across the site for routes for each user in the system. This was great because it means that http://bmark.us is a live running instance with a few different real users. So far we're running seven different users on the system. Part of that was getting all the routes updated, the queries, the templates to only show links for import/export if you're logged in, all of that stuff. The extension needed a little bit of updates, but the only big thing is changing your API URL to have your username in it.

We moved the documentation to http://docs.bmark.us and I tried to update the links in the docs as well. Much as Identi.ca is a running instance of Status.net, Bmark.us is a running instance of Bookie. I just can't get a cool url for Bookie such as http://bookie.net.

With the new site I purchased an SSL certificate. So you'll notice that the site is now all behind https. Part of the other sysadmin items were to setup twice daily pgdumps of the data which is sync'd off to S3, munin to monitor the server resources, including plugins for Nginx and Postgresql, and some Nginx config tweaks which I still need to document in the docs.

All of the code updates are available in the develop branch on Github. If you've got any questions feel free to hop into #bookie on irc and let me know.

With all this going on I'm looking for some early alpha testers. If you're interested in Bookie, but couldn't, or didn't want to, setup your own instance, let me know. I'd love to get another 6-10 people on this instance. The feedback of current users has already paid dividends. You can thank them for a pair of new features in the Chrome extension, including a new keyboard shortcut (ctrl-alt-d) and support for helping auto complete recently used tags.

Coming up, we need to work on moving the app forward by adding a user account page, creating add/edit abilities for authenticated users, and a bookmarklet that uses the add ability for mobile devices and other browsers.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 15th 2011

I just finished up reading Start Small, Stay Small and there were some good points in there. One is that writing about your progress on a project each week helps people move forward. There is something about putting down what you've accomplished and what you plan into the public that helps keep the motivation motor running.

In an effort to keep Bookie from stagnating, I think that's a good thing to start doing. Count this as the first of a series of weekly progress reports I'm going to be doing. I also like that it helps show, beyond links to commit logs, that Bookie is moving forward and getting updates.

This past week has been a bit crazy. There hasn't been a ton of time to put into things, but I've managed to move a few big things forward:

First, work on making Bookie work via user accounts and logins is moving forward. Basically all of the urls in the application needed to be updated. Currently there are two sets. If you leave out a username from the url, you get overall, full site info.

In this way, a url of /recent will pull the 50 most recent bookmarks from all users on the site. However, /rick/recent will only pull the 50 most recent bookmarks of the user rick. The API urls needed to be updated as well. There's a ton of work in getting this going, but it's a major step of progress to allowing me to host a version of Bookie that other users can sign up for. Since that's really the big goal that I've set myself by the end of the year, I'm feeling good on this one.

The idea of multiple users has me realizing that my little readability.py script that fetches url content from bookmarks and stores the clean, readble parsed html for that page needs some work. It'll never scale that way. So I've split the work into a couple of parts.

One part is a node.js script that will fetch a list of urls to go fetch and asyncronously goes out and fetches the html content. It then shoves the bookmark id and the content into a beanstalkd queue for processing. The queue is polled by a python script that then calls a new Bookie API call with the content and the id. Bookie then runs the parsing code against the content and stores it in the database. The async code on node.js can fetch the html content in a hurry. In testing with my SSD hard drive and sqlite, I'm able to pull, process, and store more than one url per second. This is with 1 node.js producer and two running instances of the python consumer.

I'm definitely looking forward to ramping this up on a real server with Postgresql running. I'd love to be able to pull down and parse content at some decent rates to be able to cope with new users signing up to the service.

So that's this week's report. Next up is more work on the multi user setup. The tag urls still need work and all of the unit tests that I had need to be updated to test the new urls. This also means some duplicate tests to check both with/without usernames in the urls. Work is never done!

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

First Bookie Sprint Good Friday

I've been cranking hard on getting Bookie going towards a 0.2 release. There's been some great work from contributers and things are really getting to be fun. To help give Bookie a boost, I'm holding a sprint at my place to hack on the 0.2 feature list. The big things we're working on include making the UI not suck 100% and getting the readable parsed content of pages integrated into the site really well.

So this Friday, 11am - 4pm is sprint time. If you want to head over let me know and I'll get your directions and such: rharding@mitechie.com.

If you can't make it, but you're interested in getting into it, jump on irc and join #bookie on freenode. Get an install of bookie going, test the importer and such. There're a few bitesite bugs in the ticket tracker that are easily fixed. So jump on in.

Introducing Bookie: only for developers

Introducing Bookie: only for developers

Delicious has been pretty stagnant over the years. I love using Instapaper and really kept waiting for Delicious to add that feature in to my pages I send into Delicious, but it never arrived. I also kept waiting for link checkers, a decent mobile view, and maybe even some support for full text search of actual web page content. Now that Yahoo has leaked out that Delicious is going to be "Sunset", I've finally gotten the push to start up my own bookmark storing application that I can add all the great features I want to have in there.

So without further ado, I give you bookie. It's built in Python using the Pyramid web framework. It's my first Pyramid app, but I do Pylons for the day job so it's not too far off.

This is the 0.1 release I'm calling "only for developers". It's not an easy one-click install, and right now it'll want you to have the -dev packages for the db engines to work right, but it does work. It can store, retrieve, and search bookmarks stored. It supports Delicious and Google bookmarks import, and will do an export. There's a fully functioning Google Chrome extension you can use to store and edit bookmarks.

This is currently a self hosting application. In time, I hope to provide a hosted version perhaps. I also hope to get a Firefox extension, as well as get started on the fancy features I noted above. So this is a LONG way from complete, but I feel like I've hit at least a 0.1 worthy milestone and want to put a stake in the ground. Here's the start of something, and hopefully it'll keep improving in future releases.

I want to thank my local guys for helping, testing, and adding great bits of code to bookie already. It's a lot of fun seeing other people get interested in things and contributing as well.

I'm trying to keep the todo list and some idea of the next milestones in the github wiki. Feel free to note bugs, add requests, etc. You can also chat with us in #bookie on freenode.