Bookie

Bookie Status Report: Aug 1st 2011

Phew, with PyOhio out of the way and my tutorial done, I can get back to Bookie. Sorry that I missed the last couple of weekly status reports. Honestly, there wasn't a ton, but there was some work going on.

So first, let's go over what's been going on.

Mobile View

I put in a bunch of work redoing the mobile view. It's still using jQuery Mobile, but I redid the javascript that ran it, I implemented an HTML5 history api to doing the tracking as you go from "page" to "page". I put that in quotes because pages are driven by just json api calls. There's no real new html page load after it gets started.

Of course, it's not totally released yet. There's a bug using it with Android. It works in Google Chrome and it seems to work on iOS. The most current version is live on the htp://bmark.us site.

Docs

There's been a lot of work to help prepare the docs for the 0.3 release coming. Craig did some great work helping going through parts of it and finding things I had missed.

API

Ok, here's the one that's going to hurt. I fleshed out most of the rest of the api to get the mobile site updated and such. There's still one view left out there, but the api is going to get reworked. Issac Kelly's talk at PyOhio reminded me that an api is all about resources, not just implementing what your app does via json. So I really need to rethink things. Of course, since the web site, the mobile site, and the extensions all use the api, there will be some work there to get everything updated. This means 0.3 is going to be delayed form where I wanted.

The docs are also pitiful when it comes to the api. Once things are updated I really want to get them really solid. A good api I think will be a big boost to Bookie going forward. So this is another heads up that 0.3 will be a bit delayed, but hopefully this will all pay off.

PyOhio Sprints

The sprints went awesome. While I was hoping to get a lot of coding myself, that was a poor expectation. I spent most of my time working on the mobile site updates and the documentation. We had some six or so people that managed to get it install. It really proved that there's some work to get for easing the installation process.

What was really cool was that once people got it installed, there was work getting done. Dave Brondsema added the urls to the main ui so you can see the url. He also added a branch flag for the bootstap.py script which should help enable bootstrapping both dev and stable versions easily.

I want to really thank everyone for taking part in the sprints and I really hope there are some people that find the project interesting and want to take part going forward.

Todo

Next up on the todo list:

  • redo the api and docs
  • fix this android mobile issue
  • keep working on the install docs. The bootstrap vs the long way was really confusing to users.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

PyOhio 2011: Another year, another great time.

Phew, another PyOhio has come and gone. This year was a great event. I can't say enough good things about the group that puts it together. It's really nice to have something somewhat local to head to every year.

Talks:

  • Data-Transfer Objects Are a Disease. Meet the cure.

    The first I went to on a whim, I wasn't really sure what kind of Data Transfer Object (DTO) was going to be discussed. It turns out, the talk was a praising of the NamedTuple as a great way to pass data in Python. It's got some nice sanity checks, is very lightweight, and helps prevent developers from going crazy adding all kinds of business logic to simple data containers. I'm not sure how they're passing them around their applications, but I can see the appeal. I know I've been bit a few times when unserializing some JSON and a typo in a dict key bites me. I wonder if there's a way to easily get back a list of NamedTuples vs dicts when loading up some JSON transferred around.

  • Aspen: A Next-generation Web Framework

    I checked out the Aspen web framework talk just to check out what new ideas people are playing with when it comes to web tools. I wasn't disappointed, Aspen has some very interesting ideas. The author has done some work to bring back the url meaning something on the project layout. The idea of your layers of the app in a single file is kind of interesting, and I can see how that'd be helpful in some development cases. I end up sitting my controller/template code side by side when I work anyway.

    The weakness I see, is that it's got the same issue PHP has when it comes to helping new developers start good practices. One of the great things about web frameworks, is that they help tell you where and how to organize your code. They give you test directories out of the box, they help bootstrap a good way to get your database connection up in a way that avoids shared sessions across requests, etc. Aspen does a lot less of that, and I could see a younger dev doing way more copy/paste of code than I'd want. It's a little bit of a bare framework, which is great to help integrate you tools of choice, but also provides a barrier to entry for new developers at times.

  • Django and Google App Engine: Why I'm using Flask and EC2

    This talk was from a friend on the west side of the state. As I've been part of the IRC discussions where he's been trying to go through various tools to build some small and simple web apps, it was cool to see the story in one swoop. He's a fan of the microframework. For his use cases, the full stack just threw hurdles in front of him. It's great to hear he's found a tool he loves in flask.

  • Evolving an internal web service

    Taavi gave a great talk on how they worked to rebuild an aging PHP app in Python over a long period of time. It's a great example of what I've been trying to get going for a while, APIs all the time. Everything seems to talk to the new application base via an array of remote methods and this great decoupling has been a boon for them to help provide data to several different systems in a clean way.

    I wish I had more time to try to chat with him. They're doing some cool things with SqlAlchemy, migrating data from old systems to new, and just some very good work and testing done on performance.

  • Creating web apis that are a joy to use

    This was something I really was interested in. Since I've been pushing apis like mad at work and I've been working on my first public one for Bookie, I knew I needed some help/guidance on this. Issac did a great job hitting home the big rules I kind of knew, but wasn't following that well.

    First, document, document, document. I loved his graph of user happiness vs amount of documentation. Users of your api don't get happy until the docs are near complete. Until then, it's just as bad as no documentation. I've got some typing ahead of me.

    The second point was something I was battling with a bit. I tend to think of the api in terms of usage. "You want to do task XXX". At that point, I'm deciding what the user wants to do. In reality it's more about the term "resouce". A resource can be data, a function (send email), or something else along those lines. However you want to expose them via the api in simple distint manner. Just taking your current html view you push to browser users and building an api that is the same doesn't work. After all, the great thing about the api is that people build and do things you didn't think to do on your current application implementation.

  • My Talk, Sqlalchemy Tutorial

    Finally, I was of course at my talk. This year I decided to really didn't like myself and I should do more than a talk, but a two hour tutorial with some hands on coding exercises. The room was full of people of all levels and was a bit more challenging than I originally thought it would be. On top of that, the AC broke and the rooms were over 85 degrees which made holding an audience's attention all that much more challenging.

    In the end, I think things came out ok. I've heard from an array of people that they enjoyed the talk. Once the first hour/talk part was over, most of the room left. We had about seven people that did the hands on code for the second hour. In the end, it sucks, but I can't blame them. If it wasn't my talk I'd have searched out cooler air as well. I hope that people still take the time to try out the hands on code and let me know if they run into any issues. If you do, feel free to email me rharding@mitechie.com.

    Thanks to PyOhio for letting me take a shot at something more classroom like. It's a new challenge to go from a talk to a tutorial and I encourage people to try it out.

Networking

I think it's true what they say, as you go to more and more conference you tend to do and learn less in the talks themselves, but make up for it with the great networking. This year it was very noticeable. What was great was that I got a chance to meet several people I've been following on Twitter for a while. These are people that are interesting, respected, and meeting them was kind of mini-starlet moments.

I got to have a great discussion around apis and self bootstrapping application installs with Issac Kelly. I met Michael Trier whos been a great Python presence on Twitter for a while now. I also caught up with the Ohio crew and guys Dave, Mike, and Catherine. If you run into these guys start a conversation, it'll be worth it.

I also had a ton of great conversations on things from new people that I really wish I did a better job of remembering and tracking down names for. Sorry that I don't call you all out.

All the hallway stuff really helped make this PyOhio a great one for me.

Sprints

Another great thing PyOhio does are the sprints. Unfortunately I could only do them on Saturday, but man what fun it was. I think we managed to get 6 or so people with their own up and running Bookie instance running. We had nearly a dozen people hacking on things at one point. We had some fun hammering pypi from the wifi network and some really good ideas came up to help make the installs a bit easier. At the end of the night we had a pull request and some definite interest going forward. I hope that the people that sprinted on Bookie found it interesting to take part in and maybe learned something. I know I've got a lot of work to do still

I'll have a separate Bookie status report out later with some details on changes and things.

A reminder as well, if you'd like to have a hosted Bookie account on bmark.us just sign up to the waiting list here: http://goo.gl/BBn2b

Bookie Status Report: July 6th 2011

A slow plodding week in Bookie progress this week. Thanks to Greg and Craig for some help in cleaning up a feature.

  • New version of the Chrome extension that will do a much better job of error checking on the options page. Make sure to get the updated extension.
  • The Account page has been cleaned up. You can now view your Api Key, change your password, and update your base account information such as name and email. I also did some ui cleanup so things should start to be a bit prettier. Let me know if you have any suggestions to the account ui.
  • I fixed the Google fonts used so things might look a bit different. I'm using the Cabin font for the body. I think I want to change it for the readable version body though. It's not great for large bodies of text. What do you think?
  • I also updated the mobile view to the jQuery Mobile beta release. I finally found the cause of a nasty bug and things should work ok. I still need to write my own history api wrapper. So navigating back and such is pretty broken still.
  • Per request of Greg and Craig, when the bookmark isn't for the current year we now add the '10 or whatever to the calendar graphic. You can see an example over here.

Next up on the todo list:

  • We need a password reset feature for those that have forgotten their passwords. There's some database side start to this, but no ui for it yet.
  • I want to start on a page in accounts for viewing your readable results such as displaying 404 counts and such. See ticket #47

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.4
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 29 2011

Shorter update list for Bookie this week. Took some vacation and not a ton of time hacking.

  • Fixed a bug in the readable/existing.py handling a non-asci url
  • Fixed all of the js/css to come via https so site should be happy lock for you
  • Added some app logging, log failed logins, imports, exports of bookmarks
  • Start the /{username}/account section with it's own sub nav and some css for that. The logout/export/import moved there

Most of this got pushed out in an incomplete form onto the live site. From here I need to add the edit form for the user's details and clean up the navigation to be reusable across the different pages.

I also want to keep adding some logging and stats counting so we can start to provide some data porn to both myself and users of their info.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 22nd 2011

Phew, it's been a good week with Bookie. First, we have some new ways to contact and follow Bookie. There's now a Google Group for better long form discussions and assistance. You have a feature idea, question, or just feedback? Then go ahead and send it to the list. Don't forget to follow BookieBmarks on Twitter. I want to start using these things to bounce ideas off of people as we move forward.

We managed to get user authentication working and added support across the site for routes for each user in the system. This was great because it means that http://bmark.us is a live running instance with a few different real users. So far we're running seven different users on the system. Part of that was getting all the routes updated, the queries, the templates to only show links for import/export if you're logged in, all of that stuff. The extension needed a little bit of updates, but the only big thing is changing your API URL to have your username in it.

We moved the documentation to http://docs.bmark.us and I tried to update the links in the docs as well. Much as Identi.ca is a running instance of Status.net, Bmark.us is a running instance of Bookie. I just can't get a cool url for Bookie such as http://bookie.net.

With the new site I purchased an SSL certificate. So you'll notice that the site is now all behind https. Part of the other sysadmin items were to setup twice daily pgdumps of the data which is sync'd off to S3, munin to monitor the server resources, including plugins for Nginx and Postgresql, and some Nginx config tweaks which I still need to document in the docs.

All of the code updates are available in the develop branch on Github. If you've got any questions feel free to hop into #bookie on irc and let me know.

With all this going on I'm looking for some early alpha testers. If you're interested in Bookie, but couldn't, or didn't want to, setup your own instance, let me know. I'd love to get another 6-10 people on this instance. The feedback of current users has already paid dividends. You can thank them for a pair of new features in the Chrome extension, including a new keyboard shortcut (ctrl-alt-d) and support for helping auto complete recently used tags.

Coming up, we need to work on moving the app forward by adding a user account page, creating add/edit abilities for authenticated users, and a bookmarklet that uses the add ability for mobile devices and other browsers.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 15th 2011

I just finished up reading Start Small, Stay Small and there were some good points in there. One is that writing about your progress on a project each week helps people move forward. There is something about putting down what you've accomplished and what you plan into the public that helps keep the motivation motor running.

In an effort to keep Bookie from stagnating, I think that's a good thing to start doing. Count this as the first of a series of weekly progress reports I'm going to be doing. I also like that it helps show, beyond links to commit logs, that Bookie is moving forward and getting updates.

This past week has been a bit crazy. There hasn't been a ton of time to put into things, but I've managed to move a few big things forward:

First, work on making Bookie work via user accounts and logins is moving forward. Basically all of the urls in the application needed to be updated. Currently there are two sets. If you leave out a username from the url, you get overall, full site info.

In this way, a url of /recent will pull the 50 most recent bookmarks from all users on the site. However, /rick/recent will only pull the 50 most recent bookmarks of the user rick. The API urls needed to be updated as well. There's a ton of work in getting this going, but it's a major step of progress to allowing me to host a version of Bookie that other users can sign up for. Since that's really the big goal that I've set myself by the end of the year, I'm feeling good on this one.

The idea of multiple users has me realizing that my little readability.py script that fetches url content from bookmarks and stores the clean, readble parsed html for that page needs some work. It'll never scale that way. So I've split the work into a couple of parts.

One part is a node.js script that will fetch a list of urls to go fetch and asyncronously goes out and fetches the html content. It then shoves the bookmark id and the content into a beanstalkd queue for processing. The queue is polled by a python script that then calls a new Bookie API call with the content and the id. Bookie then runs the parsing code against the content and stores it in the database. The async code on node.js can fetch the html content in a hurry. In testing with my SSD hard drive and sqlite, I'm able to pull, process, and store more than one url per second. This is with 1 node.js producer and two running instances of the python consumer.

I'm definitely looking forward to ramping this up on a real server with Postgresql running. I'd love to be able to pull down and parse content at some decent rates to be able to cope with new users signing up to the service.

So that's this week's report. Next up is more work on the multi user setup. The tag urls still need work and all of the unit tests that I had need to be updated to test the new urls. This also means some duplicate tests to check both with/without usernames in the urls. Work is never done!

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

First Bookie Sprint Good Friday

I've been cranking hard on getting Bookie going towards a 0.2 release. There's been some great work from contributers and things are really getting to be fun. To help give Bookie a boost, I'm holding a sprint at my place to hack on the 0.2 feature list. The big things we're working on include making the UI not suck 100% and getting the readable parsed content of pages integrated into the site really well.

So this Friday, 11am - 4pm is sprint time. If you want to head over let me know and I'll get your directions and such: rharding@mitechie.com.

If you can't make it, but you're interested in getting into it, jump on irc and join #bookie on freenode. Get an install of bookie going, test the importer and such. There're a few bitesite bugs in the ticket tracker that are easily fixed. So jump on in.

Introducing Bookie: only for developers

Introducing Bookie: only for developers

Delicious has been pretty stagnant over the years. I love using Instapaper and really kept waiting for Delicious to add that feature in to my pages I send into Delicious, but it never arrived. I also kept waiting for link checkers, a decent mobile view, and maybe even some support for full text search of actual web page content. Now that Yahoo has leaked out that Delicious is going to be "Sunset", I've finally gotten the push to start up my own bookmark storing application that I can add all the great features I want to have in there.

So without further ado, I give you bookie. It's built in Python using the Pyramid web framework. It's my first Pyramid app, but I do Pylons for the day job so it's not too far off.

This is the 0.1 release I'm calling "only for developers". It's not an easy one-click install, and right now it'll want you to have the -dev packages for the db engines to work right, but it does work. It can store, retrieve, and search bookmarks stored. It supports Delicious and Google bookmarks import, and will do an export. There's a fully functioning Google Chrome extension you can use to store and edit bookmarks.

This is currently a self hosting application. In time, I hope to provide a hosted version perhaps. I also hope to get a Firefox extension, as well as get started on the fancy features I noted above. So this is a LONG way from complete, but I feel like I've hit at least a 0.1 worthy milestone and want to put a stake in the ground. Here's the start of something, and hopefully it'll keep improving in future releases.

I want to thank my local guys for helping, testing, and adding great bits of code to bookie already. It's a lot of fun seeing other people get interested in things and contributing as well.

I'm trying to keep the todo list and some idea of the next milestones in the github wiki. Feel free to note bugs, add requests, etc. You can also chat with us in #bookie on freenode.