bookmarks

Bookie 0.4 released into the wild!

Bookie is a Python based open source bookmark managing web application that includes content archiving, a Chrome extension, and much more. Phew, that took a lot longer than expected. I've tagged Bookie 0.4 and the live site is updated to run it.

This brings a ton of work on getting an updated webui with some client side MVC, an API, Celery job running backend, some stats, and spin off projects such as breadability and a cli client.

The big thing is that signups are now there as well as a landing page. So hopefully this will spike up interest in new users checking out Bookie.

There are still a ton of long term ideas to work on with Bookie. I'd like to get a 'reading' view setup so that you can easily run through the bookmarks you've marked `toread`, especially in a mobile view. <3 my N7. I also want to work on getting suggestions for related bookmarks, suggested tags based on content, and other interesting machine learning type problems.

If you're the type that takes your bookmarks seriously give it a try. If you don't want to run your own instance, sign up to https://bmark.us and try it out there.

You can get an idea of the roadmap we're working off of on the Trello board.

Bookie weekly status report: Sept 7 2011

Updates this week

Sorry, I'm late on the weekly report. The holiday weekend threw me off. However, I've got good stuff for you. I completed the port of the fulltext search from being database specific to being Whoosh which is pure Python. This should be cool because it reduces code complexity trying to change things up based on the database used. It also been performing really well. The hosted http://bmark.us site is using it for everyone's content.

Be aware, if you do the migration, it takes a while to run. It has to go through and processes every readable record one by one. So when you upgrade, let it run.

What's next

Aside from that, I started playing with an all Javascript front end using Backbone.js. The idea is to see about setting up a very interactive front end that would allow in place edits of bookmark data easily using the existing API. We'll see how it works out.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 28th 2011

Updates this week

With 0.3 out and gone, it's time to move forward on 0.4 tickets. I started out with the big one, edit/add form ui for bookmarks on the page. This now works and when you're browsing urls from your page (/{username}/recent) you'll see a new "E" icon next to the "R" one. You'll also see a "Add" button at the top left.

Once we had the add/edit form, I could setup a bookmarklet to store your current page there. If you log into the site and go to your account page, a bookmarklet url is there for the taking. Drag it to your browser bookmark bar. On some platforms (hey Android!) you can't save a bookmarklet to the bookmarks. You first have to create a normal bookmark, and then edit the link after the fact. To help that out, there's a copy/pastable version of the bookmarklet on the account page. Just long press the text area, select all, and then paste into your bookmark from earlier.

It's my hope that this bookmarklet will help users on Firefox and other browsers use Bookie more. It's not the full extension experience, but hopefully holds people over for now. Note that when storing bookmarks in this way, the content is not immediately available as from the Chrome extension. The cron job that does fulltext parsing runs at 4am each morning.

Next I started some css work. The readable page no longer uses the Cabin font. It's also got some default styles for H1 and such. Let me know if any content looks funny and we can look to add some additional styles going forward.

Next up

For the week ahead, I'll be mainly working on tests. I don't have enough for the new add/edit feature. I'm still working on adding live api tests to the phantom.js test suite. If I get time, I also hope to start work on a private bookmark feature.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 22 2011 (0.3 yay!)

Updates this week: 0.3 out the door

We had to do some hacking of dying limbs to get there, but 0.3 is out the door. This was the biggest release cycle since things started up. However, the features of an API and multi user auth were worth it. There were a ton of small tweaks long the way as well. Here's hoping that 0.4 is a bit faster in the moving and releasing

You'll notice that we're backtracking on having a Firefox extension and a mobile site. Both are horribly broken at the moment. I really want to get this going though and they will be specific todo items in up coming releases. Well, except the Firefox extension part. Honestly, every time I try to get started with the Firefox extension I feel like the team there is just damn hostile to developers. It's beyond ridiculous how unable to get a handle on things there it's been. Confusing docs all over the place, an add on sdk that tries to run a virtualenv, but you don't built it from any of your local python's so your fubar if you have the wrong one, the fact that it's all getting redone again in JS and not Python...Sorry. I'm just not willing to be a fulltime firefox dev to make the dippy thing work.

Links

Next up

  • Work on more of the live api tests. It's running via the latest build of phantom.js and working well
  • Start work on the 0.4 todo list
  • Especially the save/edit ui and a bookmarklet for those.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 14th 2011

Updates this week

Finally, the new and fancy Bookie API is reworked and better than ever. The live running site is updated to use the new api and there's a new chrome extension that's update for it as well. Make sure you get the latest dev version of the Chrome extension.

With the api updated, it's time to get to work on finishing up 0.3 for release. This means that we need to finish updating the mobile site and some rework of how the readable content is stored. Since we pull the content out of the page based on actual page you see, we should store these separated from user to user. We don't want to leak logged in info from one user to another.

Farewell Firefox

With the api update, the Firefox is so far out of date as to be unusable. There's some work going on to help catch it up, but I think I'm going to remove it as a feature in the docs and such. We just can't claim it at this time.

New testing

With the new api we hit some testing issues. Since there's no good way I've found to functional test the extension, the tests aren't great and reliable. I also don't have tests setup on some shared code like the bookie.api.js code. I'm thinking of setting up a running instance with some live data with the sole purpose of running tests on the JS code against it. We might even go so far as to setup the extension html files as an actual website and write some functional tests around that. After all, I can load popup.html and do some testing of that as a normal web page. We'll be missing the interactions with background.html and such, but might be worth some attempt.

Any ideas and help with this is most welcome.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.11
  • Most recent code updates: develop branch
  • Current working branch: develop

Bookie status report: Aug 7th 2011

See, I can do these things once a week, I promise!

Updates this week

Progress this week is all in the api branch. It's taken some time to figure out just what things are going to look like, and I started with the docs. Lots and lots of docs updates for the api. You can check out how things look so far.

  • Gone is the custom json response object. We now send back an http status code and possibly a json message
  • The auth code is all updated. We now use a new api_auth decorator in lib/access.py that will check the api call and take care of the 403 response if things fail.
  • Most of the tests are commented out while we enable/fix them one by one as the new api methods get filled in. Once we're sure thing are where we want, we'll revisit the rest of the apis. Since parts of the functioning site use the api it really needed to be done.
  • The first few new api methods are working correctly and tests are passing in the api branch. This is mostly the /{username}/bmark/XXX urls. So adding, updating, deleting, and getting a bookmark for a user is set.

To make some of the new request changes we needed to update to Pyramid 1.1. This gave us things like request.response.status_int and some changes to the way the http exceptions work.

The goal for the next week is to keep plugging at the api. If you've got any feedback let me know. I'm really hoping to do a really solid job on this updated api and it will help with the extensions, feature updates, and encouraging new developers to get involved.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.8
  • Most recent code updates: develop branch
  • Current working branch: feature/api

Bookie Status Report: July 6th 2011

A slow plodding week in Bookie progress this week. Thanks to Greg and Craig for some help in cleaning up a feature.

  • New version of the Chrome extension that will do a much better job of error checking on the options page. Make sure to get the updated extension.
  • The Account page has been cleaned up. You can now view your Api Key, change your password, and update your base account information such as name and email. I also did some ui cleanup so things should start to be a bit prettier. Let me know if you have any suggestions to the account ui.
  • I fixed the Google fonts used so things might look a bit different. I'm using the Cabin font for the body. I think I want to change it for the readable version body though. It's not great for large bodies of text. What do you think?
  • I also updated the mobile view to the jQuery Mobile beta release. I finally found the cause of a nasty bug and things should work ok. I still need to write my own history api wrapper. So navigating back and such is pretty broken still.
  • Per request of Greg and Craig, when the bookmark isn't for the current year we now add the '10 or whatever to the calendar graphic. You can see an example over here.

Next up on the todo list:

  • We need a password reset feature for those that have forgotten their passwords. There's some database side start to this, but no ui for it yet.
  • I want to start on a page in accounts for viewing your readable results such as displaying 404 counts and such. See ticket #47

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.4
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 22nd 2011

Phew, it's been a good week with Bookie. First, we have some new ways to contact and follow Bookie. There's now a Google Group for better long form discussions and assistance. You have a feature idea, question, or just feedback? Then go ahead and send it to the list. Don't forget to follow BookieBmarks on Twitter. I want to start using these things to bounce ideas off of people as we move forward.

We managed to get user authentication working and added support across the site for routes for each user in the system. This was great because it means that http://bmark.us is a live running instance with a few different real users. So far we're running seven different users on the system. Part of that was getting all the routes updated, the queries, the templates to only show links for import/export if you're logged in, all of that stuff. The extension needed a little bit of updates, but the only big thing is changing your API URL to have your username in it.

We moved the documentation to http://docs.bmark.us and I tried to update the links in the docs as well. Much as Identi.ca is a running instance of Status.net, Bmark.us is a running instance of Bookie. I just can't get a cool url for Bookie such as http://bookie.net.

With the new site I purchased an SSL certificate. So you'll notice that the site is now all behind https. Part of the other sysadmin items were to setup twice daily pgdumps of the data which is sync'd off to S3, munin to monitor the server resources, including plugins for Nginx and Postgresql, and some Nginx config tweaks which I still need to document in the docs.

All of the code updates are available in the develop branch on Github. If you've got any questions feel free to hop into #bookie on irc and let me know.

With all this going on I'm looking for some early alpha testers. If you're interested in Bookie, but couldn't, or didn't want to, setup your own instance, let me know. I'd love to get another 6-10 people on this instance. The feedback of current users has already paid dividends. You can thank them for a pair of new features in the Chrome extension, including a new keyboard shortcut (ctrl-alt-d) and support for helping auto complete recently used tags.

Coming up, we need to work on moving the app forward by adding a user account page, creating add/edit abilities for authenticated users, and a bookmarklet that uses the add ability for mobile devices and other browsers.

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.3
  • Most recent code updates: develop branch
  • Current working branch: feature/auth

Bookie Status Report: Jun 15th 2011

I just finished up reading Start Small, Stay Small and there were some good points in there. One is that writing about your progress on a project each week helps people move forward. There is something about putting down what you've accomplished and what you plan into the public that helps keep the motivation motor running.

In an effort to keep Bookie from stagnating, I think that's a good thing to start doing. Count this as the first of a series of weekly progress reports I'm going to be doing. I also like that it helps show, beyond links to commit logs, that Bookie is moving forward and getting updates.

This past week has been a bit crazy. There hasn't been a ton of time to put into things, but I've managed to move a few big things forward:

First, work on making Bookie work via user accounts and logins is moving forward. Basically all of the urls in the application needed to be updated. Currently there are two sets. If you leave out a username from the url, you get overall, full site info.

In this way, a url of /recent will pull the 50 most recent bookmarks from all users on the site. However, /rick/recent will only pull the 50 most recent bookmarks of the user rick. The API urls needed to be updated as well. There's a ton of work in getting this going, but it's a major step of progress to allowing me to host a version of Bookie that other users can sign up for. Since that's really the big goal that I've set myself by the end of the year, I'm feeling good on this one.

The idea of multiple users has me realizing that my little readability.py script that fetches url content from bookmarks and stores the clean, readble parsed html for that page needs some work. It'll never scale that way. So I've split the work into a couple of parts.

One part is a node.js script that will fetch a list of urls to go fetch and asyncronously goes out and fetches the html content. It then shoves the bookmark id and the content into a beanstalkd queue for processing. The queue is polled by a python script that then calls a new Bookie API call with the content and the id. Bookie then runs the parsing code against the content and stores it in the database. The async code on node.js can fetch the html content in a hurry. In testing with my SSD hard drive and sqlite, I'm able to pull, process, and store more than one url per second. This is with 1 node.js producer and two running instances of the python consumer.

I'm definitely looking forward to ramping this up on a real server with Postgresql running. I'd love to be able to pull down and parse content at some decent rates to be able to cope with new users signing up to the service.

So that's this week's report. Next up is more work on the multi user setup. The tag urls still need work and all of the unit tests that I had need to be updated to test the new urls. This also means some duplicate tests to check both with/without usernames in the urls. Work is never done!

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

Introducing Bookie: only for developers

Introducing Bookie: only for developers

Delicious has been pretty stagnant over the years. I love using Instapaper and really kept waiting for Delicious to add that feature in to my pages I send into Delicious, but it never arrived. I also kept waiting for link checkers, a decent mobile view, and maybe even some support for full text search of actual web page content. Now that Yahoo has leaked out that Delicious is going to be "Sunset", I've finally gotten the push to start up my own bookmark storing application that I can add all the great features I want to have in there.

So without further ado, I give you bookie. It's built in Python using the Pyramid web framework. It's my first Pyramid app, but I do Pylons for the day job so it's not too far off.

This is the 0.1 release I'm calling "only for developers". It's not an easy one-click install, and right now it'll want you to have the -dev packages for the db engines to work right, but it does work. It can store, retrieve, and search bookmarks stored. It supports Delicious and Google bookmarks import, and will do an export. There's a fully functioning Google Chrome extension you can use to store and edit bookmarks.

This is currently a self hosting application. In time, I hope to provide a hosted version perhaps. I also hope to get a Firefox extension, as well as get started on the fancy features I noted above. So this is a LONG way from complete, but I feel like I've hit at least a 0.1 worthy milestone and want to put a stake in the ground. Here's the start of something, and hopefully it'll keep improving in future releases.

I want to thank my local guys for helping, testing, and adding great bits of code to bookie already. It's a lot of fun seeing other people get interested in things and contributing as well.

I'm trying to keep the todo list and some idea of the next milestones in the github wiki. Feel free to note bugs, add requests, etc. You can also chat with us in #bookie on freenode.