fulltext

Bookie 0.4: one week retrospective

Phew, that was a whirlwind of a week. Just over one week ago I finally released Bookie 0.4 and published the blog post to reddit as an announcement. This introduced signups and I was eager to see if there was real interest in the project now that users could sign up and try things out.

By the numbers

Traffic definitely came.

  • The blog post picked up 800 visits over the two days in the weekend.
  • https://bmark.us grabbed 360 unique new visitors.
  • We went from 58 to 126 activated user accounts.
  • Those users brought us to over 26,000 bookmarks stored in the site.

Complications

Of course, any swarm of new users finds the holes in the system and Bookie was no different. There were a few issues. First, the celery task that sends out emails on signup wasn't running because the email config wasn't setup right. This was a pretty quick fix. Next, the import system wasn't filling out the path for uploaded files correctly. This one was another pretty easy fix, but I managed imports manually until I got the fix deployed.

The big thing was that, for probably the first time, all three moving parts to the system were trying to store bookmarks at once. The celery backend, the web UI, and a cron script that looks for new bookmarks without readable content and fetches it for storing. All of these hit the Whoosh fulltext index and caused locking issues that broke both imports and saving new bookmarks from the webui until I figured out the issue and just reset the fulltext index.

It was pretty bad timing as I could see users trying to add test bookmarks via the web interface. Google realtime analytics is pretty entrancing to watch. In the end I had to run to the Whoosh docs and change things up to use the async writer instead of the default locking mechanism. This got things running again, but the problem now is that I had to remove all the existing fulltext index. I've still got to finish a background job that will walk through all bookmarks and index them.

At some point I might need to remove the fulltext indexing from the current SqlAlchemy event hooks, but as purely background celery jobs that I can control from one place easier. This would remove the lock at all from the cron job and the web ui.

Disappointments

While I could see the charts showing traffic, it was tough because it was pretty invisible traffic. There were only three new users into the #bookie irc channel, and only a few people left comments in the reddit thread. No one left a comment on the blog post. Both my Twitter account and the Bookie accountgained fewer than 5 new followers. While the repository was starred many times, only two forks were created.

Going forward

There are a few new users active over the last week, and I've gotten a pair of pull requests. While the saving of new bookmarks was broken for a lot longer than I'd have liked, the site never went down. Imports were done in a semi-reasonable time frame. All of this felt pretty great and is encouraging for future work. I still need to finish fixing up the readable parsing. It's the big selling point of Bookie, and the fact that fulltext search and readable parsed content for all bookmarks isn't there is frustrating.

Here's looking forward to great work and a more popular release announcement for Bookie 0.5.

Bookie status report: Aug 28th 2011

Updates this week

With 0.3 out and gone, it's time to move forward on 0.4 tickets. I started out with the big one, edit/add form ui for bookmarks on the page. This now works and when you're browsing urls from your page (/{username}/recent) you'll see a new "E" icon next to the "R" one. You'll also see a "Add" button at the top left.

Once we had the add/edit form, I could setup a bookmarklet to store your current page there. If you log into the site and go to your account page, a bookmarklet url is there for the taking. Drag it to your browser bookmark bar. On some platforms (hey Android!) you can't save a bookmarklet to the bookmarks. You first have to create a normal bookmark, and then edit the link after the fact. To help that out, there's a copy/pastable version of the bookmarklet on the account page. Just long press the text area, select all, and then paste into your bookmark from earlier.

It's my hope that this bookmarklet will help users on Firefox and other browsers use Bookie more. It's not the full extension experience, but hopefully holds people over for now. Note that when storing bookmarks in this way, the content is not immediately available as from the Chrome extension. The cron job that does fulltext parsing runs at 4am each morning.

Next I started some css work. The readable page no longer uses the Cabin font. It's also got some default styles for H1 and such. Let me know if any content looks funny and we can look to add some additional styles going forward.

Next up

For the week ahead, I'll be mainly working on tests. I don't have enough for the new add/edit feature. I'm still working on adding live api tests to the phantom.js test suite. If I get time, I also hope to start work on a private bookmark feature.

Alpha Testing

We have a signup for if you're interested in alpha testing the hosted install at https://bmark.us. If you'd like to try it out fill out the signup form here.

Taking Part

If you care to help or follow along make sure to follow the project on github: http://github.com/mitechie/Bookie

  • Current Chrome Extension: 0.3.16
  • Most recent code updates: develop branch
  • Current working branch: develop