Sarah Allen’s reflections on internet software and other topics

I attended the Lucene/Solr meetup this week — quite a swank event sponsored by Salesforce with tasty appetizers, beers and an incredible view of the bay. The three speakers were very knowledgeable and well spoken and I enjoyed hearing about the different applications of Lucene and Solr. Below are my rough notes.  For folks who want to learn more about Lucene and Solr, check out the upcoming conference Lucene Revolution, Oct 5-8, 2010 in Boston.

Search@salesforce.com, Bill Press, Salesforce

Salesforce uses Lucene 2.2 (not Solr) and shared some stats about their seriously large scale operation:

• millions of searches per day, hundreds of thousands of users
• hundreds of millions of doc updates per day
• force.com platform, 72,500+ customers, 150,000+ apps
• almost half a billion indexing events per day (batches can include > 1000 documents)
• Over 8 TB of searchable data
• incremental indexing (90% < 3 mins, 70% < 1 min )
• 6M queries per day, mean search time 250ms (76% < 250 ms, 89% < 500 ms)

It’s a multi-tenant architecture, each org has 1-100,000s users and had a single codebase, which means there is just 1 version to support at one time.

• consistent hashing for node affinity
• throttling for fairness
• record type bucketing, as well as by org

They use post-filtering for:

• authorization
• reranking in the DB, last update

They query db to bridge the gap with indexing lag.

They are faced with new search challenges driven by what Salesforce CEO calls “the facebook imperative.” When he started Salesforce, he used to ask “why donesn’t every enterprise app look like amazon?” Now he asks: “why doesn’t every enterprise app look like Facebook?”  (side note: this is an echo of what many folks have been saying for a while, that social networking makes sense as a feature of an app, rather than just destinations like Facebook and LinkedIn.)

Salesforce allows you to have a feed on a record, follow accounts, status updates for accounts.  They index tracked changed.  They need to search this rich set of data which is people articulating their interests. Bill noted that the needs of structured data are really different from unstructured data.

Practical Relevance, Grant Ingersoll, Lucid Imagination

Grant Ingersoll spoke of “two tales of relevance”

• The case of the missing data: you know you have poor relevance when the most important search result is on page 10.  For example, the accessories for an item are often listed higher on search results than the item itself.
• The power of suggestion.  Grant cited a specific case where just adding auto-suggest added 100s of millions to the bottom line.

Better search results = less time searching, more time acting

Other cases to consider:

• Only the first result matters, such as Google’s “I’m feeling lucky”
• Known item searches, for example: NetFlix has a high frequency of people searching for specific movies
• Are you finding all the documents that are relevant?  In the case where you want to analyze all the results returned/
• Is zero results the right answer?  Where people want to definitely know that something is not present
• Is it important that you don’t have a result that doesn’t match (e.g. Yelp doesn’t want to find a plumber talking about unclogging what you just ate when you are looking for a restaurant)

Befre undertaking any relevance tuning, you need to define what “better search” means to you.  There are many ways to test and measure:

• a/b testing
• log analysis
• empirical (top x queries, plus random sample) — read and evaluate queries, top 10, top 50, have your top biz people rate what is important –> leads to actionable data
• ask your users, thumbs up/down around your search results
• Ad Hoc evaluation
• TREC — fixed data set, fixed queries, see open relevance project (open source TREC)

Capturing user feedback:

• log analisys (click analyss)
• rating/reviws
• filters and facets

Grant notes that Lucene searches default to “or” out of the box, when “and” is typically better today.  He had a list of links that he suggested we check out (sadly I couldn’t type fast enough, but here are some I wrote down):

• code.google.com/p/luke
• solar analysis tool
• sigir.org
• Debugging Relevance Issues in Search

auto-add phrases to your questies — surround with quotes — automtric win
auto-add a “sloppy phrase” — large slop factor, like an AND, boost when words are close

Logs, Search, Cloud, Jon Gifford, Loggly

Logfile managemetn in the cloud (no Hadoop).  Logs are painful — distributed, large, ephemeral.  Most log search is hightly skewed.  “We’re just implementing grep across terabytes of data.”  This was a compelling talk, but it took most of my attention to follow, so my notes are weak and may make sense to no one except me:

syslog + 0MQ + SolrCloud
0MQ – not traditional queing, it fails, when it fails we lose data, but it is very fast
Solr give s us facets which gives us graphs

run many indexers, “hot shards” — the indexers update small shards

0MQ gives us node-specific input queues for Solr

nrt + solrCloud = Our Nirvana

Hot shards re chilled when we stop writing to them

Solr is awesome at what it does, but not so good for data mining
– plan to plug in Hadoop for large-volume analytics
Syslog is the only way in for now, adding others, http, scribe, flume,

I just got back from Google-sponsored hack session at RailsConf. Google App Engine and JRuby combine to create some real awesomeness. I created an small app that uses some classes from Lucene (written in Java) with a Rails app (written in Ruby) to search text stored in the App Engine data store. The idea was to search English text using standard linguistic analysis to distinguish whole words, as well as find variants of the same root word (e.g. find “run” and “running” when searching for run, but don’t find “runt”). It was helpful to read prior work by Ikai Lan.

First I downloaded Solr, which includes Lucene. I unzipped it into ~/src/sdk, then I set up Rails for app engine. I experimented first in irb:

$ appcfg.rb run -S irb -r config/environment

> require '/Users/sarah/src/sdk/apache-solr-1.4.0/lib/lucene-core-2.9.1.jar'
=> []
> require '/Users/sarah/src/sdk/apache-solr-1.4.0/lib/lucene-snowball-2.9.1.jar'
=> []
> java_import org.apache.lucene.analysis.snowball.SnowballAnalyzer
=> Java::OrgApacheLuceneAnalysisSnowball::SnowballAnalyzer
> snowball = SnowballAnalyzer.new("English")
=> #<Java::OrgApacheLuceneAnalysisSnowball::SnowballAnalyzer:0x2a134eca>
> s = StringReader.new("Testing Snowball")
=> #<Java::JavaIo::StringReader:0x7deb41d6>
> token_stream = snowball.tokenStream(nil, s2)
=> #<Java::OrgApacheLuceneAnalysisSnowball::SnowballFilter:0x439a8942>
> token_stream.next.term
=> "test"

Then I copied the .jar files I needed into WEB-INF/lib

$ cp ~/src/sdk/apache-solr-1.4.0/lib/lucene-core-2.9.1.jar  WEB-INF/lib/.
$ cp ~/src/sdk/apache-solr-1.4.0/lib/lucene-snowball-2.9.1.jar WEB-INF/lib/.

I ran generators to create my “content” field which is for the free-form text and another column I called index, but was mis-named, perhaps would be better to call it a “vector” or something. In any case, what I called the “index” was for the data that we will search on and I made it a special “List” type which is fast for app engine to search whole words.

./script/generate scaffold note content:string index:List -f --skip-migration
./script/generate dd_model note content:string index:List -f

Then I changed the model to use the Snowball analyzer from Lucene. I include all of the model code below, which uses DataMapper instead of ActiveRecord so that it can use the app engine datastore which is not relational. I was quite impressed with how concise the code was and how easy it was to mix Java classes in Ruby — I simply declared the class at the top of the file with a “java_import” and then I could call SnowballAnalyzer.new as if it were a regular Ruby class. Pretty sweet.

java_import org.apache.lucene.analysis.snowball.SnowballAnalyzer
java_import java.io.StringReader

class Note
  include DataMapper::Resource

  property :id,      Serial
  property :content, String,        :required => true, :length => 500
  property :index,   List,          :required => true
  timestamps :at 

  before :valid?, :update_index

  def update_index
    analyzer = SnowballAnalyzer.new("English")
    s = StringReader.new(content)
    token_stream = analyzer.tokenStream(nil, s)

    terms = []
    while (token = token_stream.next) do
      terms << token.term
    end
    self.index = terms
  end
end

I then changed the index page to include a search form (joined in late night pairing by Ian McFarland) and in the controller action I could use very familiar Rails code to find the results:

    @notes = Note.all(:index => @query)

The complete code is on github.com/ultrasaurus/full-text-search-appengine.

Create an app ID on the app engine page. Add the app name info the config file and publish the app.

vi config.ru
./script/publish.sh

Note: this is not a complete solution. There are all sorts of features of Lucene that aren't surfaced in this example, most significantly ranking. Also, the query really should be tokenized and stemmed as well as the "content" field that are being searched.

John Woodell (@johnwoodell) gave an excellent talk at the App Engine Meetup this evening on using Ruby on Google’s AppEngine. After some highlights of App Engine and a tour of three real-world use cases, John gave a series of quick demos of what seem to be very well-documented resources for getting started. I built the simple “hello” Sinatra app during his talk which was pretty satisfying.  I was distracted by my success with the first app, so I missed some of his Rails presentation, but it looks like the docs go through what he demo’d.

Overview

Key Features

• No need to install or maintain your own stack
• “We do the scaling for you” -> Google services via standard APIs
• Charge only for actual usage – Always free to get started
• Built-in application management console (pretty sweet)

Gotchas

• no native code
• no sockets or thread
• no writing to the files system
• 30 seconds to do your work, then the request times out (issues with initialization in Rails, but they’ve developed a “deferred dispatch” workaround)
• takes several seconds to “spin up” a new JRuby instance when your app first starts or your load spikes

Good things to know

• AppEngine datastore has no schema.
• Reads are always the same speed no matter how big your data store, writes are slower
• You use DataMapper rather than ActiveRecord.
• memcache API is available
• URL Fetch API is a drop-in replacement for Net::HTTP
• as of today, there will be a release where openssl “just works” for the first time

Matthew Blain, the engineer for the main bulkloader tool, gave a short talk about it.  Bulkloader lets you put data into the AppEngine data store. You can use it to import data or to move from one AppEngine app to another.  It currently supports limited input:  csv and simple xml.  It also supports “simple text” for output.  You can write any connector you want.

Some real world references to people using Ruby on App Engine:

• real world app using JRuby, Rails and App Engine (azazeal)
• http://www.codingforrent.com/
• iPhone / iPad app with JSON interface: blog post and gist by John Wang
  [update from John Woodell's comment]

Very nice getting started guide, which I followed to build a Sinatra app in about 5 minutes.

sudo gem install google-appengine
appcfg.rb generate_app hello
dev_appserver.rb hello

After this I had an app running locally at http://localhost:8080/ — I find it unsettling to hear that the dev app is running on whatever Ruby I have, but when I deploy it is JRuby.  John reassures me that the only real glitch that I’ll run into is if I have a gem with native code, for which there aren’t also java extensions (or an alternate pure Ruby implementation).

Installation notes: hasn’t been tested with rvm, the env is there, so you don’t need it. gems are stored in a jar file, kept in .gems/bundler_gems in your app directory

Then to deploy my app on App Engine, I needed to use my Google account to register an app. I picked the name “hello-sarah” which I then added to my config.ru.  I also added a little snippet of Ruby code to the Rack app to report the current time, so I could tell it was mine.

Then I typed some magic words into my local command line:

cd hello/
appcfg.rb update .

Then my app was live at: http://hello-sarah.appspot.com/

Since I tweeted it, I got a chance to see how the application dashboard works. Here’s what it looks like for this app which was first deployed a few hours ago:

I cornered Eric Ries after Startup Lessons Learned Conference yesterday. Many things he said in the course of the day resonated with me, but only one was never elaborated upon and I felt driven to ask a follow-up question.

Eric said that this was a “minimally viable conference.” They set a date and the topic, found a venue and let people start signing up before they knew exactly what it was going to be. This is exactly like what we did with the SF Ruby Outreach Workshops for Women. I know, that’s an entirely different topic, but the technique was the same. We came up with what would minimally work (we each would teach a maximum of 20 students in a couple of corporate conference rooms), then we posted the event, and received a huge outpouring of both demand from students and support from volunteers and it became much more than we had envisioned. Pretty cool, huh?

Eric has been much more successful in getting press attention than we have. So, I asked him about his techniques. Basically, he said that he just talked to everyone he could about it. He developed a core group of bloggers who were interested. In retrospect, perhaps it was a dumb question: the primary tactic he chose to create change was communication, our tactic is the more direct one of simply teaching the skills to people who are interested. Eric had to work much harder to communicate his message and stimulate interest and has a less well-defined curriculum.

Then Eric asked me a good question. You are being successful. You are having the impact you sought. Why do you care to get press? I suppose if I had a better answer to this, I would have been more driven to get the word out to a wider audience. I always argue that my strength is not PR or marketing; however, I didn’t start this year with many skills in large event planning, and I didn’t let that get in my way.

In trying to answer his question, I cited a recent NYT article: “Out of the Loop in Silicon Valley” which is a great article that highlights many successful women. The journalist, Claire Cain Miller, does a good job of presenting why anyone should care about this problem and illustrating a bit of why things are, but it comes off as rather bleak. I fear women (and girls) will read it and think: I don’t want to go there.

I think an alternate loop is forming in Silicon Valley. Our Ruby Workshops are not an isolated phenomenon. New organizations and events are springing up, like Women 2.0, DevChix, Girls in Tech, Girl Geek Dinners, and She’s Geeky to augment the work of long establish groups like the Anita Borg Institutute, SWE, WITI and others. Technical women and women founders are out there and with the predominantly male networks hard to tap, women are creating their own.  At the same time, there are women and men with experience and connections who are working to acknowledge and reach out, as individuals. There are organizations, such as Springboard, Astia, and Golden Seeds or newer incubators like Springworks, which help women obtain angel and VC funding [*].

Note: at the conference on Friday, there were only three women on stage, all on panels.  Eric admitted that he had a tough time because he had to rely on his personal network in this first conference, but he did pull in additional women as mentors and hopes to have more women speakers next year.  I wasn’t planning to mention it, but I appreciated that he brought this up, non-defensively in our conversation.

Yes, there are fewer women starting innovative, high-growth companies and fewer women in tech, but there are more than we read about, more than we notice.  Change is tough.  It happens one person, one event, one moment at a time, but it is happening.

If you are a journalist or are a blogger, and want to have a conversation, let me know: sarah _at_ ultrasaurus (dot) com.  There are a lot of great stories that I think more people would like to hear.

Steve Blank is probably most well known for this picture about embracing change:

Observations about Silicon Valley:

• $500K is the new $5M
• Seed Funds are providing entrepreneurial education
• Customer Development / Lean Startup adoption:  We are all collectively getting smarter at a scary rate.

99.7% of all companies are <500 employees and employ half of the people in the country. In contrast, a scalable startup is designed to grow big and it typically needs risk capital.  (Note: venture capital was originally called “adventure capital.”)

Your job as an entrepreneur is to search for a scalable and repeatable business model.  Then build a business around it.  Founders depart before the company grows to a large business:

1. Search
2. Build
3. Grow

No business plan survives contact with customers.

Startups search and pivot.  Small and large companies execute.

Startups should focus on:

• metrics, not accounting.
• customer validation, not sales.
• customer development, not product management.
• agile development, not engineering.

Lean startups use agile and customer development to find profits.  (A high percentage of company founders come from dysfunctional families, but that’s another topic.)