The next version will be 0.90. It will be a reliability release, but also includes performance gains. The version change will break from hadoop version numbers. 0.90 was chosen as there’s a belief it is maturing towards a 1.0 release.

The main points I picked up are:
* New batch importing allows writing hfiles directly and then just telling hbase where they are.
* Taking advantage of appends in hdfs for genuine durability.
* The namenode single point of failure is being addressed, facebook is planning to release their HA namenode.
* Replication between clusters. Allows cross data center replication. Eventually consistent.
* Tighter integration with zookeeper through a master rewrite.
* Significant work to have less temperamental behaviour during compaction and splits.
* Facebook are planning to release their distribution of hadoop and their highly available namenode.

All in all it’s very encouraging progress. I think there’s a case for us at Last.fm to look at HBase again soon.

Here are my notes from the meetup…

===Reliability===
* Master overhaul. Facebook is making it highly available.
* Test framework.
* Testing failure scenarios.
* Ops team friendly tools.
* HBase fsck.
* More performance metrics (in progress)

===HBase Master Rewrite===
Why?
* Master failover does not always work.
* ZK is patched on.
* Master to region server communication is inefficient.
* The code is a bitch.

The crux
* Better zookeeper integration
* Use zookeeper to track how far operations have progressed. eg: moving a region to another region server.
* Cleaner code.

Opens up future enhanchments
* Master need not do META edits.
* Region servers can recover their own regions on restarts.
* Reporting on shutdown.
* Limit concurrent major compactions across cluster. Because we know the progress.

Why not put META in zk?
* Could be great. It’ll take some work yet. Seems likely in the future.

===HBase cluster replication===
What?
* Fully integrated replication between clusters.
* Ships edits.
* Can be cross data centre.
* Eventually consistent across clusters.
* Master-slave, master-master, circular.

How?
* Master push.
* Write ahead log shipping.
* Meta data stored in zookeeper.
* Logs get shipped in batch and applied to the new region server locally using the htable client.
* Cute stuff.
* Timestamp based

Also…
* There is a new seperate utility program. A distcp-like map reduce job for copying tables between clusters.

===Bloom Filters, they’re back!===
Why they were originally removed?
* Tricky bugs during compaction.
** Solution: Fixed with cleverness.
* We had to estimate key size ahead of time, and under certain conditions the memory use bloated enough to make them counter productive.
** Solution: Fold / compress / cleverness.

Now.
* They don’t bloat memory uselessly.
* Defaulting to 1% error rate. Configureable.
* Great for exact queries (not good for scans, or if you know what you are always querying for already exists they’re a waste).

Usage.
* super granluarity, defaults to off.
* tweak max fold rate for compression if you know how big your rows are.
* property for turning them off at any time.
* you can turn them on on pre-existing rows, and they get added as compactions happen.

See HBASE-1200 for more information.

===HBase Bulk Loads===
* Better than the last one.
* Skips rpc paths.
* 10x faster then api use.
* Writes mapred output directly into hfiles on hdfs.
* They can be loaded into hbase easily. You just have to just tell it where they are.

steps
* Run mapred job.
* bin/hbase completebulkload /output-path/ tablename

* also new importtsv tool.

===Miscellaneous (fluffy stuff)===
Maven now the build system, all the way.
* it’s working.

Logo change.
* no one seems too attached to the bass cleff symbol.

HBASE-50 Facebook summer of code project.
* implementing snapshots (they’ll be a transaction).
* Design plan and implementation is looking so good, Stack says the committers should read it as they might learn something.

===Performance (in progress)===
Reduce IO around splits.
* Currently only triggered after compactions. So you rewrite the data both sides of the split.
* looks at the sum of all storefiles, so compactions don’t have to happen.
* checked after flush not compaction.
* Much faster.
* Avoids 50% of the io during the split.

Reduce time regions go offline.
* Splits, load balancing, region server failover.
* Fixes
** make splits faster
** Double flush memstore of region close. One before the close, one after, so that the flush after close is super fast. Which means reassigning regions wont make a region go offline for many seconds.
** Using zookeeper for more intelligent region movement.

Concurrency and priorities.
* Added multi threading to flushes and compactions.
* Multi threading of master messages.
* Flushes and compactions now have priorities associated with them.

HFile seek/reseek.
* Projections.
** seek to columns you want, not the start of the row.
** seek the the versions you want not the start of the column.

Configurable WAL
* HDFS appends to wal, provides durability.
* Optional deferred log flush that does not block requests, but constantly appends (aka scrobble server). For when 3 seconds data loss in a failure is acceptable.
* You can always disable WAL completely for speed.

Other stuff.
* Internally storing min/max timestamp of each file, for allowing skipping files that don’t over lap.
* Faster enable/disable/drop of tables.

M.I.A, Born Free from ROMAIN-GAVRAS on Vimeo.

I read a lot of people saying it’s definitely NSFW and maybe it had gone too far, etc, etc… I found it a lot less disturbing than the recent video of US soldiers in Afghanistan shooting innocent people from a helicopter and laughing (released on youtube by wikileaks). I’m really surprised there hasn’t been more comparison of it to Afghanistan and Iraq.

There is literally a blanket of white snow outside our window. Snow is weird.

Note to self: Umbrellas don’t help.

Someone should hurry up and fix their scaling problems instead of hanging out on tropical beaches. Yes, you know who you are, Mr inappropriate on irc extraordinaire.

While I’m here.. Last night was a dangerously fun flat warming. Dangerously dangerous! God that was a lot of empty champagne bottles.

I went to a place in Dalston at 4am afterwards, a gal I danced with told her friend “he’s the best dancer I’ve ever seen”. Awww thanks, she should go to Sadler’s wells at least once though. Or Soho

I also launched my pointless kafkaisms tweeter today. Read the metamorphosis one paragraph per day.

I wrote a command line scrobble client the other day. You never know when you’ll need to pipe your scrobbles.

The lease on my current place runs out on the 31st, so I’m looking for rooms again.

I saw Fever Ray at Shepard’s Bush Empire on Thursday. It was amazing.

At Last.fm we’ve been working hard on the next iteration of search (with neat auto-completion) and launched a beta to subscribers on Wednesday.

I’ve been playing with HBase, and loaded a 700 million cell table into a 5 node cluster with no problems at all. I’m also setting up this mini cluster so I can run Dumbo jobs over HBase.

Also of note: Last.fm is hiring.

But, I had pretty high expectations of being here. I thought every day I’d be hanging out with friends drinking beer, drinking coffee, having brunch etc etc etc… For the most part that’s pretty much what I’ve been doing, but I really do get exhausted by people and organizing meeting them. I need to be able to take it for granted that they’ll be there for another day we can hang out. If only everything could just be done on a whim.

I feel like having a midnight picnic on a warm night, but I just know it’ll be only me if I try to round people up at the last minute. Ah well, I refuse to plan it in advance or even think about it. I will whimsically buy wine and drink myself to sleep in a park on my own if I have to.

In some ways it’s great to be working from home now for the last of my stay here. It gives me a something to do other then mope about time and everyones busy schedules. I do like moping though.

It’ll be a good next week and a half.

I think I’m still getting over the jet lag. Today I walked around Shinjuku all day. It made me want to play The World is not Enough again. Unfortunately I left my DS in london.

The first thing I did was go to a mega book store next to Takashimaya Times Square. They have half a floor for foreign language books. Weirdly they’ve pretty good mathematics text book section. I bought an Atlas of Tokyo.

I spent the rest of the day wandering around various record shops (as well as an anarchist book shop) recommended by Tokyo Damage Report. Also found a couple of Punk clubs. I nearly went to one but was tired and too cheap to pay the 1700 yen door charge.