Harish Mallipeddi

http://www.infoq.com/presentations/Facebook-Hive-Hadoop

Thu, 25 Feb 2010 14:08:44 +0530

http://www.infoq.com/presentations/Facebook-Hive-Hadoop:

Informative talk by Ashish Thusoo and Namit Jain from Facebook’s Hive team.

Hive’s RCFile is pretty interesting - it provides record-columnar storage on top of HDFS. Apparently it results in very good compression and higher scan throughput.

http://www.dabeaz.com/GIL/

Mon, 22 Feb 2010 15:31:02 +0530

http://www.dabeaz.com/GIL/:

David Beazley continues his GIL investigation work from his ChiPy ‘09 talk. This time he also analyses the new GIL optimizations that are apparently present in the Python 3.2 svn branch. Look at his slides to see how the new GIL performs!

It took surprisingly little amount of code changes to the Python interpreter to do this kind of investigative work (Read more about his code changes). He maintains a bunch of counters for each time t, and then logs them at the end to a file. He then renders this data in a timeline for better understanding - the output of this step is some humungous PNG file. So then he splits the PNG image, tiles it up, and then uses a Google Maps kinda interface to view the tiles, zoom in/out etc.

"The spot price of MicroSD cards is nearly identical to the spot price of the very same NAND FLASH..."

Fri, 19 Feb 2010 00:30:32 +0530

“The spot price of MicroSD cards is nearly identical to the spot price of the very same NAND FLASH chips used on the inside. In other words, the extra controller IC inside the microSD card is sold to you “for free”. Incorporating the controller into the package and having it test, manage and mark bad blocks more than offsets the cost of testing each memory chip individually. A full bad block scan can take a long time on a large FLASH IC, and chip testers cost millions of dollars. Therefore, the amortized cost per chip for test alone can be comparable to the cost of silicon itself.”

- On MicroSD Problems « bunnie’s blog

Fascinating article on the MicroSD industry and the economics involved in manufacturing them by a Chumby engineer. He has a whole series of articles about hardware manufacturing and China.

http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/

Thu, 18 Feb 2010 12:41:00 +0530

http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/:

Nice article providing a quick intro to RabbitMQ/AMQP terminology.

Summary:

Virtual hosts - one RabbitMQ server can host multiple virtual hosts (each virtual host maps to one user process). One virtual host can contain multiple exchanges.
Queues - queues hold messages which consumers can consume.
Routing keys - each message has a routing key.
Exchanges - producers drop off messages at exchanges. Queues are registered with the exchanges.
Bindings - bindings are routes configured on the exchange to let the exchange know how to route messages with routing keys to queues (a message can be routed to multiple queues aka one routing key can match multiple bindings/routes).

"REPLACE INTO Tickets64 (stub) VALUES (‘a’); SELECT LAST_INSERT_ID();"

Wed, 10 Feb 2010 00:01:41 +0530

“REPLACE INTO Tickets64 (stub) VALUES (‘a’); SELECT LAST_INSERT_ID();”

- Ticket Servers: Distributed Unique Primary Keys on the Cheap - Laughing Meme

Kellan describes the hack they use at Flickr to (atomically) generate globally unique auto-increment ids across shards. Redis would be great for something like this.

http://www.techcrunch.com/2010/01/14/next-jump/

Sat, 16 Jan 2010 12:23:35 +0530

http://www.techcrunch.com/2010/01/14/next-jump/:

Interesting read on NextJump’s business model. Amidst an endless list of me-too comparison shopping sites, this one has a completely different approach. Because of the way their business works, they got their hands on 3 valuable data sources:

consumer demographic database - they’re considered a non-traditional benefits provider and hence they get access to the employment status of 30 million workers (who also happen to be consumers). It gets part of the employee record and sometimes even job title or salary grade.
transactional data - from merchants, credit card companies
consumer preference data - because it is seen as an employee perk, HR departments inside companies and the individuals themselves are willing to engage in a level of preference data sharing that has not been seen in e-commerce before. The customer preference data allows for better targeting and ultimately superior conversions (around 10-11% vs 2% for the best commerce sites today)

http://maglevity.wordpress.com/2009/12/17/kd-trees-and-maglev/

Sat, 16 Jan 2010 11:47:04 +0530

http://maglevity.wordpress.com/2009/12/17/kd-trees-and-maglev/:

Interesting post on building a KD-tree in MagLev to answer nearest neighbor searches.

Richard Hamming's "You and your research"

Sun, 03 Jan 2010 23:57:27 +0530

Richard Hamming's "You and your research":

Richard Hamming (of Bell Labs) gave a talk in 1986 on how to do great research as an individual. I particularly liked one of his comments during Q&A about reading - reading is important but read only as much as is required to understand the problems in the field but not to find the solutions.

Hard drives with 4K sectors instead of current 512B sectors

Sun, 20 Dec 2009 23:03:05 +0530

Hard drives with 4K sectors instead of current 512B sectors:

AnandTech explains why Western Digital is manufacturing hard drives with larger sectors (4K) instead of the current standard 512B sectors.

Last.fm uses MogileFS with SSDs

Tue, 15 Dec 2009 16:22:00 +0530

Last.fm uses MogileFS with SSDs:

Summary:

They used SSDs to store hot songs.
They picked the simple FIFO noop scheduler in the Linux I/O subsystem. For more info on the different schedulers, read Redhat’s IO Tuning Guide. They also set read_ahead_kb to a very low value since seek times are so low for SSDs anyway.
Modified MogileFS to differentiate between storage nodes with SSDs and nodes with regular HDDs.
1 SSD => 7000 requests vs 1 7200rpm SATA => 300 requests.

http://www.sriramkrishnan.com/blog/2009/12/stuff-ive-learned-at-microsoft.html

Sun, 13 Dec 2009 17:26:08 +0530

http://www.sriramkrishnan.com/blog/2009/12/stuff-ive-learned-at-microsoft.html:

Do’s and don’ts for programmers (not the technical stuff but the soft skills, behavior, attitude at work) by Sriram who works for Microsoft’s Azure team. I’ve already experienced and observed several things from his list since I started working for this other big software company.

How node.js exposes traditional blocking I/O calls in a non-blocking manner

Fri, 04 Dec 2009 00:20:45 +0530

If you’ve not heard about node.js, do watch the video from a talk given by Ryan Dahl (project lead) at JSConfEU 2009.

The thing which I found most interesting is how he handled the shortcomings of libraries like eventmachine. eventmachine is great for building highly-scalable single-threaded network servers using async network I/O. But while writing your network servers, you’ve to be careful not to introduce some blocking calls in your event callbacks. Since your server is single-threaded, if you block inside the event callback then you’re basically not handling any of the other incoming requests during the time that the thread was blocked. Why would you need to do blocking calls? Your MySQL client library will typically do blocking I/O. POSIX filesystem calls are blocking. There are many other such instances where the usefulness of eventmachine is reduced. So in node.js, Ryan has decided to wrap away these blocking APIs and expose them in Javascript in a non-blocking manner - basically a batteries-included arsenal of non-blocking APIs in one package if you will. Take a look at the node.js API docs to get an idea of the kinds of things that are supported.

The way node.js works underneath is that it offloads all the blocking calls to a thread-pool and when the calls finish, the results are fetched by reading off a pipe to which the threads write to. Incidentally pipes are select()-able which means these events can be handled within the main event loop of node.js in exactly the same manner as the other network socket events. I quickly implemented a barebones proof-of-concept of this idea in Python:

Note - if you’re reading this via a RSS reader, open this in a browser to see the gist code embed above.

Tumblr backup tool in 40 lines of Clojure

Tue, 01 Dec 2009 18:04:05 +0530

The code doesn’t necessarily do a lot of error handling.

Note - if you’re reading this via a RSS reader, this blog post contains a Gist embed; open the page in your browser.

Last.fm interview with CNET

Tue, 01 Dec 2009 15:26:14 +0530

Last.fm interview with CNET:

What I found interesting - they use SSDs for serving out popular radio streams and regular HDDs for the rest of their streams. Streaming is done via regular Linux boxes running MogileFS.

sniffer - Erlang NIF example

Thu, 26 Nov 2009 15:21:53 +0530

sniffer - Erlang NIF example:

Uses NIFs in otp-R13B03 to grab a binary at a specific memory location.

Riak's dets backend - too many files open error

Fri, 20 Nov 2009 17:31:50 +0530

I was playing with Riak yesterday and ran into this error on my Macbook running Snow Leopard:


{
 {badmatch,{error,{{badmatch,{error,
 {file_error,"./dets-store/1392993748081016843912887106182707253109560705024",emfile}}},
                    [   {riak_vnode,init,1},
                        {gen_server2,init_it,6},
                        {proc_lib,init_p_do_apply,3}    ]}}
 },
 [{riak_vnode_master,get_vnode,2},
  {riak_vnode_master,handle_cast,2},
  {gen_server,handle_msg,5},
  {proc_lib,init_p_do_apply,3}]
}

When using the dets storage backend, Riak seems to create/open a dets database file for each vnode (partition) in your ring. When there’s only one node in your cluster, I’m guessing all the vnodes/partitions are owned by this node which results in a whole bunch of files being opened (under the dets-store folder you configured). In Snow Leopard, I’d to do ulimit -n 8192 to increase the limit on the no. of fds a process could take. You probably won’t notice this normally - I increased the partition size in the config file and hence ran into this problem.

When Linux Runs Out of Memory - O'Reilly Media

Tue, 03 Nov 2009 14:57:44 +0530

When Linux Runs Out of Memory - O'Reilly Media:

Good article explaining Linux’s default overcommit-ing behavior (/proc/sys/vm/overcommit_memory) and the OOM killer.

Hadoop at Yahoo!

Fri, 30 Oct 2009 11:33:34 +0530

Hadoop at Yahoo!:

Slides from Eric14’s talk at UIUC.

GitHub's new architecture

Thu, 22 Oct 2009 02:10:31 +0530

GitHub's new architecture:

Very well written article highlighting the new architecture of GitHub (post-Rackspace move). Every company/team should at least have a similar internal document describing their current architecture at a high-level but unfortunately doesn’t happen in most companies.

GitHub seems to be using a lot of familiar Opensource projects in their stack - Nginx, Ruby/Rails/Unicorn, HAProxy, MySQL, memcached, redis, bunch of RPC services speaking the new BERT/BERT-RPC protocol. The only thing I’ve not really played with yet is DRBD and I’ve heard lots of good things about DRBD from others.

So looks like now you’ve Facebook’s Thrift, Google’s Protocol Buffers, Hadoop/Doug Cutting’s Avro, just plain JSON/XML and GitHub’s BERT for your (de)serialization needs!

Speaking at FOSS.MY 2009

Tue, 20 Oct 2009 01:45:25 +0530

I’ll be speaking about Hadoop at FOSS.MY 2009 in KL. They’ve an interesting bunch of speakers this year including Brain Aker, David Axmark, RMS. This will be my first time to FOSS.MY!

If you’re attending, do drop by my talk on Sunday afternoon :)