How many bugs does Google have?

After I uploaded photos to my Google drive from my iphone today, they are all corrupted. Only a few were uploaded correctly. Google service degraded so much in the recent years. It really sucks now.

I am so glad I found out this early so I didn't lost many data.

How to estimate income level / ethnic ratio from your traffic

Use this API to get demographic data:

http://www.census.gov/data/developers/data-sets.html

For each visit to your site, convert ip to zip code by maxmind; use zip code to retireve income level / ethnic ratios by the census API. Sum the ratios by proportion - the result is the estimated income level / ethnic of your traffic (assuming people in a zip visit your site at random).

JavaScript supports functional programming!

I have been using JavaScript for quite long, believing it is an old language. Today I just went to across how to define my own map reduce function in Lua, and came into my thought: if I could do this in Lua, then it is doable in JavaScript... just to discover that JavaScript has supported functional programming for long!

Open your browser console with F12:

[1,2,3].map(function(a) {return a*a})     // shows [1, 4, 9]
[1,2,3].reduce(function(a,b) {return a+b})      // shows 6
[1,2,3].filter(function(a) {return a>2})      // shows [3]

Flatten does not come out of the box, it can be added by one line:

Array.prototype.flatten = function() { return this.reduce(function(a,b) { return a.concat(b)})}

[[0,1],[2,3],[4,5,6]].flatten()      // shows [0,1,2,3,4,5,6]


It made my day! :D

JavaScript is the best language invented ever!

ps: not supported by IE 8, which covers 8% of the Internet  users.. ouch!

Dealing with HHVM 3.1.0 memory leak

A week ago, I switched my site onto HHVM (3.1.0), in replace of PHP5. The throughput increase was significant, and response time dropped from 400ms to 300ms observed from Pingdom. I was very happy with the result.

After 5 days, I got a down-time on my site.. The reason being HHVM was shut down on itself. As it is serving through cgi port 9000. Nginx cannot access it. The solution was rather simple:

      sudo /etc/init.d/hhvm restart

However, I noticed my php-fpm setting was through socket rather than port. After some reading, cgi using socket is a lot more efficient than through port, as using port involving resolving through the network. So in order for nginx to read from hhvm socket, I opened up  /etc/nginx/hhvm.conf, replace the fastcgi_pass with:

    fastcgi_pass   unix:/var/run/hhvm/hhvm.sock;

I need to tell HHVM to serve from the socket, too. So, in /etc/hhvm/server.ini, I commented the server port and added file_socket:

    hhvm.server.file_socket=/var/run/hhvm/hhvm.sock
    ;hhvm.server.port = 9000

Restarting both nginx and hhvm, now it runs peacefully, again. no significant response time improvement, but hoping it fixes the issue of shutting down itself.

5 days after, the response time of site started to catch up.. :(




I ssh-ed into the node. Well, I found multiple processes of HHVM eating up the memory - that's not good. It is a memory leak. That's why it crashed a few days ago. and it only filled up the memory when serving through socket.

Keeping reading up HHVM, people had similar issues: Laravel + HHVM will cause memory leak: https://github.com/laravel/framework/issues/4757
It became my problem as my site is still on Laravel 3.

I guess I could switch back to PHP5. but I love the performance gained from HHVM and lambda expression, and would love to have it running around the bug.

I ran httperf against my site home page on localhost, and htop observing the hhvm processes, which consists a couple of mysql query and blade template render.  The memory grew fast with the requests. To rule out it is not the problem of db connection (or not just the problem of db): I ran httperf against another page that only have a static blade template like the following:

    public function action_about()
    {
        return View::make('home.about');
    }

And the memory consumption of hhvm grew just as fast. Third test with hello world: <?php echo 'hello' ?> it didn't cause hhvm any memory leak. 

Quite possibly it was caused by blade template engine running into some corner of hhvm. I didn't go deep - will leave the part for the future.

So I'm guessing hhvm did not limit on the max threads to be created, or the connections didn't timeout accumulated in the memory, or a background process should periodically expire the cache, or maybe there should need to be a memory cap defined, etc.. Have to try out and experiment with it. There wasn't much documentation, I could only found these information about server.ini settings:

https://github.com/facebook/hhvm/wiki/INI-Settings
        https://github.com/facebook/hhvm/blob/1cd8ceb06adf9a40039ebac40c489abb811ab599/hphp/runtime/base/runtime-option.cpp#L761

Tweaked more of the parameters, none of them stopped the memory accumulation! :( Here are the settings that I tried:

hvm.resource_limit.max_rss = 268435456
hhvm.resource_limit.drop_cache_cycle = 3600
hhvm.resource_limit.max_rsspolling_cycle = 3600
;hhvm.resource_limit.max_socket = 10
hhvm.resource_limit.socket_default_timeout = 30
hhvm.server.thread_count = 3
hhvm.pagelet_server.thread_drop_cache_timeout_seconds = 60


Finally, I found people talking about JIT is the cause of the memory leak, and the parameter trying to turn off JIT seems to work:

hhvm.jit = false

Now memory with hhvm processes stopped growing with httperf adding load to it. The service usable but I lost some performance - still beat PHP5. The problem is still unsolved. The remaining issue is: why does blade template engine cause JIT to memory leak?

PHP has lambda function! :D

I just discovered that you can perform lambda function with PHP tonight! Here's how to do it:

# First, go to php console  (you will need to install HipHop)

php -a


Welcome to HipHop Debugger!
Type "help" or "?" for a complete list of commands.

Note: no server specified, debugging local scripts only.
If you want to connect to a server, launch with "-h" or use:
  [m]achine [c]onnect <servername>

hphpd> 

# mapper function in PHP is called "array_map", it takes in a function and an array, just like Python:
$result = array_map( $a==> $a+1, [1,2,3,4])

$a ==> $a+1    is a lambda function, where $a is each element in the array. The function simply takes the value and plus one. the $result is expected to have [2,3,4,5]

$result will be another array of numbers.

To get an array of values from the $result variable, we need to do something similar to .to_a method in Ruby:

var_dump($result)

array(4) {
  [0]=>
  int(2)
  [1]=>
  int(3)
  [2]=>
  int(4)
  [3]=>
  int(5)

}

The result is as expected.

Power up my PHP site with Hip-hop!

Finally I made up my mind, and decide to try out Hiphop as a replacement for PHP5.

The installation was very straight-forward. I installed following the tutorial:

 http://fideloper.com/hhvm-nginx-laravel#comment-1325788095

In short:

$ sudo add-apt-repository -y ppa:mapnik/boost
$ wget -O - http://dl.hhvm.com/conf/hhvm.gpg.key | sudo apt-key add -
$ echo deb http://dl.hhvm.com/ubuntu precise main | sudo tee /etc/apt/sources.list.d/hhvm.list
$ sudo apt-get update
$ sudo apt-get install -y hhvm

$ sudo update-rc.d hhvm defaults

# Restart the service now
$ sudo service hhvm restart                      # We'll also restart HHVM

and inside my nginx config: /etc/nginx/sites-available/default, added one line "include hhvm.conf"

... Stuff above this omitted ...
    # Make site accessible from http://localhost/
    server_name localhost;
    include hhvm.conf;  # HERE'S THE MAGIC

    location / {
... Stuff below this omitted ...

Restarted now my site is running on Hip-hop! The performance improvement is significant. The requests per second increased about 5 times without any logic update! :D

(Some remaining issue: as my site is running on Laravel 3, I had to adjust some of the library to make it. Mostly 2 parts: Laravel 3 defined "yield" as a function for blade template engine, which is a reserved word in Hip-Hop. And Hip-hop came with its own Redis client, having a name clash with the Redis client from Laravel 3. If you ran into similar trouble, here is a diff patch to apply to make it compatible:

https://gist.github.com/yuhanz/afc6189330ce0c6120a8

If you have an old PHP site and are lazy about rewriting it, it is a good option to go with Hip-hop. They supports various PHP frameworks:

http://hhvm.com/frameworks/

Google search reached error code 500! :D

It is quite uncommon to see this. Might not be able to see it in another 100 years. So I made a screenshot! :D

WeChat vs. Google Hangout

I was chatting with my aunt from Los Angeles to Beijing over WeChat. It worked fines. The voice was sometime not very clear. The picture frame rate was low. But in general the service worked.

When switching to talk to her over Google Hangout, we experienced a very frequent drop of connections. Once the connection was established, I had to shut off video, in order to hear her voice at all.

It is quite interesting how the Chinese Company Tecent creates WeChat that enables the oversea video calls between nations, while Google service is only excelling local to America.

  

Tips on using Scala interactive mode

I enjoy interactive scripting tool like irb, but scala doesn't go very friendly as ruby. As I have to specify the class path in order to import my own packages. :/

So, I wrote a shell script automating it. As I always put all jar's in a lib/ folder, this script will get all the .jar files and add them into the classpath:

LIB_PATH=`find lib | tr "\n" :`                                                 

scala -cp .:$LIB_PATH

Save this into scala.sh, and run it at the directory where you have the lib/ folder. scala

Unfortunately, Java class names are not as simple and short as ruby. My memory is so bad that I cannot remember the package name of every single class. So, I create this shell script to help myself:

find lib -name \*.jar -exec unzip -v {} \; | grep -oh --color=never '[^[:space:]]*class$' | tr / . | sed -E 's/^(.*).class$/import \1/' > /tmp/imports.scala


Run this from where you lib/ folder is, and it will find all the classes with package names, and store into /tmp/imports.scala. Then in the interactive mode, you can search in this file and copy and paste the import class from a text editor. :D

It is always helpful to script common things in a text editor, such as initialize hbase connection, etc. Load them into scala by typing in the terminal so that we are not doing the boring thing again:
:load hbase_init_script.scala

The snippets are here on Gist:
https://gist.github.com/yuhanz/02155cfa921d6ef17014
https://gist.github.com/yuhanz/8002c696fbac9dd35ff2