Thursday, February 19, 2009

What is ffp?

In my previous entry, I've posted very messed-up procedure to install ffp(f2p).
But, don't blame ffp. Ffp(http://github.com/nahi/f2p/tree/master) itself is very smart and simple(thin-layered) application. I thank NaHi, author of this product. He said "ffp is for mobile-FriendFeed users".

FriendFeed is one of the best tools for re-blogging(twitting) with powerfull aggregation.
Ffp is used for the basis of UI-customization.

Now ffp offers you
  • simple/slim UI for "traditional-mobile phone web-browser" (although iPhone has its special UI from FriendFeed)
  • grouping messages/entries from various feed-sources into comment-tree structure
  • "recent-updated" view
  • re-share(cite/quote) messages from friend feed and manage it in your feed (So, your another friend may refer interesting post of friend-of-friend. Note: privacy matter might be cleared)
Still ffp seems to be in enhancement process. Feel free to drop in f2p room at FriendFeed.
http://friendfeed.com/rooms/f2p

Monday, February 16, 2009

how to setup your ffp

ffp takes you out from your desk with FriendFeed.



source: http://github.com/nahi/ffp/tree/master



In case of CentOS5.2 default installation, rubygems is not availabe.

So, installation starts at rubygem, but it requires rdoc... anyway, let's go further.

Setup framework



install rdoc

    # yum install rdoc

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 ruby-rdoc               i386       1.8.5-5.el5_2.6  updates           136 k
Updating:
 ruby                    i386       1.8.5-5.el5_2.6  updates           280 k
 ruby-libs               i386       1.8.5-5.el5_2.6  updates           1.6 M
Installing for dependencies:
 ruby-irb                i386       1.8.5-5.el5_2.6  updates            69 k
Updating for dependencies:
 ruby-devel              i386       1.8.5-5.el5_2.6  updates           555 k







install gem

    # wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz
    # tar zxf rubygems-1.3.1.tgz
    # cd rubygems-1.3.1
    # ruby setup.rb





install rails



    # gem install rails



Successfully installed rake-0.8.3

Successfully installed activesupport-2.2.2

Successfully installed activerecord-2.2.2

Successfully installed actionpack-2.2.2

Successfully installed actionmailer-2.2.2

Successfully installed activeresource-2.2.2

Successfully installed rails-2.2.2

7 gems installed

Installing ri documentation for rake-0.8.3...

Installing ri documentation for activesupport-2.2.2...

Installing ri documentation for activerecord-2.2.2...

Installing ri documentation for actionpack-2.2.2...

Installing ri documentation for actionmailer-2.2.2...

Installing ri documentation for activeresource-2.2.2...

Installing RDoc documentation for rake-0.8.3...

Installing RDoc documentation for activesupport-2.2.2...

Installing RDoc documentation for activerecord-2.2.2...

Installing RDoc documentation for actionpack-2.2.2...

Installing RDoc documentation for actionmailer-2.2.2...

Installing RDoc documentation for activeresource-2.2.2...



test rails



$ mkdir ffp
$ rails /home/ocao/ffp
      exists
      create  app/controllers
      create  app/helpers
      create  app/models
      create  app/views/layouts
      create  config/environments
      create  config/initializers
      create  config/locales
      create  db
      create  doc
      create  lib
      create  lib/tasks
      create  log
      create  public/images
      create  public/javascripts
      create  public/stylesheets
      create  script/performance
      create  script/process
      create  test/fixtures
      create  test/functional
      create  test/integration
      create  test/performance
      create  test/unit
      create  vendor
      create  vendor/plugins
      create  tmp/sessions
      create  tmp/sockets
      create  tmp/cache
      create  tmp/pids
      create  Rakefile
      create  README
      create  app/controllers/application.rb
      create  app/helpers/application_helper.rb
      create  test/test_helper.rb
      create  test/performance/browsing_test.rb
      create  config/database.yml
      create  config/routes.rb
      create  config/initializers/inflections.rb
      create  config/initializers/mime_types.rb
      create  config/initializers/new_rails_defaults.rb
      create  config/locales/en.yml
      create  config/boot.rb
      create  config/environment.rb
      create  config/environments/production.rb
      create  config/environments/development.rb
      create  config/environments/test.rb
      create  script/about
      create  script/console
      create  script/dbconsole
      create  script/destroy
      create  script/generate
      create  script/performance/benchmarker
      create  script/performance/profiler
      create  script/performance/request
      create  script/process/reaper
      create  script/process/spawner
      create  script/process/inspector
      create  script/runner
      create  script/server
      create  script/plugin
      create  public/dispatch.rb
      create  public/dispatch.cgi
      create  public/dispatch.fcgi
      create  public/404.html
      create  public/422.html
      create  public/500.html
      create  public/index.html
      create  public/favicon.ico
      create  public/robots.txt
      create  public/images/rails.png
      create  public/javascripts/prototype.js
      create  public/javascripts/effects.js
      create  public/javascripts/dragdrop.js
      create  public/javascripts/controls.js
      create  public/javascripts/application.js
      create  doc/README_FOR_APP
      create  log/server.log
      create  log/production.log
      create  log/development.log
      create  log/test.log



Penetrate your firewall (port 3000 default)

in case of CentOS, iptables setup changed



# iptables -A INPUT -p tcp --dport 3000 -j ACCEPT


(if required) reverse proxy server setup

If you feel port 3000 specific-URL is ugly, you can setup httpd (apache) to reverse-proxy as below.



    ProxyPass /ffp/     http://yourserver:3000/

    ProxyPassReverse /ffp/      http://yourserver:3000/





kick start server(Webrick)



$ cd ffp

$ script/server

=> Booting WEBrick...

=> Rails 2.2.2 application started on http://0.0.0.0:3000

=> Ctrl-C to shutdown server; call with --help for options

[2009-02-16 09:30:40] INFO  WEBrick 1.3.1

[2009-02-16 09:30:40] INFO  ruby 1.8.5 (2006-08-25) [i386-linux]

[2009-02-16 09:30:40] INFO  WEBrick::HTTPServer#start: pid=16622 port=3000

192.168.0.51 - - [16/Feb/2009:09:30:48 JST] "GET / HTTP/1.0" 200 7385

- -> /

192.168.0.51 - - [16/Feb/2009:09:33:45 JST] "GET / HTTP/1.1" 200 7385

- -> /

192.168.0.51 - - [16/Feb/2009:09:33:45 JST] "GET /javascripts/prototype.js HTTP/1.1" 200 129738

http://b137053.ppp.asahi-net.or.jp/ffp/ -> /javascripts/prototype.js

192.168.0.51 - - [16/Feb/2009:09:33:46 JST] "GET /javascripts/effects.js HTTP/1.1" 200 38675

http://b137053.ppp.asahi-net.or.jp/ffp/ -> /javascripts/effects.js

192.168.0.51 - - [16/Feb/2009:09:33:46 JST] "GET /images/rails.png HTTP/1.1" 200 6646

http://b137053.ppp.asahi-net.or.jp/ffp/ -> /images/rails.png

192.168.0.51 - - [16/Feb/2009:09:34:02 JST] "GET /rails/info/properties HTTP/1.1" 500 13415

http://b137053.ppp.asahi-net.or.jp/ffp/ -> /rails/info/properties




Welcome ABORT!!!

you'll see the message below.



MissingSourceFile

in Rails/infoController#properties



here is the framework trace.

/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'

/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'

/usr/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:in `require'

/usr/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:521:in `new_constants_in'

(snip!)
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'

/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'

script/server:3





Thus,





no such file to load -- sqlite3

This error occurred while loading the following files:

sqlite3



but, we need SWIG before sqlite3.

The reason why SWIG is required, see http://www.machu.jp/diary/20070117.html



install SWIG



download from http://www.swig.org/download.html



# tar zxf swig-1.3.38.tar.gz

# cd swig-1.3.38

# ./configure && make && make install



install sqlite

# gem install sqlite3-ruby

Building native extensions.  This could take a while...

Successfully installed sqlite3-ruby-1.2.4

1 gem installed

Installing ri documentation for sqlite3-ruby-1.2.4...

Installing RDoc documentation for sqlite3-ruby-1.2.4...



Now your Environment seems OK

About your application’s environment


Ruby version1.8.5 (i386-linux)
RubyGems version1.3.1
Rails version2.2.2
Active Record version2.2.2
Action Pack version2.2.2
Active Resource version2.2.2
Action Mailer version2.2.2
Active Support version2.2.2
Application root/home/yourname/ffp
Environmentdevelopment
Database adaptersqlite3
Database schema version0


Setup f2p



download github code


$ cd working_directory

$ git clone git://github.com/nahi/f2p.git



initial boot to confirm setup



$ script/server

=> Booting WEBrick...

Missing these required gems:

  httpclient

  json



You're running:

  ruby 1.8.5 at /usr/bin/ruby

  rubygems 1.3.1 at /home/ocao/.gem/ruby/1.8, /usr/lib/ruby/gems/1.8



Run `rake gems:install` to install the missing gems.



install missing gems

oh, boy! where is httpclient?



$ rake gems:install

(in /home/username/f2p)

** Invoke gems:install (first_time)

** Invoke gems:base (first_time)

** Execute gems:base

** Invoke environment (first_time)

** Execute environment

rake aborted!

no such file to load -- httpclient





after all, I need to install following 2 packages (httpclient, json)

get httpclient



at http://dev.ctor.org/http-access2



# wget http://dev.ctor.org/download/httpclient-2.1.4.tar.gz

# tar zxf httpclient-2.1.4.tar.gz

# cd httpclient-2.1.4

# ruby install.rb



but, still rake is making noise on "json"

rake gems:install

(in /home/username/f2p)

rake aborted!

no such file to load -- json



(See full trace by running task with --trace)





get json

its simple.



# gem install json



Kick off again

it seems ok,



$ script/server

=> Booting WEBrick...

=> Rails 2.2.2 application started on http://0.0.0.0:3000

=> Ctrl-C to shutdown server; call with --help for options

[2009-02-16 10:53:05] INFO  WEBrick 1.3.1

[2009-02-16 10:53:05] INFO  ruby 1.8.5 (2006-08-25) [i386-linux]

[2009-02-16 10:53:05] INFO  WEBrick::HTTPServer#start: pid=25199 port=3000

192.168.0.51 - - [16/Feb/2009:10:53:15 JST] "GET / HTTP/1.1" 302 110

- -> /

192.168.0.51 - - [16/Feb/2009:10:53:16 JST] "GET /login HTTP/1.1" 200 863

- -> /login

192.168.0.51 - - [16/Feb/2009:10:54:02 JST] "POST /login/authenticate HTTP/1.1" 500 13753

http://b137053.ppp.asahi-net.or.jp/f2p/login -> /login/authenticate



but I came across login-failure




messages are below.



ActiveRecord::StatementInvalid

in LoginController#authenticate


Could not find table 'users'


/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:29:in `table_structure'

/usr/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/core_ext/object/misc.rb:39:in `returning'

/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:28:in `table_structure'

/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite_adapter.rb:189:in `columns'

/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:1220:in `columns'

/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:2839:in `attributes_from_column_definition_without_lock'

/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/locking/optimistic.rb:55:in `attributes_from_column_definition'

/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:2279:in `initialize'

app/models/user.rb:62:in `initialize'

app/controllers/login_controller.rb:26:in `new'

app/controllers/login_controller.rb:26:in `authenticate'




/usr/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:29:in `table_structure'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/core_ext/object/misc.rb:39:in `returning'
(snip!)

/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
script/server:3

Request


Parameters:

{"name"=>"user",

"commit"=>"login",

"authenticity_token"=>"xxxxxx",

"remote_key"=>"xxxxxxx"}



--- 
:csrf_id: 9xxxxxxxxxxbf
flash: !map:ActionController::Flash::FlashHash {}




Response


Headers:

{"cookie"=>[],

"Content-Type"=>"",

"Cache-Control"=>"no-cache"}


migrate db for public_environment?



reference: http://d.hatena.ne.jp/ky2009/20081226/1230278255



$ rake RAILS_ENV=production db:migrate

(in /home/ocao/f2p)

==  CreateUsers: migrating ====================================================

-- create_table(:users)

   -> 0.0057s

==  CreateUsers: migrated (0.0062s) ===========================================



==  CreateProfiles: migrating =================================================

-- create_table(:profiles)

   -> 0.0056s

==  CreateProfiles: migrated (0.0060s) ========================================



==  AddUserToProfile: migrating ===============================================

-- add_column(:profiles, :user_id, :integer)

   -> 0.0104s

==  AddUserToProfile: migrated (0.0108s) ======================================



==  InsertProfileOfUser: migrating ============================================

==  InsertProfileOfUser: migrated (0.0010s) ===================================



==  AddEntriesInThreadToProfile: migrating ====================================

-- add_column(:profiles, :entries_in_thread, :integer)

   -> 0.0097s

==  AddEntriesInThreadToProfile: migrated (0.0113s) ===========================



==  CreateLastModifieds: migrating ============================================

-- create_table(:last_modifieds)

   -> 0.0058s

==  CreateLastModifieds: migrated (0.0062s) ===================================



==  CreateCheckedModifieds: migrating =========================================

-- create_table(:checked_modifieds)

   -> 0.0057s

==  CreateCheckedModifieds: migrated (0.0061s) ================================







but, still I have the same error. maybe I'm using development environment, right?



$$ ls -l db

total 36

-rw-r--r-- 1 ocao ocao     0 Feb 16 10:49 development.sqlite3 <<< nothing!!!!

drwxrwxr-x 2 ocao ocao  4096 Feb 16 06:33 migrate

-rw-r--r-- 1 ocao ocao 10240 Feb 16 11:01 production.sqlite3

-rw-rw-r-- 1 ocao ocao  1696 Feb 16 11:01 schema.rb

$ cd ..

$ rake RAILS_ENV=development db:migrate



BINGO!!! now the application starts correctly.


Sunday, February 15, 2009

WSSE on Hatena by Erlang

I'm a user of HATENA-bookmark web service. It's a major social-bookmark service in Japan.
The API of the service is in public. So, I've written some code in Erlang.

To initiate operation, WSSE certification is required.
For details of WSSE specification, please refer
 http://www.ibm.com/developerworks/webservices/library/specification/ws-secure/
For details of HATENA-bookmark API, please refer
 http://d.hatena.ne.jp/keyword/%a4%cf%a4%c6%a4%ca%a5%d5%a5%a9%a5%c8%a5%e9%a5%a4%a5%d5AtomAPI?kid=88110#wsse (in Japanese)

Following is the wsse module.


-module(wsse).
-export([new/2]).

-define(SHA1DIGESTLENGTH, 20).

new (User, Password) ->
    {A, B, C} = now(),
    random:seed(A, B, C),
    Nonce = nonce(?SHA1DIGESTLENGTH, ""),

%% create ISO 8601 compling datetime
%% $ ruby -e " require 'open-uri' ; p Time.now.iso8601"
%% => "2008-07-31T16:16:14+09:00"
%%
%%  ref) http://www.trapexit.org/Converting_Between_struct:time_and_ISO8601_Format

    {{Year, Month, Day}, {Hour, Min, Sec}} = erlang:localtime(),
    TZ = os:cmd("date +%:z"),
    Created = io_lib:format("~4.10.0B-~2.10.0B-~2.10.0BT~2.10.0B:~2.10.0B:~2.10.0B~6s",
[Year, Month, Day, Hour, Min, Sec, TZ]),
    crypto:start(),
    Digest = binary_to_list(crypto:sha(Nonce ++ Created ++ Password)),

    "UsernameToken Username=\"" ++ User ++ "\", " ++
    "PasswordDigest=\"" ++
    base64:encode_to_string(Digest) ++ "\", " ++
    "Nonce=\"" ++
    base64:encode_to_string(Nonce) ++ "\", " ++
    "Created=\"" ++ Created ++ "\"".

nonce(0,L) -> L ++ [random:uniform(255)];
nonce(N,L) -> nonce(N -1, L ++ [random:uniform(255)]).


The HATENA-bookmark service API is using Atom format data.
Here is the module for it.
 -module(getatom).
-export([new/3]).

new(User, Password, Uri) ->
RequestHeader = [{ "X-WSSE" , wsse:new(User, Password)}],

inets:start(),
{ok, {{_Version, 200, _ReasonPhrase}, _Headers, Body}} =
http:request(get, {Uri, RequestHeader}, [ ], [ ]),
Body.


Here is a typical useage from erlang interpreter command line.

> Latest = getatom:new("username", "password", "http://b.hatena.ne.jp/atom/feed").
[60,63,120,109,108,32,118,101,114,115,105,111,110,61,34,49,
46,48,34,32,101,110,99,111,100,105,110,103,61|...]
> io:format("~s",[Latest]).

<feed version="0.3"</pp>
xmlns="http://purl.org/atom/ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/"
xml:lang="ja">
<title>kgbu\343\201\256\343\203\226\343\203\203\343\202\257\343\203\236\343\203\274\343\202\257</title>
<link rel="alternate" type="text/html" href="http://b.hatena.ne.jp/kgbu/" />
...to be continued

Thursday, February 12, 2009

Web service checker by Erlang

web assertion checker by Erlang

major functionalities

Here is my initial idea.


Multi-process test-driver : Erlang's multi-processing feature fits
        Root process checks test-completion and report summary
        page-transition => inter-process communication via messages (process transition table is given by root process)

Minimum wait for Rate limit : to save tested site, 5second minimum wait is inserted.

State transition via : request path / cookie / Authentication (Basic|Digest)
       for Identity / Secret(password, authorized keys) / Session
    passed by Messages

Assertion : any AND, OR combination of followings
    equal((text|HTML docs|XHTML docs|XML docs|), state)
    DOM-equal((XHTML docs|XML docs|), state)
    Partial-DOM-equal((XHTML docs|XML docs|), DOM-mask, state)
    Response-Header-equal(Header, state)
    Partial-Response-Header-equal(Header, Header-mask, state)
    Location-equal(URL, state)
                
               

Thursday, February 5, 2009

use vmstat command to know about disk access

I love to use vmstat command to measure load of servers. But, to check disk-intensive overload, usual vmstat output is not device-specific.

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 25924 11324 3524 135644 0 0 3 20 1 4 0 0 100 0 0
0 1 25924 11324 3528 135640 0 0 0 6 19 33 0 0 100 0 0
0 0 25924 11324 3532 135644 0 0 0 1 18 34 0 0 100 0 0


So, I looked into /proc/diskstats , but it is cryptic.

$ cat /proc/diskstats

1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
1 2 ram2 0 0 0 0 0 0 0 0 0 0 0
1 3 ram3 0 0 0 0 0 0 0 0 0 0 0
1 4 ram4 0 0 0 0 0 0 0 0 0 0 0
1 5 ram5 0 0 0 0 0 0 0 0 0 0 0
1 6 ram6 0 0 0 0 0 0 0 0 0 0 0
1 7 ram7 0 0 0 0 0 0 0 0 0 0 0
1 8 ram8 0 0 0 0 0 0 0 0 0 0 0
1 9 ram9 0 0 0 0 0 0 0 0 0 0 0
1 10 ram10 0 0 0 0 0 0 0 0 0 0 0
1 11 ram11 0 0 0 0 0 0 0 0 0 0 0
1 12 ram12 0 0 0 0 0 0 0 0 0 0 0
1 13 ram13 0 0 0 0 0 0 0 0 0 0 0
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
202 0 xvda 1900702 111759 57624068 22798740 21260401 21742422 344022684 149429332 0 25033876 172228060
202 1 xvda1 1178 2366 2 4
202 2 xvda2 2011295 57621462 43002835 344022680
253 0 dm-0 2002226 0 57550986 23818188 42992837 0 343942696 261736148 0 25033368 285554312
253 1 dm-1 8752 0 70016 102772 9998 0 79984 347408 0 16320 450180
9 0 md0 0 0 0 0 0 0 0 0 0 0 0

Then, I come back to vmstat with "-d" option. Just to see formatted information of /proc/diskstats.

$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
ram0 0 0 0 0 0 0 0 0 0 0
ram1 0 0 0 0 0 0 0 0 0 0
ram2 0 0 0 0 0 0 0 0 0 0
ram3 0 0 0 0 0 0 0 0 0 0
ram4 0 0 0 0 0 0 0 0 0 0
ram5 0 0 0 0 0 0 0 0 0 0
ram6 0 0 0 0 0 0 0 0 0 0
ram7 0 0 0 0 0 0 0 0 0 0
ram8 0 0 0 0 0 0 0 0 0 0
ram9 0 0 0 0 0 0 0 0 0 0
ram10 0 0 0 0 0 0 0 0 0 0
ram11 0 0 0 0 0 0 0 0 0 0
ram12 0 0 0 0 0 0 0 0 0 0
ram13 0 0 0 0 0 0 0 0 0 0
ram14 0 0 0 0 0 0 0 0 0 0
ram15 0 0 0 0 0 0 0 0 0 0
xvda 1900690 111759 57623940 22798620 21260398 21742409 344022556 149429324 0 25033
dm-0 2002214 0 57550858 23818068 42992821 0 343942568 261736088 0 25033
dm-1 8752 0 70016 102772 9998 0 79984 347408 0 16
md0 0 0 0 0 0 0 0 0 0 0