tangled in PowerShell scripts

Recently bogged some Windows Power shell codes.
It does following batch jobs according to the file list on AWS S3 storage. To make the list, I've modified a sample code of AWS S3 Java SDK.

1. invoke Java code (slightly modified AWS S3 Java SDK sample, too) which downloads netsh-made network trace log file which zipped in .cab or .gz format from AWS S3
2. unzip .cab or zipped trace file,
3. invoke Microsoft Network Monitor 3.4 with command line options to extract KerberosV5 specific packets and save to another network trace file
4. invoke Network Monitor again to extract more detail KerberosV5 error packets, and saved its counts to 'YYYYMMDD-HH-krb-err-ERRTYPE.txt' file. This process is iterated for each ERRTYPE.

Most of difficulties come from PowerShell specific features to handle command pipeline and asynchronous unzip process execution. Later on, I may have time to describe details of them.

bye for now.

donno why .bash_login is not used in OSX 10.8

So, I made a tweak. Please don't repeat after me. There shall be better way.

Here is my .bash_profile fragment.

ppid=`ps l $$ | awk '$3 ~ /[0-9]+/{print $3}'`
isloginshell=`ps l $ppid | awk '$13 ~ /login/{print $13}'`
if [ "$isloginshell" != "" ]
then
  /usr/bin/script ~/worklogs/`/bin/date +%Y%m%d-%H%M`.$$.console.log
fi

An anatomy of FUMBARO Japan project : why I'm watching

"FUMBARO Japan" project is not the only one.
There are so many NGO, NPO, volunteer groups. Of course, Japanese goverment and each public bodies doing best. And, as an Japanese, I thank for so much help from so many countries on the earthquakes and nuclear-plant acceident.

The reason why I'm following FUMBARO project is,

1. It concentrates in matching between needs and relief goods. Out-sourcing logistics to Amazon. This strategy keeps away from bottle-neck problem.
2. It does not wait for request. It does search for request on foot. (Network is not available there)
   
3. Find out key-person from disaster-stricken area. Then ask him/her to be a network-node of the project. Now, FUMBARO project has over 200 nodes, and that number is going up.
4. Via network nodes in disaster-stricken area, the project detecting changes of needs. Used to be, food is the first priority. Now, victims needs skills to build up their new town. The project began new plans to support them to get license to drive some heavy machinery.
   

For some people other than Japanese, it may seem ordinary NGO. But this style is very new and eccentric as a Japanese NGO. So, I'm very interested. Also I'm very worried about its success.

FUMBRO Japan project : http://fumbaro.org/

FUMBARO Japan project

Recently, I'm spending so many hours in Twitter.
There, "FUMBARO Japan" (Never say die Japan!) project http://fumbaro.org/ is going on.

The project covers "needs vs help matching" in recovery process from that earthquakes and following nuclear-plant accident.

Usually, this function is covered by the Japanese goverment and local public body. But in this disaster, some of local public body itself were severely damaged. So, thousand of people are still no-food, no-tapwater, no-electric power, no-home (some are gathering at relations home).
Many NGO, NPO, and volunteer groups are busy helping victims. But, still "needs research on-foot" is not completed yet. The "Fumbaro Japan" project covers this point.
As long as you can see on TV, and internet, most of striken areas are going back to "NORMAL". But, still there are so many people waiting help without network connection.

If you have any idea to help them, please contact to @saijotakeo on Twitter. Or, look at http://www.facebook.com/fumbaro.org on Facebook.

Further, if you can read some Japanese, you can send goods based on "Wish list" on Amazon.co.jp . "Fumbaro Japan project" is working on wish-list making from their "needs research"

Cannot click OK button in Google's OAuthAuthorizeToken page using mobile phone

I'm trying to make a Buzz viewer for Japanese mobile (not so smart) phones, which cannot execute Javascripts.

And I'm in trouble to navigate OAuth process.

To get authorization, first, you need to go
https://www.google.com/buzz/api/auth/OAuthAuthorizeToken
with parameters below (as documented in OAuth API/Buzz API docs).

oauth_token
domain
scope
iconUrl(optional)
hd(optional)
hl(optional)
btmpl(optional)

Expecting rather simple templete, I've set btmpl parameter as 'mobile'.
But it still includes Javascript-assisited-buttons.
Thus, most of the Japanese mobile phones (and its browser) cannot
click "OK, continue" button. Nor OAuth work.

Is there any solution to show simpler template?

At first, I planned to post above to Google Buzz API Developer forum.
But I've changed my mind. This is common problem in all Google's OAuth processes.
Where/Who can I ask for?

micro(nano?) Auth

just thinking now... not yet completed memo.

For mobile (or whatever computers which run modern OS) device, every "Native-application" call for kernel would better be under MAC(Mandatory Access Control).
In this case, request-token can be used.
Request tokens are issued by OS while the application is certified. The token is checked for Revokation. MAC on each kernel call realizes "real-time Revokation".

Of course, periodical check would be available, and with less load; such a s "on app booting only" or "on OS rebooting only" type.

I'd better make a survey on SELinux case...

For virtualization Hypervisors, MAC can be useful, because application ACL is almost compatible for various guest OS.
But it should not degrade performance drastically.
Anyway, every kernel call is trapped by Hypervisor (CPU may help passing thru for most of the cases). And there, MAC can be done.

How to check?
Full Certification is heavy process. Just check matching of fingerprint? Or, just booting/loaded to memory as DLL case? I guess short-handed check shall be done on each load of executable objects.
But, "on writable-memory exec" would be the path to jail-breaking of executable loading check although some CPU architectures inhibit such operation(Stack area corruption and jump to anywhere hacker'd like is the very familier opration.)

XML parsing in Erlang

XML handling in Erlang is too hard for me. So I've made some survey first.

Characteristics

XML handling operation has two major phases.

• Parse XML document and make data(tree of elements)
  
• xmerl_scan (whole element tree is processed at once)
  
• SAX type parser
  
• xmerl_eventp
• erlsom_sax (developed 3rd party. not bundled with Erlang OTP)
  

• Access (traverse) elements within data
• XPATH (xmerl_xpath)
  
• XSLT (xmerl_xs)
• callback (hook) function from SAX type parser
  
• hand-made logic
• traverse tree
• extract tuple from element tree(list) by 'list comprehension' technique
  

So, methodology of XML parsing is characterized by Parsing and Access method

matrix of each method

parse method	Acceess method	samples on the Web	by
xmerl_scan	xmerl_xpath	Parsing Atom with Erlang	Sam Ruby
xmerl_scan	xmerl_xpath (with useful MACRO)	XML processing in Erlang	Torbjörn Törnkvist
xmerl_scan	hand-made (traverse tree by lists:foldl)	Return Erlang Data from XML	Muharem Hrnjadovic
xmerl_scan	hand-made (use list comprehension)	XML processing in Erlang	Hakan Mattson
xmerl_eventp	callback(hook) function	XML processing in Erlang	Torbjörn Törnkvist
erlsom_sax	callback function	XML processing in Erlang	Willem de Jong

Operation example

example-1 : emerl_scan + xpath

If you know which elements you need exactly, and source XML file is not so huge, parse by xmerl_scan, access by xmerl_xpath.
Note: As of Erlang/OTP R13B01 supports XPATH 1.0

inspired by Torbjörn Törnkvist's code.

sample xml data ("e.xml")

<Envelope>
<Title>envelope title</Title>
<InnerEnv>
<IDNUM>403276</IDNUM>
<ItemName>Name String</ItemName>
<Pages>0</Pages>
</InnerEnv>
</Envelope>

code

-module(example1).
-export([doit/1]).
-include_lib("xmerl/include/xmerl.hrl").

-define(Val(X),
(fun() ->
[#xmlElement{name = N,
content = [#xmlText{value = V}|_]}] = X,
{N,V} end)())
.

doit(File) ->
{Xml, _} = xmerl_scan:File(File),
[
?Val(xmerl_xpath:string("/Envelope/Title", Xml)),
?Val(xmerl_xpath:string("//IDNUM", Xml)),
?Val(xmerl_xpath:string("//ItemName", Xml)),
?Val(xmerl_xpath:string("//Pages", Xml))
]
.

results

1> example1:go("e.xml").
[
{'Title',"envelope title"},
{'IDNUM',"403276"},
{'ItemName',"Name String"},
{'Pages',"0"}]

example-2 : xmerl_scan + traverse element tree by lists:foldl

if you want to translate whole XML data into other scheme, you need to traverse whole tree by lists:foldl function.

inspired by Muharem Hrnjadovic's code

sample xml data ("e.xml")

<Envelope>
<Title>envelope title</Title>
<InnerEnv>
<IDNUM>403276</IDNUM>
<ItemName>Name String</ItemName>
<Pages>0</Pages>
</InnerEnv>
</Envelope>

code

-module(example2).
-export([go/1]).
-include_lib("xmerl/include/xmerl.hrl").
go(File) -> {R, _} = xmerl_scan:file(File),
io:format("~p~n",[lists:reverse(traverse(R, []))]) .
traverse(R, L) when is_record(R, xmlElement) -> lists:foldl(fun traverse/2, L, R#xmlElement.content) ;
traverse(#xmlText{parents=[{'Title',_},_], value=V}, L) -> [{title, V}|L];
traverse(#xmlText{parents=[{'IDNUM',_},_,_], value=V}, L) -> [{idnum, V}|L];
traverse(#xmlText{parents=[{'ItemName',_},_,_], value=V}, L) -> [{itemname, V}|L];
traverse(#xmlText{parents=[{'Pages',_},_,_], value=V}, L) -> [{pages, V}|L];

traverse(_R, L) ->
L
.

results

2> example2:go("e.xml").
[{title,"envelope title"},
{idnum,"403276"},
{itemname,"Name String"},
{pages,"0"}]

example-3 : SAX type parsing operation

if the computation resouce is limited, whole XML data cannot be processed at once. So, SAX type parser callback functions to process just parsed element.

As SAX type parser, erlsom_sax is welknown. And Willem de Jong posted his code based on erlsom_sax.
see http://blog.tornkvist.org/blog.yaws?id=1193209275268448

here is the code based on xmerl_sax_parser library which is inspired by above code.

sample xml data ("e.xml")

<Envelope>

<Title>envelope title</Title>
<InnerEnv>
<IDNUM>403276</IDNUM>
<ItemName>Name String</ItemName>
<Pages>0</Pages>
</InnerEnv>
</Envelope>

code

-module(example3).
-export([go/1]).
go(File) ->
Option = [ {event_fun, fun eventfun/3}, {event_state, {[], []}} ], case xmerl_sax_parser:file(File, Option) of
　{ok,{Stack, Acc}} -> lists:reverse(Acc);
{Other} -> Other
end .

eventfun({ignorableWhitespace, _}, _, State) ->
State ;
eventfun({startElement, _, Tag, _, _}, _Location, {Stack, Acc}) -> {[Tag | Stack], Acc} ;
eventfun({characters, Value}, _Location, {[Tag | _L] = Stack, Acc}) -> {Stack, [{Tag, Value} | Acc]} ;
eventfun({endElement, _, _, _}, _Location, {[_ | L], Acc}) -> {L, Acc} ;
eventfun(_,_,State) -> State .

results

6> example3:go("e.xml").
[{"Title","envelope title"},
{"IDNUM","403276"},
{"ItemName","Name String"},
{"Pages","0"}]