The structure of RNN shows that the previous node connects to the current node. This connection is drawn in 2-dimension such that the previous node sits in the left side and the current node sits in the right side. f:id:HealthcareIT_interpreter:20190216122258p:plain However, when you see the input tensor, the structure of it is 3-dimension such that the previous input sits in the front and the current input sits in the back.

In [1]: tensor                                                                  
Out[1]: 
array([[[3, 7],
        [7, 0],
        [2, 0],
        [3, 9],
        [1, 2]],

       [[4, 4],
        [6, 0],
        [2, 4],
        [3, 4],
        [3, 0]]])

In [2]: tensor.shape                                                            
Out[2]: (2, 5, 2)

So if you revisit the RNN structure as the same as that of the tensor (3-dimension), you can draw the previous node in the front, whereas the current one in the back. This gives me the new insight that when you read a sentence you can imagine that the words stream in aligning with the front-back line, which is different from streaming in aligning with the left-right line as shown in your text. This insight may improve one's reading ability. For human's brain, the movement from left to right (or from right to left as well as from up to down) is not natural as human walks from back to front. Having this insight, I tried to read the novel and checked how my reading fluency improved.

By imagining the words stream from back to front, the kind of obsession that you have to move your eye focus from left to right has been alleviated. This alleviation makes me concentrate on just grasping the meaning of the sentence. I can name this reading method as "RNN reading". I want to know how a person with dyslexia feels when he/she tries reading using this method.

2019-02-15

How can you trim the file size of ipynb which is so big that your kernel cannot open it?

I was working on multi-variate regression analysis. There were over 80 explanatory variables so I used the AIC (Akaike information criterion) with step function in order to reduce these. BTW, the AIC step function does not exist in Python so you have to write yourself.

qiita.com This guy wrote his own step function. I copied and pasted it in my ipynb.

def step_aic(model, exog, endog, **kwargs):
    """
    This select the best exogenous variables with AIC
    Both exog and endog values can be either str or list.
    (Endog list is for the Binomial family.)

    Note: This adopt only "forward" selection

    Args:
        model: model from statsmodels.formula.api
        exog (str or list): exogenous variables
        endog (str or list): endogenous variables
        kwargs: extra keyword argments for model (e.g., data, family)

    Returns:
        model: a model that seems to have the smallest AIC
    """

    # convert exog, endog to list format
    exog = np.r_[[exog]].flatten()
    endog = np.r_[[endog]].flatten()
    remaining = set(exog)
    selected = []  # contains adopted candidates

    # calculate AIC only for constants
    formula_head = ' + '.join(endog) + ' ~ '
    formula = formula_head + '1'
    aic = model(formula=formula, **kwargs).fit().aic
    print('AIC: {}, formula: {}'.format(round(aic, 3), formula))

    current_score, best_new_score = np.ones(2) * aic

    # adopt all elements, or ends the loop if the AIC will not be improved although adding any elements
    while remaining and current_score == best_new_score:
        scores_with_candidates = []
        for candidate in remaining:

            # calculate the AIC when adding the remained elements one by one
            formula_tail = ' + '.join(selected + [candidate])
            formula = formula_head + formula_tail
            aic = model(formula=formula, **kwargs).fit().aic
            print('AIC: {}, formula: {}'.format(round(aic, 3), formula))

            scores_with_candidates.append((aic, candidate))

        # adopt the elements that improved the AIC most as the best candidate 
        scores_with_candidates.sort()
        scores_with_candidates.reverse()
        best_new_score, best_candidate = scores_with_candidates.pop()

        # if adding a candinate reduces the AIC, add it as the determined candidates 
        if best_new_score < current_score:
            remaining.remove(best_candidate)
            selected.append(best_candidate)
            current_score = best_new_score

    formula = formula_head + ' + '.join(selected)
    print('The best formula: {}'.format(formula))
    return model(formula, **kwargs).fit()

Here is the problem. "print('AIC: {}, formula: {}'.format(round(aic, 3), formula))" yeilds huge amount of text information on my notebook, which makes my file as big as 80 MB. Have you ever heard of 80 MB sized ipynb? Jupyter notebook cannot handle it and freezed. To solve this problem, you have to trim your ipynb. But how? Your local jupyter kernel cannot open it. I tried once to delete unnecessary part manually by opening ipynb in my editor (as a JSON file) but that forced me huge efforts.

github.com

My idea was to use Google Colab notebook. Colab can handle and open a big size file.

You can use Google Colab Notebooks for trimming outputs. I opened 83 MB ipynb file at Colab and Colab could handle it. From Colab GUI, you can choose the output cell you want to delete then get the file back to your local directory and reopen it. Eventually I trimmed the original file to the size of 2 MB in this way.

2018-07-08

ブログタイトルを変更しました

今年２月から転職をし、病院から外資系企業に籍を移すことになりました。もう医療通訳で診察室に呼ばれることはなくなりましたが、その代わり海外のスタッフと英語でやり取りすることが日常的になり、ますます汗だくになっております。特にテレフォンカンファレンス（TC）で相手がインド英語やフランスなまりの英語だったりすると聞き取ることすら難しい状態です。データアナリストとして日々精進していきますので今後もよろしくお願いします。

現在の取得資格：医療情報技師(2016年8月試験合格), 診療情報管理士(2018年２月試験合格、８８期生)

あ、医療情報技師の点数取得どうしよう。医療情報学会行かないと…。その点、診療情報管理士は資格維持コストが無いので楽ですね。同期の皆さんよろしくお願いします。

2018-03-25

診療情報管理士認定試験の合格者受験番号一覧をPhantomJSでスクショしようとしたらブロックされた

３月２３日に診療情報管理士認定試験の合格者受験番号が発表されました。認定試験のホームページに掲載されております。

f:id:HealthcareIT_interpreter:20180325184734p:plain

よーし、こりゃあWebscrapingしがいがあるのー、ということで、小手調べにPhantomJSでスクリーショットを撮って見よー、と試して見たことろブロックがかかってしまいました。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys as keys
import time

def main():
    url = "https://www.jha-e.com/top/certExams/resultDetails"
    
    driver = webdriver.PhantomJS()
    driver.get(url)
    
    print(driver.current_url)
    
    driver.save_screenshot("testnum.jpg")
    driver.quit()
    
    print("end")


if __name__ == '__main__':
    main()

f:id:HealthcareIT_interpreter:20180325183111j:plain

やっぱセキュリティ気にしてるのかな？このはてなブログにリンク貼ることもできないようです。

www.jha-e.com

ちなみにurlを一つ上の階層の"https://www.jha-e.com/top/certExams/result"にして見たらちゃんとスクショ撮れました。

f:id:HealthcareIT_interpreter:20180325183120j:plain

2017-10-29

Microsoft Translator Text API のcategoryを"generalnn"にすると翻訳精度が飛躍的に向上する

プログラミング医療通訳

Microsoft Translator Text APIを自作のWebアプリに組み入れて翻訳させても翻訳精度が悪くて使い物にならない、というかiOSアプリや Androidアプリと翻訳結果が違っている、という状況に陥っていました。これについて日本マイクロソフトの中の人に直接問い合わせておりましたが、先日日本マイクロソフトの某パートナー企業の人からcategory="generalnn"にすると解消するという貴重な情報をいただき、その通りにすると改善しました。何も指定しない場合は、デフォルトのcategory="general"になるそうです。ちなみに"generalnn"の最後の"nn"というのはニューラル・ネットワークの事でしょうか？こちらに良い記事がありました。

Microsoft Translator launching Neural Network based translations for all its speech languages – Translator

おそらくデフォルトのcategory="general"は、統計学的機械翻訳を行うもので、category="generalnn"は、ディープ・ニューラル・ネットワークによる翻訳を行う、という事なのでしょう。

という事で、以前Githubに公開したライブラリにもcategory="generalnn"を加えておきました。

github.com