ES7 dialog and form handling pattern

Here’s a handy pattern for handling dialogs and forms in an ES7 application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
form = {
    formfield: "",
};
processForm = null;

openForm(open_form_parameter:int)
{
    // Show dialog

    this.processForm = ()=>{
        // Use open_form_parameter in here with this.form
        // Hide dialog
    }
}

The trick is that open_form_parameter is only available when opening the form (it could be an id of something you clicked to open the dialog). It then gets captured when the processing function (processForm) is defined so that it can be used when the dialog is acknowledged. It saves having to manually store it for use later.

Extract keywords from news articles with python and nltk

In a recent unannounced project I needed to extract keywords (known as named entities) from news articles. I found this code online and modified it a bit for my purposes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import nltk

def get_named_entities(text):
    tokenized = nltk.word_tokenize(text)
    pos_tagged = nltk.pos_tag(tokenized)
    chunked = nltk.chunk.ne_chunk(pos_tagged, binary=False)
    continuous_chunk = []
    current_chunk = []
    for i in chunked:
        if type(i) == nltk.tree.Tree:
            current_chunk.append(" ".join([token for token, pos in i.leaves()]))
        elif current_chunk:
            named_entity = " ".join(current_chunk)
            if named_entity not in continuous_chunk:
                continuous_chunk.append(named_entity)
                current_chunk = []
        else:
            continue
    return continuous_chunk

On a side note, don’t call your .py files the same name as the module you’re importing (i.e. nltk.py). You’ll blow away a couple hours wondering why nothing is defined in the module you want.

Scrape ski hill snow reports with XPaths

This was a fun little project to see if I could build a web page that would gather snow reports from ski hills every day.

The first issue is that most of the web pages need javascript to run in order to fill out the data which makes it a bit of a pain. I chose Chrome headless to get the complete web page and the code is pretty simple:

1
2
3
4
5
6
7
8
9
html = subprocess.check_output(['google-chrome',
                               '--no-sandbox',
                               '--headless',
                               '--user-agent="' + USER_AGENT + '"',
                               '--disable-gpu',
                               '--blink-settings=imagesEnabled=false',
                               '--dump-dom',
                               '--virtual-time-budget=10000',
                               url])

Once you have the text, you can convert it to an element tree and extract an element like this:

1
2
tree = html.fromstring(html)
element = tree.xpath(xpath)

Now we need an actualy XPath. XPaths are pretty flexible in that you can find an element containing text, go up to it’s parent and extract values from another child. Using www.whistlerblackcomb.com as an example the XPaths would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Overnight
//*[contains(text(),"12 Hour")]/parent::div/h5

Last 24 hours
//*[contains(text(),"24 Hour")]/parent::div/h5

Last 48 hours
//*[contains(text(),"48 Hour")]/parent::div/h5

Last 7 days
//*[contains(text(),"7 Day")]/parent::div/h5

Total
//*[contains(text(),"Current")]/parent::div/h5

Base
//*[contains(text(),"Base")]/parent::div/h5

Doing similar things for a bunch of ski hills and you end up with something that looks like this:

Playing with asm.js for AI in a cross platform game

While building troikagame.com I wanted to find ways to save myself a ton of work. The game is available both as Apps (iOS and Android) as well as a web application so people can play on other platforms easily. So we used the Ionic framework. Since it’s all web based, the core of the game is the same on the web as it is in the apps. The game is also multiplayer (via websockets) so that the different platforms can play against each other.

One of the challenges is that there is a computer player that you can play against in single player and multiplayer. Single player AI runs locally on the device/web page and multiplayer AI runs on a server. I didn’t want to write the code twice so I really wanted a common language between the two.

Weirdly, it turns out that the common language is C++. The code runs in a python module on the server and is compiled to asm.js for single player via emscripten. All of the code and scripts are available at https://github.com/smellslikedonkey/troikajs_public

To actually call the module from an es7 application, I use this

1
2
3
4
5
6
7
8
9
10
// In angular.json include the asm.js file
"scripts": [
    "pathtofile/aicpp.js"
]

// Near the top of the file
declare var Module:any;  // Expose module

// Call the module
Module.ai_get_pass_cards(...parameters...);

Check out the final results at www.troikagame.com

Python aiosmtpd package: Building an automatic email responder in Tornado

Recently, after receiving a phishing email that looked pretty legit, I had an idea. What if there was a service to help people identify whether any particular email is a phishing email or not? Enter www.phishbash.com

The idea is simple. Just forward an email to a certain email address and an automated process will summarize it and send a report back. In order to do this I had to figure out how to recieve emails – this will live on AWS so we’re using SES to send emails. I also wanted to do this with Tornado which is my webserver of choice and eventually there will be some web functionality.

Turns out this is pretty easy with the python aiosmtpd package:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
import os
import email
import datetime

import tornado.httpserver
import tornado.websocket
import tornado.web
import argparse

from tornado.ioloop import IOLoop
from tornado import gen
from aiosmtpd.controller import Controller

#===============================================================================
#===============================================================================

class HealthCheckHandler(tornado.web.RequestHandler):
    @gen.coroutine
    def get(self):
        self.write('Still here ' + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") )
        self.finish()

#===============================================================================
#===============================================================================

class SMTPHandler:
    @gen.coroutine
    def handle_RCPT(self, server, session, envelope, address, rcpt_options):

        # Filter valid email addresses
        if address not in ["email@domain.com"]: # Replace with email address of service
            return '550 invalid email address'

        envelope.rcpt_tos.append(address)
        return '250 OK'

    @gen.coroutine
    def handle_DATA(self, server, session, envelope):
        try:
            parsed_email = email.message_from_string(envelope.content.decode('utf8', errors='replace'))

            header_to = parsed_email['to']
            header_from = parsed_email['from']
            header_subject = parsed_email['subject']

            if header_from.endswith("domain.com"):  # Replace with domain of service
                raise RuntimeError("header_from is from domain.com")

            payload_plain = ""
            payload_html = ""
            attachments = []

            # Multipart messages
            if parsed_email.is_multipart():
                for part in parsed_email.walk():
                    ctype = part.get_content_type()
                    cdispo = str(part.get('Content-Disposition'))

                    # skip any text/plain (txt) attachments
                    if 'attachment' in cdispo:
                        filename = part.get_filename()
                        attachments.append(filename)
                    elif 'text/plain' in ctype:
                        payload_string = part.get_payload(decode=True).decode('utf8', errors='replace')
                        payload_plain += payload_string
                    elif 'text/html' in ctype:
                        payload_string = part.get_payload(decode=True).decode('utf8', errors='replace')
                        payload_html += payload_string

            # Not multipart - i.e. plain text, no attachments, keeping fingers crossed
            else:
                ctype = parsed_email.get_content_type()

                if ctype == 'text/plain':
                    payload_string = parsed_email.get_payload(decode=True).decode('utf8', errors='replace')
                    payload_plain += payload_string
                elif ctype == 'text/html':
                    payload_string = parsed_email.get_payload(decode=True).decode('utf8', errors='replace')
                    payload_html += payload_string

            # Do further processing

        except Exception as e:
            pass

        return '250 Message accepted for delivery'

#===============================================================================
# Build Tornado web server
#===============================================================================

APP_SETTINGS = {
    "cookie_secret": "SomeCookieSecretString",
    "login_url": "/index.html"
}

application = tornado.web.Application(
    [
        # Health Checks
        (r'/healthcheck', HealthCheckHandler),
    ],
    debug=(os.environ.get('DEBUG_MODE') == 'devel'),
    **APP_SETTINGS
)

#===============================================================================
#===============================================================================

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--port', help='Server Port')
    parser.add_argument('--smtpport', help='SMTP Port')
    args = parser.parse_args()

    # Begin SMTP server
    smtp_server = Controller(SMTPHandler(), hostname='0.0.0.0', port=int(args.smtpport))
    smtp_server.start()

    # Begin Web server
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(int(args.port))

    IOLoop.instance().start()

#===============================================================================
#===============================================================================