Nathan Grigg

Persistent IPython notebook server with launchd, virtual host, and proxy

I have been using IPython for interactive Python shells for several years. For most of that time, I have resisted the web-browser-based notebook interface and mainly used the console version. Despite my love of all things texty, I finally gave in, and began using the web version almost exclusively. So much that I got annoyed at constantly needing to start and stop the IPython server and having a terminal dedicated to running it.

Always running server using Launchd

My first step was to always keep the IPython server running. I did this with a KeepAlive launchd job. Here is the plist:

xml:
<plist version="1.0">
<dict>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>net.nathangrigg.ipython</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/python3/bin/ipython</string>
        <string>notebook</string>
        <string>--no-browser</string>
        <string>--quiet</string>
        <string>--port=10223</string>
        <string>--notebook-dir=/Users/grigg/Dropbox/Notebooks</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardErrorPath</key>
    <string>/Users/grigg/Library/Logs/LaunchAgents/ipython-notebook.stderr</string>
    <key>StandardOutPath</key>
    <string>/Users/grigg/Library/Logs/LaunchAgents/ipython-notebook.stdout</string>
</dict>
</plist>

This job runs ipython notebook with the --port flag, so that the port stays the same each time.

I used LaunchControl to create and load this launch agent, but you can also just save it in ~/Library/LaunchAgents and run launchctl load.

If you want, you can be done now. The notebook browser is running at http://localhost:10223.

Virtual host and proxy using Apache

But I was not done, because I already had too many processes on my machine that were serving content at some localhost port. This required me to memorize port numbers, made Safari’s autocorrect not very useful, and felt barbaric. What I needed was a domain name that resolved to http://localhost:10223. To do this, I needed a virtual host and a proxy.

Before reading further, you should know that I am not an Apache expert. In fact, I have never managed an Apache webserver except as a hobby. The best I can promise you is that this works for me, on my OS X computer, for now.

In /etc/hosts, I created a new host called py.

127.0.0.1     py

This resolves py to 127.0.0.1, i.e., localhost.

Now in /etc/apache2/httpd.conf I created a virtual host and a proxy.

<VirtualHost 127.0.0.1:80>
    ServerName py
    ProxyPass /api/kernels/ ws://localhost:10223/api/kernels/
    ProxyPassReverse /api/kernels/ ws://localhost:10223/api/kernels/
    ProxyPass / http://localhost:10223/
    ProxyPassReverse / http://localhost:10223/
    RequestHeader set Origin "http://localhost:10223/"
</VirtualHost>

This forwards all traffic to py on port 80 to localhost on port 10223. Note that the order of the ProxyPass directives is apparently important. Also, if you use * instead of the address in the VirtualHost directive, you might also be forwarding requests originating outside of your machine, which sounds dangerous.

Then I ran sudo apachectl restart, and everything seemed to work.

Note that Safari interprets py as a Google search, so I have to type py/. Chrome does the same thing, except for that after I load py/ once, the trailing slash is optional.


Number lines

I was reading Dr. Drang’s recent post about numbering lines starting at something other than one, and my first thought was, “That’s silly, the script already does that.” Because my script does, and I was certain I was using his line-numbering Python script.

Apparently I wrote my own. It’s nothing complicated, but it is full featured. It parses the first line to get the starting line number, then replaces any existing line numbers with the correct ones.

My script changes

9: a
b
c

to

 9: a
10: b
11: c

and

 1: a
10: b
11: c

to

1: a
2: b
3: c

It can even remove line numbers if the first line starts with “X:”, so it changes

X: a
2: b
3: c

to

a
b
c

I save it as ~/bin/numberlines, so I can use it on the command line or by selecting lines and typing ':!numberlines in Vim.

python:
 1: #!/usr/bin/python
 2: """
 3: Number lines on a block of text, preserving indentation.
 4: Allow the first line to indicate start number.
 5: If the first lines starts with 'X:' then remove line numbers.
 6: """
 7: import re
 8: import sys
 9:
10: LINE_NO = re.compile(r"^\s*(\d*|[Xx]*):( |$)")
11:
12: lines = sys.stdin.readlines()
13: first_line_number = LINE_NO.match(lines[0])
14:
15: # Determine indentation.
16: indent = min(
17:     (re.match(" *|\t*", line).group(0)
18:          for line in lines if line.strip()),
19:     key=len)
20:
21: if first_line_number and first_line_number.group(1)[0] in "Xx":
22:     formatter = "{indent}{line}"
23:     # These are irrelevant.
24:     start = 1
25:     padding = 0
26: else:
27:     formatter = "{indent}{number:>{padding}}: {line}"
28:     start = int(first_line_number.group(1)) if first_line_number else 1
29:     padding = len(str(start + len(lines)))
30:
31: for i, line in enumerate(lines):
32:     line = LINE_NO.sub("", line[len(indent):])
33:     if not line: line = "\n"
34:     sys.stdout.write(formatter.format(
35:         indent=indent,
36:         number=start + i,
37:         padding=padding,
38:         line=line))

Anonymous functions in zsh

My favorite shell, zsh, allows you to define a function without assigning it a name. This turns out to be incredibly useful, and for none of the reasons that anonymous functions are usually used.

Let’s say I want to copy four or five files from one place to another, but in a more complicated way than the standard cp command allows.

bash:
$ cp 20140508/old.txt 20140508/new.txt
$ cp 20140610/old.txt 20140610/new.txt
$ cp 20140731/old.txt 20140731/new.txt
$ cp 20140802/old.txt 20140802/new.txt

Obviously, you can just run these commands by using your shell’s history and editing each time. But editing is hard, especially in the middle of a line and in more than one place per line.

One way to solve this is with a for loop:

bash:
$ for d in 20140508 20140610 20140731 20140802; do
>   cp $d/old.txt $d/new.txt
> done

But this is not very flexible. If one of the commands fails, you will probably end up with some copies completed and others not. After fixing the problem, you will have to remove the copies that succeeded from the for loop.

Another way to solve this problem is write a function and then use it. (Note that in bash you need to use three lines to define the function.)

bash:
$ mycp() { cp $1/old.txt $1/new.txt }
$ mycp 20140508
$ mycp 20140601
$ mycp 20140731
$ mycp 20140802

This solves the problem of a single command failing, since you can fix it up, rerun the failed command, and continue along. But it also turns a one-step process into two. If I later find I need to do two more copies, I have probably opened a new shell, so I have to first redefine mycp before reusing it.

Or you can use anonymous functions:

bash:
$ () { cp $1/old.txt $1/new.txt } 20140508
$ () { cp $1/old.txt $1/new.txt } 20140601
$ () { cp $1/old.txt $1/new.txt } 20140731
$ () { cp $1/old.txt $1/new.txt } 20140802

Each time you run the command, you just have to find the previous command in your shell history and edit the final word. If you need to use this same command tomorrow, you can search your shell history, and again you only need to edit the final word.


Zsh push-line-or-edit

This week Dr. Drang wrote about a useful feature in recent versions of OS X’s Terminal.app, which allows you to display a man page in its own special window. This is especially useful when you want to look something up but you are in the middle of typing a long command.

It’s not polite to respond to a good solution to a common problem by claiming to have a better solution, and even less so when the better solution requires you to change your shell. But this is the internet, so here goes.

When I need to look at a man page while writing a command, I use zsh’s push-line editing command. This clears the prompt and waits for you to type something else. After executing this new command, it restores your original prompt. The nice thing about this is that it is useful beyond just looking at manuals. Often while typing some command, I realize I need a quick mkdir or cd or even ls before I’m ready to execute.

You can bind push-line to a key (I use Ctrl-B) by putting bindkey '^B' push-line in your .zshrc.

Even better, you can use push-line-or-edit to get the same behavior with one very useful addition. Normally, if you are typing a continuation line of a multi-line command, you cannot make changes to prior lines. But push-line-or-edit redraws the lines as a single block of text, which allows you to edit anything you have typed so far.