Configuring AWS instance for Python & Jupyter Notebook server

The configuration of Amazon (AWS) Linux instance with Python Jupyter Notebook for Machine Learning.

Starting the AWS instance is out of scope, plenty of tutorials are available, however, start it in the region that GPU instances such as p2.xlarge are available.

Python

Some Python is already installed

$ python --version
The program 'python' can be found in the following packages:
* python-minimal
* python3
Try: sudo apt install
$ python3 --version
Python 3.5.2

Installing Conda (Anaconda)

$ conda --version
conda: command not found

~$ mkdir Downloads

~$ cd Downloads/

Download conda (find the newest conda install script)

The full version of Anaconda saves you time in the long run so you do not have to log into the server and install missing packages.

full version (455.91M @ 23.0MB/s takes 20s)

https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

mini (not recommended): 

https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

~/Downloads$ wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
~/Downloads$ bash Anaconda3-4.2.0-Linux-x86_64.sh
Follow the instructions and accept defaults
... a lot of packages get installed ...

PATH=/home/ubuntu/anaconda3/bin 

You need to refresh the terminal with the new bashrc settings.

$ source ~/.bashrc
$ conda --version
conda 4.2.9
$ python --version
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)
$ jupyter --version
4.2.0 

As you can see we are in pretty good shape already!

Configure iPython (Jupyter Notebook)

$ ipython
Python 3.5.2
IPython 5.1.0
In [1]: from IPython.lib import passwd
In [2]: passwd()
Enter password: Verify password: ..
Out[2]: 'sha1:5dsfdsfdsfsdfdsfdsfdsfdsfsdfdsfsfds'
In [3]: exit

Copy the password sha1 hash to use later in the configuration file:
c.NotebookApp.password=

CREATE CERTIFICATE

$ cd ~
$ mkdir certificates
$ cd certificates/
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private: /home/ubuntu/certificates
... follow the instructions ...
~/certificates$ ls
mycert.pem

 

Jupyter Notebook Server Configuration

$ jupyter notebook --generate-config

Writing default config to (note it is .jupyter, not .ipython): 

/home/ubuntu/.jupyter/jupyter_notebook_config.py

$ vi /home/ubuntu/.jupyter/jupyter_notebook_config.py
press "i" for INSERT mode

c = get_config()

### Kernel Configuration


# plotting should always be inline

c.IPKernelApp.pylab = 'inline'


### Notebook Configuration


c.NotebookApp.certfile = u'/home/ubuntu/certificates/mycert.pem'

c.NotebookApp.ip = '*'

# server does not have GUI browser

c.NotebookApp.open_browser = False

# generated in iPython shell with password() function

c.NotebookApp.password = u'sha1:9f____your_own_____cc'

# Make sure you open port 8888 in your AWS instance

# and run only one jupyther notebook

c.NotebookApp.port = 8888

Press ESC :wq to WRITE and QUIT vi

Make working directory where you synch your git

~$ mkdir dev

Start the Notebook Server

Normally, I start the notebook in the terminal and it closes when I close the terminal, I prefer to do that.

If you have to run the notebook experiment for a long time (hours, days, weeks) in which case keeping Terminal window is impossible, then you want to start it using:

$ nohup jupyter notebook ~/dev/ &
$ tail -f nohup.out

To shut it down you can look for the process ID (pid) and kill it, or restart the instance.

$ jupyter notebook ~/dev/

[I 15:54:17.847 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret

[I 15:54:18.076 NotebookApp] Serving notebooks from local directory: /home/ubuntu/dev

[I 15:54:18.077 NotebookApp] 0 active kernels 

[I 15:54:18.077 NotebookApp] The Jupyter Notebook is running at: https://[all ip addresses on your system]:8888/

[I 15:54:18.077 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Running in the browser

Make sure that your AWS has at least these ports open: 

- 22 for secure shell 

- 8888 for notebook 

Make sure you request HTTPS

From your instance grab "IPv4 Public IP" or your elastic IP (for Jupyter Notebook I do not need it)

https://your_aws_public_ip_address:8888/

You may get a HTTPS security warning, but I ignore it (in Chrome: ADVANCED: Proceed...).

You should be prompted to enter your own password.

Jupyter Notebook would be fully usable at this point.

conda update conda