Python: Removing Matrix columns that contain NaN

Removing Matrix columns that contain NaN. This is a lengthy answer, but hopefully easy to follow.
def column_to_vector(matrix, i):
    return [row[i] for row in matrix]
import numpy
def remove_NaN_columns(matrix):
    import scipy
    import math
    from numpy import column_stack, vstack

    columns = A.shape[1]
    #print("columns", columns)
    result = []
    skip_column = True
    for column in range(0, columns):
        vector = column_to_vector(A, column)
        skip_column = False
        for value in vector:
            # print(column, vector, value, math.isnan(value) )
            if math.isnan(value):
                skip_column = True
        if skip_column == False:
            result.append(vector)
    return column_stack(result)

### test it
A = vstack(([ float('NaN'), 2., 3., float('NaN')], [ 1., 2., 3., 9]))
print("A shape", A.shape, "\n", A)
B = remove_NaN_columns(A)
print("B shape", B.shape, "\n", B)

A shape (2, 4) 
 [[ nan   2.   3.  nan]
 [  1.   2.   3.   9.]]
B shape (2, 2) 
 [[ 2.  3.]
 [ 2.  3.]]


As an Amazon Associate I earn from qualifying purchases.

Python: Removing Matrix columns that contain NaN

Removing Matrix columns that contain NaN. This is a lengthy answer, but hopefully easy to follow.
def column_to_vector(matrix, i):
return [row[i] for row in matrix]
import numpy
def remove_NaN_columns(matrix):
import scipy
import math
from numpy import column_stack, vstack

columns
= A.shape[1]
#print("columns", columns)
result
= []
skip_column
= True
for column in range(0, columns):
vector
= column_to_vector(A, column)
skip_column
= False
for value in vector:
# print(column, vector, value, math.isnan(value) )
if math.isnan(value):
skip_column
= True
if skip_column == False:
result
.append(vector)
return column_stack(result)

### test it
A
= vstack(([ float('NaN'), 2., 3., float('NaN')], [ 1., 2., 3., 9]))
print("A shape", A.shape, "\n", A)
B
= remove_NaN_columns(A)
print("B shape", B.shape, "\n", B)

A shape
(2, 4)
[[ nan 2. 3. nan]
[ 1. 2. 3. 9.]]
B shape
(2, 2)
[[ 2. 3.]
[ 2. 3.]]


As an Amazon Associate I earn from qualifying purchases.

Setting TensorFlow Python on MacBook Pro (yet TBD)

In the "free moment" (2-4AM?) I would like to set up TensorFlow on my Mac.

https://gist.github.com/ageitgey/819a51afa4613649bd18

but that will have to wait until I get to it.


As an Amazon Associate I earn from qualifying purchases.

Setting TensorFlow Python on MacBook Pro (yet TBD)

In the "free moment" (2-4AM?) I would like to set up TensorFlow on my Mac.

https://gist.github.com/ageitgey/819a51afa4613649bd18

but that will have to wait until I get to it.


As an Amazon Associate I earn from qualifying purchases.

Why free AWS is not a good solution for Machine Learning research.

The free Amazon AWS instance t2.micro gives me the memory error when running my Machine Learning exercises (normalization of a large data set).

---------------------------------------------------------------------------
MemoryError  Traceback (most recent call last)
 in ()
----> 1 X_normalized = normalize_color_intensity(X_train)

Looking closer t2.micro has following congiguration: 1 CPU 1 GB RAM So basically it is as I was running my stuff on less than a Rasberry Pi (4 cores, 2GB RAM). Ouch! To surpass my MacBook Pro (4 cores 16GB RAM) I would have to run it on t2.2xlarge with: 8C PU 32 GB RAM Which is $0.376 per hour, or $279 per month. The AWS is a good option for short sessions when very powerful computer instances are needed, and it is good for corporate web servers, but for normal researcher who needs to run experiments day in and day out I would recommend  to buy gaming machine with top GPU, or two.


As an Amazon Associate I earn from qualifying purchases.

Why free AWS is not a good solution for Machine Learning research.

The free Amazon AWS instance t2.micro gives me the memory error when running my Machine Learning exercises (normalization of a large data set).

---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
in ()
----> 1 X_normalized = normalize_color_intensity(X_train)
Looking closer t2.micro has following congiguration: 1 CPU 1 GB RAM So basically it is as I was running my stuff on less than a Rasberry Pi (4 cores, 2GB RAM). Ouch! To surpass my MacBook Pro (4 cores 16GB RAM) I would have to run it on t2.2xlarge with: 8C PU 32 GB RAM Which is $0.376 per hour, or $279 per month. The AWS is a good option for short sessions when very powerful computer instances are needed, and it is good for corporate web servers, but for normal researcher who needs to run experiments day in and day out I would recommend  to buy gaming machine with top GPU, or two.


As an Amazon Associate I earn from qualifying purchases.

Configuring AWS instance for Python & Jupyter Notebook server

The configuration of Amazon (AWS) Linux instance with Python Jupyter Notebook for Machine Learning.

Starting the AWS instance is out of scope, plenty of tutorials are available, however, start it in the region that GPU instances such as p2.xlarge are available.

Python

Some Python is already installed
$ python --version
The program 'python' can be found in the following packages:
* python-minimal
* python3
Try: sudo apt install
$ python3 --version
Python 3.5.2

Installing Conda (Anaconda)

conda --version
conda: command not found


~$ mkdir Downloads

~$ cd Downloads/

Download conda (find the newest conda install script)

The full version of Anaconda saves you time in the long run so you do not have to log into the server and install missing packages.

full version (455.91M @ 23.0MB/s takes 20s)
https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

mini (not recommended): 
https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh


~/Downloads$ wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
~/Downloads$ bash Anaconda3-4.2.0-Linux-x86_64.sh
Follow the instructions and accept defaults
... a lot of packages get installed ...


PATH=/home/ubuntu/anaconda3/bin 

You need to refresh the terminal with the new bashrc settings.


$ source ~/.bashrc
$ conda --version
conda 4.2.9
$ python --version
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)
$ jupyter --version
4.2.0 
As you can see we are in pretty good shape already!

Configure iPython (Jupyter Notebook)

$ ipython
Python 3.5.2
IPython 5.1.0
In [1]: from IPython.lib import passwd
In [2]: passwd()
Enter password: Verify password: ..
Out[2]: 'sha1:5dsfdsfdsfsdfdsfdsfdsfdsfsdfdsfsfds'
In [3]: exit


Copy the password sha1 hash to use later in the configuration file:
c.NotebookApp.password=


CREATE CERTIFICATE


$ cd ~
$ mkdir certificates
$ cd certificates/
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private: /home/ubuntu/certificates
... follow the instructions ...
~/certificates$ ls
mycert.pem

 

Jupyter Notebook Server Configuration



$ jupyter notebook --generate-config
Writing default config to (note it is .jupyter, not .ipython): 
/home/ubuntu/.jupyter/jupyter_notebook_config.py



$ vi /home/ubuntu/.jupyter/jupyter_notebook_config.py
press "i" for INSERT mode



c = get_config()

### Kernel Configuration


# plotting should always be inline

c.IPKernelApp.pylab = 'inline'


### Notebook Configuration


c.NotebookApp.certfile = u'/home/ubuntu/certificates/mycert.pem'

c.NotebookApp.ip = '*'

# server does not have GUI browser

c.NotebookApp.open_browser = False

# generated in iPython shell with password() function

c.NotebookApp.password = u'sha1:9f____your_own_____cc'

# Make sure you open port 8888 in your AWS instance

# and run only one jupyther notebook

c.NotebookApp.port = 8888


Press ESC :wq to WRITE and QUIT vi



Make working directory where you synch your git



~$ mkdir dev

Start the Notebook Server

Normally, I start the notebook in the terminal and it closes when I close the terminal, I prefer to do that.

If you have to run the notebook experiment for a long time (hours, days, weeks) in which case keeping Terminal window is impossible, then you want to start it using:



$ nohup jupyter notebook ~/dev/ &
$ tail -f nohup.out

To shut it down you can look for the process ID (pid) and kill it, or restart the instance.

$ jupyter notebook ~/dev/
[I 15:54:17.847 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret
[I 15:54:18.076 NotebookApp] Serving notebooks from local directory: /home/ubuntu/dev
[I 15:54:18.077 NotebookApp] 0 active kernels 
[I 15:54:18.077 NotebookApp] The Jupyter Notebook is running at: https://[all ip addresses on your system]:8888/

[I 15:54:18.077 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).


Running in the browser

Make sure that your AWS has at least these ports open: 
- 22 for secure shell 
- 8888 for notebook 

Make sure you request HTTPS

From your instance grab "IPv4 Public IP" or your elastic IP (for Jupyter Notebook I do not need it)

https://your_aws_public_ip_address:8888/

You may get a HTTPS security warning, but I ignore it (in Chrome: ADVANCED: Proceed...).
You should be prompted to enter your own password.
Jupyter Notebook would be fully usable at this point.




conda update conda













As an Amazon Associate I earn from qualifying purchases.

Configuring AWS instance for Python & Jupyter Notebook server

The configuration of Amazon (AWS) Linux instance with Python Jupyter Notebook for Machine Learning.

Starting the AWS instance is out of scope, plenty of tutorials are available, however, start it in the region that GPU instances such as p2.xlarge are available.

Python

Some Python is already installed
$ python --version
The program 'python' can be found in the following packages:
* python-minimal
* python3
Try: sudo apt install
$ python3 --version
Python 3.5.2

Installing Conda (Anaconda)

conda --version
conda: command not found


~$ mkdir Downloads

~$ cd Downloads/

Download conda (find the newest conda install script)

The full version of Anaconda saves you time in the long run so you do not have to log into the server and install missing packages.

full version (455.91M @ 23.0MB/s takes 20s)
https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

mini (not recommended): 
https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh


~/Downloads$ wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
~/Downloads$ bash Anaconda3-4.2.0-Linux-x86_64.sh
Follow the instructions and accept defaults
... a lot of packages get installed ...


PATH=/home/ubuntu/anaconda3/bin 

You need to refresh the terminal with the new bashrc settings.


$ source ~/.bashrc
$ conda --version
conda 4.2.9
$ python --version
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)
$ jupyter --version
4.2.0 
As you can see we are in pretty good shape already!

Configure iPython (Jupyter Notebook)

$ ipython
Python 3.5.2
IPython 5.1.0
In [1]: from IPython.lib import passwd
In [2]: passwd()
Enter password: Verify password: ..
Out[2]: 'sha1:5dsfdsfdsfsdfdsfdsfdsfdsfsdfdsfsfds'
In [3]: exit


Copy the password sha1 hash to use later in the configuration file:
c.NotebookApp.password=


CREATE CERTIFICATE


$ cd ~
$ mkdir certificates
$ cd certificates/
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private: /home/ubuntu/certificates
... follow the instructions ...
~/certificates$ ls
mycert.pem

 

Jupyter Notebook Server Configuration



$ jupyter notebook --generate-config
Writing default config to (note it is .jupyter, not .ipython): 
/home/ubuntu/.jupyter/jupyter_notebook_config.py



$ vi /home/ubuntu/.jupyter/jupyter_notebook_config.py
press "i" for INSERT mode



c = get_config()

### Kernel Configuration


# plotting should always be inline

c.IPKernelApp.pylab = 'inline'


### Notebook Configuration


c.NotebookApp.certfile = u'/home/ubuntu/certificates/mycert.pem'

c.NotebookApp.ip = '*'

# server does not have GUI browser

c.NotebookApp.open_browser = False

# generated in iPython shell with password() function

c.NotebookApp.password = u'sha1:9f____your_own_____cc'

# Make sure you open port 8888 in your AWS instance

# and run only one jupyther notebook

c.NotebookApp.port = 8888


Press ESC :wq to WRITE and QUIT vi



Make working directory where you synch your git



~$ mkdir dev

Start the Notebook Server

Normally, I start the notebook in the terminal and it closes when I close the terminal, I prefer to do that.

If you have to run the notebook experiment for a long time (hours, days, weeks) in which case keeping Terminal window is impossible, then you want to start it using:



$ nohup jupyter notebook ~/dev/ &
$ tail -f nohup.out

To shut it down you can look for the process ID (pid) and kill it, or restart the instance.

$ jupyter notebook ~/dev/
[I 15:54:17.847 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret
[I 15:54:18.076 NotebookApp] Serving notebooks from local directory: /home/ubuntu/dev
[I 15:54:18.077 NotebookApp] 0 active kernels 
[I 15:54:18.077 NotebookApp] The Jupyter Notebook is running at: https://[all ip addresses on your system]:8888/

[I 15:54:18.077 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).


Running in the browser

Make sure that your AWS has at least these ports open: 
- 22 for secure shell 
- 8888 for notebook 

Make sure you request HTTPS

From your instance grab "IPv4 Public IP" or your elastic IP (for Jupyter Notebook I do not need it)

https://your_aws_public_ip_address:8888/

You may get a HTTPS security warning, but I ignore it (in Chrome: ADVANCED: Proceed...).
You should be prompted to enter your own password.
Jupyter Notebook would be fully usable at this point.




conda update conda













As an Amazon Associate I earn from qualifying purchases.

Installing Ubuntu 18.04 LTS on AMD box using USB stick created on Mac

Overview


I am installing Ubuntu 18.04.1 LTS desktop (as of November 9, 2018) on an old AMD box I had sitting around, the computer has NVidia 1050 I use for TensorFlow Machine Learning.

Download Ubuntu Desktop (amd64)


https://www.ubuntu.com/download/desktop

Prepare USB Drive

Note: At first, I tried 32GB drive, but the target computer did not recognize it, then I switched to a smaller USB stick. I used Disk Utility to format it in DOS FAT 32.
$ diskutil list
/dev/disk2 (external, physical): #:
DOS_FAT_32 UBUNTU18_04 7.9 GB disk2s1

Install UNetBooting

https://unetbootin.github.io/ 




Install disk image (ISO) on the USB drive


  • Check your USB drive name with "diskutil list" (disk2s1 for me)
  • Open "UNetBooting" and select:
  • Diskimage (radio button)
  • ISO file location
  • disk2s1 as the USB Drive
  • click OK, it will take a long while


Insert the USB into the target AMD computer


Enter BIOS Setup

Attach a keyboard to your computer.
Start the computer while pressing F10 key to enter BIOS setup. This might be different for your computer model. 
In the BIOS you may want to TRY to set up USB (or external mass storage device) as first in the BOOT priority. but my computer did not have that option.

Entering the USB boot with ESC, ESC, ESC

I was able to enter USB boot by starting the computer and pressing ESC key repeatedly.

From here I was shown "UNetBooting" utility and had the option to "Try Ubuntu without installing" which I chose.

Ubuntu Starts

My first impression of Ubuntu 18.04 LTS is that it is much cleaner and modern, it is not Mac OS just yet, but it fills nice.
  • I checked the browser and the Internet was on, via Ethernet cable.
  • The resolution is very good, 1920x1080
  • On the desktop, I had the option to "Install Ubuntu", I followed their easy steps.
  • I chose "Erase disk and Install Ubuntu" as I had old windows and Ubuntu 16.04 on it that I did not want to keep. I needed max space for machine learning datasets.
  • The installation process was very pleasant and modern.
  • Once the installation finished, remove the USB and restart the computer.
Happy Ubuntu computing!












As an Amazon Associate I earn from qualifying purchases.

Installing Ubuntu 18.04 LTS on AMD box using USB stick created on Mac

Overview


I am installing Ubuntu 18.04.1 LTS desktop (as of November 9, 2018) on an old AMD box I had sitting around, the computer has NVidia 1050 I use for TensorFlow Machine Learning.

Download Ubuntu Desktop (amd64)


https://www.ubuntu.com/download/desktop

Prepare USB Drive

Note: At first, I tried 32GB drive, but the target computer did not recognize it, then I switched to a smaller USB stick. I used Disk Utility to format it in DOS FAT 32.
$ diskutil list
/dev/disk2 (external, physical): #:
DOS_FAT_32 UBUNTU18_04 7.9 GB disk2s1

Install UNetBooting

https://unetbootin.github.io/ 




Install disk image (ISO) on the USB drive


  • Check your USB drive name with "diskutil list" (disk2s1 for me)
  • Open "UNetBooting" and select:
  • Diskimage (radio button)
  • ISO file location
  • disk2s1 as the USB Drive
  • click OK, it will take a long while


Insert the USB into the target AMD computer


Enter BIOS Setup

Attach a keyboard to your computer.
Start the computer while pressing F10 key to enter BIOS setup. This might be different for your computer model. 
In the BIOS you may want to TRY to set up USB (or external mass storage device) as first in the BOOT priority. but my computer did not have that option.

Entering the USB boot with ESC, ESC, ESC

I was able to enter USB boot by starting the computer and pressing ESC key repeatedly.

From here I was shown "UNetBooting" utility and had the option to "Try Ubuntu without installing" which I chose.

Ubuntu Starts

My first impression of Ubuntu 18.04 LTS is that it is much cleaner and modern, it is not Mac OS just yet, but it fills nice.
  • I checked the browser and the Internet was on, via Ethernet cable.
  • The resolution is very good, 1920x1080
  • On the desktop, I had the option to "Install Ubuntu", I followed their easy steps.
  • I chose "Erase disk and Install Ubuntu" as I had old windows and Ubuntu 16.04 on it that I did not want to keep. I needed max space for machine learning datasets.
  • The installation process was very pleasant and modern.
  • Once the installation finished, remove the USB and restart the computer.
Happy Ubuntu computing!












As an Amazon Associate I earn from qualifying purchases.

Post Scriptum

The views in this article are mine and do not reflect those of my employer.
I am preparing to cancel the subscription to the e-mail newsletter that sends my articles.
Follow me on:
X.com (Twitter)
LinkedIn
Google Scholar

Popular Recent Posts

Most Popular Articles

apt quotation..