Using Julia Language with AWS Lambda functions.

 

Here is a Julia language library to operate with AWS Lamda functions:

https://github.com/samoconnor/AWSLambda.jl#deploy-a-julia-lambda-function

Using Julia Language with AWS Lambda functions.

 

Here is a Julia language library to operate with AWS Lamda functions:

https://github.com/samoconnor/AWSLambda.jl#deploy-a-julia-lambda-function

aws-lambda

Install SAM CLI

$ sudo pip install --user --upgrade aws-sam-cli

Get a list of existing Lambda functions

aws lambda list-functions

Using Postman to test AWS API Gateway

• download Postman app 

aws-lambda

Install SAM CLI

$ sudo pip install --user --upgrade aws-sam-cli

Get a list of existing Lambda functions

aws lambda list-functions

Using Postman to test AWS API Gateway

• download Postman app 

Wordpress-AWS

How to install WordPress blog on the free Amazon AWS EC2 instance.

To start, there is no point re-stating the steps as they are very clear in the AWS tutorial provided below.

Why I decided to look at the WordPress self-hosting if I have this blog for last 10 years?

There are really 2 reasons:

• Extensions available for WordPress are far superior to any other blogging platform
• The WordPress hosted on EC2 instance is on well, Amazon AWS, the infrastructure lives in the same datacenter and benefits from high speed connection to my other micro-services such as AWS Lambda functions. 

Stay tuned for customization tips.

References:

• https://aws.amazon.com/getting-started/tutorials/launch-a-wordpress-website/

Wordpress-AWS

How to install WordPress blog on the free Amazon AWS EC2 instance.

To start, there is no point re-stating the steps as they are very clear in the AWS tutorial provided below.

Why I decided to look at the WordPress self-hosting if I have this blog for last 10 years?

There are really 2 reasons:

• Extensions available for WordPress are far superior to any other blogging platform
• The WordPress hosted on EC2 instance is on well, Amazon AWS, the infrastructure lives in the same datacenter and benefits from high speed connection to my other micro-services such as AWS Lambda functions. 

Stay tuned for customization tips.

References:

• https://aws.amazon.com/getting-started/tutorials/launch-a-wordpress-website/

Slack Slash Commands with AWS Lambda

Introduction

Incorporating Lambda functionality into your Slack opens endless possibilities for automation. 

The commands may provide a quick method to store the information such as "/todo" and "/memo" as well as very sophisticated methods to "/deploy" some code, or "/reserve" assets. 

With some additional work, the commands such as "/predict" or "/classify" could call on machine learning models. 

The options are truly unlimited and can serve your team, or you individually. 

If you constantly add the new automation skills to your repertoire,  it is going to give you that extra "edge" in the hyper-competitive world.

For ideas on constantly building your set of skills, read my article on Singularity on Medium.com.

Basic Architecture

• The individual user, or a team, interact with the Slack client app or website.
• The particular Slack team is identified by a unique token
• The commands are identified by a leading /slash
• The AWS API Gateway is able to provide:
• monetization for the API
• DDOS attack protection
• throttling the frequency of received API calls
• There can be an AWS Lambda function to verify the token and delegate tasks which might limit the security exposure of the system
• AWS Lambda functions fulfill a very particular skill and connect to other AWS services or APIs as needed. 
• Over many years you can have thousands of skills that you could progressively perfect and monetize 

Cost Considerations

• AWS API Gateway costs $3.50 per month per million requests. The first million is free.
• AWS Encryption costs about $1 per month

Create an AWS "IAM" role for this service

https://console.aws.amazon.com/iam/home?region=us-east-1#/roles

Create role step 1

select Lambda

Create role step 2a

Search and select "AWSLambdaBasicExecutionRole"

Create role step 2b

Search and select "AmazonDynamoDBFullAccess"

Create role step 3

Name the role

Go to the Slack you are an ADMINISTRATOR for:
e.g. https://ukidlucas.slack.com/apps

Search for "Slash Commands"

https://ukidlucas.slack.com/apps/[....]-slash-commands

It should display something like:

Click "Add Configuration"

Choose a Command: "/HelloWorld"

Click "Add Slash Command Integration"

From the "Outgoing Data" copy token=LONG_ALPHA_NUMERIC

Save the token value for next step.

Open AWS new Lambda with "slack-echo-command-python" blueprint

You can search and navigate to this blueprint, or click the link below.

https://console.aws.amazon.com/lambda/home?region=us-east-1#/create/new?bp=slack-echo-command-python

Lambda Function: Basic Information

Updating "Execution Role"

• At first, the created role was not available, but later when I edited the Lambda Function it showed up:

Lambda Function: Slack Token

• Paste the previously saved Slack token

Configuring Triggers: API name

Save the Lambda Function

• Click on the "API Gateway" trigger block
• Copy the API endpoint URL 
• Paste that URL into the Slack 
• Save the Slack Configuration

Try Calling the /helloworld from Slack

slackbot [10:12 AM]
Darn - that slash command didn't work (error message: `502_service_error`). Manage the command at text.

Go to AWS CloudWatch (your logs) by clicking Monitoring

https://console.aws.amazon.com/cloudwatch/





You can see that the initialization error has occurred having to do with Encryption

The extra level of encryption using KMS:

Set the token value as the "kmsEncryptedToken" value

Create Customer Master Key (CMK):

https://console.aws.amazon.com/kms/home?region=us-east-1#/kms/home

References

• https://www.thorntech.com/2017/03/serverless-slack-chatbot/

Slack Slash Commands with AWS Lambda

Introduction

Incorporating Lambda functionality into your Slack opens endless possibilities for automation. 

The commands may provide a quick method to store the information such as "/todo" and "/memo" as well as very sophisticated methods to "/deploy" some code, or "/reserve" assets. 

With some additional work, the commands such as "/predict" or "/classify" could call on machine learning models. 

The options are truly unlimited and can serve your team, or you individually. 

If you constantly add the new automation skills to your repertoire,  it is going to give you that extra "edge" in the hyper-competitive world.

For ideas on constantly building your set of skills, read my article on Singularity on Medium.com.

Basic Architecture

• The individual user, or a team, interact with the Slack client app or website.
• The particular Slack team is identified by a unique token
• The commands are identified by a leading /slash
• The AWS API Gateway is able to provide:
• monetization for the API
• DDOS attack protection
• throttling the frequency of received API calls
• There can be an AWS Lambda function to verify the token and delegate tasks which might limit the security exposure of the system
• AWS Lambda functions fulfill a very particular skill and connect to other AWS services or APIs as needed. 
• Over many years you can have thousands of skills that you could progressively perfect and monetize 

Cost Considerations

• AWS API Gateway costs $3.50 per month per million requests. The first million is free.
• AWS Encryption costs about $1 per month

Create an AWS "IAM" role for this service

https://console.aws.amazon.com/iam/home?region=us-east-1#/roles

Create role step 1

select Lambda

Create role step 2a

Search and select "AWSLambdaBasicExecutionRole"

Create role step 2b

Search and select "AmazonDynamoDBFullAccess"

Create role step 3

Name the role

Go to the Slack you are an ADMINISTRATOR for:
e.g. https://ukidlucas.slack.com/apps

Search for "Slash Commands"

https://ukidlucas.slack.com/apps/[....]-slash-commands

It should display something like:

Click "Add Configuration"

Choose a Command: "/HelloWorld"

Click "Add Slash Command Integration"

From the "Outgoing Data" copy token=LONG_ALPHA_NUMERIC

Save the token value for next step.

Open AWS new Lambda with "slack-echo-command-python" blueprint

You can search and navigate to this blueprint, or click the link below.

https://console.aws.amazon.com/lambda/home?region=us-east-1#/create/new?bp=slack-echo-command-python

Lambda Function: Basic Information

Updating "Execution Role"

• At first, the created role was not available, but later when I edited the Lambda Function it showed up:

Lambda Function: Slack Token

• Paste the previously saved Slack token

Configuring Triggers: API name

Save the Lambda Function

• Click on the "API Gateway" trigger block
• Copy the API endpoint URL 
• Paste that URL into the Slack 
• Save the Slack Configuration

Try Calling the /helloworld from Slack

slackbot [10:12 AM]
Darn - that slash command didn't work (error message: `502_service_error`). Manage the command at text.

Go to AWS CloudWatch (your logs) by clicking Monitoring

https://console.aws.amazon.com/cloudwatch/





You can see that the initialization error has occurred having to do with Encryption

The extra level of encryption using KMS:

Set the token value as the "kmsEncryptedToken" value

Create Customer Master Key (CMK):

https://console.aws.amazon.com/kms/home?region=us-east-1#/kms/home

References

• https://www.thorntech.com/2017/03/serverless-slack-chatbot/

I have published a new article on Medium.com

Distributed Cloud as a Pathway to Adaptable Artificial Intelligence.

https://medium.com/@ukidlucas/distributed-machine-learning-and-gp-ai-fc2a0e3f6db

I have published a new article on Medium.com

Distributed Cloud as a Pathway to Adaptable Artificial Intelligence.

https://medium.com/@ukidlucas/distributed-machine-learning-and-gp-ai-fc2a0e3f6db

AWS-Lambda-Android

• AWS Lambda for Android

AWS-Lambda-Android

• AWS Lambda for Android

AWS

Notes on how I use Amazon AWS EC2 instances for Convolutional Deep Neural Networks Machine Learning, using powerful GPU CUDA configurations.

I have moved this post to:

https://ukidlucas.github.io/posts/AWS.html

AWS

Notes on how I use Amazon AWS EC2 instances for Convolutional Deep Neural Networks Machine Learning, using powerful GPU CUDA configurations.

I have moved this post to:

https://ukidlucas.github.io/posts/AWS.html

Configuring AWS instance for Python & Jupyter Notebook server

The configuration of Amazon (AWS) Linux instance with Python Jupyter Notebook for Machine Learning.

Starting the AWS instance is out of scope, plenty of tutorials are available, however, start it in the region that GPU instances such as p2.xlarge are available.

Python

Some Python is already installed

$ python --version
The program 'python' can be found in the following packages:
* python-minimal
* python3
Try: sudo apt install
$ python3 --version
Python 3.5.2

Installing Conda (Anaconda)

$ conda --version
conda: command not found

~$ mkdir Downloads

~$ cd Downloads/

Download conda (find the newest conda install script)

The full version of Anaconda saves you time in the long run so you do not have to log into the server and install missing packages.

full version (455.91M @ 23.0MB/s takes 20s)

https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

mini (not recommended): 

https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

~/Downloads$ wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
~/Downloads$ bash Anaconda3-4.2.0-Linux-x86_64.sh
Follow the instructions and accept defaults
... a lot of packages get installed ...

PATH=/home/ubuntu/anaconda3/bin 

You need to refresh the terminal with the new bashrc settings.

$ source ~/.bashrc
$ conda --version
conda 4.2.9
$ python --version
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)
$ jupyter --version
4.2.0 

As you can see we are in pretty good shape already!

Configure iPython (Jupyter Notebook)

$ ipython
Python 3.5.2
IPython 5.1.0
In [1]: from IPython.lib import passwd
In [2]: passwd()
Enter password: Verify password: ..
Out[2]: 'sha1:5dsfdsfdsfsdfdsfdsfdsfdsfsdfdsfsfds'
In [3]: exit

Copy the password sha1 hash to use later in the configuration file:
c.NotebookApp.password=

CREATE CERTIFICATE

$ cd ~
$ mkdir certificates
$ cd certificates/
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private: /home/ubuntu/certificates
... follow the instructions ...
~/certificates$ ls
mycert.pem

 

Jupyter Notebook Server Configuration

$ jupyter notebook --generate-config

Writing default config to (note it is .jupyter, not .ipython): 

/home/ubuntu/.jupyter/jupyter_notebook_config.py

$ vi /home/ubuntu/.jupyter/jupyter_notebook_config.py
press "i" for INSERT mode

c = get_config()

### Kernel Configuration


# plotting should always be inline

c.IPKernelApp.pylab = 'inline'


### Notebook Configuration


c.NotebookApp.certfile = u'/home/ubuntu/certificates/mycert.pem'

c.NotebookApp.ip = '*'

# server does not have GUI browser

c.NotebookApp.open_browser = False

# generated in iPython shell with password() function

c.NotebookApp.password = u'sha1:9f____your_own_____cc'

# Make sure you open port 8888 in your AWS instance

# and run only one jupyther notebook

c.NotebookApp.port = 8888

Press ESC :wq to WRITE and QUIT vi

Make working directory where you synch your git

~$ mkdir dev

Start the Notebook Server

Normally, I start the notebook in the terminal and it closes when I close the terminal, I prefer to do that.

If you have to run the notebook experiment for a long time (hours, days, weeks) in which case keeping Terminal window is impossible, then you want to start it using:

$ nohup jupyter notebook ~/dev/ &
$ tail -f nohup.out

To shut it down you can look for the process ID (pid) and kill it, or restart the instance.

$ jupyter notebook ~/dev/

[I 15:54:17.847 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret

[I 15:54:18.076 NotebookApp] Serving notebooks from local directory: /home/ubuntu/dev

[I 15:54:18.077 NotebookApp] 0 active kernels 

[I 15:54:18.077 NotebookApp] The Jupyter Notebook is running at: https://[all ip addresses on your system]:8888/

[I 15:54:18.077 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Running in the browser

Make sure that your AWS has at least these ports open: 

- 22 for secure shell 

- 8888 for notebook 

Make sure you request HTTPS

From your instance grab "IPv4 Public IP" or your elastic IP (for Jupyter Notebook I do not need it)

https://your_aws_public_ip_address:8888/

You may get a HTTPS security warning, but I ignore it (in Chrome: ADVANCED: Proceed...).

You should be prompted to enter your own password.

Jupyter Notebook would be fully usable at this point.

conda update conda

Configuring AWS instance for Python & Jupyter Notebook server

The configuration of Amazon (AWS) Linux instance with Python Jupyter Notebook for Machine Learning.

Starting the AWS instance is out of scope, plenty of tutorials are available, however, start it in the region that GPU instances such as p2.xlarge are available.

Python

Some Python is already installed

$ python --version
The program 'python' can be found in the following packages:
* python-minimal
* python3
Try: sudo apt install
$ python3 --version
Python 3.5.2

Installing Conda (Anaconda)

$ conda --version
conda: command not found

~$ mkdir Downloads

~$ cd Downloads/

Download conda (find the newest conda install script)

The full version of Anaconda saves you time in the long run so you do not have to log into the server and install missing packages.

full version (455.91M @ 23.0MB/s takes 20s)

https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

mini (not recommended): 

https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

~/Downloads$ wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
~/Downloads$ bash Anaconda3-4.2.0-Linux-x86_64.sh
Follow the instructions and accept defaults
... a lot of packages get installed ...

PATH=/home/ubuntu/anaconda3/bin 

You need to refresh the terminal with the new bashrc settings.

$ source ~/.bashrc
$ conda --version
conda 4.2.9
$ python --version
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)
$ jupyter --version
4.2.0 

As you can see we are in pretty good shape already!

Configure iPython (Jupyter Notebook)

$ ipython
Python 3.5.2
IPython 5.1.0
In [1]: from IPython.lib import passwd
In [2]: passwd()
Enter password: Verify password: ..
Out[2]: 'sha1:5dsfdsfdsfsdfdsfdsfdsfdsfsdfdsfsfds'
In [3]: exit

Copy the password sha1 hash to use later in the configuration file:
c.NotebookApp.password=

CREATE CERTIFICATE

$ cd ~
$ mkdir certificates
$ cd certificates/
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private: /home/ubuntu/certificates
... follow the instructions ...
~/certificates$ ls
mycert.pem

 

Jupyter Notebook Server Configuration

$ jupyter notebook --generate-config

Writing default config to (note it is .jupyter, not .ipython): 

/home/ubuntu/.jupyter/jupyter_notebook_config.py

$ vi /home/ubuntu/.jupyter/jupyter_notebook_config.py
press "i" for INSERT mode

c = get_config()

### Kernel Configuration


# plotting should always be inline

c.IPKernelApp.pylab = 'inline'


### Notebook Configuration


c.NotebookApp.certfile = u'/home/ubuntu/certificates/mycert.pem'

c.NotebookApp.ip = '*'

# server does not have GUI browser

c.NotebookApp.open_browser = False

# generated in iPython shell with password() function

c.NotebookApp.password = u'sha1:9f____your_own_____cc'

# Make sure you open port 8888 in your AWS instance

# and run only one jupyther notebook

c.NotebookApp.port = 8888

Press ESC :wq to WRITE and QUIT vi

Make working directory where you synch your git

~$ mkdir dev

Start the Notebook Server

Normally, I start the notebook in the terminal and it closes when I close the terminal, I prefer to do that.

If you have to run the notebook experiment for a long time (hours, days, weeks) in which case keeping Terminal window is impossible, then you want to start it using:

$ nohup jupyter notebook ~/dev/ &
$ tail -f nohup.out

To shut it down you can look for the process ID (pid) and kill it, or restart the instance.

$ jupyter notebook ~/dev/

[I 15:54:17.847 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret

[I 15:54:18.076 NotebookApp] Serving notebooks from local directory: /home/ubuntu/dev

[I 15:54:18.077 NotebookApp] 0 active kernels 

[I 15:54:18.077 NotebookApp] The Jupyter Notebook is running at: https://[all ip addresses on your system]:8888/

[I 15:54:18.077 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Running in the browser

Make sure that your AWS has at least these ports open: 

- 22 for secure shell 

- 8888 for notebook 

Make sure you request HTTPS

From your instance grab "IPv4 Public IP" or your elastic IP (for Jupyter Notebook I do not need it)

https://your_aws_public_ip_address:8888/

You may get a HTTPS security warning, but I ignore it (in Chrome: ADVANCED: Proceed...).

You should be prompted to enter your own password.

Jupyter Notebook would be fully usable at this point.

conda update conda