Julia language: DataFrames: insert a column

The following code

inserts a new column into DataFrame named df
inserts this column as a first (col_ind =1) column
names this column Rata_Die
populates the column values with an array of Int64 zeros (0)
makes sure the name of the column is unique

col_ind = 1
insertcols!(df, col_ind, :Rata_Die => zeros(Int64, record_count); makeunique = true )

Julia language: DataFrames: insert a column

The following code

inserts a new column into DataFrame named df
inserts this column as a first (col_ind =1) column
names this column Rata_Die
populates the column values with an array of Int64 zeros (0)
makes sure the name of the column is unique

col_ind = 1
insertcols!(df, col_ind, :Rata_Die => zeros(Int64, record_count); makeunique = true )

Understanding lens distortion OpenCV

https://learnopencv.com/understanding-lens-distortion/

Julia language UNIX date time

Unix time date format is used in many applications, including Yahoo finance.

using Dates, Printf
unix_date = @sprintf("%.0f", Dates.datetime2unix(Dates.now()))
println( unix_date ) # String

Output:

1603013077

Julia language UNIX date time

Unix time date format is used in many applications, including Yahoo finance.

using Dates, Printf
unix_date = @sprintf("%.0f", Dates.datetime2unix(Dates.now()))
println( unix_date ) # String

Output:

1603013077

How to pay and seal the cotton caulked wooden boat

By The Art of Boat Building

Opening the Jupyter notebook in a different drive on Windows 10.

Assuming that you have Anaconda 3 installed,
you can open the "Anaconda Prompt" window and start jupyter,
however, it opens in the default C: drive.

Anaconda prompt does not allow you to change the drive from the command line. (WTF !?!)

The simplest solution I found was to specify the new drive (R: for Repos) in Jupyter startup:

> jupyter notebook --notebook-dir 'R:'

Sidenote:

In case you wonder why I am putting up with Windows 10 and not working on my MacOS, I wanted to try the NVidia GPU performance with Julia in the Jupyter notebook.

Opening the Jupyter notebook in a different drive on Windows 10.

Assuming that you have Anaconda 3 installed,
you can open the "Anaconda Prompt" window and start jupyter,
however, it opens in the default C: drive.

Anaconda prompt does not allow you to change the drive from the command line. (WTF !?!)

The simplest solution I found was to specify the new drive (R: for Repos) in Jupyter startup:

> jupyter notebook --notebook-dir 'R:'

Sidenote:

In case you wonder why I am putting up with Windows 10 and not working on my MacOS, I wanted to try the NVidia GPU performance with Julia in the Jupyter notebook.

How to build a sailboat half-model?

Filtering DataFrame

New Syntax

Syntax explained:

from original "data" (type DataFrames)
create new "dataA" (type DataFrames)
filter indices in column "Treatment" with values equal to "A"
include ALL COLUMNS indicated by ":"
Show the first 6 rows

dataA = data[isequal.(data.Treatment, "A"), : ]
first(dataA, 6)

6 rows × 5 columns

	Age	WCC	CRP	Treatment	Result
	Int64	Float64	Int64	String	String
1	37	10.2	30	A	Worse
2	67	10.4	70	A	Worse
3	42	10.4	100	A	Static
4	56	12.5	80	A	Worse
5	60	9.5	0	A	Static
6	62	13.7	10	A	Worse

Deprecated Syntax

# dataA = data[data[:Treatment] .== "A", :]

Warning: `getindex(df::DataFrame, col_ind::ColumnIndex)` is deprecated, use `df[!, col_ind]` instead.

Filtering DataFrame

New Syntax

Syntax explained:

from original "data" (type DataFrames)
create new "dataA" (type DataFrames)
filter indices in column "Treatment" with values equal to "A"
include ALL COLUMNS indicated by ":"
Show the first 6 rows

dataA = data[isequal.(data.Treatment, "A"), : ]
first(dataA, 6)

6 rows × 5 columns

	Age	WCC	CRP	Treatment	Result
	Int64	Float64	Int64	String	String
1	37	10.2	30	A	Worse
2	67	10.4	70	A	Worse
3	42	10.4	100	A	Static
4	56	12.5	80	A	Worse
5	60	9.5	0	A	Static
6	62	13.7	10	A	Worse

Deprecated Syntax

# dataA = data[data[:Treatment] .== "A", :]

Warning: `getindex(df::DataFrame, col_ind::ColumnIndex)` is deprecated, use `df[!, col_ind]` instead.

Julia packages (Pkg)

Here are the Julia packages I have been using so far, I keep this list in case I need to re-install Julia.

Un-comment, install and pre-compile the package you need:

import Pkg
# Pkg.add("Atom")
# using Atom

# Pkg.add("BinDeps")
# using BinDeps

# Pkg.add("CSV")
# using CSV

# Pkg.add("CUDAapi")
# using CUDAapi

# Pkg.add("Calculus")
# using Calculus

# Pkg.add("CodecZlib")
# using CodecZlib

# Pkg.add("Codecs")
# using Codecs

# Pkg.add("ColorTypes")
# using ColorTypes

# Pkg.add("Colors")
# using Colors

# Pkg.add("Combinatorics")
# using Combinatorics

# Pkg.add("Compat")
# using Compat

# Pkg.add("Compose")
# using Compose

# Pkg.add("Contour")
# using Contour

# Pkg.add("DataArrays")
# using DataArrays

# Pkg.add("DataFrames")
# using DataFrames

# Pkg.add("DataStructures")
# using DataStructures

# Pkg.add("Distances")
# using Distances

# Pkg.add("Distributions")
# using Distributions

# Pkg.add("FileIO")
# using FileIO

# Pkg.add("FixedPointNumbers")
# using FixedPointNumbers

# Pkg.add("Flux")
# using Flux

# Pkg.add("ForwardDiff")
# using ForwardDiff

# Pkg.add("GLM")
# using GLM

# Pkg.add("GR")
# using GR

# Pkg.add("GZip")
# using GZip

# Pkg.add("GeometryBasics")
# using GeometryBasics

# Pkg.add("Hexagons")
# using Hexagons

# Pkg.add("HypothesisTests")
# using HypothesisTests

# Pkg.add("IJulia")
# using IJulia

# Pkg.add("JSON")
# using JSON

# Pkg.add("Juno")
# using Juno

# Pkg.add("KernelDensity")
# using KernelDensity

# Pkg.add("Knet")
# using Knet

# Pkg.add("Lathe")
# using Lathe

# Pkg.add("Loess")
# using Loess

# Pkg.add("Measures")
# using Measures

# Pkg.add("Metalhead")
# using Metalhead

# Pkg.add("NaNMath")
# using NaNMath

# Pkg.add("OffsetArrays")
# using OffsetArrays

# Pkg.add("Optim")
# using Optim

# Pkg.add("PDMats")
# using PDMats

# Pkg.add("Parameters")
# using Parameters

# Pkg.add("Plots")
# using Plots

# Pkg.add("Polynomials")
# using Polynomials

# Pkg.add("Primes")
# using Primes

# Pkg.add("PyCall")
# using PyCall

# Pkg.add("PyPlot")
# using PyPlot

# Pkg.add("QuartzImageIO")
# using QuartzImageIO

# Pkg.add("RDatasets")
# using RDatasets

# Pkg.add("Reexport")
# using Reexport

# Pkg.add("Rmath")
# using Rmath

# Pkg.add("Roots")
# using Roots

# Pkg.add("Showoff")
# using Showoff

# Pkg.add("SortingAlgorithms")
# using SortingAlgorithms

# Pkg.add("StaticArrays")
# using StaticArrays

# Pkg.add("StatsBase")
# using StatsBase

# Pkg.add("StatsFuns")
# using StatsFuns

# Pkg.add("StatsPlots")
# using StatsPlots

# Pkg.add("TensorFlow")
# using TensorFlow

# Pkg.add("TextAnalysis")
# using TextAnalysis

# Pkg.add("TinySegmenter")
# using TinySegmenter

# Pkg.add("URIParser")
# using URIParser

# Pkg.add("UnicodePlots")
# using UnicodePlots

# Pkg.add("WordTokenizers")
# using WordTokenizers

# Pkg.add("Dates")
# using Dates

# Pkg.add("DelimitedFiles")
# using DelimitedFiles

# Pkg.add("SHA")
# using SHA

# Pkg.add("Statistics")
# using Statistics

Julia packages (Pkg)

Here are the Julia packages I have been using so far, I keep this list in case I need to re-install Julia.

Un-comment, install and pre-compile the package you need:

import Pkg
# Pkg.add("Atom")
# using Atom

# Pkg.add("BinDeps")
# using BinDeps

# Pkg.add("CSV")
# using CSV

# Pkg.add("CUDAapi")
# using CUDAapi

# Pkg.add("Calculus")
# using Calculus

# Pkg.add("CodecZlib")
# using CodecZlib

# Pkg.add("Codecs")
# using Codecs

# Pkg.add("ColorTypes")
# using ColorTypes

# Pkg.add("Colors")
# using Colors

# Pkg.add("Combinatorics")
# using Combinatorics

# Pkg.add("Compat")
# using Compat

# Pkg.add("Compose")
# using Compose

# Pkg.add("Contour")
# using Contour

# Pkg.add("DataArrays")
# using DataArrays

# Pkg.add("DataFrames")
# using DataFrames

# Pkg.add("DataStructures")
# using DataStructures

# Pkg.add("Distances")
# using Distances

# Pkg.add("Distributions")
# using Distributions

# Pkg.add("FileIO")
# using FileIO

# Pkg.add("FixedPointNumbers")
# using FixedPointNumbers

# Pkg.add("Flux")
# using Flux

# Pkg.add("ForwardDiff")
# using ForwardDiff

# Pkg.add("GLM")
# using GLM

# Pkg.add("GR")
# using GR

# Pkg.add("GZip")
# using GZip

# Pkg.add("GeometryBasics")
# using GeometryBasics

# Pkg.add("Hexagons")
# using Hexagons

# Pkg.add("HypothesisTests")
# using HypothesisTests

# Pkg.add("IJulia")
# using IJulia

# Pkg.add("JSON")
# using JSON

# Pkg.add("Juno")
# using Juno

# Pkg.add("KernelDensity")
# using KernelDensity

# Pkg.add("Knet")
# using Knet

# Pkg.add("Lathe")
# using Lathe

# Pkg.add("Loess")
# using Loess

# Pkg.add("Measures")
# using Measures

# Pkg.add("Metalhead")
# using Metalhead

# Pkg.add("NaNMath")
# using NaNMath

# Pkg.add("OffsetArrays")
# using OffsetArrays

# Pkg.add("Optim")
# using Optim

# Pkg.add("PDMats")
# using PDMats

# Pkg.add("Parameters")
# using Parameters

# Pkg.add("Plots")
# using Plots

# Pkg.add("Polynomials")
# using Polynomials

# Pkg.add("Primes")
# using Primes

# Pkg.add("PyCall")
# using PyCall

# Pkg.add("PyPlot")
# using PyPlot

# Pkg.add("QuartzImageIO")
# using QuartzImageIO

# Pkg.add("RDatasets")
# using RDatasets

# Pkg.add("Reexport")
# using Reexport

# Pkg.add("Rmath")
# using Rmath

# Pkg.add("Roots")
# using Roots

# Pkg.add("Showoff")
# using Showoff

# Pkg.add("SortingAlgorithms")
# using SortingAlgorithms

# Pkg.add("StaticArrays")
# using StaticArrays

# Pkg.add("StatsBase")
# using StatsBase

# Pkg.add("StatsFuns")
# using StatsFuns

# Pkg.add("StatsPlots")
# using StatsPlots

# Pkg.add("TensorFlow")
# using TensorFlow

# Pkg.add("TextAnalysis")
# using TextAnalysis

# Pkg.add("TinySegmenter")
# using TinySegmenter

# Pkg.add("URIParser")
# using URIParser

# Pkg.add("UnicodePlots")
# using UnicodePlots

# Pkg.add("WordTokenizers")
# using WordTokenizers

# Pkg.add("Dates")
# using Dates

# Pkg.add("DelimitedFiles")
# using DelimitedFiles

# Pkg.add("SHA")
# using SHA

# Pkg.add("Statistics")
# using Statistics

"Rata Die". Counting Days since...

Counting Days since...

https://en.wikipedia.org/wiki/Rata_Die

# col1 is an exported column of dates
lastDay = 54 # last row in the column sorted by the date
daysSince(x) = Dates.datetime2rata(x) - Dates.datetime2rata(col1[lastDay])
arrayDays = Array{Int64}(undef, lastDay) # array size is 54
for i = 1:lastDay
arrayDays[i] = daysSince(col1[i])
end
arrayDays

Output:

54-element Array{Int64,1}:
 613
 606
 599
 592
 582
 575
 568
 554
 547
 540
 533
 526
 512
   ⋮

"Rata Die". Counting Days since...

Counting Days since...

https://en.wikipedia.org/wiki/Rata_Die

# col1 is an exported column of dates
lastDay = 54 # last row in the column sorted by the date
daysSince(x) = Dates.datetime2rata(x) - Dates.datetime2rata(col1[lastDay])
arrayDays = Array{Int64}(undef, lastDay) # array size is 54
for i = 1:lastDay
arrayDays[i] = daysSince(col1[i])
end
arrayDays

Output:

54-element Array{Int64,1}:
 613
 606
 599
 592
 582
 575
 568
 554
 547
 540
 533
 526
 512
   ⋮

Flux: The Julia Machine Learning Library

Flux is a machine learning library writer in Julia language.

Installation

(v1.3) pkg> add Flux

Flux: The Julia Machine Learning Library

Flux is a machine learning library writer in Julia language.

Installation

(v1.3) pkg> add Flux

Julia language, read delimited files (CSV) with "readdlm"

When trying to execute:

wikiEVDraw = readdlm("wikipediaEVDraw.csv", ',') # getting quotes right is important!

I get the following error:

UndefVarError: readdlm not defined.

help?> readdlm search: readdir Couldn't find readdlm Perhaps you meant readdir, read, read!, real or readchomp No documentation found. Binding readdlm does not exist.

In this case use:

using DelimitedFiles wikiEVDraw = readdlm("wikipediaEVDraw.csv", ',') # getting quotes right is important!

Julia language, read delimited files (CSV) with "readdlm"

When trying to execute:

wikiEVDraw = readdlm("wikipediaEVDraw.csv", ',') # getting quotes right is important!

I get the following error:

UndefVarError: readdlm not defined.

help?> readdlm search: readdir Couldn't find readdlm Perhaps you meant readdir, read, read!, real or readchomp No documentation found. Binding readdlm does not exist.

In this case use:

using DelimitedFileswikiEVDraw = readdlm("wikipediaEVDraw.csv", ',') # getting quotes right is important!

Using Julia Language with AWS Lambda functions.

Here is a Julia language library to operate with AWS Lamda functions:

https://github.com/samoconnor/AWSLambda.jl#deploy-a-julia-lambda-function

Using Julia Language with AWS Lambda functions.

Here is a Julia language library to operate with AWS Lamda functions:

https://github.com/samoconnor/AWSLambda.jl#deploy-a-julia-lambda-function

Julia

The best way to start with Julia is to install JuliaPro, for free.

https://juliacomputing.com/products/juliapro

Julia

The best way to start with Julia is to install JuliaPro, for free.

https://juliacomputing.com/products/juliapro

Arrays in Julia Language

Way to create an Array in Julia language:

A simple way of creating an array

array18 = fill(18, 4) # values of 18, 4 rows

4-element Array{Int64,1}:
 18
 18
 18
 18

Float64, 2-dimensional, 4-rows, 3-columns

A = Array{Float64,2}(undef, 4,3)
# Array of floats, 2-dimentional, populate with undefiled data, 4-rows, 3-columns

4×3 Array{Float64,2}:
 2.40692e-314  5.0e-324      1.5e-323
 2.39671e-314  2.40692e-314  1.5e-323
 2.40692e-314  2.40692e-314  1.0e-323

 2.40692e-314  2.40692e-314  2.38725e-314

A[1,1] = 123
A

4×3 Array{Float64,2}:
 123.0           5.0e-324      2.5e-323
   2.38296e-314  2.38338e-314  2.5e-323
   2.38338e-314  2.38338e-314  1.0e-323
   2.38338e-314  2.38338e-314  2.64889e-32

Abstract Type Integer

A = Array{Integer}(undef,2,3)

2×3 Array{Integer,2}:
 #undef  #undef  #undef
 #undef  #undef  #undef

Concreate Type Int64

A = Array{Int64}(undef,2,3)

2×3 Array{Int64,2}:
 4826207760  4826207824  4826207888
 4826207792  4826207856  4825801360

Reference:

https://docs.julialang.org/en/v1/base/arrays/

Arrays in Julia Language

Way to create an Array in Julia language:

A simple way of creating an array

array18 = fill(18, 4) # values of 18, 4 rows

4-element Array{Int64,1}:
 18
 18
 18
 18

Float64, 2-dimensional, 4-rows, 3-columns

A = Array{Float64,2}(undef, 4,3)
# Array of floats, 2-dimentional, populate with undefiled data, 4-rows, 3-columns

4×3 Array{Float64,2}:
 2.40692e-314  5.0e-324      1.5e-323
 2.39671e-314  2.40692e-314  1.5e-323
 2.40692e-314  2.40692e-314  1.0e-323

 2.40692e-314  2.40692e-314  2.38725e-314

A[1,1] = 123
A

4×3 Array{Float64,2}:
 123.0           5.0e-324      2.5e-323
   2.38296e-314  2.38338e-314  2.5e-323
   2.38338e-314  2.38338e-314  1.0e-323
   2.38338e-314  2.38338e-314  2.64889e-32

Abstract Type Integer

A = Array{Integer}(undef,2,3)

2×3 Array{Integer,2}:
 #undef  #undef  #undef
 #undef  #undef  #undef

Concreate Type Int64

A = Array{Int64}(undef,2,3)

2×3 Array{Int64,2}:
 4826207760  4826207824  4826207888
 4826207792  4826207856  4825801360

Reference:

https://docs.julialang.org/en/v1/base/arrays/

Special Characters used for Math, ML, Logic in Jupyther Lab

Here is a quick cheat sheet of useful Unicode characters you can use in Jupyther Lab and Julia language:

Greek Alphabet

α   \alpha
β   \beta
γ   \gamma
∇   \delta

ϵ \epsilon

ζ \zeta

η \eta

θ \theta

ι \iota

κ   \kappa (should look like 'k')
λ   \lambda
μ   \mu

ν nu (not supported in Jupyther)

ξ \xi

ο omnicron (not supported in Jupyther)

π \pi

ρ \rho

σ \sigma

τ \tau

υ upsilon (not supported in Jupyther)

ϕ \phi

χ \chi

ψ \psi

ω \omega

Hebrew

ℵ \aleph wiki

Machine Learning

x̂ x\hat

Math

∡   \measuredangle
∑   \sum
±   \pm (plus minus)

≠ \ne

≥ \ge (greater or equal)

≤ \le (less or equal)
⌀ \diameter
0° 0\degree

Sets

∈ \in

Logic

∴ \therefore

⊻ \xor

Special Characters used for Math, ML, Logic in Jupyther Lab

Here is a quick cheat sheet of useful Unicode characters you can use in Jupyther Lab and Julia language:

Greek Alphabet

α   \alpha
β   \beta
γ   \gamma
∇   \delta

ϵ \epsilon

ζ \zeta

η \eta

θ \theta

ι \iota

κ   \kappa (should look like 'k')
λ   \lambda
μ   \mu

ν nu (not supported in Jupyther)

ξ \xi

ο omnicron (not supported in Jupyther)

π \pi

ρ \rho

σ \sigma

τ \tau

υ upsilon (not supported in Jupyther)

ϕ \phi

χ \chi

ψ \psi

ω \omega

Hebrew

ℵ \aleph wiki

Machine Learning

x̂ x\hat

Math

∡   \measuredangle
∑   \sum
±   \pm (plus minus)

≠ \ne

≥ \ge (greater or equal)

≤ \le (less or equal)
⌀ \diameter
0° 0\degree

Sets

∈ \in

Logic

∴ \therefore

⊻ \xor

Sublime

Download:

https://www.sublimetext.com/

Terminus

SendCode

Sublime

Download:

https://www.sublimetext.com/

Terminus

SendCode

Using Linear Regression with Julia Language

Linear Regression

What is a Linear Regression?

Watch:

https://www.youtube.com/watch?v=ZkjP5RJLQF4

Selecting the best "fit" line to predict the data.

Fitting a linear regression model to a numerical data set.

Calculating the deviations (the error)

sum of squares

References:

Using Linear Regression with Julia Language

Linear Regression

What is a Linear Regression?

Watch:

https://www.youtube.com/watch?v=ZkjP5RJLQF4

Selecting the best "fit" line to predict the data.

Fitting a linear regression model to a numerical data set.

Calculating the deviations (the error)

sum of squares

References:

The Boston Housing Data Set

Download "boston.mat"

http://www0.cs.ucl.ac.uk/staff/M.Herbster/boston

The Boston Housing Data Set

Download "boston.mat"

http://www0.cs.ucl.ac.uk/staff/M.Herbster/boston

Juno

Read page:

https://docs.junolab.org/latest/man/installation/

Install Atom

https://atom.io/

Juno

Read page:

https://docs.junolab.org/latest/man/installation/

Install Atom

https://atom.io/

pose

Body pose recognition has applications in:

image compression when only the position of the subject is needed
social signaling
ADAS driver monitoring system (DMS)

The human body pose consists of:

body detection in the image
body segmentation from the image
mapping the 2D image to the 3D model
tracking the pose over time (movie)

photo credit: posetrack.net

REFERENCES

pose

Body pose recognition has applications in:

image compression when only the position of the subject is needed
social signaling
ADAS driver monitoring system (DMS)

The human body pose consists of:

body detection in the image
body segmentation from the image
mapping the 2D image to the 3D model
tracking the pose over time (movie)

photo credit: posetrack.net

REFERENCES

Whimsical Mind Graph - Explore Machine Learning

As I am studying Machine Learning I am creating the mind graph à la Daniel Bourke:

https://whimsical.com/K85bjCAHxKg4FZBf3YW7en

Whimsical Mind Graph - Explore Machine Learning

As I am studying Machine Learning I am creating the mind graph à la Daniel Bourke:

https://whimsical.com/K85bjCAHxKg4FZBf3YW7en

Julia Flux Meatalhead.jl

## Uncomment the lines below if you have not run it previously
# import Pkg; Pkg.add("Metalhead")
# import Pkg; Pkg.add("QuartzImageIO")

using Metalhead
using Metalhead: classify
vgg = VGG19()

## local file in this lab's folder
image = load("skunk001.jpeg")
classify(vgg, image)

output:

Julia Flux Meatalhead.jl

## Uncomment the lines below if you have not run it previously
# import Pkg; Pkg.add("Metalhead")
# import Pkg; Pkg.add("QuartzImageIO")

using Metalhead
using Metalhead: classify
vgg = VGG19()

## local file in this lab's folder
image = load("skunk001.jpeg")
classify(vgg, image)

output:

Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions... Installed CUDAapi ─ v4.0.0 Updating `~/.julia/environments/v1.4/Project.toml` [3895d2a7] + CUDAapi v4.0.0 Updating `~/.julia/environments/v1.4/Manifest.toml` [3895d2a7] + CUDAapi v4.0.0

using CUDAapi
has_CUDA = "false"
if has_cuda()
@info "CUDA is on"
has_CUDA = "true"
end
println("it is ", has_CUDA, " that I have CUDA.")

Julia CUDA

import Pkg
Pkg.add("CUDAapi")

using CUDAapi
has_CUDA = "false"
if has_cuda()
@info "CUDA is on"
has_CUDA = "true"
end
println("it is ", has_CUDA, " that I have CUDA.")

Heaven 12 1/2 design by Joel White - DIY part 1

Episode 1 choosing the boat

My interests

Filtering DataFrame

New Syntax

Deprecated Syntax

Filtering DataFrame

New Syntax

Deprecated Syntax

Installation

Installation

A simple way of creating an array

Float64, 2-dimensional, 4-rows, 3-columns

Abstract Type Integer

Concreate Type Int64

A simple way of creating an array

Float64, 2-dimensional, 4-rows, 3-columns

Abstract Type Integer

Concreate Type Int64

Greek Alphabet

Hebrew

Machine Learning

Math

Sets

Logic

Greek Alphabet

Hebrew

Machine Learning

Math

Sets

Logic

Linear Regression

What is a Linear Regression?

Selecting the best "fit" line to predict the data.

Calculating the deviations (the error)

Linear Regression

What is a Linear Regression?

Selecting the best "fit" line to predict the data.

Calculating the deviations (the error)

The Boston Housing Data Set

The Boston Housing Data Set