CUDA GPU Concurrent (parallel) Programming

C, C++, and Python 3 code running asynchronously using

• threads
• queues
• other concurrent programming techniques

Relevance

• CUDA
• OpenCL
• Metal
• OpenAcc
• PyCUDA
• jCuda

Hardware:

• AMD
• Apple
• FPGA
• multi-core CPUs

Pitfalls of Concurrent Programming

• race conditions
• the expected order of thread operations is not followed
• resource contention
• two or more threads attempt to modify the same memory
• deadlock
• one or more processes are blocked by waiting for a resource
• live locks
• two or more processes are stuck in a loop, but cannot finish while waiting for resources
• resource over-utilization
• too few or too many threads, context switching
• memory required is too large
• memory changes too often
• resource under-utilization
• sitting idle

Semaphore, for all intents and purposes, is an atomic variable that has more than one thread requiring it, which means that a predefined number of threads can use the semaphore to enter a critical section of code. 

A lock is the more restrictive parent asynchronous mechanism for a single thread to enter a critical section of code. Thus a semaphore is a more relaxed form of lock.

Concurrent Programming Algorithms

• Dining Philosophers
• multiple processes require the same resources to complete their jobs
• Producer-Consumer
• consumers need to read the data
• in order
• no duplication
• Producers add data in order it needs to be processed
• Sleeping Barber
• customers are waiting
• single barber
• if the barber is sleeping customer should wake him
• If there is no space in the queue, customers are not added
• Data and Code Synchronization

References

1. https://www.coursera.org/learn/introduction-to-concurrent-programming/
2. https://cuda.juliagpu.org/stable/
3. https://cplusplus.com/
4. https://www.w3schools.com/cpp/
5. 
   

How to syntax color the code inside the blockquote HTML tag?

That is a good, if not easy, problem to solve!

In the result, I would like to take the following code in <blockquote> tag

function fold(initial, desired)
    folds = 0 # initial number of folds
    achieved = initial # initial thickness
    results = []
    while true
        achieved = achieved * 2 # fold
        #if achieved >= desired break end # if you want to exit before achieving
        push!(results, achieved) # add to results
        folds = folds + 1 # increment
        if achieved >= desired break end # if you want to exit after achieving
    end
    return results
end

 and color it by the language-specific keywords to look something like that:

What are the steps I would take?

1. In JavaScript, I would find the next blockquote element
2. Extract the text from it
3. Parse the comments ( from # to the end of the line)
4. Parse text by the white space (preserving white space)
5. Compare each WORD token to the dictionary of the reserved word
6. Do something about variable names
7. Do something about variable values
8. Wrap reserved words in <span color> tags
9. Re-assemble to content of the <blockquote>

Since this is a several-evenings project, I will leave it for later.

If you are doing this, or have done something similar, please let me know in the comments.

How to calculate number of paper folds to get thickness equal to distance to the moon?

I wanted to try how a programming exercise would translate to a blog post.

I opened Brilliant.com and the first exercise was a puzzle of folding paper. I decided to solve it using Julia language.

How thick is a sheet of printer paper?

For this exercise, let's assume 0.097mm to 0.1mm depending on the "weight" of the paper.

# define known variables
building = 828             # units: m -- Burj Khalifa building
moon = 391000000     # units: m; distance to the moon
sheet = 0.1 / 1000        # unit: m -- thickness of 100 sheets is 1 cm

function fold(initial, desired)

folds = 0                                              # initial number of folds

achieved = initial                                # initial thickness

results = []

while true

achieved = achieved * 2               # fold paper

if achieved >= desired break end # check if time to exit the loop

push!(results, achieved)               # add to results

folds = folds + 1                           # increment

end

return results 

end 

results = fold(sheet, moon)                         # replace moon with building
folds = size(results)[1]                               # size is a tuple, get first element
thickness = results[folds] /1_000_000
print("We used $folds to get $thickness thousand km.")
println()
data = results ./ 1000.0                               # data need to be Array{Float64, 1}, in km
increments = size(results)[1]

using Plots

using PlutoUI

How to create a slider in Julia?

I use that slider to show me the progression of thickness from 1 fold to as many as it takes.

@bind slider_ui Slider(1:increments) # one widget per cell

How to create a drop-down menu in Julia?

@bind plot_type Select(["scatter","line","bar"])

How to draw a simple diagram to visualize the data?

 

plot(

data[1:slider_ui], 

seriestype=Symbol(plot_type), 

title="x=$slider_ui folds, y=km, $plot_type plot"

)

 

So, what is the number of folds to get to the moon?

Only 42 folds will take us to the moon and a bit further!

Pluto for kids with Julia language

I have installed Pluto.jl for Julia language and while going thru the exercises provided by the creators, I realized that it would be a great tool to teach my kids programming. 

And, it is not any toy programming, programming in the best of the scientific languages!

How does the Pluto.jl work?

You write one, or multiple, lines of code in each "cell" and execute it.

Once you finish writing and executing a line, you can HIDE the code and only leave the result.

All other cells are able to "see" what you wrote and evaluate it, providing immediate feedback. 

The following two images illustrate this better than I can describe in words:

After adding the necessary code I was able to get a good answer:

Is Pluto.jl only for Mathematics?

Absolutely not! It is a generic programming environment. You can create word games, and anything else that you can imagine. I use it as my primary programming tool.

What are the next steps with Pluto.jl for kids?

Now, I have to write a few notebooks that are age-appropriate for my kids.
I will post them online so you can download them, but I believe YOU should learn how to create them, too. You might find a new great hobby for yourself.

Look inside the links I posted below for any updates.

Will I maintain this post with new updates on Pluto.jl for kids?

Most likely not, please check the links below for the updates.

References

• Pluto.jl folder (this article):
  https://github.com/UkiDLucas/QUANTA_QUALIA_Julia/tree/master/src/pluto
• Pluto.jl  for kids:
  https://github.com/UkiDLucas/QUANTA_QUALIA_Julia/tree/master/src/pluto/Pluto.jl%20for%20kids

Installing Pluto.jl notebook for Julia language

 What is pluto.jl

TBD

How to install Julia?

Install Julia language from https://julialang.org/

Start Julia 

    I can do that from Terminal,

    or by clicking an icon

Once Julia's prompt opens,

    click the closing bracket ] to enter the "package mode"

How to Install pluto.jl?

Simply type 

(@v1.7) pkg> add Pluto.jl

The installation is very fast.

Press BACKSPACE to exit the package mode

 Type 

julia> using Pluto
julia> Pluto.run()

At this point, the pluto website should open in your browser:

How to install Julia language Pluto notebook?

 What is pluto.jl

TBD

How to install Julia?

Install Julia language from https://julialang.org/

Start Julia 

    I can do that from Terminal,

    or by clicking an icon

How to Install pluto.jl?

Once Julia's prompt opens,

    click the closing bracket ] to enter the "package mode"

% julia
(@v1.7) pkg> add Pluto.jl

This will take a few moments.

How to stop a Julia prompt? 

julia> exit()

The installation is very fast.

Press BACKSPACE to exit the package mode

What do I need to do before running Pluto notebook?

I have learned that it is wise to change to the directory you want to work with. 

The support of Pluto notebook to jump thru many directories up or down is very sketchy.

cd /Volumes/_MY_DIRECTORIES_/Julia/Brilliant/
Brilliant % julia

julia> pwd()

How to run Pluto notebook in Julia?

julia> using Pluto
julia> Pluto.run()

At this point, the pluto website should open in your browser:

How to execute Julia code in VS code editor with Julia Language Support plugin?

In VS code,

with the Julia Language Support plugin installed,

with Julia installed on my macOS,

% which julia
/Applications/Julia-1.7.app/Contents/Resources/julia/bin/julia

 When trying to run (Shift-Enter, or RUN icon) Julia in VS code I got this error:

command 'language-julia.executableCodeBlockOrSelectionAndMove' not found

Solution

• I reinstalled the VS code plugin
• I rechecked the PATH (see above)
• I CLOSED and RE-OPENED the VS code

And voila, it works. Not very scientific, but I am happy. 

References

• Unresolved bug: https://github.com/julia-vscode/julia-vscode/issues/2116

How to execute Julia code in VS code editor with Julia Language Support plugin?

In VS code,

with the Julia Language Support plugin installed,

with Julia installed on my macOS,

% which julia
/Applications/Julia-1.7.app/Contents/Resources/julia/bin/julia

 When trying to run (Shift-Enter, or RUN icon) Julia in VS code I got this error:

command 'language-julia.executableCodeBlockOrSelectionAndMove' not found

Solution

• I reinstalled the VS code plugin
• I rechecked the PATH (see above)
• I CLOSED and RE-OPENED the VS code

And voila, it works. Not very scientific, but I am happy. 

References

• Unresolved bug: https://github.com/julia-vscode/julia-vscode/issues/2116

How to add "julia" executable to the PATH on macOS

 

% nano .zprofile

#### Julia - Uki 2022-06-26 ####
## set PATH to "julia" executable so it is available directly from Terminal
export PATH="\$PATH:/Applications/Julia-1.7.app/Contents/Resources/julia/bin"

Make sure to change to the version of Julia you have, check the path.
After adding these lines, save it in NANO by pressing 

control x 



 Do not forget to double-check your work.

% cat .zprofile

How to add "julia" executable to the PATH on macOS

 

% nano .zprofile

#### Julia - Uki 2022-06-26 ####
## set PATH to "julia" executable so it is available directly from Terminal
export PATH="\$PATH:/Applications/Julia-1.7.app/Contents/Resources/julia/bin"

Make sure to change to the version of Julia you have, check the path.
After adding these lines, save it in NANO by pressing 

control x 



 Do not forget to double-check your work.

% cat .zprofile

No more development on Turi Create

Since I have been using Apple's machine learning framework, Turi Create in the past,
today, I went to check if there is any new development and if there is support for Julia language. 

Unfortunately, it seems like it is a NO on both counts, there is a minimal code activity since the beginning of 2020, too bad.

https://github.com/apple/turicreate/graphs/code-frequency

Please let me know in the comments if you are still using it and what for.

No more development on Turi Create

Since I have been using Apple's machine learning framework, Turi Create in the past,
today, I went to check if there is any new development and if there is support for Julia language. 

Unfortunately, it seems like it is a NO on both counts, there is a minimal code activity since the beginning of 2020, too bad.

https://github.com/apple/turicreate/graphs/code-frequency

Please let me know in the comments if you are still using it and what for.

Julia language: DataFrames: insert a column

 The following code 

1. inserts a new column into DataFrame named df
2. inserts this column as a first (col_ind =1) column
3. names this column Rata_Die
4. populates the column values with an array of Int64 zeros (0)
5. makes sure the name of the column is unique

col_ind = 1

insertcols!(df, col_ind, :Rata_Die => zeros(Int64, record_count); makeunique = true )

Julia language: DataFrames: insert a column

 The following code 

1. inserts a new column into DataFrame named df
2. inserts this column as a first (col_ind =1) column
3. names this column Rata_Die
4. populates the column values with an array of Int64 zeros (0)
5. makes sure the name of the column is unique

col_ind = 1

insertcols!(df, col_ind, :Rata_Die => zeros(Int64, record_count); makeunique = true )

Julia language UNIX date time

Unix time date format is used in many applications, including Yahoo finance.

using Dates, Printf

unix_date = @sprintf("%.0f", Dates.datetime2unix(Dates.now()))

println( unix_date ) # String

Output:

1603013077

Julia language UNIX date time

Unix time date format is used in many applications, including Yahoo finance.

using Dates, Printf

unix_date = @sprintf("%.0f", Dates.datetime2unix(Dates.now()))

println( unix_date ) # String

Output:

1603013077

Opening the Jupyter notebook in a different drive on Windows 10.

Assuming that you have Anaconda 3 installed, 
you can open the "Anaconda Prompt" window and start jupyter,
however, it opens in the default C: drive.

Anaconda prompt does not allow you to change the drive from the command line. (WTF !?!)

The simplest solution I found was to specify the new drive (R: for Repos) in Jupyter startup:

> jupyter notebook --notebook-dir 'R:'

Sidenote: 

In case you wonder why I am putting up with Windows 10 and not working on my MacOS, I wanted to try the NVidia GPU performance with Julia in the Jupyter notebook. 

Opening the Jupyter notebook in a different drive on Windows 10.

Assuming that you have Anaconda 3 installed, 
you can open the "Anaconda Prompt" window and start jupyter,
however, it opens in the default C: drive.

Anaconda prompt does not allow you to change the drive from the command line. (WTF !?!)

The simplest solution I found was to specify the new drive (R: for Repos) in Jupyter startup:

> jupyter notebook --notebook-dir 'R:'

Sidenote: 

In case you wonder why I am putting up with Windows 10 and not working on my MacOS, I wanted to try the NVidia GPU performance with Julia in the Jupyter notebook. 

Julia DataFrames

 

Filtering DataFrame

New Syntax

 

Syntax explained:

1. from original "data" (type DataFrames)
2. create new "dataA" (type DataFrames)
3. filter indices in column "Treatment" with values equal to "A"
4. include ALL COLUMNS indicated by ":"
5. Show the first 6 rows

dataA = data[isequal.(data.Treatment, "A"), : ] 

first(dataA, 6)

6 rows × 5 columns

	Age	WCC	CRP	Treatment	Result
	Int64	Float64	Int64	String	String
1	37	10.2	30	A	Worse
2	67	10.4	70	A	Worse
3	42	10.4	100	A	Static
4	56	12.5	80	A	Worse
5	60	9.5	0	A	Static
6	62	13.7	10	A	Worse

 

Deprecated Syntax

# dataA = data[data[:Treatment] .== "A", :] 

 

Warning: `getindex(df::DataFrame, col_ind::ColumnIndex)` is deprecated, use `df[!, col_ind]` instead.

Julia DataFrames

 

Filtering DataFrame

New Syntax

 

Syntax explained:

1. from original "data" (type DataFrames)
2. create new "dataA" (type DataFrames)
3. filter indices in column "Treatment" with values equal to "A"
4. include ALL COLUMNS indicated by ":"
5. Show the first 6 rows

dataA = data[isequal.(data.Treatment, "A"), : ] 

first(dataA, 6)

6 rows × 5 columns

	Age	WCC	CRP	Treatment	Result
	Int64	Float64	Int64	String	String
1	37	10.2	30	A	Worse
2	67	10.4	70	A	Worse
3	42	10.4	100	A	Static
4	56	12.5	80	A	Worse
5	60	9.5	0	A	Static
6	62	13.7	10	A	Worse

 

Deprecated Syntax

# dataA = data[data[:Treatment] .== "A", :] 

 

Warning: `getindex(df::DataFrame, col_ind::ColumnIndex)` is deprecated, use `df[!, col_ind]` instead.