Wednesday, 31 March 2021

Docker compose

This is a study note from a tutorial video in YouTube: https://www.youtube.com/watch?v=fqMOX6JJhGo&t=4723s

Docker compose is used to run several differences services in one .yml file. 

 docker run WeibotopHistory 
 docker run mariadb 

docker-compose.yml
services:
    web:
        image: "WeibotopHistory"
    database:
    
    image: "mariadb"

Use "docker-compose" command to bring up all the application stacks in yml files.

 docker-compose up 

How to compose yml files?

To explain how to compose the yml files, we use the voting app example. The structure of voting app is like below:


These connections between difference services can be described using "run --link" command like below:
 docker run -d --name=redis redis 
 docker run -d --name=db postgres:9.4 
 docker run -d --name=vote -p 5000:80 --link redis:redis voting-app 
 docker run -d --name=result -p 5001:80 --link db:db result-app 
 docker run -d --name=worker --link db:db --link redis:redis worker 

Above commands can be described in yml file like below:

docker-compose.yml

redis: # This is the name of container, it is
# corresponding to "--name" in "run" command
image: redis # This is the image name
db:
image: postgres:9.4
vote:
image: voting-app
ports: # This is corresponding to "-p" in "run" command
- 5000:80
links: # This is corresponding to "--link" in "run" command
- redis
result:
image: result
ports:
- 5001:80
links:
- db
worker:
image: worker
links:
- redis
- db

However if some of the images have not been built yet, you can use "build" keywords instead of "image" to build and run. For example, since the voting-app, result app, and worker app are written by developer, they may not be built into images yet. Then we can modify above yml file to build the images.

redis:
image: redis
db:
image: postgres:9.4
vote:
build: ./vote # Here will build the image with directory "vote"
ports:
- 5000:80
links:
- redis
result:
build: ./result
ports:
- 5001:80
links:
- db
worker:
build: ./worker
links:
- redis
- db

Above format is for yml version 1.

In version 2, there are some differences:

  1. All the containers need to be defined in "services" section;
  2. If you need dependencies, you can use "depends_on" attribute to specify the services that need to be run before this service;
For example, the above yml can be modified in version 2 like below:

version: '2' # Here needs to specify the version using string
services: # All services need to be put under "services"
    redis:
    image: redis
    db:
    image: postgres:9.4
    vote:
    image: voting-app
    ports:
    - 5000:80
    depends_on: # Here to tell to run redis before run this
    - redis
    result:
    image: result
    ports:
    - 5001:80
    depends_on:
    - db
    worker:
    image: worker
    depends_on:
    - redis
    - db

Version 3 is similar to version 2. However some tags are removed and simplified.

Networking in docker compose

Let's use an example directly. 


If you want to create a network structure like above, you need to add"networks" tag in the yml files.

version: 2
services:
    redis:
    image: redis
        networks: # Here needs to specify the network you want
            - back-end # to use from the networks defined
    db:
    image: postgres:9.4
        networks:
            - back-end
    vote:
    image: voting-app
    ports:
    - 5000:80
    depends_on:
    - redis
        networks:
            - front-end
            - back-end
    result:
    image: result
    ports:
    - 5001:80
    depends_on:
    - db
        networks:
            - front-end
            - back-end
    worker:
    image: worker
    depends_on:
    - redis
    - db
        networks:
            - back-end
networks: # Here to define the networks
    front-end:
    back-end:



Reference: https://www.youtube.com/watch?v=fqMOX6JJhGo&t=4723s

Monday, 29 March 2021

Networking in docker

This is a study note from a tutorial video in YouTube: https://www.youtube.com/watch?v=fqMOX6JJhGo&t=4723s

There are three options of networks in docker. Bridge(default), host and none.

Bridge

 docker run ubuntu 

Bridge is the default network in docker. 

%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22172.17.0.3%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22380%22%20y%3D%22330%22%20width%3D%2240%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E




Internally docker will give container IP addresses ranging with 172.17.0.X. Docker0(172.17.0.1) is a bridge which is used by all containers to communicate with each other.

If you want the container to communicate externally, ports of the containers need to mapped to the ports of the host. For example, docker run -p 8282:8080 ubuntu.

In this mode, containers are isolated with the host. You can only use "run -p" to map to the host port to make it accessible.

You can create isolation in the docker network as well. In some cases, you may want certain containers connected through different bridge so that they can be isolated from other containers. By default, the docker will only create one bridge for all containers. But you can create a new bridge using below command.

 docker network create --driver bridge --subnet 102.18.0.1/16 --gateway 102.18.0.1 custom-isolated-network 


You can use docker network ls to list all the networks.

You can use docker run --network custom-isolated-network mysql:5.6 to run container in a specific network.

Host

%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22172.17.0.3%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22380%22%20y%3D%22330%22%20width%3D%2240%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E Web container can be associated with the host using below command:

 docker run --network=host ubuntu 


In this mode, some ports of host shared the same port numbers with containers, which means if the client access the port number 5000 on the host, it will automatically directed to the relevant containers. 

None

 docker run --network=none ubuntu 

In none network mode, the containers are totally isolated from each other and the outside world. They can only run programs locally in their own containers.



Reference: https://www.youtube.com/watch?v=fqMOX6JJhGo&t=4723s

Differences between CMD and ENTRYPOINT in Docker

This is a study note from a tutorial video in YouTube: https://www.youtube.com/watch?v=fqMOX6JJhGo&t=4723s

In Dockerfile, you can specify commands to execute like below:

Dockerfile:

FROM java:8
EXPOSE 8080
CMD ["java", "-jar", "/WeibotopHistorySpring.jar"]

The example has a command to execute a java program using JSON format. The first parameter "java" is an executable and the others are the parameters of "java" program. 

However in some cases we may want to change the parameter of the executable during a docker run. For example, you have below simple Dockerfile:

FROM Ubuntu
EXPOSE 8080
CMD ["sleep", "5"]

When run above Dockerfile, it will run a container and sleep for 5 seconds. What if we want it to sleep 10 seconds instead? You can run a container with a sleep 10 command because the command appended in docker run command will replace the default command in Dockerfile.

 docker run ubuntu-sleep sleep 10 

However the "sleep" command in the end looks a bit redundant because we know this docker container is used to do sleep. We can remove the sleep instruction from the run command by using ENTRYPOINT.

FROM Ubuntu
EXPOSE 8080
ENTRYPOINT ["sleep"]

So now you can simply append parameter "10" at the end of run command to change the sleep interval.

 docker run ubuntu-sleep 10 

There is a problem with the above modification using ENTRYPOINT to replace CMD because if no time is specified in the docker run command. The container will have a exception error because it misses operand for sleep command in container. You can combine both CMD and ENTRYPOINT to solve this problem:

FROM Ubuntu
EXPOSE 8080
ENTRYPOINT ["sleep"]
CMD ["5"]

In this case, when you run docker container  docker run ubuntu-sleep  directly, it will use the default 5 seconds parameters. If you append a time parameter  docker run ubuntu-sleep 10  it will sleep for 10 seconds intead.

Conclustion:

ENTRYPOINT appends the input from the docker run command, while CMD replaces the default command in Dockerfile with the input from the docker run command.


References: https://www.youtube.com/watch?v=fqMOX6JJhGo&t=4723s





Thursday, 18 March 2021

Docker commands

This post is written as a study note of docker tutorial in YouTube https://www.youtube.com/watch?v=fqMOX6JJhGo&t=1182s, and also cheat sheet for future references during work.

 run - start a container

 docker run image_name 
This command will try to find the image locally in the docker host first. If it cannot find it, it will look for it in docker hub. If image is found in docker hub, it will pull the image down from there and run it. Once the image is downloaded from docker hub, the second time run will not trigger the downloading again from docker hub.

ps - list containers

 docker ps 
This command is used to list all the running containers with some basic information. It is kinda like "ls" command to list all the files in the current folder.

 docker ps -a 
This command is used to list all the running containers as well as the recent stopped containers.

inspect - inspect container

 docker inspect silly_sammet 
You can use this command to check more details of a container.

stop - stop a container

 docker ps 

First use "ps" command to get the container name.



 docker stop silly_sammet 
Then use "stop" command to stop a container.

 docker ps -a 
Then use "ps -a" command to confirm the container is stopped.

rm - remove a container

 docker rm silly_sammet 
This command will remove the container from the memory completely.

 docker ps -a 
Then use "ps -a" command to confirm the container is removed.

images - list images

 docker images 
This command is used to list the images in the docker host.

rmi - remove images

Before remove the image, make sure the container from this image is stopped.
 docker rmi docker/whalesay 
Use this command to remove the image in the host.

 docker images 
Then you use "images" command to confirm the deletion.

pull - download an image

 docker pull docker/whalesay 
This command can be used to download docker image from docker hub directly, unlike docker run command to check the image locally then go search in docker hub. However this command will not run the image after downloading.

exec - execute a command

"exec" command is used to execute commands in a container. For example, if a ubuntu container is running, you can use "exec" to execute any linux command in this ubuntu container.
 docker ps -a 
Use "ps" command to get the container name. 



 docker exec nifty_bouman cat /etc/hosts 
 Then execute "cat /etc/hosts" with container name nifty_bouman to show the content of file hosts.








run - attach and detach

If you run a container using "docker run ubuntu", it runs a container with "attach" mode, so you cannot do anything except watching the output from the container.
However if you run container with "dettach" mode, you can still use the console with the container running at the background.
  docker run -d ubuntu sleep 100 
The output is:



However in "dettach" mode, you cannot see the log information from the console anymore since it runs in the background. But no worries, you can use below command to see the log:
 docker logs nifty_bouman 

You can use "attach" command to attach the console to a running container.
 docker attach 8ae0d 
You can just provide few characters of the container ID like above. No need to give the full ID.

run - port mapping

 docker run -p 80:5000 simple-webapp 
You can use "run -p" command to map the dock container port (5000) to host port(80). Then the user can use the host IP address + host port to access the application in docker container.
By using this command, you can map different application to different ports on the host like below:
 docker run -p 8000:5000 simple-webapp1 
 docker run -p 8001:5000 simple-webapp2 
 docker run -p 8002:5000 simple-webapp3 

run - set environment variables

 docker run -e APP_COLOR=blue simple-webapp 
Use "run -e" or "run --env" to set environment variables when run a container.

run - volume mapping


Volume mapping in docker allows you to mount a directory in host into docker container. Thus whenever you want to save some data or files into that directory in container, it will actually be saved in the host. The benefit is that when you remove the container, the data is still in the host without impact from the removing.

 docker run -v /opt/datadir:/var/lib/mysql mysql 

In above example, /opt/datadir is a directory in the host, and /var/lib/mysql is a directory in the container.

You can also create a volume in the host first using "volume create" command like below:
 docker volume create data_volume 
This command will create a volume in host in path /var/lib/docker/volumes.
/var/lib/docker
----volumes
--------data_volume
Then use "run -v" command to map volume to directory in a container.
 docker run -v data_volume:/var/lib/mysql mysql 

In fact, even if the volume is not created in the beginning using "docker volume" command, "run -v" command will automatically create a volume in the host (data_volume).

Now there is a new way to map the volume using "run --mount" command, it has more parameters to specify during execution.
 docker run --mount type=bind,source=/opt/datadir,target=/var/lib/mysql mysql 

run - links

Because services are connected to each other for most of the time, we need to specify the relationship between containers when run images.
For example, we have below voting app. It has the relationship like below:

In order to describe the above connections between containers, "run --link" command is used to run containers like below:
 docker run -d --name=redis redis 
 docker run -d --name=db postgres:9.4 
 docker run -d --name=vote -p 5000:80 --link redis:redis voting-app 
 docker run -d --name=result -p 5001:80 --link db:db result-app 
 docker run -d --name=worker --link db:db --link redis:redis worker 


run - limiting resource usage on host

You can use below command to limit the CPU usage:
 docker run --cpus=.5 ubuntu # use up to 50% of the host CPU 
Similarly you can use below command to limit the memory usage:
 docker run --memory=100m ubuntu # use up to 100m bytes of the host memory 

build - build an image

 docker build MyAppDir -t mmumshad/my-custom-app 
Use "build" command to build an image. "MyAppDir" is the directory name contains docker file used to build an image. Then specify the image name after "-t".
Below is an example of docker file.
Dockerfile:
FROM Ubuntu
RUN apt-get update && apt-get -y install python
RUN pip install flask flask-mysql
COPY . /opt/source-code
ENTRYPOINT FLASK_APP=/opt/source-code/app.py flash run




Saturday, 13 March 2021

Array assignment in Go

Array assignment in Go is very different from some other programming languages like Java, C/C+++...  When you assign an array to another array variable in C/C++ or Java, it will not create new copy of the array, the two array variables are referencing the same memory for array.

However in Go, Array in Go is considered as a value instead of a pointer like other programming languages. When assign an array variable to another array variable, it will create a copy of the first array, then assign it to the second array, which means the two array variables are referencing different memories. See the example below:

package main

import (
"fmt"
)

func main() {
var a [3]int = [...]int{1, 1, 1}
var b [3]int
b = a
b[0] = 2
b[1] = 2
b[2] = 2
fmt.Println(a)
fmt.Println(b)

}


The output is:
[1 1 1] [2 2 2]

This also applies to passing array to a function as parameters. It will also create a copy of array, then pass to function calls. 

This certainly makes array copying become much easier. However when you have a big array, it will also slow down your program due to the copying operation.

If you want to do the same like Java and C/C++ to make two array variables to point to the same array, you can do this in Go as well. You can create variable b as a array pointer using *[3]int, and then assign the address of b to a using &. This works similar to C/C++ syntax. Below is the example code:

package main

import (
"fmt"
)

func main() {
var a [3]int = [...]int{1, 1, 1}
var b *[3]int

b = &a
b[0] = 2
b[1] = 2
b[2] = 2

fmt.Println(a)
fmt.Println(b)

}

The output is:
[2 2 2] &[2 2 2]

So the value of array a is changed as well by changing values in b.


Bit manipulation using Enum in Go

 This method is learned from Youtube online course https://www.youtube.com/watch?v=YS4e4q9oBaU&t=5713s

Sometimes developers may want to use one byte to represent a combination of settings. In this case, each bit represents "Yes" or "No". 

So here you can use Enum in Go to define each bit with special meaning quickly. This allows us to manipulate each bit efficiently.

Below is an example code from the video I mentioned in the top.

package main

import (
"fmt"
)

const (
isAdmin = 1 << iota
isHeadquarters
canSeeFinancials

canSeeAfrica
canSeeAsia
canSeeEurope
canSeeNorthAmerica
canSeeSouthAmerica
)

func main() {
var roles byte = isAdmin | canSeeFinancials | canSeeEurope
fmt.Printf("%b\n", roles)

fmt.Printf("Is Admin? %v", isAdmin&roles == isAdmin)
}

The output is:

100101 Is Admin? true


This is a very good way to manage a set of settings like role management, access right management and so on.



How to pass object to between Activity objects without using Serializable or Parcelable

Using Parcelable to pass objects between Activity objects is the correct way to do so. However, occasionally developers may encounter the scenario where the object you want to pass has its class defined in another library which you don't have access to the source code. In this case, you cannot implements Serializable or Parcelable interfaces.

In this case, you can create a wrapper class implement the interfaces. But if you don't want to do this, there is another solution as well. You can use JSON to solve this problem. In sender activity, convert the object you want to pass to JSON string. And in receiver activity, convert the JSON string back to the object. 

Code in sender activity:


Intent intent = new Intent(this, ReceiverActivity.class);
Bundle bundle = new Bundle();
GsonBuilder builder = new GsonBuilder();
Gson gson = builder.create();
String objectToPassStr = gson.toJson(objectToPass); // Convert object to JSON
bundle.putString("param", objectToPassStr);
intent.putExtras(bundle);
this.startActivity(intent);


Code in receiver activity:


Bundle bundle = getIntent().getExtras();
String objectToPassStr = bundle.getString("param");
Gson gson = builder.create();
ObjectToPass objectToPass = gson.fromJson(objectToPassStr, ObjectToPass.class) // Convert
                                                                               JSON to object


So now you can use objectToPass in the receiver activity. However using Parcelable is still better solution if you can find a way to use it.

How to extend MySQL query in C# application

 In C# application, when select a pretty large table using MySQL library, you may get an exception "Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding". This is simply because the select operation take more than 30s, then it will give a timeout exception. 30s is the default setting for MySqlCommand.

To extend the query command is very simple: just set the CommandTimeout property in MySqlCommand to a larger value. Below is the complete code for a query session where you can see that the timeout for query command is extended to the maximum of integer, and then load data into a DataTable object.


string conStr = "Database=10.10.10.10;Data Source=testdb;User Id=testuser;Password=testpw";
MySqlConnection connection = new MySqlConnection(conStr);
connection.Open();
MySqlCommand cmd = new MySqlCommand("SELECT * FROM my_table", connection);
cmd.CommandTimeout = int.MaxValue; //Here execution timeout is set to maximum
DataTable table = new DataTable();
table.load(cmd.ExecuteReader());
connection.Close();





Monday, 8 March 2021

Calculate data size using Enum in Go

This method is learned from Youtube online course https://www.youtube.com/watch?v=YS4e4q9oBaU&t=5713s


I found it quite interesting and a nice demo to understand Enum in Go.

The code is like below:

package main

import (
"fmt"
)

const (
_ = iota //ignore first value by assigning to blank identifier
KB = 1 << (10 * iota)
MB
GB
TB
PB
ZB
YB
)

func main() {
fileSize := 4000000000.
fmt.Printf("%.2fGB", fileSize/GB)
}

Output:
3.73GB

Here constants KB, MB, GB, TB, PB, ZB and YB are used to calculate the responding size by dividing the constant value. Since they are written in upper case, which means they can be used in other packages as well. So it is a very useful tool.













Sunday, 7 March 2021

Variable scope in Go

There are only three levels of variable scope in Go: package level, globe level and block level.


Package Level:

Package level variables are accessible inside the package. They are defined directly on the package level and started with lower case. For example:

package main
import "fmt"

var counter int = 42

func main() {
    fmt.Println(counter);
}


Globe Level:

Globe level variables can be accessed outside of the package. Similar to package level variables, they are also defined directly on the package level, however they must be started in upper case. For example:

package main
import "fmt"

var Counter int = 42 //Here Counter is accessible outside of the package

func main() {
    fmt.Println(Counter);
}

Globe level variables can be accessed by other packages using packagename.VariableName after import the package.

Block Level:

Block level is like local variables in other programming languages. They are defined in a block like functions, if statement or else.

package main
import "fmt"

func main() {
    counter := 42
    fmt.Println(counter);

}






Difference between "docker stop" and "docker kill"

 To stop a running container, you can use either "docker stop" or "docker kill" command to do so. Although it seems doin...