some questions about connections between client and server

[httaotao]

hi, today when I saw the dev-session 7, I had some questions about the connections between client and server.

I made a small test about it.

root@gfs05:~# gluster peer status 
Number of Peers: 4

Hostname:  10.0.12.2
Uuid: 69da12d3-7d23-408b-add3-e20e60ae725f
State: Peer in Cluster (Connected)

Hostname: 10.0.12.9
Uuid: 3ed7d443-1312-4098-bd50-2a9c98ada9fd
State: Peer in Cluster (Connected)

Hostname: 10.0.12.12
Uuid: f1c72979-3502-41fb-83f2-45f9d2e47cc2
State: Peer in Cluster (Connected)

Hostname: 10.0.12.3
Uuid: 62a37f8c-77d2-4c17-93bb-e5ea1ab79024
State: Peer in Cluster (Connected)


root@gfs05:~# gluster volume info test-mem-leak
 
Volume Name: test-mem-leak
Type: Replicate
Volume ID: 139eee52-7c02-4e27-b901-add479f2d0b9
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.0.12.2:/glusterfs/test-mem-leak
Brick2: 10.0.12.9:/glusterfs/test-mem-leak
Brick3: 10.0.12.12:/glusterfs/test-mem-leak
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on
diagnostics.brick-log-level: DEBUG
diagnostics.client-log-level: DEBUG

then I mounted the volume in client and I checked the status clients info

root@gfs06:~# gluster volume status test-mem-leak clients 
Client connections for volume test-mem-leak
----------------------------------------------
Brick : 10.0.12.2:/glusterfs/test-mem-leak
Clients connected : 7
Hostname                                               BytesRead    BytesWritten       OpVersion
--------                                               ---------    ------------       ---------
10.0.12.2:49108                                          1001276         1163212           90000
10.0.12.12:49126                                            2800            2640           90000
10.0.12.9:49120                                             1716            1808           90000
10.0.12.9:49122                                         19476052         9891752           90000
10.0.12.7:49110                                             1252             972           90000
10.0.12.3:49109                                             1252             972           90000
10.0.12.6:49143                                             2628            2204           90000
----------------------------------------------
Brick : 10.0.12.9:/glusterfs/test-mem-leak
Clients connected : 7
Hostname                                               BytesRead    BytesWritten       OpVersion
--------                                               ---------    ------------       ---------
10.0.12.9:49123                                          1002452         1165336           90000
10.0.12.2:49118                                             3884            3268           90000
10.0.12.12:49125                                            2800            2640           90000
10.0.12.9:49121                                         19520780        10748578           90000
10.0.12.7:49109                                             1252             972           90000
10.0.12.3:49107                                             1252             972           90000
10.0.12.6:49142                                             2628            2204           90000
----------------------------------------------
Brick : 10.0.12.12:/glusterfs/test-mem-leak
Clients connected : 6
Hostname                                               BytesRead    BytesWritten       OpVersion
--------                                               ---------    ------------       ---------
10.0.12.12:49124                                         1000192         1162584           90000
10.0.12.2:49107                                             3884            3268           90000
10.0.12.9:49112                                             1716            1808           90000
10.0.12.9:49104                                         19476108         9891752           90000
10.0.12.7:49107                                             1252             972           90000
10.0.12.3:49106                                             1252             972           90000
----------------------------------------------

there are some questions:

1. why do the bricks have the connection with other nodes which are not the brick node but in the gluster cluster.

2. how to understand the status clients info.
   For example

Brick : 10.0.12.12:/glusterfs/test-mem-leak
Hostname                                               BytesRead    BytesWritten       OpVersion
--------                                               ---------    ------------       ---------
10.0.12.7:49107                                             1252             972           90000

Does it mean that there is a connection between 10.0.12.12 and 10.0.12.7

3. if I mount a volume with non-brick ip but it is in gluster cluster, Is this a forwarding request or replace the 'mount ip' into real brick ip ?
4. why are some brick different that it have two connections but others just has one ? such as 10.0.12.9

thanks.

[pranithk]

1. why are the bricks have the connection with other nodes which are not the brick node but in the gluster cluster.

Self-heal daemon is an internal client which performs heals which runs on each of the brick nodes. So there would be connection from this process to the brick processes, so you see them listed here.

2. how to understand the status clients info.
   For example

Brick : 10.0.12.12:/glusterfs/test-mem-leak
Hostname                                               BytesRead    BytesWritten       OpVersion
--------                                               ---------    ------------       ---------
10.0.12.7:49107                                             1252             972           90000

Does it mean that there is a connection between 10.0.12.12 and 10.0.12.7

Yes

1. if I mount a volume with non-brick ip but it is in gluster cluster, Is this a forwarding request or replace the 'mount ip' into real brick ip ?

Didn't understand this question. Could you give an example?

2. why are some brick different that it have two connections but others just has one ? such as 10.0.12.9

Do you have a mount on 10.0.12.9? Looking at the bytesread/written it looks like a mount.
10.0.12.9:49120 1716 1808 90000
10.0.12.9:49122 19476052 9891752 90000 <<<---------

thanks.

[httaotao]

1. if I mount a volume with non-brick ip but it is in gluster cluster, Is this a forwarding request or replace the 'mount ip' into real brick ip ?

Didn't understand this question. Could you give an example?

For example, if there are five nodes in gluster cluster which hostname are gfs{1..5} , there is a volume test which is replica 3 and the bricks node are in gfs{1..3}.

When I mount the volume

#mount -t glusterfs gfs4:test  /mnt/xxxx

there will have the connections with nodes gfs4 and gfs5 but they are not the brick node.

So if we do the operations, such as touching a file or cating a file, will these requests be forwarded to this node gfs4 an then proxy the requests to the brick node ?
Or the messages will sent to the brick node directly
Which one is correct ?

Self-heal daemon is an internal client which performs heals which runs on each of the brick nodes. So there would be connection from this process to the brick processes, so you see them listed here.

By the way, in my opinions, for self-heal daemon, the none-brick nodes which don't have the brick in this node but are in gluster cluster, only keep the volume info in /var/lib/glusterd/vols and they don't contain the metada and actual data , right ?
So if we heal the volume with AFR , what do the none-brick processses do ?

thanks.

[pranithk]

1. if I mount a volume with non-brick ip but it is in gluster cluster, Is this a forwarding request or replace the 'mount ip' into real brick ip ?

Didn't understand this question. Could you give an example?

For example, if there are five nodes in gluster cluster which hostname are gfs{1..5} , there is a volume test which is replica 3 and the bricks node are in gfs{1..3}.

When I mount the volume

#mount -t glusterfs gfs4:test  /mnt/xxxx

there will have the connections with nodes gfs4 and gfs5 but they are not the brick node.

So if we do the operations, such as touching a file or cating a file, will these requests be forwarded to this node gfs4 an then proxy the requests to the brick node ?
Or the messages will sent to the brick node directly
Which one is correct ?

They are sent to the bricks directly. The IP you give at the time of mount command is used to get the volfile for the mount process.

Self-heal daemon is an internal client which performs heals which runs on each of the brick nodes. So there would be connection from this process to the brick processes, so you see them listed here.

By the way, in my opinions, for self-heal daemon, the none-brick nodes which don't have the brick in this node but are in gluster cluster, only keep the volume info in /var/lib/glusterd/vols and they don't contain the metada and actual data , right ?
So if we heal the volume with AFR , what do the none-brick processses do ?

They do nothing.

thanks.

[httaotao]

Self-heal daemon is an internal client which performs heals which runs on each of the brick nodes. So there would be connection from this process to the brick processes, so you see them listed here.

By the way, in my opinions, for self-heal daemon, the none-brick nodes which don't have the brick in this node but are in gluster cluster, only keep the volume info in /var/lib/glusterd/vols and they don't contain the metada and actual data , right ?
So if we heal the volume with AFR , what do the none-brick processses do ?

They do nothing.

So if they do nothing, I am confused and I don't understand , why do the bricks have the connection with other nodes which are not the brick nodes but in the gluster cluster ? Could you tell more details about it ?

thanks.

[pranithk]

This is an optimization problem. At the moment self-heal daemon connects to all bricks. Feel free to send a patch to optimize this.

[pranithk]

Just looked at the code, it is a bit involved, maybe I will take care of this when I get time. Could you raise an issue?

[httaotao]

Just looked at the code, it is a bit involved, maybe I will take care of this when I get time. Could you raise an issue?

ok, I raise an issuse here 2594

By the way, I find that quota daemon also has the same problem.
thanks.