Linux cgroup

The Control Groups (cgroup) framework is a Linux kernel feature to limit, account, and isolate resource usage of groups of processes. A control group is a collection of processes that are bound by the same resource usage constraints. These groups can be hierarchical, where each group inherits its constraints from its parent group. The kernel provides access to multiple controllers (subsystems) through the cgroup interface.

Bbque exploits only four controllers:

  • cpu: regulation of CPUs access (typically, the CPU quota)
  • cpuset: allocation of specific CPUs and memory nodes
  • cpuacct: accounting of CPU usage
  • memory: limitation of memory usage

Depending on the logger verbosity, when starting an application your console could display errors such as “controller not found”. If they are not related to cpu, cpuset, cpuacct or memory, do not worry! Such error messages are displayed because the barbequeRTRM only mounts the controllers that it actually needs.

You won't need to manually exploit this framework; fortunately, Barbeque will manage this low level aspect in a completely transparent way. However, among the interfaces exposed by these controllers a little set is of paramount importance to the user to better understand the underlying mechanisms:

  • cpuset.cpus: the list of CPUs exploitable by the tasks running in this cgroup
  • cpuset.mems: the memory nodes exploitable by the tasks running in this cgroup
  • cpu.cfs_period_us / cpu.cfs_quota_us: the maximum CPU quota, expressed in QUOTA microseconds every PERIOD microseconds
  • memory.limit_in_bytes: the maximum amount of memory exploitable by the tasks running in this cgroup

  1. In this tutorial, the BOSP installation path is considered to be ~/BOSP. If it is not the case, modify the bash commands as needed.
  2. If you have problems accessing or visualizing the cgroups, try doing this tutorial using root privileges.

As already mentioned, you won't need to namually use Control Groups; however, for the sake of clarity, please indulge us and follow this brief tutorial.

Let us have a brief view of the cgroups hierarchy. First of all, you must source the BOSPShell.

$ . ~/BOSP/out/etc/bbque/bosp_init.env

Now, start the bbque daemon.

[BOSPShell ~] \> bbque-startd

When the daemon is started, the Control Groups are mounted in the BOSP mount folder. Let us check its contents:

[BOSPShell ~] \> cd BOSP/out/mnt/cgroup/
[BOSPShell cgroup] \> tree -d bbque/
├── host
├── res
│   ├── node1
│   ├── node2
│   └── node3
└── silos

As you can see, there are four main cgroups mount points:

  • the bbque mount point represents all the system resources, and contains host, res and silos.
  • the host mount point hosts the framework itself, along with the non-integrated applications
  • the res mount point hosts the integrated applications, thus representing the MDEV
  • the silos mount point hosts the suspended applications

As you already saw in the previous tutorial, the MDEV can be partitioned into nodes. In this case, we have three nodes. Given the following BPL, let us check how the resource partition description is reflected in the cgroups.

Obviously, being this example architecture-dependent, your BPL and results will probably be different from these.

# Main TARGET devices:
	| CPUs IDs	| Memory Nodes	| Description
HOST	| 0,4		| 0		| BBQUE Generic 1Core Hyperthreaded Host
MDEV	| 1-3,5-7	| 0		| BBQUE Generic 3Core Hyperthreaded MDEV
# Resources clusterization for MANAGED resources (single node)
	| CPUs IDs	| Time Quota	| Memory Nodes	| Memory (MB)
NODE	| 1,5		| 100		| 0		| 2000
NODE	| 2,6		| 100		| 0		| 2000
NODE	| 3,7		| 100		| 0		| 2000

Let us check, for example, the MDEV parameters.

[BOSPShell cgroup] \> cat bbque/res/cpuset.cpus
[BOSPShell cgroup] \> cat bbque/res/cpuset.mems
[BOSPShell cgroup] \> cat bbque/res/cpu.cfs_period_us 
[BOSPShell cgroup] \> cat bbque/res/cpu.cfs_quota_us 
[BOSPShell cgroup] \> cat bbque/res/memory.limit_in_bytes

The parameters set in the BPL file are correctly set in the cgroups. Note that a cpu.cfs_quota_us set to -1 means infinite CPU quota, or 100% CPU quota. The same meaning can be attributed to the huge number appearing in memory.limit_in_bytes: this value equals 2^64 bytes, the maximum addressable memory quota on the machine. As you have probably guessed, we have no need to limit the MDEV memory usage. In fact, the limitation is enforced at NODE level.

Next tutorial: Creating integrated applications using templates.

docs/bosp/cgroups.txt · Last modified: 2017/05/17 12:23 by jumanix

Page Tools