summaryrefslogtreecommitdiff
path: root/doc/source/admin/unified-limits.rst
blob: 1a37a4c5a205c9d0189eec26d08a5e187e7d67c7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
==============
Unified Limits
==============

.. WARNING::

    The unified limits API is currently labeled as experimental and can change
    in backwards incompatible ways. After we get feedback on the intricacies of
    the API and no longer expect to make API breaking changes, the API will be
    marked as stable.

As of the Queens release, keystone has the ability to store and relay
information known as a limit. Limits can be used by services to enforce quota
on resources across OpenStack. This section describes the basic concepts of
limits, how the information can be consumed by services, and how operators can
manage resource quota across OpenStack using limits.

What is a limit?
================

A limit is a threshold for resource management and helps control resource
utilization. A process for managing limits allows for reallocation of resources
to different users or projects as needs change. Some information needed to
establish a limit may include:

- project_id
- domain_id
- API service type (e.g. compute, network, object-storage)
- a resource type (e.g. ram_mb, vcpus, security-groups)
- a default limit
- a project specific limit i.e resource limit
- user_id (optional)
- a region (optional depending on the service)

.. NOTE:: The `default limit` of registered limit and the `resource limit` of
          project limit now are limited from `-1` to `2147483647` (integer). `-1` means
          no limit and `2147483647` is the max value for user to define limits.
          The length of unified limit's `resource type` now is limited from `1`
          to `255` (string).

Since keystone is the source of truth for nearly everything in the above list,
limits are a natural fit as a keystone resource. Two different limit resources
exist in this design. The first is a registered limit and the second is a
project limit.

Registered limits
-----------------

A registered limit accomplishes two important things in order to enforce quota
across multi-tenant, distributed systems. First, it establishes resource types
and associates them to services. Second, it sets a default resource limit for
all projects. The first part maps specific resource types to the services that
provide them. For example, a registered limit can map `vcpus`, to the compute
service. The second part sets a default of 20 `vcpus` per project. This
provides all the information needed for basic quota enforcement for any
resource provided by a service.

Domain limits
-------------

A domain limit is a limit associated to a specific domain and it acts as an
override for a registered limit. Similar to registered limits, domain limits
require a resource type and a service. Additionally, a registered limit must
exist before you can create a domain-specific override. For example, let's
assume a registered limit exists for `vcpus` provided by the compute service.
It wouldn't be possible to create a domain limit for `cores` on the compute
service. Domain limits can only override limits that have already been
registered. In a general sense, registered limits are likely established when a
new service or cloud is deployed. Domain limits are used continuously to manage
the flow of resource allocation.

Domain limits may affect the limits of projects within the domain. This is
particularly important to keep in mind when choosing an enforcement model,
documented below.

Project limits
--------------

Project limits have the same properties as domain limits, but are specific to
projects instead of domains. You must register a limit before creating a
project-specific override. Just like with domain limits, the flow of resources
between related projects may vary depending on the configured enforcement
model. The support enforcement models below describe how limit validation and
enforcement behave between related projects and domains.

Together, registered limits, domain limits, and project limits give deployments
the ability to restrict resources across the deployment by default, while being
flexible enough to freely marshal resources across projects.

Limits and usage
================

When we talk about a quota system, we’re really talking about two systems. A
system for setting and maintaining limits, the theoretical maximum usage, and a
system for enforcing that usage does not exceed limits. While they are coupled,
they are distinct.

Up to this point, we've established that keystone is the system for maintaining
limit information. Keystone’s responsibility is to ensure that any changes to
limits are consistent with related limits currently stored in keystone.

Individual services maintain and enforce usage. Services check enforcement
against the current limits at the time a user requests a resource. Usage
reflects the actual resource allocation in units to a consumer.

Given the above, the following is a possible and legal flow:

- User Jane is in project Foo
- Project Foo has a default CPU limit of 20
- User Jane allocated 18 CPUs in project Foo
- Administrator Kelly sets project Foo CPU limit to 10
- User Jane can no longer allocate instance resources in project Foo, until
  she (or others in the project) have deleted at least 9 CPUs to get under the
  new limit

The following would be another permutation:

- User Jane is in project Foo
- Project Foo has a default CPU limit of 20
- User Jane allocated 20 CPUs in project Foo
- User Jane attempts to create another instance, which results in a failed
  resource request since the request would violate usage based on the current
  limit of CPUs
- User Jane requests more resources
- Administrator Kelly adjust the project limit for Foo to be 30 CPUs
- User Jane resends her request for an instance, which succeeds since the
  usage for project Foo is under the project limit of 30 CPUs

This behavior lets administrators set the policy of what the future should be
when convenient, and prevent those projects from creating any more resources
that would exceed the limits in question. Members of a project can fix this for
themselves by bringing down the project usage to where there is now headroom.
If they don’t, at some point the administrators can more aggressively delete
things themselves.

Enforcement models
==================

Project resources in keystone can be organized in hierarchical structures,
where projects can be nested. As a result, resource limits and usage should
respect that hierarchy if present. It's possible to think of different cases
where limits or usage assume different characteristics, regardless of the
project structure.  For example, if a project's usage for a particular resource
hasn't been met, should the projects underneath that project assume those
limits? Should they not assume those limits? These opinionated models are
referred to as enforcement models. This section is dedicated to describing
different enforcement models that are implemented.

It is important to note that enforcement must be consistent across the entire
deployment. Grouping certain characteristics into a model makes referring to
behaviors consistent across services. Operators should be aware that switching
between enforcement models may result in backwards incompatible changes. We
recommend extremely careful planning and understanding of various enforcement
models if you're planning on switching from one model to another in a
deployment.

Keystone exposes a ``GET /limits-model`` endpoint that returns the enforcement
model selected by the deployment. This allows limit information to be
discoverable and preserves interoperability between OpenStack deployments with
different enforcement models.

Flat
----

Flat enforcement ignores all aspects of a project hierarchy. Each project is
considered a peer to all other projects. The limits associated to the parents,
siblings, or children have no affect on a particular project. This model
exercises the most isolation between projects because there are no assumptions
between limits, regardless of the hierarchy. Validation of limits via the API
will allow operations that might not be considered accepted in other models.

For example, assume project `Charlie` is a child of project `Beta`, which is a
child of project `Alpha`. All projects assume a default limit of 10 cores via a
registered limit. The labels in the diagrams below use shorthand notation for
`limit` and `usage` as `l` and `u`, respectively:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (u=0)"];
      Beta [label=" Beta (u=0)"];
      Charlie [label="Charlie (u=0)"];
   }

Each project may use up to 10 cores because of the registered limit and none of
the projects have an override. Using flat enforcement, you're allowed to
``UPDATE LIMIT on Alpha to 20``:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (l=20, u=0)", textcolor = "#00af00"];
      Beta [label=" Beta (u=0)"];
      Charlie [label="Charlie (u=0)"];
   }


You're also allowed to ``UPDATE LIMIT on Charlie to 30``, even though `Charlie`
is a sub-project of both `Beta` and `Alpha`.

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (l=20, u=0)"];
      Beta [label=" Beta (u=0)"];
      Charlie [label="Charlie (l=30, u=0)", textcolor = "#00af00"];
   }

This is allowed with flat enforcement because the hierarchy is not taken into
consideration during limit validation. Child projects may have a higher limit
than a parent project.

Conversely, you can simulate hierarchical enforcement by adjusting limits
through the project tree manually. For example, let's still assume 10 is the
default limit imposed by an existing registered limit:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (u=0)"];
      Beta [label=" Beta (u=0)"];
      Charlie [label="Charlie (u=0)"];
   }

You may set a project-specific override to ``UPDATE LIMIT on Alpha to 30``:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (l=30, u=0)", textcolor = "#00af00"];
      Beta [label=" Beta (u=0)"];
      Charlie [label="Charlie (u=0)"];
   }

Next you can ``UPDATE LIMIT on Beta to 20``:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (l=30, u=0)"];
      Beta [label=" Beta (l=20, u=0)", textcolor = "#00af00"];
      Charlie [label="Charlie (u=0)"];
   }

Theoretically, the entire project tree consisting of `Alpha`, `Beta`, and
`Charlie` is limited to 60 cores. If you'd like to ensure only 30 cores are
used within the entire hierarchy, you can ``UPDATE LIMIT on Alpha to 0``:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha [label="Alpha (l=0, u=0)", textcolor = "#00af00"];
      Beta [label=" Beta (l=20, u=0)"];
      Charlie [label="Charlie (u=0)"];
   }

You should use this model if you:

* Have project hierarchies greater than two levels
* Want extremely strict control of project usage and don't want resource usage
  to bleed across projects or domains

Advantages
~~~~~~~~~~

* Allows you to model specific and strict limits
* Works with any project hierarchy or depth
* Usage is only calculated for the project in question

Disadvantages
~~~~~~~~~~~~~

* Resources aren't allowed to flow gracefully between projects in a hierarchy
* Requires intervention and verification to move resources across projects
* Project limit validation isn't performed with respect to other projects or
  domains

Strict Two Level
----------------

The ``strict_two_level`` enforcement model assumes the project hierarchy does
not exceed two levels. The top layer can consist of projects or domains. For
example, project `Alpha` can have a sub-project called `Beta` within this
model. Project `Beta` cannot have a sub-project. The hierarchy is restrained to
two layers. `Alpha` can also be a domain that contains project `Beta`, but
`Beta` cannot have a sub-project. Regardless of the top layer consisting of
projects or domains, the hierarchical depth is limited to two layers.

Resource utilization is allowed to flow between projects in the hierarchy,
depending on the limits. This property allows for more flexibility than the
``flat`` enforcement model. The model is strict in that operators can set
limits on parent projects or domains and the limits of the children may never
exceed the parent.

For example, assume domain `Alpha` contains two projects, `Beta` and `Charlie`.
Projects `Beta` and `Charlie` are siblings so the hierarchy maintains a depth
of two. A system administrator sets the limit of a resource on `Alpha` to 20.
Both projects `Beta` and `Charlie` can consume resources until the total usage
of `Alpha`, `Beta`, and `Charlie` reach 20. At that point, no more resources
should be allocated to the tree. System administrators can also reserve
portions of domain `Alpha`'s resource in sub-projects directly. Using the
previous example, project `Beta` could have a limit of 12 resources, implicitly
leaving 8 resources for `Charlie` to consume.

The following diagrams illustrate the behaviors described above, using projects
named `Alpha`, `Beta`, `Charlie`, and `Delta`. Assume the resource in question
is cores and the default registered limit for cores is 10. Also assume we have
the following project hierarchy where `Alpha` has a limit of 20 cores and its
usage is currently 4:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=4)"];
      Beta [label="Beta (u=0)"];
      Charlie [label="Charlie (u=0)"];
   }

Technically, both `Beta` and `Charlie` can use up to 8 cores each:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=4)"];
      Beta [label="Beta (u=8)", textcolor = "#00af00"];
      Charlie [label="Charlie (u=8)", textcolor = "#00af00"];
   }

If `Alpha` attempts to claim two cores the usage check will fail
because the service will fetch the hierarchy from keystone using ``oslo.limit``
and check the usage of each project in the hierarchy to see that the total
usage of `Alpha`, `Beta`, and `Charlie` is equal to the limit of the tree, set
by `Alpha.limit`:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=6)", textcolor = "#FF0000"];
      Beta [label="Beta (u=8)"];
      Charlie [label="Charlie (u=8)"];
   }

Despite the usage of the tree being equal to the limit, we can still add
children to the tree:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;
      Alpha -> Delta;

      Alpha [label="Alpha (l=20, u=4)"];
      Beta [label="Beta (u=8)"];
      Charlie [label="Charlie (u=8)"];
      Delta [label="Delta (u=0)", textcolor = "#00af00"];
   }

Even though the project can be created, the current usage of cores across the
tree prevents `Delta` from claiming any cores:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;
      Alpha -> Delta;

      Alpha [label="Alpha (l=20, u=4)"];
      Beta [label="Beta (u=8)"];
      Charlie [label="Charlie (u=8)"];
      Delta [label="Delta (u=2)", textcolor = "#FF0000"];
   }

Creating a grandchild of project `Alpha` is forbidden because it violates the
two-level hierarchical constraint:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;
      Charlie -> Delta;

      Alpha [label="Alpha (l=20, u=4)"];
      Beta [label="Beta (u=8)"];
      Charlie [label="Charlie (u=8)"];
      Delta [label="Delta (u=0)", textcolor = "#FF0000"];
   }

This is a fundamental constraint of this design because it provides a very
clear escalation path. When a request fails because the tree limit has been
exceeded, a user has all the information they need to provide meaningful
context in a support ticket (e.g., their project ID and the parent project ID).
An administrator should be able to reshuffle usage accordingly. Providing this
information in tree structures with more than a depth of two is much harder,
but may be implemented with a separate model.

Granting `Beta` the ability to claim more cores can be done by giving `Beta` a
project-specific override for cores

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=4)"];
      Beta [label="Beta (l=12, u=8)", textcolor = "#00af00"];
      Charlie [label="Charlie (u=8)"];
   }

Note that regardless of this update, any subsequent requests to claim more
cores in the tree will be rejected since the usage is equal to the limit of the
`Alpha`. `Beta` can claim cores if they are released from `Alpha` or `Charlie`:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=2)", textcolor = "#00af00"];
      Beta [label="Beta (l=12, u=8)"];
      Charlie [label="Charlie (u=6)", textcolor = "#00af00"];
   }

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=2)"];
      Beta [label="Beta (l=12, u=12)", textcolor = "#00af00"];
      Charlie [label="Charlie (u=6)"];
   }

While `Charlie` is still under its default allocation of 10 cores, it won't be
able to claim any more cores because the total usage of the tree is equal to
the limit of `Alpha`, thus preventing `Charlie` from reclaiming the cores it
had:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=2)"];
      Beta [label="Beta (l=12, u=12)"];
      Charlie [label="Charlie (u=8)", textcolor = "#FF0000"];
   }

Creating or updating a project with a limit that exceeds the limit of `Alpha`
is forbidden. Even though it is possible for the sum of all limits under
`Alpha` to exceed the limit of `Alpha`, the total usage is capped at
`Alpha.limit`. Allowing children to have explicit overrides greater than the
limit of the parent would result in strange user experience and be misleading
since the total usage of the tree would be capped at the limit of the parent:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;

      Alpha [label="Alpha (l=20, u=0)"];
      Beta [label="Beta (l=30, u=0)", textcolor = "#FF0000"];
      Charlie [label="Charlie (u=0)"];
   }

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;
      Alpha -> Delta;

      Alpha [label="Alpha (l=20, u=0)"];
      Beta [label="Beta (u=0)"];
      Charlie [label="Charlie (u=0)"];
      Delta [label="Delta (l=30, u=0)", textcolor = "#FF0000"];
   }

Finally, let's still assume the default registered limit for cores is 10, but
we're going to create project `Alpha` with a limit of 6 cores.

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha;

      Alpha [label="Alpha (l=6, u=0)", textcolor = "#00af00"];
   }

When we create project `Beta`, which is a child of project `Alpha`, the limit
API ensures that project `Beta` doesn't assume the default of 10, despite the
registered limit of 10 cores. Instead, the child assumes the parent's limit
since no single child limit should exceed the limit of the parent:

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;

      Alpha [label="Alpha (l=6, u=0)"];
      Beta [label="Beta (l=6, u=0)", textcolor = "#00af00"];
   }

This behavior is consistent regardless of the number of children added under
project `Alpha`.

.. blockdiag::

   blockdiag {
      orientation = portrait;

      Alpha -> Beta;
      Alpha -> Charlie;
      Alpha -> Delta;

      Alpha [label="Alpha (l=6, u=0)"];
      Beta [label="Beta (l=6, u=0)"];
      Charlie [label="Charlie (l=6, u=0)", textcolor = "#00af00"];
      Delta [label="Delta (l=6, u=0)", textcolor = "#00af00"];
   }

Creating limit overrides while creating projects seems counter-productive given
the whole purpose of a registered default, but it also seems unlikely to
throttle a parent project by specifying it's default to be less than a
registered default. This behavior maintains consistency with the requirement
that the sum of all child limits may exceed the parent limit, but the limit of
any one child may not.

You should use this model if you:

* Want resources to flow between projects and domains within a hierarchy
* Don't have a project depth greater than two levels
* Are not concerned about usage calculation performance or don't have project
  trees that are wide

Advantages
~~~~~~~~~~

* Allows resources to flow between projects and domains within a strict
  two-level hierarchy
* Limits are validated when they are created and updated

Disadvantages
~~~~~~~~~~~~~

* Project depth cannot exceed two levels
* Performance may suffer in wide and flat project hierarchies during usage
  calculation