ASG scaling account for cooldown timestamp & in-progress

There are cases where it takes a long time to create a new resource as
requested by the scaling operation on an ASG resource, for instance, a
nova server creation followed by a complex SoftwareDeployment. During
this process, additional alarms may come in but failed to be blocked
by the current cooldown checking mechanism because the very first
timestamp has yet to be generated. This is leading to unexpected size
adjustment to the ASG.

This patch augments the existing cooldown checking mechanism with a
scaling-in-progress test so that additional alarms arriving during
the very first scaling operation will be ignored.

Change-Id: Ib8aa83eed366df7097c9cbb9247eca866ae4b620
Closes-Bug: #1375156

1 files changed, 23 insertions, 5 deletions

diff --git a/heat/scaling/cooldown.py b/heat/scaling/cooldown.py
index 75a55b9bb..bf7c82fb0 100644
--- a/heat/scaling/cooldown.py
+++ b/heat/scaling/cooldown.py
@@ -18,7 +18,8 @@ import six
 class CooldownMixin(object):
     '''
     Utility class to encapsulate Cooldown related logic which is shared
-    between AutoScalingGroup and ScalingPolicy
+    between AutoScalingGroup and ScalingPolicy. This logic includes both
+    cooldown timestamp comparing and scaling in progress checking.
     '''
     def _cooldown_inprogress(self):
         inprogress = False
@@ -30,16 +31,33 @@ class CooldownMixin(object):
             cooldown = 0
 
         metadata = self.metadata_get()
-        if metadata and cooldown != 0:
-            last_adjust = next(six.iterkeys(metadata))
+        if metadata.get('scaling_in_progress'):
+            return True
+
+        if 'cooldown' not in metadata:
+            # Note: this is for supporting old version cooldown checking
+            if metadata and cooldown != 0:
+                last_adjust = next(six.iterkeys(metadata))
+                if not timeutils.is_older_than(last_adjust, cooldown):
+                    inprogress = True
+        elif cooldown != 0:
+            last_adjust = next(six.iterkeys(metadata['cooldown']))
             if not timeutils.is_older_than(last_adjust, cooldown):
                 inprogress = True
+
+        if not inprogress:
+            metadata['scaling_in_progress'] = True
+            self.metadata_set(metadata)
+
         return inprogress
 
     def _cooldown_timestamp(self, reason):
-        # Save resource metadata with a timestamp and reason
+        # Save cooldown timestamp into metadata and clean the
+        # scaling_in_progress state.
         # If we wanted to implement the AutoScaling API like AWS does,
         # we could maintain event history here, but since we only need
         # the latest event for cooldown, just store that for now
-        metadata = {timeutils.utcnow().isoformat(): reason}
+        metadata = self.metadata_get()
+        metadata['cooldown'] = {timeutils.utcnow().isoformat(): reason}
+        metadata['scaling_in_progress'] = False
         self.metadata_set(metadata)