Remove vec_perm_const optab

One of the changes needed for variable-length VEC_PERM_EXPRs -- and for
long fixed-length VEC_PERM_EXPRs -- is the ability to use constant
selectors that wouldn't fit in the vectors being permuted.  E.g. a
permute on two V256QIs can't be done using a V256QI selector.

At the moment constant permutes use two interfaces:
targetm.vectorizer.vec_perm_const_ok for testing whether a permute is
valid and the vec_perm_const optab for actually emitting the permute.
The former gets passed a vec<> selector and the latter an rtx selector.
Most ports share a lot of code between the hook and the optab, with a
wrapper function for each interface.

We could try to keep that interface and require ports to define wider
vector modes that could be attached to the CONST_VECTOR (e.g. V256HI or
V256SI in the example above).  But building a CONST_VECTOR rtx seems a bit
pointless here, since the expand code only creates the CONST_VECTOR in
order to call the optab, and the first thing the target does is take
the CONST_VECTOR apart again.

The easiest approach therefore seemed to be to remove the optab and
reuse the target hook to emit the code.  One potential drawback is that
it's no longer possible to use match_operand predicates to force
operands into the required form, but in practice all targets want
register operands anyway.

The patch also changes vec_perm_indices into a class that provides
some simple routines for handling permutations.  A later patch will
flesh this out and get rid of auto_vec_perm_indices, but I didn't
want to do all that in this patch and make it more complicated than
it already is.

2018-01-02  Richard Sandiford  <richard.sandiford@linaro.org>

gcc/
	* Makefile.in (OBJS): Add vec-perm-indices.o.
	* vec-perm-indices.h: New file.
	* vec-perm-indices.c: Likewise.
	* target.h (vec_perm_indices): Replace with a forward class
	declaration.
	(auto_vec_perm_indices): Move to vec-perm-indices.h.
	* optabs.h: Include vec-perm-indices.h.
	(expand_vec_perm): Delete.
	(selector_fits_mode_p, expand_vec_perm_var): Declare.
	(expand_vec_perm_const): Declare.
	* target.def (vec_perm_const_ok): Replace with...
	(vec_perm_const): ...this new hook.
	* doc/tm.texi.in (TARGET_VECTORIZE_VEC_PERM_CONST_OK): Replace with...
	(TARGET_VECTORIZE_VEC_PERM_CONST): ...this new hook.
	* doc/tm.texi: Regenerate.
	* optabs.def (vec_perm_const): Delete.
	* doc/md.texi (vec_perm_const): Likewise.
	(vec_perm): Refer to TARGET_VECTORIZE_VEC_PERM_CONST.
	* expr.c (expand_expr_real_2): Use expand_vec_perm_const rather than
	expand_vec_perm for constant permutation vectors.  Assert that
	the mode of variable permutation vectors is the integer equivalent
	of the mode that is being permuted.
	* optabs-query.h (selector_fits_mode_p): Declare.
	* optabs-query.c: Include vec-perm-indices.h.
	(selector_fits_mode_p): New function.
	(can_vec_perm_const_p): Check whether targetm.vectorize.vec_perm_const
	is defined, instead of checking whether the vec_perm_const_optab
	exists.  Use targetm.vectorize.vec_perm_const instead of
	targetm.vectorize.vec_perm_const_ok.  Check whether the indices
	fit in the vector mode before using a variable permute.
	* optabs.c (shift_amt_for_vec_perm_mask): Take a mode and a
	vec_perm_indices instead of an rtx.
	(expand_vec_perm): Replace with...
	(expand_vec_perm_const): ...this new function.  Take the selector
	as a vec_perm_indices rather than an rtx.  Also take the mode of
	the selector.  Update call to shift_amt_for_vec_perm_mask.
	Use targetm.vectorize.vec_perm_const instead of vec_perm_const_optab.
	Use vec_perm_indices::new_expanded_vector to expand the original
	selector into bytes.  Check whether the indices fit in the vector
	mode before using a variable permute.
	(expand_vec_perm_var): Make global.
	(expand_mult_highpart): Use expand_vec_perm_const.
	* fold-const.c: Includes vec-perm-indices.h.
	* tree-ssa-forwprop.c: Likewise.
	* tree-vect-data-refs.c: Likewise.
	* tree-vect-generic.c: Likewise.
	* tree-vect-loop.c: Likewise.
	* tree-vect-slp.c: Likewise.
	* tree-vect-stmts.c: Likewise.
	* config/aarch64/aarch64-protos.h (aarch64_expand_vec_perm_const):
	Delete.
	* config/aarch64/aarch64-simd.md (vec_perm_const<mode>): Delete.
	* config/aarch64/aarch64.c (aarch64_expand_vec_perm_const)
	(aarch64_vectorize_vec_perm_const_ok): Fuse into...
	(aarch64_vectorize_vec_perm_const): ...this new function.
	(TARGET_VECTORIZE_VEC_PERM_CONST_OK): Delete.
	(TARGET_VECTORIZE_VEC_PERM_CONST): Redefine.
	* config/arm/arm-protos.h (arm_expand_vec_perm_const): Delete.
	* config/arm/vec-common.md (vec_perm_const<mode>): Delete.
	* config/arm/arm.c (TARGET_VECTORIZE_VEC_PERM_CONST_OK): Delete.
	(TARGET_VECTORIZE_VEC_PERM_CONST): Redefine.
	(arm_expand_vec_perm_const, arm_vectorize_vec_perm_const_ok): Merge
	into...
	(arm_vectorize_vec_perm_const): ...this new function.  Explicitly
	check for NEON modes.
	* config/i386/i386-protos.h (ix86_expand_vec_perm_const): Delete.
	* config/i386/sse.md (VEC_PERM_CONST, vec_perm_const<mode>): Delete.
	* config/i386/i386.c (ix86_expand_vec_perm_const_1): Update comment.
	(ix86_expand_vec_perm_const, ix86_vectorize_vec_perm_const_ok): Merge
	into...
	(ix86_vectorize_vec_perm_const): ...this new function.  Incorporate
	the old VEC_PERM_CONST conditions.
	* config/ia64/ia64-protos.h (ia64_expand_vec_perm_const): Delete.
	* config/ia64/vect.md (vec_perm_const<mode>): Delete.
	* config/ia64/ia64.c (ia64_expand_vec_perm_const)
	(ia64_vectorize_vec_perm_const_ok): Merge into...
	(ia64_vectorize_vec_perm_const): ...this new function.
	* config/mips/loongson.md (vec_perm_const<mode>): Delete.
	* config/mips/mips-msa.md (vec_perm_const<mode>): Delete.
	* config/mips/mips-ps-3d.md (vec_perm_constv2sf): Delete.
	* config/mips/mips-protos.h (mips_expand_vec_perm_const): Delete.
	* config/mips/mips.c (mips_expand_vec_perm_const)
	(mips_vectorize_vec_perm_const_ok): Merge into...
	(mips_vectorize_vec_perm_const): ...this new function.
	* config/powerpcspe/altivec.md (vec_perm_constv16qi): Delete.
	* config/powerpcspe/paired.md (vec_perm_constv2sf): Delete.
	* config/powerpcspe/spe.md (vec_perm_constv2si): Delete.
	* config/powerpcspe/vsx.md (vec_perm_const<mode>): Delete.
	* config/powerpcspe/powerpcspe-protos.h (altivec_expand_vec_perm_const)
	(rs6000_expand_vec_perm_const): Delete.
	* config/powerpcspe/powerpcspe.c (TARGET_VECTORIZE_VEC_PERM_CONST_OK):
	Delete.
	(TARGET_VECTORIZE_VEC_PERM_CONST): Redefine.
	(altivec_expand_vec_perm_const_le): Take each operand individually.
	Operate on constant selectors rather than rtxes.
	(altivec_expand_vec_perm_const): Likewise.  Update call to
	altivec_expand_vec_perm_const_le.
	(rs6000_expand_vec_perm_const): Delete.
	(rs6000_vectorize_vec_perm_const_ok): Delete.
	(rs6000_vectorize_vec_perm_const): New function.
	(rs6000_do_expand_vec_perm): Take a vec_perm_builder instead of
	an element count and rtx array.
	(rs6000_expand_extract_even): Update call accordingly.
	(rs6000_expand_interleave): Likewise.
	* config/rs6000/altivec.md (vec_perm_constv16qi): Delete.
	* config/rs6000/paired.md (vec_perm_constv2sf): Delete.
	* config/rs6000/vsx.md (vec_perm_const<mode>): Delete.
	* config/rs6000/rs6000-protos.h (altivec_expand_vec_perm_const)
	(rs6000_expand_vec_perm_const): Delete.
	* config/rs6000/rs6000.c (TARGET_VECTORIZE_VEC_PERM_CONST_OK): Delete.
	(TARGET_VECTORIZE_VEC_PERM_CONST): Redefine.
	(altivec_expand_vec_perm_const_le): Take each operand individually.
	Operate on constant selectors rather than rtxes.
	(altivec_expand_vec_perm_const): Likewise.  Update call to
	altivec_expand_vec_perm_const_le.
	(rs6000_expand_vec_perm_const): Delete.
	(rs6000_vectorize_vec_perm_const_ok): Delete.
	(rs6000_vectorize_vec_perm_const): New function.  Remove stray
	reference to the SPE evmerge intructions.
	(rs6000_do_expand_vec_perm): Take a vec_perm_builder instead of
	an element count and rtx array.
	(rs6000_expand_extract_even): Update call accordingly.
	(rs6000_expand_interleave): Likewise.
	* config/sparc/sparc.md (vec_perm_constv8qi): Delete in favor of...
	* config/sparc/sparc.c (sparc_vectorize_vec_perm_const): ...this
	new function.
	(TARGET_VECTORIZE_VEC_PERM_CONST): Redefine.

From-SVN: r256093

6 files changed, 58 insertions, 121 deletions

diff --git a/gcc/config/powerpcspe/altivec.md b/gcc/config/powerpcspe/altivec.md
index 81373f581d1..2f85e369c3e 100644
--- a/gcc/config/powerpcspe/altivec.md
+++ b/gcc/config/powerpcspe/altivec.md
@@ -2080,19 +2080,6 @@
   }
 })
 
-(define_expand "vec_perm_constv16qi"
-  [(match_operand:V16QI 0 "register_operand" "")
-   (match_operand:V16QI 1 "register_operand" "")
-   (match_operand:V16QI 2 "register_operand" "")
-   (match_operand:V16QI 3 "" "")]
-  "TARGET_ALTIVEC"
-{
-  if (altivec_expand_vec_perm_const (operands))
-    DONE;
-  else
-    FAIL;
-})
-
 (define_insn "*altivec_vpermr_<mode>_internal"
   [(set (match_operand:VM 0 "register_operand" "=v,?wo")
 	(unspec:VM [(match_operand:VM 1 "register_operand" "v,wo")
diff --git a/gcc/config/powerpcspe/paired.md b/gcc/config/powerpcspe/paired.md
index e12f07fc9b8..e950e465861 100644
--- a/gcc/config/powerpcspe/paired.md
+++ b/gcc/config/powerpcspe/paired.md
@@ -313,19 +313,6 @@
   "ps_merge11 %0, %1, %2"
   [(set_attr "type" "fp")])
 
-(define_expand "vec_perm_constv2sf"
-  [(match_operand:V2SF 0 "gpc_reg_operand" "")
-   (match_operand:V2SF 1 "gpc_reg_operand" "")
-   (match_operand:V2SF 2 "gpc_reg_operand" "")
-   (match_operand:V2SI 3 "" "")]
-  "TARGET_PAIRED_FLOAT"
-{
-  if (rs6000_expand_vec_perm_const (operands))
-    DONE;
-  else
-    FAIL;
-})
-
 (define_insn "paired_sum0"
   [(set (match_operand:V2SF 0 "gpc_reg_operand" "=f")
 	(vec_concat:V2SF (plus:SF (vec_select:SF
diff --git a/gcc/config/powerpcspe/powerpcspe-protos.h b/gcc/config/powerpcspe/powerpcspe-protos.h
index 78baeecad38..b9baae8a680 100644
--- a/gcc/config/powerpcspe/powerpcspe-protos.h
+++ b/gcc/config/powerpcspe/powerpcspe-protos.h
@@ -64,9 +64,7 @@ extern void rs6000_expand_vector_extract (rtx, rtx, rtx);
 extern void rs6000_split_vec_extract_var (rtx, rtx, rtx, rtx, rtx);
 extern rtx rs6000_adjust_vec_address (rtx, rtx, rtx, rtx, machine_mode);
 extern void rs6000_split_v4si_init (rtx []);
-extern bool altivec_expand_vec_perm_const (rtx op[4]);
 extern void altivec_expand_vec_perm_le (rtx op[4]);
-extern bool rs6000_expand_vec_perm_const (rtx op[4]);
 extern void altivec_expand_lvx_be (rtx, rtx, machine_mode, unsigned);
 extern void altivec_expand_stvx_be (rtx, rtx, machine_mode, unsigned);
 extern void altivec_expand_stvex_be (rtx, rtx, machine_mode, unsigned);
diff --git a/gcc/config/powerpcspe/powerpcspe.c b/gcc/config/powerpcspe/powerpcspe.c
index bf90cc5cd7d..9133125a3ea 100644
--- a/gcc/config/powerpcspe/powerpcspe.c
+++ b/gcc/config/powerpcspe/powerpcspe.c
@@ -1938,8 +1938,8 @@ static const struct attribute_spec rs6000_attribute_table[] =
 #undef TARGET_LEGITIMATE_CONSTANT_P
 #define TARGET_LEGITIMATE_CONSTANT_P rs6000_legitimate_constant_p
 
-#undef TARGET_VECTORIZE_VEC_PERM_CONST_OK
-#define TARGET_VECTORIZE_VEC_PERM_CONST_OK rs6000_vectorize_vec_perm_const_ok
+#undef TARGET_VECTORIZE_VEC_PERM_CONST
+#define TARGET_VECTORIZE_VEC_PERM_CONST rs6000_vectorize_vec_perm_const
 
 #undef TARGET_CAN_USE_DOLOOP_P
 #define TARGET_CAN_USE_DOLOOP_P can_use_doloop_if_innermost
@@ -38313,6 +38313,9 @@ rs6000_emit_parity (rtx dst, rtx src)
 }
 
 /* Expand an Altivec constant permutation for little endian mode.
+   OP0 and OP1 are the input vectors and TARGET is the output vector.
+   SEL specifies the constant permutation vector.
+
    There are two issues: First, the two input operands must be
    swapped so that together they form a double-wide array in LE
    order.  Second, the vperm instruction has surprising behavior
@@ -38354,22 +38357,18 @@ rs6000_emit_parity (rtx dst, rtx src)
 
    vr9  = 00000006 00000004 00000002 00000000.  */
 
-void
-altivec_expand_vec_perm_const_le (rtx operands[4])
+static void
+altivec_expand_vec_perm_const_le (rtx target, rtx op0, rtx op1,
+				  const vec_perm_indices &sel)
 {
   unsigned int i;
   rtx perm[16];
   rtx constv, unspec;
-  rtx target = operands[0];
-  rtx op0 = operands[1];
-  rtx op1 = operands[2];
-  rtx sel = operands[3];
 
   /* Unpack and adjust the constant selector.  */
   for (i = 0; i < 16; ++i)
     {
-      rtx e = XVECEXP (sel, 0, i);
-      unsigned int elt = 31 - (INTVAL (e) & 31);
+      unsigned int elt = 31 - (sel[i] & 31);
       perm[i] = GEN_INT (elt);
     }
 
@@ -38451,10 +38450,14 @@ altivec_expand_vec_perm_le (rtx operands[4])
 }
 
 /* Expand an Altivec constant permutation.  Return true if we match
-   an efficient implementation; false to fall back to VPERM.  */
+   an efficient implementation; false to fall back to VPERM.
 
-bool
-altivec_expand_vec_perm_const (rtx operands[4])
+   OP0 and OP1 are the input vectors and TARGET is the output vector.
+   SEL specifies the constant permutation vector.  */
+
+static bool
+altivec_expand_vec_perm_const (rtx target, rtx op0, rtx op1,
+			       const vec_perm_indices &sel)
 {
   struct altivec_perm_insn {
     HOST_WIDE_INT mask;
@@ -38498,19 +38501,13 @@ altivec_expand_vec_perm_const (rtx operands[4])
 
   unsigned int i, j, elt, which;
   unsigned char perm[16];
-  rtx target, op0, op1, sel, x;
+  rtx x;
   bool one_vec;
 
-  target = operands[0];
-  op0 = operands[1];
-  op1 = operands[2];
-  sel = operands[3];
-
   /* Unpack the constant selector.  */
   for (i = which = 0; i < 16; ++i)
     {
-      rtx e = XVECEXP (sel, 0, i);
-      elt = INTVAL (e) & 31;
+      elt = sel[i] & 31;
       which |= (elt < 16 ? 1 : 2);
       perm[i] = elt;
     }
@@ -38666,7 +38663,7 @@ altivec_expand_vec_perm_const (rtx operands[4])
 
   if (!BYTES_BIG_ENDIAN)
     {
-      altivec_expand_vec_perm_const_le (operands);
+      altivec_expand_vec_perm_const_le (target, op0, op1, sel);
       return true;
     }
 
@@ -38726,60 +38723,54 @@ rs6000_expand_vec_perm_const_1 (rtx target, rtx op0, rtx op1,
   return true;
 }
 
-bool
-rs6000_expand_vec_perm_const (rtx operands[4])
-{
-  rtx target, op0, op1, sel;
-  unsigned char perm0, perm1;
-
-  target = operands[0];
-  op0 = operands[1];
-  op1 = operands[2];
-  sel = operands[3];
-
-  /* Unpack the constant selector.  */
-  perm0 = INTVAL (XVECEXP (sel, 0, 0)) & 3;
-  perm1 = INTVAL (XVECEXP (sel, 0, 1)) & 3;
-
-  return rs6000_expand_vec_perm_const_1 (target, op0, op1, perm0, perm1);
-}
-
-/* Test whether a constant permutation is supported.  */
+/* Implement TARGET_VECTORIZE_VEC_PERM_CONST.  */
 
 static bool
-rs6000_vectorize_vec_perm_const_ok (machine_mode vmode, vec_perm_indices sel)
+rs6000_vectorize_vec_perm_const (machine_mode vmode, rtx target, rtx op0,
+				 rtx op1, const vec_perm_indices &sel)
 {
+  bool testing_p = !target;
+
   /* AltiVec (and thus VSX) can handle arbitrary permutations.  */
-  if (TARGET_ALTIVEC)
+  if (TARGET_ALTIVEC && testing_p)
     return true;
 
-  /* Check for ps_merge* or evmerge* insns.  */
-  if ((TARGET_PAIRED_FLOAT && vmode == V2SFmode)
-      || (TARGET_SPE && vmode == V2SImode))
+  /* Check for ps_merge*, evmerge* or xxperm* insns.  */
+  if ((vmode == V2SFmode && TARGET_PAIRED_FLOAT)
+      || (vmode == V2SImode && TARGET_SPE)
+      || ((vmode == V2DFmode || vmode == V2DImode)
+	  && VECTOR_MEM_VSX_P (vmode)))
+    {
+      if (testing_p)
+	{
+	  op0 = gen_raw_REG (vmode, LAST_VIRTUAL_REGISTER + 1);
+	  op1 = gen_raw_REG (vmode, LAST_VIRTUAL_REGISTER + 2);
+	}
+      if (rs6000_expand_vec_perm_const_1 (target, op0, op1, sel[0], sel[1]))
+	return true;
+    }
+
+  if (TARGET_ALTIVEC)
     {
-      rtx op0 = gen_raw_REG (vmode, LAST_VIRTUAL_REGISTER + 1);
-      rtx op1 = gen_raw_REG (vmode, LAST_VIRTUAL_REGISTER + 2);
-      return rs6000_expand_vec_perm_const_1 (NULL, op0, op1, sel[0], sel[1]);
+      /* Force the target-independent code to lower to V16QImode.  */
+      if (vmode != V16QImode)
+	return false;
+      if (altivec_expand_vec_perm_const (target, op0, op1, sel))
+	return true;
     }
 
   return false;
 }
 
-/* A subroutine for rs6000_expand_extract_even & rs6000_expand_interleave.  */
+/* A subroutine for rs6000_expand_extract_even & rs6000_expand_interleave.
+   OP0 and OP1 are the input vectors and TARGET is the output vector.
+   PERM specifies the constant permutation vector.  */
 
 static void
 rs6000_do_expand_vec_perm (rtx target, rtx op0, rtx op1,
-			   machine_mode vmode, unsigned nelt, rtx perm[])
+			   machine_mode vmode, const vec_perm_builder &perm)
 {
-  machine_mode imode;
-  rtx x;
-
-  imode = vmode;
-  if (GET_MODE_CLASS (vmode) != MODE_VECTOR_INT)
-    imode = mode_for_int_vector (vmode).require ();
-
-  x = gen_rtx_CONST_VECTOR (imode, gen_rtvec_v (nelt, perm));
-  x = expand_vec_perm (vmode, op0, op1, x, target);
+  rtx x = expand_vec_perm_const (vmode, op0, op1, perm, BLKmode, target);
   if (x != target)
     emit_move_insn (target, x);
 }
@@ -38791,12 +38782,12 @@ rs6000_expand_extract_even (rtx target, rtx op0, rtx op1)
 {
   machine_mode vmode = GET_MODE (target);
   unsigned i, nelt = GET_MODE_NUNITS (vmode);
-  rtx perm[16];
+  vec_perm_builder perm (nelt);
 
   for (i = 0; i < nelt; i++)
-    perm[i] = GEN_INT (i * 2);
+    perm.quick_push (i * 2);
 
-  rs6000_do_expand_vec_perm (target, op0, op1, vmode, nelt, perm);
+  rs6000_do_expand_vec_perm (target, op0, op1, vmode, perm);
 }
 
 /* Expand a vector interleave operation.  */
@@ -38806,16 +38797,16 @@ rs6000_expand_interleave (rtx target, rtx op0, rtx op1, bool highp)
 {
   machine_mode vmode = GET_MODE (target);
   unsigned i, high, nelt = GET_MODE_NUNITS (vmode);
-  rtx perm[16];
+  vec_perm_builder perm (nelt);
 
   high = (highp ? 0 : nelt / 2);
   for (i = 0; i < nelt / 2; i++)
     {
-      perm[i * 2] = GEN_INT (i + high);
-      perm[i * 2 + 1] = GEN_INT (i + nelt + high);
+      perm.quick_push (i + high);
+      perm.quick_push (i + nelt + high);
     }
 
-  rs6000_do_expand_vec_perm (target, op0, op1, vmode, nelt, perm);
+  rs6000_do_expand_vec_perm (target, op0, op1, vmode, perm);
 }
 
 /* Scale a V2DF vector SRC by two to the SCALE and place in TGT.  */
diff --git a/gcc/config/powerpcspe/spe.md b/gcc/config/powerpcspe/spe.md
index 2351152dc24..56acfdd86d0 100644
--- a/gcc/config/powerpcspe/spe.md
+++ b/gcc/config/powerpcspe/spe.md
@@ -511,19 +511,6 @@
   [(set_attr "type" "vecsimple")
    (set_attr  "length" "4")])
 
-(define_expand "vec_perm_constv2si"
-  [(match_operand:V2SI 0 "gpc_reg_operand" "")
-   (match_operand:V2SI 1 "gpc_reg_operand" "")
-   (match_operand:V2SI 2 "gpc_reg_operand" "")
-   (match_operand:V2SI 3 "" "")]
-  "TARGET_SPE"
-{
-  if (rs6000_expand_vec_perm_const (operands))
-    DONE;
-  else
-    FAIL;
-})
-
 (define_expand "spe_evmergehi"
   [(match_operand:V2SI 0 "register_operand" "")
    (match_operand:V2SI 1 "register_operand" "")
diff --git a/gcc/config/powerpcspe/vsx.md b/gcc/config/powerpcspe/vsx.md
index b669764ce8f..794ff446b8a 100644
--- a/gcc/config/powerpcspe/vsx.md
+++ b/gcc/config/powerpcspe/vsx.md
@@ -2543,19 +2543,6 @@
 }
   [(set_attr "type" "vecperm")])
 
-(define_expand "vec_perm_const<mode>"
-  [(match_operand:VSX_D 0 "vsx_register_operand" "")
-   (match_operand:VSX_D 1 "vsx_register_operand" "")
-   (match_operand:VSX_D 2 "vsx_register_operand" "")
-   (match_operand:V2DI  3 "" "")]
-  "VECTOR_MEM_VSX_P (<MODE>mode)"
-{
-  if (rs6000_expand_vec_perm_const (operands))
-    DONE;
-  else
-    FAIL;
-})
-
 ;; Extraction of a single element in a small integer vector.  Until ISA 3.0,
 ;; none of the small types were allowed in a vector register, so we had to
 ;; extract to a DImode and either do a direct move or store.