summaryrefslogtreecommitdiff
path: root/src/sds.h
Commit message (Collapse)AuthorAgeFilesLines
* Optimization: sdsRemoveFreeSpace to avoid realloc on noop (#11766)uriyage2023-01-311-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation size in order to: > reduce the need for realloc calls by making the sds implicitly take over the internal fragmentation This change was done most sds functions, excluding `sdsRemoveFreeSpace` and `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer) we call sdsRemoveFreeSpace when we see excessive free space and want to trim it. so if we don't trim it exactly to size, the caller may still see excessive free space and call it again and again. However, this resulted in some excessive calls to realloc, even when there's no need and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13). It turns out that a call for realloc with jemalloc can be expensive even if it ends up doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid the call for realloc. in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common code. the difference between them was that sdsResize would avoid using SDS_TYPE_5, since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace would permit using SDS_TYPE_5 and get an optimal memory consumption. now both methods take a `would_regrow` argument that makes it more explicit. the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both. The new test that was added to cover this concern used to pass before this PR as well, this PR is just a performance optimization and cleanup. Benchmark: `redis-benchmark -c 100 -t set -d 512 -P 10 -n 100000000` on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs). some 4.5% improvement. Co-authored-by: Oran Agra <oran@redislabs.com>
* Fix additional AOF filename issues. (#10110)Yossi Gottlieb2022-01-181-0/+1
| | | | | | | | This extends the previous fix (#10049) to address any form of non-printable or whitespace character (including newlines, quotes, non-printables, etc.) Also, removes the limitation on appenddirname, to align with the way filenames are handled elsewhere in Redis.
* Add --large-memory flag for REDIS_TEST to enable tests that consume more ↵sundb2021-11-161-1/+1
| | | | | than 100mb (#9784) This is a preparation step in order to add a new test in quicklist.c see #9776
* Query buffer shrinking improvements (#5013)Oran Agra2021-07-051-0/+1
| | | | | | | | | | | | | | | | | when tracking the peak, don't reset the peak to 0, reset it to the maximum of the current used, and the planned to be used by the current arg. when shrining, split the two separate conditions. the idle time shrinking will remove all free space. but the peak based shrinking will keep room for the current arg. when we resize due to a peak (rahter than idle time), don't trim all unused space, let the qbuf keep a size that's sufficient for the currently process bulklen, and the current peak. Co-authored-by: sundb <sundbcn@gmail.com> Co-authored-by: yoav-steinberg <yoav@monfort.co.il>
* Adjustments to recent RM_StringTruncate fix (#3718) (#9125)Oran Agra2021-06-221-0/+1
| | | | | | | | | | - Introduce a new sdssubstr api as a building block for sdsrange. The API of sdsrange is many times hard to work with and also has corner case that cause bugs. sdsrange is easy to work with and also simplifies the implementation of sdsrange. - Revert the fix to RM_StringTruncate and just use sdssubstr instead of sdsrange. - Solve valgrind warnings from the new tests introduced by the previous PR.
* Fix the wrong reisze of querybuf (#9003)sundb2021-06-151-0/+1
| | | | | | | | | | | | | | | The initialize memory of `querybuf` is `PROTO_IOBUF_LEN(1024*16) * 2` (due to sdsMakeRoomFor being greedy), under `jemalloc`, the allocated memory will be 40k. This will most likely result in the `querybuf` being resized when call `clientsCronResizeQueryBuffer` unless the client requests it fast enough. Note that this bug existed even before #7875, since the condition for resizing includes the sds headers (32k+6). ## Changes 1. Use non-greedy sdsMakeRoomFor when allocating the initial query buffer (of 16k). 1. Also use non-greedy allocation when working with BIG_ARG (we won't use that extra space anyway) 2. in case we did use a greedy allocation, read as much as we can into the buffer we got (including internal frag), to reduce system calls. 3. introduce a dedicated constant for the shrinking (same value as before) 3. Add test for querybuf. 4. improve a maxmemory test by ignoring the effect of replica query buffers (can accumulate many ACKs on slow env) 5. improve a maxmemory by disabling slowlog (it will cause slight memory growth on slow env).
* Add run all test support with define REDIS_TEST (#8570)sundb2021-03-101-1/+1
| | | | | | | | | | | | 1. Add `redis-server test all` support to run all tests. 2. Add redis test to daily ci. 3. Add `--accurate` option to run slow tests for more iterations (so that by default we run less cycles (shorter time, and less prints). 4. Move dict benchmark to REDIS_TEST. 5. fix some leaks in tests 6. make quicklist tests run on a specific fill set of options rather than huge ranges 7. move some prints in quicklist test outside their loops to reduce prints 8. removing sds.h from dict.c since it is now used in both redis-server and redis-cli (uses hiredis sds)
* Add proc-title-template option. (#8397)Yossi Gottlieb2021-01-281-0/+8
| | | | | Make it possible to customize the process title, i.e. include custom strings, immutable configuration like port, tls-port, unix socket name, etc.
* Sanitize dump payload: fail RESTORE if memory allocation failsOran Agra2020-12-061-0/+1
| | | | | When RDB input attempts to make a huge memory allocation that fails, RESTORE should fail gracefully rather than die with panic
* Mark extern definition of SDS_NOINIT in sds.hKhem Raj2019-12-211-1/+1
| | | | | | | This helps in avoiding multiple definition of this variable, its also defined globally in sds.c Signed-off-by: Khem Raj <raj.khem@gmail.com>
* Merge pull request #3828 from oranagra/sdsnewlen_prSalvatore Sanfilippo2018-02-271-0/+1
|\ | | | | add SDS_NOINIT option to sdsnewlen to avoid unnecessary memsets.
| * add SDS_NOINIT option to sdsnewlen to avoid unnecessary memsets.oranagra2017-02-231-0/+1
| | | | | | | | | | this commit also contains small bugfix in rdbLoadLzfStringObject a bug that currently has no implications.
* | fix processing of large bulks (above 2GB)Oran Agra2017-12-291-3/+3
|/ | | | | | | | | - protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer readQueryFromClient potential overflow - rioWriteBulkCount used int, although rioWriteBulkString gave it size_t - several places in sds.c that used int for string length or index. - bugfix in RM_SaveAuxField (return was 1 or -1 and not length) - RM_SaveStringBuffer was limitted to 32bit length
* Lua debugger: use sds_malloc() to allocate eval cli array.antirez2015-11-171-0/+8
| | | | | | | | | | | | | Redis-cli handles the debugger "eval" command in a special way since sdssplitargs() would not be ok: we need to send the Redis debugger the whole Lua script without any parsing. However in order to later free the argument vector inside redis-cli using just sdsfreesplitres(), we need to allocate the array of SDS pointers using the same allocator SDS is using, that may differ to what Redis is using. So now a newer version of SDS exports sds_malloc() and other allocator functions to give access, to the program it is linked to, the allocator used internally by SDS.
* SDS: Copyright updated further.antirez2015-07-251-0/+1
|
* SDS: changes to unify Redis SDS with antirez/sds repo.antirez2015-07-251-1/+1
|
* SDS: Copyright notice updated.antirez2015-07-251-2/+3
|
* SDS: sdsjoinsds() call ported from antirez/sds fork.antirez2015-07-251-0/+1
|
* SDS: New sds type 5 implemented.antirez2015-07-151-21/+47
| | | | | | | | | | | | | | This is an attempt to use the refcount feature of the sds.c fork provided in the Pull Request #2509. A new type, SDS_TYPE_5 is introduced having a one byte header with just the string length, without information about the available additional length at the end of the string (this means that sdsMakeRoomFor() will be required each time we want to append something, since the string will always report to have 0 bytes available). More work needed in order to avoid common SDS functions will pay the cost of this type. For example both sdscatprintf() and sdscatfmt() should try to upgrade to SDS_TYPE_8 ASAP when appending chars.
* sds size classes - memory optimizationOran Agra2015-07-141-9/+140
|
* sdsnative() removed: New rdb.c API can load native strings.antirez2015-01-081-1/+0
|
* Add sdsnative()Matt Stancliff2015-01-021-0/+1
| | | | | | Use the existing memory space for an SDS to convert it to a regular character buffer so we don't need to allocate duplicate space just to extract a usable buffer for native operations.
* Allow all code tests to run using Redis argsMatt Stancliff2014-12-231-0/+4
| | | | | | | | | | | | | | | | | | | | | Previously, many files had individual main() functions for testing, but each required being compiled with their own testing flags. That gets difficult when you have 8 different flags you need to set just to run all tests (plus, some test files required other files to be compiled aaginst them, and it seems some didn't build at all without including the rest of Redis). Now all individual test main() funcions are renamed to a test function for the file itself and one global REDIS_TEST define enables testing across the entire codebase. Tests can now be run with: - `./redis-server test <test>` e.g. ./redis-server test ziplist If REDIS_TEST is not defined, then no tests get included and no tests are included in the final redis-server binary.
* sdsformatip() removed.antirez2014-12-111-1/+0
| | | | | Specialized single-use function. Not the best match for sds.c btw. Also genClientPeerId() is no longer static: we need symbols.
* Add centralized IP/Peer formatting functionsMatt Stancliff2014-12-111-0/+1
| | | | | This stops us from needing to manually check against ":" to add brackets around IPv6 addresses everywhere.
* Use unsigned integers in SDS header.antirez2014-08-131-2/+2
| | | | This raises the max string to 4GB without any downside.
* Use sdscatfmt() in getClientInfoString() to make it faster.antirez2014-04-281-0/+1
|
* Fix sdsempty() prototype in sds.h.antirez2013-08-121-1/+1
|
* sdsrange() does not need to return a value.antirez2013-07-241-1/+1
| | | | | | Actaully the string is modified in-place and a reallocation is never needed, so there is no need to return the new sds string pointer as return value of the function, that is now just "void".
* sds.c: new function sdsjoin() to join strings.antirez2013-07-041-0/+1
|
* sds.c: sdssplitargs_free() removed as it was a duplicate.antirez2013-03-061-1/+0
|
* Added consts keyword where possibleErik Dubbelboer2012-03-301-11/+11
|
* sds.c: sdsAllocSize() function added.antirez2012-03-141-0/+2
|
* sds.c no longe pre-allocate more than 1MB of free space ahead. This fixes ↵antirez2012-01-161-0/+2
| | | | issue #252.
* Added sdscatsds() to sds.c/hantirez2011-11-211-0/+1
|
* sdsMakeRoomFor() exposed as public API. sdsIncrLen() added. Both the changes ↵antirez2011-11-021-0/+4
| | | | make it possible to copy stuff from a system call to an sds buffer without the need of an additional buffer and copying overhead.
* Re-use AOF buffer when it is small enoughPieter Noordhuis2011-09-131-0/+1
|
* Make sure error and status replies emitted by Lua scripts can never have ↵antirez2011-05-251-0/+1
| | | | more than a newline, otherwise it is a protocol violation and clients will desync.
* Inline sdslen and sdsavail (thanks to @bitbckt)Pieter Noordhuis2011-05-051-0/+10
|
* initial cluster config load codeantirez2011-04-071-0/+1
|
* Change function name to match what it doesPieter Noordhuis2010-12-101-1/+1
|
* Add generic function to grow an sds valuePieter Noordhuis2010-12-101-1/+1
| | | | | | Move logic concerned with setting a bit in an sds to the SETBIT command instead of keeping it in sds.c. The function to grow an sds can and will be reused for a command to set a range within a string value.
* Add commands SETBIT/GETBITPieter Noordhuis2010-12-091-0/+1
|
* Add sds function that can be called with va_listPieter Noordhuis2010-09-021-0/+2
|
* redis cli argument splitting is general and is now moved into the sds.c libantirez2010-08-051-0/+1
|
* redis.c split into many different C files.antirez2010-07-011-0/+74
networking related stuff moved into networking.c moved more code more work on layout of source code SDS instantaneuos memory saving. By Pieter and Salvatore at VMware ;) cleanly compiling again after the first split, now splitting it in more C files moving more things around... work in progress split replication code splitting more Sets split Hash split replication split even more splitting more splitting minor change
View Lorry Controller Status