summaryrefslogtreecommitdiff
path: root/Modules/ExternalData.cmake
Commit message (Collapse)AuthorAgeFilesLines
* ExternalData: Improve robustness on Windows to copy a data object to a fileBrad King2022-11-171-1/+1
| | | | | | | | | | When an external data object was recently created on disk, perhaps fetched by the current process or another process, it may be inaccessible on Windows for a short time. Tell our `file(COPY_FILE)` call to retry a few times to copy the object to the final file path. Hopefully this will resolve our long-standing spurious failures of the `Module.ExternalData` test on Windows.
* ExternalData: Improve error message when we fail to create a fileBrad King2021-04-221-1/+1
| | | | Report the underlying system error.
* ExternalData: use `file(CREATE_LINK)`Ben Boeckel2021-03-111-1/+2
| | | | This avoids spawning a new CMake command which isn't necessary.
* ExternalData: use `file(COPY_FILE)` instead of forking CMakeBen Boeckel2021-03-111-1/+1
|
* ExternalData: Avoid replacing a concurrently-created objectBrad King2021-03-051-1/+8
| | | | | | | | | | | | | | | If more than one content link references the same object, the build system may launch multiple download processes for the same object concurrently. Use whichever one finishes first, and discard the others. Without this, we replace the objects and use the last finisher instead of the first. This is okay on non-Windows platforms where `rename(2)` gives reliable atomic replacement. However, on Windows platforms and NTFS this is less reliable. I've observed `MoveFileEx` somehow cause another process to get `ERROR_SHARING_VIOLATION` when attempting to read the destination file. We may be able to improve the `file(RENAME)` implementation on modern Windows 10 versions, but for ExternalData's use case it is simpler to just not replace existing objects.
* Help: Add `.. versionadded` directives to module docsNikita Nemkin2020-12-021-10/+26
| | | | Issue: #19715
* ExternalData: add support for suppressing progress during the buildBen Boeckel2020-10-231-2/+38
| | | | | | | | | During CI builds (at least), download progress is just noise. Allow it to be suppressed. Default to `OFF` for Ninja due to the tool's behavior of not showing output until a command is complete (which makes any progress reporting of little use) and `ON` otherwise.
* Help: correct typo in ExternalData (duplicate "of")Joachim Wuttke (o)2020-09-071-1/+1
|
* Replaces execute_process calls to touch files with file(TOUCH) callsShane Parris2018-02-151-1/+1
|
* Use string(APPEND) in ModulesDaniel Pfeifer2017-05-171-1/+1
| | | | | | | Automate with: find Modules -type f -print0 | xargs -0 perl -i -0pe \ 's/set\(([a-zA-Z0-9_\$\{\}]+)(\s+)"\$\{\1\}([^"])/string(APPEND \1\2"\3/g'
* ExternalData: Add support for SHA-3 algorithmsBrad King2016-11-111-2/+7
|
* ExternalData: Add support for multiple hash algorithmsMatt McCormick2016-10-181-19/+91
| | | | | | | | | | Add support for projects to have `Data.txt.md5` *and* `Data.txt.sha512` where the content links hold hashes for the same file. Check all `ExternalData_URL_TEMPLATES` entries in order for all available hashes. The data acquisition is considered a failure if none of the available URL resources has any of the given hashes. This makes it possible to have multiple data server resources where all servers do not support all hashing algorithms.
* Simplify CMake per-source license noticesBrad King2016-09-271-13/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Per-source copyright/license notice headers that spell out copyright holder names and years are hard to maintain and often out-of-date or plain wrong. Precise contributor information is already maintained automatically by the version control tool. Ultimately it is the receiver of a file who is responsible for determining its licensing status, and per-source notices are merely a convenience. Therefore it is simpler and more accurate for each source to have a generic notice of the license name and references to more detailed information on copyright holders and full license terms. Our `Copyright.txt` file now contains a list of Contributors whose names appeared source-level copyright notices. It also references version control history for more precise information. Therefore we no longer need to spell out the list of Contributors in each source file notice. Replace CMake per-source copyright/license notice headers with a short description of the license and links to `Copyright.txt` and online information available from "https://cmake.org/licensing". The online URL also handles cases of modules being copied out of our source into other projects, so we can drop our notices about replacing links with full license text. Run the `Utilities/Scripts/filter-notices.bash` script to perform the majority of the replacements mechanically. Manually fix up shebang lines and trailing newlines in a few files. Manually update the notices in a few files that the script does not handle.
* ExternalData: Tolerate files duplicated across multiple targetsBrad King2016-09-071-29/+47
| | | | | | | | | | | | | | | | | | | | | | If multiple ExternalData_Target_Add calls generate the same output file then we need to avoid calling add_custom_command multiple times with that output. This was already done within a single target by setting a variable in the local function scope. This will not be visible in other calls though so we need to use a directory property instead to prevent adding a custom command multiple times for one output in a directory. Normally it is not safe to have multiple custom commands that produce the same output file across multiple independent targets, but since we use atomic replacement of outputs the resulting races should not be a problem. For the convenience of projects, tolerate this instead of diagnosing it. In particular, we previously allowed up to two copies of the custom command in one directory because CMake has a fallback from MAIN_DEPENDENCY to an `<output>.rule` file. While at it, add a note to the documentation that typically only one external data target should be needed for a project. Reported-by: David Manthey <david.manthey@kitware.com>
* Use string(APPEND) in ModulesDaniel Pfeifer2016-07-281-8/+8
| | | | | | | Automate with: find Modules -type f -print0 | xargs -0 perl -i -0pe \ 's/set\(([a-zA-Z0-9_]+)(\s+)"\$\{\1\}([^"])/string(APPEND \1\2"\3/g'
* ExternalData: Hide security-sensitive details from outputAlex Turbov2016-07-131-1/+2
| | | | Do not disclose credentials when printing URIs used to fetch external resources.
* ExternalData: Add option to disable use of symlinksBrad King2015-05-221-1/+8
| | | | | | | | | Add an ExternalData_NO_SYMLINKS to enable use of copies instead of symlinks to populate the real data files behind a DATA{} reference. This will be useful on UNIX-like systems when the underlying filesystem does not actually support symbolic links. Suggested-by: Matt McCormick <matt.mccormick@kitware.com>
* ExternalData: Add option to recursively match under directoriesBrad King2015-03-271-3/+20
| | | | | | Extend the ``DATA{Dir/,...}`` syntax with a new ``RECURSE:`` option to enable recursive matching of associated files. This will allow an entire directory tree of data to be referenced at once.
* ExternalData: Parameterize internal file(GLOB) operation selectionBrad King2015-03-271-6/+10
| | | | | | Extend the _ExternalData_arg_find_files signature with an option to specify the kind of file(GLOB) operation to be performed. Set CMP0009 to NEW so that GLOB_RECURSE does not follow symlinks.
* ExternalData: Add support for custom algorithm-to-URL mappingBrad King2015-02-251-1/+40
| | | | | | | | | Allow URL templates to contain a %(algo:<key>) placeholder that is replaced by mapping the canonical hash algorithm name through a map defined by the <key>. Extend the Module.ExternalData test to cover the behavior. Extend the RunCMake.ExternalData test to cover error cases.
* Help: Add notes for topic 'ExternalData-custom-download'Brad King2015-01-151-0/+2
|
* ExternalData: Add support for custom download scriptsBrad King2015-01-151-1/+104
| | | | | | | | Add support for a special URL template to map the fetch operation to a project-specified .cmake script insead of using file(DOWNLOAD). Extend the Module.ExternalData test to cover the behavior. Extend the RunCMake.ExternalData test to cover error cases.
* ExternalData: Improve documentation organizationBrad King2015-01-151-13/+18
| | | | | | | | | Move the basic DATA{} description to a section just before the file series description. Move all sections on referencing files into subsections of a common "Referencing Files" section. Subsume example usage into the introduction since it gives a high-level starting point to understand the rest of the docs.
* ExternalData: Re-order documentationBrad King2015-01-151-97/+97
| | | | | | Put example usage just after the intro. Put hash algos last. Put variables right after functions since they both cover the module API.
* ExternalData: Document all variables defined by moduleBrad King2015-01-151-5/+21
| | | | | Extend the "Module Variables" section to list variables previously mentioned only in text.
* ExternalData: Split documentation into sectionsBrad King2015-01-151-67/+111
| | | | Also explicitly mark functions and variables.
* ExternalData: Convert docs to a bracket commentBrad King2015-01-151-184/+185
| | | | | Use a bracket comment to hold the documentation instead of a block of line comments. This will make further updates easier.
* ExternalData: Drop 'not security software' disclaimer (#15060)Brad King2014-08-051-1/+1
| | | | | | | Let clients do their own research on the current strength of each hash algorithm to choose what is best for their needs. Suggested-by: Nico Schlömer <nico.schloemer@gmail.com>
* ExternalData: Warn on missing file instead of failingBrad King2014-04-171-15/+29
| | | | | | | | | | | | When the primary source tree path named by a DATA{} reference does not exist, produce an AUTHOR_WARNING instead of a FATAL_ERROR. This is useful when writing a new DATA{} reference to a test reference output that has not been created yet. This way the developer can run the test, manually verify the output, and then copy it into place to provide the reference and eliminate the warning. If the named source tree path is expected to be a file but exists as a directory, we still need to produce a FATAL_ERROR.
* Clean up usage of if(... MATCHES regex) followed string(REGEX REPLACE regex)Rolf Eike Beer2014-04-141-6/+4
| | | | | | The matches have already been calculated and can simply be taken from CMAKE_MATCH_n variables. This avoids multiple compilations of the same or very similar regular expressions.
* ExternalData: Allow local stores without any URL templatesBrad King2013-11-131-5/+10
| | | | | | | | | Allow ExternalData_URL_TEMPLATES to be empty if a value for ExternalData_OBJECT_STORES is provided. Assume in this use case that the object stores will already contain all needed objects. Extend the Module.ExternalData test to cover this case (all objects in stores). Extend the RunCMake.ExternalData test to cover the non-failure message case when stores are provided without URL templates.
* ExternalData: Show example network path URL templateBrad King2013-11-121-0/+1
| | | | | In the ExternalData_URL_TEMPLATES example, add an entry showing how to use the "file://" protocol with a network filesystem path.
* ExternalData: Format module documentationBrad King2013-11-121-107/+98
| | | | | | | | Manually revise the .rst format of the documentation. Use inline literal quotes appropriately in paragraph text. Move the :: literal block openers to the end of the preceding paragraphs. Convert the command signature documentation and examples to cmake code-block directives.
* Convert builtin help to reStructuredText source filesKitware Robot2013-10-151-116/+164
| | | | | | | | Run the convert-help.bash script to convert documentation: ./convert-help.bash "/path/to/CMake-build/bin" Then remove it.
* replace string(... MATCHES "^const$) with string(... STREQUAL "const")Rolf Eike Beer2013-06-021-1/+1
|
* ExternalData: Do not re-stage staged object filesBrad King2013-05-241-1/+4
| | | | | | | | | | | | | The ExternalData_LINK_CONTENT option tells ExternalData to convert real data files it finds into content links and to "stage" the original content in a ".ExternalData_<algo>_<hash>" file. However, after a data object has been staged it is possible that a user-provided pattern in the "REGEX:" option will later match the staged object file. We must not process staged object files even when a user pattern matches them. Fix the implementation to not match a staged object file as a normal data file for conversion. Extend the RunCMake.ExternalData test to cover this case.
* ExternalData: Preserve escaped semicolons during argument expansionBrad King2013-03-121-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The CMake language implicitly flattens lists so a ";" in a list element must be escaped with a backslash. List expansion removes backslashes escaping semicolons to leave raw semicolons in the values. Teach ExternalData_Add_Test and ExternalData_Expand_Arguments to re-escape semicolons found in list elements so the resulting argument lists work as if constructed directly by the set() command. For example: ExternalData_Add_Test(Data NAME test1 COMMAND ... "a\\;b") ExternalData_Expand_Arguments(Data args2 "c\\;d") add_test(NAME test2 COMMAND ... ${args2}) should be equivalent to set(args1 "a\\;b") add_test(NAME test1 COMMAND ... ${args1}) set(args2 "c\\;d") add_test(NAME test2 COMMAND ... ${args2}) which is equivalent to add_test(NAME test1 COMMAND ... "a;b") add_test(NAME test2 COMMAND ... "c;d") Note that it is not possible to make ExternalData_Add_Test act exactly like add_test when quoted arguments contain semicolons because the CMake language flattens lists when constructing function ARGN values. This re-escape approach at least allows test arguments to have semicolons. While at it, teach ExternalData APIs to not transform "DATA{...;...}" arguments because the contained semicolons are non-sensical. Suggested-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>
* ExternalData: Attach download rules to content links in IDEsBrad King2013-02-041-2/+2
| | | | | | | Each data file to be created in the build tree corresponds one-to-one with a raw file or content link in the source tree. Use the MAIN_DEPENDENCY of add_custom_command to attach the build rule to the source tree file. This looks much nicer in the IDE project file browser and avoids ".rule" files.
* ExternalData: Collapse ../ components in DATA{} pathsBrad King2013-01-301-1/+2
| | | | | Relative path components need to be normalized out even if they appear in the middle of a caller-supplied string.
* ExternalData: Add support for SHA 1 and 2 hash algorithmsBrad King2013-01-301-2/+7
| | | | | Update the Module.ExternalData and RunCMake.ExternalData tests to cover some of them.
* ExternalData: Generalize hash algo/ext handlingBrad King2013-01-301-11/+9
| | | | | | | Use private global variables _ExternalData_REGEX_(ALGO|EXT) to match the possible hash algorithm names and extensions in regular expressions. Use "file(<algo>)" instead of "cmake -E md5sum" to compute hashes without a child process and to support more hash algorithms.
* ExternalData: Allow DATA{} syntax to reference directoriesBrad King2013-01-301-7/+38
| | | | | | | Use a trailing slash to reference a directory. Require that a list of associated files be specified to select from within the directory. One may simply use DATA{Dir/,REGEX:.*} to reference all files but get a directory passed on the command line.
* ExternalData: Allow ()-groups in series match regexBrad King2013-01-301-5/+8
| | | | | | | Refactor use of the ExternalData_SERIES_MATCH value to avoid assuming that it has no ()-groups that interfere with group indexing. Extend the Module.ExternalData test to cover this case.
* ExternalData: Improve series matching using an explicit syntaxBrad King2013-01-291-27/+55
| | | | | | | | | | | | | | Automatic series recognition can generate false positives too easily when the default series configuration is flexible enough to handle common cases. Avoid false positives by requiring an explicit syntax to activate series recognition. Choose the syntax DATA{<name>,:} to be short, simple, and look like a vertical ellipsis. This allows us to improve the default series match configuration. Allow series references to contain one of the numbered file names. Allow '-' as a separator in addition to '.' and '_'. Document what the default configuration matches. Also provide more options to configure series <name> parsing.
* ExternalData: Remove unused private interfaceBrad King2013-01-281-10/+0
| | | | | The 'store' action was part of an older design and is no longer needed. Remove it.
* ExternalData: Cleanup stray TODO and typo in commentsBrad King2013-01-281-2/+1
| | | | | Remove an outdated TODO line left from earlier designs. Also fix a typo.
* ExternalData: Do not match directory names when resolving DATA{}Brad King2013-01-281-2/+9
| | | | | | Our content link and object download infrastructure supports only blobs (files), not trees (directories). Do not allow references to end in a slash.
* ExternalData: Remove compatibility with CMake < 2.8.5Brad King2013-01-241-17/+6
| | | | | Now that the module comes in a sufficiently new CMake it does not need to support older versions.
* Add ExternalData moduleBrad King2013-01-241-0/+710
Take files we previously distributed in ITK 4.3.1: CMake/ExternalData.cmake CMake/ExternalData_config.cmake.in and add them for distribution in upstream CMake. Update the copyright notice block format to follow CMake conventions.
View Lorry Controller Status