summaryrefslogtreecommitdiff
path: root/compiler/GHC/Unit
Commit message (Collapse)AuthorAgeFilesLines
* Desugar RecordUpd in `tcExpr`wip/T18802CarrieMY2022-05-251-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch typechecks record updates by desugaring them inside the typechecker using the HsExpansion mechanism, and then typechecking this desugared result. Example: data T p q = T1 { x :: Int, y :: Bool, z :: Char } | T2 { v :: Char } | T3 { x :: Int } | T4 { p :: Float, y :: Bool, x :: Int } | T5 The record update `e { x=e1, y=e2 }` desugars as follows e { x=e1, y=e2 } ===> let { x' = e1; y' = e2 } in case e of T1 _ _ z -> T1 x' y' z T4 p _ _ -> T4 p y' x' The desugared expression is put into an HsExpansion, and we typecheck that. The full details are given in Note [Record Updates] in GHC.Tc.Gen.Expr. Fixes #2595 #3632 #10808 #10856 #16501 #18311 #18802 #21158 #21289 Updates haddock submodule
* Change `Backend` type and remove direct dependencieswip/backend-as-recordNorman Ramsey2022-05-211-4/+3
| | | | | | | | | | | | | | | | | | | With this change, `Backend` becomes an abstract type (there are no more exposed value constructors). Decisions that were formerly made by asking "is the current back end equal to (or different from) this named value constructor?" are now made by interrogating the back end about its properties, which are functions exported by `GHC.Driver.Backend`. There is a description of how to migrate code using `Backend` in the user guide. Clients using the GHC API can find a backdoor to access the Backend datatype in GHC.Driver.Backend.Internal. Bumps haddock submodule. Fixes #20927
* Provide efficient unionMG function for combining two module graphs.Matthew Pickering2022-04-291-4/+25
| | | | | | This function is used by API clients (hls). This supercedes !6922
* Enable eventlog support in all ways by defaultBen Gamari2022-04-271-8/+2
| | | | | | | | | | | | | | | | | Here we deprecate the eventlogging RTS ways and instead enable eventlog support in the remaining ways. This simplifies packaging and reduces GHC compilation times (as we can eliminate two whole compilations of the RTS) while simplifying the end-user story. The trade-off is a small increase in binary sizes in the case that the user does not want eventlogging support, but we think that this is a fine trade-off. This also revealed a latent RTS bug: some files which included `Cmm.h` also assumed that it defined various macros which were in fact defined by `Config.h`, which `Cmm.h` did not include. Fixing this in turn revealed that `StgMiscClosures.cmm` failed to import various spinlock statistics counters, as evidenced by the failed unregisterised build. Closes #18948.
* driver: In oneshot mode, look for interface files in hidirMatthew Pickering2022-04-011-3/+10
| | | | | | | | | | | | | | | | How things should work: * -i is the search path for source files * -hidir explicitly sets the search path for interface files and the output location for interface files. * -odir sets the search path and output location for object files. Before in one shot mode we would look for the interface file in the search locations given by `-i`, but then set the path to be in the `hidir`, so in unusual situations the finder could find an interface file in the `-i` dir but later fail because it tried to read the interface file from the `-hidir`. A bug identified by #20569
* driver: Improve -Wunused-packages error message (and simplify implementation)Matthew Pickering2022-04-011-9/+12
| | | | | | | | | | | | | | | | | | | | | In the past I improved the part of -Wunused-packages which found which packages were used. Now I improve the part which detects which ones were specified. The key innovation is to use the explicitUnits field from UnitState which has the result of resolving the package flags, so we don't need to mess about with the flag arguments from DynFlags anymore. The output now always includes the package name and version (and the flag which exposed it). ``` The following packages were specified via -package or -package-id flags, but were not needed for compilation: - bytestring-0.11.2.0 (exposed by flag -package bytestring) - ghc-9.3 (exposed by flag -package ghc) - process-1.6.13.2 (exposed by flag -package process) ``` Fixes #21307
* Fix all invalid haddock comments in the compilerZubin Duggal2022-03-291-1/+1
| | | | Fixes #20935 and #20924
* hi haddock: Lex and store haddock docs in interface filesZubin Duggal2022-03-233-49/+45
| | | | | | | | | | | | | | | | | | Names appearing in Haddock docstrings are lexed and renamed like any other names appearing in the AST. We currently rename names irrespective of the namespace, so both type and constructor names corresponding to an identifier will appear in the docstring. Haddock will select a given name as the link destination based on its own heuristics. This patch also restricts the limitation of `-haddock` being incompatible with `Opt_KeepRawTokenStream`. The export and documenation structure is now computed in GHC and serialised in .hi files. This can be used by haddock to directly generate doc pages without reparsing or renaming the source. At the moment the operation of haddock is not modified, that's left to a future patch. Updates the haddock submodule with the minimum changes needed.
* driver: Properly add an edge between a .hs and its hs-boot fileMatthew Pickering2022-03-011-5/+3
| | | | | | | | | | | | | | As noted in #21071 we were missing adding this edge so there were situations where the .hs file would get compiled before the .hs-boot file which leads to issues with -j. I fixed this properly by adding the edge in downsweep so the definition of nodeDependencies can be simplified to avoid adding this dummy edge in. There are plenty of tests which seem to have these redundant boot files anyway so no new test. #21094 tracks the more general issue of identifying redundant hs-boot and SOURCE imports.
* Fix longstanding issue with moduleGraphNodes - no hs-boot files caseMatthew Pickering2022-03-011-12/+28
| | | | | | | | | | | | | | | | | | | | In the case when we tell moduleGraphNodes to drop hs-boot files the idea is to collapse hs-boot files into their hs file nodes. In the old code * nodeDependencies changed edges from IsBoot to NonBoot * moduleGraphNodes just dropped boot file nodes The net result is that any dependencies of the hs-boot files themselves were dropped. The correct thing to do is * nodeDependencies changes edges from IsBoot to NonBoot * moduleGraphNodes merges dependencies of IsBoot and NonBoot nodes. The result is a properly quotiented dependency graph which contains no hs-boot files nor hs-boot file edges. Why this didn't cause endless issues when compiling with boot files, we will never know.
* Suggestions due to hlintMatthew Pickering2022-02-245-6/+2
| | | | | It turns out this job hasn't been running for quite a while (perhaps ever) so there are quite a few failures when running the linter locally.
* driver: Remove needsTemplateHaskellOrQQ from ModuleGraphMatthew Pickering2022-02-231-19/+2
| | | | | | | | | | | | | | | | | | | | | | | | | The idea of the needsTemplateHaskellOrQQ query is to check if any of the modules in a module graph need Template Haskell then enable -dynamic-too if necessary. This is quite imprecise though as it will enable -dynamic-too for all modules in the module graph even if only one module uses template haskell, with multiple home units, this is obviously even worse. With -fno-code we already have similar logic to enable code generation just for the modules which are dependeded on my TemplateHaskell modules so we use the same code path to decide whether to enable -dynamic-too rather than using this big hammer. This is part of the larger overall goal of moving as much statically known configuration into the downsweep as possible in order to have fully decided the build plan and all the options before starting to build anything. I also included a fix to #21095, a long standing bug with with the logic which is supposed to enable the external interpreter if we don't have the internal interpreter. Fixes #20696 #21095
* Remove mg_boot field from ModuleGraphMatthew Pickering2022-02-231-11/+1
| | | | | It was unused in the compiler so I have removed it to streamline ModuleGraph.
* Track object file dependencies for TH accurately (#20604)Zubin Duggal2022-02-202-1/+23
| | | | | | | | | | | | | | | | | | | `hscCompileCoreExprHook` is changed to return a list of `Module`s required by a splice. These modules are accumulated in the TcGblEnv (tcg_th_needed_mods). Dependencies on the object files of these modules are recording in the interface. The data structures in `LoaderState` are replaced with more efficient versions to keep track of all the information required. The MultiLayerModulesTH_Make allocations increase slightly but runtime is faster. Fixes #20604 ------------------------- Metric Increase: MultiLayerModulesTH_Make -------------------------
* Tag inference work.Andreas Klebinger2022-02-121-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This does three major things: * Enforce the invariant that all strict fields must contain tagged pointers. * Try to predict the tag on bindings in order to omit tag checks. * Allows functions to pass arguments unlifted (call-by-value). The former is "simply" achieved by wrapping any constructor allocations with a case which will evaluate the respective strict bindings. The prediction is done by a new data flow analysis based on the STG representation of a program. This also helps us to avoid generating redudant cases for the above invariant. StrictWorkers are created by W/W directly and SpecConstr indirectly. See the Note [Strict Worker Ids] Other minor changes: * Add StgUtil module containing a few functions needed by, but not specific to the tag analysis. ------------------------- Metric Decrease: T12545 T18698b T18140 T18923 LargeRecord Metric Increase: LargeRecord ManyAlternatives ManyConstructors T10421 T12425 T12707 T13035 T13056 T13253 T13253-spj T13379 T15164 T18282 T18304 T18698a T1969 T20049 T3294 T4801 T5321FD T5321Fun T783 T9233 T9675 T9961 T19695 WWRec -------------------------
* Fix some notesMatthew Pickering2022-02-083-3/+3
|
* compiler: Introduce and use RoughMap for instance environmentsBen Gamari2022-02-041-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Here we introduce a new data structure, RoughMap, inspired by the previous `RoughTc` matching mechanism for checking instance matches. This allows [Fam]InstEnv to be implemented as a trie indexed by these RoughTc signatures, reducing the complexity of instance lookup and FamInstEnv merging (done during the family instance conflict test) from O(n) to O(log n). The critical performance improvement currently realised by this patch is in instance matching. In particular the RoughMap mechanism allows us to discount many potential instances which will never match for constraints involving type variables (see Note [Matching a RoughMap]). In realistic code bases matchInstEnv was accounting for 50% of typechecker time due to redundant work checking instances when simplifying instance contexts when deriving instances. With this patch the cost is significantly reduced. The larger constants in InstEnv creation do mean that a few small tests regress in allocations slightly. However, the runtime of T19703 is reduced by a factor of 4. Moreover, the compilation time of the Cabal library is slightly improved. A couple of test cases are included which demonstrate significant improvements in compile time with this patch. This unfortunately does not fix the testcase provided in #19703 but does fix #20933 ------------------------- Metric Decrease: T12425 Metric Increase: T13719 T9872a T9872d hard_hole_fits ------------------------- Co-authored-by: Matthew Pickering <matthewtpickering@gmail.com>
* Fix a few Note inconsistenciesBen Gamari2022-02-011-4/+4
|
* Multiple Home UnitsMatthew Pickering2021-12-2812-194/+814
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Multiple home units allows you to load different packages which may depend on each other into one GHC session. This will allow both GHCi and HLS to support multi component projects more naturally. Public Interface ~~~~~~~~~~~~~~~~ In order to specify multiple units, the -unit @⟨filename⟩ flag is given multiple times with a response file containing the arguments for each unit. The response file contains a newline separated list of arguments. ``` ghc -unit @unitLibCore -unit @unitLib ``` where the `unitLibCore` response file contains the normal arguments that cabal would pass to `--make` mode. ``` -this-unit-id lib-core-0.1.0.0 -i -isrc LibCore.Utils LibCore.Types ``` The response file for lib, can specify a dependency on lib-core, so then modules in lib can use modules from lib-core. ``` -this-unit-id lib-0.1.0.0 -package-id lib-core-0.1.0.0 -i -isrc Lib.Parse Lib.Render ``` Then when the compiler starts in --make mode it will compile both units lib and lib-core. There is also very basic support for multiple home units in GHCi, at the moment you can start a GHCi session with multiple units but only the :reload is supported. Most commands in GHCi assume a single home unit, and so it is additional work to work out how to modify the interface to support multiple loaded home units. Options used when working with Multiple Home Units There are a few extra flags which have been introduced specifically for working with multiple home units. The flags allow a home unit to pretend it’s more like an installed package, for example, specifying the package name, module visibility and reexported modules. -working-dir ⟨dir⟩ It is common to assume that a package is compiled in the directory where its cabal file resides. Thus, all paths used in the compiler are assumed to be relative to this directory. When there are multiple home units the compiler is often not operating in the standard directory and instead where the cabal.project file is located. In this case the -working-dir option can be passed which specifies the path from the current directory to the directory the unit assumes to be it’s root, normally the directory which contains the cabal file. When the flag is passed, any relative paths used by the compiler are offset by the working directory. Notably this includes -i and -I⟨dir⟩ flags. -this-package-name ⟨name⟩ This flag papers over the awkward interaction of the PackageImports and multiple home units. When using PackageImports you can specify the name of the package in an import to disambiguate between modules which appear in multiple packages with the same name. This flag allows a home unit to be given a package name so that you can also disambiguate between multiple home units which provide modules with the same name. -hidden-module ⟨module name⟩ This flag can be supplied multiple times in order to specify which modules in a home unit should not be visible outside of the unit it belongs to. The main use of this flag is to be able to recreate the difference between an exposed and hidden module for installed packages. -reexported-module ⟨module name⟩ This flag can be supplied multiple times in order to specify which modules are not defined in a unit but should be reexported. The effect is that other units will see this module as if it was defined in this unit. The use of this flag is to be able to replicate the reexported modules feature of packages with multiple home units. Offsetting Paths in Template Haskell splices ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When using Template Haskell to embed files into your program, traditionally the paths have been interpreted relative to the directory where the .cabal file resides. This causes problems for multiple home units as we are compiling many different libraries at once which have .cabal files in different directories. For this purpose we have introduced a way to query the value of the -working-dir flag to the Template Haskell API. By using this function we can implement a makeRelativeToProject function which offsets a path which is relative to the original project root by the value of -working-dir. ``` import Language.Haskell.TH.Syntax ( makeRelativeToProject ) foo = $(makeRelativeToProject "./relative/path" >>= embedFile) ``` > If you write a relative path in a Template Haskell splice you should use the makeRelativeToProject function so that your library works correctly with multiple home units. A similar function already exists in the file-embed library. The function in template-haskell implements this function in a more robust manner by honouring the -working-dir flag rather than searching the file system. Closure Property for Home Units ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For tools or libraries using the API there is one very important closure property which must be adhered to: > Any dependency which is not a home unit must not (transitively) depend on a home unit. For example, if you have three packages p, q and r, then if p depends on q which depends on r then it is illegal to load both p and r as home units but not q, because q is a dependency of the home unit p which depends on another home unit r. If you are using GHC by the command line then this property is checked, but if you are using the API then you need to check this property yourself. If you get it wrong you will probably get some very confusing errors about overlapping instances. Limitations of Multiple Home Units ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are a few limitations of the initial implementation which will be smoothed out on user demand. * Package thinning/renaming syntax is not supported * More complicated reexports/renaming are not yet supported. * It’s more common to run into existing linker bugs when loading a large number of packages in a session (for example #20674, #20689) * Backpack is not yet supported when using multiple home units. * Dependency chasing can be quite slow with a large number of modules and packages. * Loading wired-in packages as home units is currently not supported (this only really affects GHC developers attempting to load template-haskell). * Barely any normal GHCi features are supported, it would be good to support enough for ghcid to work correctly. Despite these limitations, the implementation works already for nearly all packages. It has been testing on large dependency closures, including the whole of head.hackage which is a total of 4784 modules from 452 packages. Internal Changes ~~~~~~~~~~~~~~~~ * The biggest change is that the HomePackageTable is replaced with the HomeUnitGraph. The HomeUnitGraph is a map from UnitId to HomeUnitEnv, which contains information specific to each home unit. * The HomeUnitEnv contains: - A unit state, each home unit can have different package db flags - A set of dynflags, each home unit can have different flags - A HomePackageTable * LinkNode: A new node type is added to the ModuleGraph, this is used to place the linking step into the build plan so linking can proceed in parralel with other packages being built. * New invariant: Dependencies of a ModuleGraphNode can be completely determined by looking at the value of the node. In order to achieve this, downsweep now performs a more complete job of downsweeping and then the dependenices are recorded forever in the node rather than being computed again from the ModSummary. * Some transitive module calculations are rewritten to use the ModuleGraph which is more efficient. * There is always an active home unit, which simplifies modifying a lot of the existing API code which is unit agnostic (for example, in the driver). The road may be bumpy for a little while after this change but the basics are well-tested. One small metric increase, which we accept and also submodule update to haddock which removes ExtendedModSummary. Closes #10827 ------------------------- Metric Increase: MultiLayerModules ------------------------- Co-authored-by: Fendor <power.walross@gmail.com>
* Properly filter for module visibility in resolvePackageImportMatthew Pickering2021-12-231-1/+1
| | | | | | | | | | | | This completes the fix for #20779 / !7123. Beforehand, the program worked by accident because the two versions of the library happened to be ordered properly (due to how the hashes were computed). In the real world I observed them being the other way around which meant the final lookup failed because we weren't filtering for visibility. I modified the test so that it failed (and it's fixed by this patch).
* package imports: Take into account package visibility when renamingMatthew Pickering2021-12-091-0/+18
| | | | | | | | | | | | In 806e49ae the package imports refactoring code was modified to rename package imports. There was a small oversight which meant the code didn't account for module visibility. This patch fixes that oversight. In general the "lookupPackageName" function is unsafe to use as it doesn't account for package visiblity/thinning/renaming etc, there is just one use in the compiler which would be good to audit. Fixes #20779
* drop instance Semigroup InstalledModuleEnvPepe Iborra2021-11-251-1/+4
| | | | Instead, introduce plusInstalledModuleEnv
* Drop instance Semigroup ModuleEnvPepe Iborra2021-11-251-1/+0
| | | | There is more than one possible Semigroup and it is not needed since plusModuleEnv can be used directly
* Monoid instance for InstalledModuleEnvPepe Iborra2021-11-251-0/+3
|
* Correct retypechecking in --make modeMatthew Pickering2021-11-251-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note [Hydrating Modules] ~~~~~~~~~~~~~~~~~~~~~~~~ What is hydrating a module? * There are two versions of a module, the ModIface is the on-disk version and the ModDetails is a fleshed-out in-memory version. * We can **hydrate** a ModIface in order to obtain a ModDetails. Hydration happens in three different places * When an interface file is initially loaded from disk, it has to be hydrated. * When a module is finished compiling, we hydrate the ModIface in order to generate the version of ModDetails which exists in memory (see Note) * When dealing with boot files and module loops (see Note [Rehydrating Modules]) Note [Rehydrating Modules] ~~~~~~~~~~~~~~~~~~~~~~~~~~~ If a module has a boot file then it is critical to rehydrate the modules on the path between the two. Suppose we have ("R" for "recursive"): ``` R.hs-boot: module R where data T g :: T -> T A.hs: module A( f, T, g ) where import {-# SOURCE #-} R data S = MkS T f :: T -> S = ...g... R.hs: module R where data T = T1 | T2 S g = ...f... ``` After compiling A.hs we'll have a TypeEnv in which the Id for `f` has a type type uses the AbstractTyCon T; and a TyCon for `S` that also mentions that same AbstractTyCon. (Abstract because it came from R.hs-boot; we know nothing about it.) When compiling R.hs, we build a TyCon for `T`. But that TyCon mentions `S`, and it currently has an AbstractTyCon for `T` inside it. But we want to build a fully cyclic structure, in which `S` refers to `T` and `T` refers to `S`. Solution: **rehydration**. *Before compiling `R.hs`*, rehydrate all the ModIfaces below it that depend on R.hs-boot. To rehydrate a ModIface, call `typecheckIface` to convert it to a ModDetails. It's just a de-serialisation step, no type inference, just lookups. Now `S` will be bound to a thunk that, when forced, will "see" the final binding for `T`; see [Tying the knot](https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/compiler/tying-the-knot). But note that this must be done *before* compiling R.hs. When compiling R.hs, the knot-tying stuff above will ensure that `f`'s unfolding mentions the `LocalId` for `g`. But when we finish R, we carefully ensure that all those `LocalIds` are turned into completed `GlobalIds`, replete with unfoldings etc. Alas, that will not apply to the occurrences of `g` in `f`'s unfolding. And if we leave matters like that, they will stay that way, and *all* subsequent modules that import A will see a crippled unfolding for `f`. Solution: rehydrate both R and A's ModIface together, right after completing R.hs. We only need rehydrate modules that are * Below R.hs * Above R.hs-boot There might be many unrelated modules (in the home package) that don't need to be rehydrated. This dark corner is the subject of #14092. Suppose we add to our example ``` X.hs module X where import A data XT = MkX T fx = ...g... ``` If in `--make` we compile R.hs-boot, then A.hs, then X.hs, we'll get a `ModDetails` for `X` that has an AbstractTyCon for `T` in the the argument type of `MkX`. So: * Either we should delay compiling X until after R has beeen compiled. * Or we should rehydrate X after compiling R -- because it transitively depends on R.hs-boot. Ticket #20200 has exposed some issues to do with the knot-tying logic in GHC.Make, in `--make` mode. this particular issue starts [here](https://gitlab.haskell.org/ghc/ghc/-/issues/20200#note_385758). The wiki page [Tying the knot](https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/compiler/tying-the-knot) is helpful. Also closely related are * #14092 * #14103 Fixes tickets #20200 #20561
* More support for optional home-unitSylvain Henry2021-11-202-13/+30
| | | | | | | | | This is a preliminary refactoring for #14335 (supporting plugins in cross-compilers). In many places the home-unit must be optional because there won't be one available in the plugin environment (we won't be compiling anything in this environment). Hence we replace "HomeUnit" with "Maybe HomeUnit" in a few places and we avoid the use of "hsc_home_unit" (which is partial) in some few others.
* driver: Cache the transitive dependency calculation in ModuleGraphMatthew Pickering2021-11-111-4/+126
| | | | | | | | Two reasons for this change: 1. Avoid computing the transitive dependencies when compiling each module, this can save a lot of repeated work. 2. More robust to forthcoming changes to support multiple home units.
* Refactor package importsSylvain Henry2021-10-224-46/+53
| | | | | | | | | Use an (Raw)PkgQual datatype instead of `Maybe FastString` to represent package imports. Factorize the code that renames RawPkgQual into PkgQual in function `rnPkgQual`. Renaming consists in checking if the FastString is the magic "this" keyword, the home-unit unit-id or something else. Bump haddock submodule
* Remove IndefiniteSylvain Henry2021-10-227-51/+26
| | | | We no longer need it after previous IndefUnitId refactoring.
* Make sure ModIface values are still forced even if not writtenMatthew Pickering2021-10-201-3/+41
| | | | | | | | | | | | | | | | | | When we are not writing a ModIface to disk then the result can retain a lot of stuff. For example, in the case I was debugging the DocDeclsMap field was holding onto the entire HomePackageTable due to a single unforced thunk. Therefore, now if we're not going to write the interface then we still force deeply it in order to remove these thunks. The fields in the data structure are not made strict because when we read the field from the interface we don't want to load it immediately as there are parts of an interface which are unused a lot of the time. Also added a note to explain why not all the fields in a ModIface field are strict. The result of this is being able to load Agda in ghci and not leaking information across subsequent reloads.
* Temporary fix for leak with -fno-code (#20509)Matthew Pickering2021-10-191-0/+4
| | | | | | | | | | | | | | | | | | This hack inserted for backpack caused a very bad leak when using -fno-code where EPS entries would end up retaining stale HomePackageTables. For any interactive user, such as HLS, this is really bad as once the entry makes it's way into the EPS then it's there for the rest of the session. This is a temporary fix which "solves" the issue by filtering the HPT to only the part which is needed for the hack to work, but in future we want to separate out hole modules from the HPT entirely to avoid needing to do this kind of special casing. ------------------------- Metric Decrease: MultiLayerModulesDefsGhci -------------------------
* Remove DT_Failed stateMatthew Pickering2021-10-192-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the moment if `-dynamic-too` fails then we rerun the whole pipeline as if we were just in `-dynamic` mode. I argue this is a misfeature and we should remove the so-called `DT_Failed` mode. In what situations do we fall back to `DT_Failed`? 1. If the `dyn_hi` file corresponding to a `hi` file is missing completely. 2. If the interface hash of `dyn_hi` doesn't match the interface hash of `hi`. What happens in `DT_Failed` mode? * The whole compiler pipeline is rerun as if the user had just passed `-dynamic`. * Therefore `dyn_hi/dyn_o` files are used which don't agree with the `hi/o` files. (As evidenced by `dynamicToo001` test). * This is very confusing as now a single compiler invocation has produced further `hi`/`dyn_hi` files which are different to each other. Why should we remove it? * In `--make` mode, which is predominately used `DT_Failed` does not work (#19782), there can't be users relying on this functionality. * In `-c` mode, the recovery doesn't fix the root issue, which is the `dyn_hi` and `hi` files are mismatched. We should instead produce an error and pass responsibility to the build system using `-c` to ensure that the prerequisites for `-dynamic-too` (dyn_hi/hi) files are there before we start compiling. * It is a misfeature to support use cases like `dynamicToo001` which allow you to mix different versions of dynamic/non-dynamic interface files. It's more likely to lead to subtle bugs in your resulting programs where out-dated build products are used rather than a deliberate choice. * In practice, people are usually compiling with `-dynamic-too` rather than separately with `-dynamic` and `-static`, so the build products always match and `DT_Failed` is only entered due to compiler bugs (see !6583) What should we do instead? * In `--make` mode, for home packages check during recompilation checking that `dyn_hi` and `hi` are both present and agree, recompile the modules if they do not. * For package modules, when loading the interface check that `dyn_hi` and `hi` are there and that they agree but fail with an error message if they are not. * In `--oneshot` mode, fail with an error message if the right files aren't already there. Closes #19782 #20446 #9176 #13616
* driver: Correct output of -fno-code and -dynamic-tooMatthew Pickering2021-10-191-11/+10
| | | | | | | | | | | | | Before we would print [1 of 3] Compiling T[boot] ( T.hs-boot, nothing, T.dyn_o ) Which was clearly wrong for two reasons. 1. No dynamic object file was produced for T[boot] 2. The file would be called T.dyn_o-boot if it was produced. Fixes #20300
* driver: Cleanups related to ModLocationMatthew Pickering2021-10-195-64/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ModLocation is the data type which tells you the locations of all the build products which can affect recompilation. It is now computed in one place and not modified through the pipeline. Important locations will now just consult ModLocation rather than construct the dynamic object path incorrectly. * Add paths for dynamic object and dynamic interface files to ModLocation. * Always use the paths from mod location when looking for where to find any interface or object file. * Always use the paths in a ModLocation when deciding where to write an interface and object file. * Remove `dynamicOutputFile` and `dynamicOutputHi` functions which *calculated* (incorrectly) the location of `dyn_o` and `dyn_hi` files. * Don't set `outputFile_` and so-on in `enableCodeGenWhen`, `-o` and hence `outputFile_` should not affect the location of object files in `--make` mode. It is now sufficient to just update the ModLocation with the temporary paths. * In `hscGenBackendPipeline` don't recompute the `ModLocation` to account for `-dynamic-too`, the paths are now accurate from the start of the run. * Rename `getLocation` to `mkOneShotModLocation`, as that's the only place it's used. Increase the locality of the definition by moving it close to the use-site. * Load the dynamic interface from ml_dyn_hi_file rather than attempting to reconstruct it in load_dynamic_too. * Add a variety of tests to check how -o -dyno etc interact with each other. Some other clean-ups * DeIOify mkHomeModLocation and friends, they are all pure functions. * Move FinderOpts into GHC.Driver.Config.Finder, next to initFinderOpts. * Be more precise about whether we mean outputFile or outputFile_: there were many places where outputFile was used but the result shouldn't have been affected by `-dyno` (for example the filename of the resulting executable). In these places dynamicNow would never be set but it's still more precise to not allow for this possibility. * Typo fixes suffices -> suffixes in the appropiate places.
* ghci: Explicitly store and restore interface file cacheMatthew Pickering2021-10-171-0/+4
| | | | | | | | | | | | | | | | | | | | | In the old days the old HPT was used as an interface file cache when using ghci. The HPT is a `ModuleEnv HomeModInfo` and so if you were using hs-boot files then the interface file from compiling the .hs file would be present in the cache but not the hi-boot file. This used to be ok, because the .hi file used to just be a better version of the .hi-boot file, with more information so it was fine to reuse it. Now the source hash of a module is kept track of in the interface file and the source hash for the .hs and .hs-boot file are correspondingly different so it's no longer safe to reuse an interface file. I took the decision to move the cache management of interface files to GHCi itself, and provide an API where `load` can be provided with a list of interface files which can be used as a cache. An alternative would be to manage this cache somewhere in the HscEnv but it seemed that an API user should be responsible for populating and suppling the cache rather than having it managed implicitly. Fixes #20217
* More strictness around HomePackageTableMatthew Pickering2021-10-121-1/+3
| | | | | | | | | This patch makes some operations to do with HomePackageTable stricter * Adding a new entry into the HPT would not allow the old HomeModInfo to be collected because the function used by insertWith wouldn't be forced. * We're careful to force the new MVar value before it's inserted into the global MVar as otherwise we retain references to old entries.
* Make ms_ghc_prim_import field strictMatthew Pickering2021-10-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | If you don't promptly force this field then it ends up retaining a lot of data structures related to parsing. For example, the following retaining chain can be observed when using GHCi. ``` PState 0x4289365ca0 0x4289385d68 0x4289385db0 0x7f81b37a7838 0x7f81b3832fd8 0x4289365cc8 0x4289365cd8 0x4289365cf0 0x4289365cd8 0x4289365d08 0x4289385e48 0x7f81b4e4c290 0x7f818f63f440 0x7f818f63f440 0x7f81925ccd18 0x7f81b4e41230 0x7f818f63f440 0x7f81925ccd18 0x7f818f63f4a8 0x7f81b3832fd8 0x7f81b3832fd8 0x4289365d20 0x7f81b38233b8 0 19 <PState:GHC.Parser.Lexer:_build-ipe/stage1/compiler/build/GHC/Parser/Lexer.hs:3779:46> _thunk( ) 0x4289384230 0x4289384160 <([LEpaComment], [LEpaComment]):GHC.Parser.Lexer:> _thunk( ) 0x4289383250 <EpAnnComments:GHC.Parser.Lexer:compiler/GHC/Parser/Lexer.x:2306:19-40> _thunk( ) 0x4289399850 0x7f818f63f440 0x4289399868 <SrcSpanAnnA:GHC.Parser:_build-ipe/stage1/compiler/build/GHC/Parser.hs:12527:13-30> L 0x4289397600 0x42893975a8 <GenLocated:GHC.Parser:_build-ipe/stage1/compiler/build/GHC/Parser.hs:12527:32> 0x4289c4e8c8 : 0x4289c4e8b0 <[]:GHC.Parser.Header:compiler/GHC/Parser/Header.hs:104:36-54> (0x4289c4da70,0x7f818f63f440) <(,):GHC.Parser.Header:compiler/GHC/Parser/Header.hs:104:36-54> _thunk( ) 0x4289c4d030 <Bool:GHC.Parser.Header:compiler/GHC/Parser/Header.hs:(112,22)-(115,27)> ExtendedModSummary 0x422e9c8998 0x7f81b617be78 0x422e9c89b0 0x4289c4c0c0 0x7f81925ccd18 0x7f81925ccd18 0x7f81925ccd18 0x7f81925ccd18 0x7f818f63f440 0x4289c4c0d8 0x4289c4c0f0 0x7f81925ccd18 0x422e9c8a20 0x4289c4c108 0x4289c4c730 0x7f818f63f440 <ExtendedModSummary:GHC.Driver.Make:compiler/GHC/Driver/Make.hs:2041:30-38> ModuleNode 0x4289c4b850 <ModuleGraphNode:GHC.Unit.Module.Graph:compiler/GHC/Unit/Module/Graph.hs:139:14-36> 0x4289c4b590 : 0x4289c4b578 <[]:GHC.Unit.Module.Graph:compiler/GHC/Unit/Module/Graph.hs:139:31-36> ModuleGraph 0x4289c4b2f8 0x4289c4b310 0x4289c4b340 0x7f818f63f4a0 <ModuleGraph:GHC.Driver.Make:compiler/GHC/Driver/Make.hs:(242,19)-(244,40)> HscEnv 0x4289d9a4a8 0x4289d9aad0 0x4289d9aae8 0x4217062a88 0x4217060b38 0x4217060b58 0x4217060b68 0x7f81b38a7ce0 0x4217060b78 0x7f818f63f440 0x7f818f63f440 0x4217062af8 0x4289d9ab10 0x7f81b3907b60 0x4217060c00 114 <HscEnv:GHC.Runtime.Eval:compiler/GHC/Runtime/Eval.hs:790:31-44> ```
* Don't link plugins' units with target code (#20218)Sylvain Henry2021-10-081-1/+12
| | | | | | | | | | | | | | Before this patch, plugin units were linked with the target code even when the unit was passed via `-plugin-package`. This is an issue to support plugins in cross-compilers (plugins are definitely not ABI compatible with target code). We now clearly separate unit dependencies for plugins and unit dependencies for target code and only link the latter ones. We've also added a test to ensure that plugin units passed via `-package` are linked with target code so that `thNameToGhcName` can still be used in plugins that need it (see T20218b).
* Refactor module dependencies codeSylvain Henry2021-09-172-8/+187
| | | | | | * moved deps related code into GHC.Unit.Module.Deps * refactored Deps module to not export Dependencies constructor to help maintaining invariants
* Fix annoying warning about Data.List unqualified importSylvain Henry2021-09-171-1/+1
|
* Export PreloadUnitClosure as it is part of the public APIFendor2021-08-241-0/+1
|
* Driver rework pt3: the upsweepMatthew Pickering2021-08-184-9/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch specifies and simplifies the module cycle compilation in upsweep. How things work are described in the Note [Upsweep] Note [Upsweep] ~~~~~~~~~~~~~~ Upsweep takes a 'ModuleGraph' as input, computes a build plan and then executes the plan in order to compile the project. The first step is computing the build plan from a 'ModuleGraph'. The output of this step is a `[BuildPlan]`, which is a topologically sorted plan for how to build all the modules. ``` data BuildPlan = SingleModule ModuleGraphNode -- A simple, single module all alone but *might* have an hs-boot file which isn't part of a cycle | ResolvedCycle [ModuleGraphNode] -- A resolved cycle, linearised by hs-boot files | UnresolvedCycle [ModuleGraphNode] -- An actual cycle, which wasn't resolved by hs-boot files ``` The plan is computed in two steps: Step 1: Topologically sort the module graph without hs-boot files. This returns a [SCC ModuleGraphNode] which contains cycles. Step 2: For each cycle, topologically sort the modules in the cycle *with* the relevant hs-boot files. This should result in an acyclic build plan if the hs-boot files are sufficient to resolve the cycle. The `[BuildPlan]` is then interpreted by the `interpretBuildPlan` function. * `SingleModule nodes` are compiled normally by either the upsweep_inst or upsweep_mod functions. * `ResolvedCycles` need to compiled "together" so that the information which ends up in the interface files at the end is accurate (and doesn't contain temporary information from the hs-boot files.) - During the initial compilation, a `KnotVars` is created which stores an IORef TypeEnv for each module of the loop. These IORefs are gradually updated as the loop completes and provide the required laziness to typecheck the module loop. - At the end of typechecking, all the interface files are typechecked again in the retypecheck loop. This time, the knot-tying is done by the normal laziness based tying, so the environment is run without the KnotVars. * UnresolvedCycles are indicative of a proper cycle, unresolved by hs-boot files and are reported as an error to the user. The main trickiness of `interpretBuildPlan` is deciding which version of a dependency is visible from each module. For modules which are not in a cycle, there is just one version of a module, so that is always used. For modules in a cycle, there are two versions of 'HomeModInfo'. 1. Internal to loop: The version created whilst compiling the loop by upsweep_mod. 2. External to loop: The knot-tied version created by typecheckLoop. Whilst compiling a module inside the loop, we need to use the (1). For a module which is outside of the loop which depends on something from in the loop, the (2) version is used. As the plan is interpreted, which version of a HomeModInfo is visible is updated by updating a map held in a state monad. So after a loop has finished being compiled, the visible module is the one created by typecheckLoop and the internal version is not used again. This plan also ensures the most important invariant to do with module loops: > If you depend on anything within a module loop, before you can use the dependency, the whole loop has to finish compiling. The end result of `interpretBuildPlan` is a `[MakeAction]`, which are pairs of `IO a` actions and a `MVar (Maybe a)`, somewhere to put the result of running the action. This list is topologically sorted, so can be run in order to compute the whole graph. As well as this `interpretBuildPlan` also outputs an `IO [Maybe (Maybe HomeModInfo)]` which can be queried at the end to get the result of all modules at the end, with their proper visibility. For example, if any module in a loop fails then all modules in that loop will report as failed because the visible node at the end will be the result of retypechecking those modules together. Along the way we also fix a number of other bugs in the driver: * Unify upsweep and parUpsweep. * Fix #19937 (static points, ghci and -j) * Adds lots of module loop tests due to Divam. Also related to #20030 Co-authored-by: Divam Narula <dfordivam@gmail.com> ------------------------- Metric Decrease: T10370 -------------------------
* Refactoring module dependenciesSylvain Henry2021-08-133-6/+22
| | | | | | | | | | | | | | * Make mkDependencies pure * Use Sets instead of sorted lists Notable perf changes: MultiLayerModules(normal) ghc/alloc 4130851520.0 2981473072.0 -27.8% T13719(normal) ghc/alloc 4313296052.0 4151647512.0 -3.7% Metric Decrease: MultiLayerModules T13719
* Improve documentation for HscTypes.usg_mod_hashAndreas Klebinger2021-08-031-2/+5
|
* rts: Introduce and use ExecPage abstractionBen Gamari2021-07-271-1/+1
| | | | | Here we introduce a very thin abstraction for allocating, filling, and freezing executable pages to replace allocateExec.
* Introduce FinderLocations for decoupling Finder from DynFlagsFendor2021-07-231-80/+110
|
* Use Ways API instead of Set specific functionsFendor2021-07-211-5/+4
|
* driver: Fix recompilation for modules importing GHC.PrimMatthew Pickering2021-07-211-0/+3
| | | | | | | | | | | | | The GHC.Prim module is quite special as there is no interface file, therefore it doesn't appear in ms_textual_imports, but the ghc-prim package does appear in the direct package dependencies. This confused the recompilation checking which couldn't find any modules from ghc-prim and concluded that the package was no longer a dependency. The fix is to keep track of whether GHC.Prim is imported separately in the relevant places. Fixes #20084
* driver: Fix interaction of -Wunused-packages and reexported-modulesMatthew Pickering2021-07-132-4/+26
| | | | | | | | | | | | | | Spurious warnings were previously emitted if an import came from a reexport due to how -Wunused-packages were implemented. Removing the dependency would cause compilation to fail. The fix is to reimplement the warning a bit more directly, by searching for which package each import comes from using the normal module finding functions rather than consulting the EPS. This has the advantage that the check could be performed at any time after downsweep rather than also relying on a populated EPS. Fixes #19518 and #19777
* Remove useless .hs-bootSylvain Henry2021-07-012-13/+0
|
View Lorry Controller Status