Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Build GHC from source (#4567) #4655

Merged
merged 3 commits into from
Apr 10, 2019
Merged

Build GHC from source (#4567) #4655

merged 3 commits into from
Apr 10, 2019

Conversation

hsyl20
Copy link
Contributor

@hsyl20 hsyl20 commented Mar 25, 2019

This patch adds the experimental support for building GHC from source. We can now specify a specific GHC to build from source in stack.yaml with the following syntax:

compiler-build:
   git: GHC_GIT_REPO
   commit: COMMIT
   flavour: FLAVOUR

In the setup phase Stack uses Hadrian to build a GHC binary distribution
with the specified flavour and installs it into
STACK_ROOT/programs/ARCH/ghc-git-COMMIT-FLAVOUR.

The built compiler can be used with the following configuration:

compiler: ghc-git-COMMIT-FLAVOUR

As GHC may rely on unreleased version of the global packages (e.g.
Cabal), you may have to add some extra-deps as follows:

extra-deps:
- git: GHC_GIT_REPO
  commit: COMMIT
  subdirs:
    - libraries/Cabal/Cabal
    - libraries/...
  • Any changes that could be relevant to users have been recorded in the ChangeLog.md
  • The documentation has been updated, if necessary.
  • A test has been added but it takes a long time to run as it builds GHC. [Also note that currently GHC head builds non-installable bindists, cf GHC#16498]

@hsyl20 hsyl20 mentioned this pull request Mar 25, 2019
Closed
@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch 3 times, most recently from a7171d4 to 3353093 Compare March 26, 2019 08:32
@snoyberg
Copy link
Contributor

Thanks! A few initial comments

  • I see that there's still a CI failure, can you address that?
  • How do you specify which version of GHC is used for bootstrapping?
  • Can you clarify a bit more about the global packages? Stack tries to assume that global packages are not the same as something on Hackage

@snoyberg snoyberg mentioned this pull request Mar 28, 2019
Closed
@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch 3 times, most recently from 295e270 to a79c607 Compare March 29, 2019 10:37
@hsyl20
Copy link
Contributor Author

hsyl20 commented Mar 29, 2019 

* I see that there's still a CI failure, can you address that?

I'm trying to fix it now. However I'm not sure I've done the correct thing in src/Stack/Types/BuildPlan.hs. I've blindly updated the version string but I'm not sure about it.

* How do you specify which version of GHC is used for bootstrapping?

I have updated the documentation. But to answer here: the bootstrapping compiler is configured via hadrian/stack.yaml in GHC source tree. Hence the commit id also "hashes" the bootstrap compiler used, which is very good for reproducible builds.

* Can you clarify a bit more about the global packages? Stack tries to assume that global packages are not the same as something on Hackage

I have tried to improve the documentation on this topic but I don't really understand " Stack tries to assume that global packages are not the same as something on Hackage". The issue we had in #4567 (cf #4567 (comment)) is that Stack was trying to build an older Cabal against a newer GHC while the only working Cabal was still unreleased.

@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch from a79c607 to 5263e0d Compare March 29, 2019 10:58
@snoyberg
Copy link
Contributor

I'm trying to fix it now. However I'm not sure I've done the correct thing in src/Stack/Types/BuildPlan.hs. I've blindly updated the version string but I'm not sure about it.

Thanks! I'll review the code in more detail soon.

I have updated the documentation. But to answer here: the bootstrapping compiler is configured via hadrian/stack.yaml in GHC source tree. Hence the commit id also "hashes" the bootstrap compiler used, which is very good for reproducible builds.

That's even better than I was hoping for!

I have tried to improve the documentation on this topic but I don't really understand " Stack tries to assume that global packages are not the same as something on Hackage". The issue we had in #4567 (cf #4567 (comment)) is that Stack was trying to build an older Cabal against a newer GHC while the only working Cabal was still unreleased.

I think the issue here is that the Stackage snapshots all specify an override of the Cabal library, which forces Stack to ignore the global Cabal version for custom Setup.hs files. We may be moving away for that for other reasons. There are more details at commercialhaskell/stackage#4425.

In #4671 I've added support for a drop-packages configuration, which will drop the Cabal library override and hopefully make this use case work more easily.

snoyberg
snoyberg reviewed Mar 31, 2019
View reviewed changes
Copy link
Contributor

@snoyberg snoyberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good.

@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch 3 times, most recently from c42870d to e93efc1 Compare April 9, 2019 14:01
@hsyl20
Copy link
Contributor Author

hsyl20 commented Apr 9, 2019

Thanks @snoyberg for the review! I've updated the PR accordingly.

@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch from e93efc1 to b51cf31 Compare April 9, 2019 14:39
snoyberg
snoyberg reviewed Apr 10, 2019
View reviewed changes
src/Stack/Setup.hs

-- RIO.Process doesn't wrap process' "shell".
-- Instead we use "proc" with the "sh" command
proc "sh" hadrianArgs runProcess_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will work on Windows, since we're trying to run a batch file, not a shell script. What about instead:

proc hadrianCmd hadrianArgs runProcess_

And removing the hadrianCmd from the args above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC it wasn't working this way on Linux: we would need shell but RIO only has proc AFAIK.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the shell script in the repo set to be executable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is. It fails with "Did not find executable at specified path: ./hadrian/build.stack.sh"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see 2768bb9. This is a really confusing aspect of the process package, I only figured this out since I've been bitten by it before.

snoyberg added a commit that referenced this pull request Apr 10, 2019
@snoyberg
Add superslow integration test capability
4b9f1b1 
See #4655 (comment)
hsyl20 added 3 commits April 10, 2019 11:37
@hsyl20
Support building GHC from source
f518855 
We can now specify a specific GHC to build from source in `stack.yaml`
with the following syntax:

```
compiler: ghc-git-COMMIT-FLAVOUR
```

In the setup phase Stack downloads the source from the specified Git
repository and commit ID, then it uses Hadrian to build a GHC binary
distribution with the specified flavour and it installs it into
STACK_ROOT/programs/ARCH/ghc-git-COMMIT-FLAVOUR.
@hsyl20
Test: use superslow and newer commit ID
0aaeaa8
@hsyl20
Remove useless code
36d6e1b
@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch from b51cf31 to 36d6e1b Compare April 10, 2019 10:02
@snoyberg
Copy link
Contributor

OK, the only thing remaining is the execution of the script on POSIX vs Windows. Everything else looks great. I'm going to futz with it a bit on my OS X and Windows machines and then merge to master. Thanks for enduring a thorough review process @hsyl20 and for the new feature!

@snoyberg snoyberg merged commit 36d6e1b into commercialhaskell:master Apr 10, 2019
@hsyl20
Copy link
Contributor Author

hsyl20 commented Apr 10, 2019

Thanks Michael for the review! My patch fixing GHC bindists has also been merged into GHC so the test passes (on Linux)!

# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@snoyberg snoyberg snoyberg left review comments

@qrilka qrilka qrilka left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hsyl20 @snoyberg @qrilka