Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Improved support for Docker #2234

Merged
merged 51 commits into from
Jun 19, 2024
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
dc34613
Add `pre-build-cmd` to `docker` target property
lhstrh Mar 7, 2024
5700b0f
Minor cleanup
lhstrh Mar 11, 2024
593ef2e
Use `pre-build-script` and `run-script` instead
lhstrh Apr 5, 2024
f7b3b46
Resolve conflict
lhstrh Apr 5, 2024
2d65776
Update reactor-c submodule
lhstrh Apr 5, 2024
bf3ef86
Update reactor-cpp submodule
lhstrh Apr 5, 2024
30bc253
WIP, amend later
lhstrh Apr 20, 2024
bbd9eb6
Handle pre-build script
lhstrh Jun 12, 2024
fd47c0d
Resolved conflicts
lhstrh Jun 12, 2024
9f25bf5
Various cleanups and bugfixes
lhstrh Jun 13, 2024
7e9dc70
Distinguish between builder and runner
lhstrh Jun 13, 2024
c85a2eb
Use correct context
lhstrh Jun 13, 2024
844d82b
Formatting
lhstrh Jun 13, 2024
999b1a5
Make it possible to skip building using docker
lhstrh Jun 14, 2024
c87d82e
Change from build to no-build and change polarity
lhstrh Jun 14, 2024
753352d
Bugfix and added test
lhstrh Jun 14, 2024
73c24f7
Fix test
lhstrh Jun 14, 2024
c8cb503
Handle federation id differently
lhstrh Jun 14, 2024
73d6915
Do not add -i argument when unfederated
lhstrh Jun 14, 2024
3b0cd7e
Fix paths in generated Dockerfile
lhstrh Jun 16, 2024
4c66716
Got rid of some code duplication
lhstrh Jun 16, 2024
8de5de4
Various cleanups
lhstrh Jun 16, 2024
91e8d22
Fix Docker support for TypeScript
lhstrh Jun 16, 2024
070ea6f
Comments
lhstrh Jun 16, 2024
0ca3d1b
First steps towards Docker support for C++
lhstrh Jun 17, 2024
b58903f
Added FIXME
lhstrh Jun 17, 2024
de08638
Apply suggestions from code review
lhstrh Jun 17, 2024
e911799
Ensure that scripts are also copied for unfederated dockerized programs
lhstrh Jun 17, 2024
fdc0854
Let runner-base default to builder-base
lhstrh Jun 17, 2024
151d540
Escape filenames referenced in Dockerfile
lhstrh Jun 18, 2024
00e4151
CppDockerGenerator.java passes smoke test
petervdonovan Jun 18, 2024
7f58039
Remove escaping from ENTRYPOINT command
petervdonovan Jun 18, 2024
279b00e
Make test more thorough
petervdonovan Jun 18, 2024
b04092f
Add back part of the escaping within ENTRYPOINT
petervdonovan Jun 18, 2024
3eba547
Format
petervdonovan Jun 18, 2024
0fbde62
Update build.yml
lhstrh Jun 18, 2024
f3f37a8
Address comments from code review
petervdonovan Jun 18, 2024
e7d510e
Update core/src/main/java/org/lflang/generator/docker/CDockerGenerato…
lhstrh Jun 18, 2024
0ddd121
Update gradle.properties
lhstrh Jun 18, 2024
7c98324
Address failure to load dynamic libraries
petervdonovan Jun 19, 2024
227c29f
Refactor according to code review
petervdonovan Jun 19, 2024
75cfbaf
clean up code for parallel builds
cmnrd Jun 19, 2024
4ba5869
keep all build information in a single place
cmnrd Jun 19, 2024
64d9be7
ros2 support for docker generation
cmnrd Jun 19, 2024
647c8eb
use more Kotlin
cmnrd Jun 19, 2024
c1f4502
run ros2 tests also with docker
cmnrd Jun 19, 2024
e5cfc62
no need to link to execinfo in ros2
cmnrd Jun 19, 2024
c5f50f3
comment and function rename
cmnrd Jun 19, 2024
6f13f01
Merge pull request #2322 from lf-lang/docker-cpp
lhstrh Jun 19, 2024
c834f2d
Address failing Epoch build
petervdonovan Jun 19, 2024
6f421f2
Address failing C++ ROS2 Docker test
petervdonovan Jun 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Various cleanups and bugfixes
  • Loading branch information
lhstrh committed Jun 13, 2024
commit 9f25bf5a8b7297a17b1db7310758dde9dcd58612
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
package org.lflang.generator.docker;

import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.lflang.LocalStrings;
import org.lflang.generator.LFGeneratorContext;
import org.lflang.generator.c.CCompiler;
import org.lflang.target.Target;
@@ -37,7 +36,9 @@ protected String generateDockerFileContent() {
var baseImage = baseImage();
return String.join(
"\n",
"# For instructions, see: https://www.lf-lang.org/docs/handbook/containerized-execution",
"# Generated by the Lingua Franca compiler version " + LocalStrings.VERSION,
"# - Docs: https://www.lf-lang.org/docs/handbook/containerized-execution",
"",
"FROM " + baseImage + " AS builder",
"WORKDIR /lingua-franca/" + lfModuleName,
generateRunForInstallingDeps(),
@@ -46,33 +47,27 @@ protected String generateDockerFileContent() {
"",
"FROM " + baseImage,
"WORKDIR /lingua-franca",
"RUN mkdir scripts",
generateCopyOfScript(),
"RUN mkdir bin",
"COPY --from=builder /lingua-franca/"
+ lfModuleName
+ "/bin/"
+ lfModuleName
+ " ./bin/"
+ lfModuleName,
"",
"# Use ENTRYPOINT not CMD so that command-line arguments go through",
"ENTRYPOINT [\"./bin/" + lfModuleName + "\"]",
generateCopyOfExecutable(),
generateEntryPoint(),
"");
}

@Override
public List<String> defaultEntryPoint() {
return List.of("./bin/" + context.getFileConfig().name);
}

@Override
protected String generateRunForInstallingDeps() {
var config = context.getTargetConfig();
var compiler = config.target == Target.CCPP ? "g++" : "gcc";
if (baseImage().equals(defaultImage())) {
return """
# Install build dependencies
RUN set -ex && apk add --no-cache %s musl-dev cmake make
"""
.formatted(compiler);
return "RUN set -ex && apk add --no-cache %s musl-dev cmake make".formatted(compiler);
} else {
return """
# Skipping installation of build dependencies (custom base image)
""";
return "# Skipping installation of build dependencies (custom base image)";
}
}

@@ -83,18 +78,14 @@ protected List<String> defaultBuildCommands() {
context.getTargetConfig(),
context.getFileConfig(),
context.getErrorReporter(),
context.getTargetConfig().target == Target.C);
context.getTargetConfig().target == Target.CCPP);

return Stream.of(
List.of("set -ex"),
getPreBuildCommand(),
List.of(
"mkdir -p bin",
String.format("%s -DCMAKE_INSTALL_BINDIR=./bin -S src-gen -B bin",
ccompile.compileCmakeCommand()),
"cd bin",
"make all"))
.flatMap(java.util.Collection::stream)
.collect(Collectors.toList());
return List.of(
"mkdir -p bin",
String.format(
"%s -DCMAKE_INSTALL_BINDIR=./bin -S src-gen -B bin", ccompile.compileCmakeCommand()),
"cd bin",
"make all",
"cd ..");
}
}
13 changes: 10 additions & 3 deletions core/src/main/java/org/lflang/generator/docker/DockerData.java
Original file line number Diff line number Diff line change
@@ -3,10 +3,10 @@
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import org.lflang.FileConfig;
import org.lflang.generator.LFGeneratorContext;
import org.lflang.target.property.DockerProperty;
import org.lflang.util.FileUtil;
import org.lflang.FileConfig;

/**
* Build configuration of a docker service.
@@ -48,8 +48,15 @@ public void writeDockerFile() throws IOException {
}

public void copyScripts(LFGeneratorContext context) throws IOException {
copyScript(context.getFileConfig(), context.getTargetConfig().get(DockerProperty.INSTANCE).preBuildScript());
copyScript(context.getFileConfig(), context.getTargetConfig().get(DockerProperty.INSTANCE).runScript());
copyScript(
context.getFileConfig(),
context.getTargetConfig().get(DockerProperty.INSTANCE).preBuildScript());
copyScript(
context.getFileConfig(),
context.getTargetConfig().get(DockerProperty.INSTANCE).postBuildScript());
copyScript(
context.getFileConfig(),
context.getTargetConfig().get(DockerProperty.INSTANCE).preRunScript());
}

private void copyScript(FileConfig fileConfig, String script) throws IOException {
Original file line number Diff line number Diff line change
@@ -2,9 +2,12 @@

import java.nio.file.Path;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.lflang.generator.LFGeneratorContext;
import org.lflang.target.property.BuildCommandsProperty;
import org.lflang.target.property.DockerProperty;
import org.lflang.target.property.DockerProperty.DockerOptions;
import org.lflang.util.StringUtil;

/**
@@ -36,15 +39,16 @@ public DockerGenerator(LFGeneratorContext context) {
/** Return the default compile commands for the C docker container. */
protected abstract List<String> defaultBuildCommands();

/** Return the commands used to build */
protected List<String> getBuildCommands() {
var customBuildCommands = context.getTargetConfig().get(BuildCommandsProperty.INSTANCE);
// FIXME: also use pre-build-script in case of custom build commands.
if (customBuildCommands != null && !customBuildCommands.isEmpty()) {
return customBuildCommands;
}
return defaultBuildCommands();
}

/** Return the command that sources the pre-build script, if there is one. */
protected List<String> getPreBuildCommand() {
var script = context.getTargetConfig().get(DockerProperty.INSTANCE).preBuildScript();
if (!script.isEmpty()) {
@@ -53,11 +57,65 @@ protected List<String> getPreBuildCommand() {
return List.of();
}

/** Return the default compile command for the C docker container. */
/** Return the command that sources the post-build script, if there is one. */
protected List<String> getPostBuildCommand() {
var script = context.getTargetConfig().get(DockerProperty.INSTANCE).postBuildScript();
if (!script.isEmpty()) {
return List.of("source src-gen/" + script);
}
return List.of();
}

/** Return the Docker RUN command used for building. */
protected String generateRunForBuild() {
return "RUN " + StringUtil.joinObjects(getBuildCommands(), " && ");
return "RUN "
+ StringUtil.joinObjects(
Stream.of(
List.of("set -ex"),
getPreBuildCommand(),
getBuildCommands(),
getPostBuildCommand())
.flatMap(java.util.Collection::stream)
.collect(Collectors.toList()),
" \\\n\t&& ");
}

protected String generateEntryPoint() {
return "ENTRYPOINT ["
+ getEntryPointCommands().stream()
.map(cmd -> "\"" + cmd + "\"")
lhstrh marked this conversation as resolved.
Show resolved Hide resolved
.collect(Collectors.joining(","))
+ "]";
}

protected String generateCopyOfExecutable() {
var lfModuleName = context.getFileConfig().name;
return "COPY --from=builder /lingua-franca/%s/bin/%s ./bin/%s"
lhstrh marked this conversation as resolved.
Show resolved Hide resolved
.formatted(lfModuleName, lfModuleName, lfModuleName);
}

protected String generateCopyOfScript() {
var script = context.getTargetConfig().get(DockerProperty.INSTANCE).preRunScript();
if (!script.isEmpty()) {
return "COPY --from=builder /lingua-franca/%s/src-gen/%s ./scripts/%s"
.formatted(context.getFileConfig().name, script, script);
petervdonovan marked this conversation as resolved.
Show resolved Hide resolved
}
return "# (No pre-run script provided.)";
}

protected List<String> getEntryPointCommands() {
var script = context.getTargetConfig().get(DockerProperty.INSTANCE).preRunScript();
if (!script.isEmpty()) {
return Stream.concat(
List.of(DockerOptions.DEFAULT_SHELL, "-c", "source scripts/" + script).stream(),
defaultEntryPoint().stream())
.toList();
}
return defaultEntryPoint();
}

public abstract List<String> defaultEntryPoint();

/** Return the default base image. */
public abstract String defaultImage();

Original file line number Diff line number Diff line change
@@ -18,13 +18,18 @@ public TSDockerGenerator(LFGeneratorContext context) {
/** Return the content of the docker file for [tsFileName]. */
public String generateDockerFileContent() {
return """
FROM %s
WORKDIR /linguafranca/$name
%s
COPY . .
ENTRYPOINT ["node", "dist/%s.js"]
"""
.formatted(baseImage(), generateRunForInstallingDeps(), context.getFileConfig().name);
FROM %s
WORKDIR /linguafranca/$name
%s
COPY . .
%s
"""
.formatted(baseImage(), generateRunForInstallingDeps(), generateEntryPoint());
}

@Override
public List<String> defaultEntryPoint() {
return List.of("node", "dist/%s.js".formatted(context.getFileConfig().name));
}

@Override
26 changes: 19 additions & 7 deletions core/src/main/java/org/lflang/target/property/DockerProperty.java
Original file line number Diff line number Diff line change
@@ -36,7 +36,9 @@ public DockerOptions fromAst(Element node, MessageReporter reporter) {
var enabled = false;
var from = "";
var rti = DockerOptions.DOCKERHUB_RTI_IMAGE;
var shell = DockerOptions.DEFAULT_SHELL;
var preBuildScript = "";
var postBuildScript = "";
var runScript = "";

if (node.getLiteral() != null) {
@@ -51,12 +53,13 @@ public DockerOptions fromAst(Element node, MessageReporter reporter) {
switch (option) {
case FROM -> from = str;
case PRE_BUILD_SCRIPT -> preBuildScript = str;
case PRE_RUN_SCRIPT -> runScript = str;
case POST_BUILD_SCRIPT -> postBuildScript = str;
case RTI_IMAGE -> rti = str;
case RUN_SCRIPT -> runScript = str;
}
}
}
return new DockerOptions(enabled, from, rti, preBuildScript, runScript);
return new DockerOptions(enabled, from, rti, shell, preBuildScript, postBuildScript, runScript);
}

@Override
@@ -87,8 +90,9 @@ public Element toAstElement(DockerOptions value) {
switch (opt) {
case FROM -> pair.setValue(ASTUtils.toElement(value.from));
case PRE_BUILD_SCRIPT -> pair.setValue(ASTUtils.toElement(value.preBuildScript));
case PRE_RUN_SCRIPT -> pair.setValue(ASTUtils.toElement(value.preRunScript));
case POST_BUILD_SCRIPT -> pair.setValue(ASTUtils.toElement(value.postBuildScript));
case RTI_IMAGE -> pair.setValue(ASTUtils.toElement(value.rti));
case RUN_SCRIPT -> pair.setValue(ASTUtils.toElement(value.runScript));
}
kvp.getPairs().add(pair);
}
@@ -106,18 +110,25 @@ public String name() {
}

/** Settings related to Docker options. */

public record DockerOptions(
boolean enabled, String from, String rti, String preBuildScript, String runScript) {
boolean enabled,
String from,
String rti,
String shell,
String preBuildScript,
String postBuildScript,
String preRunScript) {

/** Default location to pull the rti from. */
public static final String DOCKERHUB_RTI_IMAGE = "lflang/rti:rti";

public static final String DEFAULT_SHELL = "/bin/sh";

/** String to indicate a local build of the rti. */
public static final String LOCAL_RTI_IMAGE = "rti:local";

public DockerOptions(boolean enabled) {
this(enabled, "", DOCKERHUB_RTI_IMAGE, "", "");
this(enabled, "", DOCKERHUB_RTI_IMAGE, DEFAULT_SHELL, "", "", "");
}
}

@@ -130,7 +141,8 @@ public enum DockerOption implements DictionaryElement {
FROM("FROM", PrimitiveType.STRING),
RTI_IMAGE("rti-image", PrimitiveType.STRING),
PRE_BUILD_SCRIPT("pre-build-script", PrimitiveType.STRING),
RUN_SCRIPT("run-script", PrimitiveType.STRING);
PRE_RUN_SCRIPT("pre-run-script", PrimitiveType.STRING),
POST_BUILD_SCRIPT("post-build-script", PrimitiveType.STRING);

public final PrimitiveType type;

4 changes: 3 additions & 1 deletion test/C/src/docker/federated/DistributedCountContainerized.lf
Original file line number Diff line number Diff line change
@@ -10,7 +10,9 @@ target C {
coordination: centralized,
docker: {
rti-image: "rti:local",
pre-build-script: "foo.sh"
pre-build-script: "foo.sh",
pre-run-script: "bar.sh",
post-build-script: "baz.sh"
}
}

Loading