NOMAD_PORT_<label> has a regression in 0.9

[notnoop]

Nomad version

Tested with the following versions

Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)
Nomad v0.9.4 (a81aa846a45fb8248551b12616287cb57c418cd6)

Operating system and Environment details

macOS, but applies to Linux

Issue

NOMAD_PORT_<label> for a docker task in 0.9.0 series is set to the host static or dynamic port allocation. Previously, it was set to the port the service should bind to inside container.

Implementation Details

The fix requires some thought. port_map is a driver specific config currently. Task Runner and TaskEnv are not equipped to set the values properly.

At entry of taskDriver.StartTask(...) invocation yet, port_map isn't properly interpreted yet, and TaskEnv has nil driverNet in https://github.com/hashicorp/nomad/blob/v0.9.4/client/taskenv/env.go#L724-L730 . This means that TaskEnv sets NOMAD_PORT_<label> would set to the assigned host port.

Interestingly, StartTask(...) returns the port mapping and nomad attempts to set task env variables in the invocation in https://github.com/hashicorp/nomad/blob/v0.9.4/client/allocrunner/taskrunner/task_runner.go#L772 . Sadly this is too late as task already started and this env-var isn't effective.

We may need drivers to set these port environment variables (with some provider helper functions).

Reproduction steps

1. Spin up a nomad dev agent, nomad agent -dev
2. Run job: nomad job run ./example.hcl
3. inspect output: nomad alloc logs --job example

On 0.9.0, the NOMAD_PORT_db matches NOMAD_HOST_PORT_db:

$ nomad job run ./example.nomad
==> Monitoring evaluation "af179ced"
    Evaluation triggered by job "example"
    Allocation "c61fddeb" created: node "b7eb7601", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "af179ced" finished with status "complete"
$ nomad alloc logs --job example
NOMAD_HOST_PORT_db=26340
NOMAD_PORT_db=26340

Nomad 0.8.7 provides the expected behavior:

NOMAD_HOST_PORT_db=30291
NOMAD_PORT_db=6379

Job file (if appropriate)

job "example" {
  type = "batch"
  datacenters = ["dc1"]

  group "cache" {
    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"
        command = "/bin/sh"
        args = ["-c", "printenv | grep -e NOMAD_ | grep PORT"]
        port_map {
          db = 6379
        }
      }

      resources {
        cpu    = 500
        memory = 256
        network {
          mbits = 10
          port "db" {}
        }
      }
    }
  }
}

[schmichael]

Bug

This is a regression in 0.9.0 due to the driver interface no longer having a Prestart method.

Nomad v0.8 called Driver.Prestart which returned a PortMap for injecting in the environment before starting the task.

Nomad v0.9 dropped the Prestart method for task drivers and inadvertently stopped properly setting this env vars in the process.

Fix

I think we can set these env vars directly in the Docker driver.

The only code that needs access to the PortMap occurs within Driver.StartTask and after (eg service registration in a postrun hook), so we should have have drivers populate their environment from their port map inside StartTask.

Update: To clarify any driver with a port_map is affected. Updated labels.

[jazzyfresh]

It looks like only the docker, qemu and rkt drivers have a port_map so limiting the scope of this ticket to fix those drivers

[notnoop]

Fixed by #6251

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.