'Development' returned as environment name in Azure after disabling any function

[Ads5555]

Investigative information

• Timestamp: 2020-06-18T03:00:26Z
• Function App version: v3
• Function App name: FunctionApp220200618084855
• Function name(s) (as appropriate): EnvironmentTestFunction
• Invocation ID:
• Region: Australia Southeast

Repro steps

1. Create a new default Azure Function solution.
2. Add a function that simply prints out the hosting environment name via the log property.
   • _hostingEnvironment.EnvironmentName
3. Deploy to a new Azure instance and view the output of running the function via 'Code + Test' interface.
4. Note that it states the environment is 'Development'.
   • Docs state the default should be Prod in Azure. This appears to be another issue?
5. Set AZURE_FUNCTIONS_ENVIRONMENT to a value in the configuration (e.g. 'Staging').
6. Re-run the function and note its returned correctly as 'Staging'.
7. Disable the function and then re-enable.
   8 Re-run the function and note that it now says 'Development'. Restarting the functions doesn't resolve the issue. AZURE_FUNCTIONS_ENVIRONMENT is disregarded.

Expected behavior

• Disabling and enabling functions should not change if the AZURE_FUNCTIONS_ENVIRONMENT is used or not.

Actual behavior

• AZURE_FUNCTIONS_ENVIRONMENT is ignored after disabling a function and is useless for determining the environment correctly.
• I havent found a way to fix this after it becomes broken.

Known workarounds

Use your own environment key to pull from the configuration.

Related information

Source

public class EnvironmentTestFunction
{
    private readonly IHostingEnvironment _hostingEnvironment;

    public EnvironmentTestFunction(IHostingEnvironment hostingEnvironment)
    {
        _hostingEnvironment = hostingEnvironment;
    }

    [FunctionName(nameof(EnvironmentTestFunction))]
    public void EnvironmentTest([TimerTrigger("0 */1 * * * *", RunOnStartup = true)]TimerInfo myTimer, ILogger log)
    {
        try
        {
            log.LogWarning("Finding Env");
            log.LogWarning($"_hostingEnvironment.EnvironmentName: {_hostingEnvironment.EnvironmentName}");
            log.LogWarning($"_hostingEnvironment.IsProduction: {_hostingEnvironment.IsProduction()}");
        }
        catch (Exception e)
        {
            log.LogError(e, $"Exception caught {e.Message}");
        }
    }
}

[devdorn]

I've the same Problem: since 2020-06-18 the default Environment on Azure Functions (West Europe, Runtime Version 3.0.13901.0) has changed from "Production" to "Development".
I've never disabled my Function, but I had an Deployment on 2020-06-18.

[IGx89]

I'm seeing some very weird behavior with environment name as well. My function didn't have AZURE_FUNCTIONS_ENVIRONMENT set and was returning "Production" in Azure, but when I did set AZURE_FUNCTIONS_ENVIRONMENT it changed to "Development" and now no matter what I set AZURE_FUNCTIONS_ENVIRONMENT to (or even remove it) it always is "Development" now. Something definitely seems buggy in that area.

[wagich]

I seem to have the same problem. No matter what value I set for AZURE_FUNCTIONS_ENVIRONMENT, IHostEnvironment.EnvironmentName always returns "Development".

[smokedlinq]

This happened to me today, nothing I could do would change IHostEnvironment.EnvironmentName to what I set AZURE_FUNCTIONS_ENVIRONMENT value as in the configuration. I opted for a code-fix that checks the environment variable first, and if it's empty defaults to the IHostEnvironment value. Locally, when debugging with v3.0.2630 core tools, this does not happen.

[madelinegordon]

Hello, we will investigate this issue and provide information based on our findings. Thank you.

[rigdern]

Is this related to #6085?

[Ads5555]

Is this related to #6085?

My functions were respecting the prod environment config until I disabled them. It could be related but I think the investigation would need to determine that.

[wagich]

In my case it happend from the beginning on a completely new function app I deployed to via docker. It might well be two separate or related problems and me hitting #6085 too…

[madelinegordon]

Hello, we were able to repro this issue, and we believe the cause is related to the scoping of components within the Functions Host - the WebHost (outer scope) component is only fully restarted when a Function app is stopped and then started up again, whereas the ScriptHost (inner scope) is the only component that is restarted when a Function app is restarted from the portal - essentially we only do a partial restart in this case. It seems that whenever only a partial restart occurs, 'IHostEnvironment.EnvironmentName` becomes 'Development'. This can be mitigated by stopping your Function app, waiting a few seconds, and then starting it back up again.

[ericsampson]

@madelinegordon @ankitkumarr is this scheduled as a priority fix? This seems like a bad bug - as mentioned in #6085 , it's happens not only in the relatively rare case of disabling/enabling a function, but also after deploys or restarts.
If this was the same type of bug in plain App Service where customer's code was running with the wrong environment settings, there would be a big freakout and fast fix - this can cause bad things like customers calling the dev APIs instead of prod APIs and subsequent data/PII issues...

[ankitkumarr]

@fabiocav and @anthonychu -- Thoughts on the investigation and priority for this?

[pragnagopa]

Assigning to Sprint 87 as a stretch goal

[fabiocav]

This might slip into the next sprint. Updating to reflect that.

[Simon-Gregory-LG]

I've commented before on this one and even come up with a workaround, but this ticket seems to keep getting pushed back, so feel the need to re-echo the need for attention here.

The workaround is ok if you know you have the problem, but there will be plenty of people who aren't even aware this is happening. While I take small comfort that I'm not in that group, I shudder to think what could be happening elsewhere with this issue.

I struggle to think of a more critical problem than a system intermittently and indiscriminately loading its Development configuration while in Production - yet we are still not even past the needs-investigation phase here.

[krosk93]

Hello, any updates? We are experiencing right now the same behaviour. It suddenly changed from Production to Development environment. Thanks.

[fabiocav]

@krosk93 there hasn't been a lot of progress on this aside from additional investigation and identification of a couple of additional scenarios.

One thing that is worth noting, as that is what was causing the behavior for some customers, is that using any of the admin APIs (e.g. opening the portal or working with the hosted version directly or with one of the tools) will place the host in debut mode for a period of time. This enables some debug services used by the portal, like debug logging. Is it possible this is what is impacting you?

[ericsampson]

@fabiocav could you expand what that means?

I don't believe that is the only scenario we've experienced, as the issue was initially discovered after an application deploy when we noticed from logging to our ES cluster that it was calling APIs in the wrong environment.

[DavidZidar]

I pointed out the offending code in my bug report #6085 nearly a year go and the code is still there.

EnableDebugModeFilter is used by HostController when interacting with features in the portal:

azure-functions-host/src/WebJobs.Script.WebHost/Controllers/HostController.cs

Lines 64 to 65 in 9c2ce4c

	[TypeFilter(typeof(EnableDebugModeFilter))]
	public async Task<IActionResult> GetHostStatus([FromServices] IScriptHostManager scriptHostManager, [FromServices] IHostIdProvider hostIdProvider, [FromServices] IServiceProvider serviceProvider = null)

EnableDebugModeFilter flags that debug mode should be used:

azure-functions-host/src/WebJobs.Script.WebHost/Filters/EnableDebugModeFilter.cs

Lines 26 to 29 in 9c2ce4c

	public void OnActionExecuted(ActionExecutedContext context)
	{
	_debugManager?.NotifyDebug();
	}

The host then restarts and the builder sets the environment to Development:

azure-functions-host/src/WebJobs.Script.WebHost/WebScriptHostBuilderExtension.cs

Lines 140 to 144 in 9c2ce4c

	var debugStateProvider = rootServiceProvider.GetService<IDebugStateProvider>();
	if (debugStateProvider.InDebugMode)
	{
	builder.UseEnvironment(EnvironmentName.Development);
	}

Please find another way to enable debug services, DO NOT change the environment like this, it breaks all sorts of things.

[fabiocav]

@ericsampson this is not the only root cause, but wanted to make sure it was noted here as it is what impacts a lot of the apps running into this.

Right, @DavidZidar that is the logic that will be addressed.

[tiwaripradeep123]

I see intermittent issue too and today all the functions are failing because they are using wrong settings file in production.

[rmaziarka]

I experienced the same problem today. It's great that there is a @Simon-Gregory-LG workaround but such behavior of AF makes us spend hours / days on solving the wrong types of problems.

[bedwards-humana]

I'm having the same issue. Will work around for now by retrieving the AZURE_FUNCTIONS_ENVIRONMENT variable directly instead of relying on builder.GetContext().EnvironmentName.

[ericsampson]

Hi @fabiocav, I saw your PR #7304 , glad to see some movement. However, I believe that this is one of those rare occasions where a breaking change should be made, and then people can opt into the current behavior if that's what they actually want.

Also, is there going to be continued investigation as to why this issue occurs unrelated to debug mode? I don't want the issue to stop being investigated just because one way that it gets triggered was addressed.

Thanks!

[fabiocav]

@ericsampson there will be further investigation here, but we didn't want to block on that in order to address a known sceanario.

I do understand where you're coming from with the breaking change, and we did discuss this, but with the broad number of scenarios we see on Functions, we did have concrete examples and evidence of customers who would be negatively impacted by the change if that was made the default, but the new behavior will become the default with the next major version being released later this year.

[ericsampson]

Thanks @fabiocav, I figured it wasn't the end of the road but wanted to verify.

Re the second part, I hear what your saying... But most of the people on this issue would probably see the original behavior as a bug, or an misdesigned implementation at the least. Could you help me understand the scenario in which the current behavior is desirable?

[ericsampson]

I'm not trying to be a pain in the arse, I really appreciate the progress here.

Just trying to make sure the widest number of people are prevented from having to go through the pain that the people in this ticket have. Never mind everyone affected but they didn't/haven't even realize it.

[fabiocav]

@ericsampson no worries, not being a pain at all, that is our desire as well. But we do have evidence and concrete examples of applications that were relying on the "buggy behavior" and relying on the environment name change when portal/API interactions were taking place. Our tolerance for breaking changes is VERY low, so even in cases like this, we have to be safe with the approach. That, combined with the fact that our next major is just months away, impacted the decision.

[ericsampson]

I understand, thanks Fabio. Cheers

[ericsampson]

One quick question Fabio; when you exit the "Code and Test" feature in the blade, is the host being reset to the original environment?

[fabiocav]

One quick question Fabio; when you exit the "Code and Test" feature in the blade, is the host being reset to the original environment?

Not immediately. The state is eventually reset once debug times out (need to double check, but I believe that happens after 15 mins)

[psddp]

Looks like it was a design decision to always have "Development" when running local, see https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#azure_functions_environment

[unruledboy]

This is very annoying. Finally we had to directly read the environment value to determine what appsettings.json will be loaded:

        public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
        {
            var hostBuilderContext = builder.GetContext();
            var isDevelopmentEnvironment = Environment.GetEnvironmentVariable("AZURE_FUNCTIONS_ENVIRONMENT").ToLower().Contains("dev");
           //your usage of isDevelopmentEnvironment here
        }  

[UsmanMohammad]

Looks like it was a design decision to always have "Development" when running local, see https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#azure_functions_environment

It's quite short-sighted for them to assume only Development, Staging and Production are the only environment names used by people.

What I've done is to just ignore "AZURE_FUNCTIONS_ENVIRONMENT" and built my configuration this way

    private IConfiguration _configuration;
    private string _environmentName;
    public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
    {
        _environmentName = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
        
        var applicationRootPath = builder.GetContext().ApplicationRootPath;
        builder.ConfigurationBuilder
            .AddJsonFile(Path.Combine(applicationRootPath, "appsettings.json"), optional: true, reloadOnChange: false)
            .AddJsonFile(Path.Combine(applicationRootPath, $"appsettings.{ _environmentName}.json"), optional: true, reloadOnChange: false)
            .AddEnvironmentVariables();

        if (!string.Equals(_environmentName, "Development", StringComparison.OrdinalIgnoreCase))
        {
            var builtConfig = builder.ConfigurationBuilder.Build();
            var keyVaultName = builtConfig["KeyVaultName"];
            if (!string.IsNullOrWhiteSpace(keyVaultName))
            {
                var secretClient = new SecretClient(
                    new Uri($"https://{keyVaultName}.vault.azure.net/"),
                    new DefaultAzureCredential());
                builder.ConfigurationBuilder.AddAzureKeyVault(secretClient, new KeyVaultSecretManager());
            }
        }

        _configuration = builder.ConfigurationBuilder.Build();
    }

[markgibbons25]

Thanks @UsmanMohammad , it's amazing that this is still an issue years later and not documented on the DI page.

[foconnor-DS]

This just totally ruined my morning. We had an odd issue on a production AZ Function, so i disabled a trigger so that I could attempt to process it locally and debug what was going on, yet still allowing the messages to queue up. Then all hell broke loose. All of a sudden, everything to do with those hosted APIs on the AZ function started redirecting to development assets.

Not of fan of how this works.

[StefanosBekiaris]

Any updates on this? Currently setting the AZURE_FUNCTIONS_ENVIRONMENT always returns Development

[fabiocav]

Noticed this was still open without clear information from the PR linked, so I wanted to share the details:

• This issue has been addressed in 4.0, and the default behavior there matches the expectation. No action required
• If running in 3.0, the default behavior has not been changed, you you can opt into the behavior that will address the issue by defining an App Setting named AzureWebJobsFeatureFlags with a value of DisableDevModeInDebug

Hope this information helps. Closing as resolved, but let us know if there are any additional questions. Thanks!