CI fail on two tests

[adrinjalali]

The CI keeps failing on this test, I'm not sure why.

Example run: https://github.com/huggingface/huggingface_hub/runs/5628640361?check_suite_focus=true

Ping @LysandreJik @osanseviero

_______________________ HfApiPublicTest.test_model_info ________________________

self = <tests.test_hf_api.HfApiPublicTest testMethod=test_model_info>

    @with_production_testing
Downloading: 100%|██████████| 2.00/2.00 [00:00<00:00, 1.42kB/s]
    def test_model_info(self):
        _api = HfApi()
        model = _api.model_info(repo_id=DUMMY_MODEL_ID)
        self.assertIsInstance(model, ModelInfo)
        self.assertNotEqual(model.sha, DUMMY_MODEL_ID_REVISION_ONE_SPECIFIC_COMMIT)
        # One particular commit (not the top of `main`)
        model = _api.model_info(
            repo_id=DUMMY_MODEL_ID, revision=DUMMY_MODEL_ID_REVISION_ONE_SPECIFIC_COMMIT
        )
        self.assertIsInstance(model, ModelInfo)
        self.assertEqual(model.sha, DUMMY_MODEL_ID_REVISION_ONE_SPECIFIC_COMMIT)
        model = _api.model_info(
            repo_id=DUMMY_MODEL_ID,
            revision=DUMMY_MODEL_ID_REVISION_ONE_SPECIFIC_COMMIT,
            securityStatus=True,
        )
        self.assertEqual(
>           getattr(model, "securityStatus"),
            {"containsInfected": False, "infectionTypes": []},
        )
E       AttributeError: 'ModelInfo' object has no attribute 'securityStatus'

tests/test_hf_api.py:725: AttributeError
__________________ InferenceApiTest.test_inference_with_audio __________________

self = <tests.test_inference_api.InferenceApiTest testMethod=test_inference_with_audio>

    @with_production_testing
    def test_inference_with_audio(self):
        api = InferenceApi("facebook/wav2vec2-large-960h-lv60-self")
        dataset = datasets.load_dataset(
            "patrickvonplaten/librispeech_asr_dummy", "clean", split="validation"
        )
        data = self.read(dataset["file"][0])
        result = api(data=data)
        self.assertIsInstance(result, dict)
>       self.assertTrue("text" in result)
E       AssertionError: False is not true

tests/test_inference_api.py:73: AssertionError

[osanseviero]

@McPatate @Pierrci was the security status added to the staging endpoint? It seems as if setting securityStatusin the param does not do any change

[McPatate]

I would say no, but I'll let @Pierrci confirm.

[osanseviero]

For the second test, it seems the API is returning malformed soundilfe

api = InferenceApi("facebook/wav2vec2-large-960h-lv60-self")
dataset = datasets.load_dataset(
    "patrickvonplaten/librispeech_asr_dummy", "clean", split="validation"
)`

def read(filename: str) -> bytes:
  with open(filename, "rb") as f:
      bpayload = f.read()
  return 

data = read(dataset["file"][0])
result = api(data=data)
result
>>> {'error': 'Malformed soundfile'}

[osanseviero]

@Narsil, on the error above, I think the code snippet above was working some time ago. The dataset is unchanged, so I'm a bit surprised about this. Was there any change in the API?

[LysandreJik]

For the second test, it seems to be flaky; re-running it a couple of times locally makes it succeed. Couldn't reproduce on the CI, where it always fails.

[osanseviero]

In my case running the code of the second test in a clean colab always led to the malformed soundfile unfortunately. I'm surprised it passed locally for you.

[Narsil]

@osanseviero @adrinjalali ,

facebook/wav2vec2-large-960h-lv60-self is not a pinned model anymore, so you're just hitting the load message. Not sure if it was ever pinned by me.

Using facebook/wav2vec2-base-960h which is pinned (most testing purposes of the API too) will hopefully fix that.

I also suggest using a different test call

self.assertTrue("text" in result, f"We received {result} instead")

Would help understand what's going on earlier.

Another possibility would be to test the full output:

self.assertEqual(dict, {"text":"MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL"})

Should by default have a better error message and we would be also testing that output does not diverge, and no extra keys are there.

Note @osanseviero your malformed soundfile is because your are returning None in your read function :)

[osanseviero]

Thanks @Narsil! The copy paste from the test went wrong 🤦‍♂️. I will be testing this out in #788

[Pierrci]

@McPatate @Pierrci was the security status added to the staging endpoint? It seems as if setting securityStatus in the param does not do any change

It's using the same security scanner endpoint/instance as in prod (meaning that if you test with a model also in prod you should see something), but the webhook isn't plugged on it (meaning any random model added in staging will not be processed)

[McPatate]

Also I'm not sure the scanner would manage doing its job with staging stuff as it only uses the Hub in production.