What confidence can I have in BirdNET?

[ceperman]

I've build myself a sound recorder that I can deploy remotely to record birds over a period of days or weeks. It alternates between recording and sleeping (both periods configurable), creating files on an SD card which I can then analyse back home using BirdNET-Analyzer which I have locally installed on my PC ie. not the web version. I'm going to be relying on BN to do the identifications, and I'm looking at Chirpity to see how it can enhance the analysis process.

BirdNET-Analyzer appears to give credible results, that is until I compare them with Merlin (the phone app I use when I'm out and about) and sometimes the Mk1 ear. I'm interested to know how others feel about its accuracy and if they have their own comparisons.

To put the following examples in context, I live in Warwickshire, UK

First example

I've made some dawn chorus recordings over the years for a local website, and identified what I could by ear (this was before AI identification was around, or at least before I discovered it). Having found Merlin and BirdNET I analysed some of them for interest.

See (or hear!) https://www.oakleywood.org.uk/2020/05/dawn-chorus-2020/ (the second recording)
This recording was made with the equipment mentioned on the web page.

I played it with Merlin listening, and also ran it through BirdNET-Analyzer. Merlin largely agreed with what I could hear; BN at 0.7 confidence detected just the Green Woodpecker at 33 secs in. I'm not doubting this identification (confidence 0.9320), I was surprised at everything it missed. This is the comparison table I made:

Species	BirdNET	Merlin	Me	Notes
Blackbird	No	Yes	Yes	Just a distant “clucking”, not very prominent
Blackcap	No	Yes	No	Normally prominent in dawn chorus but admittedly I didn’t detect any here
Blue Tit	Yes	Yes	No	Possibly above my hearing range anyway
Carrion Crow	Yes	Yes	Yes
Chiffchaff	Yes	Yes	Yes
Firecrest	Yes	No	No	Unlikely, BirdNET low conf
Green Woodpecker	Yes	Yes	Yes
Lesser Spotted Woodpecker	Yes	No	No	Unlikely, BirdNET low conf
Ring-necked Pheasant	No	Yes	Yes
Robin	No	Yes	Yes	Not very prominent
Short-toed Treecreeper	Yes	No	No	Highly unlikely, BirdNET low conf
Song Thrush	Yes	Yes	Yes
Spotted Flycatcher	Yes	No	No	Possible, BirdNET low conf
Stock Dove	Yes	No	No	Possible, BirdNET low conf
Treecreeper	Yes	No	No	Possible, BirdNET low conf
Turtle Dove	Yes	No	No	Unlikely, BirdNET low conf
Wood Pigeon	Yes	Yes	Yes
Wood Warbler	Yes	No	No	Highly unlikely, BirdNET low conf
Wren	No	Yes	Yes	Not very prominent

			Disagreement between BirdNET and Merlin: 63%

                BirdNET confidence level set at 0.1
			Increasing the confidence level to 0.3 eliminates all but the Green Woodpecker & Song Thrush
			Increasing the confidence level to 0.4 eliminates all but the Green Woodpecker

Admittedly I was using a very low condifence threshold for BN (this was before I knew what sort of level I should typically be using, perhaps nothing less than 0.7).

Second example

This was made recently with my home-grown still-in-development recorder in a rural garden, making a 3 minute recording every 30 minutes. These are the BN results (xn = number of detections in the 3 min period):

Time slot Species

15:10 Blue Tit, Great Tit(x2) (Merlin: same)
15:39 Robin(x2) (Merlin: Blue Tit, Robin)
16:09 Robin (Merlin: same)
20:33 Tawny Owl(x2) (Merlin: nothing, although I could clearly hear it)
03:54 Barn Owl (Merlin: nothing, I couldn't hear it)
07:49 Robin(x21), Redwing, Dunnock, Long-tailed Tit (Merlin: Robin)
08:18 Wren, Robin(x3) (Merlin: Great Tit, Blue Tit, Robin, Chaffinch)
08:47 Robin(x6) (Merlin: Robin, Great Tit, Blue Tit, Great Spotted Woodpecker)
09:17 Dunnock, Robin (Merlin: Robin, Greenfinch, I only heard a Pheasant!)
09:46 Robin(x16), Pheasant (Merlin: Robin, Greenfinch, Blackbird)
10:16 Robin(x3)
10:45 Robin(x3), Great Tit(x8), Long-tailed Tit
11:15 Robin(x2)
11:44 Robin(x30), Blue Tit, Dunnock, Water Rail(!!*) (Merlin: Robin, Blue Tit, Dunnock)

Over-high gain (perhaps) in the recorder created crackle and disortion of close/loud sounds, which may have accounted for the unexpected Water Rail. Apart from this, the BN results are all quite credible.

However, when compared with Merlin identifications, it all looks a bit uncertain. Which do you believe? Both these products come from the same stable (Cornell Labs) but my understanding is that they use different AI implementations. I've used Merlin for some while now and come to have confidence in its identifications. It alerted me to the presence of Spotted Flycatchers in our local wood. Initially I didn't believe it, but it was persistent in a particular place where I eventually spotted (sic) them.

So, where does that leave me? I know that these products are not infallible, but the disagreement between them is disappointing. My feeling is that Merlin may be better at dealing with overlapping sounds, as in the dawn chorus; BN happier with descrete sounds. I obviously want to have confidence in BN because that's what I will use for my recordings.

One more thing. I think Chirpity uses the BirdNET web service. I use the desktop version and detects the Barn Owl (above) with 77% confidence level. I ran the recording through Chirpity and it detected nothing at 70% confidence level, but I dropped it to 40% and then it did detect it, at 48%. Why is there a difference? This is confusing!

[Mattk70]

Hi @ceperman , I formatted your table, as I couldn't easily read it. To be honest, as this is a comparison between BirdNET Analyzer and Merlin: the best place for feedback about your experience is https://github.com/kahst/BirdNET-Analyzer/discussions

Regarding your last paragraph, Chirpity has two Nocmig models and BirdNET available for detection, but it does not use a BirdNET web service (there isn't one, to my knowledge*). When using the BirdNET option, it uses the same model as BirdNET Analyser (v2.4) ported to JavaScript. It can be run offline and should give identical results. If it doesn't you can raise a bug report - please share the audio file if you do.

'* BirdNET is available here: https://birdnet.cornell.edu/api/, but this is indeed a very old version of BirdNET, and despite it having /api in the URL, I don't think the endpoints are documented.

[ceperman]

Hi @Mattk70, in as much as you can see this as a comparison between BirdNET and Merlin, you're right that it probably belongs elsewhere, and I may do that. I thought it would be interesting for Chirpity/BirdNET users to understand that when it comes to AI identification, opinions differ even between AIs from the same stable.

Perhaps more significant is the difference between BirdNET and what I could hear. I'm no expert but know a blackbird and pheasant when I hear one; BirdNET did pick them up but only with a low confidence (around 0.2). In order to catch these birds and others that were clearly present I would have to lower the Chirpity confident limit to 0.2, which would pick up many false positives and give me a lot more work eliminating them. Chirpity obviously does make this a lot easier to do, but it would operating at a much lower confidence level than would usually be recommended. At a more usual level of 0.7 I would have only picked up the Green Woodpecker and missed the other 8 species that were present.

I'm not sure where this leaves us, other than pointing out that using a high confidence level means eliminating false positives but also missing false negatives. To repeat what I said earlier, my gut feel is that BirdNET is much better at identifying isolated birds than when all jumbled together, as in a dawn chorus. So in the latter case, use a lower confidence level.

Re. what I said about Chirpity using the BirdNET web service, I know that BirdNET has a web interface and admittedly I was guessing that you used this somehow purely because Chirpity is getting different results from my desktop version. But as far as I can tell I'm also using v2.4 (BirdNET-Analyzer doesn't have a -version option but 2.4 is mentioned in the Readme.adoc file) so I don't know why I'm getting different results. I've included the files in question (mp3 files don't appear to be supported so I've zipped them).

Dawn chorus:
ENC0007-songt-greenp-short.mp3.zip
Barn Owl (which I cannot hear):
03395116.mp3.zip
03395116.BirdNET.results.csv

[Mattk70]

Thanks @ceperman ,

I think it's generally accepted that all AI models struggle when presented with a busy soundscape of overlapping species' calls. The very best performing ensemble models, such as those that win BirdClef achieve < 70% accuracy even after deploying many bespoke tricks to optimise for the test sounds. I am not sure of BirdNET's v2.4 model "soundscape" accuracy but I suspect it will be closer to 50% (maybe @kahst can comment?)

I looked at both files in Chirpity vs. BirdNET and can account for the different results. The main difference is that Chirpity does not report all the detections at a specific timecode, only the top one (You can see these if you look at the results for a single species - there will be a clickable grey circle indicating there are additional results) .

The other difference you will notice is that the reported confidence differs very slightly. This is due slightly different floating point rounding errors when your files are resampled to fit BirdNET's expected 48K. One of your file's rates is 44.1KHz, the other an unusual 46KHz. Neither give nice decimals when you convert them (e.g. 48/44.1 = 1.0884353741...) Python's resampling algorithm differs from that in ffmpeg - so they round in slightly different ways. For practical purposes it does not make a difference, and you don't see any difference at all if you use a sample rate that divides nicely into 48.

I've shown in the table below how that affects the results:

Start (s)	Scientific Name	Common Name	Confidence	Chirpity (BirdNET) Start (s)	Common Name	Latin Name	Confidence
3	Phylloscopus collybita	Common Chiffchaff	0.2548	3	Common Chiffchaff	Phylloscopus collybita	0.257
6	Pteruthius aeralatus	Blyth's Shrike-Babbler	0.2984	6	Blyth's Shrike-Babbler	Pteruthius aeralatus	0.3
12	Dendrocoptes medius	Middle Spotted Woodpecker	0.2483	12	Middle Spotted Woodpecker	Dendrocoptes medius	0.246
15	Phylloscopus ibericus	Iberian Chiffchaff	0.1537	15	Iberian Chiffchaff	Phylloscopus ibericus	0.154
24	Dendrocoptes medius	Middle Spotted Woodpecker	0.1074	24	Middle Spotted Woodpecker	Dendrocoptes medius	0.108
24	Columba palumbus	Common Wood-Pigeon	0.1023	-	Secondary results not reported	-	-
27	Columba palumbus	Common Wood-Pigeon	0.1421	27	Common Wood-Pigeon	Columba palumbus	0.142
30	Columba palumbus	Common Wood-Pigeon	0.1444	30	Common Wood-Pigeon	Columba palumbus	0.146
30	Dendrocoptes medius	Middle Spotted Woodpecker	0.1224	-	Secondary results not reported	-	-
33	Picus viridis	Eurasian Green Woodpecker	0.932	33	Eurasian Green Woodpecker	Picus viridis	0.93
36	Corvus corone	Carrion Crow	0.203	36	Carrion Crow	Corvus corone	0.205
39	Oriolus oriolus	Eurasian Golden Oriole	0.1855	39	Eurasian Golden Oriole	Oriolus oriolus	0.188
39	Picus viridis	Eurasian Green Woodpecker	0.1579	-	Secondary results not reported	-	-
51	Dendrocoptes medius	Middle Spotted Woodpecker	0.142	51	Middle Spotted Woodpecker	Dendrocoptes medius	0.143
51	Ficedula parva	Red-breasted Flycatcher	0.1336	-	Secondary results not reported	-	-
60	Picus viridis	Eurasian Green Woodpecker	0.4619	60	Eurasian Green Woodpecker	Picus viridis	0.456
66	Oriolus oriolus	Eurasian Golden Oriole	0.155	66	Eurasian Golden Oriole	Oriolus oriolus	0.154
66	Dryobates minor	Lesser Spotted Woodpecker	0.1212	-	Secondary results not reported	-	-
66	Dendrocoptes medius	Middle Spotted Woodpecker	0.1129	-	Tertiary results not reported	-	-
75	Dryobates minor	Lesser Spotted Woodpecker	0.1023	75	Lesser Spotted Woodpecker	Dryobates minor	0.103
81	Picus viridis	Eurasian Green Woodpecker	0.1771	81	Eurasian Green Woodpecker	Picus viridis	0.178
81	Columba palumbus	Common Wood-Pigeon	0.1738	-	Secondary results not reported	-	-
81	Dendrocoptes medius	Middle Spotted Woodpecker	0.101	-	Tertiary results not reported	-	-
84	Columba palumbus	Common Wood-Pigeon	0.171	84	Common Wood-Pigeon	Columba palumbus	0.169
87	Certhia familiaris	Eurasian Treecreeper	0.1381	87	Eurasian Treecreeper	Certhia familiaris	0.137
87	Dendrocoptes medius	Middle Spotted Woodpecker	0.1167	-	Secondary results not reported	-	-
90	Dendrocoptes medius	Middle Spotted Woodpecker	0.2527	90	Middle Spotted Woodpecker	Dendrocoptes medius	0.243
90	Dryobates minor	Lesser Spotted Woodpecker	0.2431	-	Secondary results not reported	-	-
90	Phylloscopus sibilatrix	Wood Warbler	0.2095	-	Tertiary results not reported	-	-
96	Regulus ignicapilla	Common Firecrest	0.2081	96	Common Firecrest	Regulus ignicapilla	0.21
99	Dryobates minor	Lesser Spotted Woodpecker	0.1914	99	Lesser Spotted Woodpecker	Dryobates minor	0.188
99	Phylloscopus sibilatrix	Wood Warbler	0.1457	-	Secondary results not reported	-	-
99	Dendrocoptes medius	Middle Spotted Woodpecker	0.1064	-	Tertiary results not reported	-	-
105	Dendrocoptes medius	Middle Spotted Woodpecker	0.2669	105	Middle Spotted Woodpecker	Dendrocoptes medius	0.258
108	Cyanistes caeruleus	Eurasian Blue Tit	0.1102	108	Eurasian Blue Tit	Cyanistes caeruleus	0.109
111	Dryobates minor	Lesser Spotted Woodpecker	0.121	111	Lesser Spotted Woodpecker	Dryobates minor	0.121
111	Certhia familiaris	Eurasian Treecreeper	0.1111	-	Secondary results not reported	-	-
111	Dendrocoptes medius	Middle Spotted Woodpecker	0.1032	-	Tertiary results not reported	-	-
114	Ficedula albicollis	Collared Flycatcher	0.157	114	Collared Flycatcher	Ficedula albicollis	0.157
114	Dryobates minor	Lesser Spotted Woodpecker	0.1147	-	Secondary results not reported	-	-
114	Muscicapa striata	Spotted Flycatcher	0.1104	-	Tertiary results not reported	-	-
117	Turdus philomelos	Song Thrush	0.3175	117	Song Thrush	Turdus philomelos	0.327

[ceperman]

@Mattk70 Thanks for the extremely detailed response and analysis. I've still a lot to learn about Chirpity!

FYI the file sampling rates. The 44.1KHz file was created using a commercial mp3 recorder, and this rate is fairly typical for CD quality recording. The other was from my home-grown recorder, which creates WAV format at 24KHz, 16 bits, mono. Because they are smaller, I attached mp3 versions that I'd created some while ago by exporting from Audacity, using its default export values.

When using Chirpity, I process the WAV files. Did you get to look at the Barn Owl file 03395116.mp3? I still see a significant difference in the owl detection between BirdNET used as a command (77%) and via Chirpity (48%).

BTW You said "...resampled to fit BirdNET's expected 48K...". I'm not aware of this requirement. When I use BirdNET I don't do anything special with the input files. Can you explain?

[Mattk70]

I did look at the Barn Owl file. If I run the mp3 file in BirdNET Analyzer, it picks up the Barn Owl at 47% I can see from the BirdNET csv that you got 77% when analysing the original wav file. Is this a co-incidence, or did you compare predictions from the WAV in BNA to predictions from the mp3 in Chirpity? If you think the wav file shows a discrepancy, maybe share the wav file?

Re resampling: BirdNET requires audio with a 48KHz sample rate. Both Chirpity and BirdNET Analyzer applications resample the audio internally to match that.

As a side note, I misread the properties of the barn owl file you shared. 46Kbps is the bitrate, the sample rate is actually 24KHz. A file this heavily compressed has a lot of compression artefacts, and doesn't provide the full frequency range used by BirdNET (0-15KhZ) for its predictions. I suspect this is the reason the results from the WAV and the mp3 differ so much.

An entirely different possibility is that you had applied audio filters in Chirpity and enabled "send filtered audio for analysis" in the settings. This will definitely result in differences.

[ceperman]

I used the WAV file for both, no filters. I've attached it.

I did a confidence comparison with Chirpity and BNA with the Barn Owl file, using (a) the original WAV file (b) the Audacity mp3 export I posted previously and (c) the WAV exported from Audacity to mp3, 24KHz mono:

File type	Chirpity	BirdNET-Analyzer
WAV	48%	77%
previous mp3	32%	58%
mp3 24KHz mono	not detected	not detected

The last file was created for interest. Even though the WAV file is 24KHz mono, as equivalent export is obviously of such poor quality that the Barn Owl is lost. I said originally that the owl call was so faint that I couldn't even hear it, so perhaps it's understandable.

You mention that BirdNET uses up to 15KHz for analysis. For my recorder I've implemented a 24KHz sampling rate, mostly to keep the file sizes small, which gives me an upper frequency of around 11KHz. I'd looked at sample sonograms and didn't see birds producing sounds above this. Are you suggesting it's not high enough? I can increase my sampling rate, obviously at the expense of file size. This is a consideration because, depending on the battery capacity, I might have the recorder deployed for weeks at a time, and then SD card capacity may be a limiting factor.

03395116.wav.zip

[Mattk70]

I did an analysis of the differences between BNA and Chirpity predictions for audio that isn't 48K: #205

The prediction differences between 44.1 and 48KHz sample rates is very small. Down at 24KHz they increase quite a lot. The kaiser_fast resampling algorithm attenuates the top end of the nyquist frequency quite noticeably in the spectrogram. So there will be significant perturbations in BirdNET Analyzer results if you use a low sample rate. In Chirpity, I suspect 30KHz might be sufficient, but I haven't tested it, If a minimal sample rate matters to you, you should do a similar test to the one I did to see what the optimum minimum rate really is.

The Nocmig models use 24KHz audio internally. I chose that for the same reason you landed there! The BirdNET guys use a rate 48KHz for some legacy reason, but they literally slice the audio spectrogram to remove all the frequencies above 15KHz.

[ceperman]

I've just analysed some of my recent 24KHz files using both BirdNET-Analyzer and Chirpity and got significantly different results. Reading what you said about 48KHz files, I looked at the BNA docs on GitHub. It doesn't have a requirement for 48KHz input but does say:

Model V2.4 uses the following settings: 48 kHz sampling rate (we up- and downsample automatically and can deal with artifacts (sic) from lower sampling rates).

So both Chirpity and BNA will resample my 24KHz files to 48KHz. What if they resample differently? I resampled a test file myself to 48KHz using ffmpeg and had Chirpity and BNA analyse it. Guess what? - both gave (virtually) identical results. What's more, the Chirpity results were the same for both the 24KHz and 48KHz files, which isn't surprising since we resampled in the same way. But it does mean that the reason for the differences is that BNA would appear to use a different upsampling mechanism from Chirpity/ffmpeg.

Moreover, the Chirpity results are far more convincing. In the sample I used, there was just one bird, a Song Thrush singing many times. Chirpity gave multiple Song Thrush detections, plus a few outliers which I could eliminate. BNA gave just one detection - a Tawny Owl, which could be tracked down to the first couple of notes of a Song Thrush song (to be fair, it did sound quite like a Tawny Owl and I've attached it). No mention of a Song Thrush.

My lessons from this are (a) if possible I will increase the sampling rate of my recorder to 48KHz, (b) if using lower sampling rate files I will only analyse them through Chirpity or (c) for use with BNA directly, will resample them to 48KHz with ffmpeg myself.

Mon Feb 03 2025 15-42-13.mp3.zip

[ceperman]

I've just seen your post in "Why are the Chirpity results from BirdNET sometimes different from those given when using BirdNET Analyser?"! - although I'm not sure it explains why I saw such wildly different results.

[Mattk70]

Sometimes a picture is better. This is what I was talking about in my previous comment. On the left: your original 24kHz file. In the middle, the file upscaled to 48kHz using Kaiser_fast (BirdNET Analyzer), on the right the file upscaled in Chirpity, by ffmpeg:

So, in spite of their documentation claiming that the software can deal with artifacts (this is the US English spelling) from lower sampling rates, the tests show BirdNET Analyzer doesn't do that as well as it could do if the sample rate is as low as 24kHz.

Re: What sample rate to use, 48kHz is best, but not really necessary. Results at 44.1kHz are virtually indistinguishable. Chirpity resamples in a different thread to the analysis, and delivers audio in chunks to the model. As a result, the analysis speed is almost the same regardless of the initial audio sample rate or even format (it will analyse M4A files about as fast as WAVs). BNA on the other hand does the audio conversion in the same thread, so it absolutely makes a difference. I suspect kaiser_fast is used because the time taken to resample at higher quality slows things down so much - in my BNA test a 48K file was done in c. 105 seconds, kaiser fast resampling took the analysis to c. 166 seconds, and kaiser_best took c. 266 seconds.

[ceperman]

I have to admit I can't see much of a difference between any of them, apart from the centre one being clipped at higher frequencies. The 24KHz original and Chirpity's upsampled version look identical to my eye.

I still don't understand why what appears to be a slight difference in upscaling should produce such a large difference in detections - many (and obvious) Song Thrush detections versus none at all. Still, whatever the reason, the results are clear and I shall be staying away from BNA with 24KHz files. In fact, the more I use Chirpity and get to appreciate its features the more I like it, such that I have no reason to use BNA native!

I did try recording at 48KHz (my recorder is based on an ESP32 microprocessor, and changing sampling frequency is simple) although I'm not convinced of the benefit. Analysis on my 24KHz files through Chirpity looks to work well - how much better detection could I expect with 48KHz? I'm not seeing any bird sounds above 9KHz. The issue has just been with BNA upsampling and I will be avoiding that. Downsides of 48KHz for me is larger file sizes and a significant increase in power consumption by the ESP32 - 70mA vs 110mA - and battery life is important for remote deployment. So unless something more persuasive comes to light, I'll be sticking with 24KHz.