Readlock Errors when using Set-ObjectProperty -Tags since Update 23.1.82 #325
Comments
|
When using multi-edit to set the tags in the GUI, the following displays in the webserver.log: When using Set-ObjectProperty, the call looks different: Not sure if that makes a difference and was unable to determine based on previous logs from prior PRTG version if the multi-edit always used that same syntax or if it maybe changed now. |
|
Are you able to advise whether this issue occurs when updating the tags of just a single sensor at once? |
|
Also, are you able to advise what you mean by "readlock" I ran the following command on my test server running 23.1.82 get-device ci*|get-sensor | Set-ObjectProperty -Tags a,bMy PRTG Server UI still appears to be functioning normally (although it hasn't been 20 minutes yet) |
|
It seems that when updating the tags of just a single sensor at once, the readlock issue does not occur. If possible, I could change the script to only set sensors that don't already have the desired tags, to reduce the number of sensors being changed at once. However, for some reason I cannot seem to get the Where clause to work as expected. For example, Get-Device Device | Get-Sensor | Where Tags -notlike DesiredTags does not seem to work to only show sensors without the desired tags, it seems to show all sensors on said device(s). I also tried storing the tags individually by sensor ID into a variable and converting ToString() to compare, but the logic still does not work as I expect it to - it thinks that the tags aren't applied when they actually are. Maybe there is a key piece I am missing though? Here is an output of the core.log entries when the "readlock" issue occurred yesterday. From about 15:22 > 15:41, the errors occurred essentially in a loop until the core server service restarted to correct itself. 2023-02-14 15:22:00.231576 ERRR TId 4416 Core> Wants: RL Core.BL.Users.Tree.Operations.ModifyTreeNode2 4398041330192 (4416) From: RL Warning: The current Thread ran into a timeout while entering a readlock [1073741825] . LockReason: "Core.BL.Users.Tree.Operations.ModifyTreeNode2" Lockinfos: 1. Thread(6820): ReadLocked LockReason: Core.BL.Users.Tree.Operations.ModifyTreeNode2 (4416) info: 47 |
|
After researching it more, based on similar coding examples that I saw online, it seems the Tags property is an array. It sounds like this means the logic I have for -notlike in the scanning intervals example won't work for comparing the Tags array to exclude items with tags that are already correct. It sounds like to make the script more granular and only fix the missing/incorrect values instead of it always trying to apply the tags to every single device with that name, I'll have to have the script loop through the tag array in some way after retrieving it from each device. I will try to come up with this more intricate scripting as a possible workaround to the issue. It still seems like it should be supported to change many values at once if needed (i.e. large/new installations where tagging needs to be ensured as working as expected on hundreds of sensors at once), and it has worked in the past. For sure my code will become more optimized and efficient with the above, but I am still very curious as to how this new PRTG version seems to have broken what worked OK before. |
|
Code will be something like: $Tags = Get-sensor -Device String | Select tags |
|
PRTG Support just responded to me with "We've been seeing performance issues on the latest version, none related to using this PS module as we do not support it, but in general, we'll likely have new versions soon to stabilize it." |
|
Unfortunately, I can't seem to come up with the required logic to parse the array and then automatically determine only the sensors which have the incorrect tagging set. One big reason is that the parent tags are inherited in the Tags property. In some cases, these are different based on the different objects. It's interesting that most properties can be easily interacted with as strings but in the case of Tags, they are within an array. When exporting them to Excel for example, it does not show the tags but rather System.String[]. As a partial workaround, I can look for "default tags" to target the replacement against, for example the bandwidthsensor tag that's automatically applied when applying the SNMP Traffic sensor: *Get-Device STRING | Get-Sensor -Tags bandwidth | Set-ObjectProperty -Tags MYTAGS Do you have any suggestions for how to use the Tags array to successfully target only the sensors that need changed and not all sensors with that device string? |
|
If you do I am currently investigating the readlock issue you've described to see if I can replicate it. Internally, the |
|
Thanks a lot for that suggestion of using Get-ObjectProperty. This eliminates the ParentTags problem like you said. At this point, I determined a way to at least get a listing of "offending" sensors which need to have their tags corrected. However, it looks like a simple Where -notlike statement does not work to exclude the sensors with the correct tagging. Also, since the Get-ObjectProperty does not seem to have an ID field, it seems that automating the tag application to those that need it within the conditional statement would not work. Using the sensor name property will tag ALL sensors with that name. For example, "Ping" is on every device, so it would tag all Ping sensors with the tags meant for only certain devices. Here is the code right now with comments on what I mean: If there was a way to also get the object ID with Get-ObjectProperty, the $TagSets could also include the ID property, which would then be used in the else statement, with something like Regarding investigating the readlock issue, I appreciate your testing to see if you can replicate it. It seems that it happens in our case when the tagging happens on over 200 sensors at once. The script does not crash when running against cases with ~10 sensors at once. But when it moves to the line where over 200 sensors are being tagged at once is when the readlocks start. |
|
Actually, I just realized that I can use the -Raw parameter to address the above use case and obtain the id field :) |
|
When working on lists, use The following is a useful technique for diffing two lists $first = 1,2,3
$second = 2,3,4
$missingFromFirst = $first | where { $_ -notin $second }
$missingFromSecond = $second | where { $_ -notin $first }The Sensor ID is on the $sensors = Get-Device | Get-Sensor
foreach($sensor in $sensors)
{
$props = $sensor | Get-ObjectProperty
} |
|
Thank you for the explanation/recommendation to use -notin when working with lists! I will remember this going forward because it looks to simplify the code greatly compared to what I ended up doing because I have some redundant lines. I was able to adjust the script to only apply the missing tags with this block of code for each type of device: My method for sure doesn't scale well for cases with many tags. But luckily in our case, we only need 2 tags on the sensors. |
Since applying update 23.1.82, it appears that readlock issues occur when using Set-ObjectProperty to set Tags on a bulk amount of sensors at once. The readlocks never auto-correct and the core server service restarts automatically after about 20 minutes to recover.
Example of command where problem occurs: Get-Device "StringToMatch" | Get-Sensor | Set-ObjectProperty -Tags (2 Tags)
Within the webserver.log, the respective API commands look like this (internal specific info omitted for security purposes)
#1157155 DATE TIME IPADDRESSOFREQUEST "USER" PRTG_CORE_NAME 443 GET /editsettings id=List_of_About_200_Sensors&"tags_=TAG1 TAG2"&username=USERNAME&passhash=...
I have a ticket opened with PRTG support at the moment and have delivered the log files. They are currently examining them.
Prior to update 23.1.82, this process ran nightly and never caused issues. Previously, we were on version 22.4.81. I have noticed similar readlock errors in the past, but only when experimenting with the -RawProperty values which the module documentation warns can cause undesired results.
The issue has occurred on all of our core servers. Based on this, it seems to not be related to one specific problem sensor but perhaps the underlying API call changing as part of update 23.1.82.
I will continue to test and will update based on PRTG Support's findings as well.
The text was updated successfully, but these errors were encountered: