TinyXML(2) might cause problems with the XSD schema now often attached due to the new package format (V2)

[arne48]

This is more a general problem caused by the wide use of TinyXML(2).
But to me it became the most apparent in rospack as this package provides the commandline tool and basic library to deal with ROS packages.

The problem is based on the lightweight approach of TinyXML(2) in which they don't support the parsing of schemas. see What it doesn't do
This property combined with the provided schema in REP140 led several developers to add it into their package.xmls.
In my minimal installation I found that this was already done in pluginlib, nodelet_core, bondpy, test_bond, bondcpp, bond_core, smclib, dynamic_reconfigure, bond and class_loader.

The updates containing the schema are quite new and I don't know if their are any changes already planned to address this potential issue.

According particularly to rospack one solution would be to switch to a more complex parser which supports schemas. But this will obviously not affect packages which don't use rospack but using TinyXML(2) themselves to parse package.xmls.
A more global approach would be to avoid the addition of the schema to the package.xml and maybe provide a dedicated library to verify them if needed. Doing so would preserve the wide usage of TinyXML(2).

[dirk-thomas]

From your description it is unclear to me what the actual problem is. Afaik TinyXML simply ignores the schema declaration. Maybe you can clarify.

[mikaelarguedas]

Looking at the recommendation from W3C about embedding references to xsd in xml files (see section "A Reference to an XML Schema" here) and based on recent experience working with XML embedding schemas like the DDS ones. It seems that one solution is to update the reference to the schemas in REP140 from:

<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format2.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="2">

to:

<?xml version="1.0"?>
<package format="2" xmlns:xsi="http://www.w3.org/2001/XMLSchema" xsi:noNamespaceSchemaLocation="http://download.ros.org/schema/package_format2.xsd" >

That will work around the limitation of TinyXML2 and allow rospack to keep working as expected regardless of the version of Tinyxml2 used

[dirk-thomas]

There are multiple ways to embed a schema. While we could change all manifest I would recommend to investigate why the previous working approach now fails with a newer version of TinyXML2 (I assume that is the reason?) and ticket the problem upstream.

[mikaelarguedas]

I don't know how the way to include reference to xsd was decided. Do you remember why we chose the xml-model approach and how to use it (don't see any reference of it on the W3C website)?

While there are different approaches, it looks like the one recommended by W3C and used by other projects is more widespread and works using our current parser. Based on that and the small number of packages.xml referencing xsd, it looks to me that it is worth updating the handful of packages and the example in the REP to use it.
Based on the TinyXML2 doc, If you are working with browsers or have more complete XML needs, TinyXML-2 is not the parser for you, I'm not sure that an upstream ticket would be addressed. If they submit a bugfix to not crash, I'm not sure it would make it into Zesty (even less Yakkety that is already EOL).

[mikaelarguedas]

how to use it (don't see any reference of it on the W3C website)?

Answering my own question : https://www.w3.org/TR/xml-model/#the-xml-model-processing-instruction

[dirk-thomas]

The goal is not to make TinyXML to "support" schemas. It is only about correctly parsing the markup.

[mikaelarguedas]

It is only about correctly parsing the markup.

I agree, my point is more that we won't have a fix in Yakkety and likely not have one in Zesty so we still need to touch every package using it or change rospack's parser to have a working rospack for Lunar packages

[dirk-thomas]

Please try the alternative proposal with existing linters. I can't get it to e.g. pass xmllint. That might have been one reason why the current approach was chosen.

[dirk-thomas]

It only fails with TinyXML2 version 4.0.1 which is e.g. in Ubuntu Zesty. All other versions (3.0.0 and lower as well as 5.0.0 and higher) don't have the problem and work as expected.

[mikaelarguedas]

looks like leethomason/tinyxml2#496 was the fix (or part of the fix need). It has been merged after 4.0.1 was tagged and is part of 5.0.0 (original issue: leethomason/tinyxml2#399)

[arne48]

As proposed here #606 I sent a request to the maintainers of the libtinyxml2 package of Debian and Ubuntu. I will keep you updated if I receive an answer.
Further I had a look into the ebuild script of Gentoo as the third maybe to be major platform.
They use the tarballs located on GitHub so if the GitHub developers of tinyxml2 backport the merge this problem will be solved on Gentoo as well.

Changes took place:

• Debian
  [x] Sent mail to maintainer
  [x] Received answer from maintainer, impact and eligibility to patch will be inspected

• Ubuntu
  [x] Waiting for approval by mailinglist moderator
  [x] Received response proposing as Zesty is the only affected distribution still supported to issue a request to the Stable Release Update team.
  [x] Preparing and sending the request to update the package in Zesty
  [x] Decided not to pursue this further as Zesty is almost out of support and we have an alternative binary in the ROS repository.

• Gentoo
  [x] Checking #606 if this patch gets backported
  [x] Backport was denied. Gentoo users will need to emerge 5.0.1 by themselves and ensure it to be linked against.

[mikaelarguedas]

Thanks @arne48 for the initiative.
Feel free to post here as soon as you get a reply from the maintainers.
In the meantime I created a branch with the desired commit on top of 4.0.1 here and built debs for yakkety and zesty using it as a new patch. I haven't done stretch packages yet. According to the answer you receive from the Debian maintainer we can use their updated package or distribute these homemade binaries on packages.ros.org.

[mikaelarguedas]

Every other platforms or distro are shipping versions that dont have the problem so we should be good:
Gentoo, 5.0.1
Fedora: 3.0.0
Ubuntu Xenial 2.2.0
Ubuntu Artful: 5.0.1
Debian Buster: 5.0.1

[mikaelarguedas]

libtinyxml2 with the patch is now available on packages.ros.org for Yakkety, Zesty and Stretch. So this shouldn't be an issue anymore.
@wkentaro @arne48 can any of you give it a try and confirm that the issue is solved ?

[wkentaro]

It seems fixed, thanks.

[mikaelarguedas]

Great, thanks for testing!
I'll leave it @dirk-thomas to chose when to close this ticket

[arne48]

Thanks to all of you to fix this problem so fast.
I will keep updating my post about packages in Ubuntu, Debian and Gentoo even though this issue is now closed and thankfully to @mikaelarguedas we have a patched package is the ROS repository.

[dirk-thomas]

@arne48 Do you have any links for tickets against the various distributions? That might help future readers like https://answers.ros.org/question/278733/rospack-find-throws-exception-error/ to determine the state.

[arne48]

@dirk-thomas
The current states of fixes in the three biggest Distributions affected by this issue are:

Debian Stretch:
I had email contact with the maintainer of the package Chow Loong Jin who told me he will have a look into it but neither did I get another response or saw a note in the changelog addressing this issue.

Ubuntu 17.04:
Updating a package of a released version besides security requires the issuing of an StableReleaseUpdate evaluating the impact, test cases and regression potential. If this passes the Ubuntu Stable Release Update team it would go into verification. As we already had a fixed version for in the ROS repository I stopped here. This was because 1. the EOL of 17.04 is 1/1/2018 and 2. when I figured out all the details about this process I doubt that the issue would make it through until the EOL.

Gentoo:
The build sources of tinyxml2(4.0.1) come from the Github repository and in an answer to #606 the developers pointed out that they will not update 4.0.1 anymore. (User Patches might be a solution here)

The biggest problem at the moment might be armhf on Stretch. I don't know if armhf is a big issue for Ubuntu as I saw a lot distributions use either Stretch or Xenial. Maybe we can also provide a deb-package for armhf on Stretch.

Now I contacted Chow Loong Jin again.

[justdoGIT]

@mikaelarguedas can you suggest some patch for armhf debian stretch for the above error.

[mikaelarguedas]

@kamalpandey1993 The patch I applied and released in ROS is available at https://github.com/mikaelarguedas/tinyxml2/tree/4.0.1-patch (see #77 (comment)).
Installing that version from source should solve your issue.
HTH

[arne48]

Stretch Update:
Also Debian Stretch will get no official backport of this fix.

To my understanding the last devices that can be effected are armhf SBCs running a Stretch-based distro (aka. RPi) since Ubuntu 17.04 is now out-of-support and @mikaelarguedas provided packages for arm64 and amd64.
As armhf is even not officially supported for Lunar on Stretch the impact of this issue is now minor.
So as long @mikaelarguedas is not so generous to provide a deb package for an unsupported platform (what I personally could totally understand) no further measures might be justified.

[jamesbellaero]

I managed to get everything up and running on Debian Stretch and armhf by adding the Buster mirror to my /etc/apt/sources.list file, running sudo apt-get install libtinyxml2-dev, then doing a clean build of the entire ros workspace. You still have to install every ros package by cloning them from Git repos, but at least it's possible to make everything run.

[ghost]

I am running melodic ROS on raspberry pi and debian stretch OS. My package.xml file for rospack has states 2.5.2.

I had the same issue regarding rospack, it would return the same error as #86. As advised the error was due to libtinyxml2 being version 4.0.1-1. I tried to follow @jamesbellaero advice and included a buster mirror then
sudo apt-get update | sudo apt-get remove libtinyxml2 -y | sudo apt-get install libtinyxml2-dev -y
The last command also installs libtinysml2-6a
dpkg -l libtinyxml2-6a returns version 6.2.0+dfsg-3
I go to my ROS root directory and
./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release --install-space /opt/ros/melodic
and everything is rebuilt.
I run rospack list and now it says it can't find libtinyxml2.so.4
I thought I'd be clever and make a symbolic link in the /usr/lib/arm-linux-gnueabihf directly from libtinyxml2.so.4 to libtinyxml2.so.6 but rospack list returns a Segmentation Fault.

I'm out of ideas, please help~~

[dirk-thomas]

I'm out of ideas, please help

I am not sure if you have seen the ROS support guidelines but we kindly ask to raise questions like this on answers.ros.org instead.

The rational for asking all kind of questions in a single place is:

• There are many more people reading questions there so your chances on getting an answer and in a timely manner are much higher.
• In the future other users will search there for similar problems and can find your question and the potential answers.
• The issue tracker in this repo is used to track bugs, feature requests, etc. But your question is not such an action item.
  *Additionally this ticket is already closed and your comment will therefore have even less visibility.

If you have asked your question on answers.ros.org please feel free to add a link to the question to this ticket so that future readers can find the related question.

[ghost]

@dirk-thomas Noted with thanks!