Bring back flex

[pedroCabrera]

Hi there,nice work, I saw your work back in the days, I did the xsi version at that time!
this is old I know, but still actually very fast btw. I am looking into this again.
I see in your sample scenes there is a a limit of 1.300.000 particles, I didn't have (still don't) have that limit in the xsi implementation.

I have compiled this repo for 20.5 and as I saw on one of your scenes comments, if you set up your particle limit to bigger than that, sims start to go crazy, and yes they do, did you ever figured why that was?

I don't think is a library problem as I can go way higher with xsi 2015 and my 3090 nowadays ( probably not at that time ).
So I'm thinking of porting the 1.2 flex library (never released , just source code ) for houdini and make it more "vellum style" into "sops".

I was wondering if you are interested in collaborating on that and recovering this project to live as it's still one of the fastest simulators for some kind of things out there!!

Hope to here back from you!!

[pedohorse]

Hey!
About the particle limit - as far as I remember, I've dug some forums back in the day and found enough confirmation that this was some kind of flex implementation problem, so I dropped the issue for the time being.

Haven't touched this repo in years, forgot many things and details by now.
The main reason I've stopped working on this one is just that noone seemed to have been interested in it, also vellum was added to houdini around half a year after this plugin release.

so if you say that there is a need for this plugin beyond just academic interest - then i'm interested to dig it from it's grave and take a new closer look on those old issues.

and as for building a sop framework similar to sop vellum solver - it's a simpler task as it can be built around core dop nodes, just like vellum assets are built.

[pedroCabrera]

Well I'm not sure about the interest in general, I'm actually interested on having a super fast cloth simulator for quick things, I don't think the multi solver is the key part, it's nice but I don't see it super useful for production, but cloth, skin,ropes etc with a super fast solver I do think is going to be useful.
Also I think it didn't get much attention cause it is in dops, don't get me wrong I like dops, but people since 19.5 like to use sops approach and I think they will like a fast "vellum style" for simple tasks.

So I'm going to update to flex 1.2 and will dig a bit more into that limit, as I can go further in xsi, Im able to simulate 9M at 1.2 fps there.

I'm going to fork the repo and make changes there. Will inform you if I find something related to that limit then!!

[pedohorse]

Ok, I'll then try to dust it off too during the weekend and maybe remember what's what.

as for the dops thing - back in the day when it was released - there was no vellum yet, and when vellum appeared - it was dop only too.
But yes, sop things are easier for artists to get into.

[pedroCabrera]

Nice! I must say is my first time with the Houdini sdk, so your code is super useful!!
I still need to dig more on it to understand how everything works in Houdini dops custom plugins, but I also have my flex xsi code that I can refer to to search for differences and see if I can find where the limit comes from, I want to do some tests with their original demo and see where the limit it's there!!

[pedohorse]

oh yes, houdini SDK might strike you with it's weirdness. especially DOP context, as it's logic is not very intuitive compared to SOP.
It helps if you ever worked with dops themselves and understand the concepts of dop objects and dop data, and how they are referenced and copied-on-write in different time samples

[pedroCabrera]

As you can see here im simulating 4M particles in XSI Flex, and remains stable, so we need to find where the limit is in the houdini version 🤔

[pedohorse]

I seem to have hit a deadend on linux side:
NvFlexExtReleaseCUDA_x64.a file uses symbol __sqrtf_fixed that was removed from glibc starting from 2.31, so it required mangling to rename that symbol with objcopy.
but then those static libs also require some cuda calls from cudart that were removed starting at cuda 10, and cuda toolkit 9 is not even available to download any more...
what a mess!
I wish they released flex's source code so we could recompile those deprecated static libs...
edit: ok, i've actually noticed the extensions lib code and recompiled it to aviud the objcopy workaround, but it still uses deprecated cudart calls

[pedohorse]

also while looking at all that mess - i've noticed that it seems like on windows it can use DX12 instead of cuda as a backend - which one are you using in your XSI plugin? maybe that makes difference and causes that particle limit bug?

[pedroCabrera]

Im Using cuda In the xsi plug-in so it can't be that , I think my next step is going to try to make a super simple solver (maybe even a sop node to avoid any data interaction fucking with limit) and check if I can go higher with that, basically porting the xsi plug-in to a sop node ( I think is the easiest way ) , Wich into a sop solver, can probably work for testing at least hahha

[pedohorse]

I seem to have hit a deadend on linux side: NvFlexExtReleaseCUDA_x64.a file uses symbol __sqrtf_fixed that was removed from glibc starting from 2.31, so it required mangling to rename that symbol with objcopy. but then those static libs also require some cuda calls from cudart that were removed starting at cuda 10, and cuda toolkit 9 is not even available to download any more... what a mess! I wish they released flex's source code so we could recompile those deprecated static libs... edit: ok, i've actually noticed the extensions lib code and recompiled it to aviud the objcopy workaround, but it still uses deprecated cudart calls

found cuda toolkit 9, compiled with it, and what do i get? stack smashing detected !
i guess there is no way to go without recompiling all of flex's static libs which don't have source code available.
It's surprising that it compiled and works on windows no problem

[pedroCabrera]

Oh weird , I have it compiling in windows no problem. I don't use linux son can't really help there :S

[pedroCabrera]

Im compiling with cuda 12.5 also

[pedohorse]

well, i guess nvidia cares much more about windows developers)
i guess you either have an old curart.dll, or nvidia keeps much better backwards compatibility in it.

[pedroCabrera]

now i am able to compile the 1.2 flex library with a few small changes, but when trying to load in houdini , even if im correctly specifitying to load the dynamic libraries from the 1.2 i get this error on houdini load:

NvF ERROR: Could not initialize Flex, header/library mismatch, provided: 110, expected: 120 :: ../../cuda/flex.cu : line 4595
Failed to initialize Flex library
OMEGA ERROR: Failed to initialize Flex library !

The flex.cu file doesnt exists, so im worring maybe the .lib files they provide in the branch are not compiled to the new version? as there never been a release of it .

[pedohorse]

as far as i look through their libs - it is from there, all compiled into the static lib

according to their commit log - they did update libs for 1.2 release here

But if you are sure you are linking with correct libs from 1.2 flex, and not by accident with 1.1 - then it means nvidia did not update those libs correctly, and it is what it is

[pedroCabrera]

That's the thing, to be sure I even removed the 1.1 flex version from my computer and hardcoded into the path variable the folder with the 1.2 DLLs and then recompiled. At compile time I do know I'm using the 1.2 headers as they ask for some changes, and I link against 1.2 lib folder.
So im not sure if the problem comes from the dll or the lib file. I had to change one declaration in one header of the 1.2 core.h , do you think it can be that?

[pedohorse]

now i am able to compile the 1.2 flex library with a few small changes, but when trying to load in houdini , even if im correctly specifitying to load the dynamic libraries from the 1.2 i get this error on houdini load:

NvF ERROR: Could not initialize Flex, header/library mismatch, provided: 110, expected: 120 :: ../../cuda/flex.cu : line 4595
Failed to initialize Flex library
OMEGA ERROR: Failed to initialize Flex library !

The flex.cu file doesnt exists, so im worring maybe the .lib files they provide in the branch are not compiled to the new version? as there never been a release of it .

ah, wait, this error is probably just from this like where 110 is hardcoded:

Houdini-NvFlex/nvFlexDop/SIM_NvFlexData.cpp

Line 221 in 0b674c2

nvFlexLibrary = NvFlexInit(110, &nvFlexErrorCallbackPrint, &desc);

[pedroCabrera]

hahaha ok im stupid then, makes sense. now im geting this error just at start after changing that to 120
30776: Fatal error: Segmentation fault
Crash log saved to

So, definitly that was causing the lib to say not compatible. now ill need to debug whats going on here!!

Thanks for pointing me that out!!

Btw im also working on a separate branch changing it to a geom + constraint geom separated workflow.

[pedohorse]

seeing how unsupported and abandoned flex is, i'd be cautious to put much effort into integrating with it. Also not being able to compile on linux is a full showstopper for me..

I've googled around, and it seems that flex was abandoned, but nvidia has implemented physx new particle system modeled after flex: https://nvidia-omniverse.github.io/PhysX/physx/5.1.0/docs/ParticleSystem.html and discussion here: https://forums.developer.nvidia.com/t/is-physx5-using-flex/195938
So i'd seriously consider porting/rewriting the whole thing with physx

[pedroCabrera]

Yeah i thought of implementing Physx but thats a way bigger task i think, I want to get used to the HDK and dops with this one where you did an amazing work and extend from here while learning, if at some point i fell i have the time and knowledge to implement a bigger library like physx will for sure go with it ahahha

[pedohorse]

physX is big, but you don't have to implement it all.
I'm not sure, but it looks like implementing just that PBD system might be as big of a task as flex.
maybe colliders will be a bit more complicated to deal with though...

I might try to do some trivial physX integration to see if porting is feasible, since I cannot do anything with flex any more, just as soon as I finish with my other chores