Update chen to get memory and performance improvements

[prabhu]

@cerrussell Can you check this branch with atom-tools to ensure we don't lose any endpoints especially for python?

To install the atom command based on this branch.

git clone https://github.com/AppThreat/atom.git
cd atom
git checkout feature/chen-update-mem-perf
export GITHUB_TOKEN=PAT with read access
sbt clean stage scalafmt test createDistribution
cd wrapper/nodejs
bash build.sh && sudo npm install -g .

[prabhu]

@Hritik14 can you test #144 with this branch?

[cerrussell]

@prabhu It does appear Python endpoints are lost with this. Javascript endpoints are also being lost but to a lesser extent.

[prabhu]

@cerrussell, I manually reviewed the changes to chen and unable to pinpoint to anything in particular. Let's first review the setup and try using an existing release to get the baseline, which can be used for comparison.

[Hritik14]

I'm unable to rebuild, can you help ?

; git rev-parse HEAD
0a9eda776fa175299e3d24bae92230c1e0e24859
; cd wra
; ls
README.md  build.ps1  index.js      package-lock.json  php.ini       plugins  utils.mjs
astgen.js  build.sh   node_modules  package.json       phpastgen.js  target
; npm install .

up to date, audited 113 packages in 710ms

24 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
; find . -name "*atom*"
./node_modules/eslint/lib/rules/require-atomic-updates.js
; ls node_modules/.bin
acorn  eslint  js-yaml  node-which  parser  rimraf  tsc  tsserver
; sbt clean stage scalafmt test createDistribution                                                                                    [NORMAL]
cd wrapper/nodejs
bash build.sh && npm install  .
[warn] No sbt.version set in project/build.properties, base directory: /DIRECTORY/atom/wrapper/nodejs
[info] welcome to sbt 1.10.0 (Oracle Corporation Java 21.0.1)
[info] set current project to nodejs (in build file:/DIRECTORY/atom/wrapper/nodejs/)
[success] Total time: 0 s, completed 04-Jul-2024, 2:59:38 pm
[error] Not a valid command: stage (similar: last-grep, set, last)
[error] Not a valid project ID: stage
[error] Expected ':'
[error] Not a valid key: stage (similar: state, target, tags)
[error] stage
[error]      ^
cd: no such file or directory: wrapper/nodejs
Build the atom project using 'sbt createDistribution' before running this script
All settings correct for using Composer
Downloading...

Composer (version 2.7.7) successfully installed to: /DIRECTORY/atom/wrapper/nodejs/composer.phar
Use it: php composer.phar

./composer.json has been created
Running composer update nikic/php-parser
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking nikic/php-parser (v4.18.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Generating optimized autoload files
No security vulnerability advisories found.

up to date, audited 113 packages in 623ms

24 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

up to date, audited 113 packages in 599ms

24 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
;

; find . -name "*atom*"
./node_modules/eslint/lib/rules/require-atomic-updates.js

[prabhu]

@Hritik14, I have updated the build instructions in the first comment. You can also use the below container image.

docker pull ghcr.io/appthreat/atom:feature-chen-update-mem-perf

[Hritik14]

Used the container image. Container VM has 8G memory.

Still OOM

bash-5.1# atom usages --extract-endpoints -l java .
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.lang.invoke.DirectMethodHandle.allocateInstance(DirectMethodHandle.java:501)
        at java.base/java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(DirectMethodHandle$Holder)
        at java.base/java.lang.invoke.Invokers$Holder.linkToTargetMethod(Invokers$Holder)
        at com.github.javaparser.ast.visitor.VoidVisitorAdapter.visit(VoidVisitorAdapter.java:442)

bash-5.1# find . -name "*.java" |  wc -l

123962

[prabhu]

@Hritik14 can you repeat it with 16, 32 etc to find what is the max it needs?

[prabhu]

@cerrussell This is ready for retesting on windows. I added one test for Windows here.