Elastic bi

[0ssigeno]

Description

Information about the reports can be stored inside elastic

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue).
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

• I have read and understood the rules about how to Contribute to this project
• The pull request is for the branch develop
• A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
  • I strictly followed the documentation "How to create a Plugin"
  • Usage file was updated.
  • Advanced-Usage was updated (in case the plugin provides additional optional configuration).
  • If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
  • I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. [Doc]("How to create a Plugin")
  • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
  • If you created a new analyzer and it is free (does not require API keys), please add it in the FREE_TO_USE_ANALYZERS playbook in playbook_config.json.
  • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
  • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
• If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
• Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
• I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
• If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
• If the GUI has been modified:
  • I have a provided a screenshot of the result in the PR.
  • I have created new frontend tests for the new component or updated existing ones.

Important Rules

• If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
• If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
• Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

[code-review-doctor]

Some things to consider. View full project report here.

[mlodic]

I think it could make sense to provide the schema of the index template to the users so they can insert it manually to the Elastic prior to insertion of BI data.

[mlodic]

could you please provide a sample as an example?

[0ssigeno]

[mlodic]

I would use intelowl-bi instead of intel-owl-bi. In this way we use the - correctly.
The environment would be more useful it that was put inside the index name like intelowl-bi-stag. In this way we can choose different retention mechanism based on that name (it already works like that for other indexes that we collect)
Is the timestamp related to the moment when this entry is written in elastic or when the analyzer has started?

[mlodic]

I would save the application in lowercase too, just to align with other tags we already save

[codecov]

Codecov Report

Attention: 1619 lines in your changes are missing coverage. Please review.

Comparison is base (aa8820f) 66.75% compared to head (56dc092) 75.59%.
Report is 1575 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2036      +/-   ##
===========================================
+ Coverage    66.75%   75.59%   +8.84%     
===========================================
  Files           95      481     +386     
  Lines         3706    14388   +10682     
  Branches       519     1495     +976     
===========================================
+ Hits          2474    10877    +8403     
- Misses         941     2915    +1974     
- Partials       291      596     +305     

Files	Coverage Δ
api_app/analyzers_manager/admin.py	100.00% <100.00%> (ø)
api_app/analyzers_manager/apps.py	100.00% <100.00%> (ø)
api_app/analyzers_manager/constants.py	100.00% <100.00%> (ø)
api_app/analyzers_manager/exceptions.py	100.00% <100.00%> (ø)
...app/analyzers_manager/file_analyzers/boxjs_scan.py	100.00% <100.00%> (ø)
...pi_app/analyzers_manager/file_analyzers/onenote.py	100.00% <100.00%> (ø)
...pi_app/analyzers_manager/file_analyzers/peframe.py	82.35% <100.00%> (ø)
...i_app/analyzers_manager/file_analyzers/rtf_info.py	48.64% <100.00%> (ø)
...s_manager/file_analyzers/xlm_macro_deobfuscator.py	62.50% <100.00%> (ø)
api_app/analyzers_manager/filters.py	100.00% <100.00%> (ø)
... and 237 more

... and 290 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea2abcb...56dc092. Read the comment docs.