Skip to content

Feature/json io #1013

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

Conversation

flohorovicic
Copy link
Member

Description

This PR introduces JSON input/output functionality to GemPy, allowing users to save and load geological models in JSON format. This new feature provides:

  1. Initial implementation of JSON serialization and deserialization for GemPy models
  2. Complete support for all essential model components:
    • Surface points and orientations
    • Series and structural relationships
    • Fault relations
    • Grid settings
    • Interpolation options
    • Model metadata
    • Element colors and styling

The JSON format offers several advantages:

  • Human-readable and editable model representation
  • Language-agnostic data format for interoperability
  • Structured validation of model data
  • Easy version control and model sharing

The implementation includes:

  • JsonIO class with save_model_to_json and load_model_from_json methods
  • Schema validation for JSON data
  • Sensible defaults for optional parameters
  • Comprehensive error handling

Key improvements in latest commits:

  • Colors are now correctly preserved when loading models from JSON
  • Structural relationships (ERODE, ONLAP) are properly handled
  • Fault relations matrix is correctly saved and restored
  • Metadata (creation_date, last_modification_date) is properly managed
  • Default values are provided for optional parameters
  • Comprehensive error handling for invalid data structures

Relates to #1003

Checklist

  • My code uses type hinting for function and method arguments and return values.
    • Added type hints in JsonIO class methods (load_model_from_json, save_model_to_json)
    • All helper methods include proper type annotations
  • I have created tests which cover my code.
    • Added 19 comprehensive tests in test_json_io.py
    • Tests cover both success and failure cases
    • Edge cases are properly tested
  • The test code either:
    1. demonstrates at least one valuable use case (e.g. integration tests)
      • test_multiple_series_relationships demonstrates real-world usage with multiple series
      • test_combination_model shows complex model with faults and unconformities
      • test_fault_relationships verifies fault relation handling
    2. verifies that outputs are as expected for given inputs (e.g. unit tests)
      • Unit tests for surface points loading/saving
      • Unit tests for orientations handling
      • Unit tests for metadata preservation
      • Unit tests for color preservation
  • New tests pass locally with my changes.
    • All 19 tests pass successfully
    • Only expected warnings are present (deprecation notices and known basement color warning)

- Add JSON I/O module with schema definitions

- Implement surface points loading functionality

- Add comprehensive test suite for surface points I/O

- Create tutorial demonstrating JSON I/O usage
- Add orientation schema and loading methods

- Implement orientation data validation and polarity handling

- Add comprehensive test suite for orientation loading

- Fix polarity handling in test data
- Create tutorial demonstrating JSON I/O with horizontal stratigraphic model

- Add sample JSON file with model data

- Include surface points and orientations for two horizontal layers
…id_to_name mapping from surface IDs to names - Update surface points and orientations loading to use actual surface names - Fix issue with surface names not matching between structural frame and series
…odel computation - Update surface points and orientations to match original CSV files - Add model computation step before plotting - Fix plot display using Agg backend
…s work, some problems with fault relations still remaining.
…ns requried; standard stack created by default based on point/ orientation ids.
…lization - Fixed series mapping, fault orientation, and plotting
…r visualization, fix surface points plotting, add colorbar and labels
…ervation, fixed color assignment to structural elements
…r assertions in test_multiple_series_relationships and test_combination_model
@javoha
Copy link
Member

javoha commented Apr 11, 2025

Hi,
looks good to me and tests pass. Just two notes (I also pushed them as TODOs)

  1. In 04_combination_model.py you wrote code for saving the model after computation - the resulting file is exactly the same as the one before. So this is not implemted yet - question would be if we want to do that before merging.
  2. As far as I can tell in the current version the last modification date ist nor actually updated. There is an example for getting the current date in 02_horizontal_stratigraphic.py but I am not sure if that is the final version.

Cheers,
Jan

@flohorovicic
Copy link
Member Author

Thanks a lot for the checks and the comments, @javoha!

  1. Good point - is left from a check I had done before. But it would make sense to include it, again - the functionality itself is now included in a test.
  2. True, but not required - this was mostly to show how it can be set. Maybe more something to describe in a bit more detail in an example.

Anyway, I'd say both are rather minor aspects, can be adjusted after the merge :-)

Copy link
Member

@javoha javoha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I think this feature should be part of the core gempy functionality. In the future we should include the possibility to store results. Thanks @flohorovicic!

@flohorovicic flohorovicic removed the request for review from NilsChudalla April 22, 2025 09:37
@Leguark
Copy link
Member

Leguark commented Apr 24, 2025

gempy/init.py:10: in
from .core import data
gempy/core/data/init.py:1: in
from .geo_model import GeoModel
gempy/core/data/geo_model.py:20: in
from .grid import Grid
gempy/core/data/grid.py:11: in
from .grid_modules import RegularGrid, CustomGrid, Sections
gempy/core/data/grid_modules/init.py:2: in
from .topography import Topography
gempy/core/data/grid_modules/topography.py:7: in
from ....modules.grids.create_topography import _LoadDEMArtificial
gempy/modules/init.py:5: in
from . import json_io
gempy/modules/json_io/init.py:6: in
from .json_operations import JsonIO
gempy/modules/json_io/json_operations.py:11: in
from .schema import SurfacePoint, Orientation, GemPyModelJson, IdNameMapping
gempy/modules/json_io/schema.py:6: in
from typing import TypedDict, List, Dict, Any, Optional, Union, Sequence, NotRequired
E ImportError: cannot import name 'NotRequired' from 'typing' (/usr/lib/python3.10/typing.py)

@flohorovicic
Copy link
Member Author

(Previous fail seems to be related to CI running Python 3.10 - now included an option that should work for 3.10 and >+ 3.11)

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants