Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Partial pipeline example #152

Closed
wants to merge 11 commits into from
Closed

Partial pipeline example #152

wants to merge 11 commits into from

Conversation

ajeklund
Copy link
Contributor

@ajeklund ajeklund commented Jul 11, 2023 •
edited
Loading

Description:

Example script, showing how to use partial pipelines. Converted from original Jupyter notebook demo version.

Closes issue #151.

Type of change:

  • Bug fix.
  • New feature.
  • Documentation update.

Checklist for the reviewer:

This checklist should be used as a help for the reviewer.

  • Is the change limited to one issue?
  • Does this PR close the issue?
  • Is the code easy to read and understand, including clearly named variables?
  • Do all new feature have an accompanying new test?
  • Has the documentation been updated as necessary?

@ajeklund ajeklund linked an issue Jul 11, 2023 that may be closed by this pull request
Add partial pipeline example #151
Open
@ajeklund ajeklund marked this pull request as draft July 11, 2023 09:46
@ajeklund
Copy link
Contributor Author

ajeklund commented Jul 12, 2023 •
edited
Loading

Printout of the 'siemens' error from the oteapi-services container, likely caused by the string 'S-3400' being passed to a pint.Quantity in

dlite/mappings.py", line 720, in instance_routes:

oteapi_1  | [2023-07-12 14:59:00 +0000] [49] [ERROR] Error in ASGI Framework
oteapi_1  | Traceback (most recent call last):
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/hypercorn/asyncio/task_group.py", line 23, in _handle
oteapi_1  |     await app(scope, receive, send, sync_spawn, call_soon)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/hypercorn/app_wrappers.py", line 33, in __call__
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 282, in __call__
oteapi_1  |     await super().__call__(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
oteapi_1  |     await self.middleware_stack(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
oteapi_1  |     raise exc
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
oteapi_1  |     await self.app(scope, receive, _send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
oteapi_1  |     raise exc
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
oteapi_1  |     await self.app(scope, receive, sender)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
oteapi_1  |     raise e
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
oteapi_1  |     await route.handle(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
oteapi_1  |     response = await func(request)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 241, in app
oteapi_1  |     raw_response = await run_endpoint_function(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 167, in run_endpoint_function
oteapi_1  |     return await dependant.call(**values)
oteapi_1  |   File "/app/app/routers/function.py", line 83, in get_function
oteapi_1  |     session_update = function_strategy.get(session=session_data)
oteapi_1  |   File "/oteapi-dlite/oteapi_dlite/strategies/interoperability.py", line 121, in get
oteapi_1  |     print(list(instances))
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/dlite.py", line 2194, in get_instances
oteapi_1  |     for inst in instantiate_all(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/mappings.py", line 878, in instantiate_all
oteapi_1  |     routes = instance_routes(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/mappings.py", line 720, in instance_routes
oteapi_1  |     sources[f'{inst.meta.uri}#{key}'] = quantity(value, props[key].unit)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 200, in __new__
oteapi_1  |     inst = ureg.parse_expression(value)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/registry.py", line 1309, in parse_expression
oteapi_1  |     return build_eval_tree(gen).evaluate(_define_op)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/pint_eval.py", line 146, in evaluate
oteapi_1  |     return bin_op[op_text](
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 840, in __sub__
oteapi_1  |     return self._add_sub(other, operator.sub)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 100, in wrapped
oteapi_1  |     return f(self, *args, **kwargs)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 727, in _add_sub
oteapi_1  |     raise DimensionalityError(self._units, "dimensionless")
oteapi_1  | pint.errors.DimensionalityError: Cannot convert from 'siemens' to 'dimensionless'

@ajeklund
Copy link
Contributor Author

The 'siemens' error gives an ApiError in the Jupyter notebook:

---------------------------------------------------------------------------
ApiError                                  Traceback (most recent call last)
Cell In[6], line 6
      3 pipeline = cansMprDataParser>>cansMprDataMappings>>generatorMappings>>generatorFilter>>createCollection
      5 # Execute the pipeline
----> 6 pipeline.get()

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/strategies.py:94, in AbstractBaseStrategy.get(self, session_id)
     92 self.initialize(session_id)
     93 if self.input_pipe:
---> 94     self.input_pipe.get(session_id)
     95 return self.fetch(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/pipe.py:18, in Pipe.get(self, session_id)
     16 def get(self, session_id: "Optional[str]" = None) -> bytes:
     17     """Call the input strategy's `get()` method."""
---> 18     return self.input.get(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/strategies.py:95, in AbstractBaseStrategy.get(self, session_id)
     93 if self.input_pipe:
     94     self.input_pipe.get(session_id)
---> 95 return self.fetch(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/services/base.py:89, in BaseServicesStrategy.fetch(self, session_id)
     83     return response.content
     84 strategy_name = (
     85     self.strategy_name[len("data") :]
     86     if self.strategy_name.startswith("data")
     87     else self.strategy_name
     88 )
---> 89 raise ApiError(
     90     f"Cannot fetch {self.strategy_name}: session_id={session_id!r} "
     91     f"{strategy_name}_id={self.strategy_id!r}"
     92     f"{' content=' + str(response.content) if self.debug else ''}",
     93     status=response.status_code,
     94 )

ApiError: ApiError: status=500 Cannot fetch function: session_id='session-c9a38462-a1b4-4ef3-b8d6-7688566008fe' function_id='function-a7402487-1f30-4e87-b4b0-109a668da7b7'

@ajeklund
Copy link
Contributor Author

This PR currently has its own oteapi-dlite sub-directory with code changes that probably should be added to the repository. These code changes are handled separately in #156. This PR (i.e., the example) is thus blocked by #156.

jesper-friis and others added 4 commits July 13, 2023 18:54
@jesper-friis
Simple formatting cleanup
1cbc79d
@jesper-friis
Simple formatting cleanup
b2cc13f
@CasperWA
Clean up of example
4612dd7 
Remove cached files from folders.
Add to .gitignore.

Make the partial-pipelines demo an installable package based around
"oteutils".
Move ote strategies under oteutils as demo-specific strategies.
@CasperWA
Mention the use of the mapping ttl file instead of
3c4e0f2 
hardcoding
@codecov Codecov
Copy link

codecov bot commented Feb 26, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.80%. Comparing base (52533ad) to head (3c4e0f2).
Report is 90 commits behind head on master.

Additional details and impacted files 
@@             Coverage Diff             @@
##           master     #152       +/-   ##
===========================================
+ Coverage   68.73%   89.80%   +21.06%     
===========================================
  Files          15       15               
  Lines         403      451       +48     
===========================================
+ Hits          277      405      +128     
+ Misses        126       46       -80
Flag Coverage Δ
linux 89.80% <ø> (+21.06%) ⬆️
windows 89.80% <ø> (+21.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@CasperWA CasperWA closed this Feb 26, 2024
@CasperWA CasperWA deleted the 151-partial-pipeline-example branch February 26, 2024 14:19
@ajeklund
Copy link
Contributor Author

The 'siemens' error arises because the data value is S-3400 and when a pint.Quantity is created with this as input, pint (by design) interprets this value as the mathematical expression '1 siemens - 3400' which results in a DimensionalityError since the two terms indeed have different dimensionalities.

Code for reproducing the behavior:
from pint import Quantity
q = Quantity('S-3400')
or q = Quantity('S-3400', None)

But this "works": q = Quantity('S-3400S') (it results in <Quantity(-3399, 'siemens')>)

# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add partial pipeline example
3 participants
@ajeklund @CasperWA @jesper-friis