Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Import grading report #27

Open
lamnguyen8075 opened this issue Aug 5, 2021 · 2 comments
Open

Import grading report #27

lamnguyen8075 opened this issue Aug 5, 2021 · 2 comments
Assignees
@phuocdn
Labels
bug Something isn't working

Comments

@lamnguyen8075
Copy link

lamnguyen8075 commented Aug 5, 2021 •
edited
Loading

Current behavior:

Expected behavior:
@lamnguyen8075 lamnguyen8075 added the bug Something isn't working label Aug 5, 2021
@phuocdn
Copy link

phuocdn commented Sep 29, 2021 •
edited
Loading

Error logs:

2021-09-29 08:29:09,597 INFO 2701 [staff_graded.staff_graded] [user 25994] staff_graded.py:201 - Processing 2276 byte score file aoc.csv for block-v1:STEAMforVietnam+Scratch100_Testing+2020-08+type@staffgradedxblock+block@b9ccdab206f84fbfa43e0aa45e92e62d
2021-09-29 08:29:09,651 ERROR 2701 [lms.djangoapps.courseware.module_render] [user 25994] module_render.py:1221 - error executing xblock handler
Traceback (most recent call last):
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 1196, in _invoke_xblock_handler
    resp = handler_instance.handle(handler, req, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/mixins.py", line 89, in handle
    return self.runtime.handle(self, handler_name, request, suffix)
  File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1453, in handle
    return super(MetricsMixin, self).handle(block, handler_name, request, suffix=suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/runtime.py", line 1063, in handle
    results = handler(request, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/staff_graded/staff_graded.py", line 208, in csv_import_handler
    processor.process_file(score_file, autocommit=True)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/csv_processor.py", line 187, in process_file
    self.preprocess_file(reader)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 165, in preprocess_file
    operation = self.save('error')
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 123, in save
    user=operating_user or get_current_user(),
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/models.py", line 78, in record_operation
    instance.data.save(uuid.uuid4(), ContentFile(data))
  File "/openedx/venv/lib/python3.6/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 596, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing
2021-09-29 08:29:09,656 ERROR 2701 [root] [user None] signals.py:23 - Uncaught exception from None
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/django/views/decorators/clickjacking.py", line 50, in wrapped_view
    resp = view_func(*args, **kwargs)
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 1072, in handle_xblock_callback
    return _invoke_xblock_handler(request, course_id, usage_id, handler, suffix, course=course)
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 1196, in _invoke_xblock_handler
    resp = handler_instance.handle(handler, req, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/mixins.py", line 89, in handle
    return self.runtime.handle(self, handler_name, request, suffix)
  File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1453, in handle
    return super(MetricsMixin, self).handle(block, handler_name, request, suffix=suffix)
  File "/openedx/venv/lib/python3.6/site-packages/xblock/runtime.py", line 1063, in handle
    results = handler(request, suffix)
  File "/openedx/venv/lib/python3.6/site-packages/staff_graded/staff_graded.py", line 208, in csv_import_handler
    processor.process_file(score_file, autocommit=True)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/csv_processor.py", line 187, in process_file
    self.preprocess_file(reader)
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 165, in preprocess_file
    operation = self.save('error')
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/mixins.py", line 123, in save
    user=operating_user or get_current_user(),
  File "/openedx/venv/lib/python3.6/site-packages/super_csv/models.py", line 78, in record_operation
    instance.data.save(uuid.uuid4(), ContentFile(data))
  File "/openedx/venv/lib/python3.6/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.6/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/openedx/venv/lib/python3.6/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/client.py", line 596, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/openedx/venv/lib/python3.6/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

@phuocdn
Copy link

phuocdn commented Oct 1, 2021 •
edited
Loading

Openedx uses staff_graded xblock for processing and storing data to backend storage (filesystem or s3). On production ,we set up to use s3 storage but there is a bug of django-storages when save a file to s3 with a string ContentFile arg.
jschneier/django-storages#708
Because this issue have not solved yet, we have to encode the string arg before save it. We can work around by updating super-csv package (staff_graded xblock use this package to process file).
https://github.com/edx/super-csv/blob/c5a716a71dcbaca1b3dbb8ed492a76bae0fd3267/super_csv/models.py#L72
https://github.com/edx/super-csv/blob/c5a716a71dcbaca1b3dbb8ed492a76bae0fd3267/super_csv/mixins.py#L121

    def record_operation(cls, class_name_or_obj, unique_id, operation, data, original_filename='', user=None):
        """
        Save a CSVOperation
        """
        instance = cls(
            class_name=cls._get_class_name(class_name_or_obj),
            unique_id=unique_id,
            operation=operation,
            original_filename=original_filename,
            user=user,
        )
        instance.data.save(uuid.uuid4(), ContentFile(data))
        return instance

Tested on local:
Imported Scores successfully after config s3 as storage backend & encode the data above on local open edx koa release.
image
Bucket contain data: https://cloud.digitalocean.com/spaces/edxuploads?i=2b4d6e&path=csv%2F

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@phuocdn phuocdn

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@phuocdn @lamnguyen8075