Support for chunked file uploads

[benny-adiwijaya]

Hi,

Is it possible to use elFinder chunked file upload feature in this project for handling large file?

[trannamtrung1st]

I noticed this feature before but haven't had time to do it yet. It's very welcome if you could give it a try. The documentation of elFinder client has a section about chunked upload.

[trannamtrung1st]

Hi @benny-adiwijaya,
Large file upload is an interesting topic so I decided to integrate it into the library. It was a bit tough but amazing. I released a new major version 1.3.1 with some changes in the FileController, Index.cshtml files. Please review it or better pull the repository for a working solution.

[benny-adiwijaya]

Hi @trannamtrung1st , your work is amazing.. when upload still progressing and user reload elfinder client, user can see temporary directory for chunk files.. Is it possible to hide those temporary directory from elfinder client?

[trannamtrung1st]

Hi @benny-adiwijaya, I see that too. I'm working on a way to abort the command and clean temp files. Since those temp files would occupy a large portion of the server storage if we failed to clean them, we must have an efficient approach to make sure those files are deleted if not needed anymore.

[trannamtrung1st]

Hi there @benny-adiwijaya,
I just released a new version 1.3.3 There are many changes so I suggest that you pull the repository for a working solution. Now temp files will be outside of the volume directory so it won't show up anymore.
Sorry for the inconsistency. I'm still refactoring the packages gradually. Very happy to help.

[benny-adiwijaya]

Hi @trannamtrung1st thanks for the update, I already pull the latest repository but when I try to upload file more than 15MB in demo project I get error Invalid parameters for command "upload". Do you have any idea why?

[trannamtrung1st]

Hi @benny-adiwijaya,
Can you post the stack trace for more details, please? There is an exception object in the ErrorResponse from the connector which may have some useful information.
Besides, what type of files you are uploading? Are you sure you cloned the source version with this latest commit? Did you make any changes in the demo project?
It's quite strange so I need more detailed information.

[benny-adiwijaya]

Hi @trannamtrung1st,
I get elFinder.Net.Core.Exceptions.CommandParamsException at line 552 in connector.cs
error {string[2]}
[0] "errCmdParams"
[1] "upload"

The file is mp4.. yes, I cloned the latest commit.. the only change I make is QuotaInBytes in program.cs so I can upload more than 10MB

[trannamtrung1st]

Do you have any authorization or filter applied? Can you send me that video if possible so I can make sure that is not a problem with that file?

[trannamtrung1st]

It works totally fine for me. This is a clean source without any modification or preparation.

bandicam.2021-07-26.21-52-04-222.mp4

[benny-adiwijaya]

Hi @trannamtrung1st, when using drag & drop it works but when using upload button its not.. I dont use any authorization or filter applied and I think its not file problem

[trannamtrung1st]

Hi @benny-adiwijaya,
It's weird since they are similar behaviors but submit different arguments :D. I fixed it and released a minor version 1.3.3.1.
Thanks for pointing it out. Cheers.

[benny-adiwijaya]

Hi @trannamtrung1st thank you, this project is superb.. I have a question about performance, do you think the upload speed already fast or it can be optimize?

[trannamtrung1st]

As I see, the elFinder.Net.Demo31 is faster than the advanced project since the implementation of the QuotaManagement plugin relies on some synchronization mechanisms (lock, SemaphoreSlim) which make things slower.
I also realize that running projects without debugging will increase performance significantly since Interceptors make debugging slower I think.
For optimization, we will need more contributions from others to be more objective.
For my needs, the current performance is more than enough.

[benny-adiwijaya]

Allright noted, btw I have a case.. In my project I have some volume with separate drive storage.. I want location temp folder is inside volume path, so the temporary files copied to volume storage instead of server storage.. So, is it possible to create more than one temp folder (one temp folder each volume), and customize TempFileCleanerOptions dynamically?

[trannamtrung1st]

Currently, I don't allow those temp folders to be inside the volume directory since it will be difficult to manage size, and users may modify those folders and cause trouble.

---

For multiple temp folders, it is possible since it is the volume's property. You may want to structure it like below:

• Volume-1
  -- Storage (actual volume storage)
  -- Temp
  -- Thumb
  -- etc
• Volume-n...

You can customize the request/response of the connector to support your custom logic based on the Volume-[n] directory.

---

The TempFileCleanerOptions can be retrieved using IOptionsMonitor and modified directly.
If you have any other complex use cases, I'm afraid that you have to override some default behaviors of the packages.

[benny-adiwijaya]

Okay, lets say I have 3 volumes that saved on the database(Alpha, Beta, Charlie) How to set TempFileCleanerOptions dynamically in Startup.cs? Can I set something like this:

services.AddElFinderAspNetCore()
.AddFileSystemDriver(tempFileCleanerConfig: (opt) =>
{
foreach(var vol in volumesInDB)
{
opt.ScanFolders.Add(MapStoragePath($"~/{vol.path}/.temp"), TempFileCleanerOptions.DefaultUnmanagedLifeTime);
}
})

I'm not sure if I can retrieve data from database in ConfigureServices or it should hardcode like this:

services.AddElFinderAspNetCore()
.AddFileSystemDriver(tempFileCleanerConfig: (opt) =>
{
opt.ScanFolders.Add(MapStoragePath($”/Alpha/.temp"), TempFileCleanerOptions.DefaultUnmanagedLifeTime);
opt.ScanFolders.Add(MapStoragePath($”/Beta/.temp"), TempFileCleanerOptions.DefaultUnmanagedLifeTime);
opt.ScanFolders.Add(MapStoragePath($”~/Charlie/.temp"), TempFileCleanerOptions.DefaultUnmanagedLifeTime);
})

Do you have other solution?

[trannamtrung1st]

You should inject the IOptionsMonitor<TempFileCleanerOptions> into the FilesController.
Then options.CurrentValue.ScanFolders["...somepath..."] = TempFileCleanerOptions.DefaultUnmanagedLifeTime;
Since ITempFileCleaner is a singleton service, you can update a new/current entry in ScanFolders each time a volume is requested.

See Microsoft docs for more detail about using IOptionsMonitor.

[trannamtrung1st]

The purpose of the ITempFileCleaner service is to clean the files after an amount of time since its last access time. So basically you should add all of your temp folders to the ScanFolders property. It can be achieved in many ways.

[benny-adiwijaya]

Okay, I got it thank you.. btw I am using event _driver.OnAfterUpload in FilesController for logging but don't want to log every chunks uploaded, so I changed to _driver.OnAfterChunkMerged. It works but there some issue:

1. File with small size not catched for logging because it didn't need chunkmerged
2. When uploading folder or more than one file at the same time, only one file catched for logging..

I am not using logging interceptor because I think it will make the performance slower. So, is there a better way to log all uploaded file without chunk file in FilesController?

[trannamtrung1st]

OnAfterUpload's event has the IsChunking argument which will help differentiate between chunked upload and normal upload.
Then use OnAfterChunkMerged for chunked upload.
The LoggingInterceptor is for demo purposes only and shouldn't be used in Production.