From e088d09e4705a4348444aaa8ec1b49a07128e063 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 13 Oct 2024 14:18:23 +0100 Subject: [PATCH 01/41] ZIP Export: Started defining format --- dev/docs/portable-zip-file-format.md | 82 ++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 dev/docs/portable-zip-file-format.md diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md new file mode 100644 index 00000000000..260735c58cf --- /dev/null +++ b/dev/docs/portable-zip-file-format.md @@ -0,0 +1,82 @@ +# Portable ZIP File Format + +BookStack provides exports in a "Portable ZIP" which allows the portable transfer, storage, import & export of BookStack content. +This document details the format used, and is intended for our own internal development use in addition to detailing the format for potential external use-cases (readers, apps, import for other platforms etc...). + +**Note:** This is not a BookStack backup format! This format misses much of the data that would be needed to re-create/restore a BookStack instance. There are existing better alternative options for this use-case. + +## Stability + +Following the goals & ideals of BookStack, stability is very important. We aim for this defined format to be stable and forwards compatible, to prevent breakages in use-case due to changes. Here are the general rules we follow in regard to stability & changes: + +- New features & properties may be added with any release. +- Where reasonably possible, we will attempt to avoid modifications/removals of existing features/properties. +- Where potentially breaking changes do have to be made, these will be noted in BookStack release/update notes. + +The addition of new features/properties alone are not considered as a breaking change to the format. Breaking changes are considered as such where they could impact common/expected use of the existing properties and features we document, they are not considered based upon user assumptions or any possible breakage. For example if your application, using the format, breaks because we added a new property while you hard-coded your application to use the third property (instead of a property name), then that's on you. + +## Format Outline + +The format is intended to be very simple, readable and based on open standards that could be easily read/handled in most common programming languages. +The below outlines the structure of the format: + +- **ZIP archive container** + - **data.json** - Application data. + - **files/** - Directory containing referenced files. + - *...file.ext* + +## References + +TODO - Define how we reference across content: +TODO - References to files from data.json +TODO - References from in-content to file URLs +TODO - References from in-content to in-export content (page cross links within same export). + +## Application Data - `data.json` + +The `data.json` file is a JSON format file which contains all structured data for the export. The properties are as follows: + +- `instance` - [Instance](#instance) Object, optional, details of the export source instance. +- `exported_at` - String, optional, full ISO 8601 datetime of when the export was created. +- `book` - [Book](#book) Object, optional, book export data. +- `chapter` - [Chapter](#chapter) Object, optional, chapter export data. +- `page` - [Page](#page) Object, optional, page export data. + +Either `book`, `chapter` or `page` will exist depending on export type. You'd want to check for each to check what kind of export this is, and if it's an export you can handle. It's possible that other options are added in the future (`books` for a range of books for example) so it'd be wise to specifically check for properties that can be handled, otherwise error to indicate lack of support. + +## Data Objects + +The below details the objects & their properties used in Application Data. + +#### Instance + +These details are mainly informational regarding the exporting BookStack instance from where an export was created from. + +- `version` - String, required, BookStack version of the export source instance. +- `id_ciphertext` - String, required, identifier for the BookStack instance. + +The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This is used as a simple & rough way for a BookStack instance to be able to identify if they were the source (by attempting to decrypt the ciphertext). + +#### Book + +TODO + +#### Chapter + +TODO + +#### Page + +TODO + +#### Image + +TODO + +#### Attachment + +TODO + +#### Tag + +TODO \ No newline at end of file From 1930af91cea9ab830562933f738b1c9d151b0640 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 13 Oct 2024 22:56:22 +0100 Subject: [PATCH 02/41] ZIP Export: Started types in format doc --- dev/docs/portable-zip-file-format.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 260735c58cf..c4737309f48 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -59,11 +59,22 @@ The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This i #### Book -TODO +- `id` - Number, optional, original ID for the book from exported system. +- `name` - String, required, name/title of the book. +- `description_html` - String, optional, HTML description content. +- `chapters` - [Chapter](#chapter) array, optional, chapters within this book. +- `pages` - [Page](#page) array, optional, direct child pages for this book. +- `tags` - [Tag](#tag) array, optional, tags assigned to this book. + +The `pages` are not all pages within the book, just those that are direct children (not in a chapter). To build an ordered mixed chapter/page list for the book, as what you'd see in BookStack, you'd need to combine `chapters` and `pages` together and sort by their `priority` value (low to high). #### Chapter -TODO +- `id` - Number, optional, original ID for the chapter from exported system. +- `name` - String, required, name/title of the chapter. +- `description_html` - String, optional, HTML description content. +- `pages` - [Page](#page) array, optional, pages within this chapter. +- `tags` - [Tag](#tag) array, optional, tags assigned to this chapter. #### Page @@ -79,4 +90,6 @@ TODO #### Tag -TODO \ No newline at end of file +- `name` - String, required, name of the tag. +- `value` - String, optional, value of the tag (can be empty). +- `order` - Number, optional, integer order for the tags (shown low to high). \ No newline at end of file From 42bd07d73325ed468bae2658ba130314f6fe4a21 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 15 Oct 2024 13:57:16 +0100 Subject: [PATCH 03/41] ZIP Export: Continued expanding format doc types --- dev/docs/portable-zip-file-format.md | 57 ++++++++++++++++++++++++---- 1 file changed, 49 insertions(+), 8 deletions(-) diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index c4737309f48..dc21bf8e58e 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -21,18 +21,38 @@ The format is intended to be very simple, readable and based on open standards t The below outlines the structure of the format: - **ZIP archive container** - - **data.json** - Application data. + - **data.json** - Export data. - **files/** - Directory containing referenced files. - - *...file.ext* + - *file-a* + - *file-b* + - *...* ## References +Some properties in the export data JSON are indicated as `String reference`, and these are direct references to a file name within the `files/` directory of the ZIP. For example, the below book cover is directly referencing a `files/4a5m4a.jpg` within the ZIP which would be expected to exist. + +```json +{ + "book": { + "cover": "4a5m4a.jpg" + } +} +``` + +TODO - Jotting out idea below. +Would need to validate image/attachment paths against image/attachments listed across all pages in export. +Probably good to ensure filenames are ascii-alpha-num. +`[[bsexport:image:an-image-path.png]]` +`[[bsexport:attachment:an-image-path.png]]` +`[[bsexport:page:1]]` +`[[bsexport:chapter:2]]` +`[[bsexport:book:3]]` + TODO - Define how we reference across content: -TODO - References to files from data.json TODO - References from in-content to file URLs TODO - References from in-content to in-export content (page cross links within same export). -## Application Data - `data.json` +## Export Data - `data.json` The `data.json` file is a JSON format file which contains all structured data for the export. The properties are as follows: @@ -62,6 +82,7 @@ The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This i - `id` - Number, optional, original ID for the book from exported system. - `name` - String, required, name/title of the book. - `description_html` - String, optional, HTML description content. +- `cover` - String reference, options, reference to book cover image. - `chapters` - [Chapter](#chapter) array, optional, chapters within this book. - `pages` - [Page](#page) array, optional, direct child pages for this book. - `tags` - [Tag](#tag) array, optional, tags assigned to this book. @@ -73,23 +94,43 @@ The `pages` are not all pages within the book, just those that are direct childr - `id` - Number, optional, original ID for the chapter from exported system. - `name` - String, required, name/title of the chapter. - `description_html` - String, optional, HTML description content. +- `priority` - Number, optional, integer order for when shown within a book (shown low to high). - `pages` - [Page](#page) array, optional, pages within this chapter. - `tags` - [Tag](#tag) array, optional, tags assigned to this chapter. #### Page -TODO +- `id` - Number, optional, original ID for the page from exported system. +- `name` - String, required, name/title of the page. +- `html` - String, optional, page HTML content. +- `markdown` - String, optional, user markdown content for this page. +- `priority` - Number, optional, integer order for when shown within a book (shown low to high). +- `attachments` - [Attachment](#attachment) array, optional, attachments uploaded to this page. +- `images` - [Image](#image) array, optional, images used in this page. +- `tags` - [Tag](#tag) array, optional, tags assigned to this page. + +To define the page content, either `markdown` or `html` should be provided. Ideally these should be limited to the range of markdown and HTML which BookStack supports. + +The page editor type, and edit content will be determined by what content is provided. If non-empty `markdown` is provided, the page will be assumed as a markdown editor page (where permissions allow) and the HTML will be rendered from the markdown content. Otherwise, the provided `html` will be used as editor and display content. #### Image -TODO +- `name` - String, required, name of image. +- `file` - String reference, required, reference to image file. + +File must be an image type accepted by BookStack (png, jpg, gif, webp) #### Attachment -TODO +- `name` - String, required, name of attachment. +- `link` - String, semi-optional, URL of attachment. +- `file` - String reference, semi-optional, reference to attachment file. +- `order` - Number, optional, integer order of the attachments (shown low to high). + +Either `link` or `file` must be present, as that will determine the type of attachment. #### Tag - `name` - String, required, name of the tag. - `value` - String, optional, value of the tag (can be empty). -- `order` - Number, optional, integer order for the tags (shown low to high). \ No newline at end of file +- `order` - Number, optional, integer order of the tags (shown low to high). \ No newline at end of file From 42b9700673e7b2e5a04c9f888a05d98261ed36e3 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 15 Oct 2024 16:14:11 +0100 Subject: [PATCH 04/41] ZIP Exports: Finished up format doc, move files, started builder Moved all existing export related app files into their new own dir. --- app/Exceptions/ZipExportException.php | 7 +++ .../Controllers/BookExportApiController.php | 4 +- .../Controllers/BookExportController.php | 4 +- .../ChapterExportApiController.php | 4 +- .../Controllers/ChapterExportController.php | 4 +- .../Controllers/PageExportApiController.php | 4 +- .../Controllers/PageExportController.php | 4 +- .../Tools => Exports}/ExportFormatter.php | 4 +- .../Tools => Exports}/PdfGenerator.php | 4 +- app/Exports/ZipExportBuilder.php | 48 +++++++++++++++++++ composer.json | 1 + dev/docs/portable-zip-file-format.md | 30 +++++++----- routes/api.php | 26 +++++----- routes/web.php | 27 ++++++----- tests/Entity/ExportTest.php | 2 +- 15 files changed, 119 insertions(+), 54 deletions(-) create mode 100644 app/Exceptions/ZipExportException.php rename app/{Entities => Exports}/Controllers/BookExportApiController.php (95%) rename app/{Entities => Exports}/Controllers/BookExportController.php (95%) rename app/{Entities => Exports}/Controllers/ChapterExportApiController.php (95%) rename app/{Entities => Exports}/Controllers/ChapterExportController.php (96%) rename app/{Entities => Exports}/Controllers/PageExportApiController.php (95%) rename app/{Entities => Exports}/Controllers/PageExportController.php (96%) rename app/{Entities/Tools => Exports}/ExportFormatter.php (98%) rename app/{Entities/Tools => Exports}/PdfGenerator.php (99%) create mode 100644 app/Exports/ZipExportBuilder.php diff --git a/app/Exceptions/ZipExportException.php b/app/Exceptions/ZipExportException.php new file mode 100644 index 00000000000..b2c811e0b33 --- /dev/null +++ b/app/Exceptions/ZipExportException.php @@ -0,0 +1,7 @@ +data['page'] = [ + 'id' => $page->id, + ]; + + return $this->build(); + } + + /** + * @throws ZipExportException + */ + protected function build(): string + { + $this->data['exported_at'] = date(DATE_ATOM); + $this->data['instance'] = [ + 'version' => trim(file_get_contents(base_path('version'))), + 'id_ciphertext' => encrypt('bookstack'), + ]; + + $zipFile = tempnam(sys_get_temp_dir(), 'bszip-'); + $zip = new ZipArchive(); + $opened = $zip->open($zipFile, ZipArchive::CREATE); + if ($opened !== true) { + throw new ZipExportException('Failed to create zip file for export.'); + } + + $zip->addFromString('data.json', json_encode($this->data)); + $zip->addEmptyDir('files'); + + return $zipFile; + } +} diff --git a/composer.json b/composer.json index 5c54774f1e2..3680a2c6aa2 100644 --- a/composer.json +++ b/composer.json @@ -16,6 +16,7 @@ "ext-json": "*", "ext-mbstring": "*", "ext-xml": "*", + "ext-zip": "*", "bacon/bacon-qr-code": "^3.0", "doctrine/dbal": "^3.5", "dompdf/dompdf": "^3.0", diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index dc21bf8e58e..d5635bd398d 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -39,18 +39,24 @@ Some properties in the export data JSON are indicated as `String reference`, and } ``` -TODO - Jotting out idea below. -Would need to validate image/attachment paths against image/attachments listed across all pages in export. -Probably good to ensure filenames are ascii-alpha-num. -`[[bsexport:image:an-image-path.png]]` -`[[bsexport:attachment:an-image-path.png]]` -`[[bsexport:page:1]]` -`[[bsexport:chapter:2]]` -`[[bsexport:book:3]]` - -TODO - Define how we reference across content: -TODO - References from in-content to file URLs -TODO - References from in-content to in-export content (page cross links within same export). +Within HTML and markdown content, you may require references across to other items within the export content. +This can be done using the following format: + +``` +[[bsexport::]] +``` + +Images and attachments are referenced via their file name within the `files/` directory. +Otherwise, other content types are referenced by `id`. +Here's an example of each type of such reference that could be used: + +``` +[[bsexport:image:an-image-path.png]] +[[bsexport:attachment:an-image-path.png]] +[[bsexport:page:40]] +[[bsexport:chapter:2]] +[[bsexport:book:8]] +``` ## Export Data - `data.json` diff --git a/routes/api.php b/routes/api.php index c0919d3247b..71036485597 100644 --- a/routes/api.php +++ b/routes/api.php @@ -9,6 +9,7 @@ use BookStack\Activity\Controllers\AuditLogApiController; use BookStack\Api\ApiDocsController; use BookStack\Entities\Controllers as EntityControllers; +use BookStack\Exports\Controllers as ExportControllers; use BookStack\Permissions\ContentPermissionApiController; use BookStack\Search\SearchApiController; use BookStack\Uploads\Controllers\AttachmentApiController; @@ -31,21 +32,20 @@ Route::put('books/{id}', [EntityControllers\BookApiController::class, 'update']); Route::delete('books/{id}', [EntityControllers\BookApiController::class, 'delete']); -Route::get('books/{id}/export/html', [EntityControllers\BookExportApiController::class, 'exportHtml']); -Route::get('books/{id}/export/pdf', [EntityControllers\BookExportApiController::class, 'exportPdf']); -Route::get('books/{id}/export/plaintext', [EntityControllers\BookExportApiController::class, 'exportPlainText']); -Route::get('books/{id}/export/markdown', [EntityControllers\BookExportApiController::class, 'exportMarkdown']); +Route::get('books/{id}/export/html', [ExportControllers\BookExportApiController::class, 'exportHtml']); +Route::get('books/{id}/export/pdf', [ExportControllers\BookExportApiController::class, 'exportPdf']); +Route::get('books/{id}/export/plaintext', [ExportControllers\BookExportApiController::class, 'exportPlainText']); +Route::get('books/{id}/export/markdown', [ExportControllers\BookExportApiController::class, 'exportMarkdown']); Route::get('chapters', [EntityControllers\ChapterApiController::class, 'list']); Route::post('chapters', [EntityControllers\ChapterApiController::class, 'create']); Route::get('chapters/{id}', [EntityControllers\ChapterApiController::class, 'read']); Route::put('chapters/{id}', [EntityControllers\ChapterApiController::class, 'update']); Route::delete('chapters/{id}', [EntityControllers\ChapterApiController::class, 'delete']); - -Route::get('chapters/{id}/export/html', [EntityControllers\ChapterExportApiController::class, 'exportHtml']); -Route::get('chapters/{id}/export/pdf', [EntityControllers\ChapterExportApiController::class, 'exportPdf']); -Route::get('chapters/{id}/export/plaintext', [EntityControllers\ChapterExportApiController::class, 'exportPlainText']); -Route::get('chapters/{id}/export/markdown', [EntityControllers\ChapterExportApiController::class, 'exportMarkdown']); +Route::get('chapters/{id}/export/html', [ExportControllers\ChapterExportApiController::class, 'exportHtml']); +Route::get('chapters/{id}/export/pdf', [ExportControllers\ChapterExportApiController::class, 'exportPdf']); +Route::get('chapters/{id}/export/plaintext', [ExportControllers\ChapterExportApiController::class, 'exportPlainText']); +Route::get('chapters/{id}/export/markdown', [ExportControllers\ChapterExportApiController::class, 'exportMarkdown']); Route::get('pages', [EntityControllers\PageApiController::class, 'list']); Route::post('pages', [EntityControllers\PageApiController::class, 'create']); @@ -53,10 +53,10 @@ Route::put('pages/{id}', [EntityControllers\PageApiController::class, 'update']); Route::delete('pages/{id}', [EntityControllers\PageApiController::class, 'delete']); -Route::get('pages/{id}/export/html', [EntityControllers\PageExportApiController::class, 'exportHtml']); -Route::get('pages/{id}/export/pdf', [EntityControllers\PageExportApiController::class, 'exportPdf']); -Route::get('pages/{id}/export/plaintext', [EntityControllers\PageExportApiController::class, 'exportPlainText']); -Route::get('pages/{id}/export/markdown', [EntityControllers\PageExportApiController::class, 'exportMarkdown']); +Route::get('pages/{id}/export/html', [ExportControllers\PageExportApiController::class, 'exportHtml']); +Route::get('pages/{id}/export/pdf', [ExportControllers\PageExportApiController::class, 'exportPdf']); +Route::get('pages/{id}/export/plaintext', [ExportControllers\PageExportApiController::class, 'exportPlainText']); +Route::get('pages/{id}/export/markdown', [ExportControllers\PageExportApiController::class, 'exportMarkdown']); Route::get('image-gallery', [ImageGalleryApiController::class, 'list']); Route::post('image-gallery', [ImageGalleryApiController::class, 'create']); diff --git a/routes/web.php b/routes/web.php index 81b938f32ec..5220684c0b0 100644 --- a/routes/web.php +++ b/routes/web.php @@ -7,6 +7,7 @@ use BookStack\App\HomeController; use BookStack\App\MetaController; use BookStack\Entities\Controllers as EntityControllers; +use BookStack\Exports\Controllers as ExportControllers; use BookStack\Http\Middleware\VerifyCsrfToken; use BookStack\Permissions\PermissionsController; use BookStack\References\ReferenceController; @@ -74,11 +75,11 @@ Route::get('/books/{bookSlug}/sort', [EntityControllers\BookSortController::class, 'show']); Route::put('/books/{bookSlug}/sort', [EntityControllers\BookSortController::class, 'update']); Route::get('/books/{slug}/references', [ReferenceController::class, 'book']); - Route::get('/books/{bookSlug}/export/html', [EntityControllers\BookExportController::class, 'html']); - Route::get('/books/{bookSlug}/export/pdf', [EntityControllers\BookExportController::class, 'pdf']); - Route::get('/books/{bookSlug}/export/markdown', [EntityControllers\BookExportController::class, 'markdown']); - Route::get('/books/{bookSlug}/export/zip', [EntityControllers\BookExportController::class, 'zip']); - Route::get('/books/{bookSlug}/export/plaintext', [EntityControllers\BookExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/export/html', [ExportControllers\BookExportController::class, 'html']); + Route::get('/books/{bookSlug}/export/pdf', [ExportControllers\BookExportController::class, 'pdf']); + Route::get('/books/{bookSlug}/export/markdown', [ExportControllers\BookExportController::class, 'markdown']); + Route::get('/books/{bookSlug}/export/zip', [ExportControllers\BookExportController::class, 'zip']); + Route::get('/books/{bookSlug}/export/plaintext', [ExportControllers\BookExportController::class, 'plainText']); // Pages Route::get('/books/{bookSlug}/create-page', [EntityControllers\PageController::class, 'create']); @@ -86,10 +87,10 @@ Route::get('/books/{bookSlug}/draft/{pageId}', [EntityControllers\PageController::class, 'editDraft']); Route::post('/books/{bookSlug}/draft/{pageId}', [EntityControllers\PageController::class, 'store']); Route::get('/books/{bookSlug}/page/{pageSlug}', [EntityControllers\PageController::class, 'show']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/pdf', [EntityControllers\PageExportController::class, 'pdf']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/html', [EntityControllers\PageExportController::class, 'html']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/markdown', [EntityControllers\PageExportController::class, 'markdown']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/plaintext', [EntityControllers\PageExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/pdf', [ExportControllers\PageExportController::class, 'pdf']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/html', [ExportControllers\PageExportController::class, 'html']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/markdown', [ExportControllers\PageExportController::class, 'markdown']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/plaintext', [ExportControllers\PageExportController::class, 'plainText']); Route::get('/books/{bookSlug}/page/{pageSlug}/edit', [EntityControllers\PageController::class, 'edit']); Route::get('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'showMove']); Route::put('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'move']); @@ -126,10 +127,10 @@ Route::get('/books/{bookSlug}/chapter/{chapterSlug}/edit', [EntityControllers\ChapterController::class, 'edit']); Route::post('/books/{bookSlug}/chapter/{chapterSlug}/convert-to-book', [EntityControllers\ChapterController::class, 'convertToBook']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/permissions', [PermissionsController::class, 'showForChapter']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/pdf', [EntityControllers\ChapterExportController::class, 'pdf']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/html', [EntityControllers\ChapterExportController::class, 'html']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/markdown', [EntityControllers\ChapterExportController::class, 'markdown']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/plaintext', [EntityControllers\ChapterExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/pdf', [ExportControllers\ChapterExportController::class, 'pdf']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/html', [ExportControllers\ChapterExportController::class, 'html']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/markdown', [ExportControllers\ChapterExportController::class, 'markdown']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/plaintext', [ExportControllers\ChapterExportController::class, 'plainText']); Route::put('/books/{bookSlug}/chapter/{chapterSlug}/permissions', [PermissionsController::class, 'updateForChapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/references', [ReferenceController::class, 'chapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/delete', [EntityControllers\ChapterController::class, 'showDelete']); diff --git a/tests/Entity/ExportTest.php b/tests/Entity/ExportTest.php index 7aafa3b7927..11cfddb206e 100644 --- a/tests/Entity/ExportTest.php +++ b/tests/Entity/ExportTest.php @@ -5,8 +5,8 @@ use BookStack\Entities\Models\Book; use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; -use BookStack\Entities\Tools\PdfGenerator; use BookStack\Exceptions\PdfExportException; +use BookStack\Exports\PdfGenerator; use Illuminate\Support\Facades\Storage; use Tests\TestCase; From bf0262d7d178b494e256ff5164a8d75195cdc231 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 19 Oct 2024 13:59:42 +0100 Subject: [PATCH 05/41] Testing: Split export tests into multiple files --- tests/Entity/ExportTest.php | 569 --------------------------- tests/Exports/ExportUiTest.php | 33 ++ tests/Exports/HtmlExportTest.php | 253 ++++++++++++ tests/Exports/MarkdownExportTest.php | 85 ++++ tests/Exports/PdfExportTest.php | 146 +++++++ tests/Exports/TextExportTest.php | 88 +++++ 6 files changed, 605 insertions(+), 569 deletions(-) delete mode 100644 tests/Entity/ExportTest.php create mode 100644 tests/Exports/ExportUiTest.php create mode 100644 tests/Exports/HtmlExportTest.php create mode 100644 tests/Exports/MarkdownExportTest.php create mode 100644 tests/Exports/PdfExportTest.php create mode 100644 tests/Exports/TextExportTest.php diff --git a/tests/Entity/ExportTest.php b/tests/Entity/ExportTest.php deleted file mode 100644 index 11cfddb206e..00000000000 --- a/tests/Entity/ExportTest.php +++ /dev/null @@ -1,569 +0,0 @@ -entities->page(); - $this->asEditor(); - - $resp = $this->get($page->getUrl('/export/plaintext')); - $resp->assertStatus(200); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.txt"'); - } - - public function test_page_pdf_export() - { - $page = $this->entities->page(); - $this->asEditor(); - - $resp = $this->get($page->getUrl('/export/pdf')); - $resp->assertStatus(200); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.pdf"'); - } - - public function test_page_html_export() - { - $page = $this->entities->page(); - $this->asEditor(); - - $resp = $this->get($page->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.html"'); - } - - public function test_book_text_export() - { - $book = $this->entities->bookHasChaptersAndPages(); - $directPage = $book->directPages()->first(); - $chapter = $book->chapters()->first(); - $chapterPage = $chapter->pages()->first(); - $this->entities->updatePage($directPage, ['html' => '

My awesome page

']); - $this->entities->updatePage($chapterPage, ['html' => '

My little nested page

']); - $this->asEditor(); - - $resp = $this->get($book->getUrl('/export/plaintext')); - $resp->assertStatus(200); - $resp->assertSee($book->name); - $resp->assertSee($chapterPage->name); - $resp->assertSee($chapter->name); - $resp->assertSee($directPage->name); - $resp->assertSee('My awesome page'); - $resp->assertSee('My little nested page'); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.txt"'); - } - - public function test_book_text_export_format() - { - $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); - $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); - $entities['chapter']->name = 'Export chapter'; - $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; - $entities['book']->name = 'Export Book'; - $entities['book']->description = "This is a book with stuff to export"; - $entities['chapter']->save(); - $entities['book']->save(); - - $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); - - $expected = "Export Book\nThis is a book with stuff to export\n\nExport chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; - $expected .= "My wonderful page!\nMy great page Full of great stuff"; - $resp->assertSee($expected); - } - - public function test_book_pdf_export() - { - $page = $this->entities->page(); - $book = $page->book; - $this->asEditor(); - - $resp = $this->get($book->getUrl('/export/pdf')); - $resp->assertStatus(200); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.pdf"'); - } - - public function test_book_html_export() - { - $page = $this->entities->page(); - $book = $page->book; - $this->asEditor(); - - $resp = $this->get($book->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($book->name); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.html"'); - } - - public function test_book_html_export_shows_html_descriptions() - { - $book = $this->entities->bookHasChaptersAndPages(); - $chapter = $book->chapters()->first(); - $book->description_html = '

A description with HTML within!

'; - $chapter->description_html = '

A chapter description with HTML within!

'; - $book->save(); - $chapter->save(); - - $resp = $this->asEditor()->get($book->getUrl('/export/html')); - $resp->assertSee($book->description_html, false); - $resp->assertSee($chapter->description_html, false); - } - - public function test_chapter_text_export() - { - $chapter = $this->entities->chapter(); - $page = $chapter->pages[0]; - $this->entities->updatePage($page, ['html' => '

This is content within the page!

']); - $this->asEditor(); - - $resp = $this->get($chapter->getUrl('/export/plaintext')); - $resp->assertStatus(200); - $resp->assertSee($chapter->name); - $resp->assertSee($page->name); - $resp->assertSee('This is content within the page!'); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.txt"'); - } - - public function test_chapter_text_export_format() - { - $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); - $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); - $entities['chapter']->name = 'Export chapter'; - $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; - $entities['chapter']->save(); - - $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); - - $expected = "Export chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; - $expected .= "My wonderful page!\nMy great page Full of great stuff"; - $resp->assertSee($expected); - } - - public function test_chapter_pdf_export() - { - $chapter = $this->entities->chapter(); - $this->asEditor(); - - $resp = $this->get($chapter->getUrl('/export/pdf')); - $resp->assertStatus(200); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.pdf"'); - } - - public function test_chapter_html_export() - { - $chapter = $this->entities->chapter(); - $page = $chapter->pages[0]; - $this->asEditor(); - - $resp = $this->get($chapter->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($chapter->name); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.html"'); - } - - public function test_chapter_html_export_shows_html_descriptions() - { - $chapter = $this->entities->chapter(); - $chapter->description_html = '

A description with HTML within!

'; - $chapter->save(); - - $resp = $this->asEditor()->get($chapter->getUrl('/export/html')); - $resp->assertSee($chapter->description_html, false); - } - - public function test_page_html_export_contains_custom_head_if_set() - { - $page = $this->entities->page(); - - $customHeadContent = ''; - $this->setSettings(['app-custom-head' => $customHeadContent]); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertSee($customHeadContent, false); - } - - public function test_page_html_export_does_not_break_with_only_comments_in_custom_head() - { - $page = $this->entities->page(); - - $customHeadContent = ''; - $this->setSettings(['app-custom-head' => $customHeadContent]); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($customHeadContent, false); - } - - public function test_page_html_export_use_absolute_dates() - { - $page = $this->entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertSee($page->created_at->isoFormat('D MMMM Y HH:mm:ss')); - $resp->assertDontSee($page->created_at->diffForHumans()); - $resp->assertSee($page->updated_at->isoFormat('D MMMM Y HH:mm:ss')); - $resp->assertDontSee($page->updated_at->diffForHumans()); - } - - public function test_page_export_does_not_include_user_or_revision_links() - { - $page = $this->entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertDontSee($page->getUrl('/revisions')); - $resp->assertDontSee($page->createdBy->getProfileUrl()); - $resp->assertSee($page->createdBy->name); - } - - public function test_page_export_sets_right_data_type_for_svg_embeds() - { - $page = $this->entities->page(); - Storage::disk('local')->makeDirectory('uploads/images/gallery'); - Storage::disk('local')->put('uploads/images/gallery/svg_test.svg', ''); - $page->html = ''; - $page->save(); - - $this->asEditor(); - $resp = $this->get($page->getUrl('/export/html')); - Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); - - $resp->assertStatus(200); - $resp->assertSee(''; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); - Storage::disk('local')->delete('uploads/images/gallery/svg_test2.svg'); - - $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test'); - } - - public function test_page_export_contained_html_image_fetches_only_run_when_url_points_to_image_upload_folder() - { - $page = $this->entities->page(); - $page->html = '' - . '' - . ''; - $storageDisk = Storage::disk('local'); - $storageDisk->makeDirectory('uploads/images/gallery'); - $storageDisk->put('uploads/images/gallery/svg_test.svg', 'good'); - $storageDisk->put('uploads/svg_test.svg', 'bad'); - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - - $storageDisk->delete('uploads/images/gallery/svg_test.svg'); - $storageDisk->delete('uploads/svg_test.svg'); - - $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test.svg', false); - $resp->assertSee('http://localhost/uploads/svg_test.svg'); - $resp->assertSee('src="/uploads/svg_test.svg"', false); - } - - public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local() - { - $contents = file_get_contents(public_path('.htaccess')); - config()->set('filesystems.images', 'local'); - - $page = $this->entities->page(); - $page->html = ''; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertDontSee(base64_encode($contents)); - } - - public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local_secure() - { - $testFilePath = storage_path('logs/test.txt'); - config()->set('filesystems.images', 'local_secure'); - file_put_contents($testFilePath, 'I am a cat'); - - $page = $this->entities->page(); - $page->html = ''; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertDontSee(base64_encode('I am a cat')); - unlink($testFilePath); - } - - public function test_exports_removes_scripts_from_custom_head() - { - $entities = [ - Page::query()->first(), Chapter::query()->first(), Book::query()->first(), - ]; - setting()->put('app-custom-head', ''); - - foreach ($entities as $entity) { - $resp = $this->asEditor()->get($entity->getUrl('/export/html')); - $resp->assertDontSee('window.donkey'); - $resp->assertDontSee('assertSee('.my-test-class { color: red; }'); - } - } - - public function test_page_export_with_deleted_creator_and_updater() - { - $user = $this->users->viewer(['name' => 'ExportWizardTheFifth']); - $page = $this->entities->page(); - $page->created_by = $user->id; - $page->updated_by = $user->id; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertSee('ExportWizardTheFifth'); - - $user->delete(); - $resp = $this->get($page->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertDontSee('ExportWizardTheFifth'); - } - - public function test_page_pdf_export_converts_iframes_to_links() - { - $page = Page::query()->first()->forceFill([ - 'html' => '', - ]); - $page->save(); - - $pdfHtml = ''; - $mockPdfGenerator = $this->mock(PdfGenerator::class); - $mockPdfGenerator->shouldReceive('fromHtml') - ->with(\Mockery::capture($pdfHtml)) - ->andReturn(''); - $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); - - $this->asEditor()->get($page->getUrl('/export/pdf')); - $this->assertStringNotContainsString('iframe>', $pdfHtml); - $this->assertStringContainsString('

https://www.youtube.com/embed/ShqUjt33uOs

', $pdfHtml); - } - - public function test_page_pdf_export_opens_details_blocks() - { - $page = $this->entities->page()->forceFill([ - 'html' => '
Hello

Content!

', - ]); - $page->save(); - - $pdfHtml = ''; - $mockPdfGenerator = $this->mock(PdfGenerator::class); - $mockPdfGenerator->shouldReceive('fromHtml') - ->with(\Mockery::capture($pdfHtml)) - ->andReturn(''); - $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); - - $this->asEditor()->get($page->getUrl('/export/pdf')); - $this->assertStringContainsString('
entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); - $resp->assertStatus(200); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.md"'); - } - - public function test_page_markdown_export_uses_existing_markdown_if_apparent() - { - $page = $this->entities->page()->forceFill([ - 'markdown' => '# A header', - 'html' => '

Dogcat

', - ]); - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); - $resp->assertSee('A header'); - $resp->assertDontSee('Dogcat'); - } - - public function test_page_markdown_export_converts_html_where_no_markdown() - { - $page = $this->entities->page()->forceFill([ - 'markdown' => '', - 'html' => '

Dogcat

Some bold text

', - ]); - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); - $resp->assertSee("# Dogcat\n\nSome **bold** text"); - } - - public function test_chapter_markdown_export() - { - $chapter = $this->entities->chapter(); - $page = $chapter->pages()->first(); - $resp = $this->asEditor()->get($chapter->getUrl('/export/markdown')); - - $resp->assertSee('# ' . $chapter->name); - $resp->assertSee('# ' . $page->name); - } - - public function test_book_markdown_export() - { - $book = Book::query()->whereHas('pages')->whereHas('chapters')->first(); - $chapter = $book->chapters()->first(); - $page = $chapter->pages()->first(); - $resp = $this->asEditor()->get($book->getUrl('/export/markdown')); - - $resp->assertSee('# ' . $book->name); - $resp->assertSee('# ' . $chapter->name); - $resp->assertSee('# ' . $page->name); - } - - public function test_book_markdown_export_concats_immediate_pages_with_newlines() - { - /** @var Book $book */ - $book = Book::query()->whereHas('pages')->first(); - - $this->asEditor()->get($book->getUrl('/create-page')); - $this->get($book->getUrl('/create-page')); - - [$pageA, $pageB] = $book->pages()->where('chapter_id', '=', 0)->get(); - $pageA->html = '

hello tester

'; - $pageA->save(); - $pageB->name = 'The second page in this test'; - $pageB->save(); - - $resp = $this->get($book->getUrl('/export/markdown')); - $resp->assertDontSee('hello tester# The second page in this test'); - $resp->assertSee("hello tester\n\n# The second page in this test"); - } - - public function test_export_option_only_visible_and_accessible_with_permission() - { - $book = Book::query()->whereHas('pages')->whereHas('chapters')->first(); - $chapter = $book->chapters()->first(); - $page = $chapter->pages()->first(); - $entities = [$book, $chapter, $page]; - $user = $this->users->viewer(); - $this->actingAs($user); - - foreach ($entities as $entity) { - $resp = $this->get($entity->getUrl()); - $resp->assertSee('/export/pdf'); - } - - $this->permissions->removeUserRolePermissions($user, ['content-export']); - - foreach ($entities as $entity) { - $resp = $this->get($entity->getUrl()); - $resp->assertDontSee('/export/pdf'); - $resp = $this->get($entity->getUrl('/export/pdf')); - $this->assertPermissionError($resp); - } - } - - public function test_wkhtmltopdf_only_used_when_allow_untrusted_is_true() - { - $page = $this->entities->page(); - - config()->set('exports.snappy.pdf_binary', '/abc123'); - config()->set('app.allow_untrusted_server_fetching', false); - - $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); - $resp->assertStatus(200); // Sucessful response with invalid snappy binary indicates dompdf usage. - - config()->set('app.allow_untrusted_server_fetching', true); - $resp = $this->get($page->getUrl('/export/pdf')); - $resp->assertStatus(500); // Bad response indicates wkhtml usage - } - - public function test_pdf_command_option_used_if_set() - { - $page = $this->entities->page(); - $command = 'cp {input_html_path} {output_pdf_path}'; - config()->set('exports.pdf_command', $command); - - $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); - $download = $resp->getContent(); - - $this->assertStringContainsString(e($page->name), $download); - $this->assertStringContainsString('set('exports.pdf_command', $command); - - $this->assertThrows(function () use ($page) { - $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); - }, PdfExportException::class); - } - - public function test_pdf_command_option_errors_if_command_returns_error_status() - { - $page = $this->entities->page(); - $command = 'exit 1'; - config()->set('exports.pdf_command', $command); - - $this->assertThrows(function () use ($page) { - $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); - }, PdfExportException::class); - } - - public function test_pdf_command_timout_option_limits_export_time() - { - $page = $this->entities->page(); - $command = 'php -r \'sleep(4);\''; - config()->set('exports.pdf_command', $command); - config()->set('exports.pdf_command_timeout', 1); - - $this->assertThrows(function () use ($page) { - $start = time(); - $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); - - $this->assertTrue(time() < ($start + 3)); - }, PdfExportException::class, - "PDF Export via command failed due to timeout at 1 second(s)"); - } - - public function test_html_exports_contain_csp_meta_tag() - { - $entities = [ - $this->entities->page(), - $this->entities->book(), - $this->entities->chapter(), - ]; - - foreach ($entities as $entity) { - $resp = $this->asEditor()->get($entity->getUrl('/export/html')); - $this->withHtml($resp)->assertElementExists('head meta[http-equiv="Content-Security-Policy"][content*="script-src "]'); - } - } - - public function test_html_exports_contain_body_classes_for_export_identification() - { - $page = $this->entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $this->withHtml($resp)->assertElementExists('body.export.export-format-html.export-engine-none'); - } -} diff --git a/tests/Exports/ExportUiTest.php b/tests/Exports/ExportUiTest.php new file mode 100644 index 00000000000..77b26ad8929 --- /dev/null +++ b/tests/Exports/ExportUiTest.php @@ -0,0 +1,33 @@ +whereHas('pages')->whereHas('chapters')->first(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + $entities = [$book, $chapter, $page]; + $user = $this->users->viewer(); + $this->actingAs($user); + + foreach ($entities as $entity) { + $resp = $this->get($entity->getUrl()); + $resp->assertSee('/export/pdf'); + } + + $this->permissions->removeUserRolePermissions($user, ['content-export']); + + foreach ($entities as $entity) { + $resp = $this->get($entity->getUrl()); + $resp->assertDontSee('/export/pdf'); + $resp = $this->get($entity->getUrl('/export/pdf')); + $this->assertPermissionError($resp); + } + } +} diff --git a/tests/Exports/HtmlExportTest.php b/tests/Exports/HtmlExportTest.php new file mode 100644 index 00000000000..069cf280167 --- /dev/null +++ b/tests/Exports/HtmlExportTest.php @@ -0,0 +1,253 @@ +entities->page(); + $this->asEditor(); + + $resp = $this->get($page->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.html"'); + } + + public function test_book_html_export() + { + $page = $this->entities->page(); + $book = $page->book; + $this->asEditor(); + + $resp = $this->get($book->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($book->name); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.html"'); + } + + public function test_book_html_export_shows_html_descriptions() + { + $book = $this->entities->bookHasChaptersAndPages(); + $chapter = $book->chapters()->first(); + $book->description_html = '

A description with HTML within!

'; + $chapter->description_html = '

A chapter description with HTML within!

'; + $book->save(); + $chapter->save(); + + $resp = $this->asEditor()->get($book->getUrl('/export/html')); + $resp->assertSee($book->description_html, false); + $resp->assertSee($chapter->description_html, false); + } + + public function test_chapter_html_export() + { + $chapter = $this->entities->chapter(); + $page = $chapter->pages[0]; + $this->asEditor(); + + $resp = $this->get($chapter->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($chapter->name); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.html"'); + } + + public function test_chapter_html_export_shows_html_descriptions() + { + $chapter = $this->entities->chapter(); + $chapter->description_html = '

A description with HTML within!

'; + $chapter->save(); + + $resp = $this->asEditor()->get($chapter->getUrl('/export/html')); + $resp->assertSee($chapter->description_html, false); + } + + public function test_page_html_export_contains_custom_head_if_set() + { + $page = $this->entities->page(); + + $customHeadContent = ''; + $this->setSettings(['app-custom-head' => $customHeadContent]); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertSee($customHeadContent, false); + } + + public function test_page_html_export_does_not_break_with_only_comments_in_custom_head() + { + $page = $this->entities->page(); + + $customHeadContent = ''; + $this->setSettings(['app-custom-head' => $customHeadContent]); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($customHeadContent, false); + } + + public function test_page_html_export_use_absolute_dates() + { + $page = $this->entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertSee($page->created_at->isoFormat('D MMMM Y HH:mm:ss')); + $resp->assertDontSee($page->created_at->diffForHumans()); + $resp->assertSee($page->updated_at->isoFormat('D MMMM Y HH:mm:ss')); + $resp->assertDontSee($page->updated_at->diffForHumans()); + } + + public function test_page_export_does_not_include_user_or_revision_links() + { + $page = $this->entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertDontSee($page->getUrl('/revisions')); + $resp->assertDontSee($page->createdBy->getProfileUrl()); + $resp->assertSee($page->createdBy->name); + } + + public function test_page_export_sets_right_data_type_for_svg_embeds() + { + $page = $this->entities->page(); + Storage::disk('local')->makeDirectory('uploads/images/gallery'); + Storage::disk('local')->put('uploads/images/gallery/svg_test.svg', ''); + $page->html = ''; + $page->save(); + + $this->asEditor(); + $resp = $this->get($page->getUrl('/export/html')); + Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); + + $resp->assertStatus(200); + $resp->assertSee(''; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); + Storage::disk('local')->delete('uploads/images/gallery/svg_test2.svg'); + + $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test'); + } + + public function test_page_export_contained_html_image_fetches_only_run_when_url_points_to_image_upload_folder() + { + $page = $this->entities->page(); + $page->html = '' + . '' + . ''; + $storageDisk = Storage::disk('local'); + $storageDisk->makeDirectory('uploads/images/gallery'); + $storageDisk->put('uploads/images/gallery/svg_test.svg', 'good'); + $storageDisk->put('uploads/svg_test.svg', 'bad'); + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + + $storageDisk->delete('uploads/images/gallery/svg_test.svg'); + $storageDisk->delete('uploads/svg_test.svg'); + + $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test.svg', false); + $resp->assertSee('http://localhost/uploads/svg_test.svg'); + $resp->assertSee('src="/uploads/svg_test.svg"', false); + } + + public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local() + { + $contents = file_get_contents(public_path('.htaccess')); + config()->set('filesystems.images', 'local'); + + $page = $this->entities->page(); + $page->html = ''; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertDontSee(base64_encode($contents)); + } + + public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local_secure() + { + $testFilePath = storage_path('logs/test.txt'); + config()->set('filesystems.images', 'local_secure'); + file_put_contents($testFilePath, 'I am a cat'); + + $page = $this->entities->page(); + $page->html = ''; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertDontSee(base64_encode('I am a cat')); + unlink($testFilePath); + } + + public function test_exports_removes_scripts_from_custom_head() + { + $entities = [ + Page::query()->first(), Chapter::query()->first(), Book::query()->first(), + ]; + setting()->put('app-custom-head', ''); + + foreach ($entities as $entity) { + $resp = $this->asEditor()->get($entity->getUrl('/export/html')); + $resp->assertDontSee('window.donkey'); + $resp->assertDontSee('assertSee('.my-test-class { color: red; }'); + } + } + + public function test_page_export_with_deleted_creator_and_updater() + { + $user = $this->users->viewer(['name' => 'ExportWizardTheFifth']); + $page = $this->entities->page(); + $page->created_by = $user->id; + $page->updated_by = $user->id; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertSee('ExportWizardTheFifth'); + + $user->delete(); + $resp = $this->get($page->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertDontSee('ExportWizardTheFifth'); + } + + public function test_html_exports_contain_csp_meta_tag() + { + $entities = [ + $this->entities->page(), + $this->entities->book(), + $this->entities->chapter(), + ]; + + foreach ($entities as $entity) { + $resp = $this->asEditor()->get($entity->getUrl('/export/html')); + $this->withHtml($resp)->assertElementExists('head meta[http-equiv="Content-Security-Policy"][content*="script-src "]'); + } + } + + public function test_html_exports_contain_body_classes_for_export_identification() + { + $page = $this->entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $this->withHtml($resp)->assertElementExists('body.export.export-format-html.export-engine-none'); + } +} diff --git a/tests/Exports/MarkdownExportTest.php b/tests/Exports/MarkdownExportTest.php new file mode 100644 index 00000000000..05ebbc68d75 --- /dev/null +++ b/tests/Exports/MarkdownExportTest.php @@ -0,0 +1,85 @@ +entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); + $resp->assertStatus(200); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.md"'); + } + + public function test_page_markdown_export_uses_existing_markdown_if_apparent() + { + $page = $this->entities->page()->forceFill([ + 'markdown' => '# A header', + 'html' => '

Dogcat

', + ]); + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); + $resp->assertSee('A header'); + $resp->assertDontSee('Dogcat'); + } + + public function test_page_markdown_export_converts_html_where_no_markdown() + { + $page = $this->entities->page()->forceFill([ + 'markdown' => '', + 'html' => '

Dogcat

Some bold text

', + ]); + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); + $resp->assertSee("# Dogcat\n\nSome **bold** text"); + } + + public function test_chapter_markdown_export() + { + $chapter = $this->entities->chapter(); + $page = $chapter->pages()->first(); + $resp = $this->asEditor()->get($chapter->getUrl('/export/markdown')); + + $resp->assertSee('# ' . $chapter->name); + $resp->assertSee('# ' . $page->name); + } + + public function test_book_markdown_export() + { + $book = Book::query()->whereHas('pages')->whereHas('chapters')->first(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + $resp = $this->asEditor()->get($book->getUrl('/export/markdown')); + + $resp->assertSee('# ' . $book->name); + $resp->assertSee('# ' . $chapter->name); + $resp->assertSee('# ' . $page->name); + } + + public function test_book_markdown_export_concats_immediate_pages_with_newlines() + { + /** @var Book $book */ + $book = Book::query()->whereHas('pages')->first(); + + $this->asEditor()->get($book->getUrl('/create-page')); + $this->get($book->getUrl('/create-page')); + + [$pageA, $pageB] = $book->pages()->where('chapter_id', '=', 0)->get(); + $pageA->html = '

hello tester

'; + $pageA->save(); + $pageB->name = 'The second page in this test'; + $pageB->save(); + + $resp = $this->get($book->getUrl('/export/markdown')); + $resp->assertDontSee('hello tester# The second page in this test'); + $resp->assertSee("hello tester\n\n# The second page in this test"); + } +} diff --git a/tests/Exports/PdfExportTest.php b/tests/Exports/PdfExportTest.php new file mode 100644 index 00000000000..9d85c69e23a --- /dev/null +++ b/tests/Exports/PdfExportTest.php @@ -0,0 +1,146 @@ +entities->page(); + $this->asEditor(); + + $resp = $this->get($page->getUrl('/export/pdf')); + $resp->assertStatus(200); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.pdf"'); + } + + public function test_book_pdf_export() + { + $page = $this->entities->page(); + $book = $page->book; + $this->asEditor(); + + $resp = $this->get($book->getUrl('/export/pdf')); + $resp->assertStatus(200); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.pdf"'); + } + + public function test_chapter_pdf_export() + { + $chapter = $this->entities->chapter(); + $this->asEditor(); + + $resp = $this->get($chapter->getUrl('/export/pdf')); + $resp->assertStatus(200); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.pdf"'); + } + + + public function test_page_pdf_export_converts_iframes_to_links() + { + $page = Page::query()->first()->forceFill([ + 'html' => '', + ]); + $page->save(); + + $pdfHtml = ''; + $mockPdfGenerator = $this->mock(PdfGenerator::class); + $mockPdfGenerator->shouldReceive('fromHtml') + ->with(\Mockery::capture($pdfHtml)) + ->andReturn(''); + $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); + + $this->asEditor()->get($page->getUrl('/export/pdf')); + $this->assertStringNotContainsString('iframe>', $pdfHtml); + $this->assertStringContainsString('

https://www.youtube.com/embed/ShqUjt33uOs

', $pdfHtml); + } + + public function test_page_pdf_export_opens_details_blocks() + { + $page = $this->entities->page()->forceFill([ + 'html' => '
Hello

Content!

', + ]); + $page->save(); + + $pdfHtml = ''; + $mockPdfGenerator = $this->mock(PdfGenerator::class); + $mockPdfGenerator->shouldReceive('fromHtml') + ->with(\Mockery::capture($pdfHtml)) + ->andReturn(''); + $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); + + $this->asEditor()->get($page->getUrl('/export/pdf')); + $this->assertStringContainsString('
entities->page(); + + config()->set('exports.snappy.pdf_binary', '/abc123'); + config()->set('app.allow_untrusted_server_fetching', false); + + $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); + $resp->assertStatus(200); // Sucessful response with invalid snappy binary indicates dompdf usage. + + config()->set('app.allow_untrusted_server_fetching', true); + $resp = $this->get($page->getUrl('/export/pdf')); + $resp->assertStatus(500); // Bad response indicates wkhtml usage + } + + public function test_pdf_command_option_used_if_set() + { + $page = $this->entities->page(); + $command = 'cp {input_html_path} {output_pdf_path}'; + config()->set('exports.pdf_command', $command); + + $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); + $download = $resp->getContent(); + + $this->assertStringContainsString(e($page->name), $download); + $this->assertStringContainsString('set('exports.pdf_command', $command); + + $this->assertThrows(function () use ($page) { + $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); + }, PdfExportException::class); + } + + public function test_pdf_command_option_errors_if_command_returns_error_status() + { + $page = $this->entities->page(); + $command = 'exit 1'; + config()->set('exports.pdf_command', $command); + + $this->assertThrows(function () use ($page) { + $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); + }, PdfExportException::class); + } + + public function test_pdf_command_timout_option_limits_export_time() + { + $page = $this->entities->page(); + $command = 'php -r \'sleep(4);\''; + config()->set('exports.pdf_command', $command); + config()->set('exports.pdf_command_timeout', 1); + + $this->assertThrows(function () use ($page) { + $start = time(); + $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); + + $this->assertTrue(time() < ($start + 3)); + }, PdfExportException::class, + "PDF Export via command failed due to timeout at 1 second(s)"); + } +} diff --git a/tests/Exports/TextExportTest.php b/tests/Exports/TextExportTest.php new file mode 100644 index 00000000000..c593a6585cb --- /dev/null +++ b/tests/Exports/TextExportTest.php @@ -0,0 +1,88 @@ +entities->page(); + $this->asEditor(); + + $resp = $this->get($page->getUrl('/export/plaintext')); + $resp->assertStatus(200); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.txt"'); + } + + public function test_book_text_export() + { + $book = $this->entities->bookHasChaptersAndPages(); + $directPage = $book->directPages()->first(); + $chapter = $book->chapters()->first(); + $chapterPage = $chapter->pages()->first(); + $this->entities->updatePage($directPage, ['html' => '

My awesome page

']); + $this->entities->updatePage($chapterPage, ['html' => '

My little nested page

']); + $this->asEditor(); + + $resp = $this->get($book->getUrl('/export/plaintext')); + $resp->assertStatus(200); + $resp->assertSee($book->name); + $resp->assertSee($chapterPage->name); + $resp->assertSee($chapter->name); + $resp->assertSee($directPage->name); + $resp->assertSee('My awesome page'); + $resp->assertSee('My little nested page'); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.txt"'); + } + + public function test_book_text_export_format() + { + $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); + $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); + $entities['chapter']->name = 'Export chapter'; + $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; + $entities['book']->name = 'Export Book'; + $entities['book']->description = "This is a book with stuff to export"; + $entities['chapter']->save(); + $entities['book']->save(); + + $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); + + $expected = "Export Book\nThis is a book with stuff to export\n\nExport chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; + $expected .= "My wonderful page!\nMy great page Full of great stuff"; + $resp->assertSee($expected); + } + + public function test_chapter_text_export() + { + $chapter = $this->entities->chapter(); + $page = $chapter->pages[0]; + $this->entities->updatePage($page, ['html' => '

This is content within the page!

']); + $this->asEditor(); + + $resp = $this->get($chapter->getUrl('/export/plaintext')); + $resp->assertStatus(200); + $resp->assertSee($chapter->name); + $resp->assertSee($page->name); + $resp->assertSee('This is content within the page!'); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.txt"'); + } + + public function test_chapter_text_export_format() + { + $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); + $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); + $entities['chapter']->name = 'Export chapter'; + $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; + $entities['chapter']->save(); + + $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); + + $expected = "Export chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; + $expected .= "My wonderful page!\nMy great page Full of great stuff"; + $resp->assertSee($expected); + } +} From 21ccfa97ddfe6309ff735aafbc8138cf34782563 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 19 Oct 2024 15:41:07 +0100 Subject: [PATCH 06/41] ZIP Export: Expanded page & added base attachment handling --- .../Controllers/PageExportController.php | 13 ++++ app/Exports/ZipExportBuilder.php | 66 +++++++++++++++++-- app/Exports/ZipExportFiles.php | 58 ++++++++++++++++ app/Uploads/AttachmentService.php | 11 +--- lang/en/entities.php | 1 + .../views/entities/export-menu.blade.php | 1 + routes/web.php | 1 + tests/Exports/ZipExportTest.php | 15 +++++ 8 files changed, 154 insertions(+), 12 deletions(-) create mode 100644 app/Exports/ZipExportFiles.php create mode 100644 tests/Exports/ZipExportTest.php diff --git a/app/Exports/Controllers/PageExportController.php b/app/Exports/Controllers/PageExportController.php index a4e7aae879d..01611fd2121 100644 --- a/app/Exports/Controllers/PageExportController.php +++ b/app/Exports/Controllers/PageExportController.php @@ -6,6 +6,7 @@ use BookStack\Entities\Tools\PageContent; use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; +use BookStack\Exports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; @@ -74,4 +75,16 @@ public function markdown(string $bookSlug, string $pageSlug) return $this->download()->directly($pageText, $pageSlug . '.md'); } + + /** + * Export a page to a contained ZIP export file. + * @throws NotFoundException + */ + public function zip(string $bookSlug, string $pageSlug, ZipExportBuilder $builder) + { + $page = $this->queries->findVisibleBySlugsOrFail($bookSlug, $pageSlug); + $zip = $builder->buildForPage($page); + + return $this->download()->streamedDirectly(fopen($zip, 'r'), $pageSlug . '.zip', filesize($zip)); + } } diff --git a/app/Exports/ZipExportBuilder.php b/app/Exports/ZipExportBuilder.php index d1a7b6bd461..2b8b45d0d2a 100644 --- a/app/Exports/ZipExportBuilder.php +++ b/app/Exports/ZipExportBuilder.php @@ -2,24 +2,70 @@ namespace BookStack\Exports; +use BookStack\Activity\Models\Tag; use BookStack\Entities\Models\Page; use BookStack\Exceptions\ZipExportException; +use BookStack\Uploads\Attachment; use ZipArchive; class ZipExportBuilder { protected array $data = []; + public function __construct( + protected ZipExportFiles $files + ) { + } + /** * @throws ZipExportException */ public function buildForPage(Page $page): string { - $this->data['page'] = [ - 'id' => $page->id, + $this->data['page'] = $this->convertPage($page); + return $this->build(); + } + + protected function convertPage(Page $page): array + { + $tags = array_map($this->convertTag(...), $page->tags()->get()->all()); + $attachments = array_map($this->convertAttachment(...), $page->attachments()->get()->all()); + + return [ + 'id' => $page->id, + 'name' => $page->name, + 'html' => '', // TODO + 'markdown' => '', // TODO + 'priority' => $page->priority, + 'attachments' => $attachments, + 'images' => [], // TODO + 'tags' => $tags, ]; + } - return $this->build(); + protected function convertAttachment(Attachment $attachment): array + { + $data = [ + 'name' => $attachment->name, + 'order' => $attachment->order, + ]; + + if ($attachment->external) { + $data['link'] = $attachment->path; + } else { + $data['file'] = $this->files->referenceForAttachment($attachment); + } + + return $data; + } + + protected function convertTag(Tag $tag): array + { + return [ + 'name' => $tag->name, + 'value' => $tag->value, + 'order' => $tag->order, + ]; } /** @@ -29,7 +75,7 @@ protected function build(): string { $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ - 'version' => trim(file_get_contents(base_path('version'))), + 'version' => trim(file_get_contents(base_path('version'))), 'id_ciphertext' => encrypt('bookstack'), ]; @@ -43,6 +89,18 @@ protected function build(): string $zip->addFromString('data.json', json_encode($this->data)); $zip->addEmptyDir('files'); + $toRemove = []; + $this->files->extractEach(function ($filePath, $fileRef) use ($zip, &$toRemove) { + $zip->addFile($filePath, "files/$fileRef"); + $toRemove[] = $filePath; + }); + + $zip->close(); + + foreach ($toRemove as $file) { + unlink($file); + } + return $zipFile; } } diff --git a/app/Exports/ZipExportFiles.php b/app/Exports/ZipExportFiles.php new file mode 100644 index 00000000000..d3ee70e93f2 --- /dev/null +++ b/app/Exports/ZipExportFiles.php @@ -0,0 +1,58 @@ + + */ + protected array $attachmentRefsById = []; + + public function __construct( + protected AttachmentService $attachmentService, + ) { + } + + /** + * Gain a reference to the given attachment instance. + * This is expected to be a file-based attachment that the user + * has visibility of, no permission/access checks are performed here. + */ + public function referenceForAttachment(Attachment $attachment): string + { + if (isset($this->attachmentRefsById[$attachment->id])) { + return $this->attachmentRefsById[$attachment->id]; + } + + do { + $fileName = Str::random(20) . '.' . $attachment->extension; + } while (in_array($fileName, $this->attachmentRefsById)); + + $this->attachmentRefsById[$attachment->id] = $fileName; + + return $fileName; + } + + /** + * Extract each of the ZIP export tracked files. + * Calls the given callback for each tracked file, passing a temporary + * file reference of the file contents, and the zip-local tracked reference. + */ + public function extractEach(callable $callback): void + { + foreach ($this->attachmentRefsById as $attachmentId => $ref) { + $attachment = Attachment::query()->find($attachmentId); + $stream = $this->attachmentService->streamAttachmentFromStorage($attachment); + $tmpFile = tempnam(sys_get_temp_dir(), 'bszipfile-'); + $tmpFileStream = fopen($tmpFile, 'w'); + stream_copy_to_stream($stream, $tmpFileStream); + $callback($tmpFile, $ref); + } + } +} diff --git a/app/Uploads/AttachmentService.php b/app/Uploads/AttachmentService.php index bd319fbd795..227649d8f00 100644 --- a/app/Uploads/AttachmentService.php +++ b/app/Uploads/AttachmentService.php @@ -13,14 +13,9 @@ class AttachmentService { - protected FilesystemManager $fileSystem; - - /** - * AttachmentService constructor. - */ - public function __construct(FilesystemManager $fileSystem) - { - $this->fileSystem = $fileSystem; + public function __construct( + protected FilesystemManager $fileSystem + ) { } /** diff --git a/lang/en/entities.php b/lang/en/entities.php index 35e6f050bb8..7e5a708ef62 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -39,6 +39,7 @@ 'export_pdf' => 'PDF File', 'export_text' => 'Plain Text File', 'export_md' => 'Markdown File', + 'export_zip' => 'Portable ZIP', 'default_template' => 'Default Page Template', 'default_template_explain' => 'Assign a page template that will be used as the default content for all pages created within this item. Keep in mind this will only be used if the page creator has view access to the chosen template page.', 'default_template_select' => 'Select a template page', diff --git a/resources/views/entities/export-menu.blade.php b/resources/views/entities/export-menu.blade.php index a55ab56d199..e58c842ba42 100644 --- a/resources/views/entities/export-menu.blade.php +++ b/resources/views/entities/export-menu.blade.php @@ -18,6 +18,7 @@ class="icon-list-item"
  • {{ trans('entities.export_pdf') }}.pdf
  • {{ trans('entities.export_text') }}.txt
  • {{ trans('entities.export_md') }}.md
    • +
  • {{ trans('entities.export_zip') }}.zip
    • diff --git a/routes/web.php b/routes/web.php index 5220684c0b0..6ae70983d52 100644 --- a/routes/web.php +++ b/routes/web.php @@ -91,6 +91,7 @@ Route::get('/books/{bookSlug}/page/{pageSlug}/export/html', [ExportControllers\PageExportController::class, 'html']); Route::get('/books/{bookSlug}/page/{pageSlug}/export/markdown', [ExportControllers\PageExportController::class, 'markdown']); Route::get('/books/{bookSlug}/page/{pageSlug}/export/plaintext', [ExportControllers\PageExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/zip', [ExportControllers\PageExportController::class, 'zip']); Route::get('/books/{bookSlug}/page/{pageSlug}/edit', [EntityControllers\PageController::class, 'edit']); Route::get('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'showMove']); Route::put('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'move']); diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php new file mode 100644 index 00000000000..d8ce00be33f --- /dev/null +++ b/tests/Exports/ZipExportTest.php @@ -0,0 +1,15 @@ +entities->page(); + // TODO + } +} From 7c39dd5cba7b72184adb96a8236ca1a3c99f03e3 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 20 Oct 2024 19:56:56 +0100 Subject: [PATCH 07/41] ZIP Export: Started building link/ref handling --- app/Exports/ZipExportBuilder.php | 54 +++---------- .../ZipExportModels/ZipExportAttachment.php | 37 +++++++++ .../ZipExportModels/ZipExportImage.php | 11 +++ .../ZipExportModels/ZipExportModel.php | 11 +++ app/Exports/ZipExportModels/ZipExportPage.php | 39 ++++++++++ app/Exports/ZipExportModels/ZipExportTag.php | 27 +++++++ app/Exports/ZipExportReferences.php | 55 ++++++++++++++ app/Exports/ZipReferenceParser.php | 75 +++++++++++++++++++ dev/docs/portable-zip-file-format.md | 1 + 9 files changed, 265 insertions(+), 45 deletions(-) create mode 100644 app/Exports/ZipExportModels/ZipExportAttachment.php create mode 100644 app/Exports/ZipExportModels/ZipExportImage.php create mode 100644 app/Exports/ZipExportModels/ZipExportModel.php create mode 100644 app/Exports/ZipExportModels/ZipExportPage.php create mode 100644 app/Exports/ZipExportModels/ZipExportTag.php create mode 100644 app/Exports/ZipExportReferences.php create mode 100644 app/Exports/ZipReferenceParser.php diff --git a/app/Exports/ZipExportBuilder.php b/app/Exports/ZipExportBuilder.php index 2b8b45d0d2a..720b4997d4a 100644 --- a/app/Exports/ZipExportBuilder.php +++ b/app/Exports/ZipExportBuilder.php @@ -2,10 +2,9 @@ namespace BookStack\Exports; -use BookStack\Activity\Models\Tag; use BookStack\Entities\Models\Page; use BookStack\Exceptions\ZipExportException; -use BookStack\Uploads\Attachment; +use BookStack\Exports\ZipExportModels\ZipExportPage; use ZipArchive; class ZipExportBuilder @@ -13,7 +12,8 @@ class ZipExportBuilder protected array $data = []; public function __construct( - protected ZipExportFiles $files + protected ZipExportFiles $files, + protected ZipExportReferences $references, ) { } @@ -22,50 +22,12 @@ public function __construct( */ public function buildForPage(Page $page): string { - $this->data['page'] = $this->convertPage($page); - return $this->build(); - } + $exportPage = ZipExportPage::fromModel($page, $this->files); + $this->data['page'] = $exportPage; - protected function convertPage(Page $page): array - { - $tags = array_map($this->convertTag(...), $page->tags()->get()->all()); - $attachments = array_map($this->convertAttachment(...), $page->attachments()->get()->all()); - - return [ - 'id' => $page->id, - 'name' => $page->name, - 'html' => '', // TODO - 'markdown' => '', // TODO - 'priority' => $page->priority, - 'attachments' => $attachments, - 'images' => [], // TODO - 'tags' => $tags, - ]; - } - - protected function convertAttachment(Attachment $attachment): array - { - $data = [ - 'name' => $attachment->name, - 'order' => $attachment->order, - ]; + $this->references->addPage($exportPage); - if ($attachment->external) { - $data['link'] = $attachment->path; - } else { - $data['file'] = $this->files->referenceForAttachment($attachment); - } - - return $data; - } - - protected function convertTag(Tag $tag): array - { - return [ - 'name' => $tag->name, - 'value' => $tag->value, - 'order' => $tag->order, - ]; + return $this->build(); } /** @@ -73,6 +35,8 @@ protected function convertTag(Tag $tag): array */ protected function build(): string { + $this->references->buildReferences(); + $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ 'version' => trim(file_get_contents(base_path('version'))), diff --git a/app/Exports/ZipExportModels/ZipExportAttachment.php b/app/Exports/ZipExportModels/ZipExportAttachment.php new file mode 100644 index 00000000000..d6d674a913f --- /dev/null +++ b/app/Exports/ZipExportModels/ZipExportAttachment.php @@ -0,0 +1,37 @@ +id = $model->id; + $instance->name = $model->name; + + if ($model->external) { + $instance->link = $model->path; + } else { + $instance->file = $files->referenceForAttachment($model); + } + + return $instance; + } + + public static function fromModelArray(array $attachmentArray, ZipExportFiles $files): array + { + return array_values(array_map(function (Attachment $attachment) use ($files) { + return self::fromModel($attachment, $files); + }, $attachmentArray)); + } +} diff --git a/app/Exports/ZipExportModels/ZipExportImage.php b/app/Exports/ZipExportModels/ZipExportImage.php new file mode 100644 index 00000000000..73fe3bbf594 --- /dev/null +++ b/app/Exports/ZipExportModels/ZipExportImage.php @@ -0,0 +1,11 @@ +id = $model->id; + $instance->name = $model->name; + $instance->html = (new PageContent($model))->render(); + + if (!empty($model->markdown)) { + $instance->markdown = $model->markdown; + } + + $instance->tags = ZipExportTag::fromModelArray($model->tags()->get()->all()); + $instance->attachments = ZipExportAttachment::fromModelArray($model->attachments()->get()->all(), $files); + + return $instance; + } +} diff --git a/app/Exports/ZipExportModels/ZipExportTag.php b/app/Exports/ZipExportModels/ZipExportTag.php new file mode 100644 index 00000000000..636c9ff6dc4 --- /dev/null +++ b/app/Exports/ZipExportModels/ZipExportTag.php @@ -0,0 +1,27 @@ +name = $model->name; + $instance->value = $model->value; + $instance->order = $model->order; + + return $instance; + } + + public static function fromModelArray(array $tagArray): array + { + return array_values(array_map(self::fromModel(...), $tagArray)); + } +} diff --git a/app/Exports/ZipExportReferences.php b/app/Exports/ZipExportReferences.php new file mode 100644 index 00000000000..89deb7edad3 --- /dev/null +++ b/app/Exports/ZipExportReferences.php @@ -0,0 +1,55 @@ +id) { + $this->pages[$page->id] = $page; + } + + foreach ($page->attachments as $attachment) { + if ($attachment->id) { + $this->attachments[$attachment->id] = $attachment; + } + } + } + + public function buildReferences(): void + { + // TODO - References to images, attachments, other entities + + // TODO - Parse page MD & HTML + foreach ($this->pages as $page) { + $page->html = $this->parser->parse($page->html ?? '', function (Model $model): ?string { + // TODO - Handle found link to $model + // - Validate we can see/access $model, or/and that it's + // part of the export in progress. + return '[CAT]'; + }); + // TODO - markdown + } + + // TODO - Parse chapter desc html + // TODO - Parse book desc html + } +} diff --git a/app/Exports/ZipReferenceParser.php b/app/Exports/ZipReferenceParser.php new file mode 100644 index 00000000000..6ca826bc3f4 --- /dev/null +++ b/app/Exports/ZipReferenceParser.php @@ -0,0 +1,75 @@ +modelResolvers = [ + new PagePermalinkModelResolver($queries->pages), + new PageLinkModelResolver($queries->pages), + new ChapterLinkModelResolver($queries->chapters), + new BookLinkModelResolver($queries->books), + // TODO - Image + // TODO - Attachment + ]; + } + + /** + * Parse and replace references in the given content. + * @param callable(Model):(string|null) $handler + */ + public function parse(string $content, callable $handler): string + { + $escapedBase = preg_quote(url('/'), '/'); + $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#]/"; + $matches = []; + preg_match_all($linkRegex, $content, $matches); + + if (count($matches) < 2) { + return $content; + } + + foreach ($matches[1] as $link) { + $model = $this->linkToModel($link); + if ($model) { + $result = $handler($model); + if ($result !== null) { + $content = str_replace($link, $result, $content); + } + } + } + + return $content; + } + + + /** + * Attempt to resolve the given link to a model using the instance model resolvers. + */ + protected function linkToModel(string $link): ?Model + { + foreach ($this->modelResolvers as $resolver) { + $model = $resolver->resolve($link); + if (!is_null($model)) { + return $model; + } + } + + return null; + } +} diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index d5635bd398d..7a99563d14b 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -128,6 +128,7 @@ File must be an image type accepted by BookStack (png, jpg, gif, webp) #### Attachment +- `id` - Number, optional, original ID for the attachment from exported system. - `name` - String, required, name of attachment. - `link` - String, semi-optional, URL of attachment. - `file` - String reference, semi-optional, reference to attachment file. From 06ffd8ee721a74dea9c584002b2793cc68c873a0 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 21 Oct 2024 12:13:41 +0100 Subject: [PATCH 08/41] Zip Exports: Added attachment/image link resolving & JSON null handling --- .../ZipExportModels/ZipExportAttachment.php | 2 +- .../ZipExportModels/ZipExportImage.php | 2 +- .../ZipExportModels/ZipExportModel.php | 17 +++++++--- app/Exports/ZipExportModels/ZipExportPage.php | 2 +- app/Exports/ZipExportModels/ZipExportTag.php | 2 +- app/Exports/ZipExportReferences.php | 3 ++ app/Exports/ZipReferenceParser.php | 6 ++-- .../AttachmentModelResolver.php | 22 +++++++++++++ .../ModelResolvers/ImageModelResolver.php | 33 +++++++++++++++++++ 9 files changed, 79 insertions(+), 10 deletions(-) create mode 100644 app/References/ModelResolvers/AttachmentModelResolver.php create mode 100644 app/References/ModelResolvers/ImageModelResolver.php diff --git a/app/Exports/ZipExportModels/ZipExportAttachment.php b/app/Exports/ZipExportModels/ZipExportAttachment.php index d6d674a913f..d79a16cc109 100644 --- a/app/Exports/ZipExportModels/ZipExportAttachment.php +++ b/app/Exports/ZipExportModels/ZipExportAttachment.php @@ -5,7 +5,7 @@ use BookStack\Exports\ZipExportFiles; use BookStack\Uploads\Attachment; -class ZipExportAttachment implements ZipExportModel +class ZipExportAttachment extends ZipExportModel { public ?int $id = null; public string $name; diff --git a/app/Exports/ZipExportModels/ZipExportImage.php b/app/Exports/ZipExportModels/ZipExportImage.php index 73fe3bbf594..540d3d4e55d 100644 --- a/app/Exports/ZipExportModels/ZipExportImage.php +++ b/app/Exports/ZipExportModels/ZipExportImage.php @@ -4,7 +4,7 @@ use BookStack\Activity\Models\Tag; -class ZipExportImage implements ZipExportModel +class ZipExportImage extends ZipExportModel { public string $name; public string $file; diff --git a/app/Exports/ZipExportModels/ZipExportModel.php b/app/Exports/ZipExportModels/ZipExportModel.php index e1cb616de52..26b994c018e 100644 --- a/app/Exports/ZipExportModels/ZipExportModel.php +++ b/app/Exports/ZipExportModels/ZipExportModel.php @@ -2,10 +2,19 @@ namespace BookStack\Exports\ZipExportModels; -use BookStack\App\Model; -use BookStack\Exports\ZipExportFiles; +use JsonSerializable; -interface ZipExportModel +abstract class ZipExportModel implements JsonSerializable { -// public static function fromModel(Model $model, ZipExportFiles $files): self; + /** + * Handle the serialization to JSON. + * For these exports, we filter out optional (represented as nullable) fields + * just to clean things up and prevent confusion to avoid null states in the + * resulting export format itself. + */ + public function jsonSerialize(): array + { + $publicProps = get_object_vars(...)->__invoke($this); + return array_filter($publicProps, fn ($value) => $value !== null); + } } diff --git a/app/Exports/ZipExportModels/ZipExportPage.php b/app/Exports/ZipExportModels/ZipExportPage.php index 6589ce60ae1..c7a9503546f 100644 --- a/app/Exports/ZipExportModels/ZipExportPage.php +++ b/app/Exports/ZipExportModels/ZipExportPage.php @@ -6,7 +6,7 @@ use BookStack\Entities\Tools\PageContent; use BookStack\Exports\ZipExportFiles; -class ZipExportPage implements ZipExportModel +class ZipExportPage extends ZipExportModel { public ?int $id = null; public string $name; diff --git a/app/Exports/ZipExportModels/ZipExportTag.php b/app/Exports/ZipExportModels/ZipExportTag.php index 636c9ff6dc4..09ae9f06cbe 100644 --- a/app/Exports/ZipExportModels/ZipExportTag.php +++ b/app/Exports/ZipExportModels/ZipExportTag.php @@ -4,7 +4,7 @@ use BookStack\Activity\Models\Tag; -class ZipExportTag implements ZipExportModel +class ZipExportTag extends ZipExportModel { public string $name; public ?string $value = null; diff --git a/app/Exports/ZipExportReferences.php b/app/Exports/ZipExportReferences.php index 89deb7edad3..76a7fedbec4 100644 --- a/app/Exports/ZipExportReferences.php +++ b/app/Exports/ZipExportReferences.php @@ -44,11 +44,14 @@ public function buildReferences(): void // TODO - Handle found link to $model // - Validate we can see/access $model, or/and that it's // part of the export in progress. + + // TODO - Add images after the above to files return '[CAT]'; }); // TODO - markdown } +// dd('end'); // TODO - Parse chapter desc html // TODO - Parse book desc html } diff --git a/app/Exports/ZipReferenceParser.php b/app/Exports/ZipReferenceParser.php index 6ca826bc3f4..820920da28d 100644 --- a/app/Exports/ZipReferenceParser.php +++ b/app/Exports/ZipReferenceParser.php @@ -4,9 +4,11 @@ use BookStack\App\Model; use BookStack\Entities\Queries\EntityQueries; +use BookStack\References\ModelResolvers\AttachmentModelResolver; use BookStack\References\ModelResolvers\BookLinkModelResolver; use BookStack\References\ModelResolvers\ChapterLinkModelResolver; use BookStack\References\ModelResolvers\CrossLinkModelResolver; +use BookStack\References\ModelResolvers\ImageModelResolver; use BookStack\References\ModelResolvers\PageLinkModelResolver; use BookStack\References\ModelResolvers\PagePermalinkModelResolver; @@ -24,8 +26,8 @@ public function __construct(EntityQueries $queries) new PageLinkModelResolver($queries->pages), new ChapterLinkModelResolver($queries->chapters), new BookLinkModelResolver($queries->books), - // TODO - Image - // TODO - Attachment + new ImageModelResolver(), + new AttachmentModelResolver(), ]; } diff --git a/app/References/ModelResolvers/AttachmentModelResolver.php b/app/References/ModelResolvers/AttachmentModelResolver.php new file mode 100644 index 00000000000..e870d515bbe --- /dev/null +++ b/app/References/ModelResolvers/AttachmentModelResolver.php @@ -0,0 +1,22 @@ +find($id); + } +} diff --git a/app/References/ModelResolvers/ImageModelResolver.php b/app/References/ModelResolvers/ImageModelResolver.php new file mode 100644 index 00000000000..331dd593b73 --- /dev/null +++ b/app/References/ModelResolvers/ImageModelResolver.php @@ -0,0 +1,33 @@ +where('path', '=', $fullPath)->first(); + } +} From 4fb4fe0931d220ffb9d7e173388351047b665f4c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 21 Oct 2024 13:59:15 +0100 Subject: [PATCH 09/41] ZIP Exports: Added working image handling/inclusion --- app/Exports/ZipExportBuilder.php | 2 +- app/Exports/ZipExportFiles.php | 51 ++++++++++++++- .../ZipExportModels/ZipExportImage.php | 16 ++++- app/Exports/ZipExportReferences.php | 62 ++++++++++++++++--- app/Uploads/ImageService.php | 13 ++++ app/Uploads/ImageStorageDisk.php | 9 +++ dev/docs/portable-zip-file-format.md | 13 ++-- 7 files changed, 148 insertions(+), 18 deletions(-) diff --git a/app/Exports/ZipExportBuilder.php b/app/Exports/ZipExportBuilder.php index 720b4997d4a..5c56e531b40 100644 --- a/app/Exports/ZipExportBuilder.php +++ b/app/Exports/ZipExportBuilder.php @@ -35,7 +35,7 @@ public function buildForPage(Page $page): string */ protected function build(): string { - $this->references->buildReferences(); + $this->references->buildReferences($this->files); $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ diff --git a/app/Exports/ZipExportFiles.php b/app/Exports/ZipExportFiles.php index d3ee70e93f2..27b6f937a38 100644 --- a/app/Exports/ZipExportFiles.php +++ b/app/Exports/ZipExportFiles.php @@ -4,6 +4,8 @@ use BookStack\Uploads\Attachment; use BookStack\Uploads\AttachmentService; +use BookStack\Uploads\Image; +use BookStack\Uploads\ImageService; use Illuminate\Support\Str; class ZipExportFiles @@ -14,8 +16,15 @@ class ZipExportFiles */ protected array $attachmentRefsById = []; + /** + * References for images by image ID. + * @var array + */ + protected array $imageRefsById = []; + public function __construct( protected AttachmentService $attachmentService, + protected ImageService $imageService, ) { } @@ -30,15 +39,46 @@ public function referenceForAttachment(Attachment $attachment): string return $this->attachmentRefsById[$attachment->id]; } + $existingFiles = $this->getAllFileNames(); do { $fileName = Str::random(20) . '.' . $attachment->extension; - } while (in_array($fileName, $this->attachmentRefsById)); + } while (in_array($fileName, $existingFiles)); $this->attachmentRefsById[$attachment->id] = $fileName; return $fileName; } + /** + * Gain a reference to the given image instance. + * This is expected to be an image that the user has visibility of, + * no permission/access checks are performed here. + */ + public function referenceForImage(Image $image): string + { + if (isset($this->imageRefsById[$image->id])) { + return $this->imageRefsById[$image->id]; + } + + $existingFiles = $this->getAllFileNames(); + $extension = pathinfo($image->path, PATHINFO_EXTENSION); + do { + $fileName = Str::random(20) . '.' . $extension; + } while (in_array($fileName, $existingFiles)); + + $this->imageRefsById[$image->id] = $fileName; + + return $fileName; + } + + protected function getAllFileNames(): array + { + return array_merge( + array_values($this->attachmentRefsById), + array_values($this->imageRefsById), + ); + } + /** * Extract each of the ZIP export tracked files. * Calls the given callback for each tracked file, passing a temporary @@ -54,5 +94,14 @@ public function extractEach(callable $callback): void stream_copy_to_stream($stream, $tmpFileStream); $callback($tmpFile, $ref); } + + foreach ($this->imageRefsById as $imageId => $ref) { + $image = Image::query()->find($imageId); + $stream = $this->imageService->getImageStream($image); + $tmpFile = tempnam(sys_get_temp_dir(), 'bszipimage-'); + $tmpFileStream = fopen($tmpFile, 'w'); + stream_copy_to_stream($stream, $tmpFileStream); + $callback($tmpFile, $ref); + } } } diff --git a/app/Exports/ZipExportModels/ZipExportImage.php b/app/Exports/ZipExportModels/ZipExportImage.php index 540d3d4e55d..39f1d101298 100644 --- a/app/Exports/ZipExportModels/ZipExportImage.php +++ b/app/Exports/ZipExportModels/ZipExportImage.php @@ -2,10 +2,24 @@ namespace BookStack\Exports\ZipExportModels; -use BookStack\Activity\Models\Tag; +use BookStack\Exports\ZipExportFiles; +use BookStack\Uploads\Image; class ZipExportImage extends ZipExportModel { + public ?int $id = null; public string $name; public string $file; + public string $type; + + public static function fromModel(Image $model, ZipExportFiles $files): self + { + $instance = new self(); + $instance->id = $model->id; + $instance->name = $model->name; + $instance->type = $model->type; + $instance->file = $files->referenceForImage($model); + + return $instance; + } } diff --git a/app/Exports/ZipExportReferences.php b/app/Exports/ZipExportReferences.php index 76a7fedbec4..19672db0a0e 100644 --- a/app/Exports/ZipExportReferences.php +++ b/app/Exports/ZipExportReferences.php @@ -3,8 +3,13 @@ namespace BookStack\Exports; use BookStack\App\Model; +use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExportModels\ZipExportAttachment; +use BookStack\Exports\ZipExportModels\ZipExportImage; +use BookStack\Exports\ZipExportModels\ZipExportModel; use BookStack\Exports\ZipExportModels\ZipExportPage; +use BookStack\Uploads\Attachment; +use BookStack\Uploads\Image; class ZipExportReferences { @@ -16,6 +21,9 @@ class ZipExportReferences /** @var ZipExportAttachment[] */ protected array $attachments = []; + /** @var ZipExportImage[] */ + protected array $images = []; + public function __construct( protected ZipReferenceParser $parser, ) { @@ -34,19 +42,12 @@ public function addPage(ZipExportPage $page): void } } - public function buildReferences(): void + public function buildReferences(ZipExportFiles $files): void { - // TODO - References to images, attachments, other entities - // TODO - Parse page MD & HTML foreach ($this->pages as $page) { - $page->html = $this->parser->parse($page->html ?? '', function (Model $model): ?string { - // TODO - Handle found link to $model - // - Validate we can see/access $model, or/and that it's - // part of the export in progress. - - // TODO - Add images after the above to files - return '[CAT]'; + $page->html = $this->parser->parse($page->html ?? '', function (Model $model) use ($files, $page) { + return $this->handleModelReference($model, $page, $files); }); // TODO - markdown } @@ -55,4 +56,45 @@ public function buildReferences(): void // TODO - Parse chapter desc html // TODO - Parse book desc html } + + protected function handleModelReference(Model $model, ZipExportModel $exportModel, ZipExportFiles $files): ?string + { + // TODO - References to other entities + + // Handle attachment references + // No permission check needed here since they would only already exist in this + // reference context if already allowed via their entity access. + if ($model instanceof Attachment) { + if (isset($this->attachments[$model->id])) { + return "[[bsexport:attachment:{$model->id}]]"; + } + return null; + } + + // Handle image references + if ($model instanceof Image) { + // Only handle gallery and drawio images + if ($model->type !== 'gallery' && $model->type !== 'drawio') { + return null; + } + + // We don't expect images to be part of book/chapter content + if (!($exportModel instanceof ZipExportPage)) { + return null; + } + + $page = $model->getPage(); + if ($page && userCan('view', $page)) { + if (!isset($this->images[$model->id])) { + $exportImage = ZipExportImage::fromModel($model, $files); + $this->images[$model->id] = $exportImage; + $exportModel->images[] = $exportImage; + } + return "[[bsexport:image:{$model->id}]]"; + } + return null; + } + + return null; + } } diff --git a/app/Uploads/ImageService.php b/app/Uploads/ImageService.php index 8d8da61ec18..e501cc7b12d 100644 --- a/app/Uploads/ImageService.php +++ b/app/Uploads/ImageService.php @@ -133,6 +133,19 @@ public function getImageData(Image $image): string return $disk->get($image->path); } + /** + * Get the raw data content from an image. + * + * @throws Exception + * @returns ?resource + */ + public function getImageStream(Image $image): mixed + { + $disk = $this->storage->getDisk(); + + return $disk->stream($image->path); + } + /** * Destroy an image along with its revisions, thumbnails and remaining folders. * diff --git a/app/Uploads/ImageStorageDisk.php b/app/Uploads/ImageStorageDisk.php index 798b72abdbf..8df702e0d94 100644 --- a/app/Uploads/ImageStorageDisk.php +++ b/app/Uploads/ImageStorageDisk.php @@ -55,6 +55,15 @@ public function get(string $path): ?string return $this->filesystem->get($this->adjustPathForDisk($path)); } + /** + * Get a stream to the file at the given path. + * @returns ?resource + */ + public function stream(string $path): mixed + { + return $this->filesystem->readStream($this->adjustPathForDisk($path)); + } + /** * Save the given image data at the given path. Can choose to set * the image as public which will update its visibility after saving. diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 7a99563d14b..1ba5872018c 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -46,13 +46,12 @@ This can be done using the following format: [[bsexport::]] ``` -Images and attachments are referenced via their file name within the `files/` directory. -Otherwise, other content types are referenced by `id`. +References are to the `id` for data objects. Here's an example of each type of such reference that could be used: ``` -[[bsexport:image:an-image-path.png]] -[[bsexport:attachment:an-image-path.png]] +[[bsexport:image:22]] +[[bsexport:attachment:55]] [[bsexport:page:40]] [[bsexport:chapter:2]] [[bsexport:book:8]] @@ -121,10 +120,14 @@ The page editor type, and edit content will be determined by what content is pro #### Image +- `id` - Number, optional, original ID for the page from exported system. - `name` - String, required, name of image. - `file` - String reference, required, reference to image file. +- `type` - String, required, must be 'gallery' or 'drawio' -File must be an image type accepted by BookStack (png, jpg, gif, webp) +File must be an image type accepted by BookStack (png, jpg, gif, webp). +Images of type 'drawio' are expected to be png with draw.io drawing data +embedded within it. #### Attachment From f732ef05d5b60a48ce3b42e05155e9e91d92d927 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 23 Oct 2024 10:48:26 +0100 Subject: [PATCH 10/41] ZIP Exports: Reorganised files, added page md parsing --- .../Controllers/PageExportController.php | 2 +- .../Models}/ZipExportAttachment.php | 4 ++-- .../Models}/ZipExportImage.php | 4 ++-- .../Models}/ZipExportModel.php | 2 +- .../Models}/ZipExportPage.php | 4 ++-- .../Models}/ZipExportTag.php | 2 +- .../{ => ZipExports}/ZipExportBuilder.php | 4 ++-- .../{ => ZipExports}/ZipExportFiles.php | 2 +- .../{ => ZipExports}/ZipExportReferences.php | 23 +++++++++++-------- .../{ => ZipExports}/ZipReferenceParser.php | 2 +- 10 files changed, 26 insertions(+), 23 deletions(-) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportAttachment.php (90%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportImage.php (84%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportModel.php (92%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportPage.php (91%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportTag.php (92%) rename app/Exports/{ => ZipExports}/ZipExportBuilder.php (94%) rename app/Exports/{ => ZipExports}/ZipExportFiles.php (98%) rename app/Exports/{ => ZipExports}/ZipExportReferences.php (83%) rename app/Exports/{ => ZipExports}/ZipReferenceParser.php (98%) diff --git a/app/Exports/Controllers/PageExportController.php b/app/Exports/Controllers/PageExportController.php index 01611fd2121..34e67ffcf70 100644 --- a/app/Exports/Controllers/PageExportController.php +++ b/app/Exports/Controllers/PageExportController.php @@ -6,7 +6,7 @@ use BookStack\Entities\Tools\PageContent; use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; -use BookStack\Exports\ZipExportBuilder; +use BookStack\Exports\ZipExports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; diff --git a/app/Exports/ZipExportModels/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php similarity index 90% rename from app/Exports/ZipExportModels/ZipExportAttachment.php rename to app/Exports/ZipExports/Models/ZipExportAttachment.php index d79a16cc109..8c89ae11f14 100644 --- a/app/Exports/ZipExportModels/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -1,8 +1,8 @@ pages as $page) { - $page->html = $this->parser->parse($page->html ?? '', function (Model $model) use ($files, $page) { + $handler = function (Model $model) use ($files, $page) { return $this->handleModelReference($model, $page, $files); - }); - // TODO - markdown + }; + + $page->html = $this->parser->parse($page->html ?? '', $handler); + if ($page->markdown) { + $page->markdown = $this->parser->parse($page->markdown, $handler); + } } // dd('end'); diff --git a/app/Exports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php similarity index 98% rename from app/Exports/ZipReferenceParser.php rename to app/Exports/ZipExports/ZipReferenceParser.php index 820920da28d..4d16dbc6110 100644 --- a/app/Exports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -1,6 +1,6 @@ Date: Wed, 23 Oct 2024 11:30:32 +0100 Subject: [PATCH 11/41] ZIP Exports: Added core logic for books/chapters --- app/Entities/Models/Chapter.php | 1 + .../ZipExports/Models/ZipExportBook.php | 53 ++++++++++++++++ .../ZipExports/Models/ZipExportChapter.php | 45 ++++++++++++++ .../ZipExports/Models/ZipExportPage.php | 12 ++++ app/Exports/ZipExports/ZipExportBuilder.php | 30 +++++++++ .../ZipExports/ZipExportReferences.php | 61 ++++++++++++++++--- dev/docs/portable-zip-file-format.md | 2 +- 7 files changed, 195 insertions(+), 9 deletions(-) create mode 100644 app/Exports/ZipExports/Models/ZipExportBook.php create mode 100644 app/Exports/ZipExports/Models/ZipExportChapter.php diff --git a/app/Entities/Models/Chapter.php b/app/Entities/Models/Chapter.php index c926aaa647a..088d199da67 100644 --- a/app/Entities/Models/Chapter.php +++ b/app/Entities/Models/Chapter.php @@ -60,6 +60,7 @@ public function defaultTemplate(): BelongsTo /** * Get the visible pages in this chapter. + * @returns Collection */ public function getVisiblePages(): Collection { diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php new file mode 100644 index 00000000000..5a0c5806ba8 --- /dev/null +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -0,0 +1,53 @@ +id = $model->id; + $instance->name = $model->name; + $instance->description_html = $model->descriptionHtml(); + + if ($model->cover) { + $instance->cover = $files->referenceForImage($model->cover); + } + + $instance->tags = ZipExportTag::fromModelArray($model->tags()->get()->all()); + + $chapters = []; + $pages = []; + + $children = $model->getDirectVisibleChildren()->all(); + foreach ($children as $child) { + if ($child instanceof Chapter) { + $chapters[] = $child; + } else if ($child instanceof Page) { + $pages[] = $child; + } + } + + $instance->pages = ZipExportPage::fromModelArray($pages, $files); + $instance->chapters = ZipExportChapter::fromModelArray($chapters, $files); + + return $instance; + } +} diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php new file mode 100644 index 00000000000..cd5765f48bc --- /dev/null +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -0,0 +1,45 @@ +id = $model->id; + $instance->name = $model->name; + $instance->description_html = $model->descriptionHtml(); + $instance->priority = $model->priority; + $instance->tags = ZipExportTag::fromModelArray($model->tags()->get()->all()); + + $pages = $model->getVisiblePages()->filter(fn (Page $page) => !$page->draft)->all(); + $instance->pages = ZipExportPage::fromModelArray($pages, $files); + + return $instance; + } + + /** + * @param Chapter[] $chapterArray + * @return self[] + */ + public static function fromModelArray(array $chapterArray, ZipExportFiles $files): array + { + return array_values(array_map(function (Chapter $chapter) use ($files) { + return self::fromModel($chapter, $files); + }, $chapterArray)); + } +} diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index bae46ca8225..8075595f228 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -26,6 +26,7 @@ public static function fromModel(Page $model, ZipExportFiles $files): self $instance->id = $model->id; $instance->name = $model->name; $instance->html = (new PageContent($model))->render(); + $instance->priority = $model->priority; if (!empty($model->markdown)) { $instance->markdown = $model->markdown; @@ -36,4 +37,15 @@ public static function fromModel(Page $model, ZipExportFiles $files): self return $instance; } + + /** + * @param Page[] $pageArray + * @return self[] + */ + public static function fromModelArray(array $pageArray, ZipExportFiles $files): array + { + return array_values(array_map(function (Page $page) use ($files) { + return self::fromModel($page, $files); + }, $pageArray)); + } } diff --git a/app/Exports/ZipExports/ZipExportBuilder.php b/app/Exports/ZipExports/ZipExportBuilder.php index 15edebea574..42fb03541c0 100644 --- a/app/Exports/ZipExports/ZipExportBuilder.php +++ b/app/Exports/ZipExports/ZipExportBuilder.php @@ -2,8 +2,12 @@ namespace BookStack\Exports\ZipExports; +use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exceptions\ZipExportException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; use ZipArchive; @@ -30,6 +34,32 @@ public function buildForPage(Page $page): string return $this->build(); } + /** + * @throws ZipExportException + */ + public function buildForChapter(Chapter $chapter): string + { + $exportChapter = ZipExportChapter::fromModel($chapter, $this->files); + $this->data['chapter'] = $exportChapter; + + $this->references->addChapter($exportChapter); + + return $this->build(); + } + + /** + * @throws ZipExportException + */ + public function buildForBook(Book $book): string + { + $exportBook = ZipExportBook::fromModel($book, $this->files); + $this->data['book'] = $exportBook; + + $this->references->addBook($exportBook); + + return $this->build(); + } + /** * @throws ZipExportException */ diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index c3565aaa383..1fce0fc972e 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -4,6 +4,8 @@ use BookStack\App\Model; use BookStack\Exports\ZipExports\Models\ZipExportAttachment; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportImage; use BookStack\Exports\ZipExports\Models\ZipExportModel; use BookStack\Exports\ZipExports\Models\ZipExportPage; @@ -14,8 +16,10 @@ class ZipExportReferences { /** @var ZipExportPage[] */ protected array $pages = []; - protected array $books = []; + /** @var ZipExportChapter[] */ protected array $chapters = []; + /** @var ZipExportBook[] */ + protected array $books = []; /** @var ZipExportAttachment[] */ protected array $attachments = []; @@ -41,23 +45,64 @@ public function addPage(ZipExportPage $page): void } } + public function addChapter(ZipExportChapter $chapter): void + { + if ($chapter->id) { + $this->chapters[$chapter->id] = $chapter; + } + + foreach ($chapter->pages as $page) { + $this->addPage($page); + } + } + + public function addBook(ZipExportBook $book): void + { + if ($book->id) { + $this->chapters[$book->id] = $book; + } + + foreach ($book->pages as $page) { + $this->addPage($page); + } + + foreach ($book->chapters as $chapter) { + $this->addChapter($chapter); + } + } + public function buildReferences(ZipExportFiles $files): void { - // Parse page content first - foreach ($this->pages as $page) { - $handler = function (Model $model) use ($files, $page) { - return $this->handleModelReference($model, $page, $files); + $createHandler = function (ZipExportModel $zipModel) use ($files) { + return function (Model $model) use ($files, $zipModel) { + return $this->handleModelReference($model, $zipModel, $files); }; + }; + // Parse page content first + foreach ($this->pages as $page) { + $handler = $createHandler($page); $page->html = $this->parser->parse($page->html ?? '', $handler); if ($page->markdown) { $page->markdown = $this->parser->parse($page->markdown, $handler); } } -// dd('end'); - // TODO - Parse chapter desc html - // TODO - Parse book desc html + // Parse chapter description HTML + foreach ($this->chapters as $chapter) { + if ($chapter->description_html) { + $handler = $createHandler($chapter); + $chapter->description_html = $this->parser->parse($chapter->description_html, $handler); + } + } + + // Parse book description HTML + foreach ($this->books as $book) { + if ($book->description_html) { + $handler = $createHandler($book); + $book->description_html = $this->parser->parse($book->description_html, $handler); + } + } } protected function handleModelReference(Model $model, ZipExportModel $exportModel, ZipExportFiles $files): ?string diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 1ba5872018c..6cee7356d23 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -87,7 +87,7 @@ The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This i - `id` - Number, optional, original ID for the book from exported system. - `name` - String, required, name/title of the book. - `description_html` - String, optional, HTML description content. -- `cover` - String reference, options, reference to book cover image. +- `cover` - String reference, optional, reference to book cover image. - `chapters` - [Chapter](#chapter) array, optional, chapters within this book. - `pages` - [Page](#page) array, optional, direct child pages for this book. - `tags` - [Tag](#tag) array, optional, tags assigned to this book. From 484342f26adab723b8c4625d22a8901f5bfe79af Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 23 Oct 2024 15:59:58 +0100 Subject: [PATCH 12/41] ZIP Exports: Added entity cross refs, Started export tests --- .../Controllers/BookExportController.php | 14 +++ .../Controllers/ChapterExportController.php | 13 +++ .../ZipExports/ZipExportReferences.php | 14 ++- routes/web.php | 1 + tests/Exports/ZipExportTest.php | 85 ++++++++++++++++++- tests/Exports/ZipResultData.php | 13 +++ 6 files changed, 136 insertions(+), 4 deletions(-) create mode 100644 tests/Exports/ZipResultData.php diff --git a/app/Exports/Controllers/BookExportController.php b/app/Exports/Controllers/BookExportController.php index 36906b6ad7b..f726175a086 100644 --- a/app/Exports/Controllers/BookExportController.php +++ b/app/Exports/Controllers/BookExportController.php @@ -3,7 +3,9 @@ namespace BookStack\Exports\Controllers; use BookStack\Entities\Queries\BookQueries; +use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; +use BookStack\Exports\ZipExports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; @@ -63,4 +65,16 @@ public function markdown(string $bookSlug) return $this->download()->directly($textContent, $bookSlug . '.md'); } + + /** + * Export a book to a contained ZIP export file. + * @throws NotFoundException + */ + public function zip(string $bookSlug, ZipExportBuilder $builder) + { + $book = $this->queries->findVisibleBySlugOrFail($bookSlug); + $zip = $builder->buildForBook($book); + + return $this->download()->streamedDirectly(fopen($zip, 'r'), $bookSlug . '.zip', filesize($zip)); + } } diff --git a/app/Exports/Controllers/ChapterExportController.php b/app/Exports/Controllers/ChapterExportController.php index d85b90dcb41..0d7a5c0d195 100644 --- a/app/Exports/Controllers/ChapterExportController.php +++ b/app/Exports/Controllers/ChapterExportController.php @@ -5,6 +5,7 @@ use BookStack\Entities\Queries\ChapterQueries; use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; +use BookStack\Exports\ZipExports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; @@ -70,4 +71,16 @@ public function markdown(string $bookSlug, string $chapterSlug) return $this->download()->directly($chapterText, $chapterSlug . '.md'); } + + /** + * Export a book to a contained ZIP export file. + * @throws NotFoundException + */ + public function zip(string $bookSlug, string $chapterSlug, ZipExportBuilder $builder) + { + $chapter = $this->queries->findVisibleBySlugsOrFail($bookSlug, $chapterSlug); + $zip = $builder->buildForChapter($chapter); + + return $this->download()->streamedDirectly(fopen($zip, 'r'), $chapterSlug . '.zip', filesize($zip)); + } } diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index 1fce0fc972e..8b3a4b612fe 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -3,6 +3,9 @@ namespace BookStack\Exports\ZipExports; use BookStack\App\Model; +use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Chapter; +use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\Models\ZipExportAttachment; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; @@ -107,8 +110,6 @@ public function buildReferences(ZipExportFiles $files): void protected function handleModelReference(Model $model, ZipExportModel $exportModel, ZipExportFiles $files): ?string { - // TODO - References to other entities - // Handle attachment references // No permission check needed here since they would only already exist in this // reference context if already allowed via their entity access. @@ -143,6 +144,15 @@ protected function handleModelReference(Model $model, ZipExportModel $exportMode return null; } + // Handle entity references + if ($model instanceof Book && isset($this->books[$model->id])) { + return "[[bsexport:book:{$model->id}]]"; + } else if ($model instanceof Chapter && isset($this->chapters[$model->id])) { + return "[[bsexport:chapter:{$model->id}]]"; + } else if ($model instanceof Page && isset($this->pages[$model->id])) { + return "[[bsexport:page:{$model->id}]]"; + } + return null; } } diff --git a/routes/web.php b/routes/web.php index 6ae70983d52..e6f3683c643 100644 --- a/routes/web.php +++ b/routes/web.php @@ -132,6 +132,7 @@ Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/html', [ExportControllers\ChapterExportController::class, 'html']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/markdown', [ExportControllers\ChapterExportController::class, 'markdown']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/plaintext', [ExportControllers\ChapterExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/zip', [ExportControllers\ChapterExportController::class, 'zip']); Route::put('/books/{bookSlug}/chapter/{chapterSlug}/permissions', [PermissionsController::class, 'updateForChapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/references', [ReferenceController::class, 'chapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/delete', [EntityControllers\ChapterController::class, 'showDelete']); diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index d8ce00be33f..536e23806f6 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -2,14 +2,95 @@ namespace Tests\Exports; -use BookStack\Entities\Models\Book; +use Illuminate\Support\Carbon; +use Illuminate\Testing\TestResponse; use Tests\TestCase; +use ZipArchive; class ZipExportTest extends TestCase { - public function test_page_export() + public function test_export_results_in_zip_format() + { + $page = $this->entities->page(); + $response = $this->asEditor()->get($page->getUrl("/export/zip")); + + $zipData = $response->streamedContent(); + $zipFile = tempnam(sys_get_temp_dir(), 'bstesta-'); + file_put_contents($zipFile, $zipData); + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::RDONLY); + + $this->assertNotFalse($zip->locateName('data.json')); + $this->assertNotFalse($zip->locateName('files/')); + + $data = json_decode($zip->getFromName('data.json'), true); + $this->assertIsArray($data); + $this->assertGreaterThan(0, count($data)); + + $zip->close(); + unlink($zipFile); + } + + public function test_export_metadata() { $page = $this->entities->page(); + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $this->assertEquals($page->id, $zip->data['page']['id'] ?? null); + $this->assertArrayNotHasKey('book', $zip->data); + $this->assertArrayNotHasKey('chapter', $zip->data); + + $now = time(); + $date = Carbon::parse($zip->data['exported_at'])->unix(); + $this->assertLessThan($now + 2, $date); + $this->assertGreaterThan($now - 2, $date); + + $version = trim(file_get_contents(base_path('version'))); + $this->assertEquals($version, $zip->data['instance']['version']); + + $instanceId = decrypt($zip->data['instance']['id_ciphertext']); + $this->assertEquals('bookstack', $instanceId); + } + + public function test_page_export() + { + // TODO + } + + public function test_book_export() + { + // TODO + } + + public function test_chapter_export() + { // TODO } + + protected function extractZipResponse(TestResponse $response): ZipResultData + { + $zipData = $response->streamedContent(); + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + + file_put_contents($zipFile, $zipData); + $extractDir = tempnam(sys_get_temp_dir(), 'bstestextracted-'); + if (file_exists($extractDir)) { + unlink($extractDir); + } + mkdir($extractDir); + + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::RDONLY); + $zip->extractTo($extractDir); + + $dataJson = file_get_contents($extractDir . DIRECTORY_SEPARATOR . "data.json"); + $data = json_decode($dataJson, true); + + return new ZipResultData( + $zipFile, + $extractDir, + $data, + ); + } } diff --git a/tests/Exports/ZipResultData.php b/tests/Exports/ZipResultData.php new file mode 100644 index 00000000000..b5cc2b4ca61 --- /dev/null +++ b/tests/Exports/ZipResultData.php @@ -0,0 +1,13 @@ + Date: Sun, 27 Oct 2024 14:33:43 +0000 Subject: [PATCH 13/41] ZIP Exports: Tested each type and model of export --- .../ZipExports/Models/ZipExportAttachment.php | 1 + .../ZipExports/ZipExportReferences.php | 2 +- app/Exports/ZipExports/ZipReferenceParser.php | 2 +- tests/Exports/ZipExportTest.php | 265 +++++++++++++++++- tests/Exports/ZipResultData.php | 9 + 5 files changed, 274 insertions(+), 5 deletions(-) diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index 8c89ae11f14..283ffa751c9 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -18,6 +18,7 @@ public static function fromModel(Attachment $model, ZipExportFiles $files): self $instance = new self(); $instance->id = $model->id; $instance->name = $model->name; + $instance->order = $model->order; if ($model->external) { $instance->link = $model->path; diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index 8b3a4b612fe..c630c832b37 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -62,7 +62,7 @@ public function addChapter(ZipExportChapter $chapter): void public function addBook(ZipExportBook $book): void { if ($book->id) { - $this->chapters[$book->id] = $book; + $this->books[$book->id] = $book; } foreach ($book->pages as $page) { diff --git a/app/Exports/ZipExports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php index 4d16dbc6110..da43d1b366b 100644 --- a/app/Exports/ZipExports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -38,7 +38,7 @@ public function __construct(EntityQueries $queries) public function parse(string $content, callable $handler): string { $escapedBase = preg_quote(url('/'), '/'); - $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#]/"; + $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#()]/"; $matches = []; preg_match_all($linkRegex, $content, $matches); diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 536e23806f6..ac07b33aef5 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -2,6 +2,11 @@ namespace Tests\Exports; +use BookStack\Activity\Models\Tag; +use BookStack\Entities\Repos\BookRepo; +use BookStack\Entities\Tools\PageContent; +use BookStack\Uploads\Attachment; +use BookStack\Uploads\Image; use Illuminate\Support\Carbon; use Illuminate\Testing\TestResponse; use Tests\TestCase; @@ -55,17 +60,271 @@ public function test_export_metadata() public function test_page_export() { - // TODO + $page = $this->entities->page(); + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertEquals([ + 'id' => $page->id, + 'name' => $page->name, + 'html' => (new PageContent($page))->render(), + 'priority' => $page->priority, + 'attachments' => [], + 'images' => [], + 'tags' => [], + ], $pageData); + } + + public function test_page_export_with_markdown() + { + $page = $this->entities->page(); + $markdown = "# My page\n\nwritten in markdown for export\n"; + $page->markdown = $markdown; + $page->save(); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertEquals($markdown, $pageData['markdown']); + $this->assertNotEmpty($pageData['html']); + } + + public function test_page_export_with_tags() + { + $page = $this->entities->page(); + $page->tags()->saveMany([ + new Tag(['name' => 'Exporty', 'value' => 'Content', 'order' => 1]), + new Tag(['name' => 'Another', 'value' => '', 'order' => 2]), + ]); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertEquals([ + [ + 'name' => 'Exporty', + 'value' => 'Content', + 'order' => 1, + ], + [ + 'name' => 'Another', + 'value' => '', + 'order' => 2, + ] + ], $pageData['tags']); + } + + public function test_page_export_with_images() + { + $this->asEditor(); + $page = $this->entities->page(); + $result = $this->files->uploadGalleryImageToPage($this, $page); + $displayThumb = $result['response']->thumbs->gallery ?? ''; + $page->html = '

    My image

    '; + $page->save(); + $image = Image::findOrFail($result['response']->id); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $this->assertCount(1, $pageData['images']); + $imageData = $pageData['images'][0]; + $this->assertEquals($image->id, $imageData['id']); + $this->assertEquals($image->name, $imageData['name']); + $this->assertEquals('gallery', $imageData['type']); + $this->assertNotEmpty($imageData['file']); + + $filePath = $zip->extractPath("files/{$imageData['file']}"); + $this->assertFileExists($filePath); + $this->assertEquals(file_get_contents(public_path($image->path)), file_get_contents($filePath)); + + $this->assertEquals('

    My image

    ', $pageData['html']); + } + + public function test_page_export_file_attachments() + { + $contents = 'My great attachment content!'; + + $page = $this->entities->page(); + $this->asAdmin(); + $attachment = $this->files->uploadAttachmentDataToPage($this, $page, 'PageAttachmentExport.txt', $contents, 'text/plain'); + + $zipResp = $this->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertCount(1, $pageData['attachments']); + + $attachmentData = $pageData['attachments'][0]; + $this->assertEquals('PageAttachmentExport.txt', $attachmentData['name']); + $this->assertEquals($attachment->id, $attachmentData['id']); + $this->assertEquals(1, $attachmentData['order']); + $this->assertArrayNotHasKey('link', $attachmentData); + $this->assertNotEmpty($attachmentData['file']); + + $fileRef = $attachmentData['file']; + $filePath = $zip->extractPath("/files/$fileRef"); + $this->assertFileExists($filePath); + $this->assertEquals($contents, file_get_contents($filePath)); + } + + public function test_page_export_link_attachments() + { + $page = $this->entities->page(); + $this->asEditor(); + $attachment = Attachment::factory()->create([ + 'name' => 'My link attachment for export', + 'path' => 'https://example.com/cats', + 'external' => true, + 'uploaded_to' => $page->id, + 'order' => 1, + ]); + + $zipResp = $this->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertCount(1, $pageData['attachments']); + + $attachmentData = $pageData['attachments'][0]; + $this->assertEquals('My link attachment for export', $attachmentData['name']); + $this->assertEquals($attachment->id, $attachmentData['id']); + $this->assertEquals(1, $attachmentData['order']); + $this->assertEquals('https://example.com/cats', $attachmentData['link']); + $this->assertArrayNotHasKey('file', $attachmentData); } public function test_book_export() { - // TODO + $book = $this->entities->book(); + $book->tags()->saveMany(Tag::factory()->count(2)->make()); + + $zipResp = $this->asEditor()->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $this->assertArrayHasKey('book', $zip->data); + + $bookData = $zip->data['book']; + $this->assertEquals($book->id, $bookData['id']); + $this->assertEquals($book->name, $bookData['name']); + $this->assertEquals($book->descriptionHtml(), $bookData['description_html']); + $this->assertCount(2, $bookData['tags']); + $this->assertCount($book->directPages()->count(), $bookData['pages']); + $this->assertCount($book->chapters()->count(), $bookData['chapters']); + $this->assertArrayNotHasKey('cover', $bookData); + } + + public function test_book_export_with_cover_image() + { + $book = $this->entities->book(); + $bookRepo = $this->app->make(BookRepo::class); + $coverImageFile = $this->files->uploadedImage('cover.png'); + $bookRepo->updateCoverImage($book, $coverImageFile); + $coverImage = $book->cover()->first(); + + $zipResp = $this->asEditor()->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $this->assertArrayHasKey('cover', $zip->data['book']); + $coverRef = $zip->data['book']['cover']; + $coverPath = $zip->extractPath("/files/$coverRef"); + $this->assertFileExists($coverPath); + $this->assertEquals(file_get_contents(public_path($coverImage->path)), file_get_contents($coverPath)); } public function test_chapter_export() { - // TODO + $chapter = $this->entities->chapter(); + $chapter->tags()->saveMany(Tag::factory()->count(2)->make()); + + $zipResp = $this->asEditor()->get($chapter->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $this->assertArrayHasKey('chapter', $zip->data); + + $chapterData = $zip->data['chapter']; + $this->assertEquals($chapter->id, $chapterData['id']); + $this->assertEquals($chapter->name, $chapterData['name']); + $this->assertEquals($chapter->descriptionHtml(), $chapterData['description_html']); + $this->assertCount(2, $chapterData['tags']); + $this->assertEquals($chapter->priority, $chapterData['priority']); + $this->assertCount($chapter->pages()->count(), $chapterData['pages']); + } + + + public function test_cross_reference_links_are_converted() + { + $book = $this->entities->bookHasChaptersAndPages(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + + $book->description_html = '

    Link to chapter

    '; + $book->save(); + $chapter->description_html = '

    Link to page

    '; + $chapter->save(); + $page->html = '

    Link to book

    '; + $page->save(); + + $zipResp = $this->asEditor()->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $bookData = $zip->data['book']; + $chapterData = $bookData['chapters'][0]; + $pageData = $chapterData['pages'][0]; + + $this->assertStringContainsString('href="[[bsexport:chapter:' . $chapter->id . ']]"', $bookData['description_html']); + $this->assertStringContainsString('href="[[bsexport:page:' . $page->id . ']]#section2"', $chapterData['description_html']); + $this->assertStringContainsString('href="[[bsexport:book:' . $book->id . ']]?view=true"', $pageData['html']); + } + + public function test_cross_reference_links_external_to_export_are_not_converted() + { + $page = $this->entities->page(); + $page->html = '

    Link to book

    '; + $page->save(); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $this->assertStringContainsString('href="' . $page->book->getUrl() . '"', $pageData['html']); + } + + public function test_attachments_links_are_converted() + { + $page = $this->entities->page(); + $attachment = Attachment::factory()->create([ + 'name' => 'My link attachment for export reference', + 'path' => 'https://example.com/cats/ref', + 'external' => true, + 'uploaded_to' => $page->id, + 'order' => 1, + ]); + + $page->html = '

    id}") . '?open=true">Link to attachment

    '; + $page->save(); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $this->assertStringContainsString('href="[[bsexport:attachment:' . $attachment->id . ']]?open=true"', $pageData['html']); + } + + public function test_links_in_markdown_are_parsed() + { + $chapter = $this->entities->chapterHasPages(); + $page = $chapter->pages()->first(); + + $page->markdown = "[Link to chapter]({$chapter->getUrl()})"; + $page->save(); + + $zipResp = $this->asEditor()->get($chapter->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['chapter']['pages'][0]; + + $this->assertStringContainsString("[Link to chapter]([[bsexport:chapter:{$chapter->id}]])", $pageData['markdown']); } protected function extractZipResponse(TestResponse $response): ZipResultData diff --git a/tests/Exports/ZipResultData.php b/tests/Exports/ZipResultData.php index b5cc2b4ca61..7725004c7be 100644 --- a/tests/Exports/ZipResultData.php +++ b/tests/Exports/ZipResultData.php @@ -10,4 +10,13 @@ public function __construct( public array $data, ) { } + + /** + * Build a path to a location the extracted content, using the given relative $path. + */ + public function extractPath(string $path): string + { + $relPath = implode(DIRECTORY_SEPARATOR, explode('/', $path)); + return $this->extractedDirPath . DIRECTORY_SEPARATOR . ltrim($relPath, DIRECTORY_SEPARATOR); + } } From 4051d5b8037119b382c576042bc668b8f00eee14 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 29 Oct 2024 12:11:51 +0000 Subject: [PATCH 14/41] ZIP Exports: Added new import permission Also updated new route/view to new non-book-specific flow. Also fixed down migration of old export permissions migration. --- app/Exports/Controllers/ImportController.php | 24 ++++++++ ...8_28_161743_add_export_role_permission.php | 7 ++- ...0_29_114420_add_import_role_permission.php | 61 +++++++++++++++++++ lang/en/entities.php | 1 + lang/en/settings.php | 1 + resources/views/books/index.blade.php | 7 +++ resources/views/exports/import.blade.php | 34 +++++++++++ .../views/settings/roles/parts/form.blade.php | 1 + routes/web.php | 4 ++ 9 files changed, 137 insertions(+), 3 deletions(-) create mode 100644 app/Exports/Controllers/ImportController.php create mode 100644 database/migrations/2024_10_29_114420_add_import_role_permission.php create mode 100644 resources/views/exports/import.blade.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php new file mode 100644 index 00000000000..acc803a0f0d --- /dev/null +++ b/app/Exports/Controllers/ImportController.php @@ -0,0 +1,24 @@ +middleware('can:content-import'); + } + + public function start(Request $request) + { + return view('exports.import'); + } + + public function upload(Request $request) + { + // TODO + } +} diff --git a/database/migrations/2021_08_28_161743_add_export_role_permission.php b/database/migrations/2021_08_28_161743_add_export_role_permission.php index 21f45aa0691..99416f9fcf6 100644 --- a/database/migrations/2021_08_28_161743_add_export_role_permission.php +++ b/database/migrations/2021_08_28_161743_add_export_role_permission.php @@ -11,8 +11,7 @@ */ public function up(): void { - // Create new templates-manage permission and assign to admin role - $roles = DB::table('roles')->get('id'); + // Create new content-export permission $permissionId = DB::table('role_permissions')->insertGetId([ 'name' => 'content-export', 'display_name' => 'Export Content', @@ -20,6 +19,7 @@ public function up(): void 'updated_at' => Carbon::now()->toDateTimeString(), ]); + $roles = DB::table('roles')->get('id'); $permissionRoles = $roles->map(function ($role) use ($permissionId) { return [ 'role_id' => $role->id, @@ -27,6 +27,7 @@ public function up(): void ]; })->values()->toArray(); + // Assign to all existing roles in the system DB::table('permission_role')->insert($permissionRoles); } @@ -40,6 +41,6 @@ public function down(): void ->where('name', '=', 'content-export')->first(); DB::table('permission_role')->where('permission_id', '=', $contentExportPermission->id)->delete(); - DB::table('role_permissions')->where('id', '=', 'content-export')->delete(); + DB::table('role_permissions')->where('id', '=', $contentExportPermission->id)->delete(); } }; diff --git a/database/migrations/2024_10_29_114420_add_import_role_permission.php b/database/migrations/2024_10_29_114420_add_import_role_permission.php new file mode 100644 index 00000000000..17bbe4cff26 --- /dev/null +++ b/database/migrations/2024_10_29_114420_add_import_role_permission.php @@ -0,0 +1,61 @@ +insertGetId([ + 'name' => 'content-import', + 'display_name' => 'Import Content', + 'created_at' => Carbon::now()->toDateTimeString(), + 'updated_at' => Carbon::now()->toDateTimeString(), + ]); + + // Get existing admin-level role ids + $settingManagePermission = DB::table('role_permissions') + ->where('name', '=', 'settings-manage')->first(); + + if (!$settingManagePermission) { + return; + } + + $adminRoleIds = DB::table('permission_role') + ->where('permission_id', '=', $settingManagePermission->id) + ->pluck('role_id')->all(); + + // Assign the new permission to all existing admins + $newPermissionRoles = array_values(array_map(function ($roleId) use ($permissionId) { + return [ + 'role_id' => $roleId, + 'permission_id' => $permissionId, + ]; + }, $adminRoleIds)); + + DB::table('permission_role')->insert($newPermissionRoles); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + // Remove content-import permission + $importPermission = DB::table('role_permissions') + ->where('name', '=', 'content-import')->first(); + + if (!$importPermission) { + return; + } + + DB::table('permission_role')->where('permission_id', '=', $importPermission->id)->delete(); + DB::table('role_permissions')->where('id', '=', $importPermission->id)->delete(); + } +}; diff --git a/lang/en/entities.php b/lang/en/entities.php index 7e5a708ef62..1a61b629a72 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -43,6 +43,7 @@ 'default_template' => 'Default Page Template', 'default_template_explain' => 'Assign a page template that will be used as the default content for all pages created within this item. Keep in mind this will only be used if the page creator has view access to the chosen template page.', 'default_template_select' => 'Select a template page', + 'import' => 'Import', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/lang/en/settings.php b/lang/en/settings.php index 5427cb9419e..c0b6b692a57 100644 --- a/lang/en/settings.php +++ b/lang/en/settings.php @@ -162,6 +162,7 @@ 'role_access_api' => 'Access system API', 'role_manage_settings' => 'Manage app settings', 'role_export_content' => 'Export content', + 'role_import_content' => 'Import content', 'role_editor_change' => 'Change page editor', 'role_notifications' => 'Receive & manage notifications', 'role_asset' => 'Asset Permissions', diff --git a/resources/views/books/index.blade.php b/resources/views/books/index.blade.php index 0b407a8609a..418c0fea8d1 100644 --- a/resources/views/books/index.blade.php +++ b/resources/views/books/index.blade.php @@ -49,6 +49,13 @@ @icon('tag') {{ trans('entities.tags_view_tags') }} + + @if(userCan('content-import')) + + @icon('upload') + {{ trans('entities.import') }} + + @endif diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php new file mode 100644 index 00000000000..df8f705cb45 --- /dev/null +++ b/resources/views/exports/import.blade.php @@ -0,0 +1,34 @@ +@extends('layouts.simple') + +@section('body') + +
    + +
    +
    +
    +

    {{ trans('entities.import') }}

    +

    + TODO - Desc +{{-- {{ trans('entities.permissions_desc') }}--}} +

    +
    +
    +
    + {{ csrf_field() }} +
    +
    + @include('form.checkbox', ['name' => 'images', 'label' => 'Include Images']) + @include('form.checkbox', ['name' => 'attachments', 'label' => 'Include Attachments']) +
    +
    + +
    + {{ trans('common.cancel') }} + +
    +
    +
    +
    + +@stop diff --git a/resources/views/settings/roles/parts/form.blade.php b/resources/views/settings/roles/parts/form.blade.php index 9fa76f2bfd7..a77b80e4c69 100644 --- a/resources/views/settings/roles/parts/form.blade.php +++ b/resources/views/settings/roles/parts/form.blade.php @@ -37,6 +37,7 @@
    @include('settings.roles.parts.checkbox', ['permission' => 'templates-manage', 'label' => trans('settings.role_manage_page_templates')])
    @include('settings.roles.parts.checkbox', ['permission' => 'access-api', 'label' => trans('settings.role_access_api')])
    @include('settings.roles.parts.checkbox', ['permission' => 'content-export', 'label' => trans('settings.role_export_content')])
    +
    @include('settings.roles.parts.checkbox', ['permission' => 'content-import', 'label' => trans('settings.role_import_content')])
    @include('settings.roles.parts.checkbox', ['permission' => 'editor-change', 'label' => trans('settings.role_editor_change')])
    @include('settings.roles.parts.checkbox', ['permission' => 'receive-notifications', 'label' => trans('settings.role_notifications')])
    diff --git a/routes/web.php b/routes/web.php index e6f3683c643..91aab13fecf 100644 --- a/routes/web.php +++ b/routes/web.php @@ -206,6 +206,10 @@ // Watching Route::put('/watching/update', [ActivityControllers\WatchController::class, 'update']); + // Importing + Route::get('/import', [ExportControllers\ImportController::class, 'start']); + Route::post('/import', [ExportControllers\ImportController::class, 'upload']); + // Other Pages Route::get('/', [HomeController::class, 'index']); Route::get('/home', [HomeController::class, 'index']); From a56a28fbb7eaff40a639c2d06f56de255cd654ea Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 29 Oct 2024 14:21:32 +0000 Subject: [PATCH 15/41] ZIP Exports: Built out initial import view Added syles for non-custom, non-image file inputs. Started planning out back-end handling. --- app/Exports/Controllers/ImportController.php | 8 ++++- lang/en/entities.php | 1 + resources/sass/_forms.scss | 37 ++++++++++++++++++++ resources/views/exports/import.blade.php | 31 ++++++++-------- 4 files changed, 62 insertions(+), 15 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index acc803a0f0d..9eefb097438 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -14,11 +14,17 @@ public function __construct() public function start(Request $request) { + // TODO - Show existing imports for user (or for all users if admin-level user) + return view('exports.import'); } public function upload(Request $request) { - // TODO + // TODO - Read existing ZIP upload and send through validator + // TODO - If invalid, return user with errors + // TODO - Upload to storage + // TODO - Store info/results from validator + // TODO - Send user to next import stage } } diff --git a/lang/en/entities.php b/lang/en/entities.php index 1a61b629a72..45ca4cf6b31 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -44,6 +44,7 @@ 'default_template_explain' => 'Assign a page template that will be used as the default content for all pages created within this item. Keep in mind this will only be used if the page creator has view access to the chosen template page.', 'default_template_select' => 'Select a template page', 'import' => 'Import', + 'import_validate' => 'Validate Import', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/resources/sass/_forms.scss b/resources/sass/_forms.scss index 67df4171499..1c679aaa0dd 100644 --- a/resources/sass/_forms.scss +++ b/resources/sass/_forms.scss @@ -545,6 +545,43 @@ input[type=color] { outline: 1px solid var(--color-primary); } +.custom-simple-file-input { + max-width: 100%; + border: 1px solid; + @include lightDark(border-color, #DDD, #666); + border-radius: 4px; + padding: $-s $-m; +} +.custom-simple-file-input::file-selector-button { + background-color: transparent; + text-decoration: none; + font-size: 0.8rem; + line-height: 1.4em; + padding: $-xs $-s; + border: 1px solid; + font-weight: 400; + outline: 0; + border-radius: 4px; + cursor: pointer; + margin-right: $-m; + @include lightDark(color, #666, #AAA); + @include lightDark(border-color, #CCC, #666); + &:hover, &:focus, &:active { + @include lightDark(color, #444, #BBB); + border: 1px solid #CCC; + box-shadow: 0 1px 4px 0 rgba(0, 0, 0, 0.1); + background-color: #F2F2F2; + @include lightDark(background-color, #f8f8f8, #444); + filter: none; + } + &:active { + border-color: #BBB; + background-color: #DDD; + color: #666; + box-shadow: inset 0 0 2px rgba(0, 0, 0, 0.1); + } +} + input.shortcut-input { width: auto; max-width: 120px; diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index df8f705cb45..b7030f11478 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -5,27 +5,30 @@
    -
    -
    -

    {{ trans('entities.import') }}

    -

    - TODO - Desc -{{-- {{ trans('entities.permissions_desc') }}--}} -

    -
    -
    +

    {{ trans('entities.import') }}

    {{ csrf_field() }} -
    -
    - @include('form.checkbox', ['name' => 'images', 'label' => 'Include Images']) - @include('form.checkbox', ['name' => 'attachments', 'label' => 'Include Attachments']) +
    +

    + Import content using a portable zip export from the same, or a different, instance. + Select a ZIP file to import then press "Validate Import" to proceed. + After the file has been uploaded and validated you'll be able to configure & confirm the import in the next view. +

    +
    +
    + + +
    {{ trans('common.cancel') }} - +
    From b50b7b667d2266950baa56457f2ed8b7eeda273d Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 30 Oct 2024 13:13:41 +0000 Subject: [PATCH 16/41] ZIP Exports: Started import validation --- .../ZipExportValidationException.php | 12 ++++ app/Exports/Controllers/ImportController.php | 6 ++ .../ZipExports/Models/ZipExportAttachment.php | 14 +++++ .../ZipExports/Models/ZipExportModel.php | 9 +++ .../ZipExports/Models/ZipExportTag.php | 12 ++++ app/Exports/ZipExports/ZipExportValidator.php | 63 +++++++++++++++++++ .../ZipExports/ZipFileReferenceRule.php | 26 ++++++++ .../ZipExports/ZipValidationHelper.php | 32 ++++++++++ lang/en/validation.php | 2 + resources/views/exports/import.blade.php | 2 +- 10 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 app/Exceptions/ZipExportValidationException.php create mode 100644 app/Exports/ZipExports/ZipExportValidator.php create mode 100644 app/Exports/ZipExports/ZipFileReferenceRule.php create mode 100644 app/Exports/ZipExports/ZipValidationHelper.php diff --git a/app/Exceptions/ZipExportValidationException.php b/app/Exceptions/ZipExportValidationException.php new file mode 100644 index 00000000000..2ed567d6343 --- /dev/null +++ b/app/Exceptions/ZipExportValidationException.php @@ -0,0 +1,12 @@ +validate($request, [ + 'file' => ['required', 'file'] + ]); + + $file = $request->file('file'); + $file->getRealPath(); // TODO - Read existing ZIP upload and send through validator // TODO - If invalid, return user with errors // TODO - Upload to storage diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index 283ffa751c9..ab1f5ab7559 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -3,6 +3,7 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; use BookStack\Uploads\Attachment; class ZipExportAttachment extends ZipExportModel @@ -35,4 +36,17 @@ public static function fromModelArray(array $attachmentArray, ZipExportFiles $fi return self::fromModel($attachment, $files); }, $attachmentArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'order' => ['nullable', 'integer'], + 'link' => ['required_without:file', 'nullable', 'string'], + 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], + ]; + + return $context->validateArray($data, $rules); + } } diff --git a/app/Exports/ZipExports/Models/ZipExportModel.php b/app/Exports/ZipExports/Models/ZipExportModel.php index 8d0c0a4370b..4d66f010f86 100644 --- a/app/Exports/ZipExports/Models/ZipExportModel.php +++ b/app/Exports/ZipExports/Models/ZipExportModel.php @@ -2,6 +2,7 @@ namespace BookStack\Exports\ZipExports\Models; +use BookStack\Exports\ZipExports\ZipValidationHelper; use JsonSerializable; abstract class ZipExportModel implements JsonSerializable @@ -17,4 +18,12 @@ public function jsonSerialize(): array $publicProps = get_object_vars(...)->__invoke($this); return array_filter($publicProps, fn ($value) => $value !== null); } + + /** + * Validate the given array of data intended for this model. + * Return an array of validation errors messages. + * Child items can be considered in the validation result by returning a keyed + * item in the array for its own validation messages. + */ + abstract public static function validate(ZipValidationHelper $context, array $data): array; } diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index d4e3c4290b3..ad17d5a33c6 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -3,6 +3,7 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Activity\Models\Tag; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportTag extends ZipExportModel { @@ -24,4 +25,15 @@ public static function fromModelArray(array $tagArray): array { return array_values(array_map(self::fromModel(...), $tagArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'name' => ['required', 'string', 'min:1'], + 'value' => ['nullable', 'string'], + 'order' => ['nullable', 'integer'], + ]; + + return $context->validateArray($data, $rules); + } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php new file mode 100644 index 00000000000..5ad9272de43 --- /dev/null +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -0,0 +1,63 @@ +zipPath) || !is_readable($this->zipPath)) { + $this->throwErrors("Could not read ZIP file"); + } + + // Validate file is valid zip + $zip = new \ZipArchive(); + $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); + if ($opened !== true) { + $this->throwErrors("Could not read ZIP file"); + } + + // Validate json data exists, including metadata + $jsonData = $zip->getFromName('data.json') ?: ''; + $importData = json_decode($jsonData, true); + if (!$importData) { + $this->throwErrors("Could not decode ZIP data.json content"); + } + + if (isset($importData['book'])) { + // TODO - Validate book + } else if (isset($importData['chapter'])) { + // TODO - Validate chapter + } else if (isset($importData['page'])) { + // TODO - Validate page + } else { + $this->throwErrors("ZIP file has no book, chapter or page data"); + } + } + + /** + * @throws ZipExportValidationException + */ + protected function throwErrors(...$errorsToAdd): never + { + array_push($this->errors, ...$errorsToAdd); + throw new ZipExportValidationException($this->errors); + } +} diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php new file mode 100644 index 00000000000..4f942e0e789 --- /dev/null +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -0,0 +1,26 @@ +context->zipFileExists($value)) { + $fail('validation.zip_file')->translate(); + } + } +} diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php new file mode 100644 index 00000000000..dd41e6f8b25 --- /dev/null +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -0,0 +1,32 @@ +validationFactory = app(Factory::class); + } + + public function validateArray(array $data, array $rules): array + { + return $this->validationFactory->make($data, $rules)->errors()->messages(); + } + + public function zipFileExists(string $name): bool + { + return $this->zip->statName("files/{$name}") !== false; + } + + public function fileReferenceRule(): ZipFileReferenceRule + { + return new ZipFileReferenceRule($this); + } +} diff --git a/lang/en/validation.php b/lang/en/validation.php index 2a676c7c4cc..6971edc023a 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -105,6 +105,8 @@ 'url' => 'The :attribute format is invalid.', 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', + 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + // Custom validation lines 'custom' => [ 'password-confirm' => [ diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index b7030f11478..9fe596d8888 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -10,7 +10,7 @@ {{ csrf_field() }}

    - Import content using a portable zip export from the same, or a different, instance. + Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you'll be able to configure & confirm the import in the next view.

    From c4ec50d437e52ccd831b6fb2e43baa5cf255fd1a Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 30 Oct 2024 15:26:23 +0000 Subject: [PATCH 17/41] ZIP Exports: Got zip format validation functionally complete --- .../ZipExportValidationException.php | 12 ----- app/Exports/Controllers/ImportController.php | 9 +++- .../ZipExports/Models/ZipExportAttachment.php | 2 +- .../ZipExports/Models/ZipExportBook.php | 21 ++++++++ .../ZipExports/Models/ZipExportChapter.php | 19 +++++++ .../ZipExports/Models/ZipExportImage.php | 14 +++++ .../ZipExports/Models/ZipExportPage.php | 22 ++++++++ .../ZipExports/Models/ZipExportTag.php | 2 +- app/Exports/ZipExports/ZipExportValidator.php | 53 +++++++++++-------- .../ZipExports/ZipValidationHelper.php | 31 ++++++++++- lang/en/validation.php | 3 +- resources/views/exports/import.blade.php | 3 +- 12 files changed, 149 insertions(+), 42 deletions(-) delete mode 100644 app/Exceptions/ZipExportValidationException.php diff --git a/app/Exceptions/ZipExportValidationException.php b/app/Exceptions/ZipExportValidationException.php deleted file mode 100644 index 2ed567d6343..00000000000 --- a/app/Exceptions/ZipExportValidationException.php +++ /dev/null @@ -1,12 +0,0 @@ -file('file'); - $file->getRealPath(); + $zipPath = $file->getRealPath(); + + $errors = (new ZipExportValidator($zipPath))->validate(); + if ($errors) { + dd($errors); + } + dd('passed'); // TODO - Read existing ZIP upload and send through validator // TODO - If invalid, return user with errors // TODO - Upload to storage diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index ab1f5ab7559..e586b91b0ee 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -47,6 +47,6 @@ public static function validate(ZipValidationHelper $context, array $data): arra 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], ]; - return $context->validateArray($data, $rules); + return $context->validateData($data, $rules); } } diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 5a0c5806ba8..7e1f2d8106e 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -6,6 +6,7 @@ use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportBook extends ZipExportModel { @@ -50,4 +51,24 @@ public static function fromModel(Book $model, ZipExportFiles $files): self return $instance; } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'description_html' => ['nullable', 'string'], + 'cover' => ['nullable', 'string', $context->fileReferenceRule()], + 'tags' => ['array'], + 'pages' => ['array'], + 'chapters' => ['array'], + ]; + + $errors = $context->validateData($data, $rules); + $errors['tags'] = $context->validateRelations($data['tags'] ?? [], ZipExportTag::class); + $errors['pages'] = $context->validateRelations($data['pages'] ?? [], ZipExportPage::class); + $errors['chapters'] = $context->validateRelations($data['chapters'] ?? [], ZipExportChapter::class); + + return $errors; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index cd5765f48bc..03df31b7078 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -5,6 +5,7 @@ use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportChapter extends ZipExportModel { @@ -42,4 +43,22 @@ public static function fromModelArray(array $chapterArray, ZipExportFiles $files return self::fromModel($chapter, $files); }, $chapterArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'description_html' => ['nullable', 'string'], + 'priority' => ['nullable', 'int'], + 'tags' => ['array'], + 'pages' => ['array'], + ]; + + $errors = $context->validateData($data, $rules); + $errors['tags'] = $context->validateRelations($data['tags'] ?? [], ZipExportTag::class); + $errors['pages'] = $context->validateRelations($data['pages'] ?? [], ZipExportPage::class); + + return $errors; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 05d828734a0..3388c66df36 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -3,7 +3,9 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; use BookStack\Uploads\Image; +use Illuminate\Validation\Rule; class ZipExportImage extends ZipExportModel { @@ -22,4 +24,16 @@ public static function fromModel(Image $model, ZipExportFiles $files): self return $instance; } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'file' => ['required', 'string', $context->fileReferenceRule()], + 'type' => ['required', 'string', Rule::in(['gallery', 'drawio'])], + ]; + + return $context->validateData($data, $rules); + } } diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 8075595f228..2c8b9a88abd 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -5,6 +5,7 @@ use BookStack\Entities\Models\Page; use BookStack\Entities\Tools\PageContent; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportPage extends ZipExportModel { @@ -48,4 +49,25 @@ public static function fromModelArray(array $pageArray, ZipExportFiles $files): return self::fromModel($page, $files); }, $pageArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'html' => ['nullable', 'string'], + 'markdown' => ['nullable', 'string'], + 'priority' => ['nullable', 'int'], + 'attachments' => ['array'], + 'images' => ['array'], + 'tags' => ['array'], + ]; + + $errors = $context->validateData($data, $rules); + $errors['attachments'] = $context->validateRelations($data['attachments'] ?? [], ZipExportAttachment::class); + $errors['images'] = $context->validateRelations($data['images'] ?? [], ZipExportImage::class); + $errors['tags'] = $context->validateRelations($data['tags'] ?? [], ZipExportTag::class); + + return $errors; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index ad17d5a33c6..99abb811c06 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -34,6 +34,6 @@ public static function validate(ZipValidationHelper $context, array $data): arra 'order' => ['nullable', 'integer'], ]; - return $context->validateArray($data, $rules); + return $context->validateData($data, $rules); } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index 5ad9272de43..e56394acaeb 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -2,62 +2,69 @@ namespace BookStack\Exports\ZipExports; -use BookStack\Exceptions\ZipExportValidationException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use ZipArchive; class ZipExportValidator { - protected array $errors = []; - public function __construct( protected string $zipPath, ) { } - /** - * @throws ZipExportValidationException - */ - public function validate() + public function validate(): array { - // TODO - Return type - // TODO - extract messages to translations? - // Validate file exists if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { - $this->throwErrors("Could not read ZIP file"); + return ['format' => "Could not read ZIP file"]; } // Validate file is valid zip $zip = new \ZipArchive(); $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); if ($opened !== true) { - $this->throwErrors("Could not read ZIP file"); + return ['format' => "Could not read ZIP file"]; } // Validate json data exists, including metadata $jsonData = $zip->getFromName('data.json') ?: ''; $importData = json_decode($jsonData, true); if (!$importData) { - $this->throwErrors("Could not decode ZIP data.json content"); + return ['format' => "Could not find and decode ZIP data.json content"]; } + $helper = new ZipValidationHelper($zip); + if (isset($importData['book'])) { - // TODO - Validate book + $modelErrors = ZipExportBook::validate($helper, $importData['book']); + $keyPrefix = 'book'; } else if (isset($importData['chapter'])) { - // TODO - Validate chapter + $modelErrors = ZipExportChapter::validate($helper, $importData['chapter']); + $keyPrefix = 'chapter'; } else if (isset($importData['page'])) { - // TODO - Validate page + $modelErrors = ZipExportPage::validate($helper, $importData['page']); + $keyPrefix = 'page'; } else { - $this->throwErrors("ZIP file has no book, chapter or page data"); + return ['format' => "ZIP file has no book, chapter or page data"]; } + + return $this->flattenModelErrors($modelErrors, $keyPrefix); } - /** - * @throws ZipExportValidationException - */ - protected function throwErrors(...$errorsToAdd): never + protected function flattenModelErrors(array $errors, string $keyPrefix): array { - array_push($this->errors, ...$errorsToAdd); - throw new ZipExportValidationException($this->errors); + $flattened = []; + + foreach ($errors as $key => $error) { + if (is_array($error)) { + $flattened = array_merge($flattened, $this->flattenModelErrors($error, $keyPrefix . '.' . $key)); + } else { + $flattened[$keyPrefix . '.' . $key] = $error; + } + } + + return $flattened; } } diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index dd41e6f8b25..8c285deaf5d 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -2,6 +2,7 @@ namespace BookStack\Exports\ZipExports; +use BookStack\Exports\ZipExports\Models\ZipExportModel; use Illuminate\Validation\Factory; use ZipArchive; @@ -15,9 +16,15 @@ public function __construct( $this->validationFactory = app(Factory::class); } - public function validateArray(array $data, array $rules): array + public function validateData(array $data, array $rules): array { - return $this->validationFactory->make($data, $rules)->errors()->messages(); + $messages = $this->validationFactory->make($data, $rules)->errors()->messages(); + + foreach ($messages as $key => $message) { + $messages[$key] = implode("\n", $message); + } + + return $messages; } public function zipFileExists(string $name): bool @@ -29,4 +36,24 @@ public function fileReferenceRule(): ZipFileReferenceRule { return new ZipFileReferenceRule($this); } + + /** + * Validate an array of relation data arrays that are expected + * to be for the given ZipExportModel. + * @param class-string $model + */ + public function validateRelations(array $relations, string $model): array + { + $results = []; + + foreach ($relations as $key => $relationData) { + if (is_array($relationData)) { + $results[$key] = $model::validate($this, $relationData); + } else { + $results[$key] = [trans('validation.zip_model_expected', ['type' => gettype($relationData)])]; + } + } + + return $results; + } } diff --git a/lang/en/validation.php b/lang/en/validation.php index 6971edc023a..9cf5d78b6f6 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -105,7 +105,8 @@ 'url' => 'The :attribute format is invalid.', 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', - 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + 'zip_model_expected' => 'Data object expected but ":type" found', // Custom validation lines 'custom' => [ diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index 9fe596d8888..15f33e6b7c1 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -6,7 +6,7 @@

    {{ trans('entities.import') }}

    -
    + {{ csrf_field() }}

    @@ -22,6 +22,7 @@ name="file" id="file" class="custom-simple-file-input"> + @include('form.errors', ['name' => 'file'])

    From 259aa829d42b1cd93011d5b8b531c15804741cb5 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 2 Nov 2024 14:51:04 +0000 Subject: [PATCH 18/41] ZIP Imports: Added validation message display, added testing Testing covers main UI access, and main non-successfull import actions. Started planning stored import model. Extracted some text to language files. --- app/Exports/Controllers/ImportController.php | 20 ++- app/Exports/ZipExports/ZipExportValidator.php | 9 +- lang/en/entities.php | 3 + lang/en/errors.php | 5 + lang/en/validation.php | 2 +- resources/views/exports/import.blade.php | 17 ++- tests/Exports/ZipImportTest.php | 124 ++++++++++++++++++ 7 files changed, 164 insertions(+), 16 deletions(-) create mode 100644 tests/Exports/ZipImportTest.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 5885f7991cd..323ecef268f 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -17,7 +17,9 @@ public function start(Request $request) { // TODO - Show existing imports for user (or for all users if admin-level user) - return view('exports.import'); + return view('exports.import', [ + 'zipErrors' => session()->pull('validation_errors') ?? [], + ]); } public function upload(Request $request) @@ -31,13 +33,21 @@ public function upload(Request $request) $errors = (new ZipExportValidator($zipPath))->validate(); if ($errors) { - dd($errors); + session()->flash('validation_errors', $errors); + return redirect('/import'); } + dd('passed'); - // TODO - Read existing ZIP upload and send through validator - // TODO - If invalid, return user with errors // TODO - Upload to storage - // TODO - Store info/results from validator + // TODO - Store info/results for display: + // - zip_path + // - name (From name of thing being imported) + // - size + // - book_count + // - chapter_count + // - page_count + // - created_by + // - created_at/updated_at // TODO - Send user to next import stage } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index e56394acaeb..dd56f3e70a8 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -18,21 +18,21 @@ public function validate(): array { // Validate file exists if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { - return ['format' => "Could not read ZIP file"]; + return ['format' => trans('errors.import_zip_cant_read')]; } // Validate file is valid zip $zip = new \ZipArchive(); $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); if ($opened !== true) { - return ['format' => "Could not read ZIP file"]; + return ['format' => trans('errors.import_zip_cant_read')]; } // Validate json data exists, including metadata $jsonData = $zip->getFromName('data.json') ?: ''; $importData = json_decode($jsonData, true); if (!$importData) { - return ['format' => "Could not find and decode ZIP data.json content"]; + return ['format' => trans('errors.import_zip_cant_decode_data')]; } $helper = new ZipValidationHelper($zip); @@ -47,9 +47,10 @@ public function validate(): array $modelErrors = ZipExportPage::validate($helper, $importData['page']); $keyPrefix = 'page'; } else { - return ['format' => "ZIP file has no book, chapter or page data"]; + return ['format' => trans('errors.import_zip_no_data')]; } + return $this->flattenModelErrors($modelErrors, $keyPrefix); } diff --git a/lang/en/entities.php b/lang/en/entities.php index 45ca4cf6b31..10614733533 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -45,6 +45,9 @@ 'default_template_select' => 'Select a template page', 'import' => 'Import', 'import_validate' => 'Validate Import', + 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', + 'import_zip_select' => 'Select ZIP file to upload', + 'import_zip_validation_errors' => 'Errors were detected while validating the provided ZIP file:', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/lang/en/errors.php b/lang/en/errors.php index 9c40aa9ed33..3f2f303311e 100644 --- a/lang/en/errors.php +++ b/lang/en/errors.php @@ -105,6 +105,11 @@ 'app_down' => ':appName is down right now', 'back_soon' => 'It will be back up soon.', + // Import + 'import_zip_cant_read' => 'Could not read ZIP file.', + 'import_zip_cant_decode_data' => 'Could not find and decode ZIP data.json content.', + 'import_zip_no_data' => 'ZIP file data has no expected book, chapter or page content.', + // API errors 'api_no_authorization_found' => 'No authorization token found on the request', 'api_bad_authorization_format' => 'An authorization token was found on the request but the format appeared incorrect', diff --git a/lang/en/validation.php b/lang/en/validation.php index 9cf5d78b6f6..bc01ac47b94 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -106,7 +106,7 @@ 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', - 'zip_model_expected' => 'Data object expected but ":type" found', + 'zip_model_expected' => 'Data object expected but ":type" found.', // Custom validation lines 'custom' => [ diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index 15f33e6b7c1..c4d7c881845 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -9,14 +9,10 @@ {{ csrf_field() }}
    -

    - Import books, chapters & pages using a portable zip export from the same, or a different, instance. - Select a ZIP file to import then press "Validate Import" to proceed. - After the file has been uploaded and validated you'll be able to configure & confirm the import in the next view. -

    +

    {{ trans('entities.import_desc') }}

    - +
    + @if(count($zipErrors) > 0) +

    {{ trans('entities.import_zip_validation_errors') }}

    +
      + @foreach($zipErrors as $key => $error) +
    • [{{ $key }}]: {{ $error }}
      • + @endforeach +
    + @endif +
    {{ trans('common.cancel') }} diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php new file mode 100644 index 00000000000..c9d255b1e97 --- /dev/null +++ b/tests/Exports/ZipImportTest.php @@ -0,0 +1,124 @@ +asAdmin()->get('/import'); + $resp->assertSee('Import'); + $this->withHtml($resp)->assertElementExists('form input[type="file"][name="file"]'); + } + + public function test_permissions_needed_for_import_page() + { + $user = $this->users->viewer(); + $this->actingAs($user); + + $resp = $this->get('/books'); + $this->withHtml($resp)->assertLinkNotExists(url('/import')); + $resp = $this->get('/import'); + $resp->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $resp = $this->get('/books'); + $this->withHtml($resp)->assertLinkExists(url('/import')); + $resp = $this->get('/import'); + $resp->assertOk(); + $resp->assertSeeText('Select ZIP file to upload'); + } + + public function test_zip_read_errors_are_shown_on_validation() + { + $invalidUpload = $this->files->uploadedImage('image.zip'); + + $this->asAdmin(); + $resp = $this->runImportFromFile($invalidUpload); + $resp->assertRedirect('/import'); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('Could not read ZIP file'); + } + + public function test_error_shown_if_missing_data() + { + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::CREATE); + $zip->addFromString('beans', 'cat'); + $zip->close(); + + $this->asAdmin(); + $upload = new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + $resp = $this->runImportFromFile($upload); + $resp->assertRedirect('/import'); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('Could not find and decode ZIP data.json content.'); + } + + public function test_error_shown_if_no_importable_key() + { + $this->asAdmin(); + $resp = $this->runImportFromFile($this->zipUploadFromData([ + 'instance' => [] + ])); + + $resp->assertRedirect('/import'); + $resp = $this->followRedirects($resp); + $resp->assertSeeText('ZIP file data has no expected book, chapter or page content.'); + } + + public function test_zip_data_validation_messages_shown() + { + $this->asAdmin(); + $resp = $this->runImportFromFile($this->zipUploadFromData([ + 'book' => [ + 'id' => 4, + 'pages' => [ + 'cat', + [ + 'name' => 'My inner page', + 'tags' => [ + [ + 'value' => 5 + ] + ], + ] + ], + ] + ])); + + $resp->assertRedirect('/import'); + $resp = $this->followRedirects($resp); + + $resp->assertSeeText('[book.name]: The name field is required.'); + $resp->assertSeeText('[book.pages.0.0]: Data object expected but "string" found.'); + $resp->assertSeeText('[book.pages.1.tags.0.name]: The name field is required.'); + $resp->assertSeeText('[book.pages.1.tags.0.value]: The value must be a string.'); + } + + protected function runImportFromFile(UploadedFile $file): TestResponse + { + return $this->call('POST', '/import', [], [], ['file' => $file]); + } + + protected function zipUploadFromData(array $data): UploadedFile + { + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::CREATE); + $zip->addFromString('data.json', json_encode($data)); + $zip->close(); + + return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + } +} From 74fce9640ef39a743bdb5a997724465c7c2b764c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 2 Nov 2024 17:17:34 +0000 Subject: [PATCH 19/41] ZIP Import: Added model+migration, and reader class --- app/Exports/Controllers/ImportController.php | 24 +++-- app/Exports/Import.php | 41 +++++++ app/Exports/ZipExports/ZipExportReader.php | 102 ++++++++++++++++++ app/Exports/ZipExports/ZipExportValidator.php | 26 ++--- .../ZipExports/ZipFileReferenceRule.php | 2 +- .../ZipExports/ZipValidationHelper.php | 8 +- database/factories/Exports/ImportFactory.php | 32 ++++++ ...2024_11_02_160700_create_imports_table.php | 34 ++++++ 8 files changed, 234 insertions(+), 35 deletions(-) create mode 100644 app/Exports/Import.php create mode 100644 app/Exports/ZipExports/ZipExportReader.php create mode 100644 database/factories/Exports/ImportFactory.php create mode 100644 database/migrations/2024_11_02_160700_create_imports_table.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 323ecef268f..bbf0ff57d8c 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -2,6 +2,8 @@ namespace BookStack\Exports\Controllers; +use BookStack\Exports\Import; +use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Http\Controller; use Illuminate\Http\Request; @@ -37,17 +39,23 @@ public function upload(Request $request) return redirect('/import'); } + $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); + $import = new Import(); + $import->name = $zipEntityInfo['name']; + $import->book_count = $zipEntityInfo['book_count']; + $import->chapter_count = $zipEntityInfo['chapter_count']; + $import->page_count = $zipEntityInfo['page_count']; + $import->created_by = user()->id; + $import->size = filesize($zipPath); + // TODO - Set path + // TODO - Save + + // TODO - Split out attachment service to separate out core filesystem/disk stuff + // To reuse for import handling + dd('passed'); // TODO - Upload to storage // TODO - Store info/results for display: - // - zip_path - // - name (From name of thing being imported) - // - size - // - book_count - // - chapter_count - // - page_count - // - created_by - // - created_at/updated_at // TODO - Send user to next import stage } } diff --git a/app/Exports/Import.php b/app/Exports/Import.php new file mode 100644 index 00000000000..c3ac3d52924 --- /dev/null +++ b/app/Exports/Import.php @@ -0,0 +1,41 @@ +book_count === 1) { + return self::TYPE_BOOK; + } elseif ($this->chapter_count === 1) { + return self::TYPE_CHAPTER; + } + + return self::TYPE_PAGE; + } +} diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php new file mode 100644 index 00000000000..7187a18897d --- /dev/null +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -0,0 +1,102 @@ +zip = new ZipArchive(); + } + + /** + * @throws ZipExportException + */ + protected function open(): void + { + if ($this->open) { + return; + } + + // Validate file exists + if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { + throw new ZipExportException(trans('errors.import_zip_cant_read')); + } + + // Validate file is valid zip + $opened = $this->zip->open($this->zipPath, ZipArchive::RDONLY); + if ($opened !== true) { + throw new ZipExportException(trans('errors.import_zip_cant_read')); + } + + $this->open = true; + } + + public function close(): void + { + if ($this->open) { + $this->zip->close(); + $this->open = false; + } + } + + /** + * @throws ZipExportException + */ + public function readData(): array + { + $this->open(); + + // Validate json data exists, including metadata + $jsonData = $this->zip->getFromName('data.json') ?: ''; + $importData = json_decode($jsonData, true); + if (!$importData) { + throw new ZipExportException(trans('errors.import_zip_cant_decode_data')); + } + + return $importData; + } + + public function fileExists(string $fileName): bool + { + return $this->zip->statName("files/{$fileName}") !== false; + } + + /** + * @throws ZipExportException + * @returns array{name: string, book_count: int, chapter_count: int, page_count: int} + */ + public function getEntityInfo(): array + { + $data = $this->readData(); + $info = ['name' => '', 'book_count' => 0, 'chapter_count' => 0, 'page_count' => 0]; + + if (isset($data['book'])) { + $info['name'] = $data['book']['name'] ?? ''; + $info['book_count']++; + $chapters = $data['book']['chapters'] ?? []; + $pages = $data['book']['pages'] ?? []; + $info['chapter_count'] += count($chapters); + $info['page_count'] += count($pages); + foreach ($chapters as $chapter) { + $info['page_count'] += count($chapter['pages'] ?? []); + } + } elseif (isset($data['chapter'])) { + $info['name'] = $data['chapter']['name'] ?? ''; + $info['chapter_count']++; + $info['page_count'] += count($data['chapter']['pages'] ?? []); + } elseif (isset($data['page'])) { + $info['name'] = $data['page']['name'] ?? ''; + $info['page_count']++; + } + + return $info; + } +} diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index dd56f3e70a8..e476998c216 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -2,10 +2,10 @@ namespace BookStack\Exports\ZipExports; +use BookStack\Exceptions\ZipExportException; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; -use ZipArchive; class ZipExportValidator { @@ -16,26 +16,14 @@ public function __construct( public function validate(): array { - // Validate file exists - if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { - return ['format' => trans('errors.import_zip_cant_read')]; + $reader = new ZipExportReader($this->zipPath); + try { + $importData = $reader->readData(); + } catch (ZipExportException $exception) { + return ['format' => $exception->getMessage()]; } - // Validate file is valid zip - $zip = new \ZipArchive(); - $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); - if ($opened !== true) { - return ['format' => trans('errors.import_zip_cant_read')]; - } - - // Validate json data exists, including metadata - $jsonData = $zip->getFromName('data.json') ?: ''; - $importData = json_decode($jsonData, true); - if (!$importData) { - return ['format' => trans('errors.import_zip_cant_decode_data')]; - } - - $helper = new ZipValidationHelper($zip); + $helper = new ZipValidationHelper($reader); if (isset($importData['book'])) { $modelErrors = ZipExportBook::validate($helper, $importData['book']); diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php index 4f942e0e789..bcd3c39acf0 100644 --- a/app/Exports/ZipExports/ZipFileReferenceRule.php +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -19,7 +19,7 @@ public function __construct( */ public function validate(string $attribute, mixed $value, Closure $fail): void { - if (!$this->context->zipFileExists($value)) { + if (!$this->context->zipReader->fileExists($value)) { $fail('validation.zip_file')->translate(); } } diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index 8c285deaf5d..55c86b03b5b 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -4,14 +4,13 @@ use BookStack\Exports\ZipExports\Models\ZipExportModel; use Illuminate\Validation\Factory; -use ZipArchive; class ZipValidationHelper { protected Factory $validationFactory; public function __construct( - protected ZipArchive $zip, + public ZipExportReader $zipReader, ) { $this->validationFactory = app(Factory::class); } @@ -27,11 +26,6 @@ public function validateData(array $data, array $rules): array return $messages; } - public function zipFileExists(string $name): bool - { - return $this->zip->statName("files/{$name}") !== false; - } - public function fileReferenceRule(): ZipFileReferenceRule { return new ZipFileReferenceRule($this); diff --git a/database/factories/Exports/ImportFactory.php b/database/factories/Exports/ImportFactory.php new file mode 100644 index 00000000000..55378d5832e --- /dev/null +++ b/database/factories/Exports/ImportFactory.php @@ -0,0 +1,32 @@ + 'uploads/imports/' . Str::random(10) . '.zip', + 'name' => $this->faker->words(3, true), + 'book_count' => 1, + 'chapter_count' => 5, + 'page_count' => 15, + 'created_at' => User::factory(), + ]; + } +} diff --git a/database/migrations/2024_11_02_160700_create_imports_table.php b/database/migrations/2024_11_02_160700_create_imports_table.php new file mode 100644 index 00000000000..ed188226981 --- /dev/null +++ b/database/migrations/2024_11_02_160700_create_imports_table.php @@ -0,0 +1,34 @@ +increments('id'); + $table->string('name'); + $table->string('path'); + $table->integer('size'); + $table->integer('book_count'); + $table->integer('chapter_count'); + $table->integer('page_count'); + $table->integer('created_by'); + $table->timestamps(); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::dropIfExists('imports'); + } +}; From 8ea3855e02aa5ff7782dc65e1eee8b8b24f28ce6 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 2 Nov 2024 20:48:21 +0000 Subject: [PATCH 20/41] ZIP Import: Added upload handling Split attachment service storage work out so it can be shared. --- app/Exceptions/ZipValidationException.php | 12 ++ app/Exports/Controllers/ImportController.php | 41 ++----- app/Exports/ImportRepo.php | 48 ++++++++ app/Uploads/AttachmentService.php | 86 ++------------ app/Uploads/FileStorage.php | 111 +++++++++++++++++++ 5 files changed, 195 insertions(+), 103 deletions(-) create mode 100644 app/Exceptions/ZipValidationException.php create mode 100644 app/Exports/ImportRepo.php create mode 100644 app/Uploads/FileStorage.php diff --git a/app/Exceptions/ZipValidationException.php b/app/Exceptions/ZipValidationException.php new file mode 100644 index 00000000000..aaaee792ef0 --- /dev/null +++ b/app/Exceptions/ZipValidationException.php @@ -0,0 +1,12 @@ +middleware('can:content-import'); } @@ -27,35 +28,17 @@ public function start(Request $request) public function upload(Request $request) { $this->validate($request, [ - 'file' => ['required', 'file'] + 'file' => ['required', ...AttachmentService::getFileValidationRules()] ]); $file = $request->file('file'); - $zipPath = $file->getRealPath(); - - $errors = (new ZipExportValidator($zipPath))->validate(); - if ($errors) { - session()->flash('validation_errors', $errors); + try { + $import = $this->imports->storeFromUpload($file); + } catch (ZipValidationException $exception) { + session()->flash('validation_errors', $exception->errors); return redirect('/import'); } - $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); - $import = new Import(); - $import->name = $zipEntityInfo['name']; - $import->book_count = $zipEntityInfo['book_count']; - $import->chapter_count = $zipEntityInfo['chapter_count']; - $import->page_count = $zipEntityInfo['page_count']; - $import->created_by = user()->id; - $import->size = filesize($zipPath); - // TODO - Set path - // TODO - Save - - // TODO - Split out attachment service to separate out core filesystem/disk stuff - // To reuse for import handling - - dd('passed'); - // TODO - Upload to storage - // TODO - Store info/results for display: - // TODO - Send user to next import stage + return redirect("imports/{$import->id}"); } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php new file mode 100644 index 00000000000..c8157967bc3 --- /dev/null +++ b/app/Exports/ImportRepo.php @@ -0,0 +1,48 @@ +getRealPath(); + + $errors = (new ZipExportValidator($zipPath))->validate(); + if ($errors) { + throw new ZipValidationException($errors); + } + + $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); + $import = new Import(); + $import->name = $zipEntityInfo['name']; + $import->book_count = $zipEntityInfo['book_count']; + $import->chapter_count = $zipEntityInfo['chapter_count']; + $import->page_count = $zipEntityInfo['page_count']; + $import->created_by = user()->id; + $import->size = filesize($zipPath); + + $path = $this->storage->uploadFile( + $file, + 'uploads/files/imports/', + '', + 'zip' + ); + + $import->path = $path; + $import->save(); + + return $import; + } +} diff --git a/app/Uploads/AttachmentService.php b/app/Uploads/AttachmentService.php index 227649d8f00..fa53c4ae499 100644 --- a/app/Uploads/AttachmentService.php +++ b/app/Uploads/AttachmentService.php @@ -4,59 +4,15 @@ use BookStack\Exceptions\FileUploadException; use Exception; -use Illuminate\Contracts\Filesystem\Filesystem as Storage; -use Illuminate\Filesystem\FilesystemManager; -use Illuminate\Support\Facades\Log; -use Illuminate\Support\Str; -use League\Flysystem\WhitespacePathNormalizer; use Symfony\Component\HttpFoundation\File\UploadedFile; class AttachmentService { public function __construct( - protected FilesystemManager $fileSystem + protected FileStorage $storage, ) { } - /** - * Get the storage that will be used for storing files. - */ - protected function getStorageDisk(): Storage - { - return $this->fileSystem->disk($this->getStorageDiskName()); - } - - /** - * Get the name of the storage disk to use. - */ - protected function getStorageDiskName(): string - { - $storageType = config('filesystems.attachments'); - - // Change to our secure-attachment disk if any of the local options - // are used to prevent escaping that location. - if ($storageType === 'local' || $storageType === 'local_secure' || $storageType === 'local_secure_restricted') { - $storageType = 'local_secure_attachments'; - } - - return $storageType; - } - - /** - * Change the originally provided path to fit any disk-specific requirements. - * This also ensures the path is kept to the expected root folders. - */ - protected function adjustPathForStorageDisk(string $path): string - { - $path = (new WhitespacePathNormalizer())->normalizePath(str_replace('uploads/files/', '', $path)); - - if ($this->getStorageDiskName() === 'local_secure_attachments') { - return $path; - } - - return 'uploads/files/' . $path; - } - /** * Stream an attachment from storage. * @@ -64,7 +20,7 @@ protected function adjustPathForStorageDisk(string $path): string */ public function streamAttachmentFromStorage(Attachment $attachment) { - return $this->getStorageDisk()->readStream($this->adjustPathForStorageDisk($attachment->path)); + return $this->storage->getReadStream($attachment->path); } /** @@ -72,7 +28,7 @@ public function streamAttachmentFromStorage(Attachment $attachment) */ public function getAttachmentFileSize(Attachment $attachment): int { - return $this->getStorageDisk()->size($this->adjustPathForStorageDisk($attachment->path)); + return $this->storage->getSize($attachment->path); } /** @@ -195,15 +151,9 @@ public function deleteFile(Attachment $attachment) * Delete a file from the filesystem it sits on. * Cleans any empty leftover folders. */ - protected function deleteFileInStorage(Attachment $attachment) + protected function deleteFileInStorage(Attachment $attachment): void { - $storage = $this->getStorageDisk(); - $dirPath = $this->adjustPathForStorageDisk(dirname($attachment->path)); - - $storage->delete($this->adjustPathForStorageDisk($attachment->path)); - if (count($storage->allFiles($dirPath)) === 0) { - $storage->deleteDirectory($dirPath); - } + $this->storage->delete($attachment->path); } /** @@ -213,32 +163,20 @@ protected function deleteFileInStorage(Attachment $attachment) */ protected function putFileInStorage(UploadedFile $uploadedFile): string { - $storage = $this->getStorageDisk(); $basePath = 'uploads/files/' . date('Y-m-M') . '/'; - $uploadFileName = Str::random(16) . '-' . $uploadedFile->getClientOriginalExtension(); - while ($storage->exists($this->adjustPathForStorageDisk($basePath . $uploadFileName))) { - $uploadFileName = Str::random(3) . $uploadFileName; - } - - $attachmentStream = fopen($uploadedFile->getRealPath(), 'r'); - $attachmentPath = $basePath . $uploadFileName; - - try { - $storage->writeStream($this->adjustPathForStorageDisk($attachmentPath), $attachmentStream); - } catch (Exception $e) { - Log::error('Error when attempting file upload:' . $e->getMessage()); - - throw new FileUploadException(trans('errors.path_not_writable', ['filePath' => $attachmentPath])); - } - - return $attachmentPath; + return $this->storage->uploadFile( + $uploadedFile, + $basePath, + $uploadedFile->getClientOriginalExtension(), + '' + ); } /** * Get the file validation rules for attachments. */ - public function getFileValidationRules(): array + public static function getFileValidationRules(): array { return ['file', 'max:' . (config('app.upload_limit') * 1000)]; } diff --git a/app/Uploads/FileStorage.php b/app/Uploads/FileStorage.php new file mode 100644 index 00000000000..278484e519d --- /dev/null +++ b/app/Uploads/FileStorage.php @@ -0,0 +1,111 @@ +getStorageDisk()->readStream($this->adjustPathForStorageDisk($path)); + } + + public function getSize(string $path): int + { + return $this->getStorageDisk()->size($this->adjustPathForStorageDisk($path)); + } + + public function delete(string $path, bool $removeEmptyDir = false): void + { + $storage = $this->getStorageDisk(); + $adjustedPath = $this->adjustPathForStorageDisk($path); + $dir = dirname($adjustedPath); + + $storage->delete($adjustedPath); + if ($removeEmptyDir && count($storage->allFiles($dir)) === 0) { + $storage->deleteDirectory($dir); + } + } + + /** + * @throws FileUploadException + */ + public function uploadFile(UploadedFile $file, string $subDirectory, string $suffix, string $extension): string + { + $storage = $this->getStorageDisk(); + $basePath = trim($subDirectory, '/') . '/'; + + $uploadFileName = Str::random(16) . ($suffix ? "-{$suffix}" : '') . ($extension ? ".{$extension}" : ''); + while ($storage->exists($this->adjustPathForStorageDisk($basePath . $uploadFileName))) { + $uploadFileName = Str::random(3) . $uploadFileName; + } + + $fileStream = fopen($file->getRealPath(), 'r'); + $filePath = $basePath . $uploadFileName; + + try { + $storage->writeStream($this->adjustPathForStorageDisk($filePath), $fileStream); + } catch (Exception $e) { + Log::error('Error when attempting file upload:' . $e->getMessage()); + + throw new FileUploadException(trans('errors.path_not_writable', ['filePath' => $filePath])); + } + + return $filePath; + } + + /** + * Get the storage that will be used for storing files. + */ + protected function getStorageDisk(): Storage + { + return $this->fileSystem->disk($this->getStorageDiskName()); + } + + /** + * Get the name of the storage disk to use. + */ + protected function getStorageDiskName(): string + { + $storageType = config('filesystems.attachments'); + + // Change to our secure-attachment disk if any of the local options + // are used to prevent escaping that location. + if ($storageType === 'local' || $storageType === 'local_secure' || $storageType === 'local_secure_restricted') { + $storageType = 'local_secure_attachments'; + } + + return $storageType; + } + + /** + * Change the originally provided path to fit any disk-specific requirements. + * This also ensures the path is kept to the expected root folders. + */ + protected function adjustPathForStorageDisk(string $path): string + { + $path = (new WhitespacePathNormalizer())->normalizePath(str_replace('uploads/files/', '', $path)); + + if ($this->getStorageDiskName() === 'local_secure_attachments') { + return $path; + } + + return 'uploads/files/' . $path; + } +} From c6109c708735434fdb30333ff4c24b4a80b0b749 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 3 Nov 2024 14:13:05 +0000 Subject: [PATCH 21/41] ZIP Imports: Added listing, show view, delete, activity --- app/Activity/ActivityType.php | 4 ++ app/Exports/Controllers/ImportController.php | 50 ++++++++++++++++++- app/Exports/Import.php | 23 ++++++++- app/Exports/ImportRepo.php | 32 ++++++++++++ app/Http/Controller.php | 4 +- lang/en/activities.php | 8 +++ lang/en/entities.php | 6 +++ resources/views/exports/import-show.blade.php | 38 ++++++++++++++ resources/views/exports/import.blade.php | 13 +++++ .../views/exports/parts/import.blade.php | 19 +++++++ routes/web.php | 2 + 11 files changed, 193 insertions(+), 6 deletions(-) create mode 100644 resources/views/exports/import-show.blade.php create mode 100644 resources/views/exports/parts/import.blade.php diff --git a/app/Activity/ActivityType.php b/app/Activity/ActivityType.php index 09b2ae73c56..5ec9b9cf0dc 100644 --- a/app/Activity/ActivityType.php +++ b/app/Activity/ActivityType.php @@ -67,6 +67,10 @@ class ActivityType const WEBHOOK_UPDATE = 'webhook_update'; const WEBHOOK_DELETE = 'webhook_delete'; + const IMPORT_CREATE = 'import_create'; + const IMPORT_RUN = 'import_run'; + const IMPORT_DELETE = 'import_delete'; + /** * Get all the possible values. */ diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 640b4c10891..582fff975d2 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -1,7 +1,10 @@ middleware('can:content-import'); } + /** + * Show the view to start a new import, and also list out the existing + * in progress imports that are visible to the user. + */ public function start(Request $request) { - // TODO - Show existing imports for user (or for all users if admin-level user) + // TODO - Test visibility access for listed items + $imports = $this->imports->getVisibleImports(); + + $this->setPageTitle(trans('entities.import')); return view('exports.import', [ + 'imports' => $imports, 'zipErrors' => session()->pull('validation_errors') ?? [], ]); } + /** + * Upload, validate and store an import file. + */ public function upload(Request $request) { $this->validate($request, [ @@ -39,6 +53,38 @@ public function upload(Request $request) return redirect('/import'); } - return redirect("imports/{$import->id}"); + $this->logActivity(ActivityType::IMPORT_CREATE, $import); + + return redirect($import->getUrl()); + } + + /** + * Show a pending import, with a form to allow progressing + * with the import process. + */ + public function show(int $id) + { + // TODO - Test visibility access + $import = $this->imports->findVisible($id); + + $this->setPageTitle(trans('entities.import_continue')); + + return view('exports.import-show', [ + 'import' => $import, + ]); + } + + /** + * Delete an active pending import from the filesystem and database. + */ + public function delete(int $id) + { + // TODO - Test visibility access + $import = $this->imports->findVisible($id); + $this->imports->deleteImport($import); + + $this->logActivity(ActivityType::IMPORT_DELETE, $import); + + return redirect('/import'); } } diff --git a/app/Exports/Import.php b/app/Exports/Import.php index c3ac3d52924..520d8ea6cc8 100644 --- a/app/Exports/Import.php +++ b/app/Exports/Import.php @@ -2,6 +2,7 @@ namespace BookStack\Exports; +use BookStack\Activity\Models\Loggable; use Carbon\Carbon; use Illuminate\Database\Eloquent\Factories\HasFactory; use Illuminate\Database\Eloquent\Model; @@ -17,7 +18,7 @@ * @property Carbon $created_at * @property Carbon $updated_at */ -class Import extends Model +class Import extends Model implements Loggable { use HasFactory; @@ -38,4 +39,24 @@ public function getType(): string return self::TYPE_PAGE; } + + public function getSizeString(): string + { + $mb = round($this->size / 1000000, 2); + return "{$mb} MB"; + } + + /** + * Get the URL to view/continue this import. + */ + public function getUrl(string $path = ''): string + { + $path = ltrim($path, '/'); + return url("/import/{$this->id}" . ($path ? '/' . $path : '')); + } + + public function logDescriptor(): string + { + return "({$this->id}) {$this->name}"; + } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index c8157967bc3..d7e169ad166 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -6,6 +6,7 @@ use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Uploads\FileStorage; +use Illuminate\Database\Eloquent\Collection; use Symfony\Component\HttpFoundation\File\UploadedFile; class ImportRepo @@ -15,6 +16,31 @@ public function __construct( ) { } + /** + * @return Collection + */ + public function getVisibleImports(): Collection + { + $query = Import::query(); + + if (!userCan('settings-manage')) { + $query->where('created_by', user()->id); + } + + return $query->get(); + } + + public function findVisible(int $id): Import + { + $query = Import::query(); + + if (!userCan('settings-manage')) { + $query->where('created_by', user()->id); + } + + return $query->findOrFail($id); + } + public function storeFromUpload(UploadedFile $file): Import { $zipPath = $file->getRealPath(); @@ -45,4 +71,10 @@ public function storeFromUpload(UploadedFile $file): Import return $import; } + + public function deleteImport(Import $import): void + { + $this->storage->delete($import->path); + $import->delete(); + } } diff --git a/app/Http/Controller.php b/app/Http/Controller.php index 8facf5dab3c..090cf523ad2 100644 --- a/app/Http/Controller.php +++ b/app/Http/Controller.php @@ -152,10 +152,8 @@ protected function showErrorNotification(string $message): void /** * Log an activity in the system. - * - * @param string|Loggable $detail */ - protected function logActivity(string $type, $detail = ''): void + protected function logActivity(string $type, string|Loggable $detail = ''): void { Activity::add($type, $detail); } diff --git a/lang/en/activities.php b/lang/en/activities.php index 092398ef0e1..7c3454d41ca 100644 --- a/lang/en/activities.php +++ b/lang/en/activities.php @@ -84,6 +84,14 @@ 'webhook_delete' => 'deleted webhook', 'webhook_delete_notification' => 'Webhook successfully deleted', + // Imports + 'import_create' => 'created import', + 'import_create_notification' => 'Import successfully uploaded', + 'import_run' => 'updated import', + 'import_run_notification' => 'Content successfully imported', + 'import_delete' => 'deleted import', + 'import_delete_notification' => 'Import successfully deleted', + // Users 'user_create' => 'created user', 'user_create_notification' => 'User successfully created', diff --git a/lang/en/entities.php b/lang/en/entities.php index 10614733533..e2d8e47c592 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -48,6 +48,12 @@ 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', 'import_zip_select' => 'Select ZIP file to upload', 'import_zip_validation_errors' => 'Errors were detected while validating the provided ZIP file:', + 'import_pending' => 'Pending Imports', + 'import_pending_none' => 'No imports have been started.', + 'import_continue' => 'Continue Import', + 'import_run' => 'Run Import', + 'import_delete_confirm' => 'Are you sure you want to delete this import?', + 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php new file mode 100644 index 00000000000..843a052468c --- /dev/null +++ b/resources/views/exports/import-show.blade.php @@ -0,0 +1,38 @@ +@extends('layouts.simple') + +@section('body') + +
    + +
    +

    {{ trans('entities.import_continue') }}

    + + {{ csrf_field() }} + + +
    + {{ trans('common.cancel') }} +
    + +
    +

    {{ trans('entities.import_delete_confirm') }}

    +

    {{ trans('entities.import_delete_desc') }}

    + +
    +
    + +
    +
    +
    + +
    + {{ method_field('DELETE') }} + {{ csrf_field() }} +
    + +@stop diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index c4d7c881845..be9de4c0e91 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -38,6 +38,19 @@ class="custom-simple-file-input">
    + +
    +

    {{ trans('entities.import_pending') }}

    + @if(count($imports) === 0) +

    {{ trans('entities.import_pending_none') }}

    + @else +
    + @foreach($imports as $import) + @include('exports.parts.import', ['import' => $import]) + @endforeach +
    + @endif +
    @stop diff --git a/resources/views/exports/parts/import.blade.php b/resources/views/exports/parts/import.blade.php new file mode 100644 index 00000000000..5ff6600f24b --- /dev/null +++ b/resources/views/exports/parts/import.blade.php @@ -0,0 +1,19 @@ +@php + $type = $import->getType(); +@endphp +
    +
    + @icon($type) {{ $import->name }} +
    +
    + @if($type === 'book') +
    @icon('chapter') {{ $import->chapter_count }}
    + @endif + @if($type === 'book' || $type === 'chapter') +
    @icon('page') {{ $import->page_count }}
    + @endif +
    {{ $import->getSizeString() }}
    +
    @icon('time'){{ $import->created_at->diffForHumans() }}
    +
    +
    \ No newline at end of file diff --git a/routes/web.php b/routes/web.php index 91aab13fecf..c490bb3b34e 100644 --- a/routes/web.php +++ b/routes/web.php @@ -209,6 +209,8 @@ // Importing Route::get('/import', [ExportControllers\ImportController::class, 'start']); Route::post('/import', [ExportControllers\ImportController::class, 'upload']); + Route::get('/import/{id}', [ExportControllers\ImportController::class, 'show']); + Route::delete('/import/{id}', [ExportControllers\ImportController::class, 'delete']); // Other Pages Route::get('/', [HomeController::class, 'index']); From 8f6f81948e81b4d63251bee57da57aa5809eaad2 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 3 Nov 2024 17:28:18 +0000 Subject: [PATCH 22/41] ZIP Imports: Fleshed out continue page, Added testing --- app/Exports/Controllers/ImportController.php | 17 ++- app/Exports/Import.php | 9 ++ lang/en/entities.php | 4 + resources/views/exports/import-show.blade.php | 41 ++++- routes/web.php | 1 + tests/Exports/ZipImportTest.php | 140 ++++++++++++++++++ 6 files changed, 206 insertions(+), 6 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 582fff975d2..787fd1b27e0 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -23,9 +23,8 @@ public function __construct( * Show the view to start a new import, and also list out the existing * in progress imports that are visible to the user. */ - public function start(Request $request) + public function start() { - // TODO - Test visibility access for listed items $imports = $this->imports->getVisibleImports(); $this->setPageTitle(trans('entities.import')); @@ -64,7 +63,6 @@ public function upload(Request $request) */ public function show(int $id) { - // TODO - Test visibility access $import = $this->imports->findVisible($id); $this->setPageTitle(trans('entities.import_continue')); @@ -74,12 +72,23 @@ public function show(int $id) ]); } + public function run(int $id) + { + // TODO - Test access/visibility + + $import = $this->imports->findVisible($id); + + // TODO - Run import + // Validate again before + // TODO - Redirect to result + // TOOD - Or redirect back with errors + } + /** * Delete an active pending import from the filesystem and database. */ public function delete(int $id) { - // TODO - Test visibility access $import = $this->imports->findVisible($id); $this->imports->deleteImport($import); diff --git a/app/Exports/Import.php b/app/Exports/Import.php index 520d8ea6cc8..8400382fd0d 100644 --- a/app/Exports/Import.php +++ b/app/Exports/Import.php @@ -3,11 +3,14 @@ namespace BookStack\Exports; use BookStack\Activity\Models\Loggable; +use BookStack\Users\Models\User; use Carbon\Carbon; use Illuminate\Database\Eloquent\Factories\HasFactory; use Illuminate\Database\Eloquent\Model; +use Illuminate\Database\Eloquent\Relations\BelongsTo; /** + * @property int $id * @property string $path * @property string $name * @property int $size - ZIP size in bytes @@ -17,6 +20,7 @@ * @property int $created_by * @property Carbon $created_at * @property Carbon $updated_at + * @property User $createdBy */ class Import extends Model implements Loggable { @@ -59,4 +63,9 @@ public function logDescriptor(): string { return "({$this->id}) {$this->name}"; } + + public function createdBy(): BelongsTo + { + return $this->belongsTo(User::class, 'created_by'); + } } diff --git a/lang/en/entities.php b/lang/en/entities.php index e2d8e47c592..4f5a530049b 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -51,7 +51,11 @@ 'import_pending' => 'Pending Imports', 'import_pending_none' => 'No imports have been started.', 'import_continue' => 'Continue Import', + 'import_continue_desc' => 'Review the content due to be imported from the uploaded ZIP file. When ready, run the import to add its contents to this system. The uploaded ZIP import file will be automatically removed on successful import.', 'import_run' => 'Run Import', + 'import_size' => 'Import ZIP Size:', + 'import_uploaded_at' => 'Uploaded:', + 'import_uploaded_by' => 'Uploaded by:', 'import_delete_confirm' => 'Are you sure you want to delete this import?', 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index 843a052468c..ac1b8a45d06 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -6,7 +6,44 @@

    {{ trans('entities.import_continue') }}

    -
    +

    {{ trans('entities.import_continue_desc') }}

    + +
    + @php + $type = $import->getType(); + @endphp +
    +
    +

    @icon($type) {{ $import->name }}

    + @if($type === 'book') +

    @icon('chapter') {{ trans_choice('entities.x_chapters', $import->chapter_count) }}

    + @endif + @if($type === 'book' || $type === 'chapter') +

    @icon('page') {{ trans_choice('entities.x_pages', $import->page_count) }}

    + @endif +
    +
    +
    + {{ trans('entities.import_size') }} + {{ $import->getSizeString() }} +
    +
    + {{ trans('entities.import_uploaded_at') }} + {{ $import->created_at->diffForHumans() }} +
    + @if($import->createdBy) +
    + {{ trans('entities.import_uploaded_by') }} + {{ $import->createdBy->name }} +
    + @endif +
    +
    +
    + + {{ csrf_field() }}
    @@ -23,7 +60,7 @@ class="button outline">{{ trans('common.delete') }} - +
    diff --git a/routes/web.php b/routes/web.php index c490bb3b34e..85f83352859 100644 --- a/routes/web.php +++ b/routes/web.php @@ -210,6 +210,7 @@ Route::get('/import', [ExportControllers\ImportController::class, 'start']); Route::post('/import', [ExportControllers\ImportController::class, 'upload']); Route::get('/import/{id}', [ExportControllers\ImportController::class, 'show']); + Route::post('/import/{id}', [ExportControllers\ImportController::class, 'run']); Route::delete('/import/{id}', [ExportControllers\ImportController::class, 'delete']); // Other Pages diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index c9d255b1e97..b9a8598fabe 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -2,6 +2,8 @@ namespace Tests\Exports; +use BookStack\Activity\ActivityType; +use BookStack\Exports\Import; use Illuminate\Http\UploadedFile; use Illuminate\Testing\TestResponse; use Tests\TestCase; @@ -35,6 +37,25 @@ public function test_permissions_needed_for_import_page() $resp->assertSeeText('Select ZIP file to upload'); } + public function test_import_page_pending_import_visibility_limited() + { + $user = $this->users->viewer(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $resp = $this->actingAs($user)->get('/import'); + $resp->assertSeeText('MySuperUserImport'); + $resp->assertDontSeeText('MySuperAdminImport'); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $resp = $this->actingAs($user)->get('/import'); + $resp->assertSeeText('MySuperUserImport'); + $resp->assertSeeText('MySuperAdminImport'); + } + public function test_zip_read_errors_are_shown_on_validation() { $invalidUpload = $this->files->uploadedImage('image.zip'); @@ -105,6 +126,125 @@ public function test_zip_data_validation_messages_shown() $resp->assertSeeText('[book.pages.1.tags.0.value]: The value must be a string.'); } + public function test_import_upload_success() + { + $admin = $this->users->admin(); + $this->actingAs($admin); + $resp = $this->runImportFromFile($this->zipUploadFromData([ + 'book' => [ + 'name' => 'My great book name', + 'chapters' => [ + [ + 'name' => 'my chapter', + 'pages' => [ + [ + 'name' => 'my chapter page', + ] + ] + ] + ], + 'pages' => [ + [ + 'name' => 'My page', + ] + ], + ], + ])); + + $this->assertDatabaseHas('imports', [ + 'name' => 'My great book name', + 'book_count' => 1, + 'chapter_count' => 1, + 'page_count' => 2, + 'created_by' => $admin->id, + ]); + + /** @var Import $import */ + $import = Import::query()->latest()->first(); + $resp->assertRedirect("/import/{$import->id}"); + $this->assertFileExists(storage_path($import->path)); + $this->assertActivityExists(ActivityType::IMPORT_CREATE); + } + + public function test_import_show_page() + { + $import = Import::factory()->create(['name' => 'MySuperAdminImport']); + + $resp = $this->asAdmin()->get("/import/{$import->id}"); + $resp->assertOk(); + $resp->assertSee('MySuperAdminImport'); + } + + public function test_import_show_page_access_limited() + { + $user = $this->users->viewer(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->actingAs($user); + + $this->get("/import/{$userImport->id}")->assertRedirect('/'); + $this->get("/import/{$adminImport->id}")->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $this->get("/import/{$userImport->id}")->assertOk(); + $this->get("/import/{$adminImport->id}")->assertStatus(404); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $this->get("/import/{$userImport->id}")->assertOk(); + $this->get("/import/{$adminImport->id}")->assertOk(); + } + + public function test_import_delete() + { + $this->asAdmin(); + $this->runImportFromFile($this->zipUploadFromData([ + 'book' => [ + 'name' => 'My great book name' + ], + ])); + + /** @var Import $import */ + $import = Import::query()->latest()->first(); + $this->assertDatabaseHas('imports', [ + 'id' => $import->id, + 'name' => 'My great book name' + ]); + $this->assertFileExists(storage_path($import->path)); + + $resp = $this->delete("/import/{$import->id}"); + + $resp->assertRedirect('/import'); + $this->assertActivityExists(ActivityType::IMPORT_DELETE); + $this->assertDatabaseMissing('imports', [ + 'id' => $import->id, + ]); + $this->assertFileDoesNotExist(storage_path($import->path)); + } + + public function test_import_delete_access_limited() + { + $user = $this->users->viewer(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->actingAs($user); + + $this->delete("/import/{$userImport->id}")->assertRedirect('/'); + $this->delete("/import/{$adminImport->id}")->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $this->delete("/import/{$userImport->id}")->assertRedirect('/import'); + $this->delete("/import/{$adminImport->id}")->assertStatus(404); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $this->delete("/import/{$adminImport->id}")->assertRedirect('/import'); + } + protected function runImportFromFile(UploadedFile $file): TestResponse { return $this->call('POST', '/import', [], [], ['file' => $file]); From 14578c22570d7f9ac197125ece1cf86d9d07be9b Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 4 Nov 2024 16:21:22 +0000 Subject: [PATCH 23/41] ZIP Imports: Added parent selector for page/chapter imports --- app/Exports/Controllers/ImportController.php | 14 ++++++++--- lang/en/entities.php | 2 ++ resources/sass/styles.scss | 18 +++++++++++++ resources/views/entities/selector.blade.php | 8 ++++++ resources/views/exports/import-show.blade.php | 25 +++++++++++++++---- resources/views/form/errors.blade.php | 3 +++ 6 files changed, 62 insertions(+), 8 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 787fd1b27e0..a2389c725f3 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -72,14 +72,22 @@ public function show(int $id) ]); } - public function run(int $id) + public function run(int $id, Request $request) { // TODO - Test access/visibility - $import = $this->imports->findVisible($id); + $parent = null; + + if ($import->getType() === 'page' || $import->getType() === 'chapter') { + $data = $this->validate($request, [ + 'parent' => ['required', 'string'] + ]); + $parent = $data['parent']; + } // TODO - Run import - // Validate again before + // TODO - Validate again before + // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) // TODO - Redirect to result // TOOD - Or redirect back with errors } diff --git a/lang/en/entities.php b/lang/en/entities.php index 4f5a530049b..065eb043a11 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -56,6 +56,8 @@ 'import_size' => 'Import ZIP Size:', 'import_uploaded_at' => 'Uploaded:', 'import_uploaded_by' => 'Uploaded by:', + 'import_location' => 'Import Location', + 'import_location_desc' => 'Select a target location for your imported content. You\'ll need the relevant permissions to create within the location you choose.', 'import_delete_confirm' => 'Are you sure you want to delete this import?', 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', diff --git a/resources/sass/styles.scss b/resources/sass/styles.scss index 636367e3aeb..942265d04d8 100644 --- a/resources/sass/styles.scss +++ b/resources/sass/styles.scss @@ -138,6 +138,11 @@ $loadingSize: 10px; font-size: 16px; padding: $-s $-m; } + input[type="text"]:focus { + outline: 1px solid var(--color-primary); + border-radius: 3px 3px 0 0; + outline-offset: -1px; + } .entity-list { overflow-y: scroll; height: 400px; @@ -171,6 +176,19 @@ $loadingSize: 10px; font-size: 14px; } } + &.small { + .entity-list-item { + padding: $-xs $-m; + } + .entity-list, .loading { + height: 300px; + } + input[type="text"] { + font-size: 13px; + padding: $-xs $-m; + height: auto; + } + } } .fullscreen { diff --git a/resources/views/entities/selector.blade.php b/resources/views/entities/selector.blade.php index c1280cfb2f7..0cdf4376cc7 100644 --- a/resources/views/entities/selector.blade.php +++ b/resources/views/entities/selector.blade.php @@ -1,3 +1,11 @@ +{{-- +$name - string +$autofocus - boolean, optional +$entityTypes - string, optional +$entityPermission - string, optional +$selectorEndpoint - string, optional +$selectorSize - string, optional (compact) +--}}
    getType(); + @endphp +
    @@ -9,11 +13,9 @@

    {{ trans('entities.import_continue_desc') }}

    - @php - $type = $import->getType(); - @endphp +
    -
    +

    @icon($type) {{ $import->name }}

    @if($type === 'book')

    @icon('chapter') {{ trans_choice('entities.x_chapters', $import->chapter_count) }}

    @@ -22,7 +24,7 @@

    @icon('page') {{ trans_choice('entities.x_pages', $import->page_count) }}

    @endif
    -
    +
    {{ trans('entities.import_size') }} {{ $import->getSizeString() }} @@ -45,6 +47,19 @@ action="{{ $import->getUrl() }}" method="POST"> {{ csrf_field() }} + + @if($type === 'page' || $type === 'chapter') +
    + +

    {{ trans('entities.import_location_desc') }}

    + @include('entities.selector', [ + 'name' => 'parent', + 'entityTypes' => $type === 'page' ? 'chapter,book' : 'book', + 'entityPermission' => "{$type}-create", + 'selectorSize' => 'compact small', + ]) + @include('form.errors', ['name' => 'parent']) + @endif
    diff --git a/resources/views/form/errors.blade.php b/resources/views/form/errors.blade.php index 03cd4be88f0..72d41ee56c7 100644 --- a/resources/views/form/errors.blade.php +++ b/resources/views/form/errors.blade.php @@ -1,3 +1,6 @@ +{{-- +$name - string +--}} @if($errors->has($name))
    {{ $errors->first($name) }}
    @endif \ No newline at end of file From 92cfde495e0d3141af608ea3734b612402f257dd Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 5 Nov 2024 13:17:31 +0000 Subject: [PATCH 24/41] ZIP Imports: Added full contents view to import display Reduced import data will now be stored on the import itself, instead of storing a set of totals. --- app/Exports/Controllers/ImportController.php | 5 ++- app/Exports/Import.php | 37 ++++++++---------- app/Exports/ImportRepo.php | 35 ++++++++++++++--- .../ZipExports/Models/ZipExportAttachment.php | 18 +++++++++ .../ZipExports/Models/ZipExportBook.php | 30 ++++++++++++++ .../ZipExports/Models/ZipExportChapter.php | 26 +++++++++++++ .../ZipExports/Models/ZipExportImage.php | 17 ++++++++ .../ZipExports/Models/ZipExportModel.php | 28 +++++++++++++ .../ZipExports/Models/ZipExportPage.php | 31 +++++++++++++++ .../ZipExports/Models/ZipExportTag.php | 16 ++++++++ app/Exports/ZipExports/ZipExportReader.php | 32 ++++++--------- app/Exports/ZipExports/ZipExportValidator.php | 1 - database/factories/Exports/ImportFactory.php | 5 +-- ...2024_11_02_160700_create_imports_table.php | 7 ++-- lang/en/entities.php | 8 ++-- resources/sass/styles.scss | 5 +++ resources/views/exports/import-show.blade.php | 39 ++++++------------- .../views/exports/parts/import-item.blade.php | 26 +++++++++++++ .../views/exports/parts/import.blade.php | 11 +----- tests/Exports/ZipImportTest.php | 31 +++++++++++---- 20 files changed, 303 insertions(+), 105 deletions(-) create mode 100644 resources/views/exports/parts/import-item.blade.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index a2389c725f3..3a56ed03456 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -65,10 +65,13 @@ public function show(int $id) { $import = $this->imports->findVisible($id); +// dd($import->decodeMetadata()); + $this->setPageTitle(trans('entities.import_continue')); return view('exports.import-show', [ 'import' => $import, + 'data' => $import->decodeMetadata(), ]); } @@ -89,7 +92,7 @@ public function run(int $id, Request $request) // TODO - Validate again before // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) // TODO - Redirect to result - // TOOD - Or redirect back with errors + // TODO - Or redirect back with errors } /** diff --git a/app/Exports/Import.php b/app/Exports/Import.php index 8400382fd0d..9c1771c468f 100644 --- a/app/Exports/Import.php +++ b/app/Exports/Import.php @@ -3,6 +3,9 @@ namespace BookStack\Exports; use BookStack\Activity\Models\Loggable; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Users\Models\User; use Carbon\Carbon; use Illuminate\Database\Eloquent\Factories\HasFactory; @@ -14,9 +17,8 @@ * @property string $path * @property string $name * @property int $size - ZIP size in bytes - * @property int $book_count - * @property int $chapter_count - * @property int $page_count + * @property string $type + * @property string $metadata * @property int $created_by * @property Carbon $created_at * @property Carbon $updated_at @@ -26,24 +28,6 @@ class Import extends Model implements Loggable { use HasFactory; - public const TYPE_BOOK = 'book'; - public const TYPE_CHAPTER = 'chapter'; - public const TYPE_PAGE = 'page'; - - /** - * Get the type (model) that this import is intended to be. - */ - public function getType(): string - { - if ($this->book_count === 1) { - return self::TYPE_BOOK; - } elseif ($this->chapter_count === 1) { - return self::TYPE_CHAPTER; - } - - return self::TYPE_PAGE; - } - public function getSizeString(): string { $mb = round($this->size / 1000000, 2); @@ -68,4 +52,15 @@ public function createdBy(): BelongsTo { return $this->belongsTo(User::class, 'created_by'); } + + public function decodeMetadata(): ZipExportBook|ZipExportChapter|ZipExportPage|null + { + $metadataArray = json_decode($this->metadata, true); + return match ($this->type) { + 'book' => ZipExportBook::fromArray($metadataArray), + 'chapter' => ZipExportChapter::fromArray($metadataArray), + 'page' => ZipExportPage::fromArray($metadataArray), + default => null, + }; + } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index d7e169ad166..3265e1c80dc 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,7 +2,12 @@ namespace BookStack\Exports; +use BookStack\Exceptions\FileUploadException; +use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipValidationException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Uploads\FileStorage; @@ -41,6 +46,11 @@ public function findVisible(int $id): Import return $query->findOrFail($id); } + /** + * @throws FileUploadException + * @throws ZipValidationException + * @throws ZipExportException + */ public function storeFromUpload(UploadedFile $file): Import { $zipPath = $file->getRealPath(); @@ -50,15 +60,23 @@ public function storeFromUpload(UploadedFile $file): Import throw new ZipValidationException($errors); } - $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); + $reader = new ZipExportReader($zipPath); + $exportModel = $reader->decodeDataToExportModel(); + $import = new Import(); - $import->name = $zipEntityInfo['name']; - $import->book_count = $zipEntityInfo['book_count']; - $import->chapter_count = $zipEntityInfo['chapter_count']; - $import->page_count = $zipEntityInfo['page_count']; + $import->type = match (get_class($exportModel)) { + ZipExportPage::class => 'page', + ZipExportChapter::class => 'chapter', + ZipExportBook::class => 'book', + }; + + $import->name = $exportModel->name; $import->created_by = user()->id; $import->size = filesize($zipPath); + $exportModel->metadataOnly(); + $import->metadata = json_encode($exportModel); + $path = $this->storage->uploadFile( $file, 'uploads/files/imports/', @@ -72,6 +90,13 @@ public function storeFromUpload(UploadedFile $file): Import return $import; } + public function runImport(Import $import, ?string $parent = null) + { + // TODO - Download import zip (if needed) + // TODO - Validate zip file again + // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) + } + public function deleteImport(Import $import): void { $this->storage->delete($import->path); diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index e586b91b0ee..1dbdc7333e8 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -14,6 +14,11 @@ class ZipExportAttachment extends ZipExportModel public ?string $link = null; public ?string $file = null; + public function metadataOnly(): void + { + $this->order = $this->link = $this->file = null; + } + public static function fromModel(Attachment $model, ZipExportFiles $files): self { $instance = new self(); @@ -49,4 +54,17 @@ public static function validate(ZipValidationHelper $context, array $data): arra return $context->validateData($data, $rules); } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->order = isset($data['order']) ? intval($data['order']) : null; + $model->link = $data['link'] ?? null; + $model->file = $data['file'] ?? null; + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 7e1f2d8106e..0dc4e93d43c 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -21,6 +21,21 @@ class ZipExportBook extends ZipExportModel /** @var ZipExportTag[] */ public array $tags = []; + public function metadataOnly(): void + { + $this->description_html = $this->cover = null; + + foreach ($this->chapters as $chapter) { + $chapter->metadataOnly(); + } + foreach ($this->pages as $page) { + $page->metadataOnly(); + } + foreach ($this->tags as $tag) { + $tag->metadataOnly(); + } + } + public static function fromModel(Book $model, ZipExportFiles $files): self { $instance = new self(); @@ -71,4 +86,19 @@ public static function validate(ZipValidationHelper $context, array $data): arra return $errors; } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->description_html = $data['description_html'] ?? null; + $model->cover = $data['cover'] ?? null; + $model->tags = ZipExportTag::fromManyArray($data['tags'] ?? []); + $model->pages = ZipExportPage::fromManyArray($data['pages'] ?? []); + $model->chapters = ZipExportChapter::fromManyArray($data['chapters'] ?? []); + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index 03df31b7078..50440d61a5a 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -18,6 +18,18 @@ class ZipExportChapter extends ZipExportModel /** @var ZipExportTag[] */ public array $tags = []; + public function metadataOnly(): void + { + $this->description_html = $this->priority = null; + + foreach ($this->pages as $page) { + $page->metadataOnly(); + } + foreach ($this->tags as $tag) { + $tag->metadataOnly(); + } + } + public static function fromModel(Chapter $model, ZipExportFiles $files): self { $instance = new self(); @@ -61,4 +73,18 @@ public static function validate(ZipValidationHelper $context, array $data): arra return $errors; } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->description_html = $data['description_html'] ?? null; + $model->priority = isset($data['priority']) ? intval($data['priority']) : null; + $model->tags = ZipExportTag::fromManyArray($data['tags'] ?? []); + $model->pages = ZipExportPage::fromManyArray($data['pages'] ?? []); + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 3388c66df36..691eb918fc6 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -25,6 +25,11 @@ public static function fromModel(Image $model, ZipExportFiles $files): self return $instance; } + public function metadataOnly(): void + { + // + } + public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ @@ -36,4 +41,16 @@ public static function validate(ZipValidationHelper $context, array $data): arra return $context->validateData($data, $rules); } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->file = $data['file']; + $model->type = $data['type']; + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportModel.php b/app/Exports/ZipExports/Models/ZipExportModel.php index 4d66f010f86..d3a8c35674b 100644 --- a/app/Exports/ZipExports/Models/ZipExportModel.php +++ b/app/Exports/ZipExports/Models/ZipExportModel.php @@ -26,4 +26,32 @@ public function jsonSerialize(): array * item in the array for its own validation messages. */ abstract public static function validate(ZipValidationHelper $context, array $data): array; + + /** + * Decode the array of data into this export model. + */ + abstract public static function fromArray(array $data): self; + + /** + * Decode an array of array data into an array of export models. + * @param array[] $data + * @return self[] + */ + public static function fromManyArray(array $data): array + { + $results = []; + foreach ($data as $item) { + $results[] = static::fromArray($item); + } + return $results; + } + + /** + * Remove additional content in this model to reduce it down + * to just essential id/name values for identification. + * + * The result of this may be something that does not pass validation, but is + * simple for the purpose of creating a contents. + */ + abstract public function metadataOnly(): void; } diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 2c8b9a88abd..3a876e7aaff 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -21,6 +21,21 @@ class ZipExportPage extends ZipExportModel /** @var ZipExportTag[] */ public array $tags = []; + public function metadataOnly(): void + { + $this->html = $this->markdown = $this->priority = null; + + foreach ($this->attachments as $attachment) { + $attachment->metadataOnly(); + } + foreach ($this->images as $image) { + $image->metadataOnly(); + } + foreach ($this->tags as $tag) { + $tag->metadataOnly(); + } + } + public static function fromModel(Page $model, ZipExportFiles $files): self { $instance = new self(); @@ -70,4 +85,20 @@ public static function validate(ZipValidationHelper $context, array $data): arra return $errors; } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->html = $data['html'] ?? null; + $model->markdown = $data['markdown'] ?? null; + $model->priority = isset($data['priority']) ? intval($data['priority']) : null; + $model->attachments = ZipExportAttachment::fromManyArray($data['attachments'] ?? []); + $model->images = ZipExportImage::fromManyArray($data['images'] ?? []); + $model->tags = ZipExportTag::fromManyArray($data['tags'] ?? []); + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index 99abb811c06..b6c9e338aef 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -11,6 +11,11 @@ class ZipExportTag extends ZipExportModel public ?string $value = null; public ?int $order = null; + public function metadataOnly(): void + { + $this->value = $this->order = null; + } + public static function fromModel(Tag $model): self { $instance = new self(); @@ -36,4 +41,15 @@ public static function validate(ZipValidationHelper $context, array $data): arra return $context->validateData($data, $rules); } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->name = $data['name']; + $model->value = $data['value'] ?? null; + $model->order = isset($data['order']) ? intval($data['order']) : null; + + return $model; + } } diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index 7187a18897d..c3e47da048d 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -3,6 +3,10 @@ namespace BookStack\Exports\ZipExports; use BookStack\Exceptions\ZipExportException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportModel; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use ZipArchive; class ZipExportReader @@ -71,32 +75,18 @@ public function fileExists(string $fileName): bool /** * @throws ZipExportException - * @returns array{name: string, book_count: int, chapter_count: int, page_count: int} */ - public function getEntityInfo(): array + public function decodeDataToExportModel(): ZipExportBook|ZipExportChapter|ZipExportPage { $data = $this->readData(); - $info = ['name' => '', 'book_count' => 0, 'chapter_count' => 0, 'page_count' => 0]; - if (isset($data['book'])) { - $info['name'] = $data['book']['name'] ?? ''; - $info['book_count']++; - $chapters = $data['book']['chapters'] ?? []; - $pages = $data['book']['pages'] ?? []; - $info['chapter_count'] += count($chapters); - $info['page_count'] += count($pages); - foreach ($chapters as $chapter) { - $info['page_count'] += count($chapter['pages'] ?? []); - } - } elseif (isset($data['chapter'])) { - $info['name'] = $data['chapter']['name'] ?? ''; - $info['chapter_count']++; - $info['page_count'] += count($data['chapter']['pages'] ?? []); - } elseif (isset($data['page'])) { - $info['name'] = $data['page']['name'] ?? ''; - $info['page_count']++; + return ZipExportBook::fromArray($data['book']); + } else if (isset($data['chapter'])) { + return ZipExportChapter::fromArray($data['chapter']); + } else if (isset($data['page'])) { + return ZipExportPage::fromArray($data['page']); } - return $info; + throw new ZipExportException("Could not identify content in ZIP file data."); } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index e476998c216..e27ae53c774 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -38,7 +38,6 @@ public function validate(): array return ['format' => trans('errors.import_zip_no_data')]; } - return $this->flattenModelErrors($modelErrors, $keyPrefix); } diff --git a/database/factories/Exports/ImportFactory.php b/database/factories/Exports/ImportFactory.php index 55378d5832e..74a2bcd65f3 100644 --- a/database/factories/Exports/ImportFactory.php +++ b/database/factories/Exports/ImportFactory.php @@ -23,9 +23,8 @@ public function definition(): array return [ 'path' => 'uploads/imports/' . Str::random(10) . '.zip', 'name' => $this->faker->words(3, true), - 'book_count' => 1, - 'chapter_count' => 5, - 'page_count' => 15, + 'type' => 'book', + 'metadata' => '{"name": "My book"}', 'created_at' => User::factory(), ]; } diff --git a/database/migrations/2024_11_02_160700_create_imports_table.php b/database/migrations/2024_11_02_160700_create_imports_table.php index ed188226981..0784591b8e3 100644 --- a/database/migrations/2024_11_02_160700_create_imports_table.php +++ b/database/migrations/2024_11_02_160700_create_imports_table.php @@ -16,10 +16,9 @@ public function up(): void $table->string('name'); $table->string('path'); $table->integer('size'); - $table->integer('book_count'); - $table->integer('chapter_count'); - $table->integer('page_count'); - $table->integer('created_by'); + $table->string('type'); + $table->longText('metadata'); + $table->integer('created_by')->index(); $table->timestamps(); }); } diff --git a/lang/en/entities.php b/lang/en/entities.php index 065eb043a11..ae1c1e8d4cc 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -45,7 +45,7 @@ 'default_template_select' => 'Select a template page', 'import' => 'Import', 'import_validate' => 'Validate Import', - 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', + 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', 'import_zip_select' => 'Select ZIP file to upload', 'import_zip_validation_errors' => 'Errors were detected while validating the provided ZIP file:', 'import_pending' => 'Pending Imports', @@ -53,9 +53,9 @@ 'import_continue' => 'Continue Import', 'import_continue_desc' => 'Review the content due to be imported from the uploaded ZIP file. When ready, run the import to add its contents to this system. The uploaded ZIP import file will be automatically removed on successful import.', 'import_run' => 'Run Import', - 'import_size' => 'Import ZIP Size:', - 'import_uploaded_at' => 'Uploaded:', - 'import_uploaded_by' => 'Uploaded by:', + 'import_size' => ':size Import ZIP Size', + 'import_uploaded_at' => 'Uploaded :relativeTime', + 'import_uploaded_by' => 'Uploaded by', 'import_location' => 'Import Location', 'import_location_desc' => 'Select a target location for your imported content. You\'ll need the relevant permissions to create within the location you choose.', 'import_delete_confirm' => 'Are you sure you want to delete this import?', diff --git a/resources/sass/styles.scss b/resources/sass/styles.scss index 942265d04d8..2cf3cbf8221 100644 --- a/resources/sass/styles.scss +++ b/resources/sass/styles.scss @@ -248,4 +248,9 @@ $loadingSize: 10px; transform: rotate(180deg); } } +} + +.import-item { + border-inline-start: 2px solid currentColor; + padding-inline-start: $-xs; } \ No newline at end of file diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index 63977947dba..40867377fb0 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -1,11 +1,6 @@ @extends('layouts.simple') @section('body') - - @php - $type = $import->getType(); - @endphp -
    @@ -13,29 +8,17 @@

    {{ trans('entities.import_continue_desc') }}

    - -
    + +
    -

    @icon($type) {{ $import->name }}

    - @if($type === 'book') -

    @icon('chapter') {{ trans_choice('entities.x_chapters', $import->chapter_count) }}

    - @endif - @if($type === 'book' || $type === 'chapter') -

    @icon('page') {{ trans_choice('entities.x_pages', $import->page_count) }}

    - @endif + @include('exports.parts.import-item', ['type' => $import->type, 'model' => $data])
    -
    -
    - {{ trans('entities.import_size') }} - {{ $import->getSizeString() }} -
    -
    - {{ trans('entities.import_uploaded_at') }} - {{ $import->created_at->diffForHumans() }} -
    +
    +
    {{ trans('entities.import_size', ['size' => $import->getSizeString()]) }}
    +
    {{ trans('entities.import_uploaded_at', ['relativeTime' => $import->created_at->diffForHumans()]) }}
    @if($import->createdBy) -
    - {{ trans('entities.import_uploaded_by') }} +
    + {{ trans('entities.import_uploaded_by') }} {{ $import->createdBy->name }}
    @endif @@ -48,14 +31,14 @@ method="POST"> {{ csrf_field() }} - @if($type === 'page' || $type === 'chapter') + @if($import->type === 'page' || $import->type === 'chapter')

    {{ trans('entities.import_location_desc') }}

    @include('entities.selector', [ 'name' => 'parent', - 'entityTypes' => $type === 'page' ? 'chapter,book' : 'book', - 'entityPermission' => "{$type}-create", + 'entityTypes' => $import->type === 'page' ? 'chapter,book' : 'book', + 'entityPermission' => "{$import->type}-create", 'selectorSize' => 'compact small', ]) @include('form.errors', ['name' => 'parent']) diff --git a/resources/views/exports/parts/import-item.blade.php b/resources/views/exports/parts/import-item.blade.php new file mode 100644 index 00000000000..811a3b31bda --- /dev/null +++ b/resources/views/exports/parts/import-item.blade.php @@ -0,0 +1,26 @@ +{{-- +$type - string +$model - object +--}} +
    +

    @icon($type){{ $model->name }}

    +
    +
    + @if($model->attachments ?? []) + @icon('attach'){{ count($model->attachments) }} + @endif + @if($model->images ?? []) + @icon('image'){{ count($model->images) }} + @endif + @if($model->tags ?? []) + @icon('tag'){{ count($model->tags) }} + @endif +
    + @foreach($model->chapters ?? [] as $chapter) + @include('exports.parts.import-item', ['type' => 'chapter', 'model' => $chapter]) + @endforeach + @foreach($model->pages ?? [] as $page) + @include('exports.parts.import-item', ['type' => 'page', 'model' => $page]) + @endforeach +
    +
    \ No newline at end of file diff --git a/resources/views/exports/parts/import.blade.php b/resources/views/exports/parts/import.blade.php index 5ff6600f24b..fd53095a422 100644 --- a/resources/views/exports/parts/import.blade.php +++ b/resources/views/exports/parts/import.blade.php @@ -1,18 +1,9 @@ -@php - $type = $import->getType(); -@endphp
    @icon($type) {{ $import->name }} + class="text-{{ $import->type }}">@icon($import->type) {{ $import->name }}
    - @if($type === 'book') -
    @icon('chapter') {{ $import->chapter_count }}
    - @endif - @if($type === 'book' || $type === 'chapter') -
    @icon('page') {{ $import->page_count }}
    - @endif
    {{ $import->getSizeString() }}
    @icon('time'){{ $import->created_at->diffForHumans() }}
    diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index b9a8598fabe..2b40100aabe 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -4,6 +4,9 @@ use BookStack\Activity\ActivityType; use BookStack\Exports\Import; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use Illuminate\Http\UploadedFile; use Illuminate\Testing\TestResponse; use Tests\TestCase; @@ -130,7 +133,7 @@ public function test_import_upload_success() { $admin = $this->users->admin(); $this->actingAs($admin); - $resp = $this->runImportFromFile($this->zipUploadFromData([ + $data = [ 'book' => [ 'name' => 'My great book name', 'chapters' => [ @@ -149,13 +152,13 @@ public function test_import_upload_success() ] ], ], - ])); + ]; + + $resp = $this->runImportFromFile($this->zipUploadFromData($data)); $this->assertDatabaseHas('imports', [ 'name' => 'My great book name', - 'book_count' => 1, - 'chapter_count' => 1, - 'page_count' => 2, + 'type' => 'book', 'created_by' => $admin->id, ]); @@ -168,11 +171,25 @@ public function test_import_upload_success() public function test_import_show_page() { - $import = Import::factory()->create(['name' => 'MySuperAdminImport']); + $exportBook = new ZipExportBook(); + $exportBook->name = 'My exported book'; + $exportChapter = new ZipExportChapter(); + $exportChapter->name = 'My exported chapter'; + $exportPage = new ZipExportPage(); + $exportPage->name = 'My exported page'; + $exportBook->chapters = [$exportChapter]; + $exportChapter->pages = [$exportPage]; + + $import = Import::factory()->create([ + 'name' => 'MySuperAdminImport', + 'metadata' => json_encode($exportBook) + ]); $resp = $this->asAdmin()->get("/import/{$import->id}"); $resp->assertOk(); - $resp->assertSee('MySuperAdminImport'); + $resp->assertSeeText('My exported book'); + $resp->assertSeeText('My exported chapter'); + $resp->assertSeeText('My exported page'); } public function test_import_show_page_access_limited() From 7b84558ca1deb0a605a2f632e60baaad325615e7 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 5 Nov 2024 15:41:58 +0000 Subject: [PATCH 25/41] ZIP Imports: Added parent and permission check pre-import --- app/Exceptions/ZipImportException.php | 12 ++ app/Exports/Controllers/ImportController.php | 2 - app/Exports/ImportRepo.php | 20 ++- app/Exports/ZipExports/ZipExportValidator.php | 7 +- app/Exports/ZipExports/ZipImportRunner.php | 143 ++++++++++++++++++ app/Uploads/FileStorage.php | 23 ++- 6 files changed, 195 insertions(+), 12 deletions(-) create mode 100644 app/Exceptions/ZipImportException.php create mode 100644 app/Exports/ZipExports/ZipImportRunner.php diff --git a/app/Exceptions/ZipImportException.php b/app/Exceptions/ZipImportException.php new file mode 100644 index 00000000000..2403c514477 --- /dev/null +++ b/app/Exceptions/ZipImportException.php @@ -0,0 +1,12 @@ +imports->findVisible($id); -// dd($import->decodeMetadata()); - $this->setPageTitle(trans('entities.import_continue')); return view('exports.import-show', [ diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index 3265e1c80dc..b94563545a4 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,6 +2,7 @@ namespace BookStack\Exports; +use BookStack\Entities\Queries\EntityQueries; use BookStack\Exceptions\FileUploadException; use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipValidationException; @@ -10,6 +11,7 @@ use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; +use BookStack\Exports\ZipExports\ZipImportRunner; use BookStack\Uploads\FileStorage; use Illuminate\Database\Eloquent\Collection; use Symfony\Component\HttpFoundation\File\UploadedFile; @@ -18,6 +20,8 @@ class ImportRepo { public function __construct( protected FileStorage $storage, + protected ZipImportRunner $importer, + protected EntityQueries $entityQueries, ) { } @@ -54,13 +58,13 @@ public function findVisible(int $id): Import public function storeFromUpload(UploadedFile $file): Import { $zipPath = $file->getRealPath(); + $reader = new ZipExportReader($zipPath); - $errors = (new ZipExportValidator($zipPath))->validate(); + $errors = (new ZipExportValidator($reader))->validate(); if ($errors) { throw new ZipValidationException($errors); } - $reader = new ZipExportReader($zipPath); $exportModel = $reader->decodeDataToExportModel(); $import = new Import(); @@ -90,11 +94,17 @@ public function storeFromUpload(UploadedFile $file): Import return $import; } + /** + * @throws ZipValidationException + */ public function runImport(Import $import, ?string $parent = null) { - // TODO - Download import zip (if needed) - // TODO - Validate zip file again - // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) + $parentModel = null; + if ($import->type === 'page' || $import->type === 'chapter') { + $parentModel = $parent ? $this->entityQueries->findVisibleByStringIdentifier($parent) : null; + } + + return $this->importer->run($import, $parentModel); } public function deleteImport(Import $import): void diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index e27ae53c774..889804f2013 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -10,20 +10,19 @@ class ZipExportValidator { public function __construct( - protected string $zipPath, + protected ZipExportReader $reader, ) { } public function validate(): array { - $reader = new ZipExportReader($this->zipPath); try { - $importData = $reader->readData(); + $importData = $this->reader->readData(); } catch (ZipExportException $exception) { return ['format' => $exception->getMessage()]; } - $helper = new ZipValidationHelper($reader); + $helper = new ZipValidationHelper($this->reader); if (isset($importData['book'])) { $modelErrors = ZipExportBook::validate($helper, $importData['book']); diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php new file mode 100644 index 00000000000..2f784ebea6e --- /dev/null +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -0,0 +1,143 @@ +getZipPath($import); + $reader = new ZipExportReader($zipPath); + + $errors = (new ZipExportValidator($reader))->validate(); + if ($errors) { + throw new ZipImportException(["ZIP failed to validate"]); + } + + try { + $exportModel = $reader->decodeDataToExportModel(); + } catch (ZipExportException $e) { + throw new ZipImportException([$e->getMessage()]); + } + + // Validate parent type + if ($exportModel instanceof ZipExportBook && ($parent !== null)) { + throw new ZipImportException(["Must not have a parent set for a Book import"]); + } else if ($exportModel instanceof ZipExportChapter && (!$parent instanceof Book)) { + throw new ZipImportException(["Parent book required for chapter import"]); + } else if ($exportModel instanceof ZipExportPage && !($parent instanceof Book || $parent instanceof Chapter)) { + throw new ZipImportException(["Parent book or chapter required for page import"]); + } + + $this->ensurePermissionsPermitImport($exportModel); + + // TODO - Run import + } + + /** + * @throws ZipImportException + */ + protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter|ZipExportBook $exportModel, Book|Chapter|null $parent = null): void + { + $errors = []; + + // TODO - Extract messages to language files + // TODO - Ensure these are shown to users on failure + + $chapters = []; + $pages = []; + $images = []; + $attachments = []; + + if ($exportModel instanceof ZipExportBook) { + if (!userCan('book-create-all')) { + $errors[] = 'You are lacking the required permission to create books.'; + } + array_push($pages, ...$exportModel->pages); + array_push($chapters, ...$exportModel->chapters); + } else if ($exportModel instanceof ZipExportChapter) { + $chapters[] = $exportModel; + } else if ($exportModel instanceof ZipExportPage) { + $pages[] = $exportModel; + } + + foreach ($chapters as $chapter) { + array_push($pages, ...$chapter->pages); + } + + if (count($chapters) > 0) { + $permission = 'chapter-create' . ($parent ? '' : '-all'); + if (!userCan($permission, $parent)) { + $errors[] = 'You are lacking the required permission to create chapters.'; + } + } + + foreach ($pages as $page) { + array_push($attachments, ...$page->attachments); + array_push($images, ...$page->images); + } + + if (count($pages) > 0) { + if ($parent) { + if (!userCan('page-create', $parent)) { + $errors[] = 'You are lacking the required permission to create pages.'; + } + } else { + $hasPermission = userCan('page-create-all') || userCan('page-create-own'); + if (!$hasPermission) { + $errors[] = 'You are lacking the required permission to create pages.'; + } + } + } + + if (count($images) > 0) { + if (!userCan('image-create-all')) { + $errors[] = 'You are lacking the required permissions to create images.'; + } + } + + if (count($attachments) > 0) { + if (userCan('attachment-create-all')) { + $errors[] = 'You are lacking the required permissions to create attachments.'; + } + } + + if (count($errors)) { + throw new ZipImportException($errors); + } + } + + protected function getZipPath(Import $import): string + { + if (!$this->storage->isRemote()) { + return $this->storage->getSystemPath($import->path); + } + + $tempFilePath = tempnam(sys_get_temp_dir(), 'bszip-import-'); + $tempFile = fopen($tempFilePath, 'wb'); + $stream = $this->storage->getReadStream($import->path); + stream_copy_to_stream($stream, $tempFile); + fclose($tempFile); + + return $tempFilePath; + } +} diff --git a/app/Uploads/FileStorage.php b/app/Uploads/FileStorage.php index 278484e519d..e6ac368d000 100644 --- a/app/Uploads/FileStorage.php +++ b/app/Uploads/FileStorage.php @@ -5,6 +5,7 @@ use BookStack\Exceptions\FileUploadException; use Exception; use Illuminate\Contracts\Filesystem\Filesystem as Storage; +use Illuminate\Filesystem\FilesystemAdapter; use Illuminate\Filesystem\FilesystemManager; use Illuminate\Support\Facades\Log; use Illuminate\Support\Str; @@ -70,6 +71,26 @@ public function uploadFile(UploadedFile $file, string $subDirectory, string $suf return $filePath; } + /** + * Check whether the configured storage is remote from the host of this app. + */ + public function isRemote(): bool + { + return $this->getStorageDiskName() === 's3'; + } + + /** + * Get the actual path on system for the given relative file path. + */ + public function getSystemPath(string $filePath): string + { + if ($this->isRemote()) { + return ''; + } + + return storage_path('uploads/files/' . ltrim($this->adjustPathForStorageDisk($filePath), '/')); + } + /** * Get the storage that will be used for storing files. */ @@ -83,7 +104,7 @@ protected function getStorageDisk(): Storage */ protected function getStorageDiskName(): string { - $storageType = config('filesystems.attachments'); + $storageType = trim(strtolower(config('filesystems.attachments'))); // Change to our secure-attachment disk if any of the local options // are used to prevent escaping that location. From d13e4d2eefeed427c0377be04761a639e9fdb8fc Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 9 Nov 2024 14:01:24 +0000 Subject: [PATCH 26/41] ZIP imports: Started actual import logic --- app/Entities/Tools/Cloner.php | 17 +-- .../ZipExports/Models/ZipExportAttachment.php | 6 +- .../ZipExports/Models/ZipExportTag.php | 6 +- app/Exports/ZipExports/ZipExportReader.php | 8 ++ app/Exports/ZipExports/ZipImportRunner.php | 107 ++++++++++++++++++ dev/docs/portable-zip-file-format.md | 4 +- 6 files changed, 124 insertions(+), 24 deletions(-) diff --git a/app/Entities/Tools/Cloner.php b/app/Entities/Tools/Cloner.php index 2030b050c4b..2be6083e3dd 100644 --- a/app/Entities/Tools/Cloner.php +++ b/app/Entities/Tools/Cloner.php @@ -18,17 +18,12 @@ class Cloner { - protected PageRepo $pageRepo; - protected ChapterRepo $chapterRepo; - protected BookRepo $bookRepo; - protected ImageService $imageService; - - public function __construct(PageRepo $pageRepo, ChapterRepo $chapterRepo, BookRepo $bookRepo, ImageService $imageService) - { - $this->pageRepo = $pageRepo; - $this->chapterRepo = $chapterRepo; - $this->bookRepo = $bookRepo; - $this->imageService = $imageService; + public function __construct( + protected PageRepo $pageRepo, + protected ChapterRepo $chapterRepo, + protected BookRepo $bookRepo, + protected ImageService $imageService, + ) { } /** diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index 1dbdc7333e8..c6615e1dc49 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -10,13 +10,12 @@ class ZipExportAttachment extends ZipExportModel { public ?int $id = null; public string $name; - public ?int $order = null; public ?string $link = null; public ?string $file = null; public function metadataOnly(): void { - $this->order = $this->link = $this->file = null; + $this->link = $this->file = null; } public static function fromModel(Attachment $model, ZipExportFiles $files): self @@ -24,7 +23,6 @@ public static function fromModel(Attachment $model, ZipExportFiles $files): self $instance = new self(); $instance->id = $model->id; $instance->name = $model->name; - $instance->order = $model->order; if ($model->external) { $instance->link = $model->path; @@ -47,7 +45,6 @@ public static function validate(ZipValidationHelper $context, array $data): arra $rules = [ 'id' => ['nullable', 'int'], 'name' => ['required', 'string', 'min:1'], - 'order' => ['nullable', 'integer'], 'link' => ['required_without:file', 'nullable', 'string'], 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], ]; @@ -61,7 +58,6 @@ public static function fromArray(array $data): self $model->id = $data['id'] ?? null; $model->name = $data['name']; - $model->order = isset($data['order']) ? intval($data['order']) : null; $model->link = $data['link'] ?? null; $model->file = $data['file'] ?? null; diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index b6c9e338aef..6b4720fca7e 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -9,11 +9,10 @@ class ZipExportTag extends ZipExportModel { public string $name; public ?string $value = null; - public ?int $order = null; public function metadataOnly(): void { - $this->value = $this->order = null; + $this->value = null; } public static function fromModel(Tag $model): self @@ -21,7 +20,6 @@ public static function fromModel(Tag $model): self $instance = new self(); $instance->name = $model->name; $instance->value = $model->value; - $instance->order = $model->order; return $instance; } @@ -36,7 +34,6 @@ public static function validate(ZipValidationHelper $context, array $data): arra $rules = [ 'name' => ['required', 'string', 'min:1'], 'value' => ['nullable', 'string'], - 'order' => ['nullable', 'integer'], ]; return $context->validateData($data, $rules); @@ -48,7 +45,6 @@ public static function fromArray(array $data): self $model->name = $data['name']; $model->value = $data['value'] ?? null; - $model->order = isset($data['order']) ? intval($data['order']) : null; return $model; } diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index c3e47da048d..ebc2fbbc9e1 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -73,6 +73,14 @@ public function fileExists(string $fileName): bool return $this->zip->statName("files/{$fileName}") !== false; } + /** + * @return false|resource + */ + public function streamFile(string $fileName) + { + return $this->zip->getStream("files/{$fileName}"); + } + /** * @throws ZipExportException */ diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 2f784ebea6e..2b897ff9167 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -5,18 +5,33 @@ use BookStack\Entities\Models\Book; use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Entity; +use BookStack\Entities\Models\Page; +use BookStack\Entities\Repos\BookRepo; +use BookStack\Entities\Repos\ChapterRepo; +use BookStack\Entities\Repos\PageRepo; use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipImportException; use BookStack\Exports\Import; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; +use BookStack\Exports\ZipExports\Models\ZipExportTag; use BookStack\Uploads\FileStorage; +use BookStack\Uploads\ImageService; +use Illuminate\Http\UploadedFile; class ZipImportRunner { + protected array $tempFilesToCleanup = []; // TODO + protected array $createdImages = []; // TODO + protected array $createdAttachments = []; // TODO + public function __construct( protected FileStorage $storage, + protected PageRepo $pageRepo, + protected ChapterRepo $chapterRepo, + protected BookRepo $bookRepo, + protected ImageService $imageService, ) { } @@ -51,6 +66,98 @@ public function run(Import $import, ?Entity $parent = null): void $this->ensurePermissionsPermitImport($exportModel); // TODO - Run import + // TODO - In transaction? + // TODO - Revert uploaded files if goes wrong + } + + protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book + { + $book = $this->bookRepo->create([ + 'name' => $exportBook->name, + 'description_html' => $exportBook->description_html ?? '', + 'image' => $exportBook->cover ? $this->zipFileToUploadedFile($exportBook->cover, $reader) : null, + 'tags' => $this->exportTagsToInputArray($exportBook->tags ?? []), + ]); + + // TODO - Parse/format description_html references + + if ($book->cover) { + $this->createdImages[] = $book->cover; + } + + // TODO - Pages + foreach ($exportBook->chapters as $exportChapter) { + $this->importChapter($exportChapter, $book); + } + // TODO - Sort chapters/pages by order + + return $book; + } + + protected function importChapter(ZipExportChapter $exportChapter, Book $parent, ZipExportReader $reader): Chapter + { + $chapter = $this->chapterRepo->create([ + 'name' => $exportChapter->name, + 'description_html' => $exportChapter->description_html ?? '', + 'tags' => $this->exportTagsToInputArray($exportChapter->tags ?? []), + ], $parent); + + // TODO - Parse/format description_html references + + $exportPages = $exportChapter->pages; + usort($exportPages, function (ZipExportPage $a, ZipExportPage $b) { + return ($a->priority ?? 0) - ($b->priority ?? 0); + }); + + foreach ($exportPages as $exportPage) { + // + } + // TODO - Pages + + return $chapter; + } + + protected function importPage(ZipExportPage $exportPage, Book|Chapter $parent, ZipExportReader $reader): Page + { + $page = $this->pageRepo->getNewDraftPage($parent); + + // TODO - Import attachments + // TODO - Import images + // TODO - Parse/format HTML + + $this->pageRepo->publishDraft($page, [ + 'name' => $exportPage->name, + 'markdown' => $exportPage->markdown, + 'html' => $exportPage->html, + 'tags' => $this->exportTagsToInputArray($exportPage->tags ?? []), + ]); + + return $page; + } + + protected function exportTagsToInputArray(array $exportTags): array + { + $tags = []; + + /** @var ZipExportTag $tag */ + foreach ($exportTags as $tag) { + $tags[] = ['name' => $tag->name, 'value' => $tag->value ?? '']; + } + + return $tags; + } + + protected function zipFileToUploadedFile(string $fileName, ZipExportReader $reader): UploadedFile + { + $tempPath = tempnam(sys_get_temp_dir(), 'bszipextract'); + $fileStream = $reader->streamFile($fileName); + $tempStream = fopen($tempPath, 'wb'); + stream_copy_to_stream($fileStream, $tempStream); + fclose($tempStream); + + $this->tempFilesToCleanup[] = $tempPath; + + return new UploadedFile($tempPath, $fileName); } /** diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 6cee7356d23..7e5df3f015b 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -135,12 +135,10 @@ embedded within it. - `name` - String, required, name of attachment. - `link` - String, semi-optional, URL of attachment. - `file` - String reference, semi-optional, reference to attachment file. -- `order` - Number, optional, integer order of the attachments (shown low to high). Either `link` or `file` must be present, as that will determine the type of attachment. #### Tag - `name` - String, required, name of the tag. -- `value` - String, optional, value of the tag (can be empty). -- `order` - Number, optional, integer order of the tags (shown low to high). \ No newline at end of file +- `value` - String, optional, value of the tag (can be empty). \ No newline at end of file From 378f0d595fe8aa5aca212e1c5ed22944bf8bf1b7 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 10 Nov 2024 16:03:50 +0000 Subject: [PATCH 27/41] ZIP Imports: Built out reference parsing/updating logic --- app/Entities/Repos/PageRepo.php | 13 +- .../ZipExports/ZipExportReferences.php | 8 +- .../ZipExports/ZipImportReferences.php | 142 ++++++++++++++++++ app/Exports/ZipExports/ZipImportRunner.php | 20 ++- app/Exports/ZipExports/ZipReferenceParser.php | 72 +++++++-- 5 files changed, 232 insertions(+), 23 deletions(-) create mode 100644 app/Exports/ZipExports/ZipImportReferences.php diff --git a/app/Entities/Repos/PageRepo.php b/app/Entities/Repos/PageRepo.php index 1bc15392cec..68b1c398f80 100644 --- a/app/Entities/Repos/PageRepo.php +++ b/app/Entities/Repos/PageRepo.php @@ -87,6 +87,17 @@ public function publishDraft(Page $draft, array $input): Page return $draft; } + /** + * Directly update the content for the given page from the provided input. + * Used for direct content access in a way that performs required changes + * (Search index & reference regen) without performing an official update. + */ + public function setContentFromInput(Page $page, array $input): void + { + $this->updateTemplateStatusAndContentFromInput($page, $input); + $this->baseRepo->update($page, []); + } + /** * Update a page in the system. */ @@ -121,7 +132,7 @@ public function update(Page $page, array $input): Page return $page; } - protected function updateTemplateStatusAndContentFromInput(Page $page, array $input) + protected function updateTemplateStatusAndContentFromInput(Page $page, array $input): void { if (isset($input['template']) && userCan('templates-manage')) { $page->template = ($input['template'] === 'true'); diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index c630c832b37..0de409fa19a 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -85,9 +85,9 @@ public function buildReferences(ZipExportFiles $files): void // Parse page content first foreach ($this->pages as $page) { $handler = $createHandler($page); - $page->html = $this->parser->parse($page->html ?? '', $handler); + $page->html = $this->parser->parseLinks($page->html ?? '', $handler); if ($page->markdown) { - $page->markdown = $this->parser->parse($page->markdown, $handler); + $page->markdown = $this->parser->parseLinks($page->markdown, $handler); } } @@ -95,7 +95,7 @@ public function buildReferences(ZipExportFiles $files): void foreach ($this->chapters as $chapter) { if ($chapter->description_html) { $handler = $createHandler($chapter); - $chapter->description_html = $this->parser->parse($chapter->description_html, $handler); + $chapter->description_html = $this->parser->parseLinks($chapter->description_html, $handler); } } @@ -103,7 +103,7 @@ public function buildReferences(ZipExportFiles $files): void foreach ($this->books as $book) { if ($book->description_html) { $handler = $createHandler($book); - $book->description_html = $this->parser->parse($book->description_html, $handler); + $book->description_html = $this->parser->parseLinks($book->description_html, $handler); } } } diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php new file mode 100644 index 00000000000..8062886e542 --- /dev/null +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -0,0 +1,142 @@ + */ + protected array $referenceMap = []; + + /** @var array */ + protected array $zipExportPageMap = []; + /** @var array */ + protected array $zipExportChapterMap = []; + /** @var array */ + protected array $zipExportBookMap = []; + + public function __construct( + protected ZipReferenceParser $parser, + protected BaseRepo $baseRepo, + protected PageRepo $pageRepo, + protected ImageResizer $imageResizer, + ) { + } + + protected function addReference(string $type, Model $model, ?int $importId): void + { + if ($importId) { + $key = $type . ':' . $importId; + $this->referenceMap[$key] = $model; + } + } + + public function addPage(Page $page, ZipExportPage $exportPage): void + { + $this->pages[] = $page; + $this->zipExportPageMap[$page->id] = $exportPage; + $this->addReference('page', $page, $exportPage->id); + } + + public function addChapter(Chapter $chapter, ZipExportChapter $exportChapter): void + { + $this->chapters[] = $chapter; + $this->zipExportChapterMap[$chapter->id] = $exportChapter; + $this->addReference('chapter', $chapter, $exportChapter->id); + } + + public function addBook(Book $book, ZipExportBook $exportBook): void + { + $this->books[] = $book; + $this->zipExportBookMap[$book->id] = $exportBook; + $this->addReference('book', $book, $exportBook->id); + } + + public function addAttachment(Attachment $attachment, ?int $importId): void + { + $this->attachments[] = $attachment; + $this->addReference('attachment', $attachment, $importId); + } + + public function addImage(Image $image, ?int $importId): void + { + $this->images[] = $image; + $this->addReference('image', $image, $importId); + } + + protected function handleReference(string $type, int $id): ?string + { + $key = $type . ':' . $id; + $model = $this->referenceMap[$key] ?? null; + if ($model instanceof Entity) { + return $model->getUrl(); + } else if ($model instanceof Image) { + if ($model->type === 'gallery') { + $this->imageResizer->loadGalleryThumbnailsForImage($model, false); + return $model->thumbs['gallery'] ?? $model->url; + } + + return $model->url; + } + + return null; + } + + public function replaceReferences(): void + { + foreach ($this->books as $book) { + $exportBook = $this->zipExportBookMap[$book->id]; + $content = $exportBook->description_html || ''; + $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); + + $this->baseRepo->update($book, [ + 'description_html' => $parsed, + ]); + } + + foreach ($this->chapters as $chapter) { + $exportChapter = $this->zipExportChapterMap[$chapter->id]; + $content = $exportChapter->description_html || ''; + $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); + + $this->baseRepo->update($chapter, [ + 'description_html' => $parsed, + ]); + } + + foreach ($this->pages as $page) { + $exportPage = $this->zipExportPageMap[$page->id]; + $contentType = $exportPage->markdown ? 'markdown' : 'html'; + $content = $exportPage->markdown ?: ($exportPage->html ?: ''); + $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); + + $this->pageRepo->setContentFromInput($page, [ + $contentType => $parsed, + ]); + } + } +} diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 2b897ff9167..345c22be153 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -23,8 +23,6 @@ class ZipImportRunner { protected array $tempFilesToCleanup = []; // TODO - protected array $createdImages = []; // TODO - protected array $createdAttachments = []; // TODO public function __construct( protected FileStorage $storage, @@ -32,6 +30,7 @@ public function __construct( protected ChapterRepo $chapterRepo, protected BookRepo $bookRepo, protected ImageService $imageService, + protected ZipImportReferences $references, ) { } @@ -68,6 +67,11 @@ public function run(Import $import, ?Entity $parent = null): void // TODO - Run import // TODO - In transaction? // TODO - Revert uploaded files if goes wrong + // TODO - Attachments + // TODO - Images + // (Both listed/stored in references) + + $this->references->replaceReferences(); } protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book @@ -82,15 +86,17 @@ protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader // TODO - Parse/format description_html references if ($book->cover) { - $this->createdImages[] = $book->cover; + $this->references->addImage($book->cover, null); } // TODO - Pages foreach ($exportBook->chapters as $exportChapter) { - $this->importChapter($exportChapter, $book); + $this->importChapter($exportChapter, $book, $reader); } // TODO - Sort chapters/pages by order + $this->references->addBook($book, $exportBook); + return $book; } @@ -114,6 +120,8 @@ protected function importChapter(ZipExportChapter $exportChapter, Book $parent, } // TODO - Pages + $this->references->addChapter($chapter, $exportChapter); + return $chapter; } @@ -122,7 +130,9 @@ protected function importPage(ZipExportPage $exportPage, Book|Chapter $parent, Z $page = $this->pageRepo->getNewDraftPage($parent); // TODO - Import attachments + // TODO - Add attachment references // TODO - Import images + // TODO - Add image references // TODO - Parse/format HTML $this->pageRepo->publishDraft($page, [ @@ -132,6 +142,8 @@ protected function importPage(ZipExportPage $exportPage, Book|Chapter $parent, Z 'tags' => $this->exportTagsToInputArray($exportPage->tags ?? []), ]); + $this->references->addPage($page, $exportPage); + return $page; } diff --git a/app/Exports/ZipExports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php index da43d1b366b..5929383b4dd 100644 --- a/app/Exports/ZipExports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -15,27 +15,23 @@ class ZipReferenceParser { /** - * @var CrossLinkModelResolver[] + * @var CrossLinkModelResolver[]|null */ - protected array $modelResolvers; + protected ?array $modelResolvers = null; - public function __construct(EntityQueries $queries) - { - $this->modelResolvers = [ - new PagePermalinkModelResolver($queries->pages), - new PageLinkModelResolver($queries->pages), - new ChapterLinkModelResolver($queries->chapters), - new BookLinkModelResolver($queries->books), - new ImageModelResolver(), - new AttachmentModelResolver(), - ]; + public function __construct( + protected EntityQueries $queries + ) { } /** * Parse and replace references in the given content. + * Calls the handler for each model link detected and replaces the link + * with the handler return value if provided. + * Returns the resulting content with links replaced. * @param callable(Model):(string|null) $handler */ - public function parse(string $content, callable $handler): string + public function parseLinks(string $content, callable $handler): string { $escapedBase = preg_quote(url('/'), '/'); $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#()]/"; @@ -59,13 +55,43 @@ public function parse(string $content, callable $handler): string return $content; } + /** + * Parse and replace references in the given content. + * Calls the handler for each reference detected and replaces the link + * with the handler return value if provided. + * Returns the resulting content string with references replaced. + * @param callable(string $type, int $id):(string|null) $handler + */ + public function parseReferences(string $content, callable $handler): string + { + $referenceRegex = '/\[\[bsexport:([a-z]+):(\d+)]]/'; + $matches = []; + preg_match_all($referenceRegex, $content, $matches); + + if (count($matches) < 3) { + return $content; + } + + for ($i = 0; $i < count($matches[0]); $i++) { + $referenceText = $matches[0][$i]; + $type = strtolower($matches[1][$i]); + $id = intval($matches[2][$i]); + $result = $handler($type, $id); + if ($result !== null) { + $content = str_replace($referenceText, $result, $content); + } + } + + return $content; + } + /** * Attempt to resolve the given link to a model using the instance model resolvers. */ protected function linkToModel(string $link): ?Model { - foreach ($this->modelResolvers as $resolver) { + foreach ($this->getModelResolvers() as $resolver) { $model = $resolver->resolve($link); if (!is_null($model)) { return $model; @@ -74,4 +100,22 @@ protected function linkToModel(string $link): ?Model return null; } + + protected function getModelResolvers(): array + { + if (isset($this->modelResolvers)) { + return $this->modelResolvers; + } + + $this->modelResolvers = [ + new PagePermalinkModelResolver($this->queries->pages), + new PageLinkModelResolver($this->queries->pages), + new ChapterLinkModelResolver($this->queries->chapters), + new BookLinkModelResolver($this->queries->books), + new ImageModelResolver(), + new AttachmentModelResolver(), + ]; + + return $this->modelResolvers; + } } From 48c101aa7ab5b77781f4cd536b654d037b5aa55e Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 11 Nov 2024 15:06:46 +0000 Subject: [PATCH 28/41] ZIP Imports: Finished off core import logic --- app/Exceptions/ZipImportException.php | 3 +- app/Exports/Controllers/ImportController.php | 11 +- app/Exports/ImportRepo.php | 6 +- .../ZipExports/ZipImportReferences.php | 4 +- app/Exports/ZipExports/ZipImportRunner.php | 117 +++++++++++++++--- 5 files changed, 113 insertions(+), 28 deletions(-) diff --git a/app/Exceptions/ZipImportException.php b/app/Exceptions/ZipImportException.php index 2403c514477..452365c6e88 100644 --- a/app/Exceptions/ZipImportException.php +++ b/app/Exceptions/ZipImportException.php @@ -7,6 +7,7 @@ class ZipImportException extends \Exception public function __construct( public array $errors ) { - parent::__construct(); + $message = "Import failed with errors:" . implode("\n", $this->errors); + parent::__construct($message); } } diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index ec5ac80808b..4d2c83090a3 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -79,18 +79,21 @@ public function run(int $id, Request $request) $import = $this->imports->findVisible($id); $parent = null; - if ($import->getType() === 'page' || $import->getType() === 'chapter') { + if ($import->type === 'page' || $import->type === 'chapter') { $data = $this->validate($request, [ 'parent' => ['required', 'string'] ]); $parent = $data['parent']; } - // TODO - Run import - // TODO - Validate again before - // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) + $entity = $this->imports->runImport($import, $parent); + if ($entity) { + $this->logActivity(ActivityType::IMPORT_RUN, $import); + return redirect($entity->getUrl()); + } // TODO - Redirect to result // TODO - Or redirect back with errors + return 'failed'; } /** diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index b94563545a4..d169d4845ab 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,9 +2,11 @@ namespace BookStack\Exports; +use BookStack\Entities\Models\Entity; use BookStack\Entities\Queries\EntityQueries; use BookStack\Exceptions\FileUploadException; use BookStack\Exceptions\ZipExportException; +use BookStack\Exceptions\ZipImportException; use BookStack\Exceptions\ZipValidationException; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; @@ -95,9 +97,9 @@ public function storeFromUpload(UploadedFile $file): Import } /** - * @throws ZipValidationException + * @throws ZipValidationException|ZipImportException */ - public function runImport(Import $import, ?string $parent = null) + public function runImport(Import $import, ?string $parent = null): ?Entity { $parentModel = null; if ($import->type === 'page' || $import->type === 'chapter') { diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php index 8062886e542..3bce16bbb13 100644 --- a/app/Exports/ZipExports/ZipImportReferences.php +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -110,7 +110,7 @@ public function replaceReferences(): void { foreach ($this->books as $book) { $exportBook = $this->zipExportBookMap[$book->id]; - $content = $exportBook->description_html || ''; + $content = $exportBook->description_html ?? ''; $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); $this->baseRepo->update($book, [ @@ -120,7 +120,7 @@ public function replaceReferences(): void foreach ($this->chapters as $chapter) { $exportChapter = $this->zipExportChapterMap[$chapter->id]; - $content = $exportChapter->description_html || ''; + $content = $exportChapter->description_html ?? ''; $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); $this->baseRepo->update($chapter, [ diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 345c22be153..9f19f03e2e5 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -12,17 +12,22 @@ use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipImportException; use BookStack\Exports\Import; +use BookStack\Exports\ZipExports\Models\ZipExportAttachment; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportImage; use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\Models\ZipExportTag; +use BookStack\Uploads\Attachment; +use BookStack\Uploads\AttachmentService; use BookStack\Uploads\FileStorage; +use BookStack\Uploads\Image; use BookStack\Uploads\ImageService; use Illuminate\Http\UploadedFile; class ZipImportRunner { - protected array $tempFilesToCleanup = []; // TODO + protected array $tempFilesToCleanup = []; public function __construct( protected FileStorage $storage, @@ -30,14 +35,19 @@ public function __construct( protected ChapterRepo $chapterRepo, protected BookRepo $bookRepo, protected ImageService $imageService, + protected AttachmentService $attachmentService, protected ZipImportReferences $references, ) { } /** + * Run the import. + * Performs re-validation on zip, validation on parent provided, and permissions for importing + * the planned content, before running the import process. + * Returns the top-level entity item which was imported. * @throws ZipImportException */ - public function run(Import $import, ?Entity $parent = null): void + public function run(Import $import, ?Entity $parent = null): ?Entity { $zipPath = $this->getZipPath($import); $reader = new ZipExportReader($zipPath); @@ -63,8 +73,16 @@ public function run(Import $import, ?Entity $parent = null): void } $this->ensurePermissionsPermitImport($exportModel); + $entity = null; + + if ($exportModel instanceof ZipExportBook) { + $entity = $this->importBook($exportModel, $reader); + } else if ($exportModel instanceof ZipExportChapter) { + $entity = $this->importChapter($exportModel, $parent, $reader); + } else if ($exportModel instanceof ZipExportPage) { + $entity = $this->importPage($exportModel, $parent, $reader); + } - // TODO - Run import // TODO - In transaction? // TODO - Revert uploaded files if goes wrong // TODO - Attachments @@ -72,6 +90,23 @@ public function run(Import $import, ?Entity $parent = null): void // (Both listed/stored in references) $this->references->replaceReferences(); + + $reader->close(); + $this->cleanup(); + + dd('stop'); + + // TODO - Delete import/zip after import? + // Do this in parent repo? + + return $entity; + } + + protected function cleanup() + { + foreach ($this->tempFilesToCleanup as $file) { + unlink($file); + } } protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book @@ -83,17 +118,26 @@ protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader 'tags' => $this->exportTagsToInputArray($exportBook->tags ?? []), ]); - // TODO - Parse/format description_html references - if ($book->cover) { $this->references->addImage($book->cover, null); } - // TODO - Pages - foreach ($exportBook->chapters as $exportChapter) { - $this->importChapter($exportChapter, $book, $reader); + $children = [ + ...$exportBook->chapters, + ...$exportBook->pages, + ]; + + usort($children, function (ZipExportPage|ZipExportChapter $a, ZipExportPage|ZipExportChapter $b) { + return ($a->priority ?? 0) - ($b->priority ?? 0); + }); + + foreach ($children as $child) { + if ($child instanceof ZipExportChapter) { + $this->importChapter($child, $book, $reader); + } else if ($child instanceof ZipExportPage) { + $this->importPage($child, $book, $reader); + } } - // TODO - Sort chapters/pages by order $this->references->addBook($book, $exportBook); @@ -108,17 +152,14 @@ protected function importChapter(ZipExportChapter $exportChapter, Book $parent, 'tags' => $this->exportTagsToInputArray($exportChapter->tags ?? []), ], $parent); - // TODO - Parse/format description_html references - $exportPages = $exportChapter->pages; usort($exportPages, function (ZipExportPage $a, ZipExportPage $b) { return ($a->priority ?? 0) - ($b->priority ?? 0); }); foreach ($exportPages as $exportPage) { - // + $this->importPage($exportPage, $chapter, $reader); } - // TODO - Pages $this->references->addChapter($chapter, $exportChapter); @@ -129,11 +170,13 @@ protected function importPage(ZipExportPage $exportPage, Book|Chapter $parent, Z { $page = $this->pageRepo->getNewDraftPage($parent); - // TODO - Import attachments - // TODO - Add attachment references - // TODO - Import images - // TODO - Add image references - // TODO - Parse/format HTML + foreach ($exportPage->attachments as $exportAttachment) { + $this->importAttachment($exportAttachment, $page, $reader); + } + + foreach ($exportPage->images as $exportImage) { + $this->importImage($exportImage, $page, $reader); + } $this->pageRepo->publishDraft($page, [ 'name' => $exportPage->name, @@ -147,6 +190,40 @@ protected function importPage(ZipExportPage $exportPage, Book|Chapter $parent, Z return $page; } + protected function importAttachment(ZipExportAttachment $exportAttachment, Page $page, ZipExportReader $reader): Attachment + { + if ($exportAttachment->file) { + $file = $this->zipFileToUploadedFile($exportAttachment->file, $reader); + $attachment = $this->attachmentService->saveNewUpload($file, $page->id); + $attachment->name = $exportAttachment->name; + $attachment->save(); + } else { + $attachment = $this->attachmentService->saveNewFromLink( + $exportAttachment->name, + $exportAttachment->link ?? '', + $page->id, + ); + } + + $this->references->addAttachment($attachment, $exportAttachment->id); + + return $attachment; + } + + protected function importImage(ZipExportImage $exportImage, Page $page, ZipExportReader $reader): Image + { + $file = $this->zipFileToUploadedFile($exportImage->file, $reader); + $image = $this->imageService->saveNewFromUpload( + $file, + $exportImage->type, + $page->id, + ); + + $this->references->addImage($image, $exportImage->id); + + return $image; + } + protected function exportTagsToInputArray(array $exportTags): array { $tags = []; @@ -235,7 +312,7 @@ protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter| } if (count($attachments) > 0) { - if (userCan('attachment-create-all')) { + if (!userCan('attachment-create-all')) { $errors[] = 'You are lacking the required permissions to create attachments.'; } } @@ -257,6 +334,8 @@ protected function getZipPath(Import $import): string stream_copy_to_stream($stream, $tempFile); fclose($tempFile); + $this->tempFilesToCleanup[] = $tempFilePath; + return $tempFilePath; } } From b7476a9e7fc27c27342a0a155ab256a93f19981e Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Thu, 14 Nov 2024 15:59:15 +0000 Subject: [PATCH 29/41] ZIP Import: Finished base import process & error handling Added file creation reverting and DB rollback on error. Added error display on failed import. Extracted likely shown import form/error text to translation files. --- app/Exports/Controllers/ImportController.php | 25 +++---- app/Exports/ImportRepo.php | 26 ++++++- .../ZipExports/ZipImportReferences.php | 17 +++++ app/Exports/ZipExports/ZipImportRunner.php | 69 +++++++++++-------- app/Uploads/AttachmentService.php | 2 +- app/Uploads/ImageService.php | 12 +++- lang/en/entities.php | 3 + lang/en/errors.php | 6 ++ resources/views/exports/import-show.blade.php | 49 ++++++++----- .../views/exports/parts/import.blade.php | 4 +- 10 files changed, 145 insertions(+), 68 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 4d2c83090a3..d8dceed2f8e 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -4,7 +4,7 @@ namespace BookStack\Exports\Controllers; -use BookStack\Activity\ActivityType; +use BookStack\Exceptions\ZipImportException; use BookStack\Exceptions\ZipValidationException; use BookStack\Exports\ImportRepo; use BookStack\Http\Controller; @@ -48,12 +48,9 @@ public function upload(Request $request) try { $import = $this->imports->storeFromUpload($file); } catch (ZipValidationException $exception) { - session()->flash('validation_errors', $exception->errors); - return redirect('/import'); + return redirect('/import')->with('validation_errors', $exception->errors); } - $this->logActivity(ActivityType::IMPORT_CREATE, $import); - return redirect($import->getUrl()); } @@ -80,20 +77,20 @@ public function run(int $id, Request $request) $parent = null; if ($import->type === 'page' || $import->type === 'chapter') { + session()->setPreviousUrl($import->getUrl()); $data = $this->validate($request, [ - 'parent' => ['required', 'string'] + 'parent' => ['required', 'string'], ]); $parent = $data['parent']; } - $entity = $this->imports->runImport($import, $parent); - if ($entity) { - $this->logActivity(ActivityType::IMPORT_RUN, $import); - return redirect($entity->getUrl()); + try { + $entity = $this->imports->runImport($import, $parent); + } catch (ZipImportException $exception) { + return redirect($import->getUrl())->with('import_errors', $exception->errors); } - // TODO - Redirect to result - // TODO - Or redirect back with errors - return 'failed'; + + return redirect($entity->getUrl()); } /** @@ -104,8 +101,6 @@ public function delete(int $id) $import = $this->imports->findVisible($id); $this->imports->deleteImport($import); - $this->logActivity(ActivityType::IMPORT_DELETE, $import); - return redirect('/import'); } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index d169d4845ab..f72386c47bc 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,6 +2,7 @@ namespace BookStack\Exports; +use BookStack\Activity\ActivityType; use BookStack\Entities\Models\Entity; use BookStack\Entities\Queries\EntityQueries; use BookStack\Exceptions\FileUploadException; @@ -14,8 +15,10 @@ use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Exports\ZipExports\ZipImportRunner; +use BookStack\Facades\Activity; use BookStack\Uploads\FileStorage; use Illuminate\Database\Eloquent\Collection; +use Illuminate\Support\Facades\DB; use Symfony\Component\HttpFoundation\File\UploadedFile; class ImportRepo @@ -93,25 +96,42 @@ public function storeFromUpload(UploadedFile $file): Import $import->path = $path; $import->save(); + Activity::add(ActivityType::IMPORT_CREATE, $import); + return $import; } /** - * @throws ZipValidationException|ZipImportException + * @throws ZipImportException */ - public function runImport(Import $import, ?string $parent = null): ?Entity + public function runImport(Import $import, ?string $parent = null): Entity { $parentModel = null; if ($import->type === 'page' || $import->type === 'chapter') { $parentModel = $parent ? $this->entityQueries->findVisibleByStringIdentifier($parent) : null; } - return $this->importer->run($import, $parentModel); + DB::beginTransaction(); + try { + $model = $this->importer->run($import, $parentModel); + } catch (ZipImportException $e) { + DB::rollBack(); + $this->importer->revertStoredFiles(); + throw $e; + } + + DB::commit(); + $this->deleteImport($import); + Activity::add(ActivityType::IMPORT_RUN, $import); + + return $model; } public function deleteImport(Import $import): void { $this->storage->delete($import->path); $import->delete(); + + Activity::add(ActivityType::IMPORT_DELETE, $import); } } diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php index 3bce16bbb13..b23d5e72b15 100644 --- a/app/Exports/ZipExports/ZipImportReferences.php +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -139,4 +139,21 @@ public function replaceReferences(): void ]); } } + + + /** + * @return Image[] + */ + public function images(): array + { + return $this->images; + } + + /** + * @return Attachment[] + */ + public function attachments(): array + { + return $this->attachments; + } } diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 9f19f03e2e5..c5b9da31909 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -47,14 +47,17 @@ public function __construct( * Returns the top-level entity item which was imported. * @throws ZipImportException */ - public function run(Import $import, ?Entity $parent = null): ?Entity + public function run(Import $import, ?Entity $parent = null): Entity { $zipPath = $this->getZipPath($import); $reader = new ZipExportReader($zipPath); $errors = (new ZipExportValidator($reader))->validate(); if ($errors) { - throw new ZipImportException(["ZIP failed to validate"]); + throw new ZipImportException([ + trans('errors.import_validation_failed'), + ...$errors, + ]); } try { @@ -65,15 +68,14 @@ public function run(Import $import, ?Entity $parent = null): ?Entity // Validate parent type if ($exportModel instanceof ZipExportBook && ($parent !== null)) { - throw new ZipImportException(["Must not have a parent set for a Book import"]); - } else if ($exportModel instanceof ZipExportChapter && (!$parent instanceof Book)) { - throw new ZipImportException(["Parent book required for chapter import"]); + throw new ZipImportException(["Must not have a parent set for a Book import."]); + } else if ($exportModel instanceof ZipExportChapter && !($parent instanceof Book)) { + throw new ZipImportException(["Parent book required for chapter import."]); } else if ($exportModel instanceof ZipExportPage && !($parent instanceof Book || $parent instanceof Chapter)) { - throw new ZipImportException(["Parent book or chapter required for page import"]); + throw new ZipImportException(["Parent book or chapter required for page import."]); } - $this->ensurePermissionsPermitImport($exportModel); - $entity = null; + $this->ensurePermissionsPermitImport($exportModel, $parent); if ($exportModel instanceof ZipExportBook) { $entity = $this->importBook($exportModel, $reader); @@ -81,32 +83,46 @@ public function run(Import $import, ?Entity $parent = null): ?Entity $entity = $this->importChapter($exportModel, $parent, $reader); } else if ($exportModel instanceof ZipExportPage) { $entity = $this->importPage($exportModel, $parent, $reader); + } else { + throw new ZipImportException(['No importable data found in import data.']); } - // TODO - In transaction? - // TODO - Revert uploaded files if goes wrong - // TODO - Attachments - // TODO - Images - // (Both listed/stored in references) - $this->references->replaceReferences(); $reader->close(); $this->cleanup(); - dd('stop'); + return $entity; + } - // TODO - Delete import/zip after import? - // Do this in parent repo? + /** + * Revert any files which have been stored during this import process. + * Considers files only, and avoids the database under the + * assumption that the database may already have been + * reverted as part of a transaction rollback. + */ + public function revertStoredFiles(): void + { + foreach ($this->references->images() as $image) { + $this->imageService->destroyFileAtPath($image->type, $image->path); + } - return $entity; + foreach ($this->references->attachments() as $attachment) { + if (!$attachment->external) { + $this->attachmentService->deleteFileInStorage($attachment); + } + } + + $this->cleanup(); } - protected function cleanup() + protected function cleanup(): void { foreach ($this->tempFilesToCleanup as $file) { unlink($file); } + + $this->tempFilesToCleanup = []; } protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book @@ -256,9 +272,6 @@ protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter| { $errors = []; - // TODO - Extract messages to language files - // TODO - Ensure these are shown to users on failure - $chapters = []; $pages = []; $images = []; @@ -266,7 +279,7 @@ protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter| if ($exportModel instanceof ZipExportBook) { if (!userCan('book-create-all')) { - $errors[] = 'You are lacking the required permission to create books.'; + $errors[] = trans('errors.import_perms_books'); } array_push($pages, ...$exportModel->pages); array_push($chapters, ...$exportModel->chapters); @@ -283,7 +296,7 @@ protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter| if (count($chapters) > 0) { $permission = 'chapter-create' . ($parent ? '' : '-all'); if (!userCan($permission, $parent)) { - $errors[] = 'You are lacking the required permission to create chapters.'; + $errors[] = trans('errors.import_perms_chapters'); } } @@ -295,25 +308,25 @@ protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter| if (count($pages) > 0) { if ($parent) { if (!userCan('page-create', $parent)) { - $errors[] = 'You are lacking the required permission to create pages.'; + $errors[] = trans('errors.import_perms_pages'); } } else { $hasPermission = userCan('page-create-all') || userCan('page-create-own'); if (!$hasPermission) { - $errors[] = 'You are lacking the required permission to create pages.'; + $errors[] = trans('errors.import_perms_pages'); } } } if (count($images) > 0) { if (!userCan('image-create-all')) { - $errors[] = 'You are lacking the required permissions to create images.'; + $errors[] = trans('errors.import_perms_images'); } } if (count($attachments) > 0) { if (!userCan('attachment-create-all')) { - $errors[] = 'You are lacking the required permissions to create attachments.'; + $errors[] = trans('errors.import_perms_attachments'); } } diff --git a/app/Uploads/AttachmentService.php b/app/Uploads/AttachmentService.php index fa53c4ae499..033f2334104 100644 --- a/app/Uploads/AttachmentService.php +++ b/app/Uploads/AttachmentService.php @@ -151,7 +151,7 @@ public function deleteFile(Attachment $attachment) * Delete a file from the filesystem it sits on. * Cleans any empty leftover folders. */ - protected function deleteFileInStorage(Attachment $attachment): void + public function deleteFileInStorage(Attachment $attachment): void { $this->storage->delete($attachment->path); } diff --git a/app/Uploads/ImageService.php b/app/Uploads/ImageService.php index e501cc7b12d..5c455cf8633 100644 --- a/app/Uploads/ImageService.php +++ b/app/Uploads/ImageService.php @@ -153,11 +153,19 @@ public function getImageStream(Image $image): mixed */ public function destroy(Image $image): void { - $disk = $this->storage->getDisk($image->type); - $disk->destroyAllMatchingNameFromPath($image->path); + $this->destroyFileAtPath($image->type, $image->path); $image->delete(); } + /** + * Destroy the underlying image file at the given path. + */ + public function destroyFileAtPath(string $type, string $path): void + { + $disk = $this->storage->getDisk($type); + $disk->destroyAllMatchingNameFromPath($path); + } + /** * Delete gallery and drawings that are not within HTML content of pages or page revisions. * Checks based off of only the image name. diff --git a/lang/en/entities.php b/lang/en/entities.php index ae1c1e8d4cc..26a563a7eb5 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -52,6 +52,7 @@ 'import_pending_none' => 'No imports have been started.', 'import_continue' => 'Continue Import', 'import_continue_desc' => 'Review the content due to be imported from the uploaded ZIP file. When ready, run the import to add its contents to this system. The uploaded ZIP import file will be automatically removed on successful import.', + 'import_details' => 'Import Details', 'import_run' => 'Run Import', 'import_size' => ':size Import ZIP Size', 'import_uploaded_at' => 'Uploaded :relativeTime', @@ -60,6 +61,8 @@ 'import_location_desc' => 'Select a target location for your imported content. You\'ll need the relevant permissions to create within the location you choose.', 'import_delete_confirm' => 'Are you sure you want to delete this import?', 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', + 'import_errors' => 'Import Errors', + 'import_errors_desc' => 'The follow errors occurred during the import attempt:', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/lang/en/errors.php b/lang/en/errors.php index 3f2f303311e..ced80a32c1f 100644 --- a/lang/en/errors.php +++ b/lang/en/errors.php @@ -109,6 +109,12 @@ 'import_zip_cant_read' => 'Could not read ZIP file.', 'import_zip_cant_decode_data' => 'Could not find and decode ZIP data.json content.', 'import_zip_no_data' => 'ZIP file data has no expected book, chapter or page content.', + 'import_validation_failed' => 'Import ZIP failed to validate with errors:', + 'import_perms_books' => 'You are lacking the required permissions to create books.', + 'import_perms_chapters' => 'You are lacking the required permissions to create chapters.', + 'import_perms_pages' => 'You are lacking the required permissions to create pages.', + 'import_perms_images' => 'You are lacking the required permissions to create images.', + 'import_perms_attachments' => 'You are lacking the required permission to create attachments.', // API errors 'api_no_authorization_found' => 'No authorization token found on the request', diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index 40867377fb0..e4f199aa20c 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -7,8 +7,19 @@

    {{ trans('entities.import_continue') }}

    {{ trans('entities.import_continue_desc') }}

    + @if(session()->has('import_errors')) +
    + +

    {{ trans('entities.import_errors_desc') }}

    + @foreach(session()->get('import_errors') ?? [] as $error) +

    {{ $error }}

    + @endforeach +
    +
    + @endif +
    - +
    @include('exports.parts.import-item', ['type' => $import->type, 'model' => $data]) @@ -34,32 +45,36 @@ @if($import->type === 'page' || $import->type === 'chapter')
    -

    {{ trans('entities.import_location_desc') }}

    +

    {{ trans('entities.import_location_desc') }}

    + @if($errors->has('parent')) +
    + @include('form.errors', ['name' => 'parent']) +
    + @endif @include('entities.selector', [ 'name' => 'parent', 'entityTypes' => $import->type === 'page' ? 'chapter,book' : 'book', 'entityPermission' => "{$import->type}-create", 'selectorSize' => 'compact small', ]) - @include('form.errors', ['name' => 'parent']) @endif - -
    - {{ trans('common.cancel') }} -
    - -
    -

    {{ trans('entities.import_delete_confirm') }}

    -

    {{ trans('entities.import_delete_desc') }}

    - +
    + {{ trans('common.cancel') }} +
    + +
    +

    {{ trans('entities.import_delete_confirm') }}

    +

    {{ trans('entities.import_delete_desc') }}

    + +
    +
    - -
    +
    diff --git a/resources/views/exports/parts/import.blade.php b/resources/views/exports/parts/import.blade.php index fd53095a422..2f7659c469e 100644 --- a/resources/views/exports/parts/import.blade.php +++ b/resources/views/exports/parts/import.blade.php @@ -4,7 +4,7 @@ class="text-{{ $import->type }}">@icon($import->type) {{ $import->name }}
    -
    {{ $import->getSizeString() }}
    -
    @icon('time'){{ $import->created_at->diffForHumans() }}
    +
    {{ $import->getSizeString() }}
    +
    @icon('time'){{ $import->created_at->diffForHumans() }}
    \ No newline at end of file From 7681e32dca6cb7d06c2d196bf46239a41a86852c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 16 Nov 2024 13:57:41 +0000 Subject: [PATCH 30/41] ZIP Imports: Added high level import run tests --- app/Exports/Controllers/ImportController.php | 4 +- database/factories/Exports/ImportFactory.php | 2 +- tests/Exports/ZipImportRunnerTest.php | 21 +++ tests/Exports/ZipImportTest.php | 133 +++++++++++++++++-- tests/Exports/ZipTestHelper.php | 47 +++++++ 5 files changed, 192 insertions(+), 15 deletions(-) create mode 100644 tests/Exports/ZipImportRunnerTest.php create mode 100644 tests/Exports/ZipTestHelper.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index d8dceed2f8e..a20c341fb02 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -70,9 +70,11 @@ public function show(int $id) ]); } + /** + * Run the import process against an uploaded import ZIP. + */ public function run(int $id, Request $request) { - // TODO - Test access/visibility $import = $this->imports->findVisible($id); $parent = null; diff --git a/database/factories/Exports/ImportFactory.php b/database/factories/Exports/ImportFactory.php index 74a2bcd65f3..5d0b4f89299 100644 --- a/database/factories/Exports/ImportFactory.php +++ b/database/factories/Exports/ImportFactory.php @@ -21,7 +21,7 @@ class ImportFactory extends Factory public function definition(): array { return [ - 'path' => 'uploads/imports/' . Str::random(10) . '.zip', + 'path' => 'uploads/files/imports/' . Str::random(10) . '.zip', 'name' => $this->faker->words(3, true), 'type' => 'book', 'metadata' => '{"name": "My book"}', diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php new file mode 100644 index 00000000000..7bdd8ecbb65 --- /dev/null +++ b/tests/Exports/ZipImportRunnerTest.php @@ -0,0 +1,21 @@ +runner = app()->make(ZipImportRunner::class); + } + + // TODO - Test full book import + // TODO - Test full chapter import + // TODO - Test full page import +} diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index 2b40100aabe..3644e9bdcb7 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -3,6 +3,7 @@ namespace Tests\Exports; use BookStack\Activity\ActivityType; +use BookStack\Entities\Models\Book; use BookStack\Exports\Import; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; @@ -91,7 +92,7 @@ public function test_error_shown_if_missing_data() public function test_error_shown_if_no_importable_key() { $this->asAdmin(); - $resp = $this->runImportFromFile($this->zipUploadFromData([ + $resp = $this->runImportFromFile(ZipTestHelper::zipUploadFromData([ 'instance' => [] ])); @@ -103,7 +104,7 @@ public function test_error_shown_if_no_importable_key() public function test_zip_data_validation_messages_shown() { $this->asAdmin(); - $resp = $this->runImportFromFile($this->zipUploadFromData([ + $resp = $this->runImportFromFile(ZipTestHelper::zipUploadFromData([ 'book' => [ 'id' => 4, 'pages' => [ @@ -154,7 +155,7 @@ public function test_import_upload_success() ], ]; - $resp = $this->runImportFromFile($this->zipUploadFromData($data)); + $resp = $this->runImportFromFile(ZipTestHelper::zipUploadFromData($data)); $this->assertDatabaseHas('imports', [ 'name' => 'My great book name', @@ -217,7 +218,7 @@ public function test_import_show_page_access_limited() public function test_import_delete() { $this->asAdmin(); - $this->runImportFromFile($this->zipUploadFromData([ + $this->runImportFromFile(ZipTestHelper::zipUploadFromData([ 'book' => [ 'name' => 'My great book name' ], @@ -262,20 +263,126 @@ public function test_import_delete_access_limited() $this->delete("/import/{$adminImport->id}")->assertRedirect('/import'); } - protected function runImportFromFile(UploadedFile $file): TestResponse + public function test_run_simple_success_scenario() { - return $this->call('POST', '/import', [], [], ['file' => $file]); + $import = ZipTestHelper::importFromData([], [ + 'book' => [ + 'name' => 'My imported book', + 'pages' => [ + [ + 'name' => 'My imported book page', + 'html' => '

    Hello there from child page!

    ' + ] + ], + ] + ]); + + $resp = $this->asAdmin()->post("/import/{$import->id}"); + $book = Book::query()->where('name', '=', 'My imported book')->latest()->first(); + $resp->assertRedirect($book->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSee('My imported book page'); + $resp->assertSee('Hello there from child page!'); + + $this->assertDatabaseMissing('imports', ['id' => $import->id]); + $this->assertFileDoesNotExist(storage_path($import->path)); + $this->assertActivityExists(ActivityType::IMPORT_RUN, null, $import->logDescriptor()); } - protected function zipUploadFromData(array $data): UploadedFile + public function test_import_run_access_limited() { - $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + $user = $this->users->editor(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->actingAs($user); - $zip = new ZipArchive(); - $zip->open($zipFile, ZipArchive::CREATE); - $zip->addFromString('data.json', json_encode($data)); - $zip->close(); + $this->post("/import/{$userImport->id}")->assertRedirect('/'); + $this->post("/import/{$adminImport->id}")->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $this->post("/import/{$userImport->id}")->assertRedirect($userImport->getUrl()); // Getting validation response instead of access issue response + $this->post("/import/{$adminImport->id}")->assertStatus(404); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $this->post("/import/{$adminImport->id}")->assertRedirect($adminImport->getUrl()); // Getting validation response instead of access issue response + } + + public function test_run_revalidates_content() + { + $import = ZipTestHelper::importFromData([], [ + 'book' => [ + 'id' => 'abc', + ] + ]); + + $resp = $this->asAdmin()->post("/import/{$import->id}"); + $resp->assertRedirect($import->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('The name field is required.'); + $resp->assertSeeText('The id must be an integer.'); + } + + public function test_run_checks_permissions_on_import() + { + $viewer = $this->users->viewer(); + $this->permissions->grantUserRolePermissions($viewer, ['content-import']); + $import = ZipTestHelper::importFromData(['created_by' => $viewer->id], [ + 'book' => ['name' => 'My import book'], + ]); + + $resp = $this->asViewer()->post("/import/{$import->id}"); + $resp->assertRedirect($import->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('You are lacking the required permissions to create books.'); + } + + public function test_run_requires_parent_for_chapter_and_page_imports() + { + $book = $this->entities->book(); + $pageImport = ZipTestHelper::importFromData([], [ + 'page' => ['name' => 'My page', 'html' => '

    page test!

    '], + ]); + $chapterImport = ZipTestHelper::importFromData([], [ + 'chapter' => ['name' => 'My chapter'], + ]); + + $resp = $this->asAdmin()->post("/import/{$pageImport->id}"); + $resp->assertRedirect($pageImport->getUrl()); + $this->followRedirects($resp)->assertSee('The parent field is required.'); + + $resp = $this->asAdmin()->post("/import/{$pageImport->id}", ['parent' => "book:{$book->id}"]); + $resp->assertRedirectContains($book->getUrl()); + + $resp = $this->asAdmin()->post("/import/{$chapterImport->id}"); + $resp->assertRedirect($chapterImport->getUrl()); + $this->followRedirects($resp)->assertSee('The parent field is required.'); + + $resp = $this->asAdmin()->post("/import/{$chapterImport->id}", ['parent' => "book:{$book->id}"]); + $resp->assertRedirectContains($book->getUrl()); + } + + public function test_run_validates_correct_parent_type() + { + $chapter = $this->entities->chapter(); + $import = ZipTestHelper::importFromData([], [ + 'chapter' => ['name' => 'My chapter'], + ]); + + $resp = $this->asAdmin()->post("/import/{$import->id}", ['parent' => "chapter:{$chapter->id}"]); + $resp->assertRedirect($import->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSee('Parent book required for chapter import.'); + } - return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + protected function runImportFromFile(UploadedFile $file): TestResponse + { + return $this->call('POST', '/import', [], [], ['file' => $file]); } } diff --git a/tests/Exports/ZipTestHelper.php b/tests/Exports/ZipTestHelper.php new file mode 100644 index 00000000000..3a9b3435454 --- /dev/null +++ b/tests/Exports/ZipTestHelper.php @@ -0,0 +1,47 @@ +create($importData); + $zip = static::zipUploadFromData($zipData); + rename($zip->getRealPath(), storage_path($import->path)); + + return $import; + } + + public static function deleteZipForImport(Import $import): void + { + $path = storage_path($import->path); + if (file_exists($path)) { + unlink($path); + } + } + + public static function zipUploadFromData(array $data): UploadedFile + { + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::CREATE); + $zip->addFromString('data.json', json_encode($data)); + $zip->close(); + + return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + } +} From 8645aeaa4a914c5ee7e0d07a9202b8812aefcafe Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 16 Nov 2024 16:12:45 +0000 Subject: [PATCH 31/41] ZIP Imports: Started testing core import logic Fixed image size handling, and lack of attachment reference replacements during testing. --- .../ZipExports/ZipImportReferences.php | 4 +- app/Exports/ZipExports/ZipImportRunner.php | 4 + app/Uploads/ImageService.php | 5 +- tests/Exports/ZipImportRunnerTest.php | 152 ++++++++++++++++++ tests/Exports/ZipTestHelper.php | 11 +- 5 files changed, 170 insertions(+), 6 deletions(-) diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php index b23d5e72b15..da0581df6f5 100644 --- a/app/Exports/ZipExports/ZipImportReferences.php +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -97,10 +97,12 @@ protected function handleReference(string $type, int $id): ?string } else if ($model instanceof Image) { if ($model->type === 'gallery') { $this->imageResizer->loadGalleryThumbnailsForImage($model, false); - return $model->thumbs['gallery'] ?? $model->url; + return $model->thumbs['display'] ?? $model->url; } return $model->url; + } else if ($model instanceof Attachment) { + return $model->getUrl(false); } return null; diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index c5b9da31909..27d859e5915 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -233,6 +233,10 @@ protected function importImage(ZipExportImage $exportImage, Page $page, ZipExpor $file, $exportImage->type, $page->id, + null, + null, + true, + $exportImage->name, ); $this->references->addImage($image, $exportImage->id); diff --git a/app/Uploads/ImageService.php b/app/Uploads/ImageService.php index 5c455cf8633..038e6aa417c 100644 --- a/app/Uploads/ImageService.php +++ b/app/Uploads/ImageService.php @@ -33,9 +33,10 @@ public function saveNewFromUpload( int $uploadedTo = 0, int $resizeWidth = null, int $resizeHeight = null, - bool $keepRatio = true + bool $keepRatio = true, + string $imageName = '', ): Image { - $imageName = $uploadedFile->getClientOriginalName(); + $imageName = $imageName ?: $uploadedFile->getClientOriginalName(); $imageData = file_get_contents($uploadedFile->getRealPath()); if ($resizeWidth !== null || $resizeHeight !== null) { diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php index 7bdd8ecbb65..f07b3f41b42 100644 --- a/tests/Exports/ZipImportRunnerTest.php +++ b/tests/Exports/ZipImportRunnerTest.php @@ -2,7 +2,10 @@ namespace Tests\Exports; +use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipImportRunner; +use BookStack\Uploads\Image; use Tests\TestCase; class ZipImportRunnerTest extends TestCase @@ -15,6 +18,155 @@ protected function setUp(): void $this->runner = app()->make(ZipImportRunner::class); } + public function test_book_import() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $import = ZipTestHelper::importFromData([], [ + 'book' => [ + 'id' => 5, + 'name' => 'Import test', + 'cover' => 'book_cover_image', + 'description_html' => '

    Link to chapter page

    ', + 'tags' => [ + ['name' => 'Animal', 'value' => 'Cat'], + ['name' => 'Category', 'value' => 'Test'], + ], + 'chapters' => [ + [ + 'id' => 6, + 'name' => 'Chapter A', + 'description_html' => '

    Link to book

    ', + 'priority' => 1, + 'tags' => [ + ['name' => 'Reviewed'], + ['name' => 'Category', 'value' => 'Test Chapter'], + ], + 'pages' => [ + [ + 'id' => 3, + 'name' => 'Page A', + 'priority' => 6, + 'html' => ' +

    Link to self

    +

    Link to cat image

    +

    Link to text attachment

    ', + 'tags' => [ + ['name' => 'Unreviewed'], + ], + 'attachments' => [ + [ + 'id' => 4, + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ], + [ + 'name' => 'Cats', + 'link' => 'https://example.com/cats', + ] + ], + 'images' => [ + [ + 'id' => 1, + 'name' => 'Cat', + 'type' => 'gallery', + 'file' => 'cat_image' + ], + [ + 'id' => 2, + 'name' => 'Dog Drawing', + 'type' => 'drawio', + 'file' => 'dog_image' + ] + ], + ], + ], + ], + [ + 'name' => 'Chapter child B', + 'priority' => 5, + ] + ], + 'pages' => [ + [ + 'name' => 'Page C', + 'markdown' => '[Link to text]([[bsexport:attachment:4]]?scale=big)', + 'priority' => 3, + ] + ], + ], + ], [ + 'book_cover_image' => $testImagePath, + 'file_attachment' => $testFilePath, + 'cat_image' => $testImagePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Book $book */ + $book = $this->runner->run($import); + + // Book checks + $this->assertEquals('Import test', $book->name); + $this->assertFileExists(public_path($book->cover->path)); + $this->assertCount(2, $book->tags); + $this->assertEquals('Cat', $book->tags()->first()->value); + $this->assertCount(2, $book->chapters); + $this->assertEquals(1, $book->directPages()->count()); + + // Chapter checks + $chapterA = $book->chapters()->where('name', 'Chapter A')->first(); + $this->assertCount(2, $chapterA->tags); + $firstChapterTag = $chapterA->tags()->first(); + $this->assertEquals('Reviewed', $firstChapterTag->name); + $this->assertEquals('', $firstChapterTag->value); + $this->assertCount(1, $chapterA->pages); + + // Page checks + /** @var Page $pageA */ + $pageA = $chapterA->pages->first(); + $this->assertEquals('Page A', $pageA->name); + $this->assertCount(1, $pageA->tags); + $firstPageTag = $pageA->tags()->first(); + $this->assertEquals('Unreviewed', $firstPageTag->name); + $this->assertCount(2, $pageA->attachments); + $firstAttachment = $pageA->attachments->first(); + $this->assertEquals('Text attachment', $firstAttachment->name); + $this->assertFileEquals($testFilePath, storage_path($firstAttachment->path)); + $this->assertFalse($firstAttachment->external); + $secondAttachment = $pageA->attachments->last(); + $this->assertEquals('Cats', $secondAttachment->name); + $this->assertEquals('https://example.com/cats', $secondAttachment->path); + $this->assertTrue($secondAttachment->external); + $pageAImages = Image::where('uploaded_to', '=', $pageA->id)->whereIn('type', ['gallery', 'drawio'])->get(); + $this->assertCount(2, $pageAImages); + $this->assertEquals('Cat', $pageAImages[0]->name); + $this->assertEquals('gallery', $pageAImages[0]->type); + $this->assertFileEquals($testImagePath, public_path($pageAImages[0]->path)); + $this->assertEquals('Dog Drawing', $pageAImages[1]->name); + $this->assertEquals('drawio', $pageAImages[1]->type); + + // Book order check + $children = $book->getDirectVisibleChildren()->values()->all(); + $this->assertEquals($children[0]->name, 'Chapter A'); + $this->assertEquals($children[1]->name, 'Page C'); + $this->assertEquals($children[2]->name, 'Chapter child B'); + + // Reference checks + $textAttachmentUrl = $firstAttachment->getUrl(); + $this->assertStringContainsString($pageA->getUrl(), $book->description_html); + $this->assertStringContainsString($book->getUrl(), $chapterA->description_html); + $this->assertStringContainsString($pageA->getUrl(), $pageA->html); + $this->assertStringContainsString($pageAImages[0]->getThumb(1680, null, true), $pageA->html); + $this->assertStringContainsString($firstAttachment->getUrl(), $pageA->html); + + // Reference in converted markdown + $pageC = $children[1]; + $this->assertStringContainsString("href=\"{$textAttachmentUrl}?scale=big\"", $pageC->html); + + ZipTestHelper::deleteZipForImport($import); + } + // TODO - Test full book import // TODO - Test full chapter import // TODO - Test full page import diff --git a/tests/Exports/ZipTestHelper.php b/tests/Exports/ZipTestHelper.php index 3a9b3435454..2196f361c17 100644 --- a/tests/Exports/ZipTestHelper.php +++ b/tests/Exports/ZipTestHelper.php @@ -8,7 +8,7 @@ class ZipTestHelper { - public static function importFromData(array $importData, array $zipData): Import + public static function importFromData(array $importData, array $zipData, array $files = []): Import { if (isset($zipData['book'])) { $importData['type'] = 'book'; @@ -19,7 +19,7 @@ public static function importFromData(array $importData, array $zipData): Import } $import = Import::factory()->create($importData); - $zip = static::zipUploadFromData($zipData); + $zip = static::zipUploadFromData($zipData, $files); rename($zip->getRealPath(), storage_path($import->path)); return $import; @@ -33,13 +33,18 @@ public static function deleteZipForImport(Import $import): void } } - public static function zipUploadFromData(array $data): UploadedFile + public static function zipUploadFromData(array $data, array $files = []): UploadedFile { $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); $zip = new ZipArchive(); $zip->open($zipFile, ZipArchive::CREATE); $zip->addFromString('data.json', json_encode($data)); + + foreach ($files as $name => $file) { + $zip->addFile($file, "files/$name"); + } + $zip->close(); return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); From c2c64e207f89567350eab4b40b725e8d042c9654 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 16 Nov 2024 19:52:20 +0000 Subject: [PATCH 32/41] ZIP Imports: Covered import runner with further testing --- tests/Exports/ZipImportRunnerTest.php | 194 +++++++++++++++++++++++++- 1 file changed, 191 insertions(+), 3 deletions(-) diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php index f07b3f41b42..c833fadda96 100644 --- a/tests/Exports/ZipImportRunnerTest.php +++ b/tests/Exports/ZipImportRunnerTest.php @@ -3,6 +3,7 @@ namespace Tests\Exports; use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipImportRunner; use BookStack\Uploads\Image; @@ -167,7 +168,194 @@ public function test_book_import() ZipTestHelper::deleteZipForImport($import); } - // TODO - Test full book import - // TODO - Test full chapter import - // TODO - Test full page import + public function test_chapter_import() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $parent = $this->entities->book(); + + $import = ZipTestHelper::importFromData([], [ + 'chapter' => [ + 'id' => 6, + 'name' => 'Chapter A', + 'description_html' => '

    Link to page

    ', + 'priority' => 1, + 'tags' => [ + ['name' => 'Reviewed', 'value' => '2024'], + ], + 'pages' => [ + [ + 'id' => 3, + 'name' => 'Page A', + 'priority' => 6, + 'html' => '

    Link to chapter

    +

    Link to dog drawing

    +

    Link to text attachment

    ', + 'tags' => [ + ['name' => 'Unreviewed'], + ], + 'attachments' => [ + [ + 'id' => 4, + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ] + ], + 'images' => [ + [ + 'id' => 2, + 'name' => 'Dog Drawing', + 'type' => 'drawio', + 'file' => 'dog_image' + ] + ], + ], + [ + 'name' => 'Page B', + 'markdown' => '[Link to page A]([[bsexport:page:3]])', + 'priority' => 9, + ], + ], + ], + ], [ + 'file_attachment' => $testFilePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Chapter $chapter */ + $chapter = $this->runner->run($import, $parent); + + // Chapter checks + $this->assertEquals('Chapter A', $chapter->name); + $this->assertEquals($parent->id, $chapter->book_id); + $this->assertCount(1, $chapter->tags); + $firstChapterTag = $chapter->tags()->first(); + $this->assertEquals('Reviewed', $firstChapterTag->name); + $this->assertEquals('2024', $firstChapterTag->value); + $this->assertCount(2, $chapter->pages); + + // Page checks + /** @var Page $pageA */ + $pageA = $chapter->pages->first(); + $this->assertEquals('Page A', $pageA->name); + $this->assertCount(1, $pageA->tags); + $this->assertCount(1, $pageA->attachments); + $pageAImages = Image::where('uploaded_to', '=', $pageA->id)->whereIn('type', ['gallery', 'drawio'])->get(); + $this->assertCount(1, $pageAImages); + + // Reference checks + $attachment = $pageA->attachments->first(); + $this->assertStringContainsString($pageA->getUrl(), $chapter->description_html); + $this->assertStringContainsString($chapter->getUrl(), $pageA->html); + $this->assertStringContainsString($pageAImages[0]->url, $pageA->html); + $this->assertStringContainsString($attachment->getUrl(), $pageA->html); + + ZipTestHelper::deleteZipForImport($import); + } + + public function test_page_import() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $parent = $this->entities->chapter(); + + $import = ZipTestHelper::importFromData([], [ + 'page' => [ + 'id' => 3, + 'name' => 'Page A', + 'priority' => 6, + 'html' => '

    Link to self

    +

    Link to dog drawing

    +

    Link to text attachment

    ', + 'tags' => [ + ['name' => 'Unreviewed'], + ], + 'attachments' => [ + [ + 'id' => 4, + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ] + ], + 'images' => [ + [ + 'id' => 2, + 'name' => 'Dog Drawing', + 'type' => 'drawio', + 'file' => 'dog_image' + ] + ], + ], + ], [ + 'file_attachment' => $testFilePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Page $page */ + $page = $this->runner->run($import, $parent); + + // Page checks + $this->assertEquals('Page A', $page->name); + $this->assertCount(1, $page->tags); + $this->assertCount(1, $page->attachments); + $pageImages = Image::where('uploaded_to', '=', $page->id)->whereIn('type', ['gallery', 'drawio'])->get(); + $this->assertCount(1, $pageImages); + $this->assertFileEquals($testImagePath, public_path($pageImages[0]->path)); + + // Reference checks + $this->assertStringContainsString($page->getUrl(), $page->html); + $this->assertStringContainsString($pageImages[0]->url, $page->html); + $this->assertStringContainsString($page->attachments->first()->getUrl(), $page->html); + + ZipTestHelper::deleteZipForImport($import); + } + + public function test_revert_cleans_up_uploaded_files() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $parent = $this->entities->chapter(); + + $import = ZipTestHelper::importFromData([], [ + 'page' => [ + 'name' => 'Page A', + 'html' => '

    Hello

    ', + 'attachments' => [ + [ + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ] + ], + 'images' => [ + [ + 'name' => 'Dog Image', + 'type' => 'gallery', + 'file' => 'dog_image' + ] + ], + ], + ], [ + 'file_attachment' => $testFilePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Page $page */ + $page = $this->runner->run($import, $parent); + + $attachment = $page->attachments->first(); + $image = Image::query()->where('uploaded_to', '=', $page->id)->where('type', '=', 'gallery')->first(); + + $this->assertFileExists(public_path($image->path)); + $this->assertFileExists(storage_path($attachment->path)); + + $this->runner->revertStoredFiles(); + + $this->assertFileDoesNotExist(public_path($image->path)); + $this->assertFileDoesNotExist(storage_path($attachment->path)); + + ZipTestHelper::deleteZipForImport($import); + } } From e2f6e50df4347579e3b6eb8e7c48bfcb79199a64 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 18 Nov 2024 15:53:21 +0000 Subject: [PATCH 33/41] ZIP Exports: Added ID checks and testing to validator --- .../ZipExports/Models/ZipExportAttachment.php | 2 +- .../ZipExports/Models/ZipExportBook.php | 2 +- .../ZipExports/Models/ZipExportChapter.php | 2 +- .../ZipExports/Models/ZipExportImage.php | 2 +- .../ZipExports/Models/ZipExportPage.php | 2 +- .../ZipExports/ZipFileReferenceRule.php | 1 - app/Exports/ZipExports/ZipUniqueIdRule.php | 26 +++++++ .../ZipExports/ZipValidationHelper.php | 24 ++++++ lang/en/validation.php | 1 + tests/Exports/ZipExportValidatorTests.php | 74 +++++++++++++++++++ 10 files changed, 130 insertions(+), 6 deletions(-) create mode 100644 app/Exports/ZipExports/ZipUniqueIdRule.php create mode 100644 tests/Exports/ZipExportValidatorTests.php diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index c6615e1dc49..4f5b2f23699 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -43,7 +43,7 @@ public static function fromModelArray(array $attachmentArray, ZipExportFiles $fi public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('attachment')], 'name' => ['required', 'string', 'min:1'], 'link' => ['required_without:file', 'nullable', 'string'], 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 0dc4e93d43c..47ab8f0a699 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -70,7 +70,7 @@ public static function fromModel(Book $model, ZipExportFiles $files): self public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('book')], 'name' => ['required', 'string', 'min:1'], 'description_html' => ['nullable', 'string'], 'cover' => ['nullable', 'string', $context->fileReferenceRule()], diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index 50440d61a5a..5a5fe350f3a 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -59,7 +59,7 @@ public static function fromModelArray(array $chapterArray, ZipExportFiles $files public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('chapter')], 'name' => ['required', 'string', 'min:1'], 'description_html' => ['nullable', 'string'], 'priority' => ['nullable', 'int'], diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 691eb918fc6..89083b15be1 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -33,7 +33,7 @@ public function metadataOnly(): void public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('image')], 'name' => ['required', 'string', 'min:1'], 'file' => ['required', 'string', $context->fileReferenceRule()], 'type' => ['required', 'string', Rule::in(['gallery', 'drawio'])], diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 3a876e7aaff..16e7e925539 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -68,7 +68,7 @@ public static function fromModelArray(array $pageArray, ZipExportFiles $files): public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('page')], 'name' => ['required', 'string', 'min:1'], 'html' => ['nullable', 'string'], 'markdown' => ['nullable', 'string'], diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php index bcd3c39acf0..7d6c829cf03 100644 --- a/app/Exports/ZipExports/ZipFileReferenceRule.php +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -4,7 +4,6 @@ use Closure; use Illuminate\Contracts\Validation\ValidationRule; -use ZipArchive; class ZipFileReferenceRule implements ValidationRule { diff --git a/app/Exports/ZipExports/ZipUniqueIdRule.php b/app/Exports/ZipExports/ZipUniqueIdRule.php new file mode 100644 index 00000000000..ea2b2539296 --- /dev/null +++ b/app/Exports/ZipExports/ZipUniqueIdRule.php @@ -0,0 +1,26 @@ +context->hasIdBeenUsed($this->modelType, $value)) { + $fail('validation.zip_unique')->translate(['attribute' => $attribute]); + } + } +} diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index 55c86b03b5b..7659c228bcd 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -9,6 +9,13 @@ class ZipValidationHelper { protected Factory $validationFactory; + /** + * Local store of validated IDs (in format ":". Example: "book:2") + * which we can use to check uniqueness. + * @var array + */ + protected array $validatedIds = []; + public function __construct( public ZipExportReader $zipReader, ) { @@ -31,6 +38,23 @@ public function fileReferenceRule(): ZipFileReferenceRule return new ZipFileReferenceRule($this); } + public function uniqueIdRule(string $type): ZipUniqueIdRule + { + return new ZipUniqueIdRule($this, $type); + } + + public function hasIdBeenUsed(string $type, int $id): bool + { + $key = $type . ':' . $id; + if (isset($this->validatedIds[$key])) { + return true; + } + + $this->validatedIds[$key] = true; + + return false; + } + /** * Validate an array of relation data arrays that are expected * to be for the given ZipExportModel. diff --git a/lang/en/validation.php b/lang/en/validation.php index bc01ac47b94..fdfc3d9a9b9 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -107,6 +107,7 @@ 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', 'zip_model_expected' => 'Data object expected but ":type" found.', + 'zip_unique' => 'The :attribute must be unique for the object type within the ZIP.', // Custom validation lines 'custom' => [ diff --git a/tests/Exports/ZipExportValidatorTests.php b/tests/Exports/ZipExportValidatorTests.php new file mode 100644 index 00000000000..4cacea95ec6 --- /dev/null +++ b/tests/Exports/ZipExportValidatorTests.php @@ -0,0 +1,74 @@ +filesToRemove as $file) { + unlink($file); + } + + parent::tearDown(); + } + + protected function getValidatorForData(array $zipData, array $files = []): ZipExportValidator + { + $upload = ZipTestHelper::zipUploadFromData($zipData, $files); + $path = $upload->getRealPath(); + $this->filesToRemove[] = $path; + $reader = new ZipExportReader($path); + return new ZipExportValidator($reader); + } + + public function test_ids_have_to_be_unique() + { + $validator = $this->getValidatorForData([ + 'book' => [ + 'id' => 4, + 'name' => 'My book', + 'pages' => [ + [ + 'id' => 4, + 'name' => 'My page', + 'markdown' => 'hello', + 'attachments' => [ + ['id' => 4, 'name' => 'Attachment A', 'link' => 'https://example.com'], + ['id' => 4, 'name' => 'Attachment B', 'link' => 'https://example.com'] + ], + 'images' => [ + ['id' => 4, 'name' => 'Image A', 'type' => 'gallery', 'file' => 'cat'], + ['id' => 4, 'name' => 'Image b', 'type' => 'gallery', 'file' => 'cat'], + ], + ], + ['id' => 4, 'name' => 'My page', 'markdown' => 'hello'], + ], + 'chapters' => [ + ['id' => 4, 'name' => 'Chapter 1'], + ['id' => 4, 'name' => 'Chapter 2'] + ] + ] + ], ['cat' => $this->files->testFilePath('test-image.png')]); + + $results = $validator->validate(); + $this->assertCount(4, $results); + + $expectedMessage = 'The id must be unique for the object type within the ZIP.'; + $this->assertEquals($expectedMessage, $results['book.pages.0.attachments.1.id']); + $this->assertEquals($expectedMessage, $results['book.pages.0.images.1.id']); + $this->assertEquals($expectedMessage, $results['book.pages.1.id']); + $this->assertEquals($expectedMessage, $results['book.chapters.1.id']); + } +} From 59cfc087e12c8752b4a9f1760db71a13ad6c121c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 18 Nov 2024 17:42:49 +0000 Subject: [PATCH 34/41] ZIP Imports: Added image type validation/handling Images were missing their extension after import since it was (potentially) not part of the import data. This adds validation via mime sniffing (to match normal image upload checks) and also uses the same logic to sniff out a correct extension. Added tests to cover. Also fixed some existing tests around zip functionality. --- .../ZipExports/Models/ZipExportImage.php | 3 +- app/Exports/ZipExports/ZipExportReader.php | 12 +++++++ .../ZipExports/ZipFileReferenceRule.php | 12 +++++++ app/Exports/ZipExports/ZipImportRunner.php | 8 ++++- .../ZipExports/ZipValidationHelper.php | 6 ++-- lang/en/validation.php | 1 + tests/Exports/ZipExportTest.php | 4 --- ...orTests.php => ZipExportValidatorTest.php} | 21 ++++++++++- tests/Exports/ZipImportRunnerTest.php | 35 +++++++++++++++++++ 9 files changed, 92 insertions(+), 10 deletions(-) rename tests/Exports/{ZipExportValidatorTests.php => ZipExportValidatorTest.php} (77%) diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 89083b15be1..e0e7d11986d 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -32,10 +32,11 @@ public function metadataOnly(): void public static function validate(ZipValidationHelper $context, array $data): array { + $acceptedImageTypes = ['image/png', 'image/jpeg', 'image/gif', 'image/webp']; $rules = [ 'id' => ['nullable', 'int', $context->uniqueIdRule('image')], 'name' => ['required', 'string', 'min:1'], - 'file' => ['required', 'string', $context->fileReferenceRule()], + 'file' => ['required', 'string', $context->fileReferenceRule($acceptedImageTypes)], 'type' => ['required', 'string', Rule::in(['gallery', 'drawio'])], ]; diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index ebc2fbbc9e1..6b88ef61c33 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -7,6 +7,7 @@ use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportModel; use BookStack\Exports\ZipExports\Models\ZipExportPage; +use BookStack\Util\WebSafeMimeSniffer; use ZipArchive; class ZipExportReader @@ -81,6 +82,17 @@ public function streamFile(string $fileName) return $this->zip->getStream("files/{$fileName}"); } + /** + * Sniff the mime type from the file of given name. + */ + public function sniffFileMime(string $fileName): string + { + $stream = $this->streamFile($fileName); + $sniffContent = fread($stream, 2000); + + return (new WebSafeMimeSniffer())->sniff($sniffContent); + } + /** * @throws ZipExportException */ diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php index 7d6c829cf03..90e78c060b0 100644 --- a/app/Exports/ZipExports/ZipFileReferenceRule.php +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -9,6 +9,7 @@ class ZipFileReferenceRule implements ValidationRule { public function __construct( protected ZipValidationHelper $context, + protected array $acceptedMimes, ) { } @@ -21,5 +22,16 @@ public function validate(string $attribute, mixed $value, Closure $fail): void if (!$this->context->zipReader->fileExists($value)) { $fail('validation.zip_file')->translate(); } + + if (!empty($this->acceptedMimes)) { + $fileMime = $this->context->zipReader->sniffFileMime($value); + if (!in_array($fileMime, $this->acceptedMimes)) { + $fail('validation.zip_file_mime')->translate([ + 'attribute' => $attribute, + 'validTypes' => implode(',', $this->acceptedMimes), + 'foundType' => $fileMime + ]); + } + } } } diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 27d859e5915..d25a1621f6e 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -228,6 +228,9 @@ protected function importAttachment(ZipExportAttachment $exportAttachment, Page protected function importImage(ZipExportImage $exportImage, Page $page, ZipExportReader $reader): Image { + $mime = $reader->sniffFileMime($exportImage->file); + $extension = explode('/', $mime)[1]; + $file = $this->zipFileToUploadedFile($exportImage->file, $reader); $image = $this->imageService->saveNewFromUpload( $file, @@ -236,9 +239,12 @@ protected function importImage(ZipExportImage $exportImage, Page $page, ZipExpor null, null, true, - $exportImage->name, + $exportImage->name . '.' . $extension, ); + $image->name = $exportImage->name; + $image->save(); + $this->references->addImage($image, $exportImage->id); return $image; diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index 7659c228bcd..fd9cd784472 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -33,9 +33,9 @@ public function validateData(array $data, array $rules): array return $messages; } - public function fileReferenceRule(): ZipFileReferenceRule + public function fileReferenceRule(array $acceptedMimes = []): ZipFileReferenceRule { - return new ZipFileReferenceRule($this); + return new ZipFileReferenceRule($this, $acceptedMimes); } public function uniqueIdRule(string $type): ZipUniqueIdRule @@ -43,7 +43,7 @@ public function uniqueIdRule(string $type): ZipUniqueIdRule return new ZipUniqueIdRule($this, $type); } - public function hasIdBeenUsed(string $type, int $id): bool + public function hasIdBeenUsed(string $type, mixed $id): bool { $key = $type . ':' . $id; if (isset($this->validatedIds[$key])) { diff --git a/lang/en/validation.php b/lang/en/validation.php index fdfc3d9a9b9..d9b982d1e23 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -106,6 +106,7 @@ 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + 'zip_file_mime' => 'The :attribute needs to reference a file of type :validTypes, found :foundType.', 'zip_model_expected' => 'Data object expected but ":type" found.', 'zip_unique' => 'The :attribute must be unique for the object type within the ZIP.', diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index ac07b33aef5..12531239ff3 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -107,12 +107,10 @@ public function test_page_export_with_tags() [ 'name' => 'Exporty', 'value' => 'Content', - 'order' => 1, ], [ 'name' => 'Another', 'value' => '', - 'order' => 2, ] ], $pageData['tags']); } @@ -162,7 +160,6 @@ public function test_page_export_file_attachments() $attachmentData = $pageData['attachments'][0]; $this->assertEquals('PageAttachmentExport.txt', $attachmentData['name']); $this->assertEquals($attachment->id, $attachmentData['id']); - $this->assertEquals(1, $attachmentData['order']); $this->assertArrayNotHasKey('link', $attachmentData); $this->assertNotEmpty($attachmentData['file']); @@ -193,7 +190,6 @@ public function test_page_export_link_attachments() $attachmentData = $pageData['attachments'][0]; $this->assertEquals('My link attachment for export', $attachmentData['name']); $this->assertEquals($attachment->id, $attachmentData['id']); - $this->assertEquals(1, $attachmentData['order']); $this->assertEquals('https://example.com/cats', $attachmentData['link']); $this->assertArrayNotHasKey('file', $attachmentData); } diff --git a/tests/Exports/ZipExportValidatorTests.php b/tests/Exports/ZipExportValidatorTest.php similarity index 77% rename from tests/Exports/ZipExportValidatorTests.php rename to tests/Exports/ZipExportValidatorTest.php index 4cacea95ec6..c453ef294d4 100644 --- a/tests/Exports/ZipExportValidatorTests.php +++ b/tests/Exports/ZipExportValidatorTest.php @@ -11,7 +11,7 @@ use BookStack\Uploads\Image; use Tests\TestCase; -class ZipExportValidatorTests extends TestCase +class ZipExportValidatorTest extends TestCase { protected array $filesToRemove = []; @@ -71,4 +71,23 @@ public function test_ids_have_to_be_unique() $this->assertEquals($expectedMessage, $results['book.pages.1.id']); $this->assertEquals($expectedMessage, $results['book.chapters.1.id']); } + + public function test_image_files_need_to_be_a_valid_detected_image_file() + { + $validator = $this->getValidatorForData([ + 'page' => [ + 'id' => 4, + 'name' => 'My page', + 'markdown' => 'hello', + 'images' => [ + ['id' => 4, 'name' => 'Image A', 'type' => 'gallery', 'file' => 'cat'], + ], + ] + ], ['cat' => $this->files->testFilePath('test-file.txt')]); + + $results = $validator->validate(); + $this->assertCount(1, $results); + + $this->assertEquals('The file needs to reference a file of type image/png,image/jpeg,image/gif,image/webp, found text/plain.', $results['page.images.0.file']); + } } diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php index c833fadda96..d3af6df76a6 100644 --- a/tests/Exports/ZipImportRunnerTest.php +++ b/tests/Exports/ZipImportRunnerTest.php @@ -358,4 +358,39 @@ public function test_revert_cleans_up_uploaded_files() ZipTestHelper::deleteZipForImport($import); } + + public function test_imported_images_have_their_detected_extension_added() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $parent = $this->entities->chapter(); + + $import = ZipTestHelper::importFromData([], [ + 'page' => [ + 'name' => 'Page A', + 'html' => '

    hello

    ', + 'images' => [ + [ + 'id' => 2, + 'name' => 'Cat', + 'type' => 'gallery', + 'file' => 'cat_image' + ] + ], + ], + ], [ + 'cat_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Page $page */ + $page = $this->runner->run($import, $parent); + + $pageImages = Image::where('uploaded_to', '=', $page->id)->whereIn('type', ['gallery', 'drawio'])->get(); + + $this->assertCount(1, $pageImages); + $this->assertStringEndsWith('.png', $pageImages[0]->url); + $this->assertStringEndsWith('.png', $pageImages[0]->path); + + ZipTestHelper::deleteZipForImport($import); + } } From c0dff6d4a6227549e7f756cdd0d7cd6a003b9886 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Fri, 22 Nov 2024 21:03:04 +0000 Subject: [PATCH 35/41] ZIP Imports: Added book content ordering to import preview --- app/Exports/ZipExports/Models/ZipExportBook.php | 14 ++++++++++++++ app/Exports/ZipExports/Models/ZipExportChapter.php | 7 ++++++- app/Exports/ZipExports/Models/ZipExportPage.php | 2 +- app/Exports/ZipExports/ZipExportReader.php | 1 - .../views/exports/parts/import-item.blade.php | 14 ++++++++------ 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 47ab8f0a699..4f641d25bd7 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -36,6 +36,20 @@ public function metadataOnly(): void } } + public function children(): array + { + $children = [ + ...$this->pages, + ...$this->chapters, + ]; + + usort($children, function ($a, $b) { + return ($a->priority ?? 0) - ($b->priority ?? 0); + }); + + return $children; + } + public static function fromModel(Book $model, ZipExportFiles $files): self { $instance = new self(); diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index 5a5fe350f3a..bf2dc78f8de 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -20,7 +20,7 @@ class ZipExportChapter extends ZipExportModel public function metadataOnly(): void { - $this->description_html = $this->priority = null; + $this->description_html = null; foreach ($this->pages as $page) { $page->metadataOnly(); @@ -30,6 +30,11 @@ public function metadataOnly(): void } } + public function children(): array + { + return $this->pages; + } + public static function fromModel(Chapter $model, ZipExportFiles $files): self { $instance = new self(); diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 16e7e925539..097443df02b 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -23,7 +23,7 @@ class ZipExportPage extends ZipExportModel public function metadataOnly(): void { - $this->html = $this->markdown = $this->priority = null; + $this->html = $this->markdown = null; foreach ($this->attachments as $attachment) { $attachment->metadataOnly(); diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index 6b88ef61c33..c3d5c23cfec 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -5,7 +5,6 @@ use BookStack\Exceptions\ZipExportException; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; -use BookStack\Exports\ZipExports\Models\ZipExportModel; use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Util\WebSafeMimeSniffer; use ZipArchive; diff --git a/resources/views/exports/parts/import-item.blade.php b/resources/views/exports/parts/import-item.blade.php index 811a3b31bda..5da4b21405d 100644 --- a/resources/views/exports/parts/import-item.blade.php +++ b/resources/views/exports/parts/import-item.blade.php @@ -16,11 +16,13 @@ @icon('tag'){{ count($model->tags) }} @endif
    - @foreach($model->chapters ?? [] as $chapter) - @include('exports.parts.import-item', ['type' => 'chapter', 'model' => $chapter]) - @endforeach - @foreach($model->pages ?? [] as $page) - @include('exports.parts.import-item', ['type' => 'page', 'model' => $page]) - @endforeach + @if(method_exists($model, 'children')) + @foreach($model->children() as $child) + @include('exports.parts.import-item', [ + 'type' => ($child instanceof \BookStack\Exports\ZipExports\Models\ZipExportPage) ? 'page' : 'chapter', + 'model' => $child + ]) + @endforeach + @endif
    \ No newline at end of file From f79c6aef8d1c9aa83e9ce89ec1e5ac9d9e1eb570 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Fri, 22 Nov 2024 21:36:42 +0000 Subject: [PATCH 36/41] ZIP Imports: Updated import form to show loading indicator And disable button after submit. Added here because the import could take some time, so it's best to show an indicator to the user to show that something is happening, and help prevent duplicate submission or re-submit attempts. --- resources/js/components/index.js | 1 + resources/js/components/loading-button.ts | 38 +++++++++++++++++++ resources/sass/styles.scss | 4 ++ resources/views/exports/import-show.blade.php | 4 +- 4 files changed, 45 insertions(+), 2 deletions(-) create mode 100644 resources/js/components/loading-button.ts diff --git a/resources/js/components/index.js b/resources/js/components/index.js index 8ad5e14cb2e..12c991a51d8 100644 --- a/resources/js/components/index.js +++ b/resources/js/components/index.js @@ -30,6 +30,7 @@ export {HeaderMobileToggle} from './header-mobile-toggle'; export {ImageManager} from './image-manager'; export {ImagePicker} from './image-picker'; export {ListSortControl} from './list-sort-control'; +export {LoadingButton} from './loading-button'; export {MarkdownEditor} from './markdown-editor'; export {NewUserPassword} from './new-user-password'; export {Notification} from './notification'; diff --git a/resources/js/components/loading-button.ts b/resources/js/components/loading-button.ts new file mode 100644 index 00000000000..a793d30a235 --- /dev/null +++ b/resources/js/components/loading-button.ts @@ -0,0 +1,38 @@ +import {Component} from "./component.js"; +import {showLoading} from "../services/dom"; +import {el} from "../wysiwyg/utils/dom"; + +/** + * Loading button. + * Shows a loading indicator and disables the button when the button is clicked, + * or when the form attached to the button is submitted. + */ +export class LoadingButton extends Component { + + protected button!: HTMLButtonElement; + protected loadingEl: HTMLDivElement|null = null; + + setup() { + this.button = this.$el as HTMLButtonElement; + const form = this.button.form; + + const action = () => { + setTimeout(() => this.showLoadingState(), 10) + }; + + this.button.addEventListener('click', action); + if (form) { + form.addEventListener('submit', action); + } + } + + showLoadingState() { + this.button.disabled = true; + + if (!this.loadingEl) { + this.loadingEl = el('div', {class: 'inline block'}) as HTMLDivElement; + showLoading(this.loadingEl); + this.button.after(this.loadingEl); + } + } +} \ No newline at end of file diff --git a/resources/sass/styles.scss b/resources/sass/styles.scss index 2cf3cbf8221..2106f86e625 100644 --- a/resources/sass/styles.scss +++ b/resources/sass/styles.scss @@ -106,6 +106,10 @@ $loadingSize: 10px; } } +.inline.block .loading-container { + margin: $-xs $-s; +} + .skip-to-content-link { position: fixed; top: -52px; diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index e4f199aa20c..a28b79bb35c 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -59,7 +59,7 @@ ]) @endif -
    +
    {{ trans('common.cancel') }}
    - +
    From 9ecc91929a60a66ff7e821dfbc9d2b55197988f7 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 25 Nov 2024 15:54:15 +0000 Subject: [PATCH 37/41] ZIP Import & Exports: Addressed issues during testing - Handled links to within-zip page images found in chapter/book descriptions; Added test to cover. - Fixed session showing unrelated success on failed import. Tested import file-create undo on failure as part of this testing. --- app/Exports/Controllers/ImportController.php | 2 ++ .../ZipExports/ZipExportReferences.php | 7 ++--- lang/en/errors.php | 1 + tests/Exports/ZipExportTest.php | 26 +++++++++++++++++++ 4 files changed, 33 insertions(+), 3 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index a20c341fb02..b938dac8e29 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -89,6 +89,8 @@ public function run(int $id, Request $request) try { $entity = $this->imports->runImport($import, $parent); } catch (ZipImportException $exception) { + session()->flush(); + $this->showErrorNotification(trans('errors.import_zip_failed_notification')); return redirect($import->getUrl())->with('import_errors', $exception->errors); } diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index 0de409fa19a..bf5e02133ef 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -127,11 +127,12 @@ protected function handleModelReference(Model $model, ZipExportModel $exportMode return null; } - // We don't expect images to be part of book/chapter content - if (!($exportModel instanceof ZipExportPage)) { - return null; + // Handle simple links outside of page content + if (!($exportModel instanceof ZipExportPage) && isset($this->images[$model->id])) { + return "[[bsexport:image:{$model->id}]]"; } + // Find and include images if in visibility $page = $model->getPage(); if ($page && userCan('view', $page)) { if (!isset($this->images[$model->id])) { diff --git a/lang/en/errors.php b/lang/en/errors.php index ced80a32c1f..9d738379648 100644 --- a/lang/en/errors.php +++ b/lang/en/errors.php @@ -110,6 +110,7 @@ 'import_zip_cant_decode_data' => 'Could not find and decode ZIP data.json content.', 'import_zip_no_data' => 'ZIP file data has no expected book, chapter or page content.', 'import_validation_failed' => 'Import ZIP failed to validate with errors:', + 'import_zip_failed_notification' => 'Failed to import ZIP file.', 'import_perms_books' => 'You are lacking the required permissions to create books.', 'import_perms_chapters' => 'You are lacking the required permissions to create chapters.', 'import_perms_pages' => 'You are lacking the required permissions to create pages.', diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 12531239ff3..6e8462f596f 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -274,6 +274,32 @@ public function test_cross_reference_links_are_converted() $this->assertStringContainsString('href="[[bsexport:book:' . $book->id . ']]?view=true"', $pageData['html']); } + public function test_book_and_chapter_description_links_to_images_in_pages_are_converted() + { + $book = $this->entities->bookHasChaptersAndPages(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + + $this->asEditor(); + $this->files->uploadGalleryImageToPage($this, $page); + /** @var Image $image */ + $image = Image::query()->where('type', '=', 'gallery') + ->where('uploaded_to', '=', $page->id)->first(); + + $book->description_html = '

    Link to image

    '; + $book->save(); + $chapter->description_html = '

    Link to image

    '; + $chapter->save(); + + $zipResp = $this->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $bookData = $zip->data['book']; + $chapterData = $bookData['chapters'][0]; + + $this->assertStringContainsString('href="[[bsexport:image:' . $image->id . ']]"', $bookData['description_html']); + $this->assertStringContainsString('href="[[bsexport:image:' . $image->id . ']]"', $chapterData['description_html']); + } + public function test_cross_reference_links_external_to_export_are_not_converted() { $page = $this->entities->page(); From 95d62e7f573b5bbb04a66fae926c8a33c6ab5c43 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 25 Nov 2024 16:23:59 +0000 Subject: [PATCH 38/41] ZIP Imports/Exports: Fixed some lint and test issues - Updated test handling to create imports folder when required. - Updated some tests to delete created import zip files. --- resources/js/components/index.js | 2 +- resources/js/components/page-comments.js | 1 - tests/Exports/ZipImportTest.php | 8 ++++++++ tests/Exports/ZipTestHelper.php | 9 ++++++++- 4 files changed, 17 insertions(+), 3 deletions(-) diff --git a/resources/js/components/index.js b/resources/js/components/index.js index 12c991a51d8..24e60bd97f2 100644 --- a/resources/js/components/index.js +++ b/resources/js/components/index.js @@ -30,7 +30,7 @@ export {HeaderMobileToggle} from './header-mobile-toggle'; export {ImageManager} from './image-manager'; export {ImagePicker} from './image-picker'; export {ListSortControl} from './list-sort-control'; -export {LoadingButton} from './loading-button'; +export {LoadingButton} from './loading-button.ts'; export {MarkdownEditor} from './markdown-editor'; export {NewUserPassword} from './new-user-password'; export {Notification} from './notification'; diff --git a/resources/js/components/page-comments.js b/resources/js/components/page-comments.js index 1d6abfe2044..63900888aca 100644 --- a/resources/js/components/page-comments.js +++ b/resources/js/components/page-comments.js @@ -93,7 +93,6 @@ export class PageComments extends Component { updateCount() { const count = this.getCommentCount(); - console.log('update count', count, this.container); this.commentsTitle.textContent = window.$trans.choice(this.countText, count, {count}); } diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index 3644e9bdcb7..ad0e6b24137 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -168,6 +168,8 @@ public function test_import_upload_success() $resp->assertRedirect("/import/{$import->id}"); $this->assertFileExists(storage_path($import->path)); $this->assertActivityExists(ActivityType::IMPORT_CREATE); + + ZipTestHelper::deleteZipForImport($import); } public function test_import_show_page() @@ -325,6 +327,8 @@ public function test_run_revalidates_content() $resp = $this->followRedirects($resp); $resp->assertSeeText('The name field is required.'); $resp->assertSeeText('The id must be an integer.'); + + ZipTestHelper::deleteZipForImport($import); } public function test_run_checks_permissions_on_import() @@ -340,6 +344,8 @@ public function test_run_checks_permissions_on_import() $resp = $this->followRedirects($resp); $resp->assertSeeText('You are lacking the required permissions to create books.'); + + ZipTestHelper::deleteZipForImport($import); } public function test_run_requires_parent_for_chapter_and_page_imports() @@ -379,6 +385,8 @@ public function test_run_validates_correct_parent_type() $resp = $this->followRedirects($resp); $resp->assertSee('Parent book required for chapter import.'); + + ZipTestHelper::deleteZipForImport($import); } protected function runImportFromFile(UploadedFile $file): TestResponse diff --git a/tests/Exports/ZipTestHelper.php b/tests/Exports/ZipTestHelper.php index 2196f361c17..d830d8eb6bf 100644 --- a/tests/Exports/ZipTestHelper.php +++ b/tests/Exports/ZipTestHelper.php @@ -20,7 +20,14 @@ public static function importFromData(array $importData, array $zipData, array $ $import = Import::factory()->create($importData); $zip = static::zipUploadFromData($zipData, $files); - rename($zip->getRealPath(), storage_path($import->path)); + $targetPath = storage_path($import->path); + $targetDir = dirname($targetPath); + + if (!file_exists($targetDir)) { + mkdir($targetDir); + } + + rename($zip->getRealPath(), $targetPath); return $import; } From 0a182a45ba944c807c7a5ba6d6ba3a48809e1dc2 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 26 Nov 2024 15:59:39 +0000 Subject: [PATCH 39/41] ZIP Exports: Added detection/handling of images with external storage Added test to cover. --- app/Exports/ZipExports/ZipReferenceParser.php | 23 +++++++++++++-- .../ModelResolvers/ImageModelResolver.php | 29 +++++++++++++++++-- app/Uploads/ImageStorage.php | 16 ++++++++-- tests/Exports/ZipExportTest.php | 24 +++++++++++++++ 4 files changed, 85 insertions(+), 7 deletions(-) diff --git a/app/Exports/ZipExports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php index 5929383b4dd..a6560e3f289 100644 --- a/app/Exports/ZipExports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -11,6 +11,7 @@ use BookStack\References\ModelResolvers\ImageModelResolver; use BookStack\References\ModelResolvers\PageLinkModelResolver; use BookStack\References\ModelResolvers\PagePermalinkModelResolver; +use BookStack\Uploads\ImageStorage; class ZipReferenceParser { @@ -33,8 +34,7 @@ public function __construct( */ public function parseLinks(string $content, callable $handler): string { - $escapedBase = preg_quote(url('/'), '/'); - $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#()]/"; + $linkRegex = $this->getLinkRegex(); $matches = []; preg_match_all($linkRegex, $content, $matches); @@ -118,4 +118,23 @@ protected function getModelResolvers(): array return $this->modelResolvers; } + + /** + * Build the regex to identify links we should handle in content. + */ + protected function getLinkRegex(): string + { + $urls = [rtrim(url('/'), '/')]; + $imageUrl = rtrim(ImageStorage::getPublicUrl('/'), '/'); + if ($urls[0] !== $imageUrl) { + $urls[] = $imageUrl; + } + + + $urlBaseRegex = implode('|', array_map(function ($url) { + return preg_quote($url, '/'); + }, $urls)); + + return "/(({$urlBaseRegex}).*?)[\\t\\n\\f>\"'=?#()]/"; + } } diff --git a/app/References/ModelResolvers/ImageModelResolver.php b/app/References/ModelResolvers/ImageModelResolver.php index 331dd593b73..2c6c9fecd74 100644 --- a/app/References/ModelResolvers/ImageModelResolver.php +++ b/app/References/ModelResolvers/ImageModelResolver.php @@ -3,19 +3,22 @@ namespace BookStack\References\ModelResolvers; use BookStack\Uploads\Image; +use BookStack\Uploads\ImageStorage; class ImageModelResolver implements CrossLinkModelResolver { + protected ?string $pattern = null; + public function resolve(string $link): ?Image { - $pattern = '/^' . preg_quote(url('/uploads/images'), '/') . '\/(.+)/'; + $pattern = $this->getUrlPattern(); $matches = []; $match = preg_match($pattern, $link, $matches); if (!$match) { return null; } - $path = $matches[1]; + $path = $matches[2]; // Strip thumbnail element from path if existing $originalPathSplit = array_filter(explode('/', $path), function (string $part) { @@ -30,4 +33,26 @@ public function resolve(string $link): ?Image return Image::query()->where('path', '=', $fullPath)->first(); } + + /** + * Get the regex pattern to identify image URLs. + * Caches the pattern since it requires looking up to settings/config. + */ + protected function getUrlPattern(): string + { + if ($this->pattern) { + return $this->pattern; + } + + $urls = [url('/uploads/images')]; + $baseImageUrl = ImageStorage::getPublicUrl('/uploads/images'); + if ($baseImageUrl !== $urls[0]) { + $urls[] = $baseImageUrl; + } + + $imageUrlRegex = implode('|', array_map(fn ($url) => preg_quote($url, '/'), $urls)); + $this->pattern = '/^(' . $imageUrlRegex . ')\/(.+)/'; + + return $this->pattern; + } } diff --git a/app/Uploads/ImageStorage.php b/app/Uploads/ImageStorage.php index dc4abc0f281..ddaa26a9400 100644 --- a/app/Uploads/ImageStorage.php +++ b/app/Uploads/ImageStorage.php @@ -110,10 +110,20 @@ public function urlToPath(string $url): ?string } /** - * Gets a public facing url for an image by checking relevant environment variables. + * Gets a public facing url for an image or location at the given path. + */ + public static function getPublicUrl(string $filePath): string + { + return static::getPublicBaseUrl() . '/' . ltrim($filePath, '/'); + } + + /** + * Get the public base URL used for images. + * Will not include any path element of the image file, just the base part + * from where the path is then expected to start from. * If s3-style store is in use it will default to guessing a public bucket URL. */ - public function getPublicUrl(string $filePath): string + protected static function getPublicBaseUrl(): string { $storageUrl = config('filesystems.url'); @@ -131,6 +141,6 @@ public function getPublicUrl(string $filePath): string $basePath = $storageUrl ?: url('/'); - return rtrim($basePath, '/') . $filePath; + return rtrim($basePath, '/'); } } diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 6e8462f596f..17891c73d73 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -300,6 +300,30 @@ public function test_book_and_chapter_description_links_to_images_in_pages_are_c $this->assertStringContainsString('href="[[bsexport:image:' . $image->id . ']]"', $chapterData['description_html']); } + public function test_image_links_are_handled_when_using_external_storage_url() + { + $page = $this->entities->page(); + + $this->asEditor(); + $this->files->uploadGalleryImageToPage($this, $page); + /** @var Image $image */ + $image = Image::query()->where('type', '=', 'gallery') + ->where('uploaded_to', '=', $page->id)->first(); + + config()->set('filesystems.url', 'https://i.example.com/content'); + + $storageUrl = 'https://i.example.com/content/' . ltrim($image->path, '/'); + $page->html = '

    Original URLStorage URL

    '; + $page->save(); + + $zipResp = $this->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $ref = '[[bsexport:image:' . $image->id . ']]'; + $this->assertStringContainsString("Original URLStorage URL", $pageData['html']); + } + public function test_cross_reference_links_external_to_export_are_not_converted() { $page = $this->entities->page(); From edb684c72ce0b1f1cf9be90d338ee08e24b4a0cc Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 26 Nov 2024 17:53:20 +0000 Subject: [PATCH 40/41] ZIP Exports: Updated format doc with advisories regarding html/md --- dev/docs/portable-zip-file-format.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 7e5df3f015b..fbb31785824 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -13,7 +13,8 @@ Following the goals & ideals of BookStack, stability is very important. We aim f - Where reasonably possible, we will attempt to avoid modifications/removals of existing features/properties. - Where potentially breaking changes do have to be made, these will be noted in BookStack release/update notes. -The addition of new features/properties alone are not considered as a breaking change to the format. Breaking changes are considered as such where they could impact common/expected use of the existing properties and features we document, they are not considered based upon user assumptions or any possible breakage. For example if your application, using the format, breaks because we added a new property while you hard-coded your application to use the third property (instead of a property name), then that's on you. +The addition of new features/properties alone are not considered as a breaking change to the format. Breaking changes are considered as such where they could impact common/expected use of the existing properties and features we document, they are not considered based upon user assumptions or any possible breakage. +For example if your application, using the format, breaks because we added a new property while you hard-coded your application to use the third property (instead of a property name), then that's on you. ## Format Outline @@ -57,6 +58,23 @@ Here's an example of each type of such reference that could be used: [[bsexport:book:8]] ``` +## HTML & Markdown Content + +BookStack commonly stores & utilises content in the HTML format. +Properties that expect or provided HTML will either be named `html` or contain `html` in the property name. +While BookStack supports a range of HTML, not all HTML content will be supported by BookStack and be assured to work as desired across all BookStack features. +The HTML supported by BookStack is not yet formally documented, but you can inspect to what the WYSIWYG editor produces as a basis. +Generally, top-level elements should keep to common block formats (p, blockquote, h1, h2 etc...) with no nesting or custom structure apart from common inline elements. +Some areas of BookStack where HTML is used, like book & chapter descriptions, will strictly limit/filter HTML tag & attributes to an allow-list. + +For markdown content, in BookStack we target [the commonmark spec](https://commonmark.org/) with the addition of tables & task-lists. +HTML within markdown is supported but not all HTML is assured to work as advised above. + +### Content Security + +If you're consuming HTML or markdown within an export please consider that the content is not assured to be safe, even if provided directly by a BookStack instance. It's best to treat such content as potentially unsafe. +By default, BookStack performs some basic filtering to remove scripts among other potentially dangerous elements but this is not foolproof. BookStack itself relies on additional security mechanisms such as [CSP](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) to help prevent a range of exploits. + ## Export Data - `data.json` The `data.json` file is a JSON format file which contains all structured data for the export. The properties are as follows: @@ -114,9 +132,9 @@ The `pages` are not all pages within the book, just those that are direct childr - `images` - [Image](#image) array, optional, images used in this page. - `tags` - [Tag](#tag) array, optional, tags assigned to this page. -To define the page content, either `markdown` or `html` should be provided. Ideally these should be limited to the range of markdown and HTML which BookStack supports. +To define the page content, either `markdown` or `html` should be provided. Ideally these should be limited to the range of markdown and HTML which BookStack supports. See the ["HTML & Markdown Content"](#html--markdown-content) section. -The page editor type, and edit content will be determined by what content is provided. If non-empty `markdown` is provided, the page will be assumed as a markdown editor page (where permissions allow) and the HTML will be rendered from the markdown content. Otherwise, the provided `html` will be used as editor and display content. +The page editor type, and edit content will be determined by what content is provided. If non-empty `markdown` is provided, the page will be assumed as a markdown editor page (where permissions allow) and the HTML will be rendered from the markdown content. Otherwise, the provided `html` will be used as editor & display content. #### Image From bdca9fc1ce6f3f792106e86348cfb1479f4dd27c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 27 Nov 2024 16:30:19 +0000 Subject: [PATCH 41/41] ZIP Exports: Changed the instance id mechanism Adds an instance id via app settings. --- app/Exports/ZipExports/ZipExportBuilder.php | 4 +-- ...4_11_27_171039_add_instance_id_setting.php | 30 +++++++++++++++++++ dev/docs/portable-zip-file-format.md | 6 ++-- tests/Exports/ZipExportTest.php | 6 ++-- 4 files changed, 38 insertions(+), 8 deletions(-) create mode 100644 database/migrations/2024_11_27_171039_add_instance_id_setting.php diff --git a/app/Exports/ZipExports/ZipExportBuilder.php b/app/Exports/ZipExports/ZipExportBuilder.php index 42fb03541c0..4c5c638f591 100644 --- a/app/Exports/ZipExports/ZipExportBuilder.php +++ b/app/Exports/ZipExports/ZipExportBuilder.php @@ -69,8 +69,8 @@ protected function build(): string $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ - 'version' => trim(file_get_contents(base_path('version'))), - 'id_ciphertext' => encrypt('bookstack'), + 'id' => setting('instance-id', ''), + 'version' => trim(file_get_contents(base_path('version'))), ]; $zipFile = tempnam(sys_get_temp_dir(), 'bszip-'); diff --git a/database/migrations/2024_11_27_171039_add_instance_id_setting.php b/database/migrations/2024_11_27_171039_add_instance_id_setting.php new file mode 100644 index 00000000000..ee1e90d0303 --- /dev/null +++ b/database/migrations/2024_11_27_171039_add_instance_id_setting.php @@ -0,0 +1,30 @@ +insert([ + 'setting_key' => 'instance-id', + 'value' => Str::uuid(), + 'created_at' => Carbon::now(), + 'updated_at' => Carbon::now(), + 'type' => 'string', + ]); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + DB::table('settings')->where('setting_key', '=', 'instance-id')->delete(); + } +}; diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index fbb31785824..754cb4d3e94 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -93,12 +93,10 @@ The below details the objects & their properties used in Application Data. #### Instance -These details are mainly informational regarding the exporting BookStack instance from where an export was created from. +These details are informational regarding the exporting BookStack instance from where an export was created from. +- `id` - String, required, unique identifier for the BookStack instance. - `version` - String, required, BookStack version of the export source instance. -- `id_ciphertext` - String, required, identifier for the BookStack instance. - -The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This is used as a simple & rough way for a BookStack instance to be able to identify if they were the source (by attempting to decrypt the ciphertext). #### Book diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 17891c73d73..ebe07d052bc 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -54,8 +54,10 @@ public function test_export_metadata() $version = trim(file_get_contents(base_path('version'))); $this->assertEquals($version, $zip->data['instance']['version']); - $instanceId = decrypt($zip->data['instance']['id_ciphertext']); - $this->assertEquals('bookstack', $instanceId); + $zipInstanceId = $zip->data['instance']['id']; + $instanceId = setting('instance-id'); + $this->assertNotEmpty($instanceId); + $this->assertEquals($instanceId, $zipInstanceId); } public function test_page_export()