idr0026-weigelin-immunotherapy S-BIAD860 #648
Comments
|
Conversion time: 9min |
dominikl
moved this from re-import test image
to convert all data to NGFF
in NGFF conversion
|
Trying to estimate how much space is needed for this conversion. First image is Images vary in size for the study, but about 111 .pattern images (see IDR/idr-utils#56) Maybe 300 GB or more needed (maybe up to 500 GB)? |
|
Looks like all the pattern files we need to convert are under: Corresponds to image count from IDR/idr-utils#56 |
will-moore
moved this from convert all data to NGFF
to upload data to s3
in NGFF conversion
|
|
Checking on s3... This image has only a single
cc @sbesson |
|
However, the OME.xml looks OK, e.g. https://uk1s3.embassy.ebi.ac.uk/idr0026/zarr/3.50.6-3.140922_11-36-07.00.ome.zarr/OME/METADATA.ome.xml |
will-moore
moved this from upload data to s3
to create new Fileset to replace original Fileset
in NGFF conversion
will-moore
moved this from create new Fileset to replace original Fileset
to upload some data to s3 and test
in NGFF conversion
|
On pilot-idr0125... These look good in OMERO, compared to existing IDR
|
|
@will-moore not 100% sure of what went wrong on your conversion but using the converter library shipping with the current IDR version of Bio-Formats, I get and the $ cat 3.49.6-3.140922_11-33-57.00.zarr/0/.zattrs
{
"multiscales" : [ {
"metadata" : {
"method" : "loci.common.image.SimpleImageScaler",
"version" : "Bio-Formats 0.6.10"
},
"axes" : [ {
"name" : "t",
"type" : "time"
}, {
"name" : "c",
"type" : "channel"
}, {
"name" : "z",
"type" : "space"
}, {
"name" : "y",
"type" : "space"
}, {
"name" : "x",
"type" : "space"
} ],
"name" : "11-33-57_PMT - PMT [BD2_GREEN] [00]_Time Time0000.tif",
"datasets" : [ {
"path" : "0",
"coordinateTransformations" : [ {
"scale" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"type" : "scale"
} ]
}, {
"path" : "1",
"coordinateTransformations" : [ {
"scale" : [ 1.0, 1.0, 1.0, 2.0, 2.0 ],
"type" : "scale"
} ]
} ],
"version" : "0.4"
} ],
"omero" : {
"channels" : [ {
"color" : "FF0000",
"coefficient" : 1,
"active" : true,
"label" : "BD2_GREEN",
"window" : {
"min" : 372.0,
"max" : 15788.0,
"start" : 372.0,
"end" : 15788.0
},
"family" : "linear",
"inverted" : false
}, {
"color" : "00FF00",
"coefficient" : 1,
"active" : true,
"label" : "BD8_DEEPR",
"window" : {
"min" : 373.0,
"max" : 16188.0,
"start" : 373.0,
"end" : 16188.0
},
"family" : "linear",
"inverted" : false
}, {
"color" : "0000FF",
"coefficient" : 1,
"active" : true,
"label" : "BD7_RED",
"window" : {
"min" : 759.0,
"max" : 8978.0,
"start" : 759.0,
"end" : 8978.0
},
"family" : "linear",
"inverted" : false
}, {
"color" : "FF0000",
"coefficient" : 1,
"active" : false,
"label" : "FD6_FDRED",
"window" : {
"min" : 237.0,
"max" : 12339.0,
"start" : 237.0,
"end" : 12339.0
},
"family" : "linear",
"inverted" : false
} ],
"rdefs" : {
"defaultT" : 0,
"model" : "color",
"defaultZ" : 13
}
}
}Sounds like the best way forward would be to redo the whole conversion? |
|
@sbesson Looking at #648 (comment), it looks like I used the same version: |
|
Testing... (and failing!)... |
|
Even using the same lib as @sbesson gives me same result?! |
|
I suspect there's something wrong with your environment and particularly the |
|
That didn't work either! I wanted to try on a different machine completely... Do I need to clone all of https://github.com/IDR/idr-metadata here? |
|
Just confirming... |
For the sake of testing, you might just want to copy the single
The inconsistency in the output is very concerning. Have you tried after fully deactivating Conda, not just your environment? |
No. How do you do that? |
|
|
|
I already did that. How's that different from deactivating your environment? I tried on a different machine... Then tried... WAT!? |
|
@will-moore I think I found the source of the issue. Can you try one more test with your last configuration, running |
|
The above removal of contents of the bucket ran very slowly and has only resulted in the removal of a handful of zarr filesets out of the 111 originally there. Since we want to delete ALL the filesets uploaded, probably quicker to delete the bucket and recreate.. ran This seemed to hang/time-out and doesn't seem to have had any affect: Reverted to running the rm again in a screen |
|
@sbesson - Seems that the memo issue is something it would be good to fix (or at least warn) to prevent others suffering the pain above! I can create an issue somewhere, but where? |
|
From my side, the immediate candidates are:
Possibly the outstanding action would be to retest a similar scenario using bioformats2raw 0.7.0, a multi-channel pattern dataset and identify whether it's IDR specific. /cc @melissalinkert |
|
Ah - apologies @sbesson: I just realised you meant that there is probably just 1 issue (not 4) but it needs testing to determine where the issue lies! |
|
Retested with a simpler version of the pattern file with 2 timepoints compatible with upstream Bio-Formats Placed a copy of this pattern file under The Based on the above, I am leaning towards options 1 and 2 i.e. it's an IDR/bioformats specific issue which probably will be classified as |
|
Started creating zips in a Screen |
|
With all the previous Filesets deleted from s3, uploaded just a couple of different new ones to test... Ooops - got an extra directory in there, but the images look good: |
|
Uploading zips to BioStudies... |
will-moore
moved this from upload some data to s3 and test
to BioStudies Submission
in NGFF conversion
|
francesw
changed the title
francesw
moved this from BioStudies Submission
to Data on Embassy s3
in NGFF conversion
|
Currently we have 20 out of 111 Filesets "viewable" at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD860.html... |
|
All good (missing thumbnails in screenshot are for images not included in the 20 updated by mkngff above:
|
will-moore
moved this from Data on Embassy s3
to create new Filesets in idr-next
in NGFF conversion
|
Testing on Updated to today's OMEZarrReader.jar (only on omeroreadwrite server - not proxies). Use all 111 Images in idr0026.csv - see IDR/idr-utils@003b3a3 Started |
|
mkngff just done (nearly 12:00). 25869ms is 26 secs for setId |
|
This issue has been mentioned on Image.sc Forum. There might be relevant details there: https://forum.image.sc/t/file-format-to-store-images-using-ngff-coverter/98320/10 |
idr0026-weigelin-immunotherapy
The text was updated successfully, but these errors were encountered: