Possible issue parsing large number of MSG files

[JOKAZ12345]

Hi guys, I'm trying to parse more than 500k .msg files that I have inside several subfolders. Using iped 4.2.0 I can only see that it parsed only 270k msg files with no errors. I used Axiom and confirmed it parsed the msg files.

PS: I also used an older version of iped v4.1.5 and got similar results :|

Does anyone had similar problems before?

[wladimirleite]

Were the other (failed) MSG files added to the case?
Were there any errors in the processing log related to these failed MSGs?
Try to identify a few MSG that were not parsed (or processed) and process only them.
Can you privately share some samples of these MSGs?

[lfcnassif]

Seems you are processing an E01 image, right? Are all msg files reported as allocated (not deleted) by Axiom? Are there any deleted? Processing the E01 locally shouldn't make difference. I suggest you to export all msg files to a local folder, or convert the E01 to an AD1 logical image using FTKImager, then to process the exported files or AD1 using IPED again. Please report results.

[lfcnassif]

Before that, please check if conf/HashDBLookupConfig.txt->excludeKnown option wasn't set to true.

[lfcnassif]

If the MSG files are not into compressed archives, please process the E01 image again using -profile fastmode. If they are not added to the case, that would suggest an issue with sleuthkit dependency.

[JOKAZ12345]

It's an E01 image indeed and all the files are allocated. They are split between 2 root folders, let's say A and B and then severall subfolders inside each. AFAIK it was only 3 level deep.

Today I have some free time and will do that.

EDIT 1: excludeKnown = false

[JOKAZ12345]

Fastmode solved the issue.

My IPEDConfig.txt has:

enableHash = true
processFileSignatures = true
enableFileParsing = true
expandContainers = true
processEmbeddedDisks = true
enableRegexSearch = true
enableGraphGeneration = true
indexFileContents = true

---CategoriesToExpand.txt---

Mailboxes
Compressed Archives
Emails
#Databases
Chats
#Contacts
#ISO disks
#Internet History
#User Activities
#Event Transcript
#GDrive Synced Files
#OLE files
#Georeferenced Files
#Peer-to-peer
#Chrome Cache
#Event Files

Generates registry reports:

#Windows Registry

Generates USN Journal reports

#USN Journal

#Generate Open Financial Exchange Spreadsheets
#Open Financial Exchange

#Text Documents
#Spreadsheets
#Presentations
#RTF Documents
#PDF Documents
#HTML Documents

To expand jar files, comment out the 'application/java-archive' line in ParserConfig.xml

#Programs and Libraries

[JOKAZ12345]

Now that we known thats probably related to a config file. I got only on IpedConfig

enableFileParsing = true

..
Some .msg files are taking more than 10 mins to parse ?

All the exceptions are related to:

• org.apache.tika.exception.TikaException: No TIFF image reader in classpath!
• org.apache.tika.exception.TikaException: Exception from iped.parsers.image.TiffPageParser

I got on:
--- ParsingTaskConfig.txt ---
parseUnknownFiles = true
parseCorruptedFiles = true
sortPDFChars = true
processImagesInPDFs = true

So I'm making some changes and putting all those 4 to false

[lfcnassif]

Some .msg files are taking more than 10 mins to parse ?

By the screenshots, yes. How much RAM do you have? Have you enabled OCR? Maybe it is running on attachments. Could you take a thread dump at this exact slow msg parsing moment? It would tell us the exact code point taking time to run: jstack -l pid

org.apache.tika.exception.TikaException: No TIFF image reader in classpath!

This is really strange, it shouldn't happen, it may indicate a corrupted installation (libraries missing). I suggest you to reinstall IPED from scratch again and run it using the iped.exe executable.

[lfcnassif]

enableFileParsing = true

I would also suggest testing with processFileSignatures enabled.

[JOKAZ12345]

So far so good.. The only exceptions it's giving me now for hundreds of .msg files is:

[WARN] [parsers.standard.StandardParser] ParsingThread-8 Parsing exception: PATHTOFILE/ABC.msg (null bytes) org.apache.poi.poifs.filesystem.NotOLE2FileException: Invalid header signature; read 0x0000000000000000, expected 0xE11AB1A1E011CFD0 - Your file appears not to be a valid OLE2 document

This one, ABC.msg, the size is 21,8 MB (22 868 992 bytes) and the header appears to be valid:

I checked with axiom and it parsed correctly this file.

[lfcnassif]

Is this the internal IPED Hex viewer? Maybe your temp folder run out of space or maybe you anti-virus is blocking reading file contents from the temp folder, not sure.

I checked with axiom and it parsed correctly this file.

Have you checked the opposite? Files not parsed by Axiom and parsed by IPED?

Please share the full processing log and provide sample files to reproduce the situation, I need them to continue to help you.

[JOKAZ12345]

No problem with space and AV is disabled. I've got 128 of ram.

OCR was disabled on ipedconfig at that time but I had those 4 variables enabled on taskconfig. It's not the iped hex viewer but an external one.

As soon as I can I'll share the logs and files with you.

Thanks again

[lfcnassif]

Please check the file header on IPED Hex viewer, so we can be sure if it is being read correctly from the E01 image or not.

About those 4 and the other configuration options posted, they shouldn't affect the number of allocated files added to the case, except ignoreDuplicates and excludeKnown. If you are not using triage or blind profiles, all allocated files should be added with those options disabled.

[JOKAZ12345]

Hi @lfcnassif,

The filed header on IPED is the same:

I checked at least 10 random .msg files where this error occured, and they were still parsed correctly within iped despite appearing in the log as err.

All the files parsed in axiom now appear on iped. So I can assume, it was most likely a config issue.

[lfcnassif]

Thanks for checking and returning feedback.