TimeSketch API Import Error (Plaso version 20220724 and 20221229)

[fullsendben]

Hi Folks,

I am having an error when uploading a plaso file to the TimeSketch server. The version of plaso installed on the TS server is 20221229-1ppa1~jammy and the version installed on the server where the plaso file was created is 20220724-1ppa1~focal. Is there a known compatability issue between these two?

I did not have this error when I was uploading to an older TS server that was running 20220724-1ppa1~focal.

Details:

• OS Version on host where plaso file was created: 20.04.5 LTS (Focal Fossa)
• Plaso version on host where plaso file was created: 20220724-1ppa1~focal
• OS on TimeSketch server: 20.04.5 LTS (Focal Fossa)
• Plaso version on TimeSketch server: 20221229-1ppa1~jammy

Plaso version on Forensic host(plaso file creation host):

root@hostname:/tmp# pinfo.py -V
plaso - pinfo version 20220724

root@hostname:/home/ubuntu# sudo apt show plaso-tools
Package: plaso-tools
Version: 20220724-1ppa1~focal
Priority: extra
Section: python
Source: plaso
Maintainer: Log2Timeline maintainers <log2timeline-maintainers@googlegroups.com>
Installed-Size: 23.6 kB
Depends: python3-plaso (>= 20220724-1ppa1~focal)
Download-Size: 4084 B
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/gift/stable/ubuntu focal/main amd64 Packages
Description: Tools of plaso (log2timeline)
 Plaso (log2timeline) is a framework to create super timelines. Its
 purpose is to extract timestamps from various files found on typical
 computer systems and aggregate them.
 
 -------------------------------
 
 root@hostname:/home/ubuntu# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Plaso/OS version on timesketch-web-v2 container:

# pinfo.py -V
plaso - pinfo version 20221229

# apt show plaso-tools
Package: plaso-tools
Version: 20221229-1ppa1~jammy
Status: install ok installed
Priority: extra
Section: python
Source: plaso
Maintainer: Log2Timeline maintainers <log2timeline-maintainers@googlegroups.com>
Installed-Size: 23.6 kB
Depends: python3-plaso (>= 20221229-1ppa1~jammy)
Homepage: https://github.com/log2timeline/plaso
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Tools of plaso (log2timeline)
 Plaso (log2timeline) is a framework to create super timelines. Its
 purpose is to extract timestamps from various files found on typical
 computer systems and aggregate them.
 
 -------------------------------
 
# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Web Error (timesketch server):

Elasticsearch index: f0a0f2e32c664aa7bc69ce4caa1715e0
 Created by: user
 Created at: 2023-02-28 21:12


2023-02-28T21:12:23.511623

 Provider: CLI importer tool
 Context: /usr/local/bin/timesketch_importer --host http://<TS IP address> -u <user> -p <password> ./timeline.plaso
 User: user
 File on disk: /usr/share/timesketch/upload/68f86ae4598d49f289b784ec1328308a
 File size: 82.18MB
 Original filename: timeline.plaso
 Data label: plaso


Error detail

Traceback (most recent call last):
File "/usr/bin/pinfo.py", line 85, in <module>
 if not Main():
File "/usr/bin/pinfo.py", line 62, in Main
 tool.PrintStorageInformation()
File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 1673, in PrintStorageInformation
 storage_reader = self._GetStorageReader(self._storage_file_path)
File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 587, in _GetStorageReader
 storage_factory.StorageFactory.CreateStorageReaderForFile(path))
File "/usr/lib/python3/dist-packages/plaso/storage/factory.py", line 49, in CreateStorageReaderForFile
 return sqlite_reader.SQLiteStorageFileReader(path)
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/reader.py", line 20, in __init__
 self._store.Open(path=path)
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 1150, in Open
 self._ReadAndCheckStorageMetadata(check_readable_only=True)
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 589, in _ReadAndCheckStorageMetadata
 self._CheckStorageMetadata(
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 208, in _CheckStorageMetadata
 raise IOError(
OSError: Format version: 20220716 is too old and can no longer be read.

Celery Error log (timesketch server):

root@ip-172-31-5-222:/opt/timesketch/timesketch# cat logs/worker.log
[2023-02-28 20:24:51,908] py.warnings/WARNING /usr/local/lib/python3.10/dist-packages/celery/app/utils.py:204: CDeprecationWarning:
    The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_backend instead

  deprecated.warn(description=f'The {setting!r} setting',

[2023-02-28 20:24:51,909] celery.apps.worker/WARNING Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
[2023-02-28 20:24:54,667] celery.worker.consumer.connection/INFO Connected to redis://redis:6379//
[2023-02-28 20:24:54,669] celery.worker.consumer.mingle/INFO mingle: searching for neighbors
[2023-02-28 20:24:55,676] celery.worker.consumer.mingle/INFO mingle: all alone
[2023-02-28 20:24:55,684] celery.apps.worker/INFO celery@57547aeed802 ready.
[2023-02-28 21:12:23,629] celery.worker.strategy/INFO Task timesketch.lib.tasks.run_plaso[ce294ffb778f47c8a7a5e40f0eebd86d] received
[2023-02-28 21:12:23,952] timesketch.tasks/INFO Index timeline [i-07a0bdcb2ab30939b] to index [f0a0f2e32c664aa7bc69ce4caa1715e0] (source: plaso)
[2023-02-28 21:12:24,947] celery.app.trace/ERROR Task timesketch.lib.tasks.run_plaso[ce294ffb778f47c8a7a5e40f0eebd86d] raised unexpected: CalledProcessError(1, ['pinfo.py', '--output-format', 'json', '--sections', 'events', '/usr/share/timesketch/upload/68f86ae4598d49f289b784ec1328308a'])
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/timesketch/app.py", line 225, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/tasks.py", line 647, in run_plaso
    command = subprocess.run(cmd, capture_output=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pinfo.py', '--output-format', 'json', '--sections', 'events', '/usr/share/timesketch/upload/68f86ae4598d49f289b784ec1328308a']' returned non-zero exit status 1.

pinfo on upload file, in timesketch-web-v2 container

# pinfo.py /usr/share/timesketch/upload/68f86ae4598d49f289b784ec1328308a
Traceback (most recent call last):
  File "/usr/bin/pinfo.py", line 85, in <module>
    if not Main():
  File "/usr/bin/pinfo.py", line 62, in Main
    tool.PrintStorageInformation()
  File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 1673, in PrintStorageInformation
    storage_reader = self._GetStorageReader(self._storage_file_path)
  File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 587, in _GetStorageReader
    storage_factory.StorageFactory.CreateStorageReaderForFile(path))
  File "/usr/lib/python3/dist-packages/plaso/storage/factory.py", line 49, in CreateStorageReaderForFile
    return sqlite_reader.SQLiteStorageFileReader(path)
  File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/reader.py", line 20, in __init__
    self._store.Open(path=path)
  File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 1150, in Open
    self._ReadAndCheckStorageMetadata(check_readable_only=True)
  File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 589, in _ReadAndCheckStorageMetadata
    self._CheckStorageMetadata(
  File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 208, in _CheckStorageMetadata
    raise IOError(
OSError: Format version: 20220716 is too old and can no longer be read.

[jaegeral]

Hey. The explanation is in the error message. The version you used is simply to old. Please update your plaso version to create the file and it should work.

Cheers

[fullsendben]

Hey jaegeral - Thanks for the suggestion, I figured it is likely this. Just wanted to double check since the install instructions were recommending to use Ubuntu 20.04 (I know that was for the TS server, not the plaso creation system). I'll work on getting our AMI's updated to 22.04 so we can update plaso to the newest version.