Remove unnecessary Metadata from Files

From Embedded Lab Vienna for IoT & Security
Revision as of 17:59, 14 June 2020 by Ikramer (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Summary

DescriptionCommand
Show Metadatafile /path/to/file
Show extended Metadataexiftool /path/to/file -v
Update Metadataexiftool /path/to/file -[TAG]=[VALUE]
Remove Metadataexiftool /path/to/file -[TAG]=
Remove ALL unnecessary Metadataexiftool /path/to/file -*=

Introduction

When generating data with cameras, word processing, mailing or other programs that create files, the generation program adds extended attributes to a newly created or modified file. These attributes are part of the so-called Metadata. In most cases this information is added to a file without the user being aware of it or ever having to make use of it. Metadata includes the file name, creation date, or file type, which are fairly legitimate, but they can also take the form of the author of a text document, right down to information about the device that created it. In today's world, where digital cameras are built into our mobile devices and photos took with these are shared with our beloved and the rest of the Internet, sharing metadata can be a privacy issue as well, because it can leak information about the user themselves, their hardware or software. When focusing on images, a modern smartphone camera typically stores the device model, camera settings and often the location where the photo was taken, a so-called geotag. However, the last depends on the user's settings, which most inexperienced users will never notice or change. To preserve privacy and prevent the exposure of potentially vulnerable hardware or software, it's worth reassessing the benefits of metadata storage. As for the content of the file itself, it is possible to remove unwanted metadata.

Note: Make sure to read this CNET article about metadata included with emails, and how it can be used to gather personal information and patterns.

Requirements

On UNIX like systems like GNU-Linux or macOS the command file(man) can be used to list the metadata included with the file. Additionaly when using macOS the command xattr (man) or getxattr (man) for GNU/Linux may reveal some additional metadata used by the operating system.

$ file ./LUPUS_XT2_Plus_PCB_Front.jpg

/path/to/file/LUPUS_XT2_Plus_PCB_Front.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 72x72, segment length 16, Exif Standard: [TIFF image data, little-endian, direntries=11, height=1867, manufacturer=samsung, model=SM-G965F, xresolution=164, yresolution=172, resolutionunit=2, software=PhotoScape, datetime=2020:02:24 19:14:16, width=2158], baseline, precision 8, 2158x1867, components 3
Note: Download this example image here.

Eventhough the metadata is easy to add and view, it is not so easy to remove it. A program that is able to do the job is Phil Harvey's ExifTool an can be installed using common package managers like apt for debian based systems or brew on macOS. Windows user can download the executable from the vendors webpage.

Description: "ExifTool is a platform-independent Perl library plus a command-line application for reading, writing and editing meta information in a wide variety of files." [source]

Description

Show Metadata
 $ exiftool -v LUPUS_XT2_Plus_PCB_Front.jpg

   ExifToolVersion = 11.85
   FileName = LUPUS_XT2_Plus_PCB_Front.jpg
   Directory = .
   FileSize = 3950536
   FileModifyDate = 1592068395
   FileAccessDate = 1592068396
   FileInodeChangeDate = 1592068395
   FilePermissions = 33188
   FileType = JPEG
   FileTypeExtension = JPG
   MIMEType = image/jpeg
 JPEG APP0 (14 bytes):
   + [BinaryData directory, 9 bytes]
   | JFIFVersion = 1 1
   | ResolutionUnit = 1
   | XResolution = 72
   | YResolution = 72
   | ThumbnailWidth = 0
   | ThumbnailHeight = 0
 JPEG APP1 (842 bytes):
   ExifByteOrder = II
   + [IFD0 directory with 11 entries]
   | 0)  ImageWidth = 2158
   | 1)  ImageHeight = 1867
   | 2)  Make = samsung
   | 3)  Model = SM-G965F
   | 4)  XResolution = 72 (72/1)
   | 5)  YResolution = 72 (72/1)
   | 6)  ResolutionUnit = 2
   | 7)  Software = PhotoScape
   | 8)  ModifyDate = 2020:02:24 19:14:16
   | 9)  YCbCrPositioning = 1
   | 10) ExifOffset (SubDirectory) -->
   | + [ExifIFD directory with 36 entries]
   | | 0)  ExposureTime = 0.02 (1/50)
   | | 1)  FNumber = 2.4 (240/100)
   | | 2)  ExposureProgram = 2
   | | 3)  ISO = 160
   | | 4)  ExifVersion = 0220
   | | 5)  DateTimeOriginal = 2020:02:24 19:14:16
   | | 6)  CreateDate = 2020:02:24 19:14:16
   | | 7)  ComponentsConfiguration = 1 2 3 0
   | | 8)  ShutterSpeedValue = 5.64 (564/100)
   | | 9)  ApertureValue = 2.52 (252/100)
   | | 10) BrightnessValue = 2.36 (236/100)
   | | 11) ExposureCompensation = 0 (0/10)
   | | 12) MaxApertureValue = 1.16 (116/100)
   | | 13) MeteringMode = 2
   | | 14) Flash = 0
   | | 15) FocalLength = 4.3 (430/100)
   | | 16) UserComment =
   | | 17) SubSecTime = 0020
   | | 18) SubSecTimeOriginal = 0020
   | | 19) SubSecTimeDigitized = 0020
   | | 20) FlashpixVersion = 0100
   | | 21) ColorSpace = 1
   | | 22) ExifImageWidth = 2158
   | | 23) ExifImageHeight = 1867
   | | 24) InteropOffset (SubDirectory) -->
   | | + [InteropIFD directory with 2 entries]
   | | | 0)  InteropIndex = R98
   | | | 1)  InteropVersion = 0100
   | | 25) SceneType = .
   | | 26) CustomRendered = 0
   | | 27) ExposureMode = 0
   | | 28) WhiteBalance = 0
   | | 29) DigitalZoomRatio = undef (0/0)
   | | 30) FocalLengthIn35mmFormat = 26
   | | 31) SceneCaptureType = 0
   | | 32) Contrast = 0
   | | 33) Saturation = 0
   | | 34) Sharpness = 0
   | | 35) ImageUniqueID = I12LLKF00SM
 JPEG DQT (65 bytes):
 JPEG DQT (65 bytes):
 JPEG SOF0 (15 bytes):
   ImageWidth = 2158
   ImageHeight = 1867
   EncodingProcess = 0
   BitsPerSample = 8
   ColorComponents = 3
   YCbCrSubSampling = 2 2
 JPEG DHT (29 bytes):
 JPEG DHT (179 bytes):
 JPEG DHT (29 bytes):
 JPEG DHT (179 bytes):
 JPEG SOS
Note: Make sure to checkout the man page for additional cool features within the tool like the verbose mode (-v) or the multilanguage support (-lang [COUNTRYCODE]).

In this example scenario, I'd like to remove the Make (Camera Vendor), Model (Camera Model), Software (second stage editing tool), ImageUniqueID metadata as well as all timestamps from the example photo. To write or delete metadata, tag values are assigned using -TAG=[VALUE], and/or the -geotag options. Additionally I'm going to do this operation for a bulk of photos.

Warning: Use the -v option to display the correct tag names used for modification or deletion!
Remove Metadata
$ exiftool LUPUS_XT2_Plus_*.jpg -Make= -Software= -Model= -ImageUniqueID=
Update Metadata
# Update multiple files at the same time
$ exiftool LUPUS_XT2_Plus_*.jpg -Comment="Hello darkness my old friend"

# Always update timestamps last
$ exiftool LUPUS_XT2_Plus_*.jpg -CreateDate="1970:01:01 00:00:00" -DateTimeOriginal="1970:01:01 00:00:00" -ModifyDate="1970:01:01 00:00:00"
Check modifications
$ exiftool LUPUS_XT2_Plus_PCB_Back.jpg
   ExifTool Version Number         : 11.85
   File Name                       : LUPUS_XT2_Plus_PCB_Back.jpg
   Directory                       : .
   File Size                       : 3.2 MB
   File Modification Date/Time     : 2020:06:13 20:30:20+02:00
   File Access Date/Time           : 2020:06:13 20:30:22+02:00
   File Inode Change Date/Time     : 2020:06:13 20:30:21+02:00
   File Permissions                : rw-r--r--
   File Type                       : JPEG
   File Type Extension             : jpg
   MIME Type                       : image/jpeg
   JFIF Version                    : 1.01
   Exif Byte Order                 : Little-endian (Intel, II)
   X Resolution                    : 72
   Y Resolution                    : 72
   Resolution Unit                 : inches
   Modify Date                     : 1970:01:01 00:00:00
   Y Cb Cr Positioning             : Centered
   Exposure Time                   : 1/50
   F Number                        : 2.4
   Exposure Program                : Program AE
   ISO                             : 200
   Exif Version                    : 0220
   Date/Time Original              : 1970:01:01 00:00:00
   Create Date                     : 1970:01:01 00:00:00
   Components Configuration        : Y, Cb, Cr, -
   Shutter Speed Value             : 1/50
   Aperture Value                  : 2.4
   Brightness Value                : 1.99
   Exposure Compensation           : 0
   Max Aperture Value              : 1.5
   Metering Mode                   : Center-weighted average
   Flash                           : No Flash
   Focal Length                    : 4.3 mm
   User Comment                    :
   Sub Sec Time                    : 0456
   Sub Sec Time Original           : 0456
   Sub Sec Time Digitized          : 0456
   Flashpix Version                : 0100
   Color Space                     : sRGB
   Exif Image Width                : 2158
   Exif Image Height               : 1867
   Interoperability Index          : R98 - DCF basic file (sRGB)
   Interoperability Version        : 0100
   Scene Type                      : Unknown (.)
   Custom Rendered                 : Normal
   Exposure Mode                   : Auto
   White Balance                   : Auto
   Digital Zoom Ratio              : undef
   Focal Length In 35mm Format     : 26 mm
   Scene Capture Type              : Standard
   Contrast                        : Normal
   Saturation                      : Normal
   Sharpness                       : Normal
   Comment                         : Hello darkness my old friend
   Image Width                     : 2158
   Image Height                    : 1867
   Encoding Process                : Baseline DCT, Huffman coding
   Bits Per Sample                 : 8
   Color Components                : 3
   Y Cb Cr Sub Sampling            : YCbCr4:2:0 (2 2)
   Aperture                        : 2.4
   Image Size                      : 2158x1867
   Megapixels                      : 4.0
   Scale Factor To 35 mm Equivalent: 6.0
   Shutter Speed                   : 1/50
   Create Date                     : 1970:01:01 00:00:00.0456
   Date/Time Original              : 1970:01:01 00:00:00.0456
   Modify Date                     : 1970:01:01 00:00:00.0456
   Circle Of Confusion             : 0.005 mm
   Field Of View                   : 69.4 deg
   Focal Length                    : 4.3 mm (35 mm equivalent: 26.0 mm)
   Hyperfocal Distance             : 1.55 m
   Light Value                     : 7.2

Note: The tool will store by default a copy of the original file under the same name ending _original.
Warning: Be sure to verify that the new files are OK before erasing the originals. Once in write mode, exiftool will ignore any read-specific options.

Maximus Paranoism

A camera pro may be able to still identfy the camera or the type of device by the remaining metadata. For the more paranoid, consider removing ALL metadata using -*= and you may also want to change local creation, modification, access file timestamps using touch. Read the article Linux – Fake File Access, Modify and Change TimeStamps for more information.

Remove ALL unnecessary Metadata
$ exiftool LUPUS_XT2_Plus_PCB_Back.jpg -*=
$ exiftool LUPUS_XT2_Plus_PCB_Back.jpg

  ExifTool Version Number         : 11.85
  File Name                       : LUPUS_XT2_Plus_PCB_Back.jpg
  Directory                       : .
  File Size                       : 3.2 MB
  File Modification Date/Time     : 2020:06:13 20:35:41+02:00
  File Access Date/Time           : 2020:06:13 20:36:03+02:00
  File Inode Change Date/Time     : 2020:06:13 20:36:02+02:00
  File Permissions                : rw-r--r--
  File Type                       : JPEG
  File Type Extension             : jpg
  MIME Type                       : image/jpeg
  Image Width                     : 2158
  Image Height                    : 1867
  Encoding Process                : Baseline DCT, Huffman coding
  Bits Per Sample                 : 8
  Color Components                : 3
  Y Cb Cr Sub Sampling            : YCbCr4:2:0 (2 2)
  Image Size                      : 2158x1867
  Megapixels                      : 4.0

References