Dismiss
InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Objectizer optimizations in Unified File and Object Access protocol

IP.com Disclosure Number: IPCOM000247179D
Publication Date: 2016-Aug-12
Document File: 4 page(s) / 32K

Publishing Venue

The IP.com Prior Art Database

Abstract

Disclosed is an algorithm to optimize the process of objectization in object store supporting uified file and object access. The alogrithm optimally makes use of parallel computing such that the utilization of resources of nodes participaring in the computing is deligiently used.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 32% of the total text.

Page 01 of 4

Objectizer optimizations in Unified File and Object Access protocol

Unified File and Object Storage:

Today object storage is a preferred solution to handle exponential growth of unstructured data. It is a highly scalable and economical storage solution for storing such large data. Big data analytics is one of the prominent use case to empower business these days but advanced analytics systems like MapReduce work on distributed filesystems and not object stores. To bridge the gap between traditional filesystems and object storage, a feature of unified file and object access was introduced in IBM Spectrum Scale. With this feature same data can be accessed by object as well as file protocols. This helps analytics systems to access same data from filesystem interface to run in-place analytics. This feature also helps in cases where scientific data is ingested into traditional filesystem via legacy devices and needs to be accessed over cloud via object protocol.

SwiftOnFile:

Swift is an Openstack's implementation of highly available, distributed, eventually consistent object storage. It uses a filesystem with extended attribute support as its backend. When SWIFT sits over clustered filesystem, Swiftonfile, a module in Swift can be used to enable placing of object data on the underlying filesystem such that it can also be accessed as a file using traditional file protocols like NFS/SMB/posix. This module enables data ingest using object interface and access using file protocols.

Objectizer:

The unified file and object feature in IBM Spectrum Scale uses SwiftOnFile module to enable file access to object data. Objectizer is a service implemented on top of this that enables object access to file data, i.e. it makes it possible for the files ingested via traditional file interfaces to be made visible to object interface. It corrects the object side stats like account stats and container stats as well as the container listing to accommodate ingested file information. This service uses GPFS ILM policy scans to identify files that have changed from file interface.

Problem 1: High CPU utilization

In the objectizer implementation one typically runs a parallel computing policy to identify files that have been newly added from file interface. This is done by using "swift.user.metadata is null" filter, since this xattr is set only when the file got created during object create or after the file has been objectized. When files are added from file interface, this attribute is not present. In typical implementation one will run parallel policy that will create and dispatch many threads each on given nodes to perform policy execution. Number of threads to be used by objectizer is configurable and nodelist contains the list of all protocol nodes.

To identify deleted files from file interface a policy scan cannot be used as a policy scan cannot return the files that have already been deleted. For this traditional algorithm gets a list of all objectized...