Amazon Kinesis Streams introduces server-side encryption to meet data management requirements 



Amazon Web Services announced this week that users can now encrypt data in their Amazon Kinesis streams using server-side encryption and AWS Key Management Service (KMS) keys. Server-side encryption makes it easy to meet strict data management requirements by encrypting data at rest within Kinesis Streams.

Users can also use the Kinesis management console or the AWS SDK to get the encryption status of a stream, and check if a specific read or write operation was encrypted. Consumers can also audit the encryption history using AWS CloudTrail.

Amazon Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing. The Amazon Kinesis Streams delivers accelerated log and data feed intake, so that data producers push data to an Amazon Kinesis stream as soon as the data is produced, preventing data loss in case of data producer failures. For example, system and application logs can be continuously added to a stream and be available for processing within seconds. Users can also extract metrics and generate reports from Amazon Kinesis stream data in real-time. For example, Amazon Kinesis Application can work on metrics and reporting for system and application logs as the data is streaming in, rather than wait to receive data batches.

With Amazon Kinesis Streams, users can run real-time streaming data analytics. For example, if a user can add clickstreams to the Amazon Kinesis stream and have Amazon Kinesis Application run analytics in real-time, enabling insights out of data at a scale of minutes instead of hours or days. Users can create Directed Acyclic Graphs (DAGs) of Amazon Kinesis Applications and data streams. In this scenario, one or more Amazon Kinesis Applications can add data to another Amazon Kinesis stream for further processing, enabling successive stages of stream processing.

Amazon Kinesis Streams enables real-time processing of streaming big data. It provides ordering of records, as well as the ability to read and/or replay records in the same order to multiple Amazon Kinesis Applications. The Amazon Kinesis Client Library (KCL) delivers all records for a given partition key to the same record processor, making it easier to build multiple applications reading from the same Amazon Kinesis stream (for example, to perform counting, aggregation, and filtering).

Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. Amazon SQS allows moving data between distributed application components and helps build applications in which messages are processed independently (with message-level ack/fail semantics), such as automated workflows.

Server-side encryption for Kinesis Streams automatically encrypts data using a user specified AWS KMS master key (CMK) before it is written to the stream storage layer, and decrypts the data after it is retrieved from storage. Encryption makes writes impossible and the payload and the partition key unreadable unless the user writing or reading from the stream has the permission to use the key selected for encryption on the stream. As a result, server-side encryption can make it easier to meet internal security and compliance requirements governing data.

With server-side encryption, client-side applications (producers and consumers) do not need to be aware of encryption, they do not need to manage CMKs or cryptographic operations, and data is encrypted when it is at rest and in motion within the Kinesis Streams service. All CMKs used by the server-side encryption feature are provided by the AWS KMS. AWS KMS makes it easy to use an AWS-managed CMK for Kinesis (a “one-click” encryption method), using AWS KMS generated CMK, or a CMK that has been imported for encryption.

Server-side encryption is available in US East (N. Virginia), US West (N. California and Oregon), EU (Ireland), Asia Pacific (Tokyo), and Asia Pacific (Singapore) regions. Support for server-side encryption in other regions is coming soon.

Leave a Reply

WWPI – Covering the best in IT since 1980