Deprecation of the Lake Formation ruled tables characteristic

After cautious consideration, we’ve made the choice to finish help for Ruled Tablesbeginning December 31, 2024, to concentrate on open supply transactional desk codecs reminiscent of Apache Iceberg, hudi apacheand Linux Basis Delta Lake. This choice stems from buyer choice for these open supply options, which provide ACID-compliant transactions, compaction, time journey, and different options beforehand offered by Ruled Tables. Amazon Internet Providers (AWS) has improved our help for these codecs in a number of analytics companies, together with Amazonian Athena, Amazon EMR (Elastic MapReduce), AWS Glueand Amazon redshiftwith options together with computerized compaction help for Apache Iceberg, snapshot retention and expiration and orphan file elimination for Apache Icebergenhancements to AWS Glue Knowledge Catalog CreateTable API to create Apache Iceberg tablesand help for AWS Glue Crawlers schema detection in Apache Iceberg, hudi apacheand Linux Basis Delta Lake. Clients can now use these open supply codecs to realize ACID-compliant transactions with Amazon Easy Storage Service (Amazon S3) backed by knowledge, benefiting from its wealthy options and vast compatibility.

After December 31, 2024, prospects will now not have the ability to create Ruled Desk transactions (lakeformation:StartTransaction), write to Ruled Tables (lakeformation:UpdateTableObjects), or question your ruled tables utilizing Amazon Athena. Clients will nonetheless have the ability to entry their board standing info by calling lakeformation:GetTableObjects and transaction info by calling lakeformation:ListTransactions till February 17, 2025. After February 17, 2025, all ruled desk APIs will begin to fail. Ruled desk metadata will live on throughout the AWS Glue knowledge catalog, and ruled desk knowledge will stay in your S3 buckets. No different desk varieties can be affected by this variation, together with Hive (Apache Parquet, CSV, ORC, and so on.), Iceberg, Hudi, and Delta Lake tables.

Migrating your ruled tables

Clients can migrate their ruled desk tables to one of many open supply codecs by copying their ruled desk knowledge on to Apache Iceberg utilizing Amazon Athena. Emigrate knowledge to Iceberg, you should utilize Amazon Athena CREATE TABLE AS (CTAS) assertion, as proven within the following code instance.

CREATE TABLE my_iceberg_table WITH ( 
    table_type="ICEBERG",
    is_external = false,
    location = 's3://mybucket/myicebergdata/'
) AS 
SELECT * FROM my_governed_table

You possibly can specify further properties on the desk degree, that are listed within the Amazon Athena Consumer Information. If you happen to specify partitions or buckets as a part of the Apache Iceberg desk definition, you might encounter the limitation of 100 partitions per bucket. On this case, see Use CTAS and INSERT INTO to bypass the 100 partition restrict.

If you happen to need assistance migrating your tables or have any questions, please contact us at [email protected].

Concerning the creator

Mert Hocanin is a Principal Huge Knowledge Architect at AWS Lake Formation.